Cell segmentation is the process of identifying and outlining individual cells in microscopy images so that each cell can be measured, tracked, and analyzed separately. It is a foundational step in nearly every computational workflow in biology and medicine, from counting tumor cells in a biopsy to screening thousands of drug candidates at once. Without it, a microscopy image is just a picture; with it, the image becomes quantitative data. The challenge is that cells crowd together, overlap, and vary wildly in shape, which has made accurate, automated segmentation one of the most actively researched problems in bioimage analysis for decades.
What the Process Actually Does
When researchers image living tissue or cultured cells under a microscope, the result is a dense field of objects. Some cells touch. Some overlap. Some barely contrast against the background. Cell segmentation algorithms draw a boundary around each individual cell, effectively converting an image into a labeled map where every pixel is assigned to a specific cell or to the background. Once that map exists, researchers can extract measurements for each cell: its size, shape, position, how it moves over time, what proteins it expresses, and how it interacts with neighbors.
This is why segmentation sits at the very beginning of so many analysis pipelines. The quality of everything that comes after, whether you are tracking a single cell’s movement, classifying cellular phenotypes, or mapping gene expression in a tissue section, depends directly on how accurately each cell was outlined in the first place.1PubMed Central. Evaluation of cell segmentation methods without reference segmentations A segmentation algorithm that consistently clips off a sliver of cytoplasm, or that merges two adjacent cells into one, introduces systematic errors that ripple through every downstream number.
From Thresholding to Deep Learning
Early approaches to cell segmentation relied on classical image-processing techniques. Thresholding, one of the simplest, separates cells from the background by picking a brightness cutoff: anything above a certain intensity is “cell,” everything below is “not cell.” Watershed algorithms treat the image like a topographic landscape and find the ridgelines between intensity valleys to split touching cells apart. Active contour models, sometimes called “snakes,” iteratively shrink a flexible boundary around each cell until it hugs the cell’s edge. These traditional methods, along with clustering algorithms, remain useful in certain contexts but tend to struggle with crowded fields or low-contrast images.2IET Image Processing. A Systematic Review on Cell Nucleus Instance Segmentation
Deep learning has reshaped the field. Convolutional neural networks trained on thousands of annotated cell images learn to recognize cell boundaries even in situations that stump classical algorithms. The U-Net architecture, introduced in 2015, became the workhorse of biomedical image segmentation. Its encoder-decoder design compresses an image down to essential features and then reconstructs a pixel-level segmentation map. Variants followed quickly. UNet++ added nested connections between the encoder and decoder pathways, improving segmentation accuracy across tasks including nuclei segmentation in microscopy images.3PubMed Central. UNet++: A Nested U-Net Architecture for Medical Image Segmentation Dense-UNet deepened the network and reused features across layers to handle complex in vivo cellular images.4PubMed Central. Dense-UNet: a novel multiphoton in vivo cellular image segmentation model based on a convolutional neural network Specialized versions like C-UNet were built to handle particularly tricky cases like overlapping, blurry cervical cells in Pap smear images.5PLoS ONE. Cervical cell’s nucleus segmentation through an improved UNet architecture
A major milestone came with Cellpose, a generalist deep learning method trained on over 70,000 segmented objects from highly varied image types. Unlike task-specific models, Cellpose can segment cells across a wide range of microscopy modalities without requiring the user to retrain the model or adjust parameters for each new experiment.6Nature Methods. Cellpose: a generalist algorithm for cellular segmentation That kind of out-of-the-box usability matters enormously for biologists who are not machine-learning specialists.
Why It Matters for Drug Discovery
High-content screening is one of the places where cell segmentation earns its keep most visibly. In drug discovery, researchers expose millions of cell cultures to different chemical compounds and then image the results to see which compounds changed the cells’ behavior. The images from these screens need to be analyzed automatically because no human could manually inspect that volume of data. Speed and robustness are paramount, and accuracy becomes a statistical question rather than a perfection question: you need to get cell outlines right often enough across millions of images for the aggregate measurements to be reliable.7PubMed. A fast, fully automated cell segmentation algorithm for high-throughput and high-content screening
Recent advances in artificial intelligence have made these screens more powerful. Deep learning has improved not just the segmentation step but also the downstream tasks of feature extraction and phenotypic profiling, enabling researchers to detect subtle drug-induced changes in cell morphology that older image-analysis pipelines would miss.8PubMed Central. Artificial Intelligence-Powered High-Content Analysis: Methodologies and Applications in Bioactive Compound Discovery from Natural Sources If the segmentation is sloppy, though, those subtle signals drown in noise. A compound that causes cells to round up slightly will look like nothing happened if the algorithm keeps merging rounded cells with their flat neighbors.
Reading Cancer Through Cell Boundaries
In pathology, segmentation has become central to understanding tumors. When a pathologist examines a tissue biopsy stained with standard dyes, the slide contains thousands to millions of individual cell nuclei. Manually categorizing each one as tumor cell, immune cell, stromal cell, or red blood cell is impractical. Computational tools do this automatically by first segmenting every nucleus, then classifying each one based on its visual features.
HD-Yolo, a deep learning method designed for whole-slide image analysis, segments and classifies nuclei across lung, liver, and breast cancer tissues. In breast cancer specifically, the cell-level features it extracted turned out to be more prognostically useful than standard immunohistochemistry markers for estrogen and progesterone receptor status, which are among the most established prognostic tools in breast oncology.9PubMed Central. A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization That finding underscores something important: segmentation is not just a preprocessing chore. When done well, it unlocks information that is genuinely invisible to the human eye.
Other tools in this space focus on mapping the tumor microenvironment, the ecosystem of cell types surrounding a tumor. One approach segments nuclei in standard tissue slides and then extracts dozens of features describing how those cell types are spatially arranged, capturing patterns of immune infiltration and stromal organization that correlate with patient outcomes.10PubMed Central. Computational Staining of Pathology Images to Study the Tumor Microenvironment in Lung Cancer Whole-cell segmentation pipelines, which outline the entire cell body rather than just the nucleus, are pushing this even further by capturing cytoplasmic features that nucleus-only methods miss.11Laboratory Investigation. Cell Segmentation With Globally Optimized Boundaries (CSGO)
Tracking Cells Over Time
Segmentation becomes even more powerful when applied to time-lapse imaging, where the same cells are photographed every few minutes over days or weeks. If you can segment cells in every frame, you can track them: following each cell’s movement, growth, division, and differentiation over time. This is essential for studying how stem cells decide their fate, how embryos develop, and how cell populations respond to stimuli.
The LEVER system, for example, captures phase-contrast images of cultured neural stem cells at five-minute intervals over periods of five to fifteen days, then automatically segments, tracks, and builds lineage trees showing how each cell divided and what its daughters became.12Nature Protocols. Vertebrate neural stem cell segmentation, tracking and lineaging with validation and editing Cell-ACDC, a more recent tool, extends this concept to multiple organisms including yeast and blood stem cells, automatically pairing daughter cells after division and linking them back to their mother cell for pedigree analysis.13PubMed Central. Segmentation, tracking and cell cycle analysis of live-cell imaging data with Cell-ACDC
The accuracy demands here are relentless. If the segmentation algorithm loses track of a cell for even a single frame, the lineage tree breaks. A missed division event can make one cell line appear to have stalled when it was actually proliferating. This is one reason why time-lapse studies often require human validation on top of automated segmentation, which is time-consuming but currently unavoidable for critical experiments.
Spatial Transcriptomics and the Assignment Problem
An emerging field called spatial transcriptomics measures which genes are active in individual cells while preserving information about where those cells sit within a tissue. The technologies work by detecting individual RNA molecules scattered across a tissue section, then assigning each molecule to a specific cell. That assignment step depends entirely on cell segmentation: you need to know where one cell ends and the next begins in order to decide which transcripts belong to which cell.14Nucleic Acids Research. Transforming subcellular spatial transcriptomics: deep learning models for cell segmentation
When segmentation errors misplace a cell boundary, transcripts from one cell get attributed to its neighbor. The consequences can be dramatic. A recent analysis of data from multiple tissue types and platforms found that segmentation errors confound most downstream analyses of cellular state, including differential gene expression, modeling of how neighboring cells influence each other, and detection of ligand-receptor interactions. In many cases, mis-segmented molecules dominated the list of top hits, meaning the results were being driven more by boundary errors than by real biology.15bioRxiv. Impact of Segmentation Errors in Analysis of Spatial Transcriptomics Data
How Errors Cascade
The spatial transcriptomics example points to a broader problem. Segmentation is a first step, and errors in a first step have outsized consequences. In highly multiplexed tissue imaging, where dozens of proteins are measured simultaneously across the same tissue section, researchers found that segmentation errors decrease the accuracy of cell phenotyping and lead to specific, predictable types of misclassification.16PLOS Computational Biology. Effects of segmentation errors on downstream-analysis in highly-multiplexed tissue imaging
Roughly a fifth of cells can end up assigned to the wrong phenotype. Some of these errors are mild: a helper T cell gets called a regulatory T cell, a close relative. But others are severe: an immune cell gets classified as a completely different lineage, like an epithelial cell. Those lineage-level misclassifications can fundamentally distort the picture of a tissue’s composition. If you are trying to measure how densely immune cells infiltrate a tumor, and a fraction of those immune cells are being misidentified as tumor cells, your entire analysis of the immune response is corrupted.
This cascading effect is one reason the field has invested so heavily in benchmarking. Worldwide competitions have pushed segmentation accuracy upward over the past decade, and large annotated datasets are now freely available for training and testing algorithms. Even so, no single method works well in every context, and no universally accepted benchmarking platform exists yet.17PubMed Central. Automated Cell Segmentation for Quantitative Phase Microscopy
The Software People Actually Use
Two open-source platforms dominate everyday bioimage analysis. ImageJ (and its plugin-rich distribution Fiji) is cited in over 10,000 publications per year and is especially suited for interactive, single-image work. CellProfiler, with over 10,000 publications to date, is built for automation: users construct pipelines of modular processing steps that can run unattended across thousands of images in a large experiment.18PubMed Central. ImageJ and CellProfiler: Complements in Open Source Bioimage Analysis The two are complementary rather than competing. ImageJ is the tool you reach for when you want to poke at one image and understand what you are looking at. CellProfiler is the tool you reach for when you have figured out the processing steps and need to apply them to 50,000 images overnight.
More recently, platforms like Arkitekt are bridging the gap between image analysis software and the microscope hardware itself. Arkitekt allows researchers to run segmentation algorithms in real time as images are being acquired, enabling what is sometimes called “smart microscopy,” where the microscope can react to what it sees. For example, if the segmentation algorithm detects a rare cell type, the microscope can automatically zoom in and capture a higher-resolution image.19Nature Methods. Arkitekt: streaming analysis and real-time workflows for microscopy
Foundation Models and Generalization
The newest frontier in cell segmentation borrows an idea from the large language model world: foundation models. Meta’s Segment Anything Model (SAM), originally trained to segment arbitrary objects in natural photographs, has been adapted for biomedical images. The appeal is obvious. A model that can segment any cell in any image type, without needing retraining for each new experiment, would save enormous amounts of effort.
The reality is more nuanced. When evaluated on digital pathology images in a zero-shot setting (meaning no task-specific training), SAM performed well on large connected structures but struggled with dense fields of small objects like individual cell nuclei, even when given many manual prompts per image.20PubMed Central. Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging Dense instance segmentation, where the goal is to individually outline hundreds of tightly packed nuclei in a single field of view, remains a harder problem than segmenting a single large region.
CellSAM, a purpose-built foundation model for cell segmentation, addresses some of these limitations. It achieves adequate zero-shot performance on many cell lines and can be improved with remarkably little additional data, sometimes just ten extra fields of view containing a few hundred to a thousand cells. However, it still falls short on cell morphologies that are far from its training distribution, highlighting a persistent limitation of even the most general models.21Nature Methods. CellSAM: a foundation model for cell segmentation SAMCell takes yet another approach, training a modified SAM on a large-scale microscopy dataset and demonstrating generalization to cell types and microscopes not seen during training.22PLOS ONE. SAMCell: Generalized label-free biological cell segmentation with segment anything
The pattern across all these models is consistent: generalization is improving but remains imperfect, and the hardest cases tend to involve cells that are unusually dense, unusually shaped, or imaged with an unfamiliar modality. For now, the practical advice is that foundation models are a reasonable starting point for many experiments, but researchers working with unusual cell types or demanding quantitative workflows should still expect to fine-tune or validate.
Moving Into Three Dimensions
Most segmentation work has historically been done on two-dimensional images: a single slice through a sample. But cells are three-dimensional objects, and modern microscopy techniques like confocal and light-sheet imaging produce full 3D volumes. Segmenting cells in 3D is harder because the datasets are much larger, the boundaries between cells are more complex, and the computational cost rises steeply.
One practical approach segments cells in each 2D slice of a 3D volume using existing, well-validated 2D methods, then stitches those 2D outlines into coherent 3D objects using a consensus algorithm. This strategy achieved near-perfect reconstruction accuracy on plant tissue datasets and strong performance on vascular and other tissue types.23Nature Methods. Universal consensus 3D segmentation of cells from 2D segmented stacks The advantage is that it leverages existing 2D tools rather than requiring entirely new 3D architectures.
Direct 3D segmentation methods also exist. One approach developed for high-content screening first creates a 3D nuclear mask, then applies an iterative 3D watershed algorithm on downscaled images to separate touching nuclei, and finally refines the boundaries at full resolution. Tested on a set of images containing over 2,300 nuclei, it achieved high detection precision and was more than twice as fast as the most accurate competing method.24PubMed Central. Efficient automatic 3D segmentation of cell nuclei for high-content screening Speed matters here because 3D volumes are enormous; a method that is accurate but takes hours per image is not practical for screening workflows that produce thousands of volumes per day.
Label-Free Imaging and the Contrast Problem
Many of the most successful segmentation methods rely on fluorescent labels: molecular tags that make cells or their nuclei glow brightly, providing high contrast against the background. But fluorescent labeling is not always possible or desirable. It can be toxic to living cells over time, it limits the number of things you can visualize simultaneously, and some experiments require observing cells in their natural, unstained state.
Label-free microscopy techniques like phase contrast, differential interference contrast, and quantitative phase imaging produce images where cells are visible but with much subtler contrast. Segmenting cells in these images is harder because the boundaries are less obvious to both humans and algorithms. A comprehensive comparison of segmentation methods across four label-free microscopy modalities found that method performance varies substantially depending on the imaging technique, and no single method excelled across all of them.25PubMed Central. Cell segmentation methods for label-free contrast microscopy: review and comprehensive comparison Researchers working with label-free data generally need to test multiple algorithms and validate against manual outlines before trusting their pipeline.
Quantitative phase microscopy is a particularly interesting case because it measures the optical thickness of cells, producing images that look like topographic maps. Segmentation algorithms designed specifically for this modality can automatically detect and outline cells even at high densities where individual boundaries are difficult to see, but the problem remains challenging and active.17PubMed Central. Automated Cell Segmentation for Quantitative Phase Microscopy The gap between what is achievable with fluorescent labels and what is achievable without them has narrowed thanks to deep learning, but it has not closed.
Reproducibility and the Human Factor
One underappreciated reason cell segmentation matters is reproducibility. When two labs analyze similar experiments using different segmentation settings, or when one lab manually outlines cells while another uses an automated method, the downstream results can diverge even if the underlying biology is identical. Automated, well-documented segmentation pipelines help standardize analysis across labs and across time. Researchers have specifically called for automated segmentation approaches as a way to improve reproducibility in bioimage analysis, noting that the subjectivity inherent in manual segmentation is a significant source of variability.26Synthetic Biology. Automated cell segmentation for reproducibility in bioimage analysis
At the same time, evaluating segmentation quality is itself surprisingly difficult. Most evaluation methods require a “ground truth,” a set of reference segmentations produced by expert humans, against which the algorithm is scored. But producing reference segmentations is laborious and introduces its own subjectivity. Methods have been developed to assess segmentation quality without reference segmentations, using internal consistency metrics instead, which makes it easier to catch problems in routine analysis without the overhead of creating gold-standard annotations for every experiment.1PubMed Central. Evaluation of cell segmentation methods without reference segmentations Still, the field is in a somewhat awkward position: the tool used to make biology quantitative is itself difficult to quantify objectively.