Cellpose GitHub: How to Install and Use The Tool

Cellpose is a free, open-source deep-learning tool for segmenting cells in microscopy images, and installing it takes a single pip command. Hosted on GitHub under the MouseLand organization, it ships with pre-trained models that handle fluorescence, brightfield, and even non-microscopy images out of the box. What makes the tool popular is that it works across a genuinely wide range of image types without manual tuning, but getting the most out of it means understanding which model to pick, how to set the one parameter that matters most, and when to train your own.

Installing Cellpose

The fastest path is a standard pip install. In a terminal or a Jupyter notebook cell, running pip install cellpose pulls down the package along with its core dependencies.¹ That alone gives you a working CPU-based installation. You can immediately import the library and start segmenting images.

If you want GPU acceleration, there is one extra step that trips people up: PyTorch needs to be installed separately, and the installation command varies depending on your operating system, whether you are using pip or conda, and which version of CUDA your GPU supports. The PyTorch website has an interactive selector that generates the correct command for your setup.¹ Getting this wrong is the single most common installation headache: Cellpose will still run on your CPU if PyTorch is not configured for your GPU, but it will be dramatically slower, especially on large images or batch jobs.

A few practical notes on the install. Using a virtual environment or a conda environment is strongly recommended so Cellpose’s dependencies do not collide with other Python packages you have installed. If you are on a Mac with an Apple Silicon chip, GPU support is available through PyTorch’s MPS backend, though CUDA on an NVIDIA card remains the most mature and fastest option. And if you want the graphical interface, run pip install cellpose[gui] instead, which bundles the additional packages needed to launch it.

Picking the Right Pre-trained Model

Cellpose ships with over a dozen pre-trained models, and choosing the right one is the highest-leverage decision you will make before running segmentation. All models share the same underlying neural network architecture but differ in the images they were trained on.²

The default model, called cyto, was trained on roughly 70,000 cells drawn from six different image types, including fluorescence images of well-separated neuroblastoma cells, brightfield microscopy, and even non-microscopy images like apples and jellyfish. On neuroblastoma images, it achieves a mean average precision around 0.9. The diversity of training data gives it decent generalization: when applied to fluorescence-based TissueNet images or phase-contrast LiveCell images it was never trained on, it still scores around 0.5 and 0.4, respectively.² That sounds modest, but for a model encountering entirely new image types, it is a reasonable starting point.

A second general-purpose model, cyto2, was trained on the same data as cyto plus user-contributed images, so it tends to perform a bit better across the board. Cellpose 2.0 added models fine-tuned specifically on TissueNet and LiveCell data, boosting performance on those datasets to around 0.75 and 0.7.² There are also subset-specific models (TN1 through TN3 for TissueNet subsets, LC1 through LC4 for LiveCell subsets) that target particular tissue or cell types within those larger collections.

In practice, if you are new to Cellpose, start with cyto2 for cytoplasm segmentation or the nuclei model for nuclear segmentation. If your images come from tissue sections or dense cell cultures, the TissueNet or LiveCell fine-tuned models are worth trying. If none of the pre-trained models work well, that is a signal to fine-tune, which the tool makes straightforward.

Using the Graphical Interface

Cellpose includes a GUI that lets you run segmentation without writing any code. You can launch it from the command line after installing the GUI dependencies, load an image, select a model, and hit a button to generate segmentation masks overlaid directly on your image.³ The interface lets you zoom, pan, and inspect individual masks interactively.

The GUI serves two main roles: running Cellpose on new images out of the box and manually annotating images to create training data for custom models.³ The annotation workflow is particularly useful because it feeds directly into the fine-tuning pipeline. You correct the mistakes of a pre-trained model, save the corrected masks, and use them as ground truth for retraining.

One parameter in the GUI deserves special attention: cell diameter. This is the single most important knob you will turn. Cellpose rescales images internally so that cells match the diameter the model was trained on, so if your estimate is too large or too small, segmentation quality drops fast. The GUI includes a size-calibration tool that estimates diameter automatically, and it overlays a red disk on the image so you can visually check whether the estimate matches your cells.³ If the automatic estimate looks off, you can type in a value directly. Getting this right often matters more than which model you choose.

Scripting with the Python API

For batch processing or integration into analysis pipelines, the Python API is the way to go. A minimal script loads a model, passes an image array, and gets back a mask array where each cell is labeled with a unique integer. The basic import looks like this: you pull in models from cellpose, create a Cellpose object specifying whether to use GPU and which model type you want, and then call the model’s eval method on your image.¹

The eval call returns three things: the segmented masks, the predicted flow fields that the model uses internally, and a style vector that captures the overall appearance of the image. For most users, only the masks matter. You can pass the diameter parameter directly in the eval call, or set it to None to let Cellpose estimate it automatically. Other useful parameters include channels, which tells the model which image channel contains cell bodies and which contains nuclei (if applicable), and flow_threshold, which controls how aggressively the model merges nearby regions.

Because everything runs in Python, you can wrap segmentation in a loop over a folder of images, feed the output masks into downstream analysis tools like scikit-image or pandas, or export them as labeled TIFFs. The Cellpose io module has helper functions for reading and writing common microscopy formats.

How the Segmentation Actually Works

You do not need to understand the internals to use Cellpose, but knowing the basic idea helps you troubleshoot when things go wrong. Rather than trying to draw cell boundaries directly, Cellpose predicts a set of “flow fields” for every pixel in the image. Think of it like placing a heat source at the center of each cell: the resulting gradient tells every pixel inside that cell which direction to move to reach the center. When you follow those gradients from every pixel, all the pixels belonging to the same cell converge to the same point. The model then groups pixels by which center they converge to, producing the final masks.³

This flow-based approach is why Cellpose handles odd cell shapes well. For a round cell, every pixel points roughly toward the center. For a long, curved cell with protrusions, pixels at the extremes first point toward intermediate pixels inside the cell body, which in turn point at the center. The network also predicts a probability map classifying each pixel as inside or outside a cell, which helps sharpen boundaries.³

The neural network itself is a modified U-Net with residual blocks and a “style” vector that captures the overall look of each image. That style vector lets the network adjust its processing depending on whether it is looking at fluorescence microscopy, brightfield, or something else entirely. This is part of why a single model can handle such diverse inputs without separate retraining for each imaging modality.

Training a Custom Model

When pre-trained models do not segment your images well enough, Cellpose 2.0 introduced a streamlined workflow for training your own model. The core idea is a human-in-the-loop approach: you run a pre-trained model on one of your images, correct the mistakes in the GUI, retrain a new model using that corrected image as ground truth, then repeat on a second image.⁴

The surprising finding is how little data this takes. In the Cellpose 2.0 paper, the developers found that correcting just three to five images was generally enough to produce a well-performing specialist model. Retraining uses a large learning rate and runs for only 100 epochs by default, finishing in under a minute on a GPU.⁴ In broader benchmarks, models trained on 500 to 1,000 manually segmented regions of interest performed nearly as well as models trained on entire datasets containing up to 200,000 regions, and the human-in-the-loop approach pushed the required annotations down to 100 to 200 regions while maintaining comparable accuracy.⁴

For the Python API route, training a custom model involves pointing Cellpose at a folder of images and their corresponding mask files, specifying which pre-trained model to start from, and letting it fine-tune. The key parameters are the number of epochs, the learning rate, and whether to use data augmentation. Starting from a pre-trained model rather than training from scratch is almost always the right call, because the pre-trained weights already encode general knowledge about what cells look like.

What Cellpose 3.0 Added

Cellpose 3.0 focused on a problem that earlier versions struggled with: noisy, blurry, or undersampled images. Rather than training users to pre-process their images with separate denoising tools, Cellpose 3.0 built image restoration directly into the segmentation pipeline. The developers trained the restoration module not to produce the prettiest possible pixel values, but to output images that a generalist segmentation model segments well, which is a subtle but important distinction.⁵

In practice, this means you can feed a low-quality image into Cellpose 3.0 and get substantially better segmentation results without manually tweaking denoising parameters first. The restoration runs as a one-click step before segmentation. If you have been struggling with dim fluorescence images, images taken at fast acquisition speeds, or other noisy datasets, upgrading to Cellpose 3.0 and enabling the restoration step is worth trying before investing time in custom model training.

Running Cellpose Without a Local Install

If installing Python packages locally is not an option, or if you want to test Cellpose before committing to a full setup, cloud-based notebook environments offer an alternative. Google Colab, for example, provides free access to GPU-equipped virtual machines where you can run pip install commands directly in a notebook cell. Some community tools built around Cellpose provide one-click Colab notebooks that handle all the dependency setup automatically, meaning you only need a Google account and an internet connection to get started.⁶

The tradeoff is that Colab sessions time out after a period of inactivity, and the free tier’s GPU access is limited. For a quick test or for processing a handful of images, this works well. For large-scale batch processing or iterative model training, a local or institutional GPU setup is more practical.

Integrating Cellpose into Larger Workflows

Cellpose rarely lives in isolation. Most researchers use it as one step in a pipeline that includes image pre-processing, segmentation, feature extraction, and statistical analysis. The tool’s Python API makes it straightforward to chain with libraries like scikit-image for morphological measurements, pandas for tabular data handling, and matplotlib or napari for visualization.

For researchers working in digital pathology, Cellpose models can also be integrated into QuPath, a popular open-source platform for whole-slide image analysis. This combination supports workflows where Cellpose handles the cell detection step and QuPath manages the spatial analysis, annotation, and quantification around it.⁷ The bridge between the two typically runs through exported mask files or through extensions that call Cellpose from within QuPath’s scripting environment.

Another common integration is with Fiji/ImageJ via the Cellpose wrapper plugin, which lets ImageJ users run Cellpose models without leaving their familiar interface. The plugin sends images to a local or remote Python environment running Cellpose and returns the masks to ImageJ for further analysis. This is popular among biologists who are comfortable with ImageJ but have not set up a Python workflow.

Common Pitfalls and How to Avoid Them

The most frequent mistake new users make is ignoring cell diameter. If you leave it at a default value that does not match your cells, the results will be poor regardless of which model you choose. Spend thirty seconds verifying the diameter estimate before running segmentation on an entire dataset.

A second common issue is channel assignment. Cellpose expects you to specify which channel contains the cell body signal (cytoplasm or membrane) and, optionally, which contains nuclei. If your image is grayscale, you set channels to [0, 0]. If you have a two-channel image with cytoplasm in green and nuclei in blue, you set channels to [2, 3]. Getting these numbers wrong is easy and produces confusing results, because the model will try to segment whatever signal you point it at.

Third, people sometimes expect a single pre-trained model to work perfectly on every image type. Cellpose’s generalist models are impressive, but “generalist” does not mean “omniscient.” If your cells are unusually shaped, densely packed, or imaged with unusual contrast, the default models may need help. The fine-tuning workflow exists precisely for this scenario, and as noted earlier, correcting a small handful of images is often enough to get a specialist model that dramatically outperforms the generic one on your data.

Finally, watch out for memory limits when processing very large images. Whole-slide images or high-resolution tiled acquisitions can exceed your GPU’s memory. The standard workaround is to tile the image into overlapping patches, run Cellpose on each patch, and stitch the masks back together. Cellpose’s io module and community scripts offer utilities for this, but it requires some setup. If you find yourself consistently running out of memory, processing on a machine with more GPU RAM or tiling more aggressively are the two main levers.

Cellpose Versus Other Segmentation Tools

Cellpose is not the only deep-learning segmentation tool available. StarDist, for example, excels at segmenting convex, roughly circular objects like nuclei, but struggles more with elongated or irregular cell shapes. Cellpose’s flow-based representation handles those shapes more gracefully, which is one reason it became the go-to tool for cytoplasm segmentation. Mesmer, part of the DeepCell ecosystem, targets tissue-level multiplexed imaging and was trained on the TissueNet dataset. If your images happen to match TissueNet’s distribution, Mesmer can be competitive or better, but its generalization to other image types is narrower.

The practical advice here is not to pick one tool and commit to it unconditionally. Try Cellpose’s pre-trained models first because they cover the broadest range of image types. If performance is not satisfactory, fine-tune. If you are working specifically with round nuclei and want speed, StarDist is worth benchmarking. If you are doing multiplexed tissue imaging, test DeepCell alongside Cellpose’s TissueNet-tuned models. The segmentation field moves fast, and the best tool for your images this year may not be the best tool next year.

1
STAR Protocols. Image analysis in Python for confocal microscopy
2
PubMed Central. Optimizing deep learning-based segmentation of densely packed cells using cell surface markers
3
Nature Methods. Cellpose: a generalist algorithm for cellular segmentation
4
PubMed Central. Cellpose 2.0: how to train your own model
5
PubMed Central. Cellpose3: one-click image restoration for improved cellular segmentation
6
F1000Research. BISCUIT: An Open-Source Platform for Visual Comparison of Segmentation Models in Bioimage Analysis
7
Communications Medicine. Evaluating cell AI foundation models in kidney pathology with human-in-the-loop enrichment