RNA In Situ Hybridization: Probes, Detection, and Quantification

RNA in situ hybridization (RNA ISH) is a family of techniques that lets researchers see exactly where specific RNA molecules sit inside cells and tissues, without grinding those samples into a soup first. The core idea has stayed the same since the late 1960s: design a short strand of nucleic acid, called a probe, that binds to a target RNA sequence, then make that binding event visible under a microscope. What has changed dramatically is every layer of the process, from the chemistry of the probes themselves to the software that counts individual transcripts. Modern versions can detect a single RNA molecule in a preserved tissue slice, profile thousands of genes at once, and even track RNA movement in living cells.

A Brief Origin Story

The technique traces back to 1968, when researchers used radioactively labeled probes (tagged with tritium) and detected the resulting hybrids by autoradiography, a process that involved coating slides with photographic emulsion and waiting days or weeks for an image to develop. Those first successful experiments targeted the abundant ribosomal DNA in frog oocytes, followed quickly by repetitive satellite DNA in mouse and fruit fly chromosomes. Fluorescent probes arrived roughly a decade later, opening the door to the multicolor, high-resolution approaches used today.¹

How Probes Are Designed

The probe is the heart of any ISH experiment. It is a short stretch of nucleic acid whose sequence is complementary to the target RNA, so the two strands lock together when they meet. Early probes were made from DNA or RNA and labeled with radioactive isotopes. Current designs use synthetic DNA oligonucleotides, sometimes modified with locked nucleic acid (LNA) residues. LNA modifications lock the sugar ring of certain nucleotides into a rigid shape, which raises the melting temperature of the probe–RNA hybrid and dramatically boosts signal strength compared to standard DNA probes.² That extra stability is especially useful when the target RNA is short or expressed at low levels.

Probe length, sequence composition, and the number of probes tiled along the target all affect how well the experiment works. For single-molecule FISH (smFISH), the standard approach uses a set of roughly 20 to 48 short probes, each about 20 nucleotides long, that tile along the same transcript. When each probe carries one fluorescent dye, the cluster of probes bound to one mRNA molecule creates a bright, diffraction-limited spot that stands out against the background. For targets where you cannot fit many probes, such as microRNAs or short non-coding RNAs, modified chemistries like LNA become essential because each individual probe must contribute more signal on its own.

Amplifying the Signal

When a target RNA is expressed at very low levels, the handful of fluorescent dyes on a standard probe set may not produce enough light to distinguish from noise. Several amplification strategies have been developed to solve this.

RNAscope uses pairs of target-specific probes that must bind adjacent to one another before a branching amplification tree can assemble on top of them. That dual-binding requirement suppresses background because a single probe landing in the wrong spot cannot trigger the cascade. The result is single-molecule sensitivity even in formalin-fixed, paraffin-embedded tissue, the standard preservation method for clinical samples.³ Branched DNA technology more broadly works by loading several fluorescent molecules onto each transcript through a branching scaffold, raising the signal-to-noise ratio well beyond what a simple probe set achieves.⁴

Hybridization chain reaction (HCR) takes a different route. Short DNA hairpins are designed so that the binding of a probe to its RNA target triggers a chain reaction of hairpin openings, building a long, fluorescent polymer in place. A modified version using short hairpin DNAs has been shown to outperform tyramide signal amplification, a widely used enzyme-based method, for detecting certain low-abundance transcripts in neural tissue.⁵ Enzyme-based amplification using alkaline phosphatase or horseradish peroxidase also remains common, particularly in zebrafish and other model organisms where whole-mount staining is routine.⁶

Chromogenic Versus Fluorescent Detection

Fluorescence-based detection dominates the research literature, but chromogenic detection, where the signal shows up as a colored precipitate visible under a standard bright-field microscope, has distinct advantages in clinical pathology labs. Fluorescent signals fade over time, require specialized microscopes and darkened rooms, and can be confounded by tissue autofluorescence. Chromogenic signals are permanent, viewable with the same microscopes pathologists already use, and can be digitized much faster.

In head-to-head comparisons for HER2 gene testing in breast cancer, dual-color chromogenic ISH agreed with FISH results in all 168 cases tested in one European multicenter study, with slide scoring time reduced by about 28%.⁷ A separate study confirmed roughly 99% concordance between the two methods while finding that digital scanning of chromogenic slides was vastly faster.⁸ More recently, chromogenic adaptations of HCR have been developed to bring the sensitivity of that amplification method into bright-field workflows, which is particularly useful in tissues where autofluorescence is strong.⁹

Tissue Preparation and Taming Background Noise

Getting probes inside cells without destroying the RNA or the tissue architecture is one of the trickiest parts of ISH. Samples are typically fixed with formaldehyde to cross-link molecules in place, then permeabilized so probes can enter. The standard approach uses an enzyme called proteinase K to punch holes in cell membranes, but proteinase K can chew up proteins that a researcher also wants to detect. An alternative strategy developed for fruit fly ovaries skips the enzyme and instead uses xylenes and detergents to open up the tissue, preserving antigens for subsequent protein staining while still allowing RNA probes in.¹⁰

Background fluorescence is the perennial enemy. Every tissue has molecules, particularly lipids and certain proteins, that glow when excited by a laser, creating a haze that can drown out genuine probe signals. One creative approach embeds samples in a polyacrylamide gel, anchors the RNA to the gel matrix, then strips away proteins and lipids. This clearing step reduces both off-target probe binding and autofluorescence without measurable loss of RNA.¹¹ Another strategy, called split-FISH, uses a two-part probe design where both halves must bind the target before any signal is generated, cutting down on false positives even in uncleared tissues.¹²

Choosing the right fluorescent dye also matters. In coral tissue, for example, green fluorescent proteins native to the coral and chlorophyll from symbiotic algae produce intense background fluorescence across much of the visible spectrum. Researchers working with these samples typically select dyes like Cy3 or Cy5 whose emission spectra are more distinct from the background, and rely on confocal microscopes with spectral profiling to separate real signal from noise pixel by pixel.¹³

RNA Quality and Storage

RNA degrades. That is true in any molecular biology experiment, but in ISH it has a particularly insidious consequence because the signal simply fades or vanishes and you may not realize the absence is artifact rather than biology. In formalin-fixed, paraffin-embedded tissue blocks stored at room temperature, hybridization signals drop noticeably after about a year and fall off steeply by five years. Cutting unstained slides from fresh blocks and storing them at –20°C preserves signals substantially better than leaving the blocks at room temperature and cutting slides later when needed.¹⁴ For anyone building a tissue bank with plans for future ISH studies, this is a practical detail worth knowing early.

Counting Single Molecules

One of the most powerful features of modern RNA ISH is the ability to count individual mRNA molecules inside single cells. In the smFISH approach, each mRNA appears as a distinct fluorescent dot. In mammalian cells, where transcripts are relatively spread out, counting dots works well. In bacteria, where the cell is tiny and transcripts can cluster, a different strategy has been developed: rather than counting discrete spots, the total fluorescence intensity of each focus is measured and divided by the known brightness of one probe set, yielding an estimate of copy number even when molecules overlap.¹⁵

This kind of absolute quantification is valuable because most other methods for measuring gene expression, like RNA sequencing on bulk tissue, give you relative abundances averaged over millions of cells. Knowing the actual number of copies of a transcript in an individual cell tells you something fundamentally different about how noisy a gene’s regulation is and how much cell-to-cell variation exists in a population.

Multiplexed Imaging and Spatial Transcriptomics

Standard smFISH handles a few genes at a time, limited by the number of spectrally distinct fluorescent dyes you can image simultaneously. Multiplexed methods break through this ceiling by using combinatorial labeling and sequential rounds of hybridization. In MERFISH, each RNA species is assigned a unique barcode made up of a combination of fluorescent labels applied across multiple imaging rounds. Error-robust encoding, borrowing ideas from information theory, means that even if one round fails or misreads, the barcode can still be correctly identified.¹⁶

SeqFISH takes a similar barcoding approach and its upgraded version, seqFISH+, distributes pseudo-colors across multiple channels to push the number of detectable genes up to around 24,000, effectively covering the entire transcriptome in a single experiment.¹⁷ These methods are driving the field of spatial transcriptomics, which aims to map gene expression across entire tissue sections at single-cell or even subcellular resolution. Some tools prioritize high resolution for hundreds of genes; others offer genome-scale depth at somewhat lower spatial precision.¹⁸

In Situ Sequencing With Padlock Probes

A parallel approach skips barcoding by fluorescence entirely and instead reads out RNA identity through actual sequencing reactions performed directly on a tissue section. Padlock probes are short DNA circles that snap closed only when they find their specific target sequence on a cDNA copy of the RNA. Once circularized, the probe is copied in place by rolling circle amplification, creating a bright ball of repeated sequence that can be decoded with fluorescent sequencing chemistry.¹⁹ BaristaSeq optimized this padlock-probe approach for compatibility with standard Illumina sequencing reagents, making it more accessible to labs already equipped for next-generation sequencing.²⁰ One strength of padlock-based methods is that they can discriminate single-nucleotide differences, allowing in situ genotyping of point mutations and allele-specific expression in preserved tissue.²¹

Detecting MicroRNAs and Long Non-Coding RNAs

Small RNAs like microRNAs (miRNAs) present a special challenge. They are only about 22 nucleotides long, which means there is barely enough room for a single probe to bind. Standard DNA probes form weak hybrids with such short targets. LNA-modified probes solve this by binding with unusually high affinity. A widely used protocol for miRNA detection in clinical paraffin sections relies on double-digoxigenin-labeled LNA probes and can be completed in a single day, a practical consideration when the goal is diagnostic use.²²

Long non-coding RNAs (lncRNAs) are at the opposite end of the size spectrum but pose their own difficulties: many are expressed at low levels and confined to the nucleus, making them hard to detect with conventional probe sets. LNA-based ISH has been adapted for lncRNA detection in three-dimensional cancer cell spheroids, which better mimic tumor architecture than flat cell cultures.²³

Seeing RNA and Protein at the Same Time

Gene expression does not stop at transcription. Knowing where an mRNA sits is useful, but knowing whether its protein product is also present, and in what proportion, tells a more complete story. Combined immunofluorescence and smFISH protocols let you stain for a protein and hybridize for its mRNA in the same cell. The key technical challenge is that the antibody staining and the FISH steps can interfere with each other: RNase contamination from antibody solutions can destroy target RNA, while the harsh washes in FISH protocols can strip antibodies off their targets.

Protocols that perform immunofluorescence first, followed by FISH, have been optimized to detect both RNA and protein simultaneously in single mammalian cells.²⁴ This approach has been extended to challenging sample types, including mouse preimplantation embryos, where both lncRNAs and transcription factor proteins were successfully visualized.²⁵ In plants, whole-mount smFISH combined with fluorescent protein reporters has enabled joint RNA and protein quantification at cellular and subcellular resolution.²⁶

Clinical Uses in Cancer Diagnostics

RNA ISH has moved well beyond the research bench and into clinical pathology, particularly for viral detection in cancer. In oropharyngeal squamous cell carcinoma, chromogenic RNAscope targeting the E6/E7 mRNA of high-risk HPV types showed 96.4% concordance with p16 immunohistochemistry, the current standard surrogate marker. It was also more sensitive than DNA ISH, detecting virus in cases that DNA-based probes missed.²⁷ An independent study on an automated clinical platform confirmed that RNA ISH achieved 100% sensitivity for HPV detection versus 88% for DNA ISH, with overall concordance of 93% against PCR results.²⁸

RNA ISH also detects Merkel cell polyomavirus transcripts in Merkel cell carcinoma, a rare but aggressive skin cancer. Because virus-positive and virus-negative Merkel cell carcinomas differ in prognosis and may respond differently to therapy, reliable detection of viral RNA in tissue sections has both diagnostic and prognostic value.²⁹

Computational Tools for Spot Detection

Generating images is only half the battle. Extracting meaningful data from those images, reliably identifying every fluorescent spot and distinguishing real signal from noise, requires dedicated software. Several tools have been developed and benchmarked against one another. RS-FISH provides precise and interactive spot detection, running roughly four to seven times faster than established methods like FISH-quant while maintaining comparable accuracy across datasets with varying noise levels.³⁰ TrueSpot uses three-dimensional spot detection algorithms that substantially outperform approaches working on two-dimensional image slices, which matters for thick tissue sections where RNA molecules sit at different depths.³¹ More recently, deep-learning methods like U-FISH have been developed to enhance images before spot detection, providing more consistent results across the wide range of image qualities that arise in spatial-omics experiments.³²

Adapting Protocols Across Species

An ISH protocol optimized for mouse brain tissue will not work out of the box on a plant root tip or a coral polyp. Plant cells have rigid cell walls made of cellulose and other polysaccharides that block probe entry. A whole-mount approach for plants uses a cocktail of cell-wall-digesting enzymes, carefully balanced to open up the tissue while preserving its three-dimensional structure, so that gene expression patterns can be visualized deep within intact organs.³³ The commercially available RNAscope platform, designed primarily for animal tissue, has been modified for plant samples, but the standard protocol is insufficient without optimization of fixation, permeabilization, and hybridization conditions specific to plant morphology.³⁴

Expansion Microscopy Meets FISH

Conventional light microscopes cannot resolve structures smaller than about 200 nanometers, a limit set by the physics of light diffraction. Super-resolution microscopy can beat this limit, but the instruments are expensive and technically demanding. Expansion microscopy sidesteps the problem entirely by physically swelling the sample. Tissues are embedded in a swellable hydrogel, and when the gel is expanded, the molecules within it spread apart, effectively magnifying them so that a regular confocal microscope can resolve what previously required specialized optics.³⁵

ExFISH combines this expansion approach with RNA FISH. RNAs are chemically anchored to the gel matrix before expansion, so their positions are preserved as the sample swells. The technique has been demonstrated in intact brain tissue and other thick specimens, enabling nanoscale imaging of RNA structure and location on standard diffraction-limited microscopes.³⁶ For labs that cannot justify the cost of a dedicated super-resolution system, expansion FISH offers a practical alternative for high-resolution RNA mapping.

Watching RNA Move in Living Cells

All the methods discussed so far work on fixed, dead tissue. But RNA molecules are not static. They are transcribed, transported, translated, and degraded, often in a matter of minutes. Capturing that dynamic behavior requires live-cell imaging. Fluorescent RNA aptamers are one emerging approach: an RNA sequence is engineered to fold into a shape that binds a small fluorogenic dye and lights up, functioning like a fluorescent protein tag but made entirely of RNA. Recent protocols enable real-time monitoring of RNA localization and dynamics in living cells using these bright, stable aptamer–dye systems.³⁷

Molecular beacons offer another route. These are hairpin-shaped probes that open and fluoresce only when they bind their target sequence. A beacon-based approach has been shown to detect engineered RNA transcripts containing as few as eight repeats of a target sequence with nearly 90% accuracy, using conventional wide-field fluorescence microscopy.³⁸ The trade-off with live-cell methods is that they typically require some genetic engineering of the target RNA or the cell itself, which limits their applicability to model systems rather than clinical specimens. Still, the ability to watch transcription happen in real time offers insights into gene regulation that no amount of fixed-tissue staining can provide.