A protein language model is a type of artificial intelligence that learns the patterns governing protein sequences in much the same way a large language model like GPT learns the patterns of human language. Instead of training on sentences made of words, it trains on millions of protein sequences made of amino acids, picking up on the hidden rules that dictate how proteins fold, function, and evolve. The analogy runs surprisingly deep: researchers have shown that these models capture aspects of what they call “the grammar of the language of life” as encoded in protein sequences.1PubMed. Are protein language models the new universal key? That grammar turns out to encode far more than just sequence statistics, including three-dimensional structure, evolutionary relationships, and clues about which mutations will be harmful.
Proteins as a Language
Every protein in nature is a chain of smaller building blocks called amino acids. There are twenty standard amino acids, and you can think of them as a twenty-letter alphabet. A given protein might be a few dozen letters long or several thousand. The order of those letters determines how the chain folds into a three-dimensional shape, and that shape determines what the protein does in the body. Just as a sentence in English follows grammatical rules that make it meaningful rather than random, a protein sequence follows biochemical and evolutionary “rules” that make it fold into a functional shape rather than a useless tangle. Protein language models learn those rules from data alone, without being told what the rules are.
The parallel to natural language is not just a metaphor. In both cases, meaning depends on context. A word can mean different things depending on the sentence it appears in. Similarly, an amino acid’s contribution to protein function depends on which other amino acids surround it, sometimes at positions hundreds of letters away in the sequence. Capturing those long-range dependencies is exactly what transformer-based models were designed to do, which is why the same neural network architecture behind chatbots has turned out to be powerful for proteins.
What They Train On
The raw material for training a protein language model is a database of known protein sequences. The most widely used source is UniRef, a family of databases built from UniProt that clusters sequences at different levels of similarity to reduce redundancy. UniRef100 contains every unique sequence, while UniRef90 and UniRef50 cluster sequences at 90% and 50% identity, shrinking the dataset by roughly 40% and 70% respectively.2PubMed. UniRef: comprehensive and non-redundant UniProt reference clusters Training on the less-redundant versions helps the model see a wider diversity of protein families without being overwhelmed by near-duplicate sequences.
The scale of training data matters. ProGen, a model designed for protein generation, was trained on 280 million protein sequences drawn from more than 19,000 protein families.3PubMed Central. Large language models generate functional protein sequences across diverse families ESMFold, developed by Meta AI, scaled up to 15 billion parameters trained on large sequence databases.4PubMed. Evolutionary-scale prediction of atomic-level protein structure with a language model In general, bigger models trained on more diverse data tend to develop richer internal representations of protein biology, though the relationship between size and usefulness is not always straightforward.
How Training Works
Most protein language models are trained using one of two strategies borrowed from natural language processing. The first, and more common, is masked language modeling. The model is shown a protein sequence with some amino acids hidden, and it has to predict the missing ones based on context. This is the same approach used by BERT-style models in text. ESM-2, one of the most widely used protein language models, uses this strategy. The second is autoregressive generation, where the model reads the sequence from left to right and predicts the next amino acid at each step, similar to how GPT generates text one word at a time. ProGen and ProtGPT2 are examples of this approach.
The two strategies have different strengths. Research comparing them found that autoregressive generation from a decoder-based model like ProtGPT2 outperformed a masked-language encoder like ESM-2 at proposing realistic protein structures and achieving structural diversity when generating new sequences from scratch.5PubMed Central. Pretrained protein language models choose between sequence novelty and structural completeness The masked approach, on the other hand, excels at tasks where you need to understand an existing sequence in context, like predicting the effect of a mutation. Neither approach is universally better; it depends on what you want the model to do.
What the Model Actually Learns
During training, no one tells the model about protein structure, evolutionary relationships, or biochemistry. All it sees are sequences of amino acid letters. And yet, remarkably, the internal representations it develops encode real biological information. When ESM-2 was scaled to 15 billion parameters, researchers found that an atomic-resolution picture of protein structure emerged in the learned representations.4PubMed. Evolutionary-scale prediction of atomic-level protein structure with a language model The model had never been shown a single crystal structure, yet it implicitly learned where atoms sit in space.
One window into what the model learns is its attention mechanism. Transformers work by having each position in the sequence “attend” to other positions, weighting how much information to draw from them. In protein language models, these attention patterns often mirror physical contacts between amino acids in the folded protein. Analysis of protein structures shows that about 80% of amino acid positions have four or fewer structural contacts, and the total number of contacts scales linearly with sequence length.6PubMed Central. Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention This sparse contact pattern aligns well with the way attention heads work, where each position focuses on just a few others. Researchers have extracted attention maps from models like ESM-1b and used them to predict which amino acids are physically close to each other in the folded structure.7Bioinformatics. SPOT-Contact-LM: improving single-sequence-based prediction of protein contact map using a transformer language model
Beyond structure, the model’s internal representations also capture evolutionary information. When researchers applied ESM-2 to SARS-CoV-2 spike protein variants collected throughout the pandemic, they found that the model’s representations encoded the evolutionary history between variants, distinguishing variants of concern from earlier lineages. An analysis of these representations achieved a correlation of 0.86 against actual sampling time, confirming the model inferred the evolutionary order of the sequences correctly.8Nature Communications. From single-sequences to evolutionary trajectories: protein language models capture the evolutionary potential of SARS-CoV-2 The representations also capture functional similarity: proteins that do similar jobs end up near each other in the model’s internal space, acting as numeric proxies for structure and function without any additional training.9PubMed Central. Evaluating Pretrained Protein Language Model Embeddings as Proxies for Functional Similarity
Single Sequences Versus Multiple Sequence Alignments
Before protein language models came along, the standard approach to understanding a protein’s structure was to gather a multiple sequence alignment, or MSA. You would find hundreds or thousands of related sequences from different organisms, line them up, and look for patterns of co-evolution: positions that change together, suggesting they are physically close in the fold. AlphaFold2, the breakthrough structure prediction system, relies heavily on MSAs.
Protein language models introduced an alternative: predict structure from a single sequence, with no alignment needed. RGN2, which uses a protein language model called AminoBERT, demonstrated that this approach can outperform AlphaFold2 and RoseTTAFold on orphan proteins, ones that have so few known relatives that building an MSA is impractical, while requiring up to a million-fold less compute time.10PubMed Central. Single-sequence protein structure prediction using a language model and deep learning For most well-studied proteins, MSA-based methods still hold an edge. A recent model called MSA Pairformer, which combines language modeling with alignment data, achieved roughly three-fold better precision at predicting protein-protein interaction contacts compared to the next-best method, and single-sequence models like ESMC scored below even a simple statistical baseline on that task.11Cell. Expanding the scope of protein language modeling to protein-protein interactions with MSA Pairformer So in practice, the field has not abandoned MSAs; it has gained a fast, flexible alternative that shines in specific situations.
Predicting Mutation Effects
One of the most practically important applications of protein language models is predicting what happens when you change one or more amino acids in a protein. A single mutation can cause disease, improve an enzyme’s performance, or do nothing at all. Traditionally, figuring this out required either expensive lab experiments or building a detailed evolutionary model for each protein. Language models offer a shortcut: because they have learned what “normal” protein sequences look like, they can score how likely a mutation is. If the model assigns a much lower probability to the mutated sequence than the original, that mutation is probably disruptive.
This approach is called zero-shot prediction because the model has never been specifically trained on mutation data; it uses its general understanding of protein sequences. Research has shown that large pretrained models can serve as zero-shot predictors for clinically relevant tasks, including identifying disease-causing mutations and predicting patient survival rates.12bioRxiv. Protein Language Model Predicts Mutation Pathogenicity and Clinical Prognosis More sophisticated approaches combine sequence and structure information. ProMEP, a multimodal model, quantifies the likelihood of protein variants using both sequence and structural context, allowing it to map protein fitness landscapes and identify beneficial mutations for protein engineering.13Nature Cell Research. Zero-shot prediction of mutation effects with multimodal deep representation learning guides protein engineering
Designing New Proteins
If a protein language model understands what makes a real protein sequence work, it can also generate new sequences that follow those rules but do not exist in nature. This is de novo protein design, and it represents one of the most exciting frontiers in biotechnology. ProGen demonstrated that language models can generate protein sequences with predictable function across large protein families. Artificially generated proteins fine-tuned to five distinct lysozyme families showed catalytic activity comparable to natural lysozymes, even when their sequences shared as little as 31.4% identity with any natural protein.3PubMed Central. Large language models generate functional protein sequences across diverse families That is a striking result: the model invented proteins that work even though they look quite different from anything evolution has produced.
A related task is inverse folding, which flips the usual question. Instead of asking “what shape does this sequence fold into?”, inverse folding asks “what sequence would fold into this shape?” Language models trained on millions of predicted structures have proven effective at this problem.14bioRxiv. Learning inverse folding from millions of predicted structures Newer work integrates structural and evolutionary constraints into inverse folding models to identify high-fitness mutations, bridging the gap between computational design and practical protein engineering.15PubMed. Advancing protein evolution with inverse folding models integrating structural and evolutionary constraints Some teams have even begun using feedback loops: an inverse folding model generates candidate sequences, a structure prediction model evaluates them, and the results are fed back to improve the generator.16arXiv. Protein Inverse Folding From Structure Feedback
Fine-Tuning for Specific Tasks
A pretrained protein language model is a generalist. It knows a lot about proteins in general, but it has not been optimized for any one task. Fine-tuning takes a pretrained model and adapts it to a specific job by training it a little more on task-specific data. This is far cheaper than training from scratch, because the model already understands protein language; it just needs to learn the nuances of the particular application.
Fine-tuning has proven valuable in underrepresented corners of biology. Viral proteins, for example, are poorly represented in the massive protein databases that models are typically trained on. Research has shown that fine-tuning pretrained models on viral protein sequences, using parameter-efficient strategies, significantly improves performance on downstream tasks related to viral biology.17PubMed Central. Fine-tuning protein language models unlocks the potential of underrepresented viral proteomes In another study, researchers fine-tuned ESM2 and ProtT5 to classify protein features at the amino acid level, then used the fine-tuned models to identify features enriched in pathogenic versus benign variants and to understand how mutations affect protein functionality.18Computational and Structural Biotechnology Journal. Fine-tuning protein language models to understand the functional impact of missense variants
Domain-specific pretraining offers another route. EnzGFM, a model pretrained specifically on enzyme sequences, outperformed the general-purpose ESM2 at classifying enzyme functions. The advantage was most pronounced for evolutionarily distant proteins: at a stringent 10% sequence identity threshold, EnzGFM achieved about 13% higher precision than ESM2 despite having fewer parameters.19Nature Communications. An enzyme-specific protein language model for catalytic property prediction This suggests that when you have enough data within a protein niche, specialization pays off.
Practical Applications in Drug Development
Antibody engineering is one area where protein language models have rapidly gained traction in industry. Antibodies are proteins the immune system uses to recognize and neutralize threats, and they are also the basis of many modern drugs. Designing better therapeutic antibodies traditionally involves cycles of lab work to test candidates for binding strength, stability, and manufacturability. Protein language models can accelerate this by predicting which antibody sequences are likely to have desirable properties before any lab work is done. Research has demonstrated that these models provide robust signals for antibody developability assessment, supporting their use in early-stage candidate selection and optimization.20PubMed Central. Application of protein language models for antibody developability prediction Machine learning models including protein language models are now used to optimize not just binding affinity but also specificity, stability, viscosity, and manufacturability of therapeutic antibodies.21PubMed Central. Accelerating antibody discovery and optimization with high-throughput experimentation and machine learning
Structure prediction tools built on language models are also feeding into drug discovery pipelines. ESMFold, for example, generates predicted structures that can then be used as inputs to other tools. DeepProSite uses ESMFold-generated structures combined with language model representations to predict protein binding sites, the specific regions on a protein surface where a drug molecule might attach.22PubMed Central. DeepProSite: structure-aware protein binding site prediction using ESMFold and pretrained language model Knowing where a protein can be targeted speeds up the process of designing drugs that interact with it.
Adding Structure to the Input
Most protein language models are trained only on sequences. They learn about structure indirectly, to the extent that structure is reflected in sequence patterns. But proteins are fundamentally three-dimensional objects, and some information is hard to infer from sequence alone. This has motivated the development of multimodal models that accept both sequence and structural data as input. MULAN, for instance, adds a structure adapter module to the pretrained ESM2 model. The adapter encodes backbone angles of each amino acid and fuses this structural information with ESM2’s sequence-derived representations.23Bioinformatics Advances. MULAN: multimodal protein language model for sequence and structure encoding These hybrid approaches aim to combine the generality of sequence-trained models with the precision that structural data provides, and they represent the direction much of the field is heading.
Tokenization
One seemingly mundane but practically important design choice is how to break a protein sequence into tokens before feeding it into the model. The standard approach treats each amino acid as a single token, giving you a twenty-letter alphabet. This is straightforward, but it means a typical protein of 300 amino acids becomes a 300-token sequence, which is expensive for the attention mechanism. Sub-word tokenization methods like byte-pair encoding (BPE), which are standard in text-based language models, have been explored as a way to shorten these sequences. By grouping frequently co-occurring amino acids into multi-residue tokens, BPE can reduce sequence length and speed up computation. However, long repeating patterns are rarer in proteins than in natural language, so the compression gains are limited.24Bioinformatics. Optimizing protein tokenization: reduced amino acid alphabets for efficient and accurate protein language models Comparisons between BPE and single-character tokenization in genomic models have shown mixed results, with neither approach clearly dominating across all tasks.25bioRxiv. A Comparison of Tokenization Impact in Attention Based and State Space Genomic Language Models
A different strategy is to reduce the alphabet itself. Instead of twenty amino acid types, you group chemically similar ones together, so, say, all small hydrophobic amino acids become one token type. This shrinks the alphabet, which can then make sub-word tokenization more effective because repeated patterns become more common. Whether this helps depends on the task: for some applications the lost resolution hurts, and for others the compressed representation is fine.
Making Models Smaller
State-of-the-art protein language models can have billions of parameters, making them expensive to run and impractical for many research labs. Knowledge distillation is one approach to this problem: you train a smaller “student” model to mimic the behavior of a large “teacher” model. DistilProtBert, a distilled version of the ProtBert model, can perform tasks like secondary structure prediction and distinguishing real proteins from randomly shuffled sequences on commodity hardware, enabling fine-tuning and feature extraction in a fast and efficient manner.26Bioinformatics. DistilProtBert: a distilled protein language model used to distinguish between real proteins and their randomly shuffled counterparts As the field matures, making these tools accessible to smaller labs and companies, not just those with large GPU clusters, is an active area of work.
Where They Still Struggle
For all their success, protein language models have clear limitations. Sequence-only models can plateau on tasks that require understanding of subtle biochemical or structural context. They rely on statistical patterns in training data, which means they can miss the effects of mutations at distant positions that interact through the protein’s three-dimensional fold. Generalizability across protein families remains an open challenge: models trained on millions of sequences can still struggle with rare or unseen families, and they often fail on out-of-distribution proteins like orphans that have few known relatives.27Journal of Proteome Research. Protein Language Models: Applications and Perspectives
There is also a fundamental gap between learning sequence statistics and understanding protein biophysics. A protein language model does not “know” that a particular amino acid is bulky and hydrophobic, or that a certain pair of charges attract each other. It knows only that certain letters tend to appear together in certain contexts. For many tasks that is enough, but for novel protein design or predicting the consequences of never-before-seen mutations, the absence of physical reasoning can be a real limitation. This is why hybrid approaches that combine language model representations with physics-based or structure-based features continue to be an important area of development, and why traditional bioinformatics methods remain necessary for some applications.