Neural networks are computing systems loosely inspired by biological brains, built from layers of simple processing units that learn patterns directly from data rather than following hand-written rules. A network with even a single hidden layer can, in principle, approximate any continuous function, provided it has enough units and a suitable activation function.1Neural Networks. Approximation capabilities of multilayer feedforward networks That theoretical flexibility is what makes neural networks so broadly useful, but turning it into practical results requires choosing the right architecture for the problem, training the network effectively, and managing the enormous computational costs involved.
How a Neural Network Learns
At its core, a neural network is a stack of layers. Data enters the input layer, passes through one or more hidden layers where it gets transformed, and exits through an output layer that produces a prediction. Each connection between units carries a weight, and each unit applies an activation function that decides how strongly to fire. Activation functions have multiplied in variety over the years, from classic choices like the logistic sigmoid to the now-ubiquitous ReLU and dozens of newer variants, but their shared job is the same: they introduce the nonlinearity that lets a network model complex, curved relationships instead of just straight lines.2arXiv. Activation Functions in Artificial Neural Networks: A Systematic Overview
Training means adjusting all those weights so the network’s predictions get closer to the correct answers. The dominant method is backpropagation, an algorithm that traces back through the network to figure out how much each weight contributed to the error, then nudges every weight in the direction that reduces it. Backpropagation was developed in the mid-1980s, when it was independently described by several research groups, and it remains the backbone of virtually all neural network training today.3Encyclopedia of Information Science and Technology, Sixth Edition. Artificial Neural Networks: History and State of the Art
Backpropagation tells you which direction to move the weights, but you still need a strategy for how far and how fast to move them. That is the optimizer’s job. Stochastic gradient descent (SGD) is the simplest choice: take a small batch of training examples, compute the gradient, step in that direction. More sophisticated optimizers like Adam adapt the step size for each weight individually, which often speeds up convergence. In comparative experiments on image classification tasks, Adam consistently delivered the highest accuracy, reaching above 97% on one benchmark and above 99% on another.4PubMed Central. The effect of choosing optimizer algorithms to improve computer vision tasks: a comparative study That said, SGD with careful tuning still wins in some settings. The choice depends on the task, the architecture, and how much time you want to spend fiddling with learning rates.
Feedforward Networks and the Universal Approximation Idea
The simplest architecture is the feedforward network, sometimes called a multilayer perceptron. Information flows in one direction: input to hidden layers to output, with no loops. This is the architecture behind the universal approximation theorem, which says that a feedforward network with a single hidden layer and a nonlinear activation function can approximate any continuous function over a compact region, as long as you give it enough hidden units.1Neural Networks. Approximation capabilities of multilayer feedforward networks Recent work has extended this result to deeper architectures with connections that skip between layers, showing the same approximation power holds whenever the activation function is nonlinear.5arXiv.org. Universal Approximation Theorem for Input-Connected Multilayer Perceptrons
This sounds like a feedforward network should be able to solve anything, but the theorem only guarantees that a solution exists somewhere in weight-space. It says nothing about whether training will actually find it, or how many hidden units you would need. In practice, a shallow network wide enough to model a complex task might require an absurd number of units, while a deeper but narrower network can often represent the same function more efficiently. That tension between theoretical power and practical trainability is what drove the development of specialized architectures.
Convolutional Neural Networks
Convolutional neural networks, or CNNs, were designed for data that has spatial structure, like images. Instead of connecting every input to every hidden unit, a CNN slides small filters across the input, each filter looking for a specific local pattern such as an edge, a texture, or a color gradient. Early layers detect simple features; deeper layers combine those into increasingly complex ones, like eyes, wheels, or letters. This weight-sharing scheme slashes the number of parameters compared to a fully connected network and bakes in a useful bias: the network treats a cat in the top-left corner the same as a cat in the bottom-right.
CNNs dominate image-related tasks. Modern object detection systems and semantic segmentation pipelines build on CNN backbones, often combined with more recent architectural ideas like YOLO variants and transformer-based detectors such as DETR.6arXiv. Deep Learning and Machine Learning — Object Detection and Semantic Segmentation: From Theory to Applications Extensions of the convolutional idea also work on non-image data. Graph-based CNNs apply convolution-like operations to irregularly structured data such as social networks or molecular graphs.7arXiv. Graph Based Convolutional Neural Network
Recurrent Networks and the Sequence Problem
Feedforward networks and CNNs process each input independently. If your data is a sequence where order matters, like a sentence or a time series, you need something that can remember what came before. Recurrent neural networks (RNNs) solve this by introducing loops: the output of a hidden unit at one time step feeds back in as input at the next step, giving the network a kind of memory. In theory, this lets an RNN retain information from an arbitrarily long context window.8arXiv. A Critical Review of Recurrent Neural Networks for Sequence Learning
In practice, plain RNNs struggle with long sequences because gradients tend to shrink or explode as they are propagated back through many time steps. The Long Short-Term Memory (LSTM) architecture addressed this with gating mechanisms that let the network decide what to remember and what to forget at each step. LSTM-based systems achieved breakthrough results on tasks ranging from language translation to handwriting recognition and image captioning.8arXiv. A Critical Review of Recurrent Neural Networks for Sequence Learning For over a decade, LSTMs and their close relative the GRU were the default choice for sequential data.
Transformers
Transformers have largely replaced RNNs for sequence tasks. The key innovation is self-attention: instead of processing a sequence one element at a time, a transformer lets every element attend to every other element simultaneously. This captures long-range dependencies without the vanishing-gradient issues of recurrence, and it parallelizes naturally on modern hardware, which makes training dramatically faster.9arXiv. Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
The transformer’s versatility turned out to be remarkable. In natural language processing, the field evolved from static word embeddings like Word2Vec and GloVe, which assign each word a single fixed vector regardless of context, to transformer-based models like BERT and GPT that generate different representations for the same word depending on its surroundings.10PERFECT: Journal of Smart Algorithms. From Static to Contextual: A Survey of Embedding Advances in NLP Autoregressive language models in the GPT family take a sequence of tokens as input, produce a contextual embedding for each, and predict what token comes next.11Artificial Intelligence: Foundations, Theory, and Algorithms. Pre-trained Language Models Scale that architecture up to hundreds of billions of parameters and train it on vast text corpora, and you get the large language models behind ChatGPT, Claude, and similar systems. Transformers now also power vision models, protein structure predictors, speech recognition systems, and music generators.
Graph Neural Networks
Not all data fits neatly into grids or sequences. Social networks, molecular structures, knowledge bases, and transportation maps are all naturally represented as graphs, with nodes and edges. Graph neural networks (GNNs) process these structures through message passing: each node gathers information from its neighbors, updates its own representation, and repeats for several rounds. After a few rounds, every node’s representation reflects the structure of its local neighborhood.
Standard message-passing GNNs have a known expressiveness ceiling, roughly equivalent to a classical graph-comparison test called the one-dimensional Weisfeiler-Leman test. Newer designs push past this limit. Identity-aware GNNs, for example, incorporate information about each node’s own identity during message passing, which lets them distinguish graph structures that would look identical to a vanilla GNN.12Proceedings of the AAAI Conference on Artificial Intelligence. Identity-aware Graph Neural Networks GNNs have become essential in drug discovery and materials science, where predicting a molecule’s properties from its graph structure replaces expensive lab experiments.
Diffusion Models and Generative AI
Generative models aim to create new data that looks like the training data, and diffusion models are the newest heavyweight in this space. The core idea is elegant: start with a real image and gradually add random noise to it over many steps until it becomes pure static. Then train a neural network to reverse this process, predicting and removing the noise one step at a time. Once trained, you can start from pure noise and step backward through the chain to generate a brand-new image from the learned distribution.13Scientific Reports. Denoising diffusion probabilistic models for 3D medical image generation
Diffusion models are behind systems like DALL-E, Stable Diffusion, and Midjourney for image generation. They have also been adapted for 3D medical image synthesis, video generation, audio, and even molecular design. Compared to earlier generative approaches like GANs, diffusion models tend to produce higher-quality and more diverse outputs, though they are slower at inference because they require many denoising steps.
Fine-Tuning and Alignment
Training a large model from scratch costs millions of dollars in compute. For most practical purposes, you start with a pre-trained model and adapt it to your specific task. Full fine-tuning updates every weight, but for models with hundreds of billions of parameters, that is prohibitively expensive. Low-Rank Adaptation (LoRA) offers a shortcut: freeze all the original weights and inject small trainable matrices into each layer. Compared to fully fine-tuning GPT-3’s 175 billion parameters, LoRA reduces the number of trainable parameters by a factor of 10,000 and cuts GPU memory requirements by roughly three times, while matching or beating full fine-tuning on quality.14arXiv. LoRA: Low-Rank Adaptation of Large Language Models Variations like layer-wise adaptive LoRA further refine this by adjusting the rank of the injected matrices at each layer.15Neural Networks. La-LoRA: Parameter-efficient fine-tuning with layer-wise adaptive low-rank adaptation
LoRA’s efficiency has made it practical to adapt frontier-scale models for specialized domains. In one recent study, LoRA fine-tuning of a multimodal LLaMA model achieved high accuracy on electrocardiogram image interpretation, with optimal results at a rank setting of 64.16PubMed Central. High-accuracy ECG image interpretation using parameter-efficient Low-Rank Adaptation (LoRA) fine-tuning with multimodal LLaMA V.3.2 The technique is now a standard tool in the AI practitioner’s kit, used everywhere from chatbot customization to medical imaging.
Fine-tuning gets a model good at a task, but getting it to behave in ways humans actually want is a separate challenge. Reinforcement learning from human feedback (RLHF) was the early answer: collect human judgments about which model outputs are better, train a reward model on those preferences, then use reinforcement learning to steer the language model toward higher-reward responses. It works, but the pipeline is complicated and finicky. Direct Preference Optimization (DPO) simplifies this by reformulating the reward model so that the optimal policy can be extracted directly, turning the whole alignment step into a straightforward classification problem. DPO matches or outperforms RLHF while eliminating the need for a separate reward model and the instability of reinforcement learning training loops.17arXiv. Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Scaling Laws and Why Size Matters
One of the most striking empirical findings in modern AI is that neural network performance follows remarkably predictable scaling laws. As you increase the number of parameters, the size of the training dataset, or the amount of compute, the model’s loss decreases as a smooth power law, with some trends spanning more than seven orders of magnitude.18arXiv. Scaling Laws for Neural Language Models This is not just an interesting observation. It means that researchers can predict roughly how well a model will perform before spending the money to train it, which shapes billion-dollar investment decisions.
Theoretical work has begun explaining why these power-law relationships exist. One framework identifies four distinct scaling regimes depending on whether performance is limited by dataset size or model size, and whether the bottleneck is variance (not enough data to pin down the right function) or resolution (the model lacking the capacity to represent fine-grained structure).19PubMed Central. Explaining neural scaling laws Understanding these regimes helps practitioners decide whether their next dollar is better spent on more data or a bigger model.
The downside of scaling is cost. Training frontier models now requires thousands of specialized GPUs running for weeks, consuming energy budgets comparable to small industrial operations. This creates a strong incentive to compress trained models for deployment, especially on phones, wearables, and other edge devices that lack data-center hardware.
Compression for the Real World
Three main techniques make large models small enough to run on constrained hardware. Pruning removes redundant weights or entire neurons that contribute little to accuracy. Quantization reduces the numerical precision of weights, for example from 32-bit floating point down to 8-bit integers, which shrinks memory footprint and speeds up computation on hardware designed for integer math. Knowledge distillation trains a small “student” network to mimic the outputs of a large “teacher” network, transferring learned behavior into a compact model.20arXiv. PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation
These techniques combine well. Recent work on ordered compression pipelines shows that applying pruning first, then quantization, then distillation consistently produces a better tradeoff between accuracy, model size, and inference speed than any single technique alone, reaching sub-1.5 millisecond inference latency on a standard CPU with competitive accuracy.21arXiv. Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression This kind of pipeline is what makes it possible for your phone to run speech recognition, real-time translation, or computational photography models locally without sending data to the cloud.
Applications in Drug Discovery and Protein Science
Neural networks have reshaped computational biology. AlphaFold2 used a deep neural network to predict protein structures with accuracy that stunned the field, enabling drug designers to work with structural information that previously required months of laboratory crystallography.22PubMed. AI-Based Protein Structure Prediction in Drug Discovery: Impacts and Challenges Its successor, AlphaFold 3, expanded to modeling interactions between proteins and other molecules, establishing itself as a reference tool in computational biochemistry and drug development.23PubMed Central. Review of AlphaFold 3: Transformative Advances in Drug Design and Therapeutics
Generative AI is pushing the field further. Graph-based molecular generation frameworks design novel drug candidates from scratch. Diffusion-based structural prediction pipelines like RFdiffusion engineer entirely new proteins. AI-augmented molecular docking tools like DiffDock predict how a drug molecule fits into a protein binding site, accelerating the earliest and most uncertain stage of drug development.24Medicine in Drug Discovery. Generative AI for drug discovery and protein design: the next frontier in AI-driven molecular science None of these replace wet-lab experiments entirely, but they dramatically narrow the search space, letting researchers focus expensive lab time on the most promising candidates.
The Interpretability Problem
For all their power, neural networks remain frustratingly opaque. A model can achieve superhuman accuracy on a medical image classification task, but when you ask why it flagged a particular scan, the answer is buried in millions of weight values. This is not just an academic concern. In healthcare, finance, and criminal justice, decisions need to be explainable, and regulators increasingly demand it.
Researchers have developed tools like SHAP values and Integrated Gradients that try to assign importance scores to each input feature, telling you which pixels or which words mattered most for a given prediction. These methods are widely used and often convincing. But there is a sobering theoretical limit: for model classes as expressive as neural networks, any feature attribution method that satisfies certain natural mathematical properties can provably fail to do better than random guessing at inferring how the model actually behaves.25PubMed Central. Impossibility theorems for feature attribution In other words, the explanations these tools produce may look plausible without faithfully reflecting the model’s internal reasoning.
This does not mean interpretability research is hopeless, but it does mean that popular explanation tools should be treated with healthy skepticism rather than taken as ground truth. Newer approaches like mechanistic interpretability, which tries to reverse-engineer what individual neurons and circuits inside a network have learned to do, offer a different angle. The field is active and unsettled, and the gap between what we can build and what we can understand about what we have built remains one of the defining tensions of modern AI.
Where Architectures Blur Together
The neat categories described above are increasingly messy in practice. Vision Transformers apply the transformer architecture to images, stepping on CNN territory. Graph transformers combine graph structure with self-attention. Diffusion models use U-Net backbones (a CNN variant) or transformers interchangeably. Multimodal models accept text, images, and audio simultaneously, routing each through a different encoder before fusing representations in a shared transformer.
This convergence reflects a practical lesson: the best architecture for a task depends less on theoretical elegance and more on what the data looks like and how much compute you can afford. Transformers win many benchmarks partly because they scale so well with hardware and data, not because they are inherently better at representing every kind of function. If you are working with small datasets on tabular data, a modest feedforward network or even a gradient-boosted tree still often beats a transformer. If you are processing molecular graphs, a GNN with domain-specific inductive biases will likely outperform a general-purpose language model. Matching architecture to problem remains more craft than science, guided by accumulated community experience and a lot of trial and error.