paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

AI Engineering Fundamentals: A Deep Dive into ANNs, CNNs, and RNNs

Understanding the Architecture, Trade-offs, and Practical Engineering Patterns Behind the Three Pillars of Modern Neural Networks

Introduction

Neural networks are no longer academic curiosities confined to research papers. They are production systems, running inference billions of times per day across fraud detection pipelines, image classification services, language translation APIs, and real-time recommendation engines. As AI engineering matures into a distinct discipline - with its own deployment concerns, latency budgets, observability stacks, and failure modes - it becomes critically important for software engineers to move beyond surface-level familiarity with these architectures and develop a working mental model of how each network type is designed, why it is shaped the way it is, and where it breaks down.

This post is not a gentle introduction to machine learning for beginners. It is a structured technical deep-dive aimed at software engineers who already understand gradient descent, backpropagation, and the basics of tensor operations, and who now want to understand the engineering decisions embedded in Artificial Neural Networks (ANNs), Convolutional Neural Networks (CNNs), and Recurrent Neural Networks (RNNs). We will examine each architecture from first principles, explore realistic implementation patterns in Python, and discuss trade-offs that matter in production environments - latency, memory, training stability, and inference scalability.

The Problem Space: Why Architecture Matters

A common misconception among engineers new to AI is that neural network architecture is a detail left to data scientists. In reality, choosing the wrong architecture for a problem is an engineering mistake with serious downstream consequences. A model that is structurally mismatched to its data will either fail to learn meaningful representations or will require excessive compute to compensate - both outcomes are expensive.

The core insight is this: neural network architectures encode inductive biases. An inductive bias is an assumption baked into the model structure about what kinds of patterns are likely to exist in the data. A CNN assumes spatial locality and translation invariance, which is why it works well for images. An RNN assumes sequential dependencies and time-ordered causality, which is why it was the dominant architecture for text and time series before the Transformer era. A plain ANN (often called an MLP - Multi-Layer Perceptron) makes almost no assumptions: it is the most general and therefore the least data-efficient of the three.

Understanding inductive biases helps you reason about when to reach for each architecture, what data volumes are required to make it work, and what generalisation you can expect at the boundary of the training distribution. These are the questions that matter when you are shipping AI systems, not just training notebooks.

Artificial Neural Networks (ANNs / MLPs)

Architecture and Mechanics

An Artificial Neural Network, in its feed-forward form, is a directed acyclic graph of layers. Each layer is a matrix multiplication followed by a non-linearity. Input flows in one direction - from the input layer through one or more hidden layers to the output layer - and there are no cycles. The network learns by adjusting weight matrices via backpropagation and gradient descent.

The fundamental unit is the neuron, which computes a weighted sum of its inputs, adds a bias term, and applies an activation function: y = f(Wx + b). The activation function is what gives neural networks their expressive power - without it, stacking layers would still produce a linear function, regardless of depth. Common choices include ReLU (Rectified Linear Unit) for hidden layers due to its computational simplicity and resistance to the vanishing gradient problem, sigmoid for binary output nodes, and softmax for multi-class classification heads.

What makes MLPs powerful is their status as universal function approximators. The Universal Approximation Theorem (Cybenko, 1989; Hornik, 1991) guarantees that a sufficiently wide single hidden layer network can approximate any continuous function on a compact subset of R^n to arbitrary precision. However, this theorem says nothing about how many parameters you need or how efficiently you can learn - both of which are critical in practice.

When to Use ANNs

MLPs are most appropriate when your input is a fixed-size vector of independent or weakly-ordered features - think tabular data with numerical and categorical columns. They make no assumptions about spatial structure or sequential ordering, which is a disadvantage when such structure exists (images, time series), but an advantage when your features are genuinely heterogeneous. In practice, MLPs remain highly competitive on structured tabular data and often outperform more complex architectures when the dataset is well-engineered.

They are also the building block used everywhere else: the feed-forward sub-layer inside a Transformer is an MLP. The final classification head on a CNN is an MLP. Understanding ANNs deeply is not optional - every other architecture composes on top of it.

Implementation Pattern

The following is a production-shaped MLP implementation in Python using PyTorch, demonstrating batch normalisation, dropout regularisation, and configurable depth - patterns you would actually use in a system, not a textbook:

import torch
import torch.nn as nn
from typing import List

class MLP(nn.Module):
    """
    Configurable Multi-Layer Perceptron with batch norm and dropout.
    Suitable for tabular regression and classification tasks.
    """

    def __init__(
        self,
        input_dim: int,
        hidden_dims: List[int],
        output_dim: int,
        dropout_rate: float = 0.3,
        use_batch_norm: bool = True,
    ):
        super().__init__()
        layers: List[nn.Module] = []
        prev_dim = input_dim

        for hidden_dim in hidden_dims:
            layers.append(nn.Linear(prev_dim, hidden_dim))
            if use_batch_norm:
                layers.append(nn.BatchNorm1d(hidden_dim))
            layers.append(nn.ReLU())
            layers.append(nn.Dropout(p=dropout_rate))
            prev_dim = hidden_dim

        layers.append(nn.Linear(prev_dim, output_dim))
        self.network = nn.Sequential(*layers)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.network(x)


# Usage: binary classification on tabular data
model = MLP(
    input_dim=64,
    hidden_dims=[256, 128, 64],
    output_dim=1,
    dropout_rate=0.2,
    use_batch_norm=True,
)

# Training step (simplified)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
criterion = nn.BCEWithLogitsLoss()

def train_step(batch_x: torch.Tensor, batch_y: torch.Tensor) -> float:
    model.train()
    optimizer.zero_grad()
    logits = model(batch_x).squeeze(-1)
    loss = criterion(logits, batch_y.float())
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    return loss.item()

Note the use of AdamW over plain Adam - weight decay is applied correctly to all parameters except biases. Gradient clipping via clip_grad_norm_ is included because even MLPs can experience gradient explosions on pathological batches, particularly early in training.

Convolutional Neural Networks (CNNs)

The Inductive Bias of Locality

CNNs were designed to exploit a specific property of structured grid data: local correlations matter more than global ones, and the same feature can appear anywhere in the grid. An edge in an image can appear at any position - top-left, centre, bottom-right - and it should be detected regardless of location. This is translation invariance. A convolutional filter (kernel) encodes this by sliding across the entire input and applying the same learned weights everywhere.

The convolution operation is conceptually simple: a small matrix (the kernel) is multiplied element-wise with a local patch of the input and the products are summed to produce a single output value. This is repeated as the kernel slides across the input with a defined stride. By stacking multiple kernels, the network learns multiple feature detectors at each layer. Early layers tend to learn low-level features (edges, textures), while deeper layers learn higher-level abstractions (shapes, object parts). This hierarchical feature extraction is the source of CNN's remarkable effectiveness on visual data.

Key Components in Engineering Practice

Beyond the convolution itself, several components are critical to understand from an engineering standpoint. Pooling layers (typically max pooling or average pooling) reduce spatial dimensions, decreasing computational cost and providing a degree of translation invariance. Batch Normalisation (Ioffe & Szegedy, 2015) is now standard in CNN architectures - it stabilises training, reduces sensitivity to learning rate choices, and allows much deeper networks to train reliably.

Modern CNN architectures also use residual connections (He et al., 2016 - ResNet) to allow gradients to flow directly through skip connections. Without residual connections, very deep networks suffer from vanishing or degraded gradients. With them, 50, 101, or even 152-layer networks become trainable. In production, the choice between ResNet-18, ResNet-50, EfficientNet-B0, or MobileNetV3 is often governed by the latency-accuracy trade-off, not raw accuracy alone.

Implementation Pattern

Here is a production-ready convolutional block and a simple feature extractor backbone, demonstrating the residual connection pattern:

import torch
import torch.nn as nn
import torch.nn.functional as F

class ConvBNReLU(nn.Module):
    """Standard Conv -> BatchNorm -> ReLU block."""

    def __init__(self, in_channels: int, out_channels: int, kernel_size: int = 3, stride: int = 1):
        super().__init__()
        padding = kernel_size // 2  # 'same' padding for odd kernels
        self.block = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, kernel_size, stride=stride, padding=padding, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(inplace=True),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.block(x)


class ResidualBlock(nn.Module):
    """
    Basic residual block: two ConvBNReLU layers with a skip connection.
    Handles channel mismatch and spatial downsampling via the projection shortcut.
    """

    def __init__(self, in_channels: int, out_channels: int, stride: int = 1):
        super().__init__()
        self.conv1 = ConvBNReLU(in_channels, out_channels, stride=stride)
        self.conv2 = nn.Sequential(
            nn.Conv2d(out_channels, out_channels, 3, padding=1, bias=False),
            nn.BatchNorm2d(out_channels),
        )

        # Projection shortcut when dimensions change
        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1, stride=stride, bias=False),
                nn.BatchNorm2d(out_channels),
            )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        out = self.conv1(x)
        out = self.conv2(out)
        out = out + self.shortcut(x)  # skip connection
        return F.relu(out, inplace=True)


class SimpleCNNBackbone(nn.Module):
    """
    Lightweight CNN backbone for image feature extraction.
    Outputs a spatial feature map suitable for downstream tasks.
    """

    def __init__(self, in_channels: int = 3, base_channels: int = 64):
        super().__init__()
        self.stem = ConvBNReLU(in_channels, base_channels, kernel_size=7, stride=2)
        self.pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = ResidualBlock(base_channels, base_channels)
        self.layer2 = ResidualBlock(base_channels, base_channels * 2, stride=2)
        self.layer3 = ResidualBlock(base_channels * 2, base_channels * 4, stride=2)
        self.global_avg_pool = nn.AdaptiveAvgPool2d((1, 1))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.stem(x)
        x = self.pool(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.global_avg_pool(x)
        return torch.flatten(x, 1)


# Build a full classifier
class ImageClassifier(nn.Module):
    def __init__(self, num_classes: int, in_channels: int = 3):
        super().__init__()
        self.backbone = SimpleCNNBackbone(in_channels=in_channels, base_channels=64)
        self.head = nn.Linear(64 * 4, num_classes)  # 256 -> num_classes

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        features = self.backbone(x)
        return self.head(features)

The AdaptiveAvgPool2d((1, 1)) at the end of the backbone is a key engineering pattern: it makes the model input-size agnostic, allowing inference on variable-resolution images without architectural changes - a common requirement in production image pipelines.

Recurrent Neural Networks (RNNs)

Modelling Sequential State

Where CNNs exploit spatial locality, RNNs exploit temporal locality. They are designed to process sequences - text, audio, time series, event logs - where the order of elements carries information. The defining characteristic of an RNN is the hidden state: a vector that is updated at each time step based on the current input and the previous hidden state. This creates a form of memory - information from earlier in the sequence can influence the processing of later elements.

The canonical vanilla RNN update equation is: h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b). The hidden state h_t at time t is a non-linear function of the previous state and the current input. This recursive structure is elegant but creates a severe engineering problem: when you backpropagate through many time steps (Backpropagation Through Time, or BPTT), gradients either vanish (when eigenvalues of the weight matrix are < 1, repeated multiplication drives them to zero) or explode (when eigenvalues > 1). In practice, vanilla RNNs cannot learn long-range dependencies - sequences longer than 10-20 steps become intractable.

LSTMs and GRUs: Engineering Solutions to the Vanishing Gradient

The Long Short-Term Memory (LSTM) network (Hochreiter & Schmidhuber, 1997) addresses the vanishing gradient problem through a principled gating mechanism. An LSTM cell maintains two state vectors: the hidden state h_t and the cell state c_t. The cell state is a memory channel with an additive update - gradients can flow through it unimpeded over long sequences. Three learnable gates control information flow: the forget gate (how much of the previous cell state to retain), the input gate (how much of the new candidate to write), and the output gate (what portion of the cell state to expose as the hidden state).

The Gated Recurrent Unit (GRU; Cho et al., 2014) simplifies the LSTM by merging the cell state and hidden state into a single vector and using only two gates - the reset gate and the update gate. GRUs are faster to train, use fewer parameters, and perform comparably to LSTMs on many tasks. In production, the choice between LSTM and GRU is often empirical, but GRU is a reasonable default when training time or model size is constrained.

Implementation Pattern

The following demonstrates a bidirectional LSTM for sequence classification - a common real-world pattern for tasks like sentiment analysis, anomaly detection in log streams, or event sequence classification:

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class BiLSTMClassifier(nn.Module):
    """
    Bidirectional LSTM sequence classifier.
    Handles variable-length sequences via packing/unpacking.
    """

    def __init__(
        self,
        vocab_size: int,
        embed_dim: int,
        hidden_dim: int,
        num_layers: int,
        num_classes: int,
        dropout: float = 0.3,
        pad_idx: int = 0,
    ):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
        self.lstm = nn.LSTM(
            input_size=embed_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            bidirectional=True,
            dropout=dropout if num_layers > 1 else 0.0,
        )
        self.dropout = nn.Dropout(dropout)
        # Bidirectional doubles the hidden size
        self.classifier = nn.Linear(hidden_dim * 2, num_classes)

    def forward(self, x: torch.Tensor, lengths: torch.Tensor) -> torch.Tensor:
        # x: (batch, seq_len), lengths: (batch,) - actual sequence lengths
        embedded = self.dropout(self.embedding(x))  # (batch, seq_len, embed_dim)

        # Pack sequences to avoid computing over padding tokens
        packed = pack_padded_sequence(embedded, lengths.cpu(), batch_first=True, enforce_sorted=False)
        packed_out, (h_n, _) = self.lstm(packed)
        # h_n: (num_layers * 2, batch, hidden_dim)

        # Concatenate the final forward and backward hidden states
        # h_n[-2] is last forward layer, h_n[-1] is last backward layer
        final_hidden = torch.cat([h_n[-2], h_n[-1]], dim=-1)  # (batch, hidden_dim * 2)
        final_hidden = self.dropout(final_hidden)

        return self.classifier(final_hidden)


# Inference utility
def predict_sequence(
    model: BiLSTMClassifier,
    token_ids: torch.Tensor,
    lengths: torch.Tensor,
) -> torch.Tensor:
    model.eval()
    with torch.no_grad():
        logits = model(token_ids, lengths)
        return torch.softmax(logits, dim=-1)

The use of pack_padded_sequence and pad_packed_sequence is critical in production: without packing, the LSTM processes all padding tokens, wasting compute and introducing noise from out-of-sequence positions into the final hidden state. This is a subtle but commonly missed engineering detail.

Trade-offs and Pitfalls in Production

ANN Pitfalls

The primary pitfall with MLPs on tabular data is insufficient regularisation. Because MLPs make no structural assumptions, they have a high capacity to overfit, particularly on datasets with many features and relatively few samples. Engineers frequently reach for deeper networks when a better-regularised shallower network would generalise better. The standard toolkit - dropout, weight decay, early stopping, and batch normalisation - is well-established, but requires careful tuning.

A second issue is feature engineering dependency. Unlike CNNs, which learn spatial features automatically, MLPs on tabular data benefit enormously from hand-crafted features: log transforms for skewed distributions, polynomial interaction terms, proper encoding of categorical variables (target encoding, embeddings for high-cardinality categories). Skipping this engineering work and relying on raw features is a common cause of underperformance. The MLP is a strong learner, but it is not a substitute for domain-informed feature construction.

CNN Pitfalls

CNNs are remarkably sensitive to input normalisation. Training on unnormalised pixel values (0-255) without mean-subtraction and standard deviation scaling will produce unstable training dynamics and degraded performance. In production, you must also apply the same normalisation at inference time - a mismatch between training and inference pre-processing is one of the most common and hardest-to-debug sources of accuracy degradation in deployed image models.

Transfer learning, while powerful, introduces its own pitfalls. When fine-tuning a pre-trained backbone (e.g., a ResNet pretrained on ImageNet), it is important to use a much smaller learning rate for the backbone layers than for the newly initialised head. Applying a uniform learning rate will corrupt the pre-learned feature representations in early layers before the head has had time to adapt, leading to worse final accuracy than training from scratch. Discriminative learning rates - where different parameter groups receive different learning rates - are standard practice and supported natively in libraries like fastai and PyTorch Lightning.

RNN Pitfalls

Despite gating mechanisms, LSTMs and GRUs struggle with sequences longer than a few hundred tokens. The hidden state is a fixed-size bottleneck - it must compress the entire context into a single vector, losing fine-grained information from earlier positions as the sequence grows. The attention mechanism, originally introduced to augment RNNs (Bahdanau et al., 2015) and later the basis of the Transformer architecture (Vaswani et al., 2017), was developed precisely to address this bottleneck.

In production, RNNs are also harder to parallelise than CNNs or Transformers. Each time step depends on the previous hidden state, making true parallel processing impossible within a sequence. This manifests as longer training times and higher latency at inference, particularly for long sequences. For most new sequence modelling tasks initiated today, a Transformer or a state-space model (e.g., Mamba) is a stronger default choice than an LSTM - but understanding RNNs remains essential for maintaining legacy systems and for constrained-resource environments where Transformer quadratic attention costs are prohibitive.

Best Practices for AI Engineering

Choose Architecture Based on Data Structure, Not Trend

The single most important engineering decision is architecture selection, and it should be driven by the structure of your data and the inductive biases you want to encode. Use CNNs for spatial grid data with local correlations. Use RNNs (or Transformers) for sequential data with temporal dependencies. Use MLPs for tabular data with heterogeneous features. When in doubt, start with the simplest architecture that could plausibly work - complexity should be introduced only when simpler models have been shown to underperform.

In practice, many real-world systems combine architectures. A video understanding model might use a CNN backbone to extract per-frame features and an LSTM to model temporal dynamics across frames. A time series forecasting system might use a 1D CNN to extract local patterns and an MLP head for the final prediction. Understanding each architecture as a composable module - not as a standalone solution - is the mindset that produces robust AI engineering.

Instrument Training and Inference Like Any Other System

Neural networks are software systems, and they should be treated as such. Log training loss and validation metrics at every epoch. Track gradient norms - a sudden spike or collapse in gradient norms is an early warning sign of instability. Use learning rate schedulers (cosine annealing, warmup schedules) and track the effective learning rate over time. At inference time, instrument latency at the p50, p95, and p99 percentiles - not just the mean.

Equally important is data pipeline observability. Training bugs frequently originate in the data loader, not the model: silent errors in augmentation, incorrect label mappings, or batch sizes that silently drop the last few samples. Validate input shapes, dtypes, and value ranges at batch boundaries using assertions or schema validators. The time investment in a robust data pipeline pays dividends in debugging speed throughout the project lifecycle.

Prefer Pre-trained Models for Transfer Learning Where Possible

Training large neural networks from scratch requires significant compute, large datasets, and careful hyperparameter tuning. For most production tasks, fine-tuning a pre-trained model from a model hub (Hugging Face Hub, PyTorch Hub, TensorFlow Hub) is more efficient and produces better results than training from random initialisation. This applies to all three architecture families: pre-trained CNNs (EfficientNet, ResNet, ViT) on vision tasks, pre-trained LSTMs or Transformers on sequence tasks, and pre-trained MLP mixers on structured data tasks.

The key engineering concern with pre-trained models is distribution shift: if your production data differs significantly from the pre-training distribution, the pre-learned representations may not transfer well. Monitor model performance on recent production samples and retrain or fine-tune periodically. Use tools like Evidently AI, WhyLabs, or custom statistical tests to detect data drift before it silently degrades model accuracy.

Analogies and Mental Models

Understanding these architectures becomes significantly easier with the right mental models. Think of an MLP as a committee of generalists: each layer transforms the representation independently, with no knowledge of spatial or temporal structure baked in. The committee can learn almost anything, but it needs a lot of examples and careful training to do so.

A CNN is like a team of specialists with magnifying glasses, each scanning a different patch of the input looking for a specific pattern. Every specialist uses the same magnifying glass (shared weights) regardless of where in the image they are looking. The specialists at lower layers look for simple patterns (edges, corners), while those at higher layers combine their findings to recognise complex structures (faces, vehicles).

An RNN is like a reader moving through a sentence one word at a time, carrying a notepad. At each word, the reader updates the notepad based on what they just read and what was already written. The problem is that the notepad has limited space - if the sentence is very long, early information gets overwritten. An LSTM gives the reader a more sophisticated notepad with explicit mechanisms for deciding what to write, what to erase, and what to read back out.

The 80/20 Insight

If you take one structural insight from each architecture, let it be these: for ANNs, depth enables compositionality of features, but regularisation is what allows depth to generalise. For CNNs, weight sharing through convolution is the fundamental efficiency - a single kernel applied across an entire image requires a fraction of the parameters a fully connected layer would. For RNNs, the gating mechanism in LSTMs and GRUs is the key innovation - it creates additive gradient pathways that allow long-range learning where vanilla recurrence cannot.

These three insights - regularised depth, weight sharing, and gated memory - account for the majority of the practical performance gains you will observe when engineering AI systems with these architectures. Everything else - optimiser choices, learning rate schedules, augmentation strategies - is important but second-order relative to getting the architecture's core mechanism right.

Key Takeaways

  1. Match architecture to data structure: CNNs for spatial data, RNNs for sequential data, MLPs for tabular data. Mismatched architectures waste compute and underperform regardless of tuning effort.
  2. Use residual connections for deep CNNs and regularisation for deep MLPs: without them, deep networks either fail to train (vanishing gradients) or overfit (insufficient regularisation).
  3. Pack variable-length sequences in RNNs: using pack_padded_sequence in PyTorch is a non-optional production pattern - padding tokens contaminate the hidden state if not masked.
  4. Instrument your training pipeline as rigorously as your inference pipeline: gradient norms, data loader validation, and learning rate tracking are not optional diagnostics; they are operational hygiene.
  5. Default to pre-trained models and fine-tune: for the vast majority of production tasks, fine-tuning a well-chosen pre-trained model will outperform training from scratch in both accuracy and total engineering time.

Conclusion

Artificial Neural Networks, Convolutional Neural Networks, and Recurrent Neural Networks are not competing technologies - they are complementary tools, each designed to exploit a specific structural property of data. ANNs offer generality at the cost of data efficiency. CNNs offer spatial efficiency through local weight sharing. RNNs offer temporal modelling through recurrent state, at the cost of parallelism and long-range capacity.

For AI engineers, the practical skill is not knowing how to implement each architecture from scratch (though that understanding matters), but knowing how to select, compose, instrument, and deploy them reliably. As the field continues to evolve - with state-space models challenging RNNs, and Vision Transformers challenging CNNs - the foundational reasoning that explains why these architectures are designed the way they are will remain the most durable knowledge you can carry forward. Architecture is encoding assumptions about data. Get that right, and everything else becomes tractable.

References