Artificial Intelligence: A Conceptual Learner's Series

Part 7: Deep learning

Author: Christopher DanosAI by Claude

Deep learning drives most of today's headline AI. This part builds it up from a single artificial neuron to the architectures behind image recognition, language models and image generators.

Concept 1: Artificial neurons and layers

In one line: A neural network is a stack of simple units, each of which weighs its inputs, adds them up and passes on a signal, and stacking many layers lets the network represent very complex patterns.

Example: Think of a hiring committee screening résumés. The first row of reviewers each look for one simple thing: one checks for a degree, one for years of experience, one for typos. The second row combines those notes into bigger judgments: "strong technical background," "careful communicator." A final reviewer combines those into yes or no. Nobody handles the whole decision, yet together they make a nuanced one. Now imagine the committee tuning, by trial and error, how much each reviewer trusts each note.

Formally:

An artificial neuron computes a weighted sum of its inputs plus a bias, then applies an activation function g:

output = g (i \sum w_{i} x_{i} + b)

Common activations include ReLU (max(0, z)), sigmoid (squashes to 0–1) and tanh. Without a nonlinear activation, stacked layers collapse into a single linear model.
A feedforward network arranges neurons in layers: an input layer, one or more hidden layers and an output layer. "Deep" means many hidden layers.
The universal approximation theorem says a network with one sufficiently large hidden layer can approximate any reasonable function. Depth makes many functions far more efficient to represent.

Connections: A single neuron without the activation is the linear model from Part 6, Concept 3. Networks are loosely inspired by brains (Part 1, Concept 6) but are not brain simulations.

Watch out:

"Neuron" is a metaphor. Biological neurons are vastly more complex.
Representable isn't learnable. The theorem says a good network exists, not that training will find it.

Concept 2: Training with backpropagation

In one line: A network learns by making a prediction, measuring how wrong it was, and then working backward to figure out how much each weight contributed to the error, nudging every weight to do a little better.

Example: A relay team runs a race and finishes 2 seconds behind the record. The coach watches the replay backward from the finish: the anchor lost 0.5 seconds, the third runner's handoff cost 1 second, and so on back to the start. Each runner gets feedback in proportion to their share of the gap. After many races and small corrections, the team gets faster. Backpropagation does this for millions of weights at once.

Formally:

Forward pass: compute the network's output for an input.
Loss: measure the error, such as squared error for regression or cross-entropy for classification.
Backpropagation: use the chain rule of calculus to compute the gradient of the loss with respect to every weight, layer by layer from output back to input.
Gradient descent update: move each weight a small step against its gradient. Stochastic gradient descent (SGD) uses small random batches of data per step; variants like Adam adapt the step size per weight.
One pass through the whole training set is an epoch; training often takes many epochs.
Vanishing gradients: in deep networks, gradients can shrink toward zero in early layers. ReLU activations, careful initialization, normalization layers and residual connections (shortcuts that skip layers) all help.

Connections: This is gradient descent from Part 6, Concept 3, scaled up. Overfitting (Part 6, Concept 2) is countered with dropout (randomly silencing neurons during training), weight decay and more data.

Watch out:

The loss surface is not a simple bowl. Training finds a good solution, rarely the single best one.
Hyperparameters matter. Learning rate, batch size and architecture choices can make or break training.

Concept 3: Convolutional networks for images

In one line: Convolutional networks recognize images by sliding small pattern detectors across the picture, building up from edges to textures to parts to whole objects.

Example: To spot a cat in a photo, you don't memorize what a cat looks like in every possible position. You notice pointed ears, whiskers and fur texture wherever they appear. A convolutional network does something similar. Its first layer detects edges and color blobs anywhere in the image. The next combines edges into textures and curves. Deeper layers combine those into eyes and ears, and the final layers into "cat." The same ear detector works whether the cat sits top-left or bottom-right.

Formally:

A convolution slides a small grid of weights (a filter or kernel, such as 3 × 3) across the image, producing a feature map showing where the pattern appears.
Weight sharing: the same filter is used at every position, giving translation invariance and far fewer parameters than a fully connected layer.
Pooling layers shrink feature maps by summarizing small regions, building tolerance to small shifts.
Stacking layers builds a hierarchy of features, with deeper layers seeing larger regions of the image.
Transfer learning: a network pretrained on millions of images can be fine-tuned for a new task with far less data.

Connections: AlexNet's 2012 win with a convolutional network launched the deep learning era (Part 1, Concept 7). These networks are a core tool of computer vision (Part 9).

Watch out:

Networks can latch onto shortcuts. A model may learn "snow in background" means wolf rather than learning what wolves look like.
Adversarial examples. Tiny, invisible pixel changes can flip a confident prediction.

Concept 4: Sequences, attention and transformers

In one line: To understand sequences like sentences, a network needs to relate each part to the others, and attention lets every word directly look at every other word to decide which ones matter.

Example: Read: "The trophy didn't fit in the suitcase because it was too big." What does "it" refer to? The trophy. Change "big" to "small," and "it" becomes the suitcase. To get this right, the system must connect "it" with words several positions away, weighing them by relevance. Older models read word by word and struggled to carry information across long distances. Attention lets the model connect "it" to "trophy" and "suitcase" in one step and weigh which fits.

Formally:

Recurrent neural networks (RNNs) process a sequence one element at a time, carrying a hidden state forward. LSTMs add gates that help preserve information over longer spans.
Attention computes, for each position, a weighted mix of all other positions. Each word produces a query, a key and a value; relevance is the match between one word's query and another's key.
Self-attention applies this within one sequence; multi-head attention runs several attention patterns in parallel to capture different relationships.
The transformer (2017) is built from layers of self-attention and feedforward networks, with positional encodings to represent word order. It processes all positions in parallel, which makes training on huge datasets practical.
Embeddings represent each word or word fragment (token) as a vector of numbers, so similar meanings sit near each other.

Connections: Transformers trained with self-supervised learning (Part 6, Concept 4) are the foundation of large language models (Part 9).

Watch out:

Attention cost grows with the square of sequence length. Long documents are expensive, which drives much current research.
Attention weights aren't explanations. Where the model "looks" doesn't fully explain why it decided.

Concept 5: Generative models and scale

In one line: Generative models learn what real data looks like well enough to produce new examples, and making models and datasets much bigger has repeatedly unlocked new abilities.

Example: Show a model millions of photos of faces. A recognition model learns to say "face or not." A generative model learns the patterns so thoroughly that it can produce a face of a person who has never existed: plausible skin, lighting and features. Text works the same way: a model trained to predict the next word, at enormous scale, can write essays, code and poems.

Formally:

A generative model learns the probability distribution of the data so it can sample new examples. A discriminative model only learns to separate categories.
Variational autoencoders (VAEs) compress data into a compact latent space and learn to decode samples from it.
Generative adversarial networks (GANs) pit a generator against a discriminator that tries to spot fakes; each improves by competing with the other.
Diffusion models learn to reverse a gradual noising process, turning random noise into an image step by step. They power many current image generators.
Autoregressive models generate one token at a time, each conditioned on everything before it. Large language models work this way.
Scaling laws: model performance has improved predictably with more parameters, data and compute, a pattern that has guided recent AI development.

Connections: GANs are a two-player game (Part 5, Concept 5). Generated content raises the ethics questions in Part 10.

Watch out:

Fluent isn't accurate. Generative models produce plausible output, which can include confident fabrications.
Scaling trends can bend. Past improvements with scale don't guarantee future ones; data and compute both have limits.

Retention kit

Answer from memory, then check. Review at about 1 day, 3 days, 1 week, 2 weeks and 1 month.

What does a single artificial neuron compute, and why are nonlinear activations essential?
Explain backpropagation using the relay-team analogy. What problem do residual connections help with?
Why does weight sharing make convolutional networks efficient and translation-invariant?
How does attention resolve "it" in the trophy-and-suitcase sentence? What are queries, keys and values?
Name three kinds of generative models and the core idea behind each.

Build exercise (1–2 hours): Using PyTorch, train a small feedforward network on the MNIST handwritten digits. Then swap in a two-layer convolutional network and compare test accuracy and parameter count. Finally, add tiny random noise to a test image and see how confident each model stays.

Python algorithms

The three building blocks of this part in plain NumPy: a two-layer network trained with hand-written backpropagation to learn XOR, which no single linear model can (Concepts 1–2); a convolution that detects a vertical edge (Concept 3); and scaled dot-product self-attention (Concept 4). Needs Python 3 and NumPy.

import numpy as np

rng = np.random.default_rng(0)

# 1. A two-layer neural network trained with backpropagation (learns XOR)
def train_xor(hidden=8, lr=0.5, epochs=5000):
    X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
    y = np.array([[0], [1], [1], [0]], dtype=float)
    W1, b1 = rng.normal(0, 1, (2, hidden)), np.zeros(hidden)
    W2, b2 = rng.normal(0, 1, (hidden, 1)), np.zeros(1)
    sigmoid = lambda z: 1 / (1 + np.exp(-z))
    for _ in range(epochs):
        # Forward pass
        h = np.tanh(X @ W1 + b1)
        out = sigmoid(h @ W2 + b2)
        # Backward pass (chain rule), cross-entropy loss
        d_out = (out - y) / len(X)
        dW2, db2 = h.T @ d_out, d_out.sum(0)
        d_h = (d_out @ W2.T) * (1 - h ** 2)            # tanh derivative
        dW1, db1 = X.T @ d_h, d_h.sum(0)
        # Gradient descent update
        W1, b1, W2, b2 = W1 - lr * dW1, b1 - lr * db1, W2 - lr * dW2, b2 - lr * db2
    return out.ravel()

# 2. Convolution: slide a small filter across an image
def conv2d(image, kernel):
    kh, kw = kernel.shape
    H, W = image.shape[0] - kh + 1, image.shape[1] - kw + 1
    out = np.zeros((H, W))
    for i in range(H):
        for j in range(W):
            out[i, j] = (image[i:i + kh, j:j + kw] * kernel).sum()   # same weights everywhere
    return out

# 3. Scaled dot-product self-attention
def softmax(z, axis=-1):
    z = z - z.max(axis=axis, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=axis, keepdims=True)

def self_attention(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv                   # queries, keys, values
    weights = softmax(Q @ K.T / np.sqrt(K.shape[1]))  # relevance of every word to every word
    return weights @ V, weights

if __name__ == "__main__":
    print("XOR predictions for 00, 01, 10, 11:", np.round(train_xor(), 2))

    image = np.zeros((6, 6)); image[:, 3:] = 1         # dark left half, bright right half
    vertical_edge = np.array([[-1, 0, 1]] * 3)
    print("Edge-filter response (row 0):", conv2d(image, vertical_edge)[0])

    tokens = ["the", "trophy", "suitcase", "it"]
    X = rng.normal(0, 1, (4, 8))
    X[3] = X[1] + 0.3 * rng.normal(0, 1, 8)            # make "it" resemble "trophy"
    X = 2 * X / np.linalg.norm(X, axis=1, keepdims=True)  # equal-length embeddings
    I = np.eye(8)
    _, w = self_attention(X, I, I, I)
    print("Where 'it' attends:", {t: round(float(v), 2) for t, v in zip(tokens, w[3])})

Output when run:

XOR predictions for 00, 01, 10, 11: [0. 1. 1. 0.]
Edge-filter response (row 0): [0. 3. 3. 0.]
Where 'it' attends: {'the': 0.06, 'trophy': 0.33, 'suitcase': 0.24, 'it': 0.37}

The edge filter fires only where dark meets bright. In the attention demo the embeddings are random toys, with "it" built to resemble "trophy" and identity matrices standing in for learned weights, so the result shows the mechanism, not real language understanding. In a trained transformer, Wq, Wk and Wv are learned.

Try it:

Remove the hidden layer (connect inputs straight to the output) and watch XOR become impossible to learn.
Rotate the edge filter 90° (vertical_edge.T) and run it on an image with a horizontal boundary.