Artificial Intelligence: A Conceptual Learner's Series

Part 6: Machine learning

Author: Christopher DanosAI by Claude

Earlier parts assumed someone handed the agent its knowledge. This part covers how agents acquire knowledge from data instead, and how to tell whether what they learned will hold up.

Concept 1: Learning from examples

In one line: Instead of writing the rules yourself, show the computer many examples with the right answers and let it find a rule that reproduces them.

Example: You want to flag fraudulent credit card charges. Writing rules by hand ("flag any purchase over $2,000 abroad") catches some fraud and annoys honest travelers. Instead, you gather a million past transactions, each labeled fraud or not. The system finds patterns you'd never think to write down, such as a specific combination of time of day, merchant type and spending speed. When a new transaction arrives, it applies what it learned.

Formally:

Supervised learning: learn a function from inputs to outputs using labeled examples (input, correct output).
Classification predicts a category (fraud or not). Regression predicts a number (a house's price).
Features are the measurable properties of each input (amount, time, location).
The learner chooses a hypothesis h from a hypothesis space (all the functions it's allowed to consider) that fits the training data.
Unsupervised learning finds structure without labels (Concept 4). Reinforcement learning learns from rewards (Part 8).

Connections: Learning lets a rational agent improve beyond its designer's built-in knowledge (Part 1, Concept 4). It's also where the shift in AI history toward data came from (Part 1, Concept 7).

Watch out:

Labels can be wrong or biased. A model learns whatever the labels encode, including human mistakes and prejudices.
The features you choose limit what's learnable. If the deciding factor isn't in the data, no algorithm can find it.

Concept 2: Generalization and overfitting

In one line: The real goal isn't to fit the examples you have but to predict new ones well, and fitting the training data too closely is a common way to fail at that.

Example: A student preparing for an exam memorizes the answers to every practice question word for word. She scores 100% on the practice set. On the real exam, the questions are worded differently, and she struggles. Her classmate who learned the underlying ideas scored 85% on practice but does just as well on the exam. The first student overfit; the second generalized.

Formally:

Generalization: performing well on data not seen during training.
Overfitting: a model fits noise and quirks of the training set, so training error is low but error on new data is high. Underfitting: the model is too simple to capture the real pattern.
Split data into a training set (to fit), a validation set (to tune choices) and a test set (touched once, to estimate real performance).
Cross-validation rotates which part of the data is held out, giving a more reliable estimate from limited data.
Regularization penalizes model complexity, nudging the learner toward simpler explanations.
The bias–variance trade-off: simple models err systematically (bias); flexible models are sensitive to the particular training sample (variance).

Connections: Preferring simpler hypotheses is an old principle often called Ockham's razor, borrowed from philosophy (Part 1, Concept 6).

Watch out:

Never tune on the test set. Each peek leaks information, and the final score becomes optimistic.
Big models can generalize surprisingly well. Modern deep networks sometimes defy the classic trade-off, so treat it as a guide rather than a law.

Concept 3: Two classic model families

In one line: Decision trees learn by asking a series of yes/no questions, while linear models learn by weighting each feature and adding them up; together they show the core trade-offs in machine learning.

Example: A bank decides on loan applications. A decision tree works like a flowchart: Is income above $50,000? If yes, is existing debt below $10,000? If yes, approve. Anyone can read and audit it. A linear model instead gives each factor a weight (income counts +3 per $10,000, debt counts −2 per $10,000, years at job +1), adds up the score and approves above a threshold. It handles gradual trade-offs smoothly, where a tree has hard cutoffs.

Formally:

A decision tree splits the data on the feature that best separates the classes, measured by information gain (reduction in entropy), then repeats on each branch. Pruning removes branches that only fit noise.
Ensembles of trees, such as random forests and gradient-boosted trees, combine many trees and are among the strongest methods for tabular data.
A linear model predicts with a weighted sum: y = w₁x₁ + w₂x₂ + … + b. Logistic regression passes that sum through a squashing function to output a probability.
Weights are learned by minimizing a loss function (how wrong the predictions are) with gradient descent: repeatedly nudge each weight in the direction that reduces the loss.

w \leftarrow w - α \frac{\partial Loss}{\partial w}

In words: move each weight a small step (size α, the learning rate) downhill on the loss.

Connections: Gradient descent is local search (Part 2, Concept 4) on the loss surface. Neural networks (Part 7) stack many linear models with nonlinear steps between them.

Watch out:

Learning rate is a balancing act. Too large and training bounces around or diverges; too small and it crawls.
Interpretability varies. A single small tree is easy to explain; a forest of 500 trees is not.

Concept 4: Finding structure without labels

In one line: Even without right answers, a computer can discover useful structure in data, such as natural groupings or a simpler way to describe it.

Example: A grocery chain has purchase histories for two million shoppers and no labels at all. A clustering algorithm groups them by buying patterns. One group buys diapers, formula and coffee: new parents. Another buys protein bars and sports drinks: gym regulars. No one told the system these categories exist. The marketing team names them afterward and tailors offers to each.

Formally:

Clustering groups similar items. k-means picks k cluster centers, assigns each point to the nearest center, moves each center to the average of its points and repeats.
Dimensionality reduction compresses many features into a few that preserve most of the information. Principal component analysis (PCA) finds the directions along which the data varies most.
Density estimation models where data is likely to occur, useful for spotting anomalies (data that looks unlike anything seen before).
Self-supervised learning creates labels from the data itself, such as predicting the next word in a sentence. It underlies modern language models (Part 9).

Connections: Unsupervised learning often feeds supervised learning (Concept 1) by producing better features. It also relates to learning the structure of Bayesian networks (Part 4).

Watch out:

Clusters are suggestions, not truths. k-means will produce k groups whether or not the data really has them.
Choosing k is a judgment call. Different values can tell different, equally plausible stories.

Concept 5: Measuring performance honestly

In one line: A single accuracy number can badly mislead, so measure the kinds of mistakes a model makes and check that the data didn't secretly give away the answers.

Example: A model for detecting a rare disease reports 99.5% accuracy. Impressive, until you learn only 0.5% of patients have the disease. A model that says "healthy" to everyone scores 99.5% too, while catching zero cases. What you need to know is: of the sick patients, how many did it catch, and of the patients it flagged, how many were actually sick?

Formally:

A confusion matrix counts true positives, false positives, true negatives and false negatives.
Precision: of the items flagged positive, the fraction that truly are. Recall: of the truly positive items, the fraction flagged.
Raising the decision threshold usually trades recall for precision. The ROC curve and precision–recall curve show that trade-off across all thresholds.
Data leakage happens when training data contains information unavailable at prediction time, such as a "treatment started" field in a diagnosis model. It produces great test scores and useless real-world performance.
Distribution shift: the data the model sees after deployment differs from its training data, so performance quietly degrades.

Connections: Which mistake is worse is a utility question (Part 5, Concept 1): a missed cancer and a false alarm have very different costs.

Watch out:

Accuracy hides imbalance. On rare-event problems, always check precision and recall.
Too-good results deserve suspicion. A sudden jump to near-perfect scores more often means leakage than a breakthrough.

Retention kit

Answer from memory, then check. Review at about 1 day, 3 days, 1 week, 2 weeks and 1 month.

What's the difference between classification and regression, and between supervised and unsupervised learning?
Explain overfitting using the exam-memorizing example. What are training, validation and test sets each for?
How does a decision tree choose its splits? How does gradient descent update a weight?
How does k-means work, and what's one danger in trusting its clusters?
Why is accuracy misleading for rare events? Define precision and recall.

Build exercise (1–2 hours): Using scikit-learn and any public tabular dataset, train a decision tree at depths 1 through 20. Plot training accuracy and validation accuracy against depth and find where overfitting begins. Then compare against a random forest on the same split.

Python algorithms

The core models of this part written from scratch with NumPy, so nothing is hidden inside a library: a decision tree chosen by information gain and logistic regression trained by gradient descent (Concept 3), k-means with restarts (Concept 4), and precision and recall exposing the always-healthy model (Concept 5). Needs Python 3 and NumPy (pip install numpy).

import numpy as np
from collections import Counter

rng = np.random.default_rng(0)

# 1. Decision tree (ID3-style) with information gain
def entropy(labels):
    counts = np.array(list(Counter(labels).values()), dtype=float)
    p = counts / counts.sum()
    return -(p * np.log2(p)).sum()

def best_split(X, y):
    """Try every feature and threshold; keep the largest information gain."""
    base, best = entropy(y), (0, None, None)
    for f in range(X.shape[1]):
        for t in np.unique(X[:, f]):
            left = X[:, f] <= t
            if left.all() or (~left).all():
                continue
            gain = base - (left.mean() * entropy(y[left]) + (~left).mean() * entropy(y[~left]))
            if gain > best[0]:
                best = (gain, f, t)
    return best

def build_tree(X, y, depth=0, max_depth=3):
    gain, f, t = best_split(X, y)
    if depth == max_depth or gain == 0:
        return Counter(y).most_common(1)[0][0]           # leaf: majority class
    left = X[:, f] <= t
    return (f, t, build_tree(X[left], y[left], depth + 1, max_depth),
                  build_tree(X[~left], y[~left], depth + 1, max_depth))

def predict_tree(tree, x):
    while isinstance(tree, tuple):
        f, t, left, right = tree
        tree = left if x[f] <= t else right
    return tree

# 2. Logistic regression trained by gradient descent
def train_logistic(X, y, lr=0.1, epochs=2000):
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(epochs):
        p = 1 / (1 + np.exp(-(X @ w + b)))               # predicted probability
        grad_w, grad_b = X.T @ (p - y) / len(y), (p - y).mean()
        w, b = w - lr * grad_w, b - lr * grad_b          # step downhill on the loss
    return w, b

# 3. k-means clustering
def kmeans(X, k, iters=50, seed=0, restarts=10):
    """Run several random starts and keep the tightest clustering."""
    runs = [kmeans_once(X, k, iters, seed + i) for i in range(restarts)]
    return min(runs, key=lambda run: run[2])[:2]

def kmeans_once(X, k, iters, seed):
    r = np.random.default_rng(seed)
    centers = X[r.choice(len(X), k, replace=False)]
    for _ in range(iters):
        labels = np.argmin(((X[:, None] - centers) ** 2).sum(-1), axis=1)   # nearest center
        new = np.array([X[labels == j].mean(0) if (labels == j).any() else centers[j] for j in range(k)])
        if np.allclose(new, centers):
            break
        centers = new
    inertia = ((X - centers[labels]) ** 2).sum()      # total squared distance
    return centers, labels, inertia

# 4. Honest evaluation
def precision_recall(y_true, y_pred):
    tp = int(((y_pred == 1) & (y_true == 1)).sum())
    fp = int(((y_pred == 1) & (y_true == 0)).sum())
    fn = int(((y_pred == 0) & (y_true == 1)).sum())
    return tp / max(tp + fp, 1), tp / max(tp + fn, 1)

if __name__ == "__main__":
    # Loan data: [income in $10k, debt in $10k]; approve if income is high and debt low
    X = rng.uniform(0, 10, size=(400, 2))
    y = ((X[:, 0] > 5) & (X[:, 1] < 4)).astype(int)
    train, test = slice(0, 300), slice(300, 400)

    tree = build_tree(X[train], y[train])
    acc = np.mean([predict_tree(tree, x) == t for x, t in zip(X[test], y[test])])
    print(f"Decision tree test accuracy: {acc:.2f}")

    w, b = train_logistic(X[train], y[train])
    pred = ((X[test] @ w + b) > 0).astype(int)
    print(f"Logistic regression test accuracy: {np.mean(pred == y[test]):.2f}, weights {np.round(w, 2)}")

    blobs = np.vstack([rng.normal(c, 0.5, size=(50, 2)) for c in ([0, 0], [5, 5], [0, 5])])
    centers, _ = kmeans(blobs, 3)
    print("k-means centers:", np.round(centers[np.argsort(centers[:, 0] + centers[:, 1])], 1).tolist())

    # Rare disease: a model that always says "healthy"
    y_true = np.array([1] * 5 + [0] * 995)
    lazy = np.zeros(1000, dtype=int)
    print("Always-healthy accuracy:", np.mean(lazy == y_true), "| precision, recall:", precision_recall(y_true, lazy))

Output when run:

Decision tree test accuracy: 0.98
Logistic regression test accuracy: 0.89, weights [ 1.09 -1.73]
k-means centers: [[-0.0, -0.0], [-0.1, 4.9], [4.9, 4.9]]
Always-healthy accuracy: 0.995 | precision, recall: (0.0, 0.0)

The tree beats logistic regression here because the true rule is two hard cutoffs, which a tree represents exactly and a single straight line can't. The weights still make sense: income pushes toward approval, debt pushes against it.

Try it:

Set restarts=1 in kmeans and try a few seeds: some runs put two centers in one cluster.
Build trees with max_depth from 1 to 10 on noisy labels (flip 10% of y) and compare training versus test accuracy to find where overfitting starts.