Artificial Intelligence: A Conceptual Learner's Series
Part 9: Language, vision and robotics
This part shows how the methods from earlier parts come together in three applied areas: understanding language, seeing the world and acting physically in it.
Concept 1: Language models
In one line: A language model learns how likely each next word is given the words before it, and a big enough model, trained on enough text, picks up grammar, facts and a surprising amount of reasoning along the way.
Example: Your phone's keyboard suggests "you" after "thank." That's a tiny language model: it has counted which words tend to follow which. Now scale that idea up enormously. A model trained on a vast amount of text, with billions of parameters, predicting the next word of "The capital of France is" must have absorbed the fact. Predicting the next line of a proof requires following the logic. Prediction at scale forces the model to learn a lot about the world.
Formally:
A language model assigns probabilities to sequences of tokens (words or word fragments), usually by predicting each token from the ones before it.
n-gram models count sequences of n words. They are simple but can't capture long-range context.
Large language models (LLMs) are transformers (Part 7, Concept 4) trained by self-supervised learning on next-token prediction over massive text collections. This stage is called pretraining.
Fine-tuning then adapts the model, for example with instruction tuning on example conversations and RLHF (Part 8, Concept 5) on human preferences.
Generation samples one token at a time; a temperature setting controls how random the choices are.
Connections: LLMs combine embeddings, attention and scale from Part 7 with reinforcement learning from Part 8.
Watch out:
Fluency isn't truth. Models can state false things confidently, often called hallucination.
Training data shapes everything. Biases, gaps and errors in the text carry into the model.
Concept 2: Putting language models to work
In one line: A pretrained language model becomes useful for a specific job through careful instructions, connections to outside information and tools, and checks on its output.
Example: A company wants a support assistant that answers questions about its own products. A general model knows nothing about their return policy. So the system first searches the company's help articles for passages related to the customer's question, then gives those passages to the model along with the question and instructions to answer only from them. If the model needs an order status, it calls the company's order-lookup tool instead of guessing.
Formally:
Prompting: steering behavior through instructions and examples in the input. Showing a few worked examples is called few-shot prompting.
Retrieval-augmented generation (RAG): retrieve relevant documents, often by comparing embeddings, and include them in the prompt so answers are grounded in specific sources.
Tool use: the model emits structured calls to external functions (search, calculators, databases) and uses the results.
Agents: language models placed in a loop of planning, acting with tools and observing results, pursuing multi-step goals.
Evaluation: test outputs systematically against reference answers, rubrics or other checks, since a single impressive demo says little about reliability.
Connections: A tool-using LLM agent is a rational agent (Part 1, Concept 4) whose percepts are text and tool results and whose actions are text and tool calls.
Watch out:
Retrieval quality caps answer quality. If the right passage isn't retrieved, the model can't use it.
Untrusted text can carry instructions. Documents and web pages fed to a model may try to redirect it, a risk called prompt injection.
Concept 3: Computer vision
In one line: Computer vision turns raw pixels into useful descriptions of a scene: what objects are present, where they are, and how far away.
Example: A self-checkout camera watches items go by. It must answer several questions at once. Is this a banana or a plantain (classification)? Where exactly is each item in the frame (detection)? Which pixels belong to the bag and which to the hand holding it (segmentation)? Did the shopper scan the item before bagging it (tracking over time)? Each question is a different vision task, often handled by one network with several outputs.
Formally:
Image classification: assign a label to the whole image.
Object detection: find each object and draw a bounding box around it with a label.
Semantic segmentation: label every pixel by category. Instance segmentation separates individual objects of the same category.
Depth estimation and 3D reconstruction recover the scene's geometry from one or more images, using cues like stereo disparity and motion.
Most modern systems use convolutional networks (Part 7, Concept 3) or vision transformers, which apply attention to patches of the image.
Multimodal models connect images and text in a shared embedding space, enabling search by description and image captioning.
Connections: Vision supplies the percepts for many agents, especially robots (Concepts 4 and 5). Its failures illustrate the evaluation pitfalls from Part 6, Concept 5.
Watch out:
Benchmarks flatter. Accuracy on a curated dataset often drops in unusual lighting, angles or environments.
Uneven performance across groups. Vision systems have shown higher error rates for underrepresented groups in their training data (Part 10).
Concept 4: Robot perception and localization
In one line: A robot has to work out where it is and what's around it from noisy sensors, and it often has to build a map while using that same map to find itself.
Example: A robot vacuum is placed in a new home. It has never seen the floor plan. As it moves, its wheel sensors say it traveled 2 meters, but wheels slip, so that's only approximate. Its laser scanner sees a wall ahead. It uses the wall to correct its position estimate and adds the wall to its map. After a few rooms, it recognizes the doorway it started from and fixes the small errors that accumulated along the way.
Formally:
Robots sense with cameras, lidar (laser range-finding), sonar, wheel encoders and inertial measurement units (IMUs).
Localization: estimate the robot's position given a map, typically by filtering (Part 4, Concept 5) with Kalman or particle filters. Particle-filter localization is often called Monte Carlo localization.
Mapping: build a map given known positions.
SLAM (simultaneous localization and mapping): do both at once, when neither the map nor the position is known.
Loop closure: recognizing a previously visited place, which allows accumulated errors to be corrected across the whole map.
Connections: This is reasoning over time (Part 4, Concept 5) applied to physical position, and a POMDP-style belief state (Part 5, Concept 4) in practice.
Watch out:
Errors accumulate without correction. Dead reckoning from wheel sensors alone drifts steadily.
Changing environments confuse maps. Moved furniture or crowds can make familiar places look new.
Concept 5: Robot planning and control
In one line: Once a robot knows where it is, it has to plan a collision-free path and then control its motors precisely enough to follow it, correcting continuously as reality differs from the plan.
Example: A robot arm in a warehouse must pick a mug from a cluttered shelf. First it plans a path for its joints that reaches the mug without hitting the shelf or neighboring items. Then, as it moves, small errors creep in: the motor overshoots slightly, the mug isn't exactly where the camera said. A control loop measures the gap between where the hand is and where it should be, many times per second, and corrects. Finally, it grips with just enough force to hold the mug without crushing it.
Formally:
Configuration space: the space of all possible robot poses (for an arm, all combinations of joint angles). Obstacles become forbidden regions in this space.
Motion planning finds a path through free configuration space. Sampling-based planners, such as probabilistic roadmaps and rapidly exploring random trees (RRTs), handle high-dimensional spaces by sampling random poses.
Trajectory optimization refines a path to be smooth, fast and energy-efficient.
Feedback control corrects deviations continuously. A PID controller combines proportional, integral and derivative terms of the error.
Learning-based control uses reinforcement learning or imitation of human demonstrations, increasingly combined with classical planning.
Connections: Motion planning is search (Part 2) in a continuous space, and feedback control comes from control theory (Part 1, Concept 6).
Watch out:
Physical mistakes are expensive. A bad action can damage hardware or hurt people, so robots need safety limits beyond what software agents do.
Manipulation is harder than it looks. Tasks easy for a toddler, like folding a towel, remain difficult for robots, while tasks hard for people, like chess, are easy for computers.
Retention kit
Answer from memory, then check. Review at about 1 day, 3 days, 1 week, 2 weeks and 1 month.
What does a language model predict, and what are pretraining and fine-tuning?
How does retrieval-augmented generation work, and what's one risk of feeding outside text to a model?
Distinguish classification, detection and segmentation using the self-checkout example.
What is SLAM, and why does loop closure matter?
What is configuration space? What does a feedback controller do that a plan alone can't?
Build exercise (1–2 hours): Build a tiny retrieval-augmented question answerer. Split a long public-domain text into paragraphs, embed them with an open-source embedding model, retrieve the top three paragraphs for a question by cosine similarity, and print them alongside the question. Test with questions whose answers are and aren't in the text, and note when retrieval fails.
Python algorithms
Miniature versions of each system in this part: a bigram language model with temperature (Concept 1), TF-IDF retrieval for RAG (Concept 2), a particle filter that localizes a robot in a hallway (Concept 4), and a PID controller (Concept 5). Runs with plain Python 3.
import math, random
from collections import Counter, defaultdict
# 1. A bigram language model: predict the next word from the previous one
def train_bigram(text):
words = text.lower().split()
counts = defaultdict(Counter)
for a, b in zip(words, words[1:]):
counts[a][b] += 1
return counts
def generate(model, start, length=8, temperature=1.0, seed=0):
rng, word, out = random.Random(seed), start, [start]
for _ in range(length):
options = model.get(word)
if not options:
break
weights = [c ** (1 / temperature) for c in options.values()] # temperature sharpens/flattens
word = rng.choices(list(options), weights=weights)[0]
out.append(word)
return " ".join(out)
# 2. Retrieval for RAG: TF-IDF vectors and cosine similarity
def tfidf_index(docs):
tokenized = [d.lower().split() for d in docs]
df = Counter(w for doc in tokenized for w in set(doc))
idf = {w: math.log(len(docs) / df[w]) + 1 for w in df}
vecs = [{w: c * idf[w] for w, c in Counter(doc).items()} for doc in tokenized]
return vecs, idf
def cosine(a, b):
dot = sum(v * b.get(w, 0) for w, v in a.items())
na, nb = math.sqrt(sum(v * v for v in a.values())), math.sqrt(sum(v * v for v in b.values()))
return dot / (na * nb) if na and nb else 0.0
def retrieve(query, docs, k=1):
vecs, idf = tfidf_index(docs)
q = {w: c * idf.get(w, 0) for w, c in Counter(query.lower().split()).items()}
ranked = sorted(range(len(docs)), key=lambda i: cosine(q, vecs[i]), reverse=True)
return [docs[i] for i in ranked[:k]]
# 3. Particle filter: a robot localizing itself in a hallway with doors
DOORS = {2, 3, 7}
def particle_filter(moves, sensed, n=1000, length=10, seed=0):
rng = random.Random(seed)
particles = [rng.uniform(0, length) for _ in range(n)]
for move, saw_door in zip(moves, sensed):
particles = [(p + move + rng.gauss(0, 0.2)) % length for p in particles] # predict
weights = [0.8 if (round(p) in DOORS) == saw_door else 0.2 for p in particles] # weigh by sensor
particles = rng.choices(particles, weights=weights, k=n) # resample
return Counter(round(p) % length for p in particles).most_common(1)[0][0]
# 4. PID controller: steer a value toward a target
def pid(target=10.0, kp=0.6, ki=0.05, kd=0.2, steps=40):
value, integral, prev_err = 0.0, 0.0, target
for _ in range(steps):
err = target - value
integral += err
control = kp * err + ki * integral + kd * (err - prev_err)
value += control * 0.5 # simple system response
prev_err = err
return value
if __name__ == "__main__":
corpus = ("the robot sees the door the robot moves to the door "
"the door opens and the robot enters the room")
print("Generated:", generate(train_bigram(corpus), "the"))
docs = ["Returns are accepted within 30 days with a receipt.",
"Shipping takes 3 to 5 business days.",
"Gift cards cannot be returned or exchanged."]
print("Retrieved for 'can I return this within 30 days':",
retrieve("can i return this within 30 days", docs))
# Robot starts at 1 (unknown to it), moves 1 per step: sees door, door, no door
estimate = particle_filter(moves=[1, 1, 1], sensed=[True, True, False])
print(f"Particle filter: most likely position {estimate} (true position 4)")
print(f"PID final value: {pid():.2f} (target 10)")Output when run:
Generated: the door opens and the door the door the
Retrieved for 'can I return this within 30 days': ['Returns are accepted within 30 days with a receipt.']
Particle filter: most likely position 4 (true position 4)
PID final value: 10.08 (target 10)The bigram model's babble shows why scale matters: with one word of context and a tiny corpus, it loops. The particle filter starts with no idea where it is; seeing two doors in a row then no door only fits one stretch of hallway, so the particles collapse onto the right spot.
Try it:
Generate with
temperature=0.3andtemperature=3and compare how repetitive versus random the text gets.Change the sensor readings to
[True, False, False]and print the top three particle positions: the robot can no longer tell where it is.