Joyous Brands

Artificial Intelligence: A Visual Learner's Series

Part 9: Language, vision and robotics

Author: Christopher DanosAI by Claude

How the methods from earlier parts come together to read, see and act in the physical world.

Language models

In one line: Predict the next word well enough, at a large enough scale, and a model picks up grammar, facts and a lot of reasoning.

  Huge text collection ─▶ Pretraining      ─▶ Fine-tuning               ─▶ Assistant
                          predict next token   example chats + human feedback

  • "The capital of France is …" can only be predicted by a model that has absorbed the fact.

  • Tokens are words or word pieces; temperature controls how random generation is.

Watch out: Fluent isn't true; models can state falsehoods confidently (hallucination).

Putting language models to work

In one line: A general model becomes useful for a specific job through instructions, outside information, tools and checks.

Retrieval and tools ground the model in real informationThe model answers from passages it was handed, and calls a tool instead of guessingCustomer question“Can I return this?”Retrievefind related passagesLanguage modelquestion + passagesGrounded answerbased on the policyHelp articlesthe company's own docsOrder lookup toolreal order statustool call and result

  • Prompting steers with instructions and examples; RAG retrieves relevant passages; tool use fetches facts the model can't know; agents loop through plan, act, observe.

Watch out: Outside text can contain instructions aimed at the model (prompt injection), so treat it as data.

Computer vision

In one line: Turn pixels into useful descriptions: what's there, where it is, and its shape.

Three questions a vision system can ask about the same picturegroceriesClassificationone label for the whole imageapplecereal boxDetectiona labeled box around each objectSegmentationevery pixel gets a category

  • Further tasks: depth estimation and 3D reconstruction, plus tracking objects across video frames.

  • Built on convolutional networks or vision transformers; multimodal models link images and text.

Watch out: Accuracy on curated benchmarks often drops in unusual lighting, angles or settings.

Robots: knowing where you are, and moving well

In one line: A robot estimates where it is from noisy sensors, plans a collision-free path and corrects continuously as it moves.

A robot loops: sense, estimate, plan, actThe loop repeats many times per second, correcting as reality drifts from the planSensecamera, lidar, wheel sensorsLocalize and mapwhere am I, what's here?Plan a pathroute that avoids obstaclesControl motorsfeedback corrects errors

  • SLAM builds a map while locating itself in it; loop closure (recognizing a place seen before) fixes accumulated drift.

  • Motion planning searches configuration space with planners like RRTs; PID controllers correct the gap between where the arm is and where it should be.

Watch out: Folding a towel is still harder for robots than winning at chess.

Self-test

  1. Redraw the retrieval diagram for a study assistant that answers from your own notes.

  2. Sketch a street photo and mark what classification, detection and segmentation would each output.

  3. Draw the robot loop for a lawn mower and name one sensor and one failure for each stage.