AI Expert Roadmap

76
222Total
146Done
47Partial
29Todo
CONCEPTPRACTICALSTRATEGICEVAL
Phase 1 of 6

Foundations

Mental models for how AI systems work

73%
20Done
7Partial
5Todo
◐CONCEPT
AI vs ML vs deep learning vs generative AI — the taxonomy
→Write a taxonomy doc comparing your 12+ projects by AI type — classify each (Dosh=GenAI text+vision, ServicePulse=ML classification pipeline, FPL BlackBox=GenAI+edge ML, Local LLM Stack=self-hosted inference)2 hours
  1. Create a spreadsheet with columns: project, AI type, model(s) used, input modality, output type
  2. Classify each project: pure GenAI, ML pipeline, hybrid, or rule-based with AI augmentation
  3. Write a 1-page summary explaining where each project sits on the AI taxonomy and why
  4. Identify gaps — which AI types are you not using yet (e.g. reinforcement learning)?
Google's ML Crash Course: https://developers.google.com/machine-learning/crash-courseStanford CS229 lecture 1 (Andrew Ng) — the taxonomy explained in 20 minutes
○CONCEPT
Neural networks: weights, activations, backpropagation
→Follow Karpathy's Zero to Hero series, then train a tiny classifier on your Dosh transaction categories using PyTorch on LLM-Studio1 week evenings
  1. Watch Karpathy's 'The spelled-out intro to neural networks' (2.5 hours) — code along in a Jupyter notebook
  2. Export 500 labelled Dosh transactions from SQLite with their categories
  3. Build a simple 2-layer neural network in PyTorch that classifies transactions by category
  4. Train it on LLM-Studio, plot loss curves, and compare accuracy to your Haiku prompt-based categoriser
  5. Write up what backpropagation actually does in plain English — test by explaining it to someone
Karpathy's Neural Networks: Zero to Hero playlist: https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ3Blue1Brown 'Neural Networks' series (4 videos, ~1 hour total): https://www.3blue1brown.com/topics/neural-networksPyTorch 60-minute blitz: https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html
◐CONCEPT
Supervised / unsupervised / reinforcement learning
→Your Dosh categoriser is supervised learning (labelled transactions). Add k-means clustering (unsupervised) to automatically group similar transactions and discover spending patterns you haven't labelled yet1 afternoon
  1. Export Dosh transactions with embeddings (or generate embeddings with Qwen3-Embedding-4B on the KC-Studio embed server)
  2. Run k-means clustering (scikit-learn) with k=10,20,30 and compare cluster quality with silhouette scores
  3. Visualise clusters with UMAP or t-SNE — do they match your existing categories?
  4. Write a comparison: supervised (your Haiku categoriser) vs unsupervised (clusters) — when would you use each?
scikit-learn clustering guide: https://scikit-learn.org/stable/modules/clustering.htmlGoogle's ML Crash Course — Clustering module: https://developers.google.com/machine-learning/clustering
◐CONCEPT
Transformer architecture — why it matters
→Watch Karpathy's 'Let's build GPT' while running inference on LLM-Studio with MLX — pause at each architecture component and map it to what's happening in your local Gemma 4 26B-A4B model1 afternoon
  1. Watch Karpathy's 'Let's build GPT from scratch' (2 hours) — take notes on each component
  2. Run Gemma 4 26B-A4B on LLM-Studio via MLX and inspect the model config (layers, heads, dimensions)
  3. Map paper concepts to real values: how many attention heads? What's the embedding dimension? How many parameters per layer?
  4. Write a 1-pager: 'What happens inside Gemma when I send it a prompt' — trace the full path
Karpathy's 'Let's build GPT': https://www.youtube.com/watch?v=kCc8FmEb1nYThe Illustrated Transformer by Jay Alammar: https://jalammar.github.io/illustrated-transformer/
✓CONCEPT
Tokens, embeddings, and vector spacesclick to expand
✓CONCEPT
Temperature, top-p, top-k — how sampling worksclick to expand
✓CONCEPT
Context windows: limits, growth, implicationsclick to expand
◐CONCEPT
Training vs inference — economics and trade-offs
Done so far: Inference side done: sized the Qwen lane from 30 days of InfluxDB request data (41k requests, 1.57M output tokens; busiest hour used 18% of capacity) and costs cloud alternatives per task before choosing a failover model.
→Cover the training half: estimate what a LoRA fine-tune would cost on LLM-Studio (electricity and machine time) and compare it with the inference spend you have already measured, so the 'where money goes' picture includes training2 hours
  1. Take the inference baseline you already have: monthly requests, tokens and cost per lane from InfluxDB, plus cloud-alternative pricing per task
  2. Get a reference training cost: the per-run cost of the Replicate FLUX LoRA trainings you already pay for
  3. Measure LLM-Studio's wattage during a small MLX LoRA run and multiply by the hours a realistic fine-tune would need
  4. Write a summary: 'My AI economics — training vs inference, where money goes and where local saves it'
Anthropic pricing page: https://www.anthropic.com/pricingArticles on the economics of inference cost — search the reader-mcp feeds for 'inference cost'
✓CONCEPT
Fine-tuning vs prompting vs RAG — when to use whichclick to expand
◐CONCEPT
RLHF — what it is and why it matters
→Read the Constitutional AI paper (Anthropic, 2022) — you reference it in your prompt library constantly. Your ha-log-agent's risk-based approval tiers are a practical form of alignment via human feedback1 hour
  1. Read the Constitutional AI paper — focus on sections 2 (method) and 4 (results), skip appendix (45 minutes)
  2. Map ha-log-agent's approval tiers to the paper's concepts: where is the 'constitution'? Where is the 'human feedback'?
  3. Write a short note: 'How RLHF connects to what I build' — reference Telegram bot, ha-log-agent, and prompt library
Constitutional AI paper: https://arxiv.org/abs/2212.08073Anthropic's RLHF blog post: https://www.anthropic.com/research/training-a-helpful-and-harmless-assistantChip Huyen's RLHF explainer: https://huyenchip.com/2023/05/02/rlhf.html
7/8▼
✓CONCEPT
GPT-4o/5, Claude, Gemini, Llama, Mistral, Qwen — capabilities and trade-offsclick to expand
✓CONCEPT
Open vs closed source modelsclick to expand
✓CONCEPT
Multimodal models — vision, audio, videoclick to expand
✓CONCEPT
Model benchmarks: MMLU, HumanEval, HELM, LMSys Arenaclick to expand
○CONCEPT
Scaling laws — compute, data, parameters
→Use your reader-mcp feeds (news-monitor) to flag scaling laws papers. Read the Chinchilla paper and 2 follow-ups — your feeds already track DeepMind and Anthropic who publish on this3 hours
  1. Search reader-mcp for 'scaling laws' articles from the last 6 months — pick the 3 that news-monitor scored highest
  2. Read the Chinchilla paper (Hoffmann et al., 2022) — focus on the key finding: optimal compute allocation
  3. Read one follow-up (e.g. 'Scaling Data-Constrained Language Models') to see how the field has evolved
  4. Write a summary: 'What scaling laws mean for my model choices' — why Gemma 4 26B-A4B exists, why Haiku is cheap
Chinchilla paper: https://arxiv.org/abs/2203.15556Scaling Data-Constrained LMs: https://arxiv.org/abs/2305.16264Epoch AI scaling laws dashboard: https://epochai.org/data/notable-ai-models
✓CONCEPT
Reasoning models: o3, DeepSeek-R1, extended thinkingclick to expand
✓CONCEPT
SLMs — Phi, Gemma, Qwen 0.5B — when small beats bigclick to expand
✓STRATEGIC
How to evaluate and select models for use casesclick to expand
○CONCEPT
Attention mechanism — self-attention, multi-head, cross-attention
→Local LLM Stack — add attention weight visualisation to Open WebUI or use BertViz with a small model. MLX exposes attention weights from Gemma 4 26B-A4B during inference1 afternoon
  1. Watch 3Blue1Brown's 'Attention in transformers, visually explained' (26 min)
  2. Install BertViz (pip install bertviz) and visualise attention patterns on a small BERT model with your own text
  3. Use MLX's model inspection to extract attention weights from Gemma 4 26B-A4B on a sample prompt
  4. Compare attention patterns across layers — which heads attend to syntax vs semantics?
  5. Write a 1-pager explaining self-attention, multi-head, and cross-attention with your own examples
3Blue1Brown 'Attention in transformers': https://www.youtube.com/watch?v=eMlx5fFNoYcBertViz interactive attention visualisation: https://github.com/jessevig/bertvizThe Illustrated Transformer (attention section): https://jalammar.github.io/illustrated-transformer/
○CONCEPT
Positional encoding — absolute, relative, RoPE
→Karpathy's Zero to Hero covers positional encoding. Compare how Gemma 4 26B-A4B (uses RoPE) handles long context vs older absolute-position models. Your local stack can test this empirically2 hours
  1. Watch the positional encoding section of Karpathy's 'Let's build GPT' (timestamp ~45 min)
  2. Read the RoPE paper abstract + method section (15 minutes)
  3. Test Gemma 4 26B-A4B on prompts with key info at positions 100, 1000, and 3000 tokens — does retrieval accuracy degrade?
  4. Compare with Qwen 3.6 35B-A3B (mlx_lm) (also RoPE) at the same positions — do they behave similarly?
RoPE paper (Su et al., 2021): https://arxiv.org/abs/2104.09864Karpathy's 'Let's build GPT' positional encoding section: https://www.youtube.com/watch?v=kCc8FmEb1nY&t=2700sEleuther AI blog on positional encodings: https://blog.eleuther.ai/rotary-embeddings/
✓CONCEPT
KV cache — what it is and why it matters for inference costclick to expand
✓CONCEPT
Mixture of Experts (MoE) — how Mixtral and GPT-4 likely workclick to expand
◐CONCEPT
Speculative decoding — how it speeds up inference
Done so far: Tested a dense Qwen 27B MTP (multi-token prediction) build on MLX — 3.2x slower in aggregate and rejected — and found Qwen 3.6's MTP head unrunnable because no mlx-lm implements it. A classic draft-model setup is still untested.
→Local LLM Stack — try mlx-lm `--draft-model` with a small Qwen 3 draft model against the Qwen 3.6 35B-A3B lane on LLM-Studio, scored through workload_eval so that both tokens/second and output quality are measured2 hours
  1. Read the speculative decoding paper abstract + method (15 minutes)
  2. Download a small standalone Qwen 3 model (same tokenizer family as Qwen 3.6) to LLM-Studio via MLX as the draft model
  3. Serve Qwen 3.6 35B-A3B with `mlx_lm.server --draft-model` on a spare port by borrowing the production lane briefly (both lanes are resident, so pause its health monitor first)
  4. Benchmark with workload_eval: baseline vs draft-assisted, recording tokens/second, draft acceptance and det_score per task
  5. Document the speedup and when it helps most (long outputs vs short answers)
Speculative decoding paper (Leviathan et al.): https://arxiv.org/abs/2211.17192mlx-lm speculative decoding: https://github.com/ml-explore/mlx-lm
○PRACTICAL
Watch Karpathy's "Let's build GPT" and implement along with it
→Do it on LLM-Studio. Train a tiny GPT on your iMessage corpus (already indexed in Chroma) — a character-level model that generates text in your writing style1 full afternoon (4 hours)
  1. Set aside a Saturday afternoon. Watch the full 2-hour video, pausing to code along in a Jupyter notebook
  2. Export your iMessage text from the Chroma personal_docs collection as training data
  3. Replace Shakespeare with your iMessage data — train a character-level GPT
  4. Generate sample outputs and compare with your actual writing style
  5. Write up: 'What I learned building a GPT from scratch' — key insights about attention, training, and loss
Karpathy's 'Let's build GPT from scratch': https://www.youtube.com/watch?v=kCc8FmEb1nYKarpathy's nanoGPT repo: https://github.com/karpathy/nanoGPTMLX training examples: https://github.com/ml-explore/mlx-examples
◐EVAL
MILESTONE: Explain transformer inference pipeline from token to probability
Done so far: Built and documents the full serving path — Open WebUI → llm-proxy → MLX servers — with per-lane queues, failover, per-request token/TTFT logging and real-completion health probes, so he can show where latency goes (queueing, prefill, decode). The model-internals half (token → embedding → attention → logits) is not yet written up.
→Write it up as an explainer-kit video or a wiki concept article (`~/wiki/concepts/`) walking through exactly what happens when Gemma 4 26B-A4B processes a query — from raw text to tokenisation to embeddings to attention (and MoE expert routing) to output probabilities — using its real config values3 hours
  1. Draft the outline: tokenisation → embedding lookup → positional encoding → N transformer blocks (attention + expert routing) → layer norm → logits → sampling
  2. For each step, reference the actual Gemma 4 26B-A4B config values (vocab size, hidden dim, num layers, num heads, experts and experts per token)
  3. Add diagrams showing tensor shapes at each stage
  4. Have someone technical read it and ask questions — if you can answer, you understand it
Andrej Karpathy's 'Let's build GPT' as reference: https://www.youtube.com/watch?v=kCc8FmEb1nYThe Illustrated GPT-2 by Jay Alammar: https://jalammar.github.io/illustrated-gpt2/