AI Reading List

This is my reading list for building deeper intuition about modern LLMs: what they are, why they work, how they are trained, what might be happening inside them, and how they are served in production.

1. Neural Network Foundations

• Neural Networks: Zero to Hero — Andrej Karpathy
  • Build the base intuition: tensors, gradients, backpropagation, MLPs, embeddings, optimization, and language modeling.
• 3Blue1Brown Neural Networks Series
  • Visual intuition for gradients, backpropagation, and what neural networks are doing geometrically.

2. Transformer Foundations

• The Illustrated Transformer — Jay Alammar
  • Conceptual pass on tokens, embeddings, attention, encoder/decoder structure, and transformer blocks.
• The Annotated Transformer — Harvard NLP
  • Code-level walkthrough of the original transformer.
• Attention Is All You Need
  • Original transformer paper. Read for the core architecture, not modern training practice.

3. Build a Tiny LLM

• Let’s build GPT from scratch — Andrej Karpathy
  • Practical bridge from transformer theory to a working language model.
• nanoGPT
  • Minimal real GPT training code: small enough to understand and real enough to matter.

4. Scaling and Pretraining

• GPT-3: Language Models are Few-Shot Learners
  • Scaling, few-shot prompting, and in-context learning.
• Training Compute-Optimal Large Language Models / Chinchilla
  • Compute/data tradeoffs and why more data can beat simply making the model larger.
• How to Scale Your Model
  • Systems spine for scaling models. Use chapters 1, 4, 7, and 12.

5. Modern Open LLM Recipes

• Llama 2 Technical Report
  • Training, instruction tuning, RLHF, safety, and evaluation.
• Llama 3 Technical Report
  • Modern recipe covering data quality, scale, post-training, safety, and evaluation.

6. Post-Training and Alignment

• InstructGPT
  • Core RLHF and instruction-following paper.
• Direct Preference Optimization
  • Preference optimization as a simpler alternative to full RLHF.
• Constitutional AI
  • AI feedback and written behavioral principles for training.

7. Mechanistic Interpretability

• A Mathematical Framework for Transformer Circuits
  • Attention heads, residual streams, circuits, and internal mechanisms.
• In-context Learning and Induction Heads
  • A concrete example of models learning algorithmic behavior.
• Toy Models of Superposition
  • Why models represent many features in compressed internal spaces.

8. Inference and Serving Systems

• Learn Inference
  • Serving generative models in production: TTFT, throughput, hardware, engines, optimization, multimodality, autoscaling, and deployment.
• Inference Engineering
  • Runtime, infrastructure, tooling, product requirements, hardware, optimization, modalities, and operations.
• The Memory Wall: Interactive Explainer
  • Weights, KV-cache, bandwidth, context length, batching, latency, and throughput.
• LLM Engineer’s Almanac
  • Workload shapes, batching, token timing, quantization, speculative decoding, and performance rooflines.

9. GPU and Kernel Performance

• Learn Kernels
  • From one inference request down to CUDA/Triton kernels and back out to distributed serving; covers profiling, KV-cache systems, quantization, speculative decoding, topology, and accelerators.
• AI Performance Engineering
  • Index for deeper GPU, accelerator, and AI performance study.

10. Agents and Production AI Systems

• The Hitchhiker’s Guide to Agentic AI
  • LLM substrate, reasoning, memory, tool use, multi-agent systems, evaluation, and production. Read chapters 1–2 and selected parts of chapter 11.