How is an LLM made?
From raw text predictor to usable assistant. We'll walk through pre-training, supervised fine-tuning, RLHF and Constitutional AI. Then: how the Transformer and attention work, and what the context window, the working memory, means for your daily prompts.
What you'll learn: the three training steps, RAG vs fine-tuning, attention, context window, “lost in the middle”, and three very concrete prompt rules of thumb that follow from this.
Lesson 2.1: The three training steps
Goal: know the difference between “completing text” and “becoming a chatbot”.
Every modern model you use (ChatGPT, Claude, Gemini) has gone through three steps:
Read ~everything ever digitised
- Goal: predict the next token.
- Data: web, books, papers, code (15–30 trillion tokens).
- Duration: months, millions of euros.
- Result: complete language, but it doesn't know what you want.
Teach it to answer questions
- Humans write tens of thousands of question→answer pairs.
- The model learns the right format and tone.
- Also called: instruction tuning, dialog data.
- Result: a chatbot that explains instead of completing the sentence.
Teach it what we prefer
- Humans rate two answers; preference teaches 'alignment'.
- This is also where most safety work sits.
- DPO is a more efficient variant of RLHF.
- Result: ChatGPT, Claude, Gemini as you know them.
A model's “personality”, does it lean toward caution? Does it probe further? Does it write directly or only after nuance?, largely sits in step 3.
Questions for lesson 2.1
1. Why doesn't “raw pre-training alone” work as a chatbot?
2. Where does most “safety” and style work sit in a modern LLM?
Lesson 2.2: Domain fine-tuning vs RAG
Goal: understand why “medical LLM” sounds attractive but is often not the best route.
If you want to make an LLM “medical”, you have two fundamentally different routes:
Training on medical text
- Continued training on large medical corpora.
- Examples: Med-PaLM, MedGemma, MedLM.
- Modern alternative: LoRA, small adapters instead of retraining the whole model.
- Trade-off: sharper here, sometimes worse elsewhere. And the model still doesn't 'know' exact guideline citations.
Pasting documents into the prompt
- Documents (guideline, paper, record excerpt) in a vector database.
- On a question: retrieve + put in the prompt + let the LLM answer based on that source.
- No retraining needed; can be kept up to date.
- Often a better choice for clinical use cases, verifiable, auditable, traceable to the source.
Rule of thumb: retrieve facts (RAG). Style or a fixed report format: fine-tuning. LoRA is only a cheap way to fine-tune, not a third route.
The practical rule
Just want an ESC guideline summarised neatly with good source attribution? Then RAG (route B) is almost always better than trusting a “medical model” relying on its memory. Fine-tuning is more interesting for style (clinical reports, a certain output format) than for facts.
Questions for lesson 2.2
3. You want an LLM to cite reliably from the ESC endocarditis 2023 guideline. What's usually the best route?
4. What is “LoRA” in the context of fine-tuning?
Lesson 2.3: The Transformer and “attention”
Goal: remember one concept that explains three prompt habits.
In 2017 Vaswani et al. published the paper “Attention is All You Need”. It describes the Transformer architecture, which sits under every modern LLM. Before that, language models (RNNs) processed words one by one and “forgot” those earlier in the sentence.
Attention does something fundamentally different: when predicting the next token, the model looks at all earlier tokens at once, and weighs which are most important.
Dark blue: tokens that matter for the next word. That is why a clear context above your question works better than a vague start.
Two consequences you can use in your prompts today:
- Parallel trainable on GPUs. That's why the current scale is possible at all.
- The model sees EVERYTHING in front of it, simultaneously. That means: how you order information in your prompt directly influences what gets “attention”.
Questions for lesson 2.3
5. What does “attention” do in a Transformer?
Lesson 2.4: The context window: the working memory
Goal: know how much the model can hold “in its head at once”, and when it breaks.
The context window is everything a model can currently see: your prompt + its own answer so far + possibly a system message. It's not long-term memory, when the chat closes, it's gone (unless you've turned on a “memory” feature).
But bigger ≠ better in practice. Research calls it the lost-in-the-middle effect: information in the middle of a very long prompt is picked up less reliably than info at the top or (especially) bottom.
Three practical habits that follow directly
- Put your question at the bottom. Context above, question below. The last token has the most direct line to the output.
- Paste critical source info just before your question. Not in the middle of a 50-page block.
- Be sparing with “noise”. A long dump of irrelevant data dilutes attention, literally.
Questions for lesson 2.4
6. You paste an extensive consultation with “write a discharge letter” at the bottom. Then you try it the other way: question at the top, consultation at the bottom. What's the proven rule of thumb?
7. What does the “lost-in-the-middle” effect mean?
8. Best mental image of the context window?
Live exercise: test context order yourself
Run these two prompts one after the other in an empty chat (first the A version, then the B version as a new chat). Compare the answers.
With most models you'll notice a subtle difference: version B (question at the bottom, after the source) more often has a cleaner structure and contains fewer “omitted” clinical details. Not a law, but a rule of thumb.
Lesson 2.5: How a token is “made” (the inference cycle)
Goal: understand why it sometimes responds slowly and why you see the first tokens before the end is generated.
Tokens are generated one by one. For each new token the model runs through ~100 layers of attention and feed-forward computations. That's why you see the answer appear as if someone is typing: the model has only computed up to token T at moment T.
Two consequences:
- KV caching. Earlier token representations are reused, so only the newest token has to be fully recomputed. That's why the first tokens are often the slowest (“time to first token”) and the pace speeds up afterward.
- The model can't undo its first token. Once it answered “A”, it's stuck with it and builds the rest of the answer around it, even if A was wrong. This is one of the mechanisms behind stubborn errors in long answers.
Working principle: cut off a shaky answer early
If you see a hallucinating start in the first two sentences (“According to chapter 5.3 of the guideline …”), interrupt and rephrase. Continuing costs more time and the model has trouble reversing its own false assumption.
Questions for lesson 2.5
9. Why is a mistake in the first sentence of the answer more problematic than one halfway through?
Take-home from module 2
Three steps
Pre-training (complete language) → SFT (answer questions) → RLHF/DPO (preferences + safety). The personality sits in step 3.
RAG > medical FT (often)
For factual citations, “source in the prompt” is more reliable than a medical-fine-tuned model working ‘from memory’.
Question at the bottom
Context above, question at the bottom. Avoid info in the middle for long prompts. Works on every model that's a Transformer, so all those you use.