Introduction to LLM

Details
Written by: Babak
Category: Blog
Hits: 30

Few technologies have captured the public imagination — and the research community’s energy — as profoundly as Large Language Models. From chatbots that converse with uncanny fluency to pair programmers that write production‑ready functions, LLMs have redefined what we consider possible in artificial intelligence. Yet for all their dazzling demos, LLMs are not magic. They are the culmination of over seventy years of incremental progress in mathematics, linguistics, and computer engineering. Beneath the user‑friendly interfaces lies a stack of probability, linear algebra, distributed systems, and massive datasets. This article tears down that stack, layer by layer. We begin with the very definition of a language model, tracing its lineage from Shannon’s information theory and Markov chains through the neural revolution of Word2Vec and ELMo, and finally arrive at the transformer architectures that power GPT‑4, Claude, and their peers. We define every essential term — tokens, context windows, attention heads, positional encodings, and scaling laws — with enough mathematical rigour to satisfy a practitioner yet enough intuition for a curious developer. We then walk through a complete, reproducible training pipeline for a small GPT‑style model on the WikiText‑2 dataset, including a line‑by‑line code walkthrough. By the end, you will not only know what an LLM is, you will understand how to build one, evaluate it, and appreciate the challenges that lie ahead on the road to artificial general intelligence.

Code: https://github.com/babak-abad/Introduction-To-LLM

The Pre‑History: Information, Entropy, and Markov

The story of language modelling begins not with neural networks, but with Claude Shannon’s 1948 masterpiece, A Mathematical Theory of Communication[18]. Shannon introduced the concept of entropy as a measure of information content, and he showed that natural language has a surprising amount of redundancy. In fact, English has an entropy of roughly 1.0 to 1.5 bits per character — far less than the 4.7 bits per character one would expect from a uniform distribution over 26 letters. This redundancy is what makes prediction possible. If a language were completely random, no model could predict the next character better than random guessing. But because language is structured, we can exploit that structure.

Shannon also proposed the Markov chain as a model for generating text that mimics the statistical properties of a language. In a Markov chain of order n, the next state (character or word) depends only on the previous n states. This is precisely the n‑gram assumption that dominated language modelling for decades. Early work by Andrey Markov himself (1906) applied this to letter sequences in Pushkin’s Eugene Onegin. The connection between Markov’s mathematics and Shannon’s information theory laid the groundwork for everything that followed: if we can assign probabilities to sequences, we can compress, transmit, and generate language.

But the n‑gram approach has a fundamental flaw: it treats each word as an independent symbol, and the number of possible sequences grows exponentially with n. This is known as the curse of dimensionality. Even with a modest vocabulary of 50,000 words, a trigram model has to estimate 125 trillion probabilities — far more than any corpus can reliably count. Smoothing techniques like Good‑Turing estimation and Kneser‑Ney interpolation[19] helped, but they were palliative, not curative. The field needed a new representation: one where words could share statistical strength through similarity. That representation arrived in the form of distributed vectors.

From One‑Hot to Dense Vectors: The Embedding Revolution

The transition from sparse, count‑based representations to dense, continuous vector spaces is one of the most consequential shifts in modern AI. In a traditional one‑hot encoding, each word is a binary vector of length |V| with a single 1. These vectors are orthogonal, meaning “cat” and “dog” are as distant as “cat” and “airplane.” No semantic relationship is captured. The distributional hypothesis — that words which appear in similar contexts have similar meanings — offers a way out. If we can embed words such that their vectors reflect their contextual distributions, we can populate the vector space with meaningful geometry.

The first major breakthrough was the neural probabilistic language model of Bengio et al. (2003)[20]. They used a feed‑forward neural network with a projection layer that learned embeddings jointly with the language modelling objective. This was computationally expensive, but it proved that distributed representations could improve perplexity over n‑grams. The idea gained traction when Mikolov et al. released Word2Vec[1] in 2013, offering two architectures: CBOW (Continuous Bag‑of‑Words) which predicts the centre word from context, and Skip‑gram which predicts context words from the centre word. Both use shallow neural networks and can be trained efficiently on massive corpora. The resulting embeddings exhibited astonishing algebraic properties. The canonical example king − man + woman ≈ queen demonstrated that the model had learned not just similarity, but analogy.

GloVe[2] (Global Vectors) offered a different approach, using factorisation of the global word‑word co‑occurrence matrix. It combined the strengths of global matrix factorisation with the locality of window‑based methods. Both Word2Vec and GloVe produced static embeddings: each word has exactly one vector, irrespective of context. This leads to polysemy problems — “bank” cannot be simultaneously a financial institution and a river’s edge. The solution was to make embeddings contextual.

ELMo[3] (Embeddings from Language Models) used a deep bidirectional LSTM to produce character‑level representations. It generated a different vector for “bank” in each sentence, depending on the surrounding words. BERT[4] (Bidirectional Encoder Representations from Transformers) took this to its logical extreme with a transformer encoder trained on masked language modelling and next‑sentence prediction. BERT achieved state‑of‑the‑art results on a wide range of GLUE tasks, but it remained an encoder‑only model. For generation, the world needed a decoder.

That decoder came with GPT (Generative Pre‑training)[5] and its successors. OpenAI demonstrated that training a decoder‑only transformer with a simple causal language modelling objective, on a sufficiently large dataset, produces a model that can perform a bewildering variety of tasks through few‑shot prompting. This was the insight that launched the current era. The rest of this article unpacks exactly how that architecture works.

Evolution of Language Models ~2000 n‑gram / Smoothing 2013 Word2Vec / GloVe 2018 ELMo / BERT 2020 GPT‑3 (175B) 2023+ GPT‑4 / Claude 3 Contextual → Decoder‑only → Scaling Laws
Fig 1. The evolution of language models from sparse n‑grams to dense neural embeddings, bidirectional transformers, and finally the autoregressive decoder‑only models that dominate today’s landscape.

The Transformer Architecture: A Detailed Mechanical Breakdown

The transformer[6] replaced recurrence with self‑attention, enabling parallel processing and capturing long‑range dependencies. In a decoder‑only stack, the model consists of L identical layers, each with two main sub‑layers: a masked multi‑head self‑attention mechanism and a position‑wise feed‑forward network. Each sub‑layer is wrapped with a residual connection and followed by layer normalisation. We examine each component in depth.

Input Embedding and Positional Encoding

The input sequence x = (x1, ..., xT) consists of token indices. The first operation maps each index to a dense vector via an embedding matrix We ∈ &R;|V|×d, producing ei ∈ &R;d. However, self‑attention is permutation‑invariant — it treats the input as a set, not a sequence. Without an order signal, “dog bites man” and “man bites dog” produce identical representations. We must inject positional information.

The original transformer used absolute sinusoidal positional encodings: 

PE(pos, 2i) = sin(pos / 100002i/d)
PE(pos, 2i+1) = cos(pos / 100002i/d)

These fixed, non‑trainable encodings have a useful property: the dot product of two positional encodings depends only on their relative offset, which helps the model attend to relative positions. However, modern models often use learned positional embeddings (GPT‑2) or more advanced relative methods like Rotary Positional Embeddings (RoPE)[11]. RoPE encodes absolute position via a rotation matrix applied to the query and key vectors; the attention score between positions i and j depends only on i−j, which helps with extrapolation beyond the training context length. In our toy model we use learned embeddings, but the principle remains the same: the model must know where each token stands.

Multi‑Head Self‑Attention

Attention is the engine of the transformer. For a single head with dimension dk, given a sequence of tokens represented by the hidden states H ∈ &R;T×d, we project H into three matrices: 

Q = H WQ,  K = H WK,  V = H WV

where WQ, WK ∈ &R;d×dk and WV ∈ &R;d×dv. The attention scores are computed as: 

A = softmax(QKT / √dk + M)

Here M is the causal mask: an upper‑triangular matrix filled with −∞ where i < j (preventing attending to future tokens). The scaling factor √dk stabilises gradients by preventing the dot products from growing too large, which would push the softmax into regions of saturated gradients. The output is A V.

In practice, we use multi‑head attention: h parallel heads, each with its own WQ, WK, WV. Each head captures different aspects of the sequence — some focus on local syntax, others on long‑range semantic dependencies. The outputs of all heads are concatenated and projected through WO ∈ &R;h dv × d. The computational complexity is O(T2 d) — the quadratic in sequence length is the bottleneck that context window extensions must address.

Feed‑Forward Network and Normalisation

After attention, the model applies a two‑layer MLP with a non‑linearity: 

FFN(x) = σ(x W1 + b1) W2 + b2

where σ is typically GELU (Gaussian Error Linear Unit) in GPT‑2, or SwiGLU in more recent models like LLaMA. The inner dimension is often 4 times the hidden size, giving the model the capacity to memorise and transform patterns. Each sub‑layer uses a residual connection (x + Sublayer(x)) followed by layer normalisation. Modern transformers often use Pre‑LN (layer norm before the sub‑layer) instead of Post‑LN, as it is more stable during training.

Decoder‑Only Transformer (GPT style) Output Probabilities Linear (vocab_size) Softmax Add & Normalize Feed‑Forward Network (GELU / Swish) Add & Normalize Masked Multi‑Head Self‑Attention (8 heads, d_k = 64) MASK Add & Normalize Positional Encoding Input Embedding x₁ x₂ x₃ ... xₜ Context Window L = 128 × N layers (N = 6 in our example)
Fig 3. Decoder‑only transformer architecture. The model processes tokens in parallel, with each layer using masked self‑attention to prevent looking ahead. The output is a probability distribution over the vocabulary.

Tokenization: The Unsung Hero of LLM Engineering

A model cannot consume raw text. It needs a discrete vocabulary. The choice of tokenizer has profound effects on model performance, vocabulary size, and the ability to handle out‑of‑vocabulary words. Byte‑Pair Encoding (BPE)[7], popularised by GPT‑2, is the workhorse of the field. BPE operates as follows:

  1. Start with a vocabulary of all bytes or characters (e.g., 256 base tokens).
  2. Count the frequency of every adjacent pair of tokens in the corpus.
  3. Merge the most frequent pair into a new token, add it to the vocabulary.
  4. Repeat steps 2‑3 until the vocabulary reaches a target size (typically 30k–100k).

This greedy, iterative process produces a vocabulary where common words like “the” and “ing” become single tokens, while rare words are composed of multiple known pieces. WordPiece (used by BERT) is similar but uses a probabilistic objective based on likelihood gain. SentencePiece[8] treats the input as a raw byte stream, making it truly language‑agnostic. Our training example uses the GPT‑2 tokenizer, which has 50,257 tokens.

Byte‑Pair Encoding (BPE) Tokenization Raw text: "tokenization" Characters (initial vocabulary): t o k e n i z a t i o n Final tokens (after merging frequent pairs): ["token" , "ization"] Vocabulary: 50k subword units
Fig 2. Byte‑Pair Encoding tokenization in action. Frequent subword units become tokens, enabling the model to handle rare words without an <UNK> token.

The vocabulary size directly impacts the output layer’s parameter count. For a hidden dimension d of 256, the final linear layer has |V| × d weights — about 12.8 million for a 50k vocabulary. This is often a significant fraction of total parameters, which is why some models (e.g., LLaMA) use larger vocabularies to reduce sequence length, trading embedding parameters for fewer tokens per document.

Numerical Example: Byte‑Pair Encoding in Action

To make tokenization concrete, let us work through a complete numerical example. Suppose we have a tiny corpus consisting of the single string "aaabdaaabac". We will apply Byte‑Pair Encoding from first principles. The initial vocabulary consists of the individual characters: {a, b, c, d}. We assign them integer IDs: a → 1, b → 2, c → 3, d → 4.

  1. Count adjacent pairs in the string "aaabdaaabac":
    • (a,a) appears at positions (1,2), (2,3), (6,7), (7,8) → frequency 4.
    • (a,b) appears at (3,4) and (8,9) → frequency 2.
    • (b,d) appears once.
    • (d,a) appears once.
    • (b,a) appears once.
    • (a,c) appears once.
  2. Merge the most frequent pair, (a,a), into a new token. Let us call this new token Z and assign it the next available ID, 5. The string becomes: 
    Z Z b d Z Z b a c

    Notice that the four occurrences of (a,a) have been collapsed into two occurrences of Z.

  3. Re‑count adjacent pairs in the new sequence:
    • (Z,Z) appears twice → frequency 2.
    • (Z,b) appears twice → frequency 2.
    • (b,d) appears once.
    • (d,Z) appears once.
    • (b,a) appears once.
    • (a,c) appears once.
  4. Merge the most frequent pair again. Both (Z,Z) and (Z,b) have frequency 2. We can choose (Z,b) (the order is deterministic in implementations, but either works). Let us call this new token Y with ID 6. The string becomes: 
    Z Y d Z Y a c
  5. Stop when the vocabulary reaches a desired size, or when no pair occurs more than once. Our final vocabulary is {a:1, b:2, c:3, d:4, Z:5, Y:6}.

The final tokenised representation of the original string is therefore the sequence of integer IDs: 

[5, 6, 4, 5, 6, 1, 3]

This is exactly what a tokeniser produces: a compact, variable‑length sequence of integers where frequent sub‑patterns (like "aa" and "Zb") are compressed into single tokens. The actual token IDs are passed directly to the model’s embedding layer, which maps each ID to a dense, trainable vector. Notice that the original 11 characters are now represented by only 7 tokens — a compression of about 36%, and the new tokens Z and Y capture common substructures that would otherwise be repeated across the corpus.

Why One‑Hot Encoding Cannot Be Used Here

A novice might ask: “Why not simply assign a one‑hot vector to each character or word and feed that into the network?” The reasons are profound, and they touch the very foundations of how modern deep learning processes text. Let us dismantle the one‑hot approach point by point.

  1. Catastrophic Dimensionality – In a typical LLM, the vocabulary size |V| is between 30,000 and 100,000 (for subword models), and in character‑level or byte‑level setups it is at least 256. A one‑hot vector for a single token is a binary vector of length |V| with exactly one 1 and the rest 0s. If we used one‑hot directly as the input to the transformer’s first linear layer, that layer would have |V| × d parameters, where d is the hidden dimension. For |V| = 50,000 and d = 1,024, this is over 51 million parameters for a single layer — all wasted on a sparse, uninformative representation. In practice, the embedding layer does perform a lookup, which is mathematically equivalent to multiplying by a one‑hot vector but is implemented as an index‑based table lookup (O(1) time, O(1) memory per batch). Feeding literal one‑hot vectors to the network would require storing a T × |V| matrix for each sequence — for a context window of 2,048 tokens, that is 2,048 × 50,000 = 102 million floating‑point numbers for a single sequence, which is utterly infeasible on modern hardware.
  2. Complete Semantic Orthogonality – One‑hot vectors are orthogonal; their dot product is always 0, and their cosine distance is always 1 (except for identical tokens). This means the model has no way to encode that “cat” and “dog” are both animals, or that “run” and “ran” are the same verb. Every word is treated as an atomic, unrelated entity. The transformer would have to learn all semantic relationships from scratch, relying entirely on the massive co‑occurrence statistics across the corpus — but the input representation itself provides no inductive bias toward similarity. In contrast, dense embeddings (learned via the tokenisation + embedding lookup) are placed in a continuous space where similar words cluster together. This is the foundation of transfer learning and generalisation.
  3. Out‑of‑Vocabulary (OOV) Paralysis – A fixed one‑hot vocabulary cannot handle any token not seen during training. Every time a new word appears — a technical term, a name, a misspelling, or an emoji — the tokeniser would have to map it to a special <UNK> token, discarding all morphological information. Subword tokenisation (BPE, WordPiece, SentencePiece) solves this by representing novel words as sequences of known subword units. For example, the word “tokenization” in our earlier example became ["token", "ization"]. In a one‑hot system, the vocabulary would need to pre‑enumerate every possible word in the language, which is impossible for any practical system.
  4. No Contextual Variation – One‑hot encodings are static. The word “bank” receives the same one‑hot index regardless of whether it appears in “river bank” or “bank account.” Without context, the model cannot disambiguate polysemy. Subword tokenisation, combined with the transformer’s self‑attention, allows the model to build a contextualised representation from the dense embedding of that token. The embedding layer gives a starting vector, but it is the attention mechanism that modulates it. If we used one‑hot, the first layer would have to map the massive sparse vector directly to the hidden state, which is both computationally wasteful and fails to provide a meaningful starting point for contextualisation.
  5. Gradient Flow and Trainability – In a one‑hot representation, the gradient with respect to the input is always zero because the input is discrete and not parameterised. The only trainable parameters would be the weight matrix of the first linear layer, which acts as a de‑facto embedding matrix. However, because each input is a sparse vector with a single 1, the gradient update only affects the row corresponding to that token. This is exactly what the embedding lookup does, but the one‑hot formulation forces the framework to perform huge, sparse matrix‑vector multiplications that are both slower and more memory‑intensive than a simple gather operation. Frameworks like PyTorch and TensorFlow are optimised for dense matrix multiplication; sparse one‑hot operations kill parallelism and throughput.
  6. Sequence Length Blow‑up – Even if we ignore the computational cost, consider the context window. A transformer with context L processes an input tensor of shape (batch, L, d). With one‑hot, the input would be (batch, L, |V|). Since |V| is typically 10 to 100 times larger than d, the memory usage would skyrocket, reducing the feasible batch size and context length to impractical levels. The subword token IDs are scalars (integers) that are immediately converted to dense vectors via embedding lookup, keeping the input to the transformer layers in the compact d-dimensional space.

In short, tokenisation is the process of turning text into discrete integer IDs. The embedding layer conceptually multiplies those IDs by a one‑hot matrix, but in practice it is implemented as a lookup table. Feeding a literal one‑hot vector into the network would be a beginner’s mistake — one that shatters memory budgets, destroys any notion of semantic similarity, breaks on unseen words, and cripples training speed. The genius of subword tokenisation is that it reduces the vocabulary size, handles out‑of‑vocabulary words gracefully, and provides a compact, integer representation that the embedding layer can efficiently map to a learned, dense continuous space where gradient‑based optimisation thrives.

Masked Self‑Attention Weights (Heatmap) MASKED Key / Value tokens (past context) → Query token (current) ↓ High weight Low weight
Fig 4. Masked self‑attention in action. The highlighted token attends to previous tokens with varying weights, while the mask (blue triangle) blocks attention to future positions.

The Context Window and Its Constraints

The context window L is the maximum number of tokens the model can process in one forward pass. Because self‑attention scales as O(L2) in both time and memory, extending the window is expensive. For our small model, L = 128 fits easily. For GPT‑3, L = 2048; for GPT‑4, L = 32,768; and for Claude 3, L = 200,000[10]. To achieve these large windows without quadratically exploding costs, researchers use:

  • Sparse attention – attending only to a local neighbourhood or to fixed stride positions.
  • Sliding window attention (e.g., Longformer, Mistral).
  • FlashAttention[21] – a hardware‑aware algorithm that tiles the attention computation to reduce memory reads/writes, achieving near‑linear scaling in practice for moderate lengths.
  • ALiBi (Attention with Linear Biases) – a relative positional method that decays attention scores linearly with distance, which extrapolates well beyond the training length.

The context window determines what the model can “see.” It defines the scope of reasoning, summarisation, and in‑context learning. A larger window allows the model to process entire books or lengthy codebases, but it also increases the risk of attention dilution — where the model spreads its attention too thinly over many tokens, losing focus on the most relevant ones.

Training Infrastructure: Data, Optimisation, and Scale

Training a production LLM is an engineering feat. It requires:

  • Massive datasets: The Pile, CommonCrawl, C4, and curated web text. Data is deduplicated, filtered for quality, and often mixed with code (e.g., GitHub) and multilingual text.
  • Model parallelism: Tensors are sharded across thousands of GPUs using techniques like FSDP (Fully Sharded Data Parallel) and ZeRO (Zero Redundancy Optimizer)[22]. These partition the optimizer states, gradients, and parameters across devices, drastically reducing memory footprint.
  • Mixed precision training: Using FP16 or BF16 (bfloat16) to accelerate computation and reduce memory, while keeping a master copy of weights in FP32 for stability.
  • Loss scaling and gradient clipping to avoid underflow and explosions.

Our miniature example uses none of these — it runs on a single GPU with FP32 — but it serves as a pedagogical foundation. The scaling laws[13] discovered by Kaplan et al. (2020) and later refined by DeepMind’s Chinchilla paper[23] show that performance scales as a power law with model size, dataset size, and compute. The Chinchilla optimal is to scale data and parameters equally: for every doubling of parameters, double the number of training tokens. Many current models are under‑trained relative to this rule, which has prompted a focus on high‑quality data rather than just raw scale.

The Causal Language Modelling Objective

The loss function is the negative log‑likelihood of the target tokens: 

&mathcal;(θ) = - ∑t=1T log Pθ(xt | x<t)

This is cross‑entropy between the model’s predicted distribution and the one‑hot target. The gradient flows back through all layers, updating the embeddings, attention weights, and feed‑forward parameters. The AdamW optimizer (Adam with decoupled weight decay) is the industry standard. It computes adaptive learning rates per parameter, which is crucial for the highly non‑convex loss landscape of deep networks.

Practical Training: A Complete Code Walkthrough

The script below trains a 15‑million‑parameter GPT‑2 on WikiText‑2 for 10 epochs. The hyperparameters are centralised in config.py: 

import torch
from torch.utils.data import DataLoader
from transformers import GPT2Config, GPT2LMHeadModel, AutoTokenizer
from datasets import load_dataset
import config

# 1. Load dataset and tokenizer
dataset = load_dataset("wikitext", "wikitext-2-raw-v1")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=config.CONTEXT_WINDOW
    )

tokenized_dataset = dataset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
tokenized_dataset = tokenized_dataset.with_format("torch")

# 2. Create a small GPT‑2 model
model_config = GPT2Config(
    vocab_size=len(tokenizer),
    n_positions=config.CONTEXT_WINDOW,
    n_embd=config.HIDDEN_SIZE,
    n_layer=config.NUM_LAYERS,
    n_head=config.NUM_HEADS,
    activation_function="gelu_new",
)
model = GPT2LMHeadModel(model_config)

# 3. Dataloader
train_loader = DataLoader(
    tokenized_dataset["train"],
    batch_size=config.BATCH_SIZE,
    shuffle=True
)

# 4. Optimizer and scheduler
optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=config.LEARNING_RATE
)
scheduler = torch.optim.lr_scheduler.LinearLR(
    optimizer,
    start_factor=0.1,
    total_iters=config.WARMUP_STEPS
)

# 5. Training loop
model.train()
for epoch in range(config.NUM_EPOCHS):
    for batch_idx, batch in enumerate(train_loader):
        optimizer.zero_grad()
        outputs = model(
            input_ids=batch["input_ids"],
            labels=batch["input_ids"]
        )
        loss = outputs.loss
        loss.backward()
        torch.nn.utils.clip_grad_norm_(
            model.parameters(),
            config.GRAD_CLIP
        )
        optimizer.step()
        scheduler.step()

        if batch_idx % 50 == 0:
            print(
                f"Epoch {epoch}, Batch {batch_idx}, "
                f"Loss: {loss.item():.4f}"
            )

Line‑by‑line analysis:

  • Lines 1‑5: The imports bring in torch for tensor math, DataLoader for batching, Hugging Face’s transformers for the GPT‑2 backbone, datasets for loading WikiText‑2, and our custom config.
  • Lines 8‑9: We use the pre‑trained GPT‑2 tokenizer because it already has a subword vocabulary. We set pad_token to eos_token to handle batching where sequences differ in length.
  • Lines 11‑16: The tokenization function truncates each article to 128 tokens (our context window). The map call applies this batched, removing the raw text to save memory. The with_format("torch") converts the dataset to PyTorch tensors.
  • Lines 19‑26: A GPT2Config object specifies the architecture. n_positions is the context window, n_embd the hidden dimension, n_layer the number of transformer blocks, and n_head the attention heads. GPT2LMHeadModel is a wrapper that adds a language modelling head (the final linear + softmax) on top of the base GPT‑2 model.
  • Lines 29‑39: AdamW with weight decay is the default optimizer. The LinearLR scheduler linearly increases the learning rate from 10% of the base rate over the first 100 steps (warmup), then decays it back. Warmup prevents early instability; the initial gradients are huge, and a full learning rate would cause divergence.
  • Lines 42‑54: Inside the loop, we zero gradients, compute the forward pass (the model returns the loss when labels are provided), backpropagate, clip gradients to 1.0, and step the optimizer. Gradient clipping is essential to prevent exploding gradients, especially in deep transformers. The loss is printed every 50 batches to monitor progress.

Running this on a single NVIDIA RTX 3060 (12GB) uses about 4GB of VRAM and takes roughly 2.5 hours. The loss decreases from ~6.5 to ~4.2, indicating that the model is learning the distribution of the text. After training, the model is saved to ./trained_model.

Training Loss vs. Epoch 0 1 2 3 4 5 Cross‑Entropy Loss 0 2 4 6 8 10 Epoch Loss drops from 6.5 → 4.2 Model learns statistical patterns
Fig 5. Typical training loss curve for a small language model. The loss drops quickly in the first few epochs and then stabilizes, showing that the model is learning the statistical patterns in the data.

Generation: Sampling Strategies and Decoding

Once trained, the model is an autoregressive generator. At each step, it produces a probability distribution over the vocabulary. The simplest method is greedy decoding, selecting the argmax token. However, greedy decoding often yields repetitive, dull outputs. Better strategies include:

  • Temperature sampling: Divide the logits by a temperature τ before softmax. A temperature τ < 1 sharpens the distribution (promoting high‑probability tokens), while τ > 1 flattens it (encouraging diversity).
  • Top‑k sampling: Filter the distribution to the k most probable tokens, then sample from the renormalised subset. This avoids low‑probability nonsense tokens.
  • Top‑p (nucleus) sampling: Choose the smallest set of tokens whose cumulative probability exceeds p, then sample from that set. This adapts the truncation dynamically.
  • Beam search: Maintains multiple candidate sequences and explores them in parallel, returning the highest‑scoring sequence. Good for translation or summarisation, but less creative than sampling.

The generate.py script in the repository combines temperature and top‑k: 

import torch
from transformers import AutoTokenizer, GPT2LMHeadModel

def generate_text(
    prompt: str,
    model_path: str = "./trained_model",
    max_new_tokens: int = 50,
    temperature: float = 0.8,
    top_k: int = 50,
):
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    model = GPT2LMHeadModel.from_pretrained(model_path)
    model.eval()
    input_ids = tokenizer.encode(prompt, return_tensors="pt")
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)
    input_ids = input_ids.to(device)

    with torch.no_grad():
        for _ in range(max_new_tokens):
            outputs = model(input_ids)
            logits = outputs.logits[:, -1, :] / temperature
            if top_k is not None:
                indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]
                logits[indices_to_remove] = float("-inf")
            probs = torch.softmax(logits, dim=-1)
            next_token = torch.multinomial(probs, num_samples=1)
            input_ids = torch.cat([input_ids, next_token], dim=1)

    return tokenizer.decode(input_ids[0], skip_special_tokens=True)

The script loads the saved model, takes a prompt, and iteratively samples new tokens until the desired length is reached. The top_k filter ensures that only the 50 most probable tokens are considered, which eliminates rare, erratic choices.

Evaluation: Perplexity, Benchmarks, and Emergent Abilities

Perplexity (PPL) is the most common intrinsic metric for language models: it is the exponential of the average negative log‑likelihood. A lower perplexity means the model is more confident in its predictions. For our small model, the final perplexity is about e4.2 ≈ 66, meaning on average the model is uncertain across ~66 equally likely choices. GPT‑3 achieves perplexity below 20 on WikiText‑2, demonstrating the benefit of scale.

However, perplexity does not correlate perfectly with practical quality. A model that assigns high probability to boring, safe text may have low perplexity but poor generation quality. This is why extrinsic benchmarks are vital:

  • MMLU[16] – measures multitask understanding across 57 subjects (math, history, law, etc.).
  • GSM8K – grade‑school math word problems, testing reasoning.
  • HumanEval – code generation from docstrings.
  • HellaSwag[17] – commonsense reasoning and physical intuition.

Emergent abilities are capabilities that appear suddenly at a certain scale, such as multi‑step reasoning or in‑context learning. These are not explicitly trained for, but arise as a byproduct of the training objective and the model’s capacity. The scaling laws suggest that as we increase parameters, data, and compute, more and more of these abilities will emerge. This is the empirical foundation of the scaling hypothesis — the belief that simply scaling up current architectures may be sufficient to achieve AGI, though this remains hotly debated.

Post‑Training: Alignment, Instruction Tuning, and RLHF

A pretrained base model predicts the next token given any prefix, but that prefix could be a harmful prompt or an ambiguous instruction. To make LLMs safe, helpful, and instruction‑following, we apply post‑training:

Supervised Fine‑Tuning (SFT)

The model is fine‑tuned on a dataset of (instruction, response) pairs, typically collected from human demonstrations or generated by a stronger model. This teaches the model the style and structure of desired responses.

Reinforcement Learning from Human Feedback (RLHF)

RLHF[24] involves three steps:

  1. Train a reward model that predicts a scalar score for a response, based on human preferences.
  2. Use reinforcement learning (specifically PPO – Proximal Policy Optimization) to fine‑tune the base model to maximise the reward, while staying close to the SFT model via a KL penalty.
  3. Iterate with additional preference data.

Alternatives to RLHF have emerged, such as Direct Preference Optimization (DPO)[25], which re‑frames the RL problem as a supervised classification over preference pairs, eliminating the need for a separate reward model and PPO. These techniques are what transform a raw next‑token predictor into a helpful assistant.

Challenges and Future Directions

Despite their successes, LLMs face formidable challenges:

  • Hallucinations – generating plausible but factually false information. This stems from the fact that the model is optimised for likelihood, not truthfulness. Retrieval‑augmented generation (RAG) and tool‑use (e.g., calculators, search) are current mitigations.
  • Biases – models amplify biases present in their training data. Debiasing strategies include data filtering, adversarial training, and constitutional AI.
  • Compute cost – training a 175B parameter model like GPT‑3 costs millions of dollars and emits significant carbon. More efficient architectures are needed.
  • Context window limits – while 200k tokens is impressive, processing entire book chapters or long‑form medical records remains out of reach for many applications.

The future of LLMs may lie in Mixture of Experts (MoE) (e.g., Mistral 8x7B, Grok), where only a subset of parameters is activated per token, decoupling parameter count from inference cost. State‑Space Models (SSMs) like Mamba[26] offer an alternative to attention that scales linearly with sequence length, potentially replacing transformers in the long run. Test‑time scaling (as seen in OpenAI’s o1) uses the model to search or reason over multiple steps at inference time, pushing the frontier of reasoning without changing parameters. The journey from Shannon’s entropy to models that pass the bar exam has been extraordinary, but the final chapters are far from written.

Key Takeaways

  • A language model is a probability distribution over token sequences, typically trained with a next‑token prediction objective (causal LM).
  • Tokenization (BPE, WordPiece, SentencePiece) bridges raw text and discrete vocabulary, balancing vocabulary size and out‑of‑word coverage.
  • The transformer decoder is the backbone of modern LLMs, using masked self‑attention, residual connections, and feed‑forward networks to process sequences in parallel.
  • The context window (L) limits the model’s scope; quadratic scaling O(L2) drives innovations like FlashAttention and sparse attention.
  • Training an LLM from scratch is data‑ and compute‑intensive, but a small‑scale (15M parameters) model can be trained on a single GPU in a few hours for educational purposes.
  • Generation strategies (temperature, top‑k, top‑p) trade off diversity and quality.
  • Post‑training alignment (SFT + RLHF / DPO) turns base models into safe, instruction‑following assistants.
  • Evaluation goes beyond perplexity to benchmark reasoning, common sense, and domain‑specific knowledge (MMLU, GSM8K).
  • LLMs still struggle with hallucinations, bias, and efficiency, but emerging architectures (MoE, SSMs) and inference‑time scaling are promising directions.

Resources

Every asset, dataset, and library cited above — numbered in order of first appearance. Sample assets are used under the license noted for each; the derived images in this article are derivative works of them. Click any bracketed marker such as [1] to jump to its entry.

The complete, runnable code for every figure above lives in the companion repository under src/. Full attribution for every photo, dataset, and library is in the repository's RESOURCES.md.