Karpathy microgpt.py 해부

When you open the code to properly understand GPT, surprisingly, the peripherals come out before the essence.

PyTorch, CUDA, tokenizer, dataset loader, optimizer, scheduler, checkpoint, distributed training. In practice, this is a necessary device. But when you first try to figure out “what does GPT actually calculate,” these devices obscure your view.

The reason Andrej Karpathy'smicrogpt.pyis good isn't because it's small. This is because it peels off almost all hidden layers and makes the GPT learning loop visible all the way through in one file. The path starting from the character tokenizer to scalar autograd, Q/K/V attention, negative log likelihood, Adam update, and autoregressive sampling is revealed.

The goal of this article is not to “introduce a small GPT implementation”. The goal is to check what calculations occur behindloss.backward()andoptimizer.step(), why attention is divided into Q/K/V, and what kind of loop the next token learning turns into in the actual code.

microgpt.pyis a short file of around 200 lines. So, rather than avoiding the original text and explaining it, read this article following the flow of the original text, including enough key code blocks and dismantling them right below. It's a safe bet to check back at Karpathy's microgpt.py gist for the most up-to-date source at the time of publication.

Analysis base date: 2026-05-05
Analysis target: Andrej Karpathymicrogpt.pygist
Analysis method: Excerpt of key code blocks following the flow of the original text, explanation by line, connection to GPT structure
Caution: Since the gist can be modified, you must re-check the original revision before actual study or reproduction.

Key takeaways

• microgpt.pyshows GPT not as a “library call” but as an execution loop with a computational graph, attention, and optimizer connected.
• The key roles ofTensor,autograd,Module, andoptimizer.step()hidden by PyTorch are directly revealed as small Python objects and lists.
• The model is not a large LLM, but a small decoder-only Transformer that learns the name dataset character by character. So the structure is visible, but performance is not the goal.
• Attention is implemented in the following flows:Q,K,V, scaled dot product, softmax, and weighted sum. This part is the reference point for reading Transformer.
• After reading this code, GPT learning can be summarized as “the process of creating a computational graph to increase the probability of the next token and modifying the weight with gradient.”

Note: The code blocks in this article are excerpts to be read along the flow of the original text, which is about 200 lines long. This is not an implementation of GPT for production, but rather a reference for learning.

Order of reading this article

You can read it from beginning to end, but it's faster if you change the viewing order depending on your purpose.

If you want to see only the core flow, you can look in the following order:Value.backward()togpt()to loss calculation to Adam update to sampling.

1. What code ismicrogpt.py?

This code can be defined in one sentence as follows:

1Learn a small GPT with a character-level name dataset,2A pure Python executable that generates a string that looks like a new name.

The original file only uses standard libraries such asos,math, andrandom. The code includes data download, tokenizer configuration, autograd engine, Transformer forward pass, Adam update, and inference sampling, in that order. So this file is less of a “small model” and more of a transparent laboratory for observing GPT execution paths.

The core message is this:

1The center of GPT is the calculation that increases the probability of the next token.2However, separate system engineering is required to make the calculation quickly and large.

2. Entire execution flow

Let's start with the flow of the program.

1Check input.txt2If you don't have to, download names.txt3Create a to docs list4to create character vocabulary5Configure scalar autograd with to Value class6to embed / attention / MLP / lm_head weight initialization7To learn next-token prediction one by one8-> loss.backward()9parameter update to Adam10to Generate character samples starting from BOS

This flow is a microcosm of actual GPT training.

In an actual large model, the dataset is web-scale, the tokenizer is a BPE series, and the computation is performed with a GPU tensor kernel. But the core of the algorithm is still the same.

1Convert tokens to vectors2to attention and pass MLP3to predict the next token probability4to create a loss by comparing it with the correct answer.5Calculate the to gradient6Update weight with to optimizer7During to inference, the next tokens are sampled one by one.

3. Import and seed fixation

The first part to look at is import and seed.

1import os      # os.path.exists2import math    # math.log, math.exp3import random  # seed, choices, gauss, shuffle4 5random.seed(42)

These four lines may seem simple, but they show the nature of the code.

The important thing is that there is nonumpy,torch,tensorflow. In other words, this code shows the learning and inference flow of GPT without the tensor library. Instead, all operations occur on Python scalars and lists, so the speed is very slow.

In practice, we don't learn this way. But it's a good form to understand what PyTorch does under the hood.

4. Dataset preparation

Preparing training data looks something like this:

1if not os.path.exists('input.txt'):2    import urllib.request3    names_url = 'https://raw.githubusercontent.com/karpathy/makemore/988aa59/names.txt'4    urllib.request.urlretrieve(names_url, 'input.txt')5 6docs = [line.strip() for line in open('input.txt') if line.strip()]7random.shuffle(docs)8print(f"num docs: {len(docs)}")

The original code downloadsnames.txtofmakemoreif there is noinput.txt, reads each line as one document, and creates adocslist.

docslooks roughly like this.

1["emma", "olivia", "ava", "isabella"]

In general GPT learning, when you think of a document, you might think of a long text such as a web page, book, code file, or conversation record. But here, one name is one document.

1doc = "emma"

This model is not a model that generates sentences, but a character-level GPT that generates strings that look like names.

5. Character-level tokenizer

Next is the tokenizer.

1uchars = sorted(set(''.join(docs)))2BOS = len(uchars)3vocab_size = len(uchars) + 14 5print(f"vocab size: {vocab_size}")

Sort all unique characters that appear in the dataset, and use the last index as theBOSspecial token.

For example, if your dataset only has 26 lowercase letters of the alphabet, it would look like this:

1uchars = ['a', 'b', 'c',..., 'z']2BOS = 263vocab_size = 27

In actual learning, addBOSbefore and after the name.

1tokens = [BOS] + [uchars.index(ch) for ch in doc] + [BOS]

For example,"emma"is conceptually converted to this:

1[BOS, e, m, m, a, BOS]

The firstBOSmeans "Now the name begins". The finalBOSis also used to mean "the end of the name."

In practice, tokenizers usually separateBOS,EOS,PAD,UNK, etc. This code uses one special token for both start and end for simplicity.

6.Valueclass and scalar autograd

One of the most important parts of this file is theValueclass.

1class Value:2    __slots__ = ('data', 'grad', '_children', '_local_grads')3 4    def __init__(self, data, children=(), local_grads=()):5        self.data = data                  # forward value6        self.grad = 0                     # gradient for loss7        self._children = children         # Previous node in the computation graph8        self._local_grads = local_grads   # local derivative for each child

This one object looks like a simple number, but it actually has four pieces of information.

By analogy to PyTorch,Valueis a very smallTensor. However, PyTorch Tensor is a multidimensional array, andValueis a single scalar.

Karpathy's separate project, micrograd, is also a small autograd engine that dynamically creates a DAG on scalar values ​​and performs reverse-mode autodiff.

Ultimately, theValueclass ofmicrogpt.pyis a device for answering the following questions:

1If loss.backward() isn't magic,2What exactly happens inside?

7. Addition, multiplication, local gradient

Valuesaves the calculation graph while creating the calculation result as a newValue.

1def __add__(self, other):2    other = other if isinstance(other, Value) else Value(other)3    return Value(self.data + other.data, (self, other), (1, 1))4 5def __mul__(self, other):6    other = other if isinstance(other, Value) else Value(other)7    return Value(self.data * other.data, (self, other), (other.data, self.data))

Starting with addition, the forward value isx + y, and the local gradient is 1 on both sides.

1z = x + y2 3dz/dx = 14dz/dy = 1

Multiplication is different.

1z = x * y2 3dz/dx = y4dz/dy = x

So the local gradient of multiplication is the counterpart value.

Thanks to this structure, when you later callloss.backward(), you can obtain the gradient of all parameters by following the calculation graph backwards.

8.backward()and chain rule

The core of autograd isbackward().

1def backward(self):2    topo = []3    visited = set()4 5    def build_topo(v):6        if v not in visited:7            visited.add(v)8            for child in v._children:9                build_topo(child)10            topo.append(v)11 12    build_topo(self)13    self.grad = 114 15    for v in reversed(topo):16        for child, local_grad in zip(v._children, v._local_grads):17            child.grad += local_grad * v.grad

This function can be understood in three steps.

11. Starting from loss, collect the entire computational graph.22. Topologically sort the graph.33. Apply the chain rule by rotating from back to front.

The most important line is this:

1child.grad += local_grad * v.grad

When written as a formula, it is as follows:

1dLoss/dChild += dCurrent/dChild * dLoss/dCurrent

In other words, it is a chain rule. The principle behind the next line in PyTorch is the same.

1loss.backward()

The difference is the execution unit. PyTorch processes quickly in tensor units, andmicrogpt.pyprocesses slowly but transparently in units of scalarValueobjects.

9. Initialize model parameters

Now we create parameters for the GPT model to learn.

1n_layer = 12n_embd = 163block_size = 164n_head = 45head_dim = n_embd // n_head

These values ​​are extremely small compared to actual LLM. However, the structure retains the basic form of GPT.

1embedding2-> attention3-> MLP4-> output logits

The weight matrix is ​​also filled withValueobjects rather than regular numbers.

1matrix = lambda nout, nin, std=0.08: [2    [Value(random.gauss(0, std)) for _ in range(nin)]3    for _ in range(nout)4]

This way, each weight can have its own gradient after the forward pass.

10.state_dict: Gather GPT components

The parameters are stored instate_dict.

1state_dict = {2    'wte': matrix(vocab_size, n_embd),3    'wpe': matrix(block_size, n_embd),4    'lm_head': matrix(vocab_size, n_embd),5}

The weights inside the Transformer layer have the following characteristics.

MLP quadruplesn_embdand then decreases it again.

116 -> 64 -> 16

Finally, expand all parameters into a one-dimensional list. This is to make it easier for the optimizer to traverse and update all parameters.

11.linear,softmax,rmsnorm

There are three helpers that are repeatedly used in the model forward.

1def linear(x, w):2    return [dot(row, x) for row in w]

This function is a matrix-vector product. When written in PyTorch, it looks roughly like this:

1y = W @ x

But here it is not a tensor operation. All multiplication and addition areValueoperations. Therefore,linear()is a simple calculation and at the same time creates an autograd graph.

Next is softmax.

1def softmax(logits):2    max_val = max(val.data for val in logits)3    exps = [(val - max_val).exp() for val in logits]4    total = sum(exps)5    return [e / total for e in exps]

softmax turns logits into a probability distribution.

1softmax(x_i) = exp(x_i) / sum(exp(x_j))

The reason for subtracting the maximum value first here is because of numerical stability. For example, if logits is[1000, 1001, 1002], overflow may occur when applying the exponential function as is. Subtracting the maximum gives[-2, -1, 0], the softmax result is the same, but the calculation is much more stable.

Next is RMSNorm.

1def rmsnorm(x):2    ms = sum(xi * xi for xi in x) / len(x)3    scale = (ms + 1e-5) ** -0.54    return [xi * scale for xi in x]

RMSNorm normalizes values ​​based on the root mean square of the input vector. RMSNorm paper removes the mean-subtracting re-centering in LayerNorm and proposes RMS statistics-based normalization.

12.gpt()function and embedding

Now this is the model body, thegpt()function.

1def gpt(token_id, pos_id, keys, values):2    tok_emb = state_dict['wte'][token_id]3    pos_emb = state_dict['wpe'][pos_id]4    x = [t + p for t, p in zip(tok_emb, pos_emb)]5    x = rmsnorm(x)

Transformer does not process the token id itself directly. First, change the token id to vector.

1token_id -> token embedding vector

However, attention alone cannot tell the order of tokens. So, you need to include location information as well.

1token embedding + position embedding

[Transformer paper] (https://arxiv.org/abs/1706.03762) also explains that in structures without recurrence and convolution, positional encoding is added to input embedding to use sequence order information.

13. Making Q, K, V

When entering the Transformer block, we first prepare attention.

1x_residual = x2x = rmsnorm(x)3 4q = linear(x, state_dict[f'layer{li}.attn_wq'])5k = linear(x, state_dict[f'layer{li}.attn_wk'])6v = linear(x, state_dict[f'layer{li}.attn_wv'])7 8keys[li].append(k)9values[li].append(v)

Q, K, and V are the core concepts of attention.

keys[layer].append(k)andvalues[layer].append(v)are small KV caches. Because only the key/value up to the current location is accumulated, the model cannot see future tokens.

In a typical decoder-only transformer, a causal mask is used to mask future tokens.microgpt.pyprocesses tokens one by one from left to right rather than inserting the entire sequence at once. So the autoregressive condition holds without an explicit causal mask.

1What you can see at your current position:2[BOS, previous characters, current character]3 4What you can't see in your current position:5future characters

14. Deconstructing multi-head attention

Ifn_embd = 16,n_head = 4, the dimension of head is 4.

1Full embedding 16 dimensions2 3head 0: 0~34head 1: 4~75head 2: 8~116head 3: 12~15

Instead of performing one large attention, we divide the vector into pieces and perform attention from multiple perspectives.

1for h in range(n_head):2    hs = h * head_dim3    q_h = q[hs:hs+head_dim]4    k_h = [ki[hs:hs+head_dim] for ki in keys[li]]5    v_h = [vi[hs:hs+head_dim] for vi in values[li]]

The attention score is calculated using the scaled dot-product.

1attn_logits = [2    sum(q_h[j] * k_h[t][j] for j in range(head_dim)) / head_dim**0.53    for t in range(len(k_h))4]5attn_weights = softmax(attn_logits)

[Transformer thesis] (https://arxiv.org/abs/1706.03762) explains that the dot product of the query and key is divided by the square root of the key dimension and then softmax is applied to create a weight for the value.

Then the values ​​are weighted.

1head_out = [2    sum(attn_weights[t] * v_h[t][j] for t in range(len(v_h)))3    for j in range(head_dim)4]5x_attn.extend(head_out)

In other words, attention answers the following questions:

1For the current token to predict the next token,2Which of the previous tokens should I refer to and how much?

15. Attention output and residual connection

After concatenating the results of each head, output projection is performed.

1x = linear(x_attn, state_dict[f'layer{li}.attn_wo'])2x = x + x_residual

The first line shuffles the output of multiple heads back into the embedding dimension.

1multi-head output -> output projection

The second line is the residual connection.

1x = attention_output + original_input

Residual connections help information and gradients flow well in deep networks. Transformer papers also use residual connections around each sub-layer.

Althoughmicrogpt.pyhas only one layer, the structure itself follows the basic pattern of the GPT block.

1RMSNorm2-> Multi-head Attention3-> Output Projection4-> Residual Add

16. MLP block andlm_head

Attention is followed by an MLP block.

1x_residual = x2x = rmsnorm(x)3 4x = linear(x, state_dict[f'layer{li}.mlp_fc1'])5x = [xi.relu() for xi in x]6x = linear(x, state_dict[f'layer{li}.mlp_fc2'])7 8x = x + x_residual

If you write the structure in one line, it looks like this.

1x -> RMSNorm -> Linear -> ReLU -> Linear -> Residual Add

In size, it looks like this:

116 -> 64 -> 16

If attention determines “which past token to look at,” MLP nonlinearly transforms the hidden representation of each location. The Transformer paper also uses a position-wise feed-forward network in addition to the attention sub-layer.

Finally, it passes through the output head.

1logits = linear(x, state_dict['lm_head'])2return logits

logitsis not yet a probability.

1logits = raw score for next token candidates

For example, if it isvocab_size = 27, there are also 27 logits. If softmax is applied here, it becomes the next token probability.

1probs = softmax(logits)

Ultimately, thegpt()function does the following:

1Receive the current token, current position, and past KV cache2Returns logits for the next token.

17. Learning loop: turning one name into a next-token problem

The learning loop selects one document at each step and creates a token sequence prefixed withBOS.

1for step in range(num_steps):2    doc = docs[step % len(docs)]3    tokens = [BOS] + [uchars.index(ch) for ch in doc] + [BOS]4    n = min(block_size, len(tokens) - 1)

For example,doc = "emma"would be the following sequence.

1[BOS, e, m, m, a, BOS]

This sequence changes to the following problems during learning.

This is the basic goal of GPT learning.

1Guess the next token based on the tokens you have seen so far.

18. Loss calculation

The forward pass and loss calculations have the following structure.

1keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]2losses = []3 4for pos_id in range(n):5    token_id, target_id = tokens[pos_id], tokens[pos_id + 1]6    logits = gpt(token_id, pos_id, keys, values)7    probs = softmax(logits)8    loss_t = -probs[target_id].log()9    losses.append(loss_t)10 11loss = (1 / n) * sum(losses)

The key is the negative log likelihood of the correct answer token probability.

1loss = -log(P(correct_next_token))

The higher the probability of the correct answer token, the smaller the loss.

1P(correct) = 0.92-log(0.9) ~= 0.105

If the probability of the correct answer token is low, the loss increases.

1P(correct) = 0.012-log(0.01) ~= 4.605

In other words, the model is trained to change the weight to give a higher probability to the actual next character.

19.loss.backward(): Flow gradient to all parameters

Once loss is created, backpropagation is performed.

1loss.backward()

This one line callsValue.backward()created earlier. In reality, we follow the following path backwards:

1loss2-> log3-> softmax4-> logits5-> lm_head6-> MLP7-> attention8-> embedding9-> parameters

Each parameter is aValueobject. Therefore, afterloss.backward(), thegradfield of each parameter is filled.

1p.grad

This value has the following meaning:

1If I change this parameter slightly, how much does the loss change?

In practical terms, this perspective is also important when understanding PyTorch'sloss.backward(). This one line isn't magic, it's the process of applying the chain rule by following the computation graph backwards.

20. Direct implementation of Adam optimizer

Now we update the parameters using gradient.

1learning_rate = 0.012beta1 = 0.853beta2 = 0.994eps_adam = 1e-85 6m = [0.0] * len(params)7v = [0.0] * len(params)

Adam paper introduces Adam as a stochastic optimization algorithm that uses adaptive estimates of the lower-order moments of the gradient.

The update proceeds as follows:

11. As the step progresses, the learning rate decreases.22. Update the moving average m of the gradient.33. Update the moving average v of the gradient squared.44. Correct the bias that occurs in the initial step.55. Modify the parameter value directly.66. Initialize gradient to 0.

The core of the original update loop is this part.

1lr_t = learning_rate * (1 - step / num_steps)2 3for i, p in enumerate(params):4    m[i] = beta1 * m[i] + (1 - beta1) * p.grad5    v[i] = beta2 * v[i] + (1 - beta2) * p.grad ** 26 7    m_hat = m[i] / (1 - beta1 ** (step + 1))8    v_hat = v[i] / (1 - beta2 ** (step + 1))9 10    p.data -= lr_t * m_hat / (v_hat ** 0.5 + eps_adam)11    p.grad = 0

The last line is especially important.

1p.grad = 0

If you do not initialize the gradient, the gradient of the next step will be accumulated in the previous step. In PyTorch, the role is similar to the following code.

1optimizer.zero_grad()2optimizer.step()

21. Inference: Starting from BOS and generating one letter at a time

Once training is complete, a new name is generated.

1temperature = 0.52 3for sample_idx in range(20):4    keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]5    token_id = BOS6    sample = []

The creation loop looks like this:

1for pos_id in range(block_size):2    logits = gpt(token_id, pos_id, keys, values)3    probs = softmax([l / temperature for l in logits])4 5    token_id = random.choices(6        range(vocab_size),7        weights=[p.data for p in probs]8    )[0]9 10    if token_id == BOS:11        break12 13    sample.append(uchars[token_id])

The creation flow is as follows:

1BOS2to sample first letter3to sample the second letter4->...5Quit when to BOS appears.

temperature controls the randomness of generation.

Here it istemperature = 0.5. Rather than generating something completely random, it gives a stronger preference to characters that the model deems plausible.

22. Revisit the entire structure at once

The overall structure ofmicrogpt.pycan be summarized as follows.

If you boil the main point down to one sentence, it's like this.

1microgpt.py is an executable anatomy diagram that allows GPT's training and inference to be traced by hand without PyTorch.

The differences from the actual LLM should also be clarified.

23. What’s especially worth learning from this code?

It would be a shame to see this code as simply a "small GPT implementation". There are three more important learning points: Once these three things are captured, Transformer code written in PyTorch becomes much less opaque.

23.1 Autograd is a computational graph and chain rule

TheValueclass is a miniature version of PyTorch autograd.

1Graph creation when forwarding2Apply chain rule when to backward3to parameter.grad fill

If you see this process in person, the following code will be much clearer.

1loss.backward()2optimizer.step()3optimizer.zero_grad()

23.2 Attention is the comparison of Q and K, the weighted sum of V

The core of attention is three lines:

1score = q dot k / sqrt(d)2weight = softmax(score)3output = sum(weight * value)

In other words, the current token does not view all past tokens equally, but gives higher weight to the necessary tokens.

23.3 GPT learning is a process of increasing the probability of the next token.

The learning objectives are not complicated.

1Guess the next token with the tokens seen so far.

loss is the negative log likelihood of the correct answer token probability.

1loss = -log(P(correct_next_token))

This principle remains a central idea in both small name generation models and large language models.

24. Practical checklist

If you're following the original code directly, it's a good idea to look at it in the order below.

1microgpt.py reading checklist2 3[ ] I checked the original gist reference date.4[ ] I understand the flow of downloading names.txt when there is no input.txt.5[ ] I understood that one docs is one name.6[ ] I understand that uchars, BOS, and vocab_size act as tokenizers.7[ ] Can explain the difference between Value.data and Value.grad.8[ ] I understand how add and multiply store local gradients.9[ ] Can explain topological sort and chain rule of backward().10[ ] The roles of wte, wpe, and lm_head can be distinguished.11[ ] Can explain the roles of Q, K, and V projection.12[ ] I understood attention score = q dot k / sqrt(head_dim).13[ ] I understand that the keys and values ​​list acts as a small KV cache.14[ ] I understand that the MLP block has a 16 to 64 to 16 structure.15[ ] The meaning of loss = -log(prob[target]) can be explained.16[ ] Adam's m, v, and bias correction can be roughly explained.17[ ] I understand what temperature does in inference.

25. Q&A

Q1. Is this code really GPT?

Structurally, it is a decoder-only Transformer-based next-token model. However, the size is very small, the tokenizer is in character units, and the training data is also a list of names.

Therefore, it is more accurate to view it as an educational miniature to understand the GPT structure rather than an actual implementation of GPT.

Q2. Why didn't you use PyTorch?

This is because the goal is algorithmic description, not performance.

With PyTorch,Tensor,Module,autograd,optimizerhide a lot of work. In practice, that is an advantage. However, for learning purposes, the internal structure may not be visible.

microgpt.pydeliberately exposes its hidden parts.

Q3. Why isBOSalso used as the end token?

Because of simplification.

In actual tokenizer,BOSandEOSare usually separated. However, since this code deals with the small problem of name generation, it uses one special token for both the start and end.

Q4. There is no causal mask, so why not look at the future token?

This code does not process the entire sequence at once. Each token is processed one by one from left to right, and only the current key/value is stored in the cache.

Therefore, at the current position, the key/value of the future token does not yet exist.

1Instead of covering the future with a mask,2We proceed without calculating the future at all.

Q5. If I grow this code as is, will I get an LLM?

Conceptually, it's the right direction, but it's difficult as is. A real LLM requires the following elements:

1large-scale tokenizer2Large dataset pipeline3batch training4tensor operation5GPU/TPU acceleration6mixed precision7distributed training8checkpoint9evaluation10serving stack11Security and cost management

microgpt.pyis a code that shows the framework of the algorithm. The practical LLM has a huge system engineering on top of it.

Q6. What is the most important line in this code?

Personally, there are three streams.

1child.grad += local_grad * node.grad

This line is the chain rule, the core of autograd.

1attention_score = q dot k / sqrt(head_dim)

This expression is the core of attention.

1loss = -log(P(correct_next_token))

This expression shows the next-token prediction learning goal.

26. References and uncertainty

References

• Andrej Karpathymicrogpt.pygist: Contains the GPT learning and inference process in one file in pure Python.
• Karpathymicrograd: A scalar-valued autograd engine that implements reverse-mode autodiff on top of dynamically created DAGs.
• Attention Is All You Need: Explains the basic structure of scaled dot-product attention, multi-head attention, residual connection, feed-forward network, and positional encoding.
• Adam: A Method for Stochastic Optimization: Explains the first moment, second moment, and bias correction ideas of Adam optimizer.
• Root Mean Square Layer Normalization: Describes RMS statistics-based normalization and re-scaling invariance.

confirmed facts

• As of confirmation on 2026-05-05, the original gist contains themicrogpt.pyfile, and is displayed asLast active May 5, 2026 01:01on the GitHub Gist page.
• The original code includes data loading, tokenizer,Valueautograd, Transformer forward, Adam learning loop, and inference sampling.
• The original code comments explain that this model follows GPT-2, but uses RMSNorm instead of LayerNorm, no bias, and ReLU instead of GeLU.

Author's interpretation

• This code is not an implementation to actually service GPT, but is more of an educational reference to follow the GPT algorithm by hand.
• TheValueclass is a good starting point for understanding PyTorch autograd.
• Attention implementation is not a tensor operation, but a Python list and scalar operation, so it is slow, but the structure is easy to read.

uncertainty

• Since the gist may be modified, you must re-check the original revision before actual learning or reproduction.
• Execution time may vary depending on Python version, CPU performance, and local environment.
• The commentary in this article is based on the original gist structure confirmed on 2026-05-05.

finish

In summary,microgpt.pyis closer to a code that removes unnecessary devices to understand GPT, rather than a code that makes GPT smaller.

Creating a real LLM requires a lot more systems. The tokenizer changes, the training data grows, and batch training, GPU kernel, distributed training, checkpoint, and serving stack are added. However, most of the central flow of GPT is contained within this file.

1character to token2token -> embedding3embedding -> attention4attention -> MLP5MLP -> logits6logits -> loss7loss -> gradient8gradient -> Adam update9BOS -> sampling -> generated text

When first reading, it is better to look atValue.backward()before thegpt()function. The reason is simple. The greatest value of this file is not that it “implements a Transformer,” but that it shows how forward computation becomes a graph, and how that graph returns to a gradient.

Then, if you follow the sequence oflinear(),softmax(), attention score calculation, and loss calculation, the entire structure becomes clear. If you understand this code, Transformer code written in PyTorch will no longer look like two magic lines:loss.backward()andoptimizer.step().