Core Concept #5: Encoder vs Decoder (the original design)...

ChatGPT and Me

The last of what GPT referred to as its Core Concepts. A closer look at the difference between Encoder and Decoder Transformers.

My input (prompts) are in the boxed elements. The rest is ChatGPT's output. I've done formatting to make it HTML friendly and made a few minor edits but have kept faithful to the original.

“The original Transformer had two halves: an Encoder that reads and understands input and a Decoder that generates output. Models like GPT mostly use the decoder side. Models like BERT mostly use the encoder side.”… How different is the architecture of encoders and decoders?

They’re less like “two completely different machines” and more like “the same machine wearing different hats because humans insist on splitting everything into departments.” The encoder and decoder in the original Transformer are built from very similar building blocks. The real differences are in how they use attention and what information they’re allowed to see.

Both encoder and decoder stacks are made of repeated layers containing: So structurally, they’re cousins. Close cousins. Slightly unsettling family resemblance.

The big differences:

  1. Encoder attention is bidirectional

    An encoder can look at the entire input sequence at once. If the sentence is: “The bank near the river flooded.” the encoder can use both “river” and “flooded” to interpret “bank.”
    So every token attends to:
    • previous words
    • future words
    • itself

    The encoder’s job is: Build rich contextual representations.

    That’s why encoder models like BERT are good at:
    • understanding
    • classification
    • search
    • embeddings
    • extracting meaning

  2. Decoder attention is causal (masked)

    A decoder generates text one token at a time. When predicting the next word, it cannot peek into the future, because future words don’t exist yet. Tiny inconvenience imposed by causality. Physics ruins everything. So if the decoder is generating: “The cat sat on the …” it can only see: and must predict the next token from that. This is implemented with causal masking. That’s the defining trait of decoder models like GPT. Their job is: Predict the next token autoregressively.

  3. Original Transformer decoders also had cross-attention

    In the original paper, the decoder had an extra attention module: That’s how translation worked. Example: So the original decoder was actually more complex than the encoder.

    Modern GPT-style models remove the encoder entirely and just use giant decoder stacks.

  4. Why GPT can still “understand” things

    People often say: “Encoders understand, decoders generate.” That’s directionally true but oversimplified. Large decoder-only models absolutely learn deep representations internally. They have to. You can’t generate coherent language without understanding structure, semantics, and context to some degree.

    The difference is mainly:

    One studies.
    One performs live on stage hoping nobody notices the panic.

    Technically, the self-attention equations are nearly identical except for masking:

    Encoder attention:

    \[ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left( \frac{QK^{T}}{\sqrt{d_k}} \right)V \]

    Decoder attention:
    Same equation, but with a causal mask added before softmax:

    \[ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left( \frac{QK^{T}}{\sqrt{d_k}} + M \right)V \]

    where \(M\) blocks attention to future positions.

    That tiny masking change completely changes behavior:

    Human civilization also works this way. Tiny policy change, completely different outcome. Usually worse.

Green Hosting Badge