Key takeaways
- A transformer is a kind of neural network, not an alternative to one.
- Attention lets each word take account of the other words in the passage, however far away they are.
- Transformers process a passage in parallel, which made training on very large text collections practical.
- GPT stands for generative pre-trained transformer.
How attention works in plain words
Attention works by letting each word in a passage look at the other words and decide which ones matter most for its meaning.
Take the word "it" in "The prospect liked the proposal but said it was too long". To work out what "it" refers to, the model gives more weight to "proposal" than to "prospect". A transformer does this for every word, in many layers and from several angles at once, building up a picture of how the whole passage fits together.
Why the transformer architecture mattered
The transformer architecture mattered because it let models follow long passages and be trained on far more text than earlier designs.
Earlier language networks read text one word at a time, in order, and tended to lose track of what came many words before. A transformer looks at the whole passage at once, so distant words can still influence each other, and the work can be spread across many processors. That combination is what made today's large language models possible.
Transformer vs neural network
A transformer is one type of neural network, so the comparison is between a specific design and the general family.
All neural networks learn weights from examples. What sets the transformer apart is its use of attention as the main way information moves between positions in the input. Other designs suit other jobs, but for language, and more and more for images and audio, transformers are the standard choice.
What GPT stands for
GPT stands for generative pre-trained transformer.
Generative means it produces text. Pre-trained means it first learned from a large body of general text before any narrower tuning. Transformer names the design. Many other model families use the same design under different names, so transformer describes how a model is built, not who built it.
A transformer at work in an email thread
In a long email thread, attention is what lets a model connect "the second option" in the last reply to the pricing tier named five messages earlier.
When you ask for a summary of the thread or a draft reply, the model has to link references that are far apart: who "she" is, which date "that slot" means, what "the second option" was. Attention handles these links well within the text the model can see at once. It does not make the model check that the pricing tier still exists, so a reply that names a price is worth a second look.
Where Pebbles Ai fits
Common questions
Who invented the transformer?
A team of researchers at Google introduced it in the 2017 paper "Attention is all you need". Many labs have since adapted the design.
Do transformers understand language?
They model how words relate across very large amounts of text, which lets them use language well. Whether that counts as understanding is debated; in practice they can still misread a passage or state things that are not true.
Are transformers only used for text?
No. The same design is used for images, audio, code and models that mix several kinds of input.
About this page. Written by the Pebbles Ai team for people who run sales, marketing and growth. Last updated .
