Token
The smallest piece of text an AI language model processes—usually a word or part of a word.
In Plain English
When you type words into an AI language model, the model doesn't read whole sentences at once. Instead, it breaks your text into tiny chunks called tokens—typically individual words, or parts of words, or even punctuation marks. The model processes these tokens one at a time to understand meaning and generate responses. Think of tokens like the individual letter tiles in Scrabble; the model works with those pieces to build understanding.
💡Real-World Example
If you type 'I cannot do that,' a language model might split it into tokens like 'I,' 'cannot,' 'do,' 'that,' and the period. The model then analyzes these five tokens to understand your sentence and prepare a response. Longer texts mean more tokens, which affects how long the AI takes to respond.
Related Terms
What did you think of our explanation?