Meme detail
About this meme
Text detected in the image
Andrej Karpathy @karpathy To help explain the weirdness of LLM Tokenization I thought it could be amusing to translate every token to a unique emoji. This is a lot closer to truth - each token is basically its own little hieroglyph and the LLM has to learn (from scratch) what it all means based on training data statistics. So have some empathy the next time you ask an LlM how many letters 'r' there are in the word 'strawberry', because your question looks like this: Play with it here :) colab.research.google.com/drive/1SVs-ALf.. Перевести пост What you see What LLMsees Words vs Tokens A word is most likely what you think it is - the most simple form or unit of language as 2 understood by humans. In the sentence, "I like cats" there are three words - "" "like" and "cats." We can think of words as the primary building blocks of language; the fundamental pieces of language that we are taught from a very young age. → A token is a bit more complex. Tokenization is the process of converting pieces of language into bits of data that are usable for a program, and a tokenizer is an algorithm or function that performs this process, i.e., takes language and converts it into these usable bits of data. Thus, a token is a unit of text that is intentionally segmented for a 2 M→T large language model to process efficiently. These units can be words or any other © ~MCO subset of language - parts of words, combinations of words, or punctuation. 1 → There are a variety of different tokenizers out there which reflect a variety of trade offs. Well-known tokenizers include NLTK (Natural Language Toolkit), Spacy, BERT → tokenizer and Keras. Whether or not to select one of these or a different tokenizer 8→ depends upon your specific use case. On average, there are roughly O.75 words per token, but there can be meaningful differences among tokenizers. O↑
