Meme detail

Text detected in the image

what you see What LLM sees Words vs Tokens X T A word is most likely what you think it is - the most simple form or unit of language as understood by humans. In the sentence, "I like cats", there are three words - "" "like" C and "cats." We can think of words as the primary building blocks of language; the fundamental pieces of language that we are taught from a very young age. A token is a bit more complex. Tokenization is the process of converting pieces of language into bits of data that are usable for a program, and a tokenizer is an algorithm or function that performs this process, i.e., takes language and converts it into these usable bits of data. Thus, a token is a unit of text that is intentionally segmented for a large language model to process efficiently. These units can be words or any other subset of language - parts of words, combinations of words, or punctuation. There are a variety of different tokenizers out there which reflect a variety of trade offs. Well-known tokenizers include NLTK (Natural Language Toolkit), Spacy, BERT tokenizer and Keras. Whether or not to select one of these or a different tokenizer depends upon your specific use case. On average, there are roughly O.75 words per token, but there can be meaningful differences among tokenizers.

Popularity