Tokens are an AI context of units of text. Could be a character or a word. Why use the term tokens instead of words? Who knows. Maybe because some units aren’t words? The tech industry delights in creating stupid terms for things that already had terms.
There are to many words to have each be it’s own input, but bundling frequently used snippets is more efficient than using characters.
Then the investors came and decided to measure success by wasted energy and the rest is history.
A token can range from individual letters to several words in a stock phrase, from what little I understand. It’s a piece of whatever they’re doing behind the scenes leaking into the public-facing terminology.
According to my SO, it’s safe enough to just say “a token is a word” as long as you remember this is only an approximation.
Tokens are an AI context of units of text. Could be a character or a word. Why use the term tokens instead of words? Who knows. Maybe because some units aren’t words? The tech industry delights in creating stupid terms for things that already had terms.
Well, yeah. You answered yourself?
There are to many words to have each be it’s own input, but bundling frequently used snippets is more efficient than using characters.
Then the investors came and decided to measure success by wasted energy and the rest is history.
A token can range from individual letters to several words in a stock phrase, from what little I understand. It’s a piece of whatever they’re doing behind the scenes leaking into the public-facing terminology.
According to my SO, it’s safe enough to just say “a token is a word” as long as you remember this is only an approximation.