Tokens, Embeddings and Vectors: The Three Ideas Under Modern AI

If you want to understand how an AI model turns your words into something a machine can actually work with, three ideas do almost all of the explaining: tokens, embeddings, and vectors. In one chain: an AI breaks your text into tokens (small chunks), then turns each token into an embedding (a list of numbers called a vector that captures its meaning), and because meaning is now numbers, the machine can measure how close two pieces of text are in significance. That chain, text to tokens to meaningful numbers, is the foundation under large language models, semantic search, and the retrieval that grounds modern chatbots.

These three words come up constantly once you go even slightly below the surface of AI, and they sound far more mathematical than the ideas actually are. You do not need the maths to understand them. You need the intuition, which is genuinely elegant: these are the tricks that let a machine, which only understands numbers, work with the meaning of human language. Here they are, one at a time, in plain words.

Tokens: how AI chops up text

A token is a chunk of text, the unit an AI model actually reads. It is usually a word or a piece of a word, very roughly three-quarters of a word in English on average. The model does not see your sentence as a flowing line of letters the way you do. It first runs it through a tokenizer that splits it into these chunks, and from then on the token is the basic unit it works with.

Why chunks of words rather than whole words, or single letters? It is a practical balance. Whole words would mean an impossibly large dictionary and total confusion at the first unusual or misspelled word. Single letters would be too fine-grained to carry much meaning efficiently. Sub-word tokens hit the sweet spot: common words become a single token, while rarer words get split into recognisable pieces, so "tokenization" might break into "token" and "ization." This is also why a model can handle a word it has never seen, it just assembles it from familiar smaller pieces. Tokens matter to you for two concrete reasons I have covered elsewhere: they are the unit your AI bill is usually measured in, and they are the unit the context window is measured in. When people say a model has a "128,000-token window," tokens are these chunks. So tokens are step one: text gets chopped into manageable pieces. But a chunk of text is still text, and machines only understand numbers. That is the next step.

Embeddings and vectors: turning meaning into numbers

Here is the genuinely clever part, and the one worth slowing down for. Once text is in tokens, each token gets converted into an embedding: a list of numbers that represents its meaning. That list of numbers is called a vector. So an embedding is the meaning, and a vector is the form it takes, a row of numbers. People use the words almost interchangeably, and for practical purposes you can treat "embedding" and "the vector that holds it" as the same thing.

But a list of numbers representing "meaning" sounds like hand-waving until you see what the numbers actually do, so here is the intuition that makes it click. Picture a map. On a map, two cities that are close in the real world are close on the page, and their position encodes something real about them. An embedding does the same thing for meaning. Each word becomes a point in a kind of vast multidimensional map of meaning, and words with similar meanings end up positioned close together, while unrelated words end up far apart. "Dog" and "puppy" sit near each other. "Dog" and "lion" are a bit further but still in the same neighbourhood. "Dog" and "algorithm" are on opposite sides of the map. The numbers in the vector are simply the coordinates of that point. That is the whole idea: meaning encoded as position, so that distance equals similarity.

The remarkable thing is that nobody assigns these positions by hand. The model learns them by reading enormous amounts of text and noticing which words appear in similar contexts. Words used the same way drift together; words used differently drift apart. And it captures relationships, not just similarity. The classic, almost eerie example: the model learns that the relationship between "Paris" and "France" is the same as between "Berlin" and "Germany", capital to country, encoded as a consistent direction in the map. Modern embeddings go a step further and are contextual, meaning the same word gets a slightly different position depending on its surroundings, so "bank" in "river bank" lands somewhere different from "bank" in "savings bank." The map adjusts to context.

Why this powers search that understands meaning

This is where the three ideas pay off in something you actually use. Because meaning is now numbers, and similar meanings sit close together, a computer can find related text by simple geometry: convert everything to vectors, and to find the closest match to a query, look for the nearest points on the map. This is called semantic search, and it is a real upgrade over old keyword search.

The difference is worth feeling concretely. Keyword search matches words. Ask it for "affordable laptop" and it looks for the words "affordable" and "laptop," and misses a product described as "budget notebook" because the words do not match, even though the meaning is identical. Semantic search matches meaning. "Affordable laptop" and "budget notebook" land close together on the map, so it finds the match the keyword search missed. This is exactly the machinery underneath RAG and the e-commerce product search I have written about: when a system retrieves "the most relevant documents" or matches a vague shopper's description to the right product, this is how. Everything is stored as vectors in a vector database, and "relevant" means "near in the map of meaning." The clever-sounding capability rests entirely on the simple idea that meaning has been turned into coordinates.

So the full chain, plainly: your text is split into tokens, each token is turned into an embedding (a vector) that places its meaning as a point on a vast map, and because meaning is now position, machines can measure similarity, find related text, and feed the right context to a model. Tokens are the chunks, vectors are the numbers, embeddings are the meaning those numbers capture. Three words that sound like advanced maths, resting on one elegant idea: turn meaning into position, and a machine can finally work with it. Understand that, and a surprising amount of how modern AI works stops being mysterious.

A few common questions

What is a token in AI? A chunk of text, the basic unit an AI model reads. Usually a word or part of a word, roughly three-quarters of an English word on average. Text is split into tokens by a tokenizer before the model processes it, which is why both AI pricing and the context window are typically measured in tokens.

What is the difference between an embedding and a vector? A vector is simply a list of numbers. An embedding is a vector used to represent the meaning of a token or piece of text. So the embedding is the meaning, and the vector is the numerical form it takes. In everyday use the terms are treated as nearly interchangeable.

How do embeddings capture meaning? By placing each word as a point in a vast multidimensional "map of meaning," where words with similar meanings sit close together and unrelated words sit far apart. The model learns these positions by reading huge amounts of text and noticing which words appear in similar contexts. Distance encodes similarity; direction can encode relationships (Paris is to France as Berlin is to Germany).

What is semantic search? Search that matches meaning rather than exact words. Text is converted to vectors, and the system finds results by looking for the nearest points on the map of meaning. So "affordable laptop" can match a "budget notebook" even with no shared words. It is the machinery behind RAG and meaning-based product search.