Neural networks only understand numbers — vectors and matrices. But most interesting data (words, categories, user IDs) starts out as discrete symbols with no inherent numeric meaning. An embedding is a learned mapping from a discrete symbol to a dense vector of numbers, chosen so that similar symbols end up at similar vectors.
From one-hot to dense
One-hot encoding
- Vector of all zeros, one 1
- Length = vocabulary size (huge)
- Every word equally 'far' from every other
- No notion of similarity
Learned embedding
- Dense vector, e.g. 768 numbers
- Fixed small size regardless of vocab
- 'king' and 'queen' end up close together
- Captures learned semantic structure
The famous example: in a well-trained word embedding space, the vector arithmetic lands very close to the vector for — not because anyone programmed that relationship, but because it emerged from the statistics of how those words are used in text.