Embeddings · L2 Mechanics

How embeddings are actually learned via lookup tables and training signal, and how similarity is measured and used.

token id (integer)
lookup row in embedding table
dense vector
used by rest of the network

An embedding layer is literally a big lookup table of learned vectors, one row per vocabulary item.

Mechanically, an embedding layer is just a matrix $E \in R^{V \times d}$ (vocabulary size $V$ , embedding dimension $d$ ). Looking up token $i$ means reading row $i$ of $E$ . Critically, $E$ 's rows are learned parameters updated by gradient descent, just like any other weight matrix — the network discovers useful vector positions purely by trying to reduce its training loss.