The loss function underneath the loop
At every position in every training sequence, the model outputs a probability distribution over the entire vocabulary for 'what comes next.' The loss compares that distribution to the actual next token using cross-entropy: it penalizes the model in proportion to how much probability mass it failed to put on the correct token. A confident, wrong prediction is punished far more than a hesitant, wrong one — which is what makes cross-entropy a much better training signal than simple accuracy.