– :

Delving a bit into the fascinating world of Large Language Models (LLMs).

At their core, LLMs take text, break it into tokens, convert those tokens into vectors, pass them through layers of mathematical transformations, and predict the next token in a sequence.

In this first post, I focus on the very first step in that pipeline: how raw text becomes vectors the model can reason about — covering tokenization, subword units (BPE), and embedding vectors.