The ability to predict brain activity from words before they occur can be explained by information shared between neighbouring words, without requiring next-word prediction by the brain.

At the core of these advancements lies the concept of tokenization — a fundamental process that dictates how user inputs are interpreted, processed and ultimately billed. Understanding tokenization is ...