Back to AI Labs

Tokenization Visualizer

See how the GPT-4 tokenizer (cl100k_base) splits any text into tokens, with running counts for characters, tokens, and chars-per-token.

Tokenization
LLM Internals
Loading interactive demo…