Back to AI Labs
Tokenization Visualizer
See how the GPT-4 tokenizer (cl100k_base) splits any text into tokens, with running counts for characters, tokens, and chars-per-token.
Tokenization
LLM Internals
Loading interactive demo…
See how the GPT-4 tokenizer (cl100k_base) splits any text into tokens, with running counts for characters, tokens, and chars-per-token.