Token Counter: Count Tokens for GPT, Claude & Any LLM Free

A token is not a word, and it is not a character. It's a sub-word unit — a chunk of text drawn from a specific model's fixed vocabulary, the list of every string that model's tokenizer is allowed to emit. Common words like the or is usually collapse into a single token. Rare words, unfamiliar names, and technical jargon routinely split into two, three, or more pieces, because no single entry for them ever earned a spot in the vocabulary during training.

This distinction matters because every large language model reads and writes tokens, not characters and not words. Before your prompt ever reaches the model's weights, a tokenizer converts it into a sequence of integers — each one an index into that fixed vocabulary — and the model's entire understanding of "length" is measured in that sequence, never in the character count you'd get from string.length.

The part that surprises people who haven't looked closely: the same input string produces a different token count depending on which model's tokenizer reads it.GPT, Claude, and Llama are not just different neural networks running on the same text representation — each family trained its own vocabulary, on its own training corpus, using its own byte-pair-encoding parameters. A sentence that costs 42 tokens under GPT's tokenizer might cost 46 under Claude's and 39 under Llama's, with no consistent direction to the difference. A token counterthat actually runs each model family's real tokenizer, rather than approximating with a shared formula, is the only way to know which number is true for the model you're actually calling.

The mechanism behind almost every modern LLM tokenizer is Byte-Pair Encoding (BPE), or a close variant of it (SentencePiece, used by Llama and several other open-weight families, applies the same core idea to Unicode text before it ever reaches raw bytes). The algorithm is simple enough to walk through by hand, and understanding it is what makes the rest of this article click.

Training starts with every individual byte or character treated as its own token — the smallest possible alphabet. The algorithm then scans a huge training corpus, counts every adjacent pair of tokens that appears, and merges the single most frequent pair into one new token. That merged token is added to the vocabulary, and the process repeats — scan, count, merge — thousands of times, until the vocabulary reaches a target size (roughly 100K entries for GPT's cl100k_base, around 200K for the newer o200k_base, and around 128K for Llama 3's tokenizer).

A simplified walk-through, on a toy corpus of just four words, makes the mechanics concrete:

Training corpus (toy example): "lower", "lowest", "newer", "wider"

Start — every character is its own token:
l o w e r
l o w e s t
n e w e r
w i d e r

Step 1 — the most frequent adjacent pair across the whole corpus is "e r",
appearing in "lower" and "newer". Merge it into a single token "er":
l o w er
l o w e s t
n e w er
w i d er

Step 2 — "l o" is now the most frequent remaining pair (in "lower" and
"lowest"). Merge it into "lo":
lo w er
lo w e s t
n e w er
w i d er

... this scan-count-merge cycle repeats tens of thousands of times over a
training corpus of hundreds of billions of characters, not four toy words,
until the vocabulary hits its target size.

The consequence of building a vocabulary this way is exactly what you'd expect from a frequency-driven process: whole words and common subwords that appeared constantly in training — the, -ing, tion, common punctuation — earn their own single merged token early, because they were frequent enough to win the merge race over and over. A word that rarely or never appeared in training gets no such shortcut. It falls back to being assembled from smaller, more generic pieces at inference time — sometimes down to individual characters or bytes — which is why an unfamiliar person's name, a rare technical term, or a string of non-English text so often costs noticeably more tokens than its character count would suggest.

Because GPT, Claude, and Llama each ran that BPE training process independently — different corpus, different target vocabulary size, different starting byte handling — the resulting vocabularies genuinely diverge. Two tokenizers can both be well-trained BPE vocabularies and still split the same sentence into a different number of pieces, because the specific merges that survived training differ. Here's the shape of that divergence, illustrated conceptually rather than as a guaranteed live output (always confirm the exact count with the real tokenizer, which is exactly what a live llm token counter is for):

Input: "The transformer's self-attention mechanism revolutionized NLP."

GPT-style vocabulary (o200k_base, ~200K merged tokens):
[The][ transformer][ 's][ self][-][attention][ mechanism][ revolution][ized][ NLP][.]
→ 11 tokens

Claude-style vocabulary (Anthropic's own, independently trained):
[The][ transform][er][ 's][ self][-][attention][ mechanism][ revolution][ized][ NLP][.]
→ 12 tokens

Llama-style vocabulary (SentencePiece, ~128K tokens, "▁" marks a
leading space):
[▁The][▁transform][er][▁'s][▁self][-][attention][▁mechanism][▁revolution][ized][▁NLP][.]
→ 12 tokens

Notice where the split lands: GPT's vocabulary happened to earn a merged transformer token, while the other two fall back to transform + erbecause that exact whole word didn't win a merge in their training runs. Multiply that kind of one-token difference across a multi-thousand-token prompt and the gap between what GPT, Claude, and Llama each report for the identical input becomes something you actually have to check, not assume. Running GenKitLab's token counter against the same prompt across all three model families in one place is the fast way to see that gap for your own text, rather than someone else's example sentence.

"One token is roughly four characters of English text" is a real, commonly cited heuristic, and it's not wrong exactly — it's a reasonable average over typical English prose, where common words and common subwords dominate and BPE merges do their job efficiently. The problem is that it's an average, and averages hide exactly the cases where accuracy matters most for a working prompt token counter.

The heuristic breaks down hardest in three specific situations, each traceable straight back to how BPE merges get earned:

• Code. Source code is dense with whitespace runs, symbols ({, }, =>, ::), and camelCase or snake_case identifiers that never appeared often enough in natural-language training data to earn their own merged token. A single identifier like calculateTotalPriceWithDiscount typically fragments into five or six pieces — far more, character for character, than a sentence of ordinary English.
• Non-English text. Every major tokenizer vocabulary is trained on a corpus that skews heavily English. Languages with different scripts or word-formation patterns — Japanese, Korean, Arabic, Hindi — get proportionally fewer of their common words and subwords represented as single merged tokens, so the same sentence, translated, routinely costs two to three times as many tokens in a non-English language as it does in English.
• Unusual formatting.Deeply nested JSON, Markdown tables, long runs of repeated punctuation, or unusual capitalization all interrupt the exact character sequences a tokenizer's merges were trained to expect, forcing more, smaller tokens than a same-length string of plain prose would need.

A quick, illustrative comparison of the gap the heuristic misses:

Input: "function calculateTotalPriceWithDiscount(items, discountRate) {"

Character count: 66
"1 token ≈ 4 characters" estimate: ~17 tokens
Typical real BPE tokenizer count: ~24 tokens

The gap comes entirely from the identifier and the punctuation —
neither camelCase words nor "(", ")", "{" earn cheap single-token
treatment the way common English words do.

None of this means the 4-characters-per-token rule is useless — for a quick gut check on a paragraph of plain English prose, it's close enough. It means treating it as a substitute for an actual count on anything that isn't plain English prose — a code snippet, a prompt with embedded JSON, a non-English document — is exactly where the estimate and the bill (or the context-window limit) start to diverge.

An inexact token count isn't an abstract inaccuracy — it fails in two concrete, practical ways, and both show up the moment a prompt actually goes to production.

1. Billing is metered per token

Every major LLM API — OpenAI, Anthropic, the hosted Llama providers — bills by the token, with input and output tokens usually priced separately and differently. A rough character-count estimate that's off by even 15-20% translates directly into a cost forecast that's off by the same margin, at the scale of an entire workload rather than one request. Getting the count right per prompt is the input to getting the cost right across a whole application — the natural next step once you have an exact count is turning it into an actual dollar figure, which is exactly what GenKitLab's prompt cost calculator and the companion LLM pricing calculator guide walk through in detail.

2. Every model has a hard context-window limit

This is the failure mode that actually costs more than a billing surprise. A model's context window — 128K tokens, 200K tokens, whatever the specific limit is — is a hard ceiling enforced by the API, not a soft guideline. If a character-based estimate puts a prompt safely under that limit but the real tokenizer counts it over, the request doesn't degrade gracefully or truncate quietly — it fails outright, at request time, in production, usually with an error a user or downstream system has to handle. A slightly conservative estimate that occasionally undercounts your available budget costs you a little headroom. An estimate that occasionally overcounts your available budget — telling you a prompt fits when it doesn't — costs you a failed request. That asymmetry is the whole argument for running the actual tokenizer instead of a formula.

Once a prompt's exact token count is known and it's either too expensive or too close to the context limit, the next move is shrinking it — GenKitLab's prompt optimizer trims redundant instructions and verbose phrasing without changing what the prompt asks the model to do.

A browser tool is the right call for checking a single prompt before you send it. The moment token counting needs to run inside a pipeline — pre-flight validation before an API call, batch cost estimation across a dataset, or a CI check on prompt length — it belongs in code, one library per model family, since each model family ships its own tokenizer implementation and there's no universal one to install.

GPT: tiktoken

import tiktoken

# encoding_for_model() picks the right vocabulary for the model name —
# o200k_base for GPT-4o and newer, cl100k_base for GPT-4/GPT-3.5.
enc = tiktoken.encoding_for_model("gpt-4o")

text = "The transformer's self-attention mechanism revolutionized NLP."
tokens = enc.encode(text)

print(len(tokens))  # the exact token count for this specific model family
print(tokens)        # the actual integer token IDs the model will see

Claude: the count_tokens API

import anthropic

client = anthropic.Anthropic()

result = client.messages.count_tokens(
    model="claude-sonnet-4-5",
    messages=[
        {"role": "user", "content": "The transformer's self-attention mechanism revolutionized NLP."}
    ],
)

print(result.input_tokens)  # exact count from Anthropic's own tokenizer

Anthropic doesn't publish its BPE vocabulary as an installable offline library the way OpenAI does with tiktoken — count_tokensis a live API call, which means an exact Claude count normally costs a network round trip unless it's reproduced locally, which is exactly what a browser-based token calculator covering Claude alongside GPT and Llama saves you from having to wire up.

Llama: SentencePiece via Hugging Face

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

text = "The transformer's self-attention mechanism revolutionized NLP."
ids = tok.encode(text)

print(len(ids))  # exact count under Llama 3's SentencePiece-derived vocabulary

Three model families, three separate libraries, three different exact counts for the identical input string — which is the whole reason a llm token counter that runs all three tokenizers side by side in one place, without three separate installs or an API key, is worth reaching for before writing any of this code into a pipeline.

Plenty of quick online "token counters" are actually word-count or character-count calculators wearing a token-counter label, applying a fixed ratio (often exactly the 4-characters-per-word heuristic debunked above) rather than running a real tokenizer. The difference matters most in exactly the cases this article covers — code, non-English text, and unusual formatting — where a fixed ratio and a real BPE count diverge the furthest.

The honest read: a word-count ratio is fine for a rough, back-of-envelope sense of a plain-English paragraph's size. It stops being fine the moment the answer feeds into a real cost forecast or a context-window budget, which is exactly the two cases this article opened with. GenKitLab's token counter counts tokens with a real tokenizer per model family, shows GPT, Claude, and Llama side by side for the same prompt, and estimates cost from that exact count — all running client-side, with nothing you paste ever uploaded anywhere.

A token count is one input into a larger decision about what a prompt actually costs and whether it fits. A few tools that pair naturally with counting one:

• Prompt Cost Calculator — turn an exact token count into a dollar figure across GPT, Claude, and Llama pricing tiers, for a single prompt or a whole workload.
• LLM Pricing Calculator guide — the deeper walkthrough of how per-token pricing actually adds up across input tokens, output tokens, and different model tiers.
• Prompt Optimizer — once you know what a prompt costs, this trims redundant instructions and verbose phrasing to shrink the token count without changing what the model is asked to do.

All AI tools →