Token counting in PHP

Rough token estimation and basic text chunking

Before sending text to an LLM, it helps to estimate context size in tokens. The example below shows a rough character-based estimate, simple chunking, and chunking with overlap.

Example of use


                
<?php

function chunkText(string $text, int $chunkSize = 1000): array {
    return mb_str_split($text, $chunkSize);
}

function chunkWithOverlap(string $text, int $size = 1000, int $overlap = 200): array {
    $chunks = [];

    if ($size <= $overlap) {
        return $chunks;
    }

    for ($i = 0; $i < mb_strlen($text); $i += ($size - $overlap)) {
        $chunks[] = mb_substr($text, $i, $size);
    }

    return $chunks;
}

$documentFileName = 'document.txt';
$documentPath = __DIR__ . '/' . $documentFileName;

$pureText = file_get_contents($documentPath);
$text = str_replace("\n", '', $pureText);

$chars = mb_strlen($text);
$estimatedTokens = (int) ($chars / 3);
$chunks = chunkText($text, 90);
$overlapChunks = chunkWithOverlap($text, 90, 20);

echo 'Data:' . PHP_EOL;
echo "Chars in document.txt: {$chars}" . PHP_EOL;
echo "Estimated tokens (chars / 3): {$estimatedTokens}" . PHP_EOL . PHP_EOL;

echo 'Chunk samples without overlap:' . PHP_EOL;

foreach ($chunks as $index => $chunk) {
    echo ((int)$index + 1) . '. ' . trim($chunk) . PHP_EOL;
}

echo PHP_EOL . 'Chunk samples with overlap:' . PHP_EOL;

foreach ($overlapChunks as $index => $chunk) {
    echo ((int)$index + 1) . '. ' . trim($chunk) . PHP_EOL;
}

File: document.txt


            Large language models process input as tokens, not characters.
Before sending a long document, we can quickly estimate whether
it fits into the model context window and whether we need to split it into chunks.

When a text exceeds the available context window, chunking helps process information
safely. Each chunk can be embedded, indexed, and retrieved later when it is relevant to
the user query. This keeps prompts smaller, faster, and often more accurate in production
systems.

Overlap between chunks preserves nearby meaning at the boundaries.
Without overlap, one important sentence may be cut in half and lose context.
With overlap, the neighboring chunk still carries that sentence and improves retrieval
quality.

In practice, teams tune chunk size and overlap by measurement.
They compare answer quality, latency, and cost for different configurations.
There is no universal best value, but a clear evaluation loop gives reliable results.

Result: Memory: 0.01 Mb Time running: 0.001 sec.


                Data:
Chars in document.txt: 934
Estimated tokens (chars / 3): 311

Chunk samples without overlap:
1. Large language models process input as tokens, not characters.Before sending a long docume
2. nt, we can quickly estimate whetherit fits into the model context window and whether we ne
3. ed to split it into chunks.When a text exceeds the available context window, chunking help
4. s process informationsafely. Each chunk can be embedded, indexed, and retrieved later when
5. it is relevant tothe user query. This keeps prompts smaller, faster, and often more accur
6. ate in productionsystems.Overlap between chunks preserves nearby meaning at the boundaries
7. .Without overlap, one important sentence may be cut in half and lose context.With overlap,
8. the neighboring chunk still carries that sentence and improves retrievalquality.In practi
9. ce, teams tune chunk size and overlap by measurement.They compare answer quality, latency,
10. and cost for different configurations.There is no universal best value, but a clear evalu
11. ation loop gives reliable results.

Chunk samples with overlap:
1. Large language models process input as tokens, not characters.Before sending a long docume
2. ending a long document, we can quickly estimate whetherit fits into the model context wind
3. e model context window and whether we need to split it into chunks.When a text exceeds the
4. n a text exceeds the available context window, chunking helps process informationsafely. E
5. informationsafely. Each chunk can be embedded, indexed, and retrieved later when it is rel
6. later when it is relevant tothe user query. This keeps prompts smaller, faster, and often
7. , faster, and often more accurate in productionsystems.Overlap between chunks preserves ne
8. chunks preserves nearby meaning at the boundaries.Without overlap, one important sentence
9. e important sentence may be cut in half and lose context.With overlap, the neighboring chu
10. the neighboring chunk still carries that sentence and improves retrievalquality.In practi
11. valquality.In practice, teams tune chunk size and overlap by measurement.They compare answ
12. nt.They compare answer quality, latency, and cost for different configurations.There is no
13. urations.There is no universal best value, but a clear evaluation loop gives reliable resu
14. gives reliable results.