Token counting in PHP
Rough token estimation and basic text chunking
Before sending text to an LLM, it helps to estimate context size in tokens. The example below shows a rough character-based estimate, simple chunking, and chunking with overlap.
Example of code:
<?php
function chunkText(string $text, int $chunkSize = 1000): array {
return mb_str_split($text, $chunkSize);
}
function chunkWithOverlap(string $text, int $size = 1000, int $overlap = 200): array {
$chunks = [];
// Guard against invalid settings that would cause a non-progressing loop.
if ($size <= $overlap) {
return $chunks;
}
// Move by (size - overlap) so neighboring chunks share context.
for ($i = 0; $i < mb_strlen($text); $i += ($size - $overlap)) {
$chunks[] = mb_substr($text, $i, $size);
}
return $chunks;
}