Case 1. Will the document fit the model context window?
Fast context-size estimation in plain PHP
In this case we build a fast pre-check before sending text to an LLM: estimate tokens by chars / 3, compare with the model context limit, and decide whether to send the document as is or switch to chunking.
Example of code:
<?php
function estimateTokensByChars(string $text): int {
return (int) (mb_strlen($text) / 3);
}
function fitsContext(int $tokens, int $limit): bool {
return $tokens <= $limit;
}
$knowledgeBasePath = __DIR__ . '/knowledge-base.txt';
$text = file_get_contents($knowledgeBasePath);
$chars = mb_strlen($text);
$estimatedTokens = estimateTokensByChars($text);
$contextLimit = 128_000;
echo "Chars: {$chars}" . PHP_EOL;
echo "Estimated tokens: {$estimatedTokens}" . PHP_EOL;
echo fitsContext($estimatedTokens, $contextLimit)
? 'The document fits into the model context window.'
: 'The document is too large. Use chunking.';