Case 1. Will the document fit the model context window?

Fast context-size estimation in plain PHP

In this case we build a fast pre-check before sending text to an LLM: estimate tokens by chars / 3, compare with the model context limit, and decide whether to send the document as is or switch to chunking.

Example of use


                
<?php

function estimateTokensByChars(string $text): int {
    return (int) (mb_strlen($text) / 3);
}

function fitsContext(int $tokens, int $limit): bool {
    return $tokens <= $limit;
}

$knowledgeBaseFileName = 'knowledge-base.txt';
$knowledgeBasePath = __DIR__ . '/' . $knowledgeBaseFileName;

$knowledgeBaseText = file_get_contents($knowledgeBasePath);
$chars = mb_strlen($knowledgeBaseText);
$estimatedTokens = estimateTokensByChars($knowledgeBaseText);
$contextLimit = 128_000;

echo 'Knowledge base quick context check:' . PHP_EOL;
echo "Chars: " . number_format($chars) . PHP_EOL;
echo "Estimated tokens (chars / 3): {$estimatedTokens}" . PHP_EOL;
echo "Model context limit: " . number_format($contextLimit) . PHP_EOL;
echo fitsContext($estimatedTokens, $contextLimit)
    ? 'Result: the document fits into the model context window.' . PHP_EOL
    : 'Result: the document is too large, chunking is required.' . PHP_EOL;

echo PHP_EOL . 'Two size scenarios:' . PHP_EOL;

$sampleDocs = [
    210_000,
    600_000,
];

foreach ($sampleDocs as $sampleChars) {
    $sampleTokens = (int) ($sampleChars / 3);
    $fits = fitsContext($sampleTokens, $contextLimit)
        ? 'fits (can be sent directly)'
        : 'does not fit (chunking needed)';

    echo '- '.number_format($sampleChars) . ' chars -> '.number_format($sampleTokens) . " tokens -> {$fits}" . PHP_EOL;
}

File: knowledge-base.txt


            Product setup guide:
1. Create a workspace and invite support team members.
2. Connect your email inbox and live chat channel.
3. Configure canned responses for frequently asked questions.
4. Use tags to route billing, technical, and onboarding requests.

Support playbook:
- If API calls fail with 401, rotate the access token and retry.
- If customer import fails, validate CSV encoding and required columns.
- If webhook events are delayed, check delivery logs and replay failed events.

Technical notes:
The assistant must estimate document size before sending context to an LLM.
A rough estimate uses 1 token ~= 3 chars for Russian-heavy texts.
If estimated tokens exceed the model window, split text into chunks first.

Result: Memory: 0.003 Mb Time running: 0.002 sec.


                Knowledge base quick context check:
Chars: 725
Estimated tokens (chars / 3): 241
Model context limit: 128,000
Result: the document fits into the model context window.

Two size scenarios:
- 210,000 chars -> 70,000 tokens -> fits (can be sent directly)
- 600,000 chars -> 200,000 tokens -> does not fit (chunking needed)