Case 2. Under the tokenizer microscope
Counting real tokens with a model tokenizer
In the previous case we used a rough char-based estimate. Here we count real tokens: text is passed through the tokenizer used by the model, then we compare the number of characters and tokens.
Example of code:
<?php
use Yethee\Tiktoken\EncoderProvider;
$provider = new EncoderProvider();
$encoder = $provider->getForModel('gpt-4o');
$text = file_get_contents('customer-manual.txt');
$tokens = $encoder->encode($text);
echo "Chars: ";
echo mb_strlen($text) . "\n";
echo "Tokens: ";
echo count($tokens) . "\n";