Case 2. Under the tokenizer microscope

Counting real tokens with a model tokenizer

In the previous case we used a rough char-based estimate. Here we count real tokens: text is passed through the tokenizer used by the model, then we compare the number of characters and tokens.

Example of use


                
<?php

use Yethee\Tiktoken\EncoderProvider;

$customerManualFileName = 'customer-manual.txt';
$customerManualPath = __DIR__ . '/' . $customerManualFileName;
$customerManualText = file_get_contents($customerManualPath);

$provider = new EncoderProvider();
$encoder = $provider->getForModel('gpt-4o');
$tokens = $encoder->encode($customerManualText);

echo 'Counting real tokens:' . PHP_EOL;
echo 'Chars: ' . mb_strlen($customerManualText) . PHP_EOL;
echo 'Tokens: ' . count($tokens) . PHP_EOL . PHP_EOL;

echo 'Let\'s look at the tokens for the text: "User Authentication System"' . PHP_EOL;
$tokens = $encoder->encode('User Authentication System');
print_r($tokens);
echo PHP_EOL;

echo 'Let\'s recover text from tokens:' . PHP_EOL;
$decoded = $encoder->decode($tokens);
echo '"' . $decoded . '"';

File: customer-manual.txt


            Custom Manual: SmartSupport Desk

Welcome to SmartSupport Desk. This guide helps you set up your team workspace
and process customer requests effectively.

Getting started:
1. Create your workspace and invite team members.
2. Connect communication channels (email, chat, and web form).
3. Configure categories for billing, technical issues, and onboarding.

Daily workflow:
- Review the incoming queue and assign priorities.
- Use predefined replies for repetitive questions.
- Escalate critical incidents to the engineering team.

Troubleshooting notes:
- If message sync is delayed, check webhook delivery logs.
- If attachments fail to upload, validate file size and MIME type.
- If search quality is poor, reindex the knowledge base.

Result: Memory: 21.876 Mb Time running: 1.813 sec.


                Counting real tokens:
Chars: 738
Tokens: 144

Let's look at the tokens for the text: "User Authentication System"
Array
(
    [0] => 1844
    [1] => 43684
    [2] => 1219
)

Let's recover text from tokens:
"User Authentication System"