DATA OBFUSCATOR — TOKEN REPORT
==============================
Tokenizer: Primary: cl100k_base pretokenization + BPE-length model (GPT-4 family); other models estimated below
Special tokens: not counted (raw file content, no BOS/EOS)

FILE                                BYTES       TOKENS
------------------------------------------------------
INPUT
  pk_study.csv                      27894        12276

OUTPUT DATASET
  obfuscated.csv                    17393         8905

OTHER OUTPUT FILES (excluding log)
  column_report.txt                  2293          504
  config.used.txt                     756          205
  keymap.txt                         6966         2228
  llm.txt                            9901         2428
  summary.txt                        1835          518
  tokens.txt                         3096          766

EXCLUDED (log / not counted in totals)
  obfuscator.log                     6224         2023

------------------------------------------------------
TOTAL INPUT                         27894        12276
TOTAL OUTPUT DATASET                17393         8905
TOTAL OTHER FILES                   24847         6649
TOTAL ALL COUNTED                   70134        27830

MULTI-MODEL ESTIMATES (approximate - same files, other tokenizers)
MODEL                                                   INPUT   OUT DATA      OTHER ALL COUNTED
-----------------------------------------------------------------------------------------------
GPT-4 / GPT-3.5 (cl100k_base)                           12276       8905       6649       27830
GPT-4o / GPT-4.1 / o-series / GPT-5 (o200k_base)        12274       8903       6487       27664
Claude 3 and later (Anthropic)                          12405       8906       7061       28372
Gemini (Google)                                         14636      10184       6874       31694
Llama 3 / 3.1 / 3.3 (Meta)                              12276       8905       6649       27830
Qwen 2 / 2.5 / 3 (Alibaba)                              16266      12534       7237       36037
Mistral 7B / Mixtral / Llama 2 (32k vocab)              19457      13826       9125       42408

Notes:
- Shared pretokenization follows the published cl100k_base regex rules
  (contractions, letter runs, 1-3 digit groups, punctuation, newlines);
  each model family is then scored with its own profile.
- Digit handling dominates numeric CSV files. GPT-4/4o, Llama 3, Claude 3+
  and Gemini group digits (up to 3 per token); Qwen, Mistral 7B and
  Llama 2 spend one token per digit, so expect noticeably higher counts.
  The grouped-digit families therefore land close together on a dataset;
  they diverge more on prose such as llm.txt and summary.txt.
- Treat every figure as an estimate (typically within about +-10-25 %).
  For exact numbers use the provider's counter: tiktoken (OpenAI), the
  count_tokens endpoint (Anthropic), countTokens (Gemini), or the model's
  Hugging Face tokenizer (Llama, Qwen, Mistral).
- No BOS/EOS special tokens are added (file content, not a chat turn).
- The log file is excluded from totals as specified.