Tools
Free AI Detector Tools: How Accurate Are They?
We test popular free AI detector tools and explain why their accuracy varies, using tokenization and perplexity concepts from the LLM lifecycle.
Read the guide
Guides
Page 6 of 7
Fundamentals
What Is AGI? Artificial General Intelligence Explained Simply
AGI explained simply: what artificial general intelligence means, how it differs from today's LLMs, which skills it needs, and why it is still hypothetical.
Read the guide
LLM Internals
How Emojis and Whitespace Change BPE Token Counts
See how emojis, symbols, and whitespace inflate BPE token counts, with worked examples and practical ways to estimate tokens for mixed-content prompts.
Read the guide
LLM Internals
Will AI Take Over Humanity? A Token-Level Answer
Explore the "AI takeover" question through the LLM lifecycle: how tokenization shapes meaning, limits context, and reframes what "control" really means.
Read the guide
Fundamentals
Machine Learning Lifecycle: From Linear Regression to LLMs
Explore the full machine learning and LLM lifecycle, from linear regression and backpropagation to attention, LoRA, RLHF, and KV-cache inference.
Read the guide
LLM Internals
How BPE Tokenization Works: A Step-by-Step Merge Walkthrough
Walk through the BPE merge algorithm on a tiny vocabulary with frequency counts, see how "low" + "er" becomes "lower", and check token-to-cost impact.
Read the guide
Inference & Cost
How to Calculate TPM Rate Limits for LLM APIs
A worked budgeting walkthrough showing how prompt tokens, max output tokens, and request frequency combine to hit provider TPM/RPM limits, with a formula and checklist.
Read the guide
LLM Internals
Counting Tokens in Code and JSON Before an LLM Call
Learn why code and JSON tokenize differently from prose, with worked examples of chat overhead and formatting tokens plus a pre-flight cost checklist.
Read the guide
LLM Internals
Why Token Counts Drift Across Tokenizer Versions
Learn why identical text yields different token counts across tokenizer versions, how BPE merge tables shift, and how to version-pin counts for accurate API cost estimates.
Read the guide
LLM Internals
What Counts as One Token in BPE? A Walkthrough
See how "unbelievable" splits into BPE subword tokens, then use a rule-of-thumb checklist to predict token counts before you run a token counter.
Read the guide
LLM Internals
Tokens per Word for LLM APIs: A Practical Ratio Guide
Learn how to calculate tokens per word for LLM API requests. See ratios for English, code and punctuation, plus a worked example and character-based estimation.
Read the guide
LLM Internals
Counting Tokens Before Sending Text to an LLM
Learn how to estimate token counts from file size and text traits, why "characters divided by 4" fails, and how to validate counts before calling LLM APIs.
Read the guide