Lilly Tech Systems
Tokens in AI
Master the fundamental unit of AI language models. Understand what tokens are, how tokenization works, how to count and optimize them, and how they affect pricing, context windows, and model performance.
What You'll Learn
By the end of this course, you'll understand tokens deeply enough to optimize costs, manage context windows, and make informed decisions about AI model usage.
Token Fundamentals
Understand what tokens actually are — not words, not characters, but subword units created by tokenization algorithms.
Counting & Measurement
Learn to count tokens programmatically using tiktoken, Anthropic's tokenizer, and other tools across all major models.
Pricing & Costs
Understand API pricing models, calculate costs, and build budget plans for production AI applications.
Optimization
Master techniques to reduce token usage, compress prompts, leverage caching, and choose cost-efficient models.
Course Lessons
Follow the lessons in order for a complete understanding, or jump to any topic.
1. Introduction
What are tokens? Why they matter for pricing, context limits, and performance. Visual examples and token-to-word ratios.
2. How Tokenization Works
Tokenization algorithms: BPE, WordPiece, SentencePiece, Unigram. How vocabularies are built and subwords are split.
3. Tokenizers by Model
Compare tokenizers across OpenAI, Anthropic, Google, and Meta models. Vocabulary sizes, performance, and multilingual handling.
4. Counting Tokens
Count tokens programmatically with tiktoken, Hugging Face tokenizers, and online tools. Handle images, code, and multiple languages.
5. Token Limits & Context Windows
Context windows by model, input vs output limits, what happens when you exceed them, and strategies to manage context.
6. Cost & Pricing
API pricing models, input vs output costs, price comparisons, batch discounts, prompt caching, and budget planning.
7. Optimization
Reduce token usage with prompt compression, caching, batching, model selection, and token-efficient formatting.
8. Best Practices
Production checklist for token management: monitoring, budgets, long documents, multi-language, and common mistakes.
Prerequisites
What you need before starting this course.
- Basic understanding of AI language models (helpful but not required)
- Python installed for hands-on token counting examples
- Curiosity about how AI models process text