Lilly Tech Systems
Intermediate
Traffic Classification
Implement ML-based traffic classification that separates legitimate users from bots, scrapers, and DDoS attack sources in real time.
Classification Categories
- Legitimate users: Real humans using browsers, mobile apps, or APIs normally
- Good bots: Search engine crawlers, monitoring services, partner integrations
- Bad bots: Scrapers, credential stuffers, vulnerability scanners
- Attack traffic: Volumetric floods, application-layer attacks, protocol abuse
Behavioral Fingerprinting
| Signal | Human Behavior | Bot/Attack Behavior |
|---|---|---|
| Request patterns | Variable, follows navigation paths | Repetitive, systematic, or random |
| Timing | Variable inter-request gaps | Consistent or machine-precise timing |
| JavaScript execution | Full browser JS execution | No JS or limited execution |
| Mouse/keyboard | Natural movement patterns | No interaction or robotic patterns |
| Header diversity | Diverse user agents, cookies | Identical or missing headers |
Evolving bots: Sophisticated bots use headless browsers, mimic human timing, and rotate user agents. AI classifiers must continuously learn from new bot behaviors. Features that worked last month may not catch today's bots.
ML Classification Pipeline
- Feature extraction: Extract behavioral features from requests within sliding time windows
- Real-time scoring: Score each source IP / session with the classification model
- Risk tiering: Assign risk levels (clean, suspicious, likely bot, confirmed attack)
- Action mapping: Apply appropriate mitigation per risk tier
- Feedback loop: Validated classifications improve the model continuously
Challenge-Response Verification
- JavaScript challenges: Require JS execution to prove browser capability
- CAPTCHAs: Human verification for suspicious sessions (use sparingly)
- Proof-of-work: Require computational effort to access resources
- Cookie validation: Set and verify cookies to identify legitimate browser sessions
Balance: Never block legitimate users to stop bots. Use progressive challenges: clean traffic passes freely, suspicious traffic gets a JS challenge, and only confirmed attacks get blocked. Monitor false positive rates carefully.