Lilly Tech Systems
Build an AI Search Engine
Build a complete, production-ready search engine that combines keyword search (BM25), semantic vector search, and hybrid re-ranking. You will index documents, generate embeddings with sentence-transformers, build a FastAPI backend with Elasticsearch, create a polished search UI, and deploy the entire system with Docker — all in 6 hands-on steps.
What You Will Build
A fully functional AI-powered search engine that returns relevant results using keyword matching, semantic understanding, and hybrid fusion. The system ingests documents, generates dense vector embeddings, indexes them alongside traditional inverted-index fields in Elasticsearch, and serves results through a clean search interface with autocomplete, facets, and highlighting.
Hybrid Search
Combine BM25 keyword matching with dense vector cosine similarity using Reciprocal Rank Fusion. Get the best of both lexical and semantic retrieval in a single query.
Semantic Understanding
Sentence-transformers encode documents and queries into 384-dimensional vectors. The engine understands meaning, not just keywords — "car repair" finds "automobile maintenance."
Cross-Encoder Re-ranking
A cross-encoder model re-scores the top candidates for maximum precision. The final ranking considers both relevance signals and contextual understanding.
Rich Search UI
Autocomplete suggestions, faceted filtering, highlighted snippets, and paginated results. A production-quality interface built with vanilla HTML, CSS, and JavaScript.
Tech Stack
Every component is open source or has a generous free tier. Total cost to run: $0 for development, under $5/month in production.
Python 3.11+
The core language for the backend API, indexing pipeline, embedding generation, and re-ranking logic.
FastAPI
High-performance async web framework for the search API, autocomplete endpoint, and document ingestion routes.
Elasticsearch 8
Industry-standard search engine with built-in BM25 scoring, dense_vector fields for kNN search, and powerful analyzers.
sentence-transformers
Open-source embedding models (all-MiniLM-L6-v2) that run locally — no API key required. Generates 384-dimensional dense vectors.
cross-encoder
ms-marco-MiniLM-L-6-v2 cross-encoder for re-ranking. Scores query-document pairs for precise final ordering.
Docker
Containerized deployment with docker-compose for reproducible builds across development, staging, and production environments.
Prerequisites
Make sure you have these installed before starting.
Required
- Python 3.11 or higher
- Docker and docker-compose
- 8 GB RAM minimum (Elasticsearch + embedding model)
- Basic Python knowledge (functions, classes, async/await)
- A terminal (bash, zsh, PowerShell, or CMD)
Helpful but Not Required
- Experience with Elasticsearch or Lucene
- Familiarity with REST APIs and FastAPI
- Basic understanding of embeddings and vector similarity
- HTML/CSS/JavaScript basics for the frontend step
Build Steps
Follow these lessons in order. Each step builds on the previous one. By the end, you will have a fully deployable AI search engine.
1. Project Setup & Architecture
Create the project structure, install dependencies, configure Elasticsearch with Docker, and set up the FastAPI skeleton. You will have a running search API by the end.
2. Data Indexing Pipeline
Build a pipeline that processes documents, generates sentence-transformer embeddings, and indexes them into Elasticsearch with both text and dense_vector fields.
3. Keyword Search (BM25)
Implement traditional keyword search with Elasticsearch BM25, custom analyzers, multi-field matching, and relevance tuning with boosting.
4. Semantic Search
Add vector search using dense_vector fields and kNN queries. The engine now understands meaning — "programming tutorials" finds "coding lessons."
5. Hybrid Search & Re-ranking
Combine BM25 and vector scores with Reciprocal Rank Fusion. Add cross-encoder re-ranking for maximum precision on the final result set.
6. Search Interface
Build a polished search UI with autocomplete suggestions, faceted filters, highlighted snippets, and paginated results. No framework required.
7. Deploy & Scale
Containerize the entire stack with Docker, add Redis caching, configure query analytics, and optimize for production traffic.
8. Enhancements & Best Practices
Add personalization, A/B testing, query understanding, and spell correction. Includes a comprehensive FAQ for search engine builders.