Lilly Tech Systems
Build a RAG Chatbot
Build a complete, production-ready Retrieval-Augmented Generation chatbot from scratch. You will ingest documents, create vector embeddings, build a retrieval pipeline, stream AI responses, and deploy the entire system with Docker — all in 6 hands-on steps.
What You Will Build
A fully functional RAG chatbot with a clean chat interface that answers questions using your own documents. The system ingests PDFs and HTML files, chunks and embeds them into a Qdrant vector store, retrieves relevant passages with re-ranking, and streams responses with source citations.
Chat Interface
A responsive HTML/JS chat UI with message history, typing indicators, copy-to-clipboard buttons, and streaming responses that appear word-by-word in real time.
Smart Retrieval
Multi-query retrieval with cross-encoder re-ranking finds the most relevant document chunks. Citations link every answer back to its source document and page number.
Streaming API
A FastAPI backend that streams token-by-token responses via Server-Sent Events. The UI updates in real time, just like ChatGPT.
Docker Deployment
One-command deployment with docker-compose. The entire stack — API server, Qdrant vector database, and ingestion worker — runs in containers ready for production.
Tech Stack
Every component is open source or has a generous free tier. Total cost to run: $0 for development, under $5/month in production.
Python 3.11+
The core language for the backend API, document ingestion pipeline, and embedding logic.
FastAPI
High-performance async web framework for the REST API and Server-Sent Events streaming endpoint.
LangChain
Document loaders, text splitters, and retriever abstractions that simplify the RAG pipeline.
Qdrant
Open-source vector database with built-in hybrid search, filtering, and payload storage.
OpenAI API
text-embedding-3-small for embeddings ($0.02/1M tokens) and gpt-4o-mini for generation ($0.15/1M tokens).
Docker
Containerized deployment with docker-compose for reproducible builds across dev, staging, and production.
Prerequisites
Make sure you have these installed before starting.
Required
- Python 3.11 or higher
- Docker and docker-compose
- An OpenAI API key (get one at
platform.openai.com) - Basic Python knowledge (functions, classes, async/await)
- A terminal (bash, zsh, PowerShell, or CMD)
Helpful but Not Required
- Experience with FastAPI or Flask
- Familiarity with REST APIs
- Basic understanding of embeddings and vector search
- HTML/CSS/JavaScript basics for the frontend step
Build Steps
Follow these lessons in order. Each step builds on the previous one. By the end, you will have a fully deployable RAG chatbot.
1. Project Setup & Architecture
Create the project structure, install dependencies, configure Docker, and set up environment variables. You will have a running FastAPI server and Qdrant instance by the end.
2. Document Ingestion Pipeline
Build a pipeline that loads PDFs and HTML files, splits them into chunks with metadata, and prepares them for embedding. Full working Python code included.
3. Embedding & Vector Store
Generate OpenAI embeddings for every chunk, store them in Qdrant with metadata payloads, and set up hybrid search with dense + sparse vectors.
4. Retrieval Pipeline
Implement multi-query retrieval, cross-encoder re-ranking, context assembly with deduplication, and citation tracking that links answers to source documents.
5. Generation & Streaming
Build the prompt engineering layer, stream responses token-by-token with FastAPI SSE, and add hallucination prevention with grounding checks.
6. Chat UI
Create a clean HTML/JS chat interface with message history, typing indicators, copy buttons, and real-time streaming display. No framework required.
7. Deploy to Production
Containerize the entire stack with Docker, configure environment variables, set up health checks, monitoring, and cost tracking for production use.
8. Enhancements & Next Steps
Add multi-tenant support, authentication, analytics dashboards, and explore advanced patterns. Includes a comprehensive FAQ for RAG chatbot builders.