Lilly Tech Systems
Multi-Modal AI
Explore AI systems that perceive and reason across multiple modalities — vision, language, audio, and video. Learn how models like GPT-4V, Gemini, and Claude combine different input types to solve complex real-world problems, and build your own multi-modal applications from scratch.
What You'll Learn
By the end of this course, you'll understand how multi-modal AI works and be able to build applications that combine text, images, audio, and video.
Vision + Language
Understand how models process images alongside text to perform visual question answering, image captioning, and visual reasoning tasks.
Audio + Text
Learn how speech recognition, audio understanding, and text combine to create powerful conversational and analytical AI systems.
Video Understanding
Explore temporal reasoning across video frames, action recognition, and how AI can understand and describe video content.
Building Applications
Build real-world multi-modal applications using APIs and frameworks, from document analysis to content moderation systems.
Course Lessons
Follow the lessons in order to build a comprehensive understanding of multi-modal AI, or jump to any topic you need.
1. Introduction
Understand what multi-modal AI is, why it matters, and how combining modalities creates more capable and useful AI systems.
2. Vision + Language
Explore architectures that combine visual and textual understanding, including image captioning, VQA, and visual grounding.
3. Audio + Text
Learn how models process speech and audio alongside text for transcription, translation, audio analysis, and conversational AI.
4. Video Understanding
Dive into temporal reasoning, action recognition, and video-language models that can describe and analyze video content.
5. Building Applications
Build multi-modal applications using APIs from OpenAI, Google, and Anthropic. Cover document analysis, content moderation, and more.
6. Best Practices
Production deployment strategies, evaluation metrics, cost optimization, and emerging trends in multi-modal AI systems.
Prerequisites
What you need before starting this course.
- Basic understanding of machine learning and neural networks
- Familiarity with Python programming
- Understanding of how LLMs work (recommended)
- Experience with at least one AI API (helpful but not required)