Lilly Tech Systems
Introduction to TPU & AI Accelerators
AI workloads demand specialized hardware. Understanding the landscape of AI accelerators — from TPUs to custom ASICs — is essential for choosing the right platform.
Why Specialized Hardware?
General-purpose CPUs are versatile but inefficient for the repetitive matrix operations that dominate deep learning. AI accelerators are chips designed specifically for neural network computation, trading generality for massive performance gains on AI workloads.
The key insight: neural networks are dominated by matrix multiplications and element-wise operations. Custom hardware that optimizes these specific operations can be 10-100x more efficient than CPUs.
Types of AI Accelerators
| Type | Examples | Key Feature |
|---|---|---|
| GPU | NVIDIA A100, H100, B200 | Massively parallel, programmable, dominant ecosystem |
| TPU | Google TPU v5e, v6 | Systolic arrays optimized for matrix ops, tight Google Cloud integration |
| NPU/Neural Engine | Apple Neural Engine, Qualcomm Hexagon | On-device inference, power-efficient, mobile-first |
| Custom ASIC | AWS Trainium, Intel Gaudi | Purpose-built for specific workloads, cost-optimized |
| FPGA | Intel Stratix, Xilinx Alveo | Reconfigurable hardware, low-latency inference |
The Accelerator Landscape
NVIDIA GPUs (Market Leader)
Dominant in training and inference. CUDA ecosystem, Tensor Cores, and comprehensive software stack (cuDNN, TensorRT, Triton).
Google TPUs
Custom ASICs designed for TensorFlow and JAX. Available via Google Cloud. Systolic array architecture optimized for large-scale training.
Apple Silicon
Neural Engine integrated into M-series chips. Optimized for on-device ML inference via Core ML.
Cloud-Specific Chips
AWS Trainium/Inferentia, Microsoft Maia — cloud providers building custom chips to reduce dependency on NVIDIA.
Key Metrics
- FLOPS: Floating-point operations per second — raw computational throughput (e.g., H100: 1,979 TFLOPS FP8)
- Memory bandwidth: How fast data moves between memory and compute units (e.g., H100: 3.35 TB/s HBM3)
- Memory capacity: Total memory available for model weights and activations (e.g., H100: 80 GB)
- Power efficiency: FLOPS per watt — critical for edge deployment and data center costs
- Software ecosystem: Framework support, tools, and community. NVIDIA's CUDA ecosystem is the benchmark.
Training vs Inference Hardware
| Requirement | Training | Inference |
|---|---|---|
| Precision | FP32/BF16/FP16 | INT8/INT4/FP8 |
| Memory | Very high (weights + gradients + optimizer) | Lower (weights only) |
| Throughput | Critical (time to train) | Important (tokens/sec) |
| Latency | Less important | Critical (user experience) |
| Typical hardware | H100, TPU v5, Trainium | L4, Inferentia, Neural Engine |