Data Architecture Patterns

Lambda Architecture for ML Feature Pipeline:

                    [Raw Events (Kafka)]
                     /              \
                    v                v
        [Batch Layer]            [Speed Layer]
        Spark/Dataflow           Flink/Kafka Streams
        Runs daily               Processes in real-time
        Full history             Last few hours
        Complex features         Simple aggregations
             |                        |
             v                        v
    [Batch Feature Store]     [Real-Time Feature Store]
    (BigQuery/S3)             (Redis/DynamoDB)
             \                      /
              v                    v
           [Serving Layer - merges both]
           batch_features + real_time_features → model input

Example features:
  Batch:     user_avg_purchase_90d, category_affinity_score, lifetime_value
  Real-time: items_viewed_this_session, cart_value, time_since_last_click

Kappa Architecture for ML Feature Pipeline:

    [Raw Events (Kafka with long retention)]
                    |
                    v
            [Stream Processor]
            Flink / Kafka Streams
            Handles both real-time
            and historical replay
                    |
                    v
            [Feature Store]
            Single unified store
            (Redis + cold storage)
                    |
                    v
            [Model Serving]

Retraining: Replay Kafka topic from offset 0
Feature backfill: Replay with new feature logic

# Feature Store Architecture

class FeatureStore:
    """
    Dual-store architecture used by Feast, Tecton, and most custom implementations.
    """

    def __init__(self):
        # Online store: low-latency lookups for serving
        # Backed by Redis, DynamoDB, or Bigtable
        # Stores latest feature values per entity
        self.online_store = RedisClient(
            host="redis-cluster.internal",
            read_timeout_ms=5  # Must be fast
        )

        # Offline store: full history for training
        # Backed by BigQuery, S3/Parquet, or Delta Lake
        # Stores all historical feature values with timestamps
        self.offline_store = BigQueryClient(
            dataset="ml_features"
        )

    def get_online_features(self, entity_id: str, feature_names: list) -> dict:
        """Called during inference. Must return in < 10ms p99."""
        return self.online_store.hmget(f"features:{entity_id}", feature_names)

    def get_training_data(self, entity_ids: list, features: list,
                          start_date: str, end_date: str) -> DataFrame:
        """Called during training. Point-in-time correct join."""
        return self.offline_store.query(f"""
            SELECT e.entity_id, e.timestamp, {', '.join(features)}
            FROM entities e
            LEFT JOIN feature_table f
              ON e.entity_id = f.entity_id
              AND f.timestamp <= e.timestamp  -- Point-in-time correctness!
              AND f.timestamp >= TIMESTAMP_SUB(e.timestamp, INTERVAL 1 DAY)
            WHERE e.timestamp BETWEEN '{start_date}' AND '{end_date}'
        """)

    def materialize(self, feature_name: str):
        """Sync offline store → online store (runs on schedule)."""
        latest_values = self.offline_store.get_latest(feature_name)
        self.online_store.bulk_set(latest_values)

# Data Architecture Decision Matrix

QUESTION 1: Do you need real-time features for inference?
  YES → You need an online feature store (Redis/DynamoDB)
  NO  → Batch precomputation is sufficient (simpler, cheaper)

QUESTION 2: How complex are your features?
  Simple aggregations (count, sum, avg) → Kappa architecture (streaming only)
  Complex joins + window functions     → Lambda architecture (batch + streaming)
  Just raw data, no aggregation        → Skip feature engineering, embed at serving time

QUESTION 3: How much data do you have?
  < 10GB    → Just use PostgreSQL + CSV files. Seriously.
  10-500GB  → Data lake (S3/GCS) + DVC for versioning
  500GB-10TB → Lakehouse (Delta Lake/Iceberg) + feature store
  > 10TB    → Full Lambda architecture + managed feature store (Tecton/Vertex)

QUESTION 4: How many teams share the data?
  1 team    → Keep it simple. Local feature pipelines, DVC.
  2-5 teams → Feature store becomes critical for consistency
  5+ teams  → Managed feature store + data catalog + governance

QUESTION 5: What are your compliance requirements?
  None      → Flexible schema, minimal governance
  SOC2      → Audit logs, access control, encryption at rest
  GDPR      → Data lineage, deletion capabilities, consent tracking
  HIPAA     → All of above + BAA with cloud provider, strict access control