Feature Engineering

Master Databricks Feature Store, Spark ML pipelines, and data preparation techniques — covering approximately 20% of the Databricks ML Professional exam.

Databricks Feature Store

The Databricks Feature Store is a centralized repository for storing, discovering, and sharing features across ML projects. It is a critical exam topic because it solves the feature consistency problem — ensuring training and serving use the same feature computation logic.

Key Concepts

• Feature Table — A Delta table managed by the Feature Store with a primary key and optional timestamp key for point-in-time lookups
• Feature Lookup — Joining feature tables with training data using FeatureLookup objects at training time
• Online Store — A low-latency store (e.g., Amazon DynamoDB, Azure Cosmos DB) synced from the offline Feature Store for real-time serving
• Feature Freshness — How recently the features were computed; critical for time-sensitive models

Creating a Feature Table

The exam tests your knowledge of the Feature Store API. Here is the standard pattern:

from databricks.feature_store import FeatureStoreClient

fs = FeatureStoreClient()

# Create a feature table from a Spark DataFrame
fs.create_table(
    name="ml_features.customer_features",
    primary_keys=["customer_id"],
    timestamp_keys=["event_date"],  # for point-in-time lookups
    df=customer_features_df,
    description="Customer behavioral features for churn prediction"
)

Training with Feature Lookups

Instead of manually joining features, the Feature Store handles lookups automatically:

from databricks.feature_store import FeatureLookup

feature_lookups = [
    FeatureLookup(
        table_name="ml_features.customer_features",
        feature_names=["total_purchases", "avg_session_duration"],
        lookup_key="customer_id"
    ),
    FeatureLookup(
        table_name="ml_features.product_features",
        feature_names=["category_popularity", "price_percentile"],
        lookup_key="product_id"
    )
]

training_set = fs.create_training_set(
    df=labels_df,
    feature_lookups=feature_lookups,
    label="churned"
)

training_df = training_set.load_df()

Publishing to Online Store

For real-time serving, features must be published to an online store:

# Publish feature table to online store
fs.publish_table(
    name="ml_features.customer_features",
    online_store=online_store_spec,
    filter_condition="event_date >= '2026-01-01'"
)

Spark ML Pipelines

Spark ML pipelines provide a standardized way to chain feature transformations and model training into a single, reproducible workflow.

Pipeline Components

• Transformer — Takes a DataFrame and produces a new DataFrame (e.g., StringIndexer, VectorAssembler, StandardScaler)
• Estimator — An algorithm that can be fit() on a DataFrame to produce a Transformer (e.g., StandardScaler before fitting)
• Pipeline — A sequence of Transformers and Estimators chained together
• PipelineModel — A fitted Pipeline that can transform() new data

from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, VectorAssembler, StandardScaler

# Define pipeline stages
indexer = StringIndexer(inputCol="category", outputCol="category_idx")
assembler = VectorAssembler(
    inputCols=["category_idx", "amount", "frequency"],
    outputCol="features_raw"
)
scaler = StandardScaler(inputCol="features_raw", outputCol="features")

pipeline = Pipeline(stages=[indexer, assembler, scaler])

# Fit the pipeline
pipeline_model = pipeline.fit(train_df)

# Transform new data
transformed_df = pipeline_model.transform(test_df)

Data Preparation Techniques

Handling Missing Values

# Imputation with Spark
from pyspark.ml.feature import Imputer

imputer = Imputer(
    inputCols=["age", "income"],
    outputCols=["age_imputed", "income_imputed"],
    strategy="median"  # "mean", "median", or "mode"
)

Encoding Categorical Variables

• StringIndexer — Maps string values to numeric indices (frequency-based ordering)
• OneHotEncoder — Converts indexed values to binary vectors (use for low-cardinality features)
• Feature Hashing — Maps high-cardinality features to fixed-size vectors using FeatureHasher

Feature Scaling

• StandardScaler — Standardizes features to zero mean and unit variance (use with linear models, neural nets)
• MinMaxScaler — Scales features to [0, 1] range (use when you need bounded values)
• MaxAbsScaler — Scales by dividing by max absolute value (preserves sparsity)

Point-in-Time Lookups

For time-series and event-driven features, point-in-time correctness prevents data leakage:

# Feature table with timestamp key enables point-in-time lookups
fs.create_table(
    name="ml_features.user_daily_stats",
    primary_keys=["user_id"],
    timestamp_keys=["date"],  # ensures no future data leaks into training
    df=daily_stats_df
)

# At training time, the Feature Store automatically performs
# as-of joins using the timestamp key

Practice Questions

Question 1 — Feature Store

Question 2 — Feature Lookup

Question 3 — Spark ML Pipeline

Question 4 — Point-in-Time

Question 5 — Data Preparation