Kubeflow on EKS

Deploy and use Kubeflow on Amazon EKS for ML pipelines, notebook servers, distributed training, and experiment tracking.

What is Kubeflow?

Kubeflow is an open-source ML platform for Kubernetes that provides tools for the entire ML lifecycle. On EKS, it integrates with AWS services for storage, authentication, and compute.

Installing Kubeflow on EKS

# Clone the AWS Kubeflow distribution
git clone https://github.com/awslabs/kubeflow-manifests.git
cd kubeflow-manifests

# Install prerequisites
make install-tools

# Deploy Kubeflow with AWS integrations
make deploy-kubeflow INSTALLATION_OPTION=kustomize \
  DEPLOYMENT_OPTION=vanilla

Distributed Training with PyTorchJob

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llm-finetune
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
            - name: pytorch
              image: training-image:latest
              resources:
                limits:
                  nvidia.com/gpu: 8
    Worker:
      replicas: 3
      template:
        spec:
          containers:
            - name: pytorch
              image: training-image:latest
              resources:
                limits:
                  nvidia.com/gpu: 8

Kubeflow Pipelines Example

from kfp import dsl, compiler

@dsl.component(base_image="python:3.10")
def preprocess(data_path: str) -> str:
    # Data preprocessing logic
    return processed_path

@dsl.component(base_image="pytorch/pytorch:2.1.0-cuda12.1")
def train(data_path: str, epochs: int) -> str:
    # Training logic
    return model_path

@dsl.pipeline(name="ML Training Pipeline")
def training_pipeline(data_path: str, epochs: int = 10):
    preprocess_task = preprocess(data_path=data_path)
    train_task = train(
        data_path=preprocess_task.output,
        epochs=epochs
    )

compiler.Compiler().compile(training_pipeline, "pipeline.yaml")