Deploying on Inferentia

Learn production deployment patterns for Inferentia-based inference, including SageMaker endpoints, containerized deployments on ECS/EKS, and direct EC2 setups.

Deployment Options

SageMaker Deployment

import sagemaker
from sagemaker.pytorch import PyTorchModel

# Create a SageMaker model with Inferentia
pytorch_model = PyTorchModel(
    model_data="s3://bucket/model_neuron.tar.gz",
    role="arn:aws:iam::role/SageMakerRole",
    framework_version="2.1",
    py_version="py310",
    entry_point="inference.py",
)

# Deploy to an Inf2 endpoint
predictor = pytorch_model.deploy(
    instance_type="ml.inf2.xlarge",
    initial_instance_count=1,
    endpoint_name="my-inferentia-endpoint",
)

# Run inference
result = predictor.predict({"inputs": "Hello world"})

EKS Deployment with Neuron Plugin

Deploy on Kubernetes using the AWS Neuron device plugin:

# Install Neuron device plugin DaemonSet
kubectl apply -f https://raw.githubusercontent.com/aws-neuron/
  aws-neuron-sdk/master/src/k8/k8s-neuron-device-plugin.yml

# Pod spec requesting Neuron devices
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: inference
    image: your-neuron-container:latest
    resources:
      limits:
        aws.amazon.com/neuron: 1  # Request 1 Neuron device
      requests:
        cpu: "4"
        memory: "8Gi"

Auto-Scaling Strategies