Lilly Tech Systems
Intermediate
Data Synchronization for Hybrid AI
Synchronize training data, model artifacts, and experiment metadata between on-premises and cloud environments with consistency, efficiency, and regulatory compliance.
What Needs Synchronizing?
| Data Type | Direction | Frequency | Volume |
|---|---|---|---|
| Training data | On-prem to cloud | Daily/weekly batch | TB-PB |
| Model artifacts | Bidirectional | Per training run | MB-GB |
| Experiment metadata | Bidirectional | Real-time | KB |
| Feature store data | On-prem to cloud | Streaming/batch | GB-TB |
| Inference logs | Cloud to on-prem | Streaming | GB/day |
Synchronization Tools
AWS DataSync
Automated data transfer between on-premises storage and S3. Handles scheduling, integrity verification, and bandwidth throttling.
Azure Data Box / AzCopy
Bulk data transfer via physical appliance (Data Box) or high-speed network copy (AzCopy) for large dataset migrations.
Rclone / MinIO
Open-source tools for S3-compatible sync between on-premises MinIO clusters and cloud object storage. Flexible and vendor-neutral.
Bash - Rclone Sync Example
# Sync on-prem training data to cloud (incremental) rclone sync \ /data/training/processed/ \ s3-remote:ml-training-data/processed/ \ --transfers 32 \ --checkers 16 \ --s3-upload-concurrency 8 \ --exclude "*.tmp" \ --log-file /var/log/rclone-sync.log # Sync model artifacts from cloud back to on-prem rclone sync \ s3-remote:ml-models/production/ \ /models/production/ \ --transfers 8
Data Governance in Sync
- Data classification: Tag data with sensitivity levels. Only sync data that is approved for the target environment.
- Anonymization pipeline: Apply PII scrubbing and anonymization before syncing sensitive data to cloud environments.
- Audit trail: Log every data transfer with source, destination, timestamp, and data classification for compliance reporting.
- Encryption in transit: Always encrypt data during transfer using TLS 1.3. Use dedicated network connections for large transfers.
Best practice: Synchronize data incrementally, not fully. Use checksums to detect changes and transfer only modified files. For TB-scale datasets, initial sync may require physical transfer (AWS Snowball, Azure Data Box) followed by incremental network sync.