ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

When to Use This Skill

• Building new ML pipelines from scratch
• Designing workflow orchestration for ML systems
• Implementing data → model → deployment automation
• Setting up reproducible training workflows
• Creating DAG-based ML orchestration
• Integrating ML components into production systems

What This Skill Provides

Core Capabilities

1. Pipeline Architecture

   • End-to-end workflow design
   • DAG orchestration patterns (Airflow, Dagster, Kubeflow)
   • Component dependencies and data flow
   • Error handling and retry strategies

2. Data Preparation

   • Data validation and quality checks
   • Feature engineering pipelines
   • Data versioning and lineage
   • Train/validation/test splitting strategies

3. Model Training

   • Training job orchestration
   • Hyperparameter management
   • Experiment tracking integration
   • Distributed training patterns

4. Model Validation

   • Validation frameworks and metrics
   • A/B testing infrastructure
   • Performance regression detection
   • Model comparison workflows

5. Deployment Automation

   • Model serving patterns
   • Canary deployments
   • Blue-green deploy

…(truncated)