Flights Price Prediction MLOps Pipeline¶
How I engineered a zero-overhead, multi-cloud MLOps deployment pipeline¶
Project Type: MLOps & CI/CD Engineering
Goal: Prove I can architect secure, automated, and reproducible machine learning systems
Context: Many ML models fail to reach production due to deployment friction, environment discrepancies, and lack of tracking.
The Business Problem¶
Building an accurate model is only half the battle. This project tackled the operational challenges of deploying ML in the real world:
The Pain Points
- Deployment Friction: Manual, error-prone processes for deploying models to production servers.
- "It Works on My Machine": Environment discrepancies leading to inconsistent predictions and broken dependencies.
- Zero Traceability: Difficulty tracking which data version produced which model, making rollbacks impossible.
The Solution Impact
By automating the entire CI/CD pipeline and implementing centralized tracking, I achieved a 100% reduction in manual deployment overhead while ensuring total model reproducibility.
My Process (How I Approached It)¶
This project demonstrates how I bridge the gap between data science and reliable software engineering:
1. Centralized Experiment Tracking
Deployed an MLflow Tracking Server on AWS EC2 (backed by RDS and S3) to provide a single source of truth for all model metadata, parameters, and artifacts.
2. Immutable Data Versioning
Implemented a Medallion Architecture (Bronze, Silver, Gold) using Parquet and DVC. This linked exact data versions to the code that processed them, guaranteeing reproducibility.
3. Automated CI/CD Workflows
Architected GitHub Actions to automatically lint, test, build Docker images, and push to Google Artifact Registry upon versioned Git tag pushes.
4. Secure Serverless Deployment
Configured Google Cloud Run for blue-green deployments, integrating with Google Secret Manager to inject credentials at runtime—ensuring zero exposure of sensitive keys.
Technical Highlights¶
Why This Stack?¶
- Google Cloud Run: Serverless serving means zero-maintenance scaling and lower costs for fluctuating API traffic.
- LightGBM: Chosen as the champion model for its superior stability and rapid training times (~2.5 mins).
- Multi-Cloud Setup (AWS + GCP): AWS utilized for cost-effective centralized tracking, while GCP handled high-performance serverless inference.
Key Engineering Decisions
- Validation Gates: Integrated Great Expectations at every pipeline stage to enforce strict schema adherence and prevent data corruption.
- Parquet vs. CSV: Switched to Parquet for intermediate layers, achieving massive storage savings and faster columnar I/O.
- Strict Environment Parity: Leveraged `uv` for lightning-fast dependency resolution and multi-stage Docker builds to ensure identical local and production environments.
Results & Impact¶
$7.60
Test Set RMSE (LightGBM)
100%
Manual Deployment Reduction
0.999
R² Score Reliability
Operational Wins:¶
- Zero-Downtime Deployments: Automated blue-green deployments triggered entirely via Git tags.
- High-Performance API: Built a fast prediction API using FastAPI, complemented by an interactive Streamlit frontend for explainability.
- Infrastructure as Code: The entire ML pipeline can be reproduced by any developer using a single command (
dvc repro).
What This Proves About My Process¶
- I Build Systems, Not Just Models: I understand that an ML model is useless if it can't be safely deployed and monitored.
- I Prioritize Security: Integrating Secret Manager and AWS IAM Roles proves I follow production-grade security practices.
- I Engineer for Reliability: Automated testing, validation gates, and strict version control are non-negotiable in my workflow.