Python Data & ML
Engineering Training
— Islamabad 2026

Data engineering is the work of acquiring, cleaning, transforming, and storing data so it is ready for analysis or model training. This phase covers Pandas (the industry standard) and Polars (the performance-first modern replacement) so you can work in either ecosystem — and make an informed choice for new projects.

1. Series and DataFrame internals: dtypes, memory layout, and why they matter for performance
2. Data ingestion: reading CSV, JSON, Parquet, Excel, and SQL databases with Pandas
3. Data cleaning: handling nulls, duplicates, type coercion, and string normalisation
4. Data transformation: groupby, merge, pivot, melt, and window functions
5. Time series data: DatetimeIndex, resampling, rolling windows, and timezone handling
6. Categorical data and memory optimisation: reducing DataFrame memory footprint by 60–80%
7. Method chaining and pipe: writing readable, composable transformation pipelines
8. Vectorisation vs loops: why loc/iloc/apply patterns matter for performance

9. Why Polars: lazy evaluation, zero-copy memory model, true parallelism — 10–100x performance over Pandas on large datasets
10. Polars expressions: the composable, lazy query API — the fundamental building block of Polars code
11. Lazy vs eager execution: building query plans before materialising results — the key to Polars performance
12. Polars with Parquet: the native storage format for Polars workflows
13. Migrating Pandas code to Polars: the common patterns and where they differ
14. When to use Pandas vs Polars: practical decision framework based on data size and team context

15. Parquet: columnar storage, compression codecs, and why it is the production standard for data pipelines
16. Apache Arrow: the in-memory columnar format that Polars, DuckDB, and modern data tools are built on
17. JSON Lines (JSONL): streaming-friendly format for log and event data
18. DuckDB: in-process SQL analytics engine that queries Parquet and CSV files directly — the fastest path from raw file to SQL query
19. Working with large files that do not fit in memory: chunked processing and streaming reads with both Pandas and Polars
20. Data lake patterns on AWS S3: partitioned Parquet datasets and querying with Athena
21. Data validation with Great Expectations and Pandera: asserting data quality before it enters your pipeline

A one-off data transformation script is not a data pipeline. A pipeline runs on a schedule, handles failures gracefully, retries, logs, alerts, and produces auditable outputs. This phase covers how to build them with the tools the industry has standardised on.

1. ETL vs ELT: the architectural difference and when each pattern applies — why modern data stacks have shifted to ELT
2. Building ETL pipelines in pure Python: extract → validate → transform → load as composable, testable steps
3. Apache Airflow fundamentals: DAGs, operators, sensors, XComs, connections, and the task lifecycle
4. Writing Airflow DAGs in Python: scheduling pipelines, managing task dependencies, and setting retry policies
5. Airflow best practices: idempotent tasks, templating with Jinja, and avoiding common DAG pitfalls
6. Airflow on AWS MWAA: deploying managed Airflow, configuring plugins, and monitoring DAG health
7. Prefect as a modern Airflow alternative: flows, tasks, deployments, and work pools
8. Database pipelines: incremental loads, upserts, and change data capture (CDC) patterns
9. AWS Glue: serverless ETL jobs for large-scale data transformation on S3 using PySpark
10. PostgreSQL as a data warehouse at medium scale: schemas, materialized views, BRIN and GIN indexes for analytics
11. Pipeline monitoring: alerting on failures, data quality regressions, SLO tracking, and dead-letter queues for failed records

Data and ML systems need APIs — to receive data, trigger pipelines, serve predictions, and expose results to applications. FastAPI is the dominant Python API framework in 2026 for anything that needs to be fast, well-typed, and auto-documented. Used by Uber, Netflix, and Microsoft internally.

1. FastAPI fundamentals: routing, path parameters, query parameters, request bodies, and response models
2. Pydantic v2 for request and response validation: models, field validators, computed fields, model_validator, and serialisation
3. Dependency injection in FastAPI: building composable, testable service layers — database sessions, authentication, and configuration
4. Async endpoints: using asyncio properly in FastAPI for I/O-bound operations — when to use def vs async def
5. Database integration with SQLAlchemy 2.0 async: async sessions, transactions, and connection pooling with asyncpg
6. Alembic for database migrations: versioned schema changes in production without downtime
7. Authentication: JWT tokens, OAuth2 password flow, and API key authentication — middleware-based route protection
8. Background tasks: offloading heavy work (data processing, email, ML inference) with FastAPI BackgroundTasks and Celery
9. File upload and streaming: receiving large data files, validating them, and processing them asynchronously
10. Streaming responses: server-sent events for streaming ML inference output token by token
11. OpenAPI documentation: auto-generated Swagger and ReDoc docs, customising schemas, and using the docs UI for testing
12. Testing FastAPI applications: TestClient, async test patterns with httpx, and mocking external dependencies
13. Deploying FastAPI: Docker + Gunicorn + Uvicorn workers, AWS ECS, and AWS Lambda with Mangum

This phase is split into two tiers: classical ML with scikit-learn (the workhorse of most production ML systems) and deep learning with PyTorch (for neural networks, NLP, and computer vision). Both are taught from an engineering perspective — not academic theory or research papers.

1. The ML workflow: problem framing, data preparation, model selection, evaluation, and deployment — the engineering lifecycle
2. Feature engineering: encoding categorical variables, scaling, imputation, and feature selection techniques
3. scikit-learn Pipelines: chaining preprocessing and model steps into a single, deployable, serialisable object
4. Supervised learning: linear and logistic regression, decision trees, random forests, gradient boosting (XGBoost, LightGBM)
5. Unsupervised learning: K-means clustering, DBSCAN, PCA for dimensionality reduction
6. Model evaluation: cross-validation, confusion matrices, ROC-AUC, precision/recall trade-offs, and RMSE
7. Hyperparameter tuning: GridSearchCV, RandomizedSearchCV, and Optuna for Bayesian optimisation
8. Handling imbalanced datasets: SMOTE, class weights, and threshold tuning for precision/recall balance
9. Model explainability: SHAP values for understanding feature importance in production models — a requirement for regulated industries
10. Saving and loading models: joblib, pickle, and ONNX for cross-framework portability

11. PyTorch fundamentals: tensors, autograd, and the computational graph — how backpropagation actually works
12. Building neural networks with nn.Module: layers, activations, dropout, batch normalisation, and forward passes
13. Training loops: loss functions, optimisers (Adam, AdamW), learning rate schedulers, gradient clipping, and early stopping
14. Datasets and DataLoaders: batching, shuffling, and custom dataset classes for structured and unstructured data
15. Transfer learning: fine-tuning pre-trained models from HuggingFace Transformers for classification tasks
16. NLP with transformers: tokenisation, embeddings, and using BERT/DistilBERT for text classification and named entity recognition
17. Computer vision basics: CNNs, image classification, and object detection with torchvision and ResNet fine-tuning
18. GPU training: moving tensors to CUDA, mixed precision training with torch.amp, and gradient accumulation
19. Model checkpointing: saving and resuming training, best model selection, and checkpoint management
20. Exporting models: TorchScript, ONNX export, and preparing PyTorch models for Triton Inference Server

Training a model once in a notebook is not ML engineering. MLOps is the discipline of making ML reproducible, auditable, and maintainable across teams and over time. This phase covers the tools and practices that enterprise ML teams operate on.

1. MLOps principles: reproducibility, versioning, automation, and monitoring as first-class engineering concerns
2. MLflow fundamentals: tracking experiments, logging parameters, metrics, and artefacts from every training run
3. MLflow Projects: packaging ML code for reproducible runs in any environment — Docker and conda backends
4. MLflow Model Registry: versioning trained models, managing staging/production transitions, and model lineage tracking
5. DVC (Data Version Control): versioning large datasets and model artefacts alongside code in Git without LFS overhead
6. Feature stores: what they are and when you need one — Feast for online and offline feature serving
7. Weights & Biases (W&B) as an MLflow alternative: rich deep learning experiment tracking with gradient histograms and media logging
8. Model cards: documenting model capabilities, limitations, training data, and evaluation results — a requirement for responsible AI deployment
9. CI/CD for ML: automated retraining pipelines triggered by data drift detection or scheduled cadences with GitHub Actions
10. Experiment comparison: systematically analysing runs across hyperparameter sweeps with the MLflow UI and programmatic comparison API

A model that is not serving predictions is not doing anything. This phase covers every practical pattern for getting models into production — from self-managed FastAPI endpoints to fully managed SageMaker endpoints with automated monitoring and retraining.

1. Serving scikit-learn and PyTorch models with FastAPI: prediction endpoints, batch inference, and async request queuing
2. BentoML: packaging models with their dependencies into portable, OCI-compliant, deployable services
3. Triton Inference Server: high-performance model serving for GPU-accelerated PyTorch and ONNX models
4. Model caching and warm-up: ensuring low-latency responses on the first request after cold start
5. Batching inference requests: grouping incoming requests for GPU efficiency — dynamic batching in Triton

6. SageMaker overview: training jobs, processing jobs, pipelines, model registry, and real-time endpoints — the full managed ML platform
7. SageMaker Training Jobs: running scikit-learn and PyTorch training at scale on managed compute with custom Docker containers
8. SageMaker Processing: running data preprocessing and post-processing jobs at scale — the alternative to Airflow for ML-specific data work
9. SageMaker Model Registry: versioning and approving models for deployment — approval workflows from the AWS console and API
10. SageMaker Real-Time Endpoints: deploying models to auto-scaling inference endpoints — instance selection and scaling policy
11. SageMaker Serverless Inference: cost-efficient endpoints for low-traffic prediction APIs without idle compute cost
12. SageMaker Batch Transform: running inference over large datasets without a persistent endpoint — cost-effective for periodic scoring
13. SageMaker Pipelines: building end-to-end ML pipelines that chain data processing, training, evaluation, and deployment in a single workflow
14. SageMaker with MLflow: using MLflow tracking with SageMaker training jobs — unified experiment tracking across environments

15. Data drift detection: monitoring input feature distributions over time with Evidently AI — statistical tests for distribution shift
16. Model performance monitoring: tracking prediction accuracy, latency, and error rates in production with Prometheus and CloudWatch
17. SageMaker Model Monitor: automated data quality and model quality monitoring on SageMaker endpoints
18. Retraining triggers: detecting when model performance has degraded and automatically scheduling retraining via EventBridge
19. A/B testing models in production: shadow deployments and traffic splitting between model versions with SageMaker multi-variant endpoints