Open to new opportunities

Osama Shahid

AI Data Engineer

I build data platforms that move 50M+ records a day — reliably, observably, and on time.

0years in production data
0TB migrated with <5 min downtime
0records processed daily
Osama Shahid

About

Data pipelines are products.
I treat them that way.

I'm an AI Data Engineer with 4 years building production data platforms on AWS and Azure Databricks for a Fortune 500 pharmaceutical client — from CDC-based migrations of 15+ TB to medallion lakehouses serving 40+ dashboards.

My happy place is the unglamorous middle: schema evolution that doesn't page anyone at 3 AM, backfills that just work, and streaming pipelines with honest latency numbers.

0fewer schema incidents
0fewer pipeline failures
0faster Spark jobs

Experience

Where I've shipped

IT Analyst — Senior Data Engineer, Tech Lead

Nov 2022 — Present

Tata Consultancy Services · Client: Fortune 500 pharmaceutical (Life Sciences) · Noida, India

  • Migrated 15+ TB of biomedical and commercial data from SQL Server to Aurora PostgreSQL with under 5 minutes of downtime — CDC pipelines on DMS, Glue PySpark and Step Functions, 120+ stored procedures converted.
  • Built bronze–silver–gold lakehouses on Azure Databricks processing 50M+ records/day at 15-minute freshness — Auto Loader ingestion, Delta Live Tables, Unity Catalog governance.
  • Cut schema-related incidents ~90% with metadata-driven ingestion and data-quality frameworks adopted across 80+ pipelines.
  • Tech lead of a 4-engineer pod; cut pipeline failures 40% across 30+ Airflow DAGs with retries, backfills and SLA alerting.

Projects

Things I've built

sparklens

Client-side Spark event-log visualizer. Drop in a log and get a stage DAG plus skew, spill and shuffle findings — everything runs in the browser, your log never leaves it.

ReactTypeScriptSparkVite

ragdrop

Ephemeral bring-your-own-data RAG. Upload a CSV, chat with it through retrieval — then it self-deletes after 7 days. No accounts, no data kept.

PythonRAGQdrantReact

sql-rag

Natural-language to SQL studio. Hybrid dense + BM25 retrieval with reciprocal-rank fusion, schema pruning, and a repair loop that fixes its own bad queries.

PythonRAGFastAPIReact

pharma-rag

Pharmaceutical Q&A over DailyMed drug labels. Hybrid retrieval, conversational follow-ups, and a ReAct agent mode — every answer fully cited.

PythonAgentsFastAPIReact

privatewire-vpn

Personal WireGuard VPN. Kotlin Android client paired with one-command server provisioning scripts.

KotlinWireGuardUbuntu

osuconnect

Self-hosted chat platform — communities, channels, DMs, voice and screen sharing. NestJS + React over Postgres and Redis, LiveKit for real-time media, all running on a free-tier Oracle VM.

TypeScriptNestJSReactLiveKit

Skills

The toolbox

Languages

PythonPySparkSQLSpark SQLJavaScript

AWS

GlueLambdaStep FunctionsDMSS3Aurora PostgreSQLMSKEventBridgeAthena

Azure & Databricks

DatabricksDelta LakeUnity CatalogDelta Live TablesAuto LoaderDatabricks SQLADF

Data Engineering

ETL / ELTCDCSpark Structured StreamingMedallion ArchitectureDimensional ModelingData QualityData LineageREST APIs

DevOps

AirflowGitHub ActionsDockerCloudFormationTerraformGit

Certifications

AWS Certified Data Engineer — Associate ↗Azure Database Administrator Associate

Verify the AWS cert at the link above with validation no. 7314d9d0ce9349b5b35313dc1d25dbb6

Contact

Let's build something
that scales.

Currently open to data engineering roles and interesting collaborations.