
Data Integration Engineer, ERP, CRM, Supply-Chain Pipelines
Dublin
Wigi highlights
2+ years, most recently Mid-level Data Integration Engineer
Built a Databricks Data Lakehouse pipeline with Medallion Architecture
Rewrote a nightly batch job, cutting refresh latency to under 1 hour
In a past position, built and maintained batch and streaming data pipelines in Python, Spark and SQL on Databricks, processing ∼800 GB/day of ERP, CRM and supply-chain data within a platform handling multi-TB daily volumes. Tuned the team's heaviest Spark jobs through repartitioning, broadcast joins and caching, reducing runtimes by ∼30% and compute costs on those workloads by ∼15%.
They also built Kafka-based ingestion into a Data Lake consuming ∼2M business events/day, enabling near-real-time order-status and CRM updates. A reusable Python data-quality module (schema, freshness, row-count checks) was developed, now used across a dozen pipelines, catching bad loads before they reach downstream reports. The general skill set includes Cloud Storage, Data Modeling, Delta Lake and FastAPI.
Available in 2 weeks
Ready to start in 2 weeks. Interview now. Actively looking.
Skills
Work experience
Unlocks when they connect with you.
- projects· 1
- education· 2
Similar available Data Engineers
View all- Data Engineer — Hive & Modélisation Des Données · Rouen, France · 9 Yrs Exp · On-site · $6,600/mo
- Apache Spark Data Engineer Specialized on PySpark · Alicante, Spain · 4 Yrs Exp · Remote · $3,600/mo
- Data Analytics Engineer (Amazon Redshift, Amazon S3) · London, UK · 5 Yrs Exp · Remote · $5,700/mo
- Qlik Data Engineer | Jenkins · Barcelona, Spain · 10+ Yrs Exp · On-site · $5,100/mo
- Senior Data Engineer / Data Pipelines & PySpark · Utrecht, Netherlands · 10+ Yrs Exp · Remote · $7,600/mo