
Senior Real-Time Analytics Pipeline Engineer
Amsterdam
Wigi highlights
9+ years, currently Senior Real-Time Analytics Pipeline Engineer
Built a sync system for 150 databases and 250 Kafka topics
Led 200+ ELT & ETL jobs ingesting data from 250+ sources
Implemented automated reporting ETL processes with Apache Airflow, reducing manual labor by 35 hours per month and allowing for the processing of millions of rows of data using SQL. Engineered a real-time monitoring and alerting system with Prometheus and Grafana, enabling proactive identification and resolution of data infrastructure incidents, minimizing downtime and data loss.
Successfully migrated Airflow from 1.x to 2.x, implementing role-based access control for enhanced team autonomy in DAG management, resulting in a considerable reduction in data engineering troubleshooting requests. In another role, implemented and built a high-throughput data scraping architecture using Apache Storm and Java, capable of processing data from 1000+ websites concurrently, enabling real-time data analysis for market research using Apache Solr.
Open to offers
Open to new opportunities
Skills
Work experience
Unlocks when they connect with you.
- projects· 1
- certifications· 5
- education· 2
- Photos· 1
Similar available Data Engineers
View all- Data Analyst, Data-Driven Decision-Making · Wrocław, Poland · 10+ Yrs Exp · Hybrid · $3,700/mo
- Senior Data Engineer (Optimizing Analytics Pipelines) · London, UK · 10+ Yrs Exp · On-site · $8,000/mo
- Data Analytics Engineer (C, C++) · Tampere, Finland · 9 Yrs Exp · Remote · $6,600/mo
- Senior Data Engineer (Cloud-Native AI/ML Pipelines) · Berlin, Germany · 6 Yrs Exp · On-site · $4,400/mo
- Data Engineer | Reliability-Focused AI Systems · Berlin, Germany · 10+ Yrs Exp · Remote · $5,300/mo