
Lead Data Engineer, Enterprise-Scale Data Warehousing
Wrocław
Wigi highlights
16+ years, currently operating as Lead Data Engineer
Built a Python ETL framework, moving ~90% of development
Led a legacy data solution redesign, cutting runtime from 12+ hours to 1 - 2 hours
Built a robust testing framework with schema, type and data validation, achieving ~80% test coverage, significantly improving reliability. In another role, segmented complex pipelines into clear, manageable steps and provided a solution for fast materialization of all intermediate steps, publishing each process's output to Athena tables, making data validation by end users and QAs seamless and ensuring transparency across stages.
They also implemented auto Cloudera Data Science Workbench (CDSW) Cleanup, automating user alerting for storage quota breaches, disabling users post-notification and migrating their data to AWS S3 after a set period, and implemented scripts for migration of all Hive tables from Alluxio to AWS S3 due to Alluxio's scalability issues in enterprise environments.
Open to offers
Open to new opportunities
Skills
Work experience
Unlocks when they connect with you.
- certifications· 7
- education· 2
- Files· 4
Similar available Data Engineers
View all- Data Engineer, Biodiversity Data Infrastructure · Amsterdam, Netherlands · 6 Yrs Exp · Remote · $6,100/mo
- Data Engineer (Solutions De Données Massives) · Paris, France · 6 Yrs Exp · Remote · $7,300/mo
- Data Engineer & Tech Lead · Katowice, Poland · 10+ Yrs Exp · Remote · $3,900/mo
- Data Engineer (Scalable Pipeline Engineering) · Kyiv, Ukraine · 4 Yrs Exp · Remote · $2,800/mo
- Enterprise Data Integration Engineer · Hamburg, Germany · 8 Yrs Exp · Hybrid · $7,000/mo