
Staff Data Engineer, AI-Optimized Data Pipelines
Aachen
Wigi highlights
10+ years, currently working as Staff Data Engineer
Led modern data format work, cutting RAM use 10x and latency 50x
Redesign enabled access to tens of thousands of cores
Designed and implemented the modern and highly scalable data platform of Research from scratch, built on AWS EKS and on-prem Kubernetes for PiB-scale data processing pipelines; led investigations and redesigns of the multi-modal data stack using novel Rust-based file formats beyond parquet, such as Lance, enabling large-scale processing and memory- and compute-efficient shuffled data ingress for distributed training of Vision and Audio models.
In a previous role, also evaluated and performed performance analysis of AI algorithms for detection of fractures, strokes and more from MRI and CT imaging. They also implemented numeric algorithms on x64 and μC architectures. Data Science and Data Warehousing (DWH) are among the general capabilities.
Open to offers
Open to new opportunities
Skills
Work experience
Unlocks when they connect with you.
- education· 3
- Files· 1
Similar available Data Engineers
View all- Team Lead Data Engineer (Apache Airflow, Snowflake) · Lviv, Ukraine · 10+ Yrs Exp · Remote · $3,400/mo
- Sr Data Engineer (Optimizing Pipelines Across Cloud Platforms) · Porto, Portugal · 5 Yrs Exp · Remote · $3,300/mo
- Data Engineer (Automated Analytics Infrastructure) · Lutsk, Ukraine · 4 Yrs Exp · Remote · $2,500/mo
- Data Engineer & Analyst · Leuven, Belgium · 10+ Yrs Exp · On-site · $5,700/mo
- ETL Developer | Talend & ETL · Warsaw, Poland · 5 Yrs Exp · Remote · $3,600/mo