
Lead Data Engineer (Scalable Analytics Infrastructure)
London
Wigi highlights
7+ years, currently working as Lead Data Engineer
Developed PySpark scripts for Amazon EMR and EC2
Built Airflow pipelines in Python with retry mechanisms for Snowflake
As a data engineer, performed benchmark testing, Hive performance tuning and optimized PySpark DAGs in data processing pipelines, and designed an Amazon Cloud solution for various ingestions, considering required data storage layers, suitable file formats, data serialization, partitioning and rejection strategy based on the nature of datasets and business requirements, adhering to system design principles.
In another role, implemented a cloud-to-cloud batch solution using Python to ingest files from AWS S3 to Google Cloud Storage through Amazon Lambda and restricted access using Amazon IAM. They also built a cloud-to-cloud real-time ingestion solution in Google Cloud Platform using Python to transfer data from Amazon SQS and Amazon SNS to Google Pub/Sub. Their general capabilities include AWS Glue, Azure Databricks, Computerized System Validation (CSV), Engineering Data Management and GitHub.
Available in 2 weeks
Ready to start in 2 weeks. Interview now. Actively looking.
Skills
Work experience
Unlocks when they connect with you.
- education· 1
Similar available Data Engineers
View all- Data Engineer, Financial Sector Data Pipelines · Ede, Netherlands · 10+ Yrs Exp · Remote · $7,500/mo
- Data Engineer (Geospatial & AI-driven Data Platforms) · Zurich, Switzerland · 10+ Yrs Exp · Remote · $9,900/mo
- Data Engineer (Data Infrastructure And Insight Enablement) · Berlin, Germany · 10+ Yrs Exp · Remote · $6,600/mo
- Team Lead Data Engineer (Apache Airflow, Snowflake) · Lviv, Ukraine · 10+ Yrs Exp · Remote · $3,400/mo
- Sr Data Engineer (Optimizing Pipelines Across Cloud Platforms) · Porto, Portugal · 5 Yrs Exp · Remote · $3,300/mo