
Apache Airflow & PySpark Data Engineer
Stafford, VA
Wigi highlights
4+ years, currently operating as Mid-level Data Engineer
Developed Spark applications using Python and Apache Spark
Created Spark ETL pipelines on AWS EMR for various data sources
Designed and implemented data pipelines utilizing AWS Redshift stored procedures to facilitate seamless data movement and transformation; implemented data validation and cleansing procedures using SQL to maintain high data quality standards and accuracy throughout the data processing pipeline.
In a previous role, they also developed ETL/ELT pipelines using big data and cloud technologies such as Apache Hive, Apache Spark, Azure Data Factory and Azure Databricks and extracted, transformed and loaded data from source systems to Azure Data Storage services using Azure Data Factory, T-SQL, Spark-SQL and U-SQL Azure Data Lake Analytics.
Open to offers
Open to new opportunities
Skills
Work experience
Unlocks when they connect with you.
- education· 2
- Files· 1
Similar available Data Engineers
View all- Senior Apache Kafka Big Data Engineer | Scala · Odesa, Ukraine · 10+ Yrs Exp · Remote · $2,400/mo
- Data Engineer — Jira & Email Automation · Adelaide, Australia · 10+ Yrs Exp · Remote · $5,900/mo
- Senior Data Engineer / Log Analysis & Petroleum Geology · Cluj-Napoca, Romania · 10+ Yrs Exp · Remote · $3,000/mo
- Scalable Data Infrastructure Engineer · London, UK · 6 Yrs Exp · Hybrid · $7,800/mo
- Clinical Data & SAS Programmer · Timișoara, Romania · 5 Yrs Exp · Remote · $2,400/mo