
Data Integration Engineer / Azure Data Factory & Apache Hadoop
New York City, NY
Wigi highlights
6+ years, currently working as Senior Data Integration Engineer
Optimized PySpark data workflows with efficient joins and caching
Set up Azure DevOps and Jenkins CI/CD pipelines for data solutions
In the current role, designed and implemented data pipelines for banking transaction data, integrating real-time data streaming via AWS Kinesis for faster data processing, and built a data warehouse using AWS Redshift to enable scalable and efficient analytics and reporting for banking data.
In a past role, they also implemented Slowly Changing Dimensions (SCD) Type 1 and Type 2 in Azure Synapse Analytics, enabling accurate historical tracking and reliable reporting. Separate work involved designing data storage solutions using Hadoop HDFS for storing raw network data and making it available for downstream analytics. Amazon DynamoDB is among the listed skills.
Open to offers
Open to new opportunities
Skills
Work experience
Unlocks when they connect with you.
- certifications· 1
- education· 1
Similar available Data Engineers
View all- Data Engineer, Real Estate Data Platform · Lviv, Ukraine · 8 Yrs Exp · Remote · $2,300/mo
- Azure Data Engineer (Amazon RDS, Microsoft Azure) · Belfast, UK · 8 Yrs Exp · Remote · $6,300/mo
- Data Engineer - Anti-Fraud Detection · Nice, France · 2 Yrs Exp · Hybrid · $6,100/mo
- Data Integration Engineer, Business Intelligence Solutions · Berlin, Germany · 10+ Yrs Exp · Remote · $7,700/mo
- Data Engineer, Cross-Sector Analytical Infrastructure · Birmingham, UK · 10+ Yrs Exp · Remote · $6,100/mo