
Data Engineer, Optimizing Retail Data Workflows
Cambridge
Wigi highlights
5+ years, currently working as Mid-level Data Engineer
Used complex SQL queries to help reduce costs by 15%
Built a star-schema database enabling sub-millisecond data analysis
Created a Python class which cleans and transforms over 120k rows of data before being loaded into a Postgres database; developed a system that extracts retail sales data from five different data sources: PDF documents, an AWS RDS database, a RESTful API, JSON and CSV files.
Developed an end-to-end data processing pipeline hosted on AWS based on Pinterest's experimental processing pipeline, using Lambda architecture for both batch and stream-processing; streamed data in real-time using AWS Kinesis and performed near real-time analysis leveraging a Spark cluster on Databricks.
Open to offers
Open to new opportunities
Skills
Work experience
Unlocks when they connect with you.
- projects· 4
- certifications· 6
- education· 5
Similar available Data Engineers
View all- Data Engineer | Reliability-Focused AI Systems · Berlin, Germany · 10+ Yrs Exp · Remote · $5,300/mo
- Senior Data Engineer — Apache Airflow & Data Build Tool (DBT) · Berlin, Germany · 10+ Yrs Exp · Remote · $6,400/mo
- Data Engineer, Modernizing BI Data Infrastructure · Singapore · 10+ Yrs Exp · Remote · $5,700/mo
- Data Engineer (Real-Time Analytics Pipelines) · Białystok, Poland · 10+ Yrs Exp · Remote · $5,100/mo
- Data Engineer (Real-Time Well Analytics) · Nowy Sącz, Poland · 10+ Yrs Exp · On-site · $4,900/mo