Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
summerfortress's avatar
@summerfortress

Big Data Engineer, Enterprise-Scale Data Processing

$3,000/mo

Sofia

Experience: 10+ yearsOpen to: Sofia·

Wigi highlights

10+ years, most recently Senior Big Data Engineer

Reduced data quality processing from ~36000 core-hours to ~25 core-hours

Designed and maintained 100+ data pipelines for datasets from hundreds of GB to TB

In the most recent role, repartitioned petabyte-scale datasets to eliminate small file inefficiencies and improve query performance, and migrated and operated pipelines across on-prem (HDFS/YARN) and AWS (EMR/S3/Delta Lake).

In a previous role as a QA Engineer, they also implemented monitoring and reporting solutions across a range of environments and platforms and developed statistical analysis of call center and its operators' efficiency. Skills include Apache Airflow, AWS Glue, CI, Distributed Systems and Django.

Available in 2 weeks

Ready to start in 2 weeks. Interview now. Actively looking.

Skills
Tech stack
AWS GlueAWS Glue
Apache AirflowApache Airflow
Amazon S3Amazon S3
Apache HadoopApache Hadoop
Apache HiveApache Hive
Apache SparkApache Spark
PythonPython
JenkinsJenkins
DjangoDjango
SQLSQL
Tools & workflows
Machine Learning (ML) Pipelines
ETL
AWS EMR
Distributed Systems
Data Architecture
Livy
YARN
Selenium WebDriver
CI
Continuous Delivery
Work experience
· 5 roles

Senior Big Data Engineer

Company hidden · 3-5 years

Data Pipeline Engineer

Company hidden · under 1 year

Python Backend Engineer

Company hidden · 1-3 years

QA Engineer

Company hidden · 1-3 years

Python Backend Engineer

Company hidden · under 1 year

More on this profile

Unlocks when they connect with you.

  • certifications· 1
  • education· 2

Similar available Data Engineers

View all