Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
devcapuchin's avatar
@devcapuchin

Lead Data Engineer, Enterprise-Scale Data Warehousing

$4,500/mo

Wrocław

Experience: 10+ yearsOpen to: Wrocław·

Wigi highlights

16+ years, currently operating as Lead Data Engineer

Built a Python ETL framework, moving ~90% of development

Led a legacy data solution redesign, cutting runtime from 12+ hours to 1 - 2 hours

Built a robust testing framework with schema, type and data validation, achieving ~80% test coverage, significantly improving reliability. In another role, segmented complex pipelines into clear, manageable steps and provided a solution for fast materialization of all intermediate steps, publishing each process's output to Athena tables, making data validation by end users and QAs seamless and ensuring transparency across stages.

They also implemented auto Cloudera Data Science Workbench (CDSW) Cleanup, automating user alerting for storage quota breaches, disabling users post-notification and migrating their data to AWS S3 after a set period, and implemented scripts for migration of all Hive tables from Alluxio to AWS S3 due to Alluxio's scalability issues in enterprise environments.

Open to offers

Open to new opportunities

Skills
Tech stack
Apache AirflowApache Airflow
AWSAWS
AWS GlueAWS Glue
DockerDocker
ScalaScala
Apache SparkApache Spark
Apache HadoopApache Hadoop
PythonPython
SQLSQL
BashBash
LinuxLinux
AnsibleAnsible
Oracle DatabaseOracle Database
Tools & workflows
PySpark
Data Pipelines
Cloudera
EMR
Data Architects
Data Warehousing (DWH)
ETL
Work experience
· 11 roles

Data Engineering Lead

Company hidden · 1-3 years

Data Engineering Lead

Company hidden · 1-3 years

Data Engineering Lead

Company hidden · 1-3 years

Data Engineering Lead

Company hidden · under 1 year

Data Warehousing Architect

Company hidden · 3-5 years

Senior Oracle/ETL Developer & Team Lead

Company hidden · under 1 year

Senior Oracle/ETL/BI Developer & Team Lead

Company hidden · 1-3 years

Senior Oracle/ETL Engineer

Company hidden · 1-3 years

Oracle Analytics Architect

Company hidden · under 1 year

ERP Software Developer

Company hidden · 1-3 years

Junior ERP Developer

Company hidden · 1-3 years

More on this profile

Unlocks when they connect with you.

  • certifications· 7
  • education· 2
  • Files· 4

Similar available Data Engineers

View all