Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
smoothstarship's avatar
@smoothstarship

Lead Data Engineer (Scalable Analytics Infrastructure)

$6,700/mo

London

Experience: 8 yearsOpen to: London·

Wigi highlights

7+ years, currently working as Lead Data Engineer

Developed PySpark scripts for Amazon EMR and EC2

Built Airflow pipelines in Python with retry mechanisms for Snowflake

As a data engineer, performed benchmark testing, Hive performance tuning and optimized PySpark DAGs in data processing pipelines, and designed an Amazon Cloud solution for various ingestions, considering required data storage layers, suitable file formats, data serialization, partitioning and rejection strategy based on the nature of datasets and business requirements, adhering to system design principles.

In another role, implemented a cloud-to-cloud batch solution using Python to ingest files from AWS S3 to Google Cloud Storage through Amazon Lambda and restricted access using Amazon IAM. They also built a cloud-to-cloud real-time ingestion solution in Google Cloud Platform using Python to transfer data from Amazon SQS and Amazon SNS to Google Pub/Sub. Their general capabilities include AWS Glue, Azure Databricks, Computerized System Validation (CSV), Engineering Data Management and GitHub.

Available in 2 weeks

Ready to start in 2 weeks. Interview now. Actively looking.

Skills
Tech stack
AWS GlueAWS Glue
Tools & workflows
Optimization
Azure Databricks
Business Requirements
Kafka Streams
Release Management
GitHub
Engineering Data Management
Stored Procedures
Computerized System Validation (CSV)
Metrics Reporting
Query Writing
Visualization
Enterprise Solution Design
Code Review
Workload Prioritization
Logics
Datasets
Workflow Management
JSON
Work experience
· 3 roles

Risk & Regulatory Data Lead Engineer

Company hidden · 3-5 years

Senior Data Engineer

Company hidden · 1-3 years

Data Engineer - Nike Project

Company hidden · 1-3 years

More on this profile

Unlocks when they connect with you.

  • education· 1

Similar available Data Engineers

View all