Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
vastshortfilm's avatar
@vastshortfilm

Senior Data Engineer | DataQuality Engineer

$5,900/mo· Negotiable

Singapore

Experience: 7 yearsOpen to: Singapore·

Wigi highlights

7+ years, currently working as Senior Data Engineer

Designed ETL for millions of daily counterparty records

Built ETL workflows with Spark, Airflow, Python, S3 and Hive

As part of the current responsibilities, partitioning and compression techniques were implemented to optimize data storage, enhancing query performance and reducing costs for large volumes of historical data. In a previous role, they also used Apache Spark, HDFS, S3, Airflow, Hive and Athena to build scalable, efficient workflows.

Other work included designing and implementing ETL pipelines to enable statistical insights on large datasets, comparing on-prem and cloud-based data pipelines. Also built and optimized data pipelines to process student activity from RDBMS and S3, ensuring accurate and timely academic performance reporting. Their general capabilities include AWS Glue, AWS Lambda, Big Data Analytics and Data Streaming.

Open to offers

Open to new opportunities

Skills
Tech stack
LinuxLinux
Microsoft SQL ServerMicrosoft SQL Server
AWS GlueAWS Glue
Amazon S3Amazon S3
AWS LambdaAWS Lambda
Apache SparkApache Spark
Apache AirflowApache Airflow
SQLSQL
PythonPython
Tools & workflows
Athena
Glue
Business Requirements
Query Languages
Relational Databases
Big Data Analytics
Data Integrity
Data Streaming
GCP
Data Processing
Work experience
· 4 roles

Senior Engineer (Data Pipelines & QA)

Company hidden · 3-5 years

Data Systems Engineer

Company hidden · 1-3 years

Team Member

Company hidden · under 1 year

Data Quality Analyst

Company hidden · 1-3 years

More on this profile

Unlocks when they connect with you.

  • certifications· 6
  • education· 1

Similar available Data Engineers

View all