Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
pleiadesloop's avatar
@pleiadesloop

AWS Lambda & Apache Spark Data Engineer

$11,300/mo

Allen, TX

Experience: 7 yearsOpen to: Allen, TX

Wigi highlights

7+ years, currently Senior AWS Lambda & Apache Spark Data Engineer

Developed Delta Lake and Apache Iceberg workflows with Spark/PySpark

Implemented ETL workflows to migrate data to OneStream

In the current role, implemented modern Lakehouse data management practices, including incremental processing, MERGE/upsert operations, partitioning, schema evolution, snapshots and time travel to support scalable and reliable data processing. During a previous tenure, they also built Databricks and Delta Lake pipelines using PySpark for real-time data ingestion, cleansing, validation and transformation, while monitoring and optimizing Spark workloads.

Separate work included implementing Delta Lake Bronze/Silver/Gold pipelines with incremental ingestion, MERGE/upserts, schema management, deduplication, null handling and source-to-target reconciliation. Also developed data migration workflows from Amazon S3 to Snowflake using Scala and Spark to support efficient and reliable data processing. General capabilities include Amazon Kinesis, Amazon QuickSight and Amazon RDS.

Open to offers

Open to new opportunities

Skills
Tech stack
Apache SparkApache Spark
Amazon RDSAmazon RDS
AWSAWS
Amazon DynamoDBAmazon DynamoDB
AWS LambdaAWS Lambda
Amazon QuickSightAmazon QuickSight
Amazon KinesisAmazon Kinesis
Amazon S3Amazon S3
Amazon CloudWatchAmazon CloudWatch
Amazon RedshiftAmazon Redshift
Tools & workflows
EMR
ETL
Data Migration
Data Pipelines
Data Quality
Data Integration
Glue
IAM
Athena
Work experience
· 4 roles

Data Systems Engineer

Company hidden · 1-3 years

Data Systems Engineer

Company hidden · 1-3 years

Data Systems Engineer

Company hidden · under 1 year

Data Systems Engineer

Company hidden · 1-3 years

More on this profile

Unlocks when they connect with you.

  • education· 1

Similar available Data Engineers

View all