Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
rhythmsecond's avatar
@rhythmsecond

Python Data Engineer (Google BigQuery)

$5,500/mo

Amsterdam

Experience: 10+ yearsOpen to: Amsterdam·

Wigi highlights

11+ years, currently working as Mid-level Python Data Engineer

Implemented CI/CD pipelines with Jenkins for Spark

Optimized Spark jobs processing 200-300 GB of data

Fine-tuned Spark applications to handle complex join operations involving more than 15 sources in a single dataset, significantly improving the performance and efficiency of the data processing pipeline; developed data assets using PySpark to perform transformations and mappings, facilitating loading of data from multiple banking system portfolios into GCS.

Developed Sqoop scripts to import data from Teradata, ensuring data integration accuracy and efficiency; employed a Test-Driven Development approach, resulting in functional and unit test cases with 100% code coverage.

Open to offers

Open to new opportunities

Skills
Tech stack
dbtdbt
LookerLooker
DockerDocker
KubernetesKubernetes
Apache KafkaApache Kafka
Amazon S3Amazon S3
AWS LambdaAWS Lambda
Amazon EMRAmazon EMR
Amazon AthenaAmazon Athena
ScalaScala
PythonPython
TerraformTerraform
Google Cloud DataflowGoogle Cloud Dataflow
Tools & workflows
Problem Solving
GCP Cloud Functions
GCP Pub/Sub
Google Dataproc
Google BigQuery
Work experience
· 7 roles

Data Systems Engineer

Company hidden · 3-5 years

Data Engineer - Tier 3

Company hidden · under 1 year

Senior Business Consultant - Risk Mitigation

Company hidden · under 1 year

Software Development Engineer II

Company hidden · 1-3 years

Tech Systems Analyst

Company hidden · under 1 year

Senior Systems Architect

Company hidden · 1-3 years

IT Systems Specialist

Company hidden · 1-3 years

More on this profile

Unlocks when they connect with you.

  • certifications· 8
  • education· 2

Similar available Data Engineers

View all