Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
summitsnapper's avatar
@summitsnapper

Data Integration Engineer, ERP, CRM, Supply-Chain Pipelines

$6,100/mo

Dublin

Experience: 2 yearsOpen to: Dublin·

Wigi highlights

2+ years, most recently Mid-level Data Integration Engineer

Built a Databricks Data Lakehouse pipeline with Medallion Architecture

Rewrote a nightly batch job, cutting refresh latency to under 1 hour

In a past position, built and maintained batch and streaming data pipelines in Python, Spark and SQL on Databricks, processing ∼800 GB/day of ERP, CRM and supply-chain data within a platform handling multi-TB daily volumes. Tuned the team's heaviest Spark jobs through repartitioning, broadcast joins and caching, reducing runtimes by ∼30% and compute costs on those workloads by ∼15%.

They also built Kafka-based ingestion into a Data Lake consuming ∼2M business events/day, enabling near-real-time order-status and CRM updates. A reusable Python data-quality module (schema, freshness, row-count checks) was developed, now used across a dozen pipelines, catching bad loads before they reach downstream reports. The general skill set includes Cloud Storage, Data Modeling, Delta Lake and FastAPI.

Available in 2 weeks

Ready to start in 2 weeks. Interview now. Actively looking.

Skills
Tech stack
Apache HadoopApache Hadoop
FastAPIFastAPI
JenkinsJenkins
GitGit
DockerDocker
PythonPython
MongoDBMongoDB
SnowflakeSnowflake
PostgreSQLPostgreSQL
Amazon AthenaAmazon Athena
Tools & workflows
Databricks Products
Delta Lake
ETL Pipeline Design
Data Modeling
CI/CD
GitHub
ETL
GCP
Cloud Storage
Google BigQuery
Work experience
· 1 role

Code Engineer

Company hidden · 1-3 years

More on this profile

Unlocks when they connect with you.

  • projects· 1
  • education· 2

Similar available Data Engineers

View all