
Enterprise Data Pipeline Optimization Engineer
Edinburgh
Wigi highlights
11+ years, currently Enterprise Data Pipeline Optimization Engineer
Migrated IBM DataStage from 9.1 to 11.7 Version
Optimized BigQuery performance by partitioning, clustering and denormalizing data
In one role, optimized a source data extraction pipeline by implementing partition techniques on primary keys to distribute data evenly across DataStage nodes. They also developed and deployed microservices on a GKE cluster in RTL environments using Jenkins, Harbor and Spinnaker tools to automate application deployment and testing.
Other work included streamlining routine problems by implementing automation scripts for recurring issues and minimizing manual intervention. Critical batches were monitored with vigilance and failures promptly addressed alongside routine maintenance tasks. General capabilities include cloud computing, critical thinking, data mining, data visualization and Google Data Studio.
Open to a signing bonus
For the right offer and opportunity, it will happen.
Skills
Work experience
Unlocks when they connect with you.
- education· 1
Similar available Data Engineers
View all- Geodata Engineer / FME Server & ArcGIS Enterprise · Leeuwarden, Netherlands · 10+ Yrs Exp · Hybrid · $5,300/mo
- Data Engineer (Spatial Data Pipelines) · Lille, France · 4 Yrs Exp · Remote · $5,500/mo
- Data Engineer, Data Pipeline Optimization · Lille, France · 9 Yrs Exp · Remote · $5,800/mo
- Data Engineer, Optimizing Supply Chain Data Flows · Manchester, UK · 7 Yrs Exp · Remote · $5,000/mo
- Data Engineer | Databricks Products & Azure Databricks · Timișoara, Romania · 10+ Yrs Exp · Hybrid · $2,300/mo