Wigiwork — Remote Tech, AI & Fintech Talent MarketplaceWigiwork — Remote Tech, AI & Fintech Talent Marketplace
Log inSign up
thyonelocksmith's avatar
@thyonelocksmith

Senior Real-Time Analytics Pipeline Engineer

$7,100/mo

Amsterdam

Experience: 9 yearsOpen to: Amsterdam·

Wigi highlights

9+ years, currently Senior Real-Time Analytics Pipeline Engineer

Built a sync system for 150 databases and 250 Kafka topics

Led 200+ ELT & ETL jobs ingesting data from 250+ sources

Implemented automated reporting ETL processes with Apache Airflow, reducing manual labor by 35 hours per month and allowing for the processing of millions of rows of data using SQL. Engineered a real-time monitoring and alerting system with Prometheus and Grafana, enabling proactive identification and resolution of data infrastructure incidents, minimizing downtime and data loss.

Successfully migrated Airflow from 1.x to 2.x, implementing role-based access control for enhanced team autonomy in DAG management, resulting in a considerable reduction in data engineering troubleshooting requests. In another role, implemented and built a high-throughput data scraping architecture using Apache Storm and Java, capable of processing data from 1000+ websites concurrently, enabling real-time data analysis for market research using Apache Solr.

Open to offers

Open to new opportunities

Skills
Tech stack
PythonPython
GrafanaGrafana
MySQLMySQL
TensorFlowTensorFlow
PyTorchPyTorch
TerraformTerraform
Apache HadoopApache Hadoop
ScalaScala
Apache SparkApache Spark
AWSAWS
Apache AirflowApache Airflow
Apache KafkaApache Kafka
LinuxLinux
KubernetesKubernetes
DockerDocker
SQLSQL
PostgreSQLPostgreSQL
MongoDBMongoDB
DjangoDjango
Tools & workflows
YARN
Work experience
· 6 roles

Senior Big Data Engineer

Company hidden · under 1 year

Senior Big Data Engineer

Company hidden · 1-3 years

Data Systems Engineer

Company hidden · 3-5 years

Data Systems Engineer

Company hidden · 1-3 years

Data Engineering Trainee

Company hidden · 1-3 years

Code Engineer

Company hidden · under 1 year

More on this profile

Unlocks when they connect with you.

  • projects· 1
  • certifications· 5
  • education· 2
  • Photos· 1

Similar available Data Engineers

View all