
Big Data Developer, Enterprise Data Lake Optimization
New York City, NY
Wigi highlights
9+ years, currently operating as Senior Big Data Developer
Developed Spark code with Scala and Spark-SQL/Streaming
Built a job server and job shell for job submission and monitoring
In the current role, they used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive and implemented solutions for ingesting data from various sources and processing data at rest using Hadoop, MapReduce frameworks, MongoDB, Hive, Oozie, Flume, Sqoop and Talend.
In a previous role, real-time feed was extracted using Kafka and Spark streaming, converted to RDD, processed as Data Frame and saved as Parquet format in HDFS. Also created multi-node Hadoop and Spark clusters in AWS instances to generate terabytes of data and stored it in AWS HDFS.
Available in 2 weeks
Ready to start in 2 weeks. Interview now. Actively looking.
Skills
Work experience
Unlocks when they connect with you.
- education· 1
Similar available Data Engineers
View all- Data Engineer | Azure Databricks & Modelado De Datos · Badalona, Spain · 4 Yrs Exp · Remote · $3,800/mo
- Data Engineer (Generative AI Infrastructure) · Madrid, Spain · 10+ Yrs Exp · Remote · $2,900/mo
- Associate Data Engineer, Process Optimization · Mannheim, Germany · 7 Yrs Exp · On-site · $5,400/mo
- Senior Data Engineer, Enterprise Data Analytics · Chicago, IL · 7 Yrs Exp · Remote · $12,700/mo
- Data Analyst, Data-Driven Decision-Making · Wrocław, Poland · 10+ Yrs Exp · Hybrid · $3,700/mo