We seek a Senior Big Data Engineer to join our remote team for an exciting project focused on developing ETL and ELT pipelines for aggregating commerce purchase and catalogue data from various sources into the data lake.
In this role, you will be responsible for architecting and maintaining our code base for ETL and ELT pipelines, large batch/micro-batch processing, and streaming systems. You will also build the infrastructure required for optimal extraction, transformation, and loading of data from various sources using ADF, Spark, Kafka, or similar technologies.
Experience the freedom of remote work from anywhere in Georgia, whether from the comfort of your home, our modern offices in Tbilisi and Batumi or a coworking space in Kutaisi.
Want more jobs like this?
Get jobs in Tbilisi, Georgia delivered to your inbox every week.
#LI-DNI#LI-TG3
Responsibilities
- Design and maintain our code base for ETL and ELT pipelines, large batch/micro-batch processing, and streaming systems
- Build out the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using ADF, Spark, Kafka, or similar technologies
- Identify, design, and implement internal process improvements, automate manual processes, optimize data delivery, re-designing infrastructure for greater scalability, etc
- Act as a single point of responsibility over any delivery-related matters, including escalations, upsells, ramp-downs for top management and stakeholders
- Ensure a sound and future-proof architecture is planned and the implementation meets the technical quality standards
- Coordinate between multiple disciplines and stakeholders, and ensure that projects are delivered in line with the client's processes and methodologies
- Establish a strategy of continuous delivery risk management that enables proactive decisions and actions throughout the delivery life cycle
- Measure and improve delivery productivity, and serve as a consultant to the Data Engineers in support of quality and timely delivery of work
- Perform production support and deployment activities
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field
- Minimum of 3 years of experience in Data Software Engineering
- Expertise in Apache Spark, Databricks, Scala, and SQL
- Proficiency in Apache Kafka, PySpark, and Spark Streaming
- Skills in building and deploying data pipelines in a cloud environment
- Strong knowledge of data warehousing concepts, data modeling, and ETL/ELT processes
- B2+ English level, both verbal and written
- Familiarity with Microsoft Azure cloud platform