We are looking for a Data Infrastructure Engineer tasked with supporting an infrastructure cluster for data processing pipelines.
The ideal candidate will play a critical role in maintaining, enhancing, and troubleshooting issues from the AWS level down to specific configurations of data pipelines. This individual will also implement changes, set up new instances, and manage continuous deployment pipelines to ensure smooth operations.

#LI-DNI

Responsibilities

• React to incidents and diagnose issues stemming from AWS all the way to data pipeline configurations
• Address and fix identified issues or escalate them to the data team with clear and substantiated arguments
• Implement changes within the infrastructure to improve performance and reliability
• Set up new instances as per project requirements
• Manage and optimize continuous deployment (CD) pipelines
• Collaborate with teams across engineering to integrate their work into the infrastructure
• Ensure system security and compliance with relevant regulations
• Monitor system performance and make recommendations for improvements
• Document infrastructure changes and update systems accordingly

• Minimum 2 years of relevant working experience in a data infrastructure environment
• Knowledge of AWS and Kubernetes (K8S)
• Familiarity with Terraform for infrastructure as code (IaC) practices
• Proficiency in either Airflow or Spark for data pipeline implementation
• Background in setting up and managing CI/CD tools such as Jenkins
• Ability to work independently and in a team-oriented environment
• Exceptional problem-solving skills and attention to detail

• Understanding of Flink and Databricks
• Competency in Spinnaker for CD management
• Flexibility to adapt and learn new technologies quickly