We are seeking a seasoned Data Integration Engineer to take charge of onboarding a new Data Provider.
This role involves crafting new pipelines for Data Ingestion, Raw, Curated, and Consolidated stages using Databricks, employing existing templates as a foundation. The ideal candidate will leverage their expertise in Databricks and PySpark to enhance data processing and integration.
#LI-DNI
Responsibilities
- Design and develop scalable data pipelines for Ingestion, Raw, Curated, and Consolidated phases
- Utilize Databricks to build efficient and reliable data flows
- Implement PySpark to process large datasets within the pipelines
- Collaborate with data teams to establish pipeline architecture and best practices
- Optimize data pipelines for maximum performance and scalability
- Ensure data integrity and quality throughout the pipeline process
- Monitor and troubleshoot any issues that arise in data pipelines
- Conduct regular system tests and updates to enhance functionality
- Document pipeline designs and modifications comprehensively
Want more jobs like this?
Get jobs in Novoukrayinka, Ukraine delivered to your inbox every week.
- Minimum of 2 years of working experience in data pipeline engineering or a related field
- Proficiency in Databricks and PySpark
- Background in designing and implementing complex data pipelines
- Skills in data processing and workflow management
- Qualifications in system architecture related to data management
- Capability to debug and optimize pipeline performance
- Familiarity with Amazon Web Services
- Knowledge of Apache Airflow
- Understanding of Python and SQL programming languages
- Work on a flexible schedule remotely or from any of our comfortable offices or coworking spaces in Ukraine
- Receive the necessary equipment to perform your work tasks
- Change projects and technology stacks within EPAM
- Gain experience in various business domains (Insurance, E-commerce, Healthcare, Finance, Travelling, Media, Artificial Intelligence, and more)
- Consider relocation options in over 30 countries worldwide
- Participate in volunteer, charity programs and communities (both technical and interest-based)
- You can plan your individual career path together with your manager
- Receive regular feedback from colleagues
- Improve your English for free with certified teachers (Speaking Clubs, client interview preparation courses, etc.)
- Get the opportunity to undergo free training and certification in AWS, GCP, or Azure Clouds
- Use the internal E-learn training program (18,200+ specialized training and mentoring programs)
- Access corporate accounts on LinkedIn Learning, Get Abstract and other partner resources
- Study at EPAM Solution Architecture School with the instructors who are practicing architects
- Develop as a leader, join Delivery Management, Resource Management, Leadership Essentials school and more
- Participate in internal communities (500+ meetups, technical discussions, brainstorming sessions, online events and conferences annually)
- Vacation and sick leave (including a sick leave without a medical certificate)
- A wide range of Voluntary Medical Insurance programs providing both medical treatment and various preventive options (including sports activities)
- Medical insurance for family members at corporate rates
- Company support during significant life events (childbirth or adoption, marriage, etc.)
- Support for psychological comfort: discounts on services from mental health specialists or coaches, thematic training
- E-kids program - a free programming language training program for EPAMers' children