We're seeking an experienced Data Engineering Technical Lead to drive our data
infrastructure and analytics initiatives. The ideal candidate will have extensive experience
with AWS data services and Azure Databricks, leading teams, and implementing best
practices in data engineering.
Responsibilities
• Lead and mentor a team of data engineers, fostering a culture of innovation and
continuous improvement
• Architect, design, and implement scalable data solutions using AWS and Azure
Databricks
• Collaborate with cross-functional teams to understand business requirements and
translate them into technical solutions
• Establish and enforce data engineering best practices, including data governance,
Want more jobs like this?
Get jobs in Bangalore, India delivered to your inbox every week.
security, and quality standards
• Drive the adoption of new technologies and methodologies to enhance our data
capabilities
• Manage end-to-end data pipeline development, from ingestion to transformation
and delivery
• Optimize data infrastructure for performance, cost-efficiency, and reliability
Requirements
• 7-10 years of experience in data engineering, with a strong focus on cloud-based
solutions
• Extensive hands-on experience with AWS data services (e.g., Redshift, Glue, EMR,
Athena, Lake Formation)
• Expert knowledge of SQL, Python, and Scala
• Experience with big data technologies such as Apache Spark, Hadoop, and Hive
• Strong understanding of data modeling, ETL processes, and data warehousing
concepts
• Proven track record of leading and mentoring data engineering teams
• Excellent communication skills and ability to work with both technical and non[1]technical stakeholders
• Proficiency in Azure Databricks and its ecosystem
• Design and implement data pipelines using Databricks' ETL capabilities
• Optimize data workflows using Databricks' performance tuning features
• Implement and manage Delta Lake tables for improved data reliability and
performance
• Set up and maintain Databricks workspaces, including cluster configurations and
access controls
• Integrate Databricks with other Azure services and external data sources
• Leverage Databricks' built-in visualization tools for data exploration and reporting
• Implement automated testing and CI/CD pipelines for Databricks projects
• Train team members on Databricks best practices and new features
Preferred Qualifications
• Experience with real-time data processing and streaming technologies (e.g., Kafka,
Kinesis)
• History of successful large-scale data projects using Databricks
• Experience with Databricks' Unity Catalog for fine-grained access control
• Familiarity with Databricks' Auto Loader and Change Data Capture (CDC) features
• Knowledge of integrating Databricks with Azure Synapse Analytics
• Experience with Databricks' Koalas for pandas-like operations on large datasets
• Understanding of Databricks' machine learning capabilities, including AutoML
• Familiarity with machine learning workflows and MLOps
• Knowledge of data visualization tools (e.g., Tableau, Power BI)
• Relevant certifications (e.g., AWS Certified Data Analytics, Azure Data Engineer
Job Segment: Database, SQL, Data Warehouse, Technology