Introduction
At IBM, work is more than a job - it's a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better, but to attempt things you've never thought possible. Are you ready to lead in this new era of technology and solve some of the world's most challenging problems? If so, lets talk.
Your Role and Responsibilities
We are seeking an experienced Senior Spark / AWS Engineer to join our team.
As a key member of our technical team, you will be responsible for designing, developing, and deploying scalable, efficient, and reliable Spark-based applications on AWS.
If you have hands-on experience working with AWS, EMR, Airflow, Terraform, and GitLab, and are passionate about Spark development, refactoring, performance tuning, and AWS engineering, we'd love to hear from you!
Want more jobs like this?
Get Software Engineering jobs delivered to your inbox every week.
Responsibilities:
Spark Development
Design and develop scalable, efficient, and reliable Spark-based applications
Collaborate with cross-functional teams to understand requirements and deliver high-quality solutions
Spark Refactoring
Analyze and optimize existing Spark code for performance, scalability, and maintainability
Refactor code to improve readability, efficiency, and reliability
Collaborate with the team to ensure smooth code deployments
Spark Performance Tuning
Identify performance bottlenecks in Spark applications
Analyze and optimize Spark configurations, such as memory, CPU, and network settings
Implement optimizations to improve application performance and scalability
AWS Engineering
Collaborate with cross-functional teams to ensure seamless integration with AWS services
Ensure AWS security, monitoring, and logging best practices
Required Technical and Professional Expertise
Strong understanding of Spark: APIs, configurations, performance tuning, and best practices
Hands-on experience working with AWS: EC2, S3, EMR, Glue, Lambda, API Gateway, etc.
EMR experience: creating, managing, and optimizing EMR clusters
Airflow experience: creating, managing, and optimizing Airflow workflows
Terraform experience: designing and deploying infrastructure as code
GitLab experience: using GitLab for version control, CI/CD, and collaboration
Scala knowledge: preferred but not required
Excellent problem-solving skills: ability to troubleshoot complex technical issues
* Strong communication skills: ability to collaborate with cross-functional teams and communicate technical ideas effectively
Preferred Technical and Professional Expertise
- Relevant applicable certifications