We are hiring a Senior Data Integration Engineer, who will support a critical digital transformation project for one of EPAM's top clients.
This is a high-impact role, with the opportunity to advance your skills and grow within a global organization. If you'd like to learn more about this position and project, apply now! Connect with a recruiter today!
Req.#671805849
Responsibilities:
We are seeking an experienced Data Engineer to join our team and work on developing, implementing, and maintaining big data solutions. Our Client's Cloud Platform supports multiple deployed routers (BHR4, BHRx1, K2, CHR) and collects data from those. The ideal candidate will have a strong background in data processing, AWS cloud services, and various ETL tools. The candidate will be responsible for optimizing performance, managing data pipelines, and working with distributed systems. The tasks will include acceptance testing, end-to-end testing of API parameters from the routers to the cloud API, and end-to-end testing of TR-069 parameters from the routers.
Want more jobs like this?
Get jobs in Gunnison, CO delivered to your inbox every week.
#LI-DNI
Responsibilities
- Creating and integrating tables from different data sources on cloud for data acquisition, integration, and analysis
- Collecting router logs, storing them in AWS S3, and analyzing them with Athena to generate statistics
- Performing transformations & actions on RDDs that can be used as data stage for ETL
- Developing automation frameworks by connecting to multiple clusters and databases on the cloud like HBase, Mongo DB, Oracle, Teradata, and SQL Server for achieving simultaneous dataflow
- Configuring Glue crawlers, tables, and Athena external tables from the S3 data source to run SQL queries
- Querying different data sets with Boto3, Pandas, and Python to connect to AWS S3, HBase, and Athena for querying a large set of data and scheduling multiple jobs on EMR
- Using Python, pandas, boto3, and spark to write a reusable code module to handle a large number of datasets
- Developing APIs using Lambda and API Gateway to achieve an abstraction layer to integrate with on-prem systems
- Building end-to-end test cases to validate the data flow of the APIs. Performing testing on router for the API's sent from cloud to router and verify E-E functionality. Testing end-to-end API parameters from the routers to the cloud API, and end-to-end testing of TR-69 parameters from the routers
- Critical: Self-driven with the ability to work independently, get a deep understanding of the domain and context, identify risks and issues, and develop solutions without close supervision
- Hands-on experience with AWS services
- Very good understanding of big data pipeline and common database like relational (MySQL & PostgreSQL), MongoDB, HBase
- Extensive experience with SQL, including modifying and writing complex queries, particularly using window functions
- Experience performing batch/real time processing using Spark
- Experience designing and developing appropriate test automation frameworks and data validation techniques to ensure optimized product performance
- Understanding of networking stack and any wireless protocol - preferably 802.11
- Experience working with routers & embedded consumer products, preferably knowledge of TR-69 CPE WAN Management Protocol
- Experience with Cloud testing tools
- Very strong scripting experience using Spark and Python is a must
- Experience in the Telecommunications Industry is preferred
- Medical, Dental and Vision Insurance (Subsidized)
- Health Savings Account
- Flexible Spending Accounts (Healthcare, Dependent Care, Commuter)
- Short-Term and Long-Term Disability (Company Provided)
- Life and AD&D Insurance (Company Provided)
- Employee Assistance Program
- Unlimited access to LinkedIn learning solutions
- Matched 401(k) Retirement Savings Plan
- Paid Time Off - the employee will be eligible to accrue 15-25 paid days, depending on specific level and tenure with EPAM (accrual eligibility may change over time)
- Paid Holidays - nine (9) total per year
- Legal Plan and Identity Theft Protection
- Accident Insurance
- Employee Discounts
- Pet Insurance
- Employee Stock Purchase Program
- If otherwise eligible, participation in the discretionary annual bonus program
- If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program
EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.
YouTube video player
This posting includes a good faith range of the salary EPAM would reasonably expect to pay the selected candidate. The range provided reflects base salary only. Individual compensation offers within the range are based on a variety of factors, including, but not limited to: geographic location, experience, credentials, education, training; the demand for the role; and overall business and labor market considerations. Most candidates are hired at a salary within the range disclosed. Salary range: $90K - $150K. In addition, the details highlighted in this job posting above are a general description of all other expected benefits and compensation for the position.
In accordance with the LA County Fair Chance Ordinance, you may find a copy of the Notice containing a summary of the Ordinance's key provisions here: Concept FCO Posting 8 27 24 (lacounty.gov)
It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.