Principal Systems Support

Job Description:

Our Site Reliability Engineering group within Enterprise Infrastructure combines Operations Excellence with the Development Experience to deliver services at high scale, high availability with resilience by using automation and Infrastructure Code. We build reliability into our ecosystem by applying best practices in Resiliency Engineering, Automation, Observability & Chaos Testing.

The team comes from diverse technical backgrounds, and the responsibilities provide the opportunity for a variety of challenges. Ideal candidates will have a background in either software engineering or systems engineering with a desire to learn the other or previous experience as an SRE. We are looking for a Systems Thinking, SRE Engineer who has helped teams scale through production insights, operational automation, developer guidance, real-time metrics, automation, automation, automation...!

• Bachelor's degree or higher in a technology related field (e.g. Engineering, Computer Science, etc.) required, Master's degree a plus
• 5+ years of hands-on experience deploying and/or supporting highly distributed multi-tiered systems at scale.
• 1-2 years of experience in Cloud development (AWS) and migration skills; Experience with building and operating highly resilient platforms in AWS cloud environments
• 2-4 years of experience in software development with Python, NodeJS, or Java with a focus on SDLC and automation
• Hands-on experience with container orchestration, preferably with Kubernetes
• Experience operating and implementing distributed & highly concurrent service-based

• Ability to automate with various scripting languages (Python, Shell scripting, etc...)
• Experience managing systems using infrastructure as code tools (IAM, ARM, Terraform, Chef, ...)
• Solid understanding of Cloud Computing and DevOps concepts including CI/CD pipelines
• Hands-on Kubernetes skills and knowledge.
• Hands on experience with one or more observability tools (Prometheus, Grafana, ELK/OpenSearch, OpenTelemetry, Datadog, etc...)
• Experienced in Instrumentation with systems skills on building and operating, monitoring, logging, alerting services of distributed systems at scale
• Proven experience in maintaining scalability and resiliency of complex environment.
• Proven experience in implementing advanced observability practices and techniques at scale.
• Demonstrated ability to utilize modern monitoring tools (DataDog, Prometheus, Splunk, ...)
• Experienced in Instrumentation with systems skills on building and operating, monitoring, logging, alerting services of distributed systems at scale
• Ability to triage, execute root cause analysis, and be decisive under pressure
• Experience managing and interpreting large datasets using query languages and visualization tools
• Proficient communication skills with an ability to reach both technical and non-technical audience
• Ability to learn new software, method and practices and bringing them to our developers
• Ability to work with a variety of individuals and groups, both in person and virtually, in a constructive and collaborative manner and build and maintain effective relationships

• Help define and execute a comprehensive reliability and observability strategy, ensuring that Fidelity's systems are always available when our customers need them.
• Bring together technical, procedural, and financial data to reduce toil and increase efficiency
• You will execute plans for technical standardization and process refinement within the engineering organization, especially for Site Reliability Engineers.
• Troubleshoot stack-wide engineering issues related to hardware, software, network, applications, and cloud service providers.
• Coach peer SREs and development teams on how to build highly available systems