- Setting up and automating data pipelines using tools such as Airflow, Kafka amd Rabbitmq
- Providing best practices and executing POC for automated and efficient model operations at scale.
- Good to have hands-on experience using large foundation models (e.g. LLMs) and associated tool chains (e.g. langchain) and APIs to build applications, tools and workflows for production.
- Ability to talk to customer architect team and drive and own the deployment cycle and co-ordinate with AIML developers to deploy the solution

? Gaining direct experience in LLM and LLM-relevant product performance measurement, including creating metrics, designing and conducting tests, to help tech and product teams to understand LLM and product performance.
? Delivering performance and feedback analysis to identify issues in LLM training and product design.
? Collaborating with product managers and engineers to build testing workflows, create feedback loops, and enhance the effectiveness of testing processes and standards.
? Working closely with the annotation team to gauge the quality of training data sets.
Qualification ?
? Professional proficiency in English with excellent or native-level writing and critique skills. Multilingual candidates are preferred.
? Critical thinking and analytical skills are required. Experience in consulting, market research, auditing, operational process design and improvement is valued. Experience as NLP model and search product performance evaluators is strongly preferred.
? Comfort with meeting deadlines and generating creative solutions, including the use of technology and tools, to enhance the quality of both individual and team outputs.
? A deep interest in LLM is essential. The ideal candidate is an enthusiastic learner who finds engagement with diverse case studies stimulating.

LLM Ops