AI/ML Engineer – Observability, Evaluation & Agent

Location

Charlotte, North Carolina, United States

Salary

Not specified

Company

Akash pawar

Job description

Role & Company / Project Overview: We are looking for a talented and experienced AI/ML Engineer (Observability, Evaluation & Agent Reliability) to build, evaluate, monitor, and improve next-generation LLM-powered applications and AI agents. This position is based in Charlotte, NC and is open to local or nearby candidates only. Key Responsibilities & Daily Duties: • Build, evaluate, monitor, and improve next-generation LLM-powered applications and AI agents. • Implement and maintain AI observability, telemetry, and distributed tracing. • Define, track, and monitor metrics, dashboards, and SLOs/SLIs for AI and agent systems. • Conduct prompt and model performance analysis to ensure accuracy and efficiency. • Develop and execute AI/ML testing and regression frameworks. • Oversee CI/CD and production reliability engineering for machine learning pipelines and agent workflows. Required Qualifications, Skills & Experience: • Total Experience: 12+ Years • Core expertise in AI/ML & Generative AI • In-depth knowledge of LLM & Agent Evaluation methodologies • Proven experience with AI/ML testing & regression frameworks • Strong programming proficiency in Python & test automation • Hands-on expertise with AI observability, telemetry & distributed tracing • Experience with metrics, dashboards & SLOs/SLIs • Strong background in prompt & model performance analysis • Solid understanding of CI/CD & production reliability • Technical tooling: OpenTelemetry, Prometheus, Grafana, or Datadog • Experience with Cloud, Kubernetes & MLOps Working Hours, Shifts & Location Details: • Location: Charlotte, NC (Local or nearby candidates only) Compensation & Benefits: • Not specified Application Instructions: • Interested and qualified candidates are requested to submit their updated resume for consideration.