Senior ETL Tester with AI Experience
- Location
- Chicago, Illinois, United States
- Salary
- Not specified
- Company
- akash pawar
Job description
Role & Company / Project Overview: We are seeking a Senior QA professional skilled in ETL, SQL, Azure Databricks, Agentic AI evaluation, web testing, and test automation tools to drive data integrity, compliance, and scalable validation strategies. In this project, we are building AI agents that generate extraction queries and transformation logic, along with the Workbench through which human reviewers approve everything produced. The agents emit metadata which the client's existing data platform executes. Key Responsibilities & Daily Duties: • Own the overall test strategy and reconciliation methodology. • Define the equivalence standard: specify what must match exactly, what may vary within tolerance, and identify at which layer each rule applies. • Design source-to-target validation across the redesigned path covering raw ingestion, consolidated transformation, and domain refinement. • Establish risk-based prioritisation to address schedule constraints that do not permit exhaustive coverage of hundreds of tables. • Author the Test Strategy and Test Plan for formal review with the client in Phase 2. • Own agent evaluation workflows. • Build and maintain a golden reference set from existing production extracts, validated with the Epic SME, enforcing strict separation between evaluation data and anything used to develop or tune agents. • Stand up and maintain the evaluation harness in MLflow for scoring runs, versioning, and candidate comparison. • Define and report critical measures including mapping precision and recall, override rate, confidence calibration, run-to-run consistency, escalation correctness, and cost per case. • Operate the regression gate ensuring no prompt, skill, or model change is promoted without passing evaluation, preventing silent degradation of quality. • Characterise how agents fail qualitatively and quantitatively, including hallucinated tables and plausible-but-wrong columns. Required Qualifications, Skills & Experience: • Senior-level QA experience specializing in ETL testing, SQL, and Azure Databricks. • Direct experience with Agentic AI evaluation and test automation tools. • Strong background in web testing, data reconciliation, and establishing compliance and validation frameworks. • Familiarity with MLflow evaluation harnesses, scoring runs, and prompt/model regression gating. • Experience working alongside Subject Matter Experts (e.g., Epic SME) and managing gold reference datasets. • Local candidates only. Working Hours, Shifts & Location Details: • Location: Chicago, Illinois, United States. • Work Arrangement: Hybrid (3 days work from office / onsite). Application Instructions: • Interested and local candidates should share their updated resume with the hiring team.