Job Description:
We are looking for a Senior AI AQA Engineer to lead the design, development, and execution of test strategies for our AI/ML-driven systems. In this role, you will play a critical part in shaping how we ensure quality, reliability, fairness, and robustness across our AI products. You will be responsible for ensuring the quality, reliability, and performance of our AI/ML-driven products and systems. You will collaborate with cross-functional teams to test end-to-end pipelines, automate validation processes, and drive industry best practices for AI quality assurance.
What You’ll Do
Work with the QA lead to collaborate on QA strategy for AI/ML systems, including model evaluation, data validation, and pipeline integrity
Design and implement robust automated and manual testing frameworks tailored to AI workflows, model validation primarlily using AI Agents
Validate model performance with metrics such as accuracy, F1 score, precision/recall, bias, drift, and explainability
Drive continuous testing in ML pipelines (CI/CD for AI), enabling scalable model deployment and monitoring including regression testing
Identify and address quality risks associated with data labeling, training data, and model behavior in production.
Collaborate with engineering, DevOps, and product teams to ensure alignment on quality goals.
Test AI systems for accuracy, fairness, bias, drift, robustness, and scalability.
Analyze test results, detect anomalies, and report bugs or inconsistencies using AI agents.
Assist in designing monitoring systems for model performance in production.
Document test cases, tools, and processes for internal teams and audits.
Stay updated on QA best practices for machine learning and AI systems.
Your First 90 Days
You will start by learning about and understanding our products, including from the experiences of our current team. You will be responsible for building Product mind maps to help design, lead and propose QA test strategies for our AI Product by collaborating with the product and eng teams. You will be responsible for building the first set of QA Test cases (manual or automation driven) to start building the QA pipelines.
Who You Are
5+ years of experience in QA engineering, with at least 2+ years focused on AI/ML systems.
Strong proficiency in Python, with hands-on experience testing ML code and pipelines.
Familiarity with testing concepts such as adversarial testing, edge case discovery, and synthetic data generation.
Experience with CI/CD, Git, Docker, and at least one major cloud platform (AWS, GCP, Azure).
Strong communication skills with the ability to influence technical and non-technical stakeholders.
Understanding of test methodologies, test case coverage analysis, and effective hardening and exploration strategies.
Experience building AI testing agents for both manual and automation testing.
Your Differentiators
Ability to work and collaborate with others remotely
Efficiency mindset
Leads by example
Mentor to the team
Problem-solver
Strong analytical skillset
Flexible and adaptable to changing priorities and projects/applications
