Senior QA Automation Engineer (AI-Native Testing) Team: Quality Engineering Level: Senior
About the Client We’re a Spec-Driven Development platform that turns specifications into working software with AI at the core. We’re building the tooling and workflows that let teams ship faster without sacrificing quality — and we hold ourselves to that same bar. Our QA function isn’t an afterthought; it’s an AI-native discipline where engineers use LLMs and agents as first-class tools to design, generate, and evaluate tests.
The Role We’re looking for a Senior QA Automation Engineer to own quality across the platform and set the standard for how AI-native testing is done here. You’ll write robust Playwright suites in TypeScript, orchestrate AI agents to produce test artifacts at scale, and define the evaluation strategy for our LLM-driven features. Beyond testing the platform itself, you’ll use it to validate third-party applications — giving you a rare dual vantage point as both builder and power user. As a senior member of the team, you’ll do more than execute: you’ll shape test strategy, raise the quality bar, mentor others, and influence how quality is engineered across the product. You should be able to operate with a high degree of autonomy and turn ambiguous quality problems into clear, durable systems.
What You’ll Do
Own quality for the platform — define and drive the test strategy for the Spec-Driven Development experience, end to end, from spec ingestion through generated output.
Scale test creation with AI agents — build repeatable workflows using Claude Code, Skills, Agents, and Prompts to author, scaffold, and maintain test artifacts; establish patterns the rest of the team adopts.
Architect API test suites — design reliable, well-structured, maintainable API automation.
Architect UI automation — lead Playwright (TypeScript) end-to-end and component testing, including framework design and best practices.
Direct exploratory and manual testing — cover what automation can’t, and convert findings into durable automated coverage.
Own pipelines and reporting — keep test pipelines fast, green, and trustworthy; define coverage strategy and produce reporting that leadership can act on.
Define LLM evaluation — design and own evaluations of AI-generated output using structured evaluation frameworks, establishing what “correct” means for non-deterministic systems.
Validate third-party apps with the platform — dogfood it against real external applications and feed insight back to the product team.
Mentor and raise the bar — set standards, review others’ test work, and grow the team’s AI-native testing capability.
What We’re Looking For
5+ years in QA automation/test engineering, with a track record of owning quality for complex products.
Deep TypeScript and Playwright expertise (UI and API), including framework and architecture decisions.
Strong test design fundamentals — clear cases, meaningful assertions, zero tolerance for flakiness, honest reporting.
Proven experience designing and maintaining CI test pipelines, coverage strategy, and reporting.
Hands-on experience using LLMs and AI agents in real workflows — and the judgment to know where they help and where they don’t.
Experience with LLM response evaluation and evaluation frameworks for non-deterministic outputs.
Demonstrated autonomy, ownership, and the ability to mentor and influence beyond your own work.
Nice to Have
Hands-on Claude Code and agent-based tooling experience (Skills / Agents / Prompts).
Experience testing AI-powered or spec-driven products.
Track record of standing up a quality function or strategy from scratch.
Why Join
Work at the frontier of AI-native quality engineering — redefining testing workflows, not bolting AI onto old ones.
Real ownership and influence over how quality is engineered on a platform that changes how software is built.
Direct impact on a product you’ll use every day.
