Context
We are building a complex agent that analyzes documents and produces defined data structures extracted from them. Three properties are critical: the structures must be accurate — faithful to the source documents, with exact references and exact quotes; valid — types conforming to their schemas, required fields filled, and so on; and free of duplicates — not repeating structures that were already accepted or rejected before.
Must-have
- 3+ shipped LLM-powered systems in production, each with a measurable quality harness the candidate built personally (golden sets, precision/recall tracking, regression gates in CI). Be ready to walk through one in depth — at whatever level your own NDAs allow — including a concrete regression the harness caught and what changed because of it. - Proven ability to split model work from deterministic validation: pipelines where extraction is model-driven but every acceptance check (dedup, linkage, schema, rule-conformance) is executable code. Prompt-only quality control is a disqualifier. - Information-retrieval and data-modeling depth: hybrid exact+semantic indexing; identity-keyed deduplication (record identity over embedding similarity); append-only stores with supersession semantics; incremental re-indexing without full rebuilds. - Multi-pass agent-pipeline engineering: idempotent, sumable fan-out/verify loops (extract -> validate -> dedup/link -> adversarial verify -> loop-until-dry), token/cost budgeting, caching. - Evidence-first design method: for every architectural element you propose, you state its precondition and how it is verified; you back your designs with executable acceptance checks against golden sets, not documents alone.
Nice-to-have
- Experience working with and analyzing regulated-domain documents (insurance/finance/legal). - Event-sourced or lakehouse-style record stores (append-only, versioned, time-travel). - Knowledge-graph construction and record linkage / entity resolution. - Adversarial-verification agent patterns (a dedicated verifier stage that tries to refute extractions). - Custom eval-suite tooling built when off-the-shelf eval frameworks did not fit.
