Our Customer:
A technology company developing an AI-based application for enterprise users. The project focuses on delivering accurate, reliable, secure, and user-friendly AI-powered functionality.
You will join a cross-functional team and work closely with developers, AI engineers, product owners, and business stakeholders to ensure the quality of AI features throughout the product lifecycle.
Your Tasks:
Analyze project requirements, user stories, and AI-related business scenarios.
Create and execute test cases and test scenarios for AI features and workflows.
Perform functional, system, integration, regression, and end-to-end testing.
Validate AI-generated responses for accuracy, relevance, completeness, clarity, and consistency.
Test different types of user inputs, including valid, invalid, incomplete, and ambiguous queries.
Identify hallucinations, incorrect or inconsistent responses, bias, and gaps in AI understanding.
Prepare prompt-based test scenarios and test different prompt variations.
Verify that AI follows defined instructions, guardrails, business rules, and expected tone.
Validate AI responses against approved knowledge sources and expected outcomes.
Test integrations between AI components, APIs, databases, knowledge bases, and third-party or enterprise systems.
Verify authentication, authorization, role-based access, and data correctness.
Perform regression testing after changes to prompts, models, APIs, or business logic.
Maintain reusable regression test suites.
Test AI behavior in edge cases and unexpected user scenarios.
Log defects with clear reproduction steps, test data, expected and actual results, and severity.
Track defects using Jira, Azure DevOps, or similar tools.
Work closely with developers, AI engineers, product owners, and business stakeholders to resolve issues.
Provide testing results and release sign-off input.
Required Experience and Skills:
3+ years of commercial experience in software testing / QA.
Strong understanding of functional, regression, system, integration, and end-to-end testing.
Experience creating test cases, test scenarios, and test data.
Experience testing AI-based applications, chatbots, APIs, enterprise applications, or similar solutions.
Experience validating AI-generated outputs for correctness, relevance, and quality.
Understanding of AI concepts such as prompts, LLM responses, hallucinations, context, and knowledge retrieval.
Experience with defect tracking tools such as Jira, Azure DevOps, or similar.
Good analytical and problem-solving skills.
Strong attention to detail and ability to identify inconsistencies in AI-generated responses.
Good communication skills and ability to work effectively with cross-functional teams.
Would be a Plus:
Hands-on experience testing Generative AI, NLP, ML, chatbot, or conversational AI solutions.
Experience testing RAG-based or knowledge-based AI applications.
Basic understanding of prompt engineering and LLM behavior.
Experience with API testing tools such as Postman or Swagger.
Experience with automation testing tools such as Selenium, Playwright, or similar.
Knowledge of Azure OpenAI, Azure AI, Microsoft Copilot Studio, Power Platform, or similar AI platforms.
Understanding of prompt injection, data leakage, access control, and responsible AI testing.
Experience testing AI integrations with databases, knowledge bases, or third-party services.
