Blackthorn Vision is looking for a Senior AI Engineer. Our client, established in 2011 and based in Colorado, is a leading player in the U.S. application management market. We’ve had a successful partnership with this client and are seeking someone who is passionate about exploring the potential of AI, solving real customer problems, and bringing impactful AI capabilities to a product trusted by over 6,000 enterprises.
What you’ll work on
Build AI Insights and agentic product experiences
Develop our AI orchestration microservice using C# / .NET 10, MongoDB, Redis, RabbitMQ, and Kubernetes, working closely with architects and technical leads.
Integrate LLMs using Microsoft Agent Framework and Microsoft.Extensions.AI with Azure AI Foundry / Azure OpenAI, including the Responses API.
Design prompt templates and structured JSON-schema outputs, select the right model for each task, and implement token metering, cost estimation, and budget controls.
Decide what customer data an LLM should see for each task, including devices, vulnerabilities, installed applications, and patch deployments, and design efficient retrieval strategies that balance context quality with token cost.
Evolve the platform from batch-generated insights toward chat and agentic workflows, including tool calling over customer data, multi-step agents, and appropriate guardrails.
Make AI quality measurable through evaluation datasets, prompt A/B experiments on real tenant data, LLM-as-judge evaluation, duplicate detection, source attribution, and latency benchmarking.
Manage AI model deployments and capacity across development, staging, and production environments in US and EU Azure regions.
Improve AI-assisted software development We also use AI inside the engineering process itself. You’ll help maintain a shared GitHub Copilot instruction and tooling catalog used across 40+ engineering repositories, covering:
coding, security, and testing conventions;
reusable prompts;
agents and skills;
Copilot plugins.
You’ll evolve the delivery pipeline that distributes updates to consumer repositories through automated PRs, while supporting repository-specific overrides and opt-outs using Azure DevOps YAML, PowerShell, and CI guards.
Discover and ship new AI opportunities
Identify product problems where AI can provide capabilities that conventional software cannot.
Research ideas and test their feasibility against real product data.
Build prototypes and compare them against meaningful baselines.
Take successful experiments through production rather than stopping at the demo stage.
Help other engineering teams design and integrate AI capabilities.
Write technical proposals for product and engineering leadership.
Turn ambiguous ideas into scoped, deliverable engineering work.
What we’re looking for
5+ years of professional software engineering or ML engineering experience.
2+ years building and shipping LLM-powered features to production, with hands-on experience in areas such as:
prompt design;
structured outputs;
offline evaluation datasets;
A/B experimentation;
LLM-as-judge evaluation;
latency and cost optimization.
Experience building at least one tool-calling agent or conversational assistant over real data, using a framework such as Microsoft Agent Framework, Semantic Kernel, LangGraph, OpenAI Agents SDK, or equivalent.
Strong experimental discipline: you can define a baseline, choose meaningful metrics, run experiments on real data, and interpret disappointing results as honestly as successful ones.
Strong Python skills for experimentation, evaluation, and data analysis.
Practical C# / .NET experience. You should be comfortable implementing production features in an existing ASP.NET Core codebase with support from technical leads.
Hands-on experience using GitHub Copilot or another AI coding assistant as part of your daily engineering workflow, ideally including custom instructions, prompts, agents, or similar extensions


