Eximia IT AB
Tillbaka till bloggen

From Work Item to Pull Request: Building an AI Agent for Azure DevOps

JS

Johan Spännare

#ai #azure-devops #automation #open-source

What if developers could tag a work item and have an AI agent analyze it, propose a solution, and create a pull request—all automatically?

We built exactly that. In four days.

The Problem

Developers spend significant time on routine implementation tasks. Reading requirements, understanding context, writing boilerplate code, creating branches, and opening pull requests. These steps are predictable and often repetitive.

Meanwhile, AI coding assistants have become capable enough to handle substantial implementation work—if given the right context and workflow.

The Solution

We created a template repository that connects Azure DevOps work items to an AI agent running in Azure Pipelines. The workflow is straightforward:

  1. Developer creates a work item with clear requirements
  2. Adds the ai-ready tag
  3. AI analyzes the work item and posts an implementation proposal as a comment
  4. Developer reviews and adds the ai-approved tag
  5. AI implements the solution and creates a pull request
  6. Developer reviews and merges

The AI can also respond to questions via @ai mentions in comments, making it an interactive assistant throughout the development process.

Key Features

Attachment Analysis

The agent uses vision capabilities to analyze screenshots, PDFs, Word documents, and other attachments. UI mockups, error messages, and specification documents become part of the context.

Three Operating Modes

  • Analyze: Reviews work item and proposes implementation approach
  • Implement: Creates branch, writes code, runs quality gates, opens PR
  • Command: Responds to @ai questions in comments

Quality Gates

Before creating a pull request, the agent runs configurable quality checks: build verification, tests, and linting. Failed gates are reported back to the work item.

System-Specific Configuration

Different repositories can have different contexts, prompts, and quality requirements. The agent resolves which configuration to use based on the work item’s target system.

Building With AI

An interesting aspect of this project: we attempted to build it using GitHub Copilot but were unsuccessful. The agentic workflow required—iterative debugging, complex pipeline syntax, cross-system integration—exceeded its capabilities at the time.

The solution was built with Claude Opus 4.5, which provided the autonomous problem-solving needed for rapid development. Four days, 88 commits, 65% of planned features complete.

This highlights a gap: while GitHub Copilot works well for code completion, true AI agentic development requires different capabilities. The agent itself runs on GitHub Copilot for enterprise compliance, but building it required a more capable model.

Enterprise Ready

The solution is designed for enterprise environments:

  • Uses GitHub Enterprise Copilot—all data stays within existing infrastructure
  • Full audit trail in Azure DevOps and pipeline logs
  • Agent can only create PRs, never merge
  • Works as a template or git submodule for organizations

Try It

The template is available on GitHub:

Repository: github.com/JohanSpannare/ado-ai-agent-template

Fork it, configure your Azure DevOps organization, and deploy to a pilot team. The documentation covers setup, local testing, and customization.

What We Learned

Building this reinforced a few things:

  1. Context is everything. The AI performs dramatically better when given work item details, attachments, and codebase context together.

  2. Human-in-the-loop matters. The analyze-approve-implement workflow keeps developers in control while automating the tedious parts.

  3. The right tool for the job. Different AI models excel at different tasks. Using Claude for building and Copilot for running made sense for this project.

  4. Templates accelerate adoption. Making the solution a template repository with clear customization points lowers the barrier for other teams.

AI agents are not replacing developers. They are handling the predictable work so developers can focus on the interesting problems.