Tag Archives: Forecasting

AI Velocity: Building a Human vs AI Delivery Report Part eight of our series on running an AI agent as a real teammate: what AI velocity actually means, and the delivery report we built to find out. Earlier parts: onboarding the agent, its first task, assigning work to humans, supporting a Sprint Retrospective, building a structured Jira plan, the Jira operating loop, and eight Sprints of results. After integrating Sophie with Jira and running the workflow through several weekly Sprints, we wanted to answer a question that sounded simple but was not: How much of our completed delivery should be classified as AI work, and how much should be classified as human work? The first challenge was not building a chart. It was defining what the chart could honestly claim. We did not have reliable historical logs showing how many minutes Sophie worked on each issue, how many prompts a task required, how much of a deliverable was written by a person, or how much AI assistance a human used on a human-assigned issue. We also did not have a consistent Jira field recording AI contribution at the time the work finished. So we did not want to publish a report that pretended to measure authorship, productivity, or causal impact. Instead, we built a narrower retrospective report based on the one historical signal Jira could verify consistently: who held an issue when it closed. That decision produced a useful report, but it also defined its limits. The question behind the report The original Jira ticket framed this as a request to compare regular velocity with AI velocity. In practical terms, that meant comparing completed story points associated with Sophie against completed story points associated with other team members. Before building anything, we had to separate three different questions: Jira gave us a reasonably reliable answer to the first question. It did not give us a complete historical answer to the second or third. That distinction became the foundation of the report. The report uses the following assignment-based classification: What AI work and human work actually mean In practice those two categories mean something concrete. Sophie executed her own tickets: she moved each task through the board, did the work, and recorded the result. A team member carried out the tickets under their own name. The assignee field is not a label of convenience; it reflects who actually performed the work. This is still an operational classification rather than a measurement of authorship, and the limitation is not symmetrical. On Sophie’s side the figure is sound, because the agent did the work. What a person contributed there was definition and review: the team specified the task and checked it before closure. On the human side the figure is incomplete, because a team member may use AI tools for research, drafting, code or debugging without any of it appearing in Jira. The practical consequence is that the AI share reported below is a floor rather than a ceiling. The report makes that limitation visible instead of hiding it. Assignment is not the same as creation One clarification belongs before the numbers, because the previous post reported a very different-looking figure for the same Sprints. There, Sophie was the recorded creator of almost every Sprint issue from Sprint 150 onward. Here, she is the assignee on a much smaller share of completed story points. Both statements are accurate, because they describe different fields. Creating an issue is the mechanical step of turning an agreed specification into a Jira record. Being assigned an issue means carrying the work itself. Sophie did the first for most of the board and the second for a minority of it. A reader comparing the two posts should expect the creation figure to be high and the assignment figure to be much lower. Why the first Jira report was not enough The first instinct was to create the report inside Jira. That was attractive because Jira already contained the issues, Sprint membership, assignees, statuses, and story-point estimates. The problem was that a native Jira report did not provide the flexibility needed for this analysis. We needed to combine historical Sprint data, classify issues into AI, Human, and Unassigned groups, calculate story-point totals, and show the results with an explanation of the methodology. We considered Jira marketplace options, including tools such as eazyBI and Custom Charts. They could provide more sophisticated reporting, but the licensing cost was disproportionate to the narrow question we were trying to answer. For our user count, both quotes came in around USD 3,000 per year. Pricing on these products scales with tier, so another team should check its own numbers rather than assume ours. That did not mean those products were bad choices. It meant that, for this experiment, the cost of a specialized Jira plugin was difficult to justify before we knew whether the metric itself was useful. The more practical alternative was to use the data we already controlled: That architecture separated data collection from visualization and gave us a dashboard we could revise without changing the Jira workflow itself. The historical export The historical analysis covered TYD Sprints 146 through 151. The export contained 107 completed issues. That is fewer than the total number of issues in those Sprints, because the report counts only work that reached a completed status. The resulting story-point totals were: Classification Story points Share of assigned story points AI-assigned 42 14.63% Human-assigned 245 85.37% Unassigned buffer 27 Excluded from the assigned-work percentage The unassigned figure deserves an explanation, because it is not missing data. Each Sprint normally carries a deliberate five-point cross-project buffer that we leave without an assignee on purpose. At planning time we do not know who will need it. One person may absorb it, or several may share it, in whatever proportion the Sprint turns out to require. Leaving it unassigned is the honest representation of shared capacity. About the buffer The buffer is sized deliberately rather than fixed. Five points is the normal allowance, but the team raises it when experienceContinue reading “AI Velocity: Building a Human vs AI Delivery Report”