AI Velocity: Building a Human vs AI Delivery Report

Part eight of our series on running an AI agent as a real teammate: what AI velocity actually means, and the delivery report we built to find out. Earlier parts: onboarding the agent, its first task, assigning work to humans, supporting a Sprint Retrospective, building a structured Jira plan, the Jira operating loop, and eight Sprints of results.

After integrating Sophie with Jira and running the workflow through several weekly Sprints, we wanted to answer a question that sounded simple but was not:

How much of our completed delivery should be classified as AI work, and how much should be classified as human work?

The first challenge was not building a chart. It was defining what the chart could honestly claim.

We did not have reliable historical logs showing how many minutes Sophie worked on each issue, how many prompts a task required, how much of a deliverable was written by a person, or how much AI assistance a human used on a human-assigned issue. We also did not have a consistent Jira field recording AI contribution at the time the work finished.

So we did not want to publish a report that pretended to measure authorship, productivity, or causal impact. Instead, we built a narrower retrospective report based on the one historical signal Jira could verify consistently: who held an issue when it closed.

That decision produced a useful report, but it also defined its limits.

The question behind the report

The original Jira ticket framed this as a request to compare regular velocity with AI velocity. In practical terms, that meant comparing completed story points associated with Sophie against completed story points associated with other team members.

Before building anything, we had to separate three different questions:

1. Assignment. Who was responsible for the Jira issue?
2. Contribution. How much of the work was actually performed or assisted by AI?
3. Outcome. Did AI change delivery speed, quality, or team capacity?

Jira gave us a reasonably reliable answer to the first question. It did not give us a complete historical answer to the second or third.

That distinction became the foundation of the report.

The report uses the following assignment-based classification:

• AI: completed issues assigned to Sophie Hermes;
• Human: completed issues assigned to team members other than Sophie;
• Unassigned: completed issues without an assignee, shown separately rather than forced into either category.

What AI work and human work actually mean

In practice those two categories mean something concrete. Sophie executed her own tickets: she moved each task through the board, did the work, and recorded the result. A team member carried out the tickets under their own name. The assignee field is not a label of convenience; it reflects who actually performed the work.

This is still an operational classification rather than a measurement of authorship, and the limitation is not symmetrical. On Sophie’s side the figure is sound, because the agent did the work. What a person contributed there was definition and review: the team specified the task and checked it before closure. On the human side the figure is incomplete, because a team member may use AI tools for research, drafting, code or debugging without any of it appearing in Jira. The practical consequence is that the AI share reported below is a floor rather than a ceiling.

The report makes that limitation visible instead of hiding it.

Assignment is not the same as creation

One clarification belongs before the numbers, because the previous post reported a very different-looking figure for the same Sprints.

There, Sophie was the recorded creator of almost every Sprint issue from Sprint 150 onward. Here, she is the assignee on a much smaller share of completed story points. Both statements are accurate, because they describe different fields.

Creating an issue is the mechanical step of turning an agreed specification into a Jira record. Being assigned an issue means carrying the work itself. Sophie did the first for most of the board and the second for a minority of it. A reader comparing the two posts should expect the creation figure to be high and the assignment figure to be much lower.

Why the first Jira report was not enough

The first instinct was to create the report inside Jira. That was attractive because Jira already contained the issues, Sprint membership, assignees, statuses, and story-point estimates.

The problem was that a native Jira report did not provide the flexibility needed for this analysis. We needed to combine historical Sprint data, classify issues into AI, Human, and Unassigned groups, calculate story-point totals, and show the results with an explanation of the methodology.

We considered Jira marketplace options, including tools such as eazyBI and Custom Charts. They could provide more sophisticated reporting, but the licensing cost was disproportionate to the narrow question we were trying to answer. For our user count, both quotes came in around USD 3,000 per year. Pricing on these products scales with tier, so another team should check its own numbers rather than assume ours.

That did not mean those products were bad choices. It meant that, for this experiment, the cost of a specialized Jira plugin was difficult to justify before we knew whether the metric itself was useful.

The more practical alternative was to use the data we already controlled:

Jira REST API
    -> Python export
    -> Google Sheets
    -> Looker Studio

That architecture separated data collection from visualization and gave us a dashboard we could revise without changing the Jira workflow itself.

The historical export

The historical analysis covered TYD Sprints 146 through 151. The export contained 107 completed issues. That is fewer than the total number of issues in those Sprints, because the report counts only work that reached a completed status.

The resulting story-point totals were:

Classification	Story points	Share of assigned story points
AI-assigned	42	14.63%
Human-assigned	245	85.37%
Unassigned buffer	27	Excluded from the assigned-work percentage

The unassigned figure deserves an explanation, because it is not missing data. Each Sprint normally carries a deliberate five-point cross-project buffer that we leave without an assignee on purpose. At planning time we do not know who will need it. One person may absorb it, or several may share it, in whatever proportion the Sprint turns out to require. Leaving it unassigned is the honest representation of shared capacity.

About the buffer

The buffer is sized deliberately rather than fixed. Five points is the normal allowance, but the team raises it when experience says five will not be enough. Sprint 148 carried seven for exactly that reason: five had proved short in the Sprint before it, and we expected the following Sprint to need more. Across Sprints 146 to 151 the buffer totalled 27 points.

Sprint 149 is the instructive one. It shows zero unassigned points, not because no buffer was planned, but because a team member picked up the buffer during the Sprint and closed it under their name. That is the correct outcome operationally, and it is worth noting for anyone building a similar report: the unassigned column records who held an issue when it closed, not what was planned at the start. Buffer that gets used stops looking like buffer.

The 14.63% figure is calculated as:

AI-assigned story points
------------------------------------------
AI-assigned + Human-assigned story points

In this dataset:

42 / (42 + 245) = 14.63%

The unassigned category is deliberately excluded from that denominator. Including unassigned work as human or AI work would make the result look more complete than the underlying data justified.

The table is useful as a description of assignment distribution. It is not evidence that AI independently delivered 14.63% of the team’s work, and it is not evidence that human delivery was 85.37% free of AI assistance.

Velocity is a team measure

One more framing point belongs here, because the series has been careful about the framework elsewhere.

In Scrum, velocity is a property of the team, not of individuals. Breaking it down per person invites exactly the comparison the framework avoids, because it turns a planning aid into a performance measure. This report splits completed story points by assignee, so it is fair to ask whether it crosses that line.

The distinction we would draw is that the split here is operational rather than evaluative. It exists to describe how much work sat with an automated account, not to rank contributors or set expectations for any individual. The team’s velocity remains the team’s velocity. If a breakdown like this ever started being used to compare people, it would have stopped serving its purpose.

Building the export scripts

The implementation used two complementary pieces.

Python: extracting and classifying Jira data

The Python export script queried Jira for the issues in the target Sprint range and collected the fields required for analysis:

• Sprint;
• Sprint start and end dates;
• issue key;
• summary;
• status;
• assignee;
• story points;
• creator and other audit fields where useful.

It then assigned each issue to one of the three reporting groups:

if issue_is_unassigned:
    group = "Unassigned"
elif assignee_is_sophie:
    group = "AI"
else:
    group = "Human"

The export generated two CSV files:

sprint_issue_data.csv
sprint_metrics.csv

The issue-level file was important for auditability. A dashboard total is easier to trust when someone can trace it back to the individual Jira issues that produced it.

The metrics file fed the summary charts. It contained one row per Sprint with the calculated AI, Human, and Unassigned totals, plus the derived AI share.

Google Apps Script: refreshing the data

The Apps Script connected the export process to Google Sheets so that the dashboard did not depend on manually uploading a new CSV after every Sprint.

The intended flow was:

Jira
  -> scheduled Apps Script refresh
  -> Google Sheets tabs
  -> Looker Studio data source

The historical report was initially generated from the completed Sprint range. We then structured the refresh process so it could take future Sprint data without redesigning the dashboard.

This was an important architectural choice. We wanted the dashboard to update its numbers from the data source while keeping the explanatory text stable.

Designing the Looker Studio report

We organized the report around a few simple questions rather than a large collection of charts.

1. How many story points went to AI and to humans?

The main chart was a stacked column chart:

Completed Story Points by Sprint: AI vs Human

Its dimensions and metrics were:

Dimension:
  Sprint

Metrics:
  AI story points
  Human story points
  Unassigned story points

The stacked format made it possible to compare total delivery while still showing the distribution between categories.

2. How did the AI share change over time?

The second chart showed:

AI Share of Assigned Story Points by Sprint

This was a trend view, not a performance verdict. A rising AI share could mean that the team gave Sophie more work, that they trusted her with more tasks, or simply that the Sprint composition changed. It could not, by itself, prove higher productivity.

3. What are the current totals?

The report used scorecards for:

• AI completed story points;
• Human completed story points;
• average AI share.

It also included a Sprint summary table with the complete breakdown.

4. Can someone audit the total?

The issue-level detail table was the most important chart for trust. It allowed a reviewer to see:

• Sprint;
• issue key;
• summary;
• assignee;
• assignment group;
• status;
• story points.

That table answers a basic but essential question:

Which Jira issues count as AI work, and why?

Why the dashboard text should stay general

One practical lesson came from editing the Looker Studio canvas. The text boxes initially contained Sprint-specific numbers and descriptions. That made the dashboard harder to maintain because every new Sprint required manually rewriting explanatory text.

We changed the explanatory text to describe the method rather than the current result:

This report uses assignment-based classification. AI-assigned issues are assigned to Sophie Hermes. Human-assigned issues are assigned to other team members. Unassigned issues are reported separately. This is not a measurement of authorship or AI assistance within human-assigned work. Jira remains the source of truth for Sprint membership, status, assignee, and story-point estimates.

The numbers belong in scorecards, charts, and tables. The methodology belongs in stable text.

This made the report easier to refresh and reduced the risk that an old Sprint number would remain in a dashboard explanation after the underlying data had changed.

The measurement problem we did not solve

The retrospective report answered an operational question, but it did not solve the deeper measurement problem.

Assignment is not contribution.

The agent executed Sophie’s tasks, but they did not exist in isolation. A person defined the objective, supplied the context, and checked the result before closing it. The Jira assignee field records who did the work, not that collaboration around it.

Likewise, a task assigned to a human may involve AI assistance that is invisible to Jira. A developer might use an AI tool for research, code generation, testing, documentation, or debugging without changing the issue’s assignee.

That is why we would not call the 14.63% result “AI productivity” without qualification. The defensible statement is narrower:

In Sprints 146 to 151, 14.63% of completed assigned story points belonged to issues whose Jira assignee was Sophie Hermes.

That sentence is less dramatic than “AI delivered 14.63% of our work,” but the available evidence supports it.

The next measurement design

For future work, we proposed adding explicit Jira fields to capture contribution at completion time:

• AI Contribution %
• AI Collaboration Mode
• AI Contribution Notes

A simple contribution scale could use:

0   = no AI contribution
25  = minor AI assistance
50  = substantial collaboration
75  = AI produced most of the initial output
100 = AI produced the deliverable with human validation

Nobody should read the contribution percentage as an objective measurement of minutes or authorship. It would be a declared operational estimate, recorded while the work is still fresh.

We could then calculate a weighted story-point metric as:

Story points x AI Contribution %

For example, a five-point task with an estimated 50% AI contribution would contribute 2.5 AI-weighted points to the prospective report.

This approach has limitations of its own. It is subjective, it adds process overhead, and different people may apply the scale inconsistently. But it would answer a more relevant question than assignment alone: how much AI assistance did the people doing the work believe was present?

A later evolution could combine manual estimates with evidence from pull requests, commits, comments, or tool logs. That would be more complex and would introduce privacy and governance questions, so we would start with a simple field rather than pretend that a perfect metric is available.

What the report taught us

The most important result was not the 14.63% number. It was the process of discovering what the number could and could not mean.

Three lessons stood out.

1. Define the unit before building the chart

“AI velocity” sounds precise, but it can refer to assignment, contribution, time saved, output volume, or delivery impact. Those are different metrics.

We started with assignment because it was available and auditable. That was a reasonable first step only because we named the limitation clearly.

2. An auditable imperfect metric is better than an impressive unsupported one

The issue-level CSV and Jira source data made the report inspectable. A reviewer could trace the totals back to issues and assignments.

That transparency mattered more than creating a more ambitious label for the result.

3. Reporting infrastructure should be separate from agent authority

The report did not need Sophie to decide what counted as success. The export read Jira, classified the records using an explicit rule, and calculated the totals outside the agent’s operational loop.

That separation reduced the risk that the system would quietly redefine its own performance.

Our conclusion

We would build the report again, but we would describe it more carefully than we did at the beginning.

The first version of the question was:

How fast is AI compared with humans?

The version we could actually answer was:

How were completed story points distributed between issues assigned to Sophie, issues assigned to other team members, and unassigned issues across the selected Sprints?

That is a narrower question, but it is a real one.

The report gave us a repeatable data pipeline, an auditable dashboard, and a starting point for better future measurement. It also showed why “Human vs. AI” is not a simple race between two workers. The work was collaborative, the evidence was incomplete, and the boundary between assistance and authorship was not stored automatically in Jira.

For us, that is the correct conclusion of this stage:

Before measuring whether AI is faster, define what counts as AI work, record it consistently, and make sure the dashboard does not claim more than the underlying data can support.

More in this series: Part one, onboarding the agent · Part two, its first task in Jira · Part three, assigning work to humans · Part four, supporting a Sprint Retrospective · Part five, a structured Jira plan · Part six, the Jira operating loop · Part seven, eight Sprints in