Eight Sprints with an AI Agent: What Actually Changed

Part seven of our series on running an AI agent as a real teammate: eight Sprints with an AI agent on the Jira board, and what actually changed. Earlier parts: onboarding the agent, its first task, assigning work to humans, supporting a Sprint Retrospective, building a structured Jira plan, and the Jira operating loop.

In the previous post, we described the constrained Jira operating loop we built around Sophie: a scheduled agent that works only from explicit instructions, processes work assigned to its own account, and stops when a human decision is required.

This is the more important question: what happened after we ran that system repeatedly with a real team?

The short answer is that Sophie became useful, but not because she could make product decisions. Her value came from performing clearly specified operational work consistently: creating and assigning Sprint tasks, processing tasks assigned to her account, and producing bounded analysis. The failures were equally instructive. When a ticket carried a title but not the business context behind it, she could produce something well formed and completely wrong.

This review covers eight consecutive weekly Sprints, TYD Sprints 146 through 153. At the time of writing, Sprint 153 is still in progress. The completed-Sprint evidence therefore covers Sprints 146 to 152, while Sprint 153 shows the current operating pattern rather than a final outcome.

We use she/her for Sophie throughout this post, and “it” only for the integration, configuration, or workflow itself.

The evidence we did and did not collect

Jira is the source of truth for the table below. It reports the issues currently in each Sprint and the number whose recorded creator is Sophie. It does not prove that every issue was autonomously designed by an AI, and it does not substitute for human product ownership. It documents an operational fact: who created the Jira record.

Sprint	Issues in Sprint	Created by Sophie	Model period
146	17	0	DeepSeek V4 Pro
147	22	0	DeepSeek V4 Pro
148	23	0	DeepSeek V4 Pro
149	18	0	DeepSeek V4 Pro
150	15	14	DeepSeek V4 Pro
151	16	15	DeepSeek V4 Pro
152	23	21	Transition: DeepSeek V4 Pro to GPT-5.6 Luna to GPT-5.6 Terra
153*	24	24	GPT-5.6 Terra

* Sprint 153 was active when this article was prepared.

Two things stand out.

First, the recurring creation workflow did not become visible in Jira all at once. Sophie was not the recorded creator of the Sprint issues in Sprints 146 to 149. From Sprint 150 onward, she became the recorded creator of almost all Sprint work items: 14 of 15 in Sprint 150, 15 of 16 in Sprint 151, 21 of 23 in Sprint 152, and all 24 currently listed in Sprint 153.

Second, we did not record every metric that would make a perfect case study. We did not systematically track correction counts per Sprint, clarification round-trips, or human minutes spent maintaining the system. Rather than reconstruct numbers from memory, we are leaving those cells out. A useful article should distinguish measured operational data from impressions after the fact.

What the agent was actually doing

By this stage, Sophie had two recurring responsibilities:

1. Create the tasks for the following Sprint and assign them to the appropriate team members from an explicit specification.
2. Process the work explicitly assigned to Sophie’s own Jira account.

That is a narrower role than “AI Sprint planning.” Humans still own priorities, scope, and the decisions that make a Sprint meaningful. But the mechanical conversion of an agreed task list into Jira records is repetitive work. Once the structure and assignments are explicit, it is a good candidate for automation.

Worth being precise about the framework here. Scrum leaves task distribution to the Developers, who internally decide who does what. The specification, including the assignments in it, came from the team. Sophie converted an agreed list into Jira records rather than deciding who should carry which item. Part three covers that boundary in more detail.

The evidence in Jira supports that distinction. From Sprint 150 onward, the agent regularly created the Sprint’s work items. That is useful operational leverage, not a claim that the agent independently chose the work.

What broke in production

The most valuable production failures were not syntax errors or API outages. They were cases where the system returned a plausible result that was wrong in the team’s real context.

1. A title is not a specification

One task asked Sophie to create email copy for a Moth Ball follow-up. The Jira issue contained a title but no description.

Sophie completed the literal request: she produced several polished email options. The problem was that she interpreted “moth ball” as a consumer product and wrote a product follow-up email. The task was actually related to an event, so the output was irrelevant to the intended audience and purpose.

This was a complete failure of the deliverable, even though the prose was competent.

That distinction matters. Sophie did not fail to generate text. She failed because the ticket did not provide enough context to determine what the text was for. A title can name work; it cannot always specify it.

The process lesson is simple:

When a task depends on event context, audience, project vocabulary, or an internal acronym, that context must be present in the issue, not assumed to exist in the agent’s head.

We saw a related pattern with the internal label TYY, which our team uses for Tending Your Yard. That is different from TYD, the Jira project key. A compact team abbreviation may be obvious to people who work with it every day, but it is insufficient context for an agent unless the ticket defines it or links to the relevant project information.

2. Jira authentication was not the same as usable access

Our initial Jira integration used scoped API tokens. Those tokens did not give Sophie the board access or data retrieval the workflow required.

We changed to a Classic Jira API token. The integration then worked on the first attempt.

The lesson was not that one token type is universally better. It was that a configured token is not proof that the agent has the specific board access its workflow needs. We now treat board visibility and required API reads as explicit post-setup checks.

3. Jira field names carry more meaning than they appear to

We also had to correct our handling of parent. We initially treated it as if it could refer to an ordinary work item that sounded like a parent task. In our Jira configuration, it had to refer to an Epic for the issue types involved.

That was a valuable correction because it exposed a broader rule: natural-language labels are not identifiers. The automation must validate the real Jira entity and its key before writing a relationship that affects project structure.

What we changed as a result

The changes were mostly process changes, not a rewrite of the system.

Before	After
A short title could be treated as enough instruction for a content task.	A task that relies on business context needs a description, relevant source material, or a clear human-review boundary.
The automation treated parent as a generic relationship.	It resolves and validates the required Epic relationship before setting the field.
A successful auth response could be treated as setup completion.	Board access and required data retrieval are verified after authentication.
The agent moved finished work straight to Done, and some of it still needed corrections.	We added a Review column that agent work has to pass through. Human-assigned work does not use it, because it is a control on agent output rather than a change to the whole board.

The Review column came out of a specific failure. Sophie was moving completed work directly to Done, and we found tasks marked complete that still needed correction. The column now sits between her work and closure, and nothing she finishes is closed until a person has checked it.

It is important because it reflects a change in how we think about autonomy. Completion is not one thing. A finished, self-contained analytical deliverable may be safe to close when its task explicitly permits that. Content, ambiguous work, or anything that depends on team context needs a human handoff before it becomes a final result.

What one-week Sprints changed

Our team already worked in weekly Sprints, so we cannot compare this directly with a previous two-week or monthly planning cadence. What we can say is that a one-week Sprint made the value of recurring operational work immediately visible.

Creating and assigning the next Sprint’s task list is not difficult work in isolation. It is work that returns every week, often at the point when the team has the least spare attention for administrative steps. Once Sophie had an explicit task to turn an agreed specification into Jira issues and assignments, teammates reported that the Sprint-creation step took less time.

Weekly Sprints also make mistakes more expensive. There is little slack. A context failure on a Monday can consume a meaningful portion of a five-day work cycle, especially if it creates the appearance of progress before a person discovers that the output is off-target.

That is why the Moth Ball failure mattered. It was not a theoretical hallucination benchmark. It was a reminder that, in a one-week Sprint, reviewing the assumptions behind a polished answer is part of the delivery process.

Changing models without rebuilding the integration

The model changed twice during this period:

• The system used deepseek-v4-pro through Sprint 151 and the beginning of Sprint 152.
• We switched to gpt-5.6-luna on 28 July.
• We switched again to gpt-5.6-terra on 30 July.

The reason was output quality and efficiency. We did not record a single reproducible Luna failure that triggered the second switch; the decision from Luna to Terra was a qualitative model-choice decision rather than the result of a controlled benchmark. We did not reset the Jira integration, change the scheduler, or rewrite the operating instructions. We changed the model configuration and continued with the same workflow.

The current model configuration is:

model:
  default: gpt-5.6-terra
  provider: openai-codex
  base_url: <your OpenAI-compatible endpoint>
  # api_mode is not present in this installation. Verify the provider's
  # protocol default before copying this block to another endpoint.

The endpoint is intentionally omitted because it is environment-specific. The important reproducibility point is the model identifier: gpt-5.6-terra.

Why there is no api_mode here

That block will look like a contradiction to anyone who read Part 1, where the second hard-won lesson was to pin the wire protocol explicitly instead of letting it be inferred. There is no api_mode here. The distinction is where the protocol comes from.

In Part 1, the broken configuration pointed at a raw endpoint URL and declared nothing about which protocol to speak, so the protocol was inferred from the URL shape, and inferred wrongly. The corrected configuration in that post fixed the problem by naming a generic provider and pinning api_mode explicitly. Here, provider names a known provider type, and the protocol follows from that identifier rather than from the endpoint string. The lesson from Part 1 has not been abandoned. It is satisfied by a different field.

The practical rule for a reader is unchanged: never leave the protocol to be guessed. If your configuration points at a generic endpoint rather than a named provider, pin api_mode explicitly, and confirm the effective protocol after any provider change rather than assuming the default is what you want.

This was a clean portability test. It supports the practical claim that Hermes separates the model choice from the Jira operating loop. It does not support a claim that Terra is better across a completed Sprint sample: Sprint 152 contained the transition, and Sprint 153 was still active when this article was prepared. The next completed Sprints are the evidence needed for that comparison.

Trust calibration: what Sophie can do, and what still needs a person

At the beginning, the experiment was intentionally small. Sophie checked the board, took work assigned to her own account, moved a task into progress, and followed the instruction attached to that task.

Over time, we permitted a larger but still bounded set of operational actions:

• create tasks for an explicitly defined upcoming Sprint;
• assign those tasks according to the supplied specification;
• process work explicitly assigned to Sophie;
• complete self-contained analysis when the task’s expected output is clear.

The permissions did not become unlimited. Sophie should not decide Sprint priorities, invent missing requirements, assign herself unassigned work, or treat a vague title as approval to fill in business context.

The most important restriction is not technical. It is social: humans remain responsible for deciding whether a task has enough context and whether the resulting work is ready for review or closure.

Part 1 describes this as an approval step in which the agent proposes and a person confirms. The implementation detail is more precise: the approval gate is behavioural, not a universal Hermes dialog. Part six documents the actual mechanism, where ambiguous work receives analysis and waits, while some explicitly self-contained tasks may be completed automatically.

What we abandoned

We did not abandon an entire automation during this period. The integration remained useful enough to continue.

What we did abandon was a risky assumption: that the agent could infer a Jira parent relationship or business context from a label alone. The parent/Epic correction and the Moth Ball example both led to the same conclusion. When the underlying meaning is ambiguous, the correct automation behaviour is to ask, wait, or hand off, not to guess more confidently.

Cost: what we know and what we do not

For DeepSeek, we loaded US$20 into the API balance. That balance lasted nearly two months, including usage that happened before the Jira integration began.

This is useful directional evidence, but it is not a clean monthly Jira-automation cost. We did not separately meter model usage by Jira task, and we did not record human maintenance time. We would rather state that limitation plainly than turn a partial balance history into a misleading ROI calculation.

Our verdict and the smallest useful version

We would do the integration again.

From our perspective, the integration itself was straightforward. The difficult parts were not the agent framework or Jira API mechanics once the correct token was in place. The real work was defining the operating boundary: what the agent may execute, what it must escalate, and what a sufficiently specified task looks like.

If we rebuilt the smallest useful version, we would keep:

• an agent-specific Jira account;
• a scheduled check for tasks explicitly assigned to that agent;
• one explicit task format for creating and assigning Sprint work;
• a read-back check after any create, update, or transition;
• a human-review handoff for context-dependent work.

We would not begin with broad board access, autonomous prioritization, or an expectation that the agent can recover missing project context from short titles.

What we still have not solved

Two open problems remain.

First, the Review step covers agent work but not the wider question it raises. The column expresses the right distinction between work completed by an agent and work accepted by a team, and it does that job for Sophie. What we have not settled is what a good review actually consists of, or how much of it can eventually be automated without losing the point of having a person there at all.

Second, we need better task-context discipline. The Moth Ball failure was not fixed by changing models. It was caused by a ticket that lacked the information required to produce the right answer. We need task descriptions to carry the relevant context, links, audience, and definition of success, or to explicitly require a clarification before work begins.

That is the real conclusion after eight weekly Sprints. The agent added value when the work was explicit and the authority boundary was clear. It did not eliminate the need for good tickets, human judgment, or review. It made those requirements more visible.

More in this series: Part one, onboarding the agent · Part two, its first task in Jira · Part three, assigning work to humans · Part four, supporting a Sprint Retrospective · Part five, a structured Jira plan · Part six, the Jira operating loop