Making AI Agent Tasks Observable

1. Introduction: What Commit Logs Can No Longer Explain

Once AI coding tools are introduced to a team, code grows quickly, but the process by which that code was produced becomes even harder to see. In the past, a quick look through the commit log gave you a rough sense of who had been working on what and for how long. Now, even when looking at the same log, it is impossible to tell whether a result came from an hour of human deliberation or an agent producing it in three minutes. The repository is unchanged, but what is happening inside it has become invisible.

We set two goals. First, even if the code was produced by an agent, someone who joins later should be able to receive only the repository, understand it immediately, and take over. Second, regardless of which project the repository belongs to, the moment it is registered for analysis, metrics and trends should be available without any additional configuration. These two goals determined every decision that followed.

2. What Should Be Counted?

2.1 Reporting, Not Detection

At first, we tried to analyze code changes and identify the parts written by AI, but we soon abandoned that approach. A commit contains an author, a timestamp, changes, and a message, but it has no place to record “how this line was produced.” Code drafted by AI and then refined and committed by a human is byte-for-byte identical to code typed by hand. The problem is not low detection accuracy; the data contains no subject that can be detected.

So we gave up on detection and decided to collect reports instead. We count only what a tool records or a person explicitly reports. In return, we firmly named the metric “reported AI,” not “AI usage rate.” This raises the question of what happens when people do not report it, but we decided to treat that not as a defect, but as another metric. The fact that the reporting rate is low is itself a signal that the convention is not being followed.

2.2 Use Sessions, Not Commits, as the Unit

The more important decision was what to regard as a single piece of work. Work with an agent does not end in one step. You give instructions, review the result, ask for changes, and receive it again. The process is left split across three or four commits.

If commits are counted as the unit, all that remains is the fact that “the agent made three commits.” What we actually want to know is “how many attempts did it take to complete this work?” A task completed in one attempt and a task rolled back five times tell entirely different stories even if both consist of three commits. So we decided to include a session identifier in commits and group commits from the same session into a single piece of work.

3. The Approach We Adopted

3.1 Use Commit Messages as the Interface

When you do not understand code written by a human, you can ask them. You cannot do that with code written by an agent. Once a session ends, its context disappears, and even if you invoke the same tool later, it cannot explain why it made that choice last time. Therefore, the more an agent is involved in the code, the more its explanation needs to remain alongside the code itself. That is the only way for someone who joins later to understand the project with nothing but the repository.

We could have accumulated records in a separate repository or pulled them through a tool API, but we chose commit messages. Commits are preserved regardless of which tool is used, so this approach is tool-agnostic; the records move together with the code and survive replication or mirroring; and because the original is preserved as-is, we can reread it later even if the interpretation rules change. For the format, we used Git’s existing trailer convention as-is.

[SPEC-008] feat(member): 세션에서 사용자 신원을 읽도록 변경

X-Agent: dev
X-Agent-Session: 3ae7dbc4
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

There was one trap. Co-authored-by is originally a convention for indicating collaboration between people, so using it as-is would also classify commits created by two people working together as AI commits. This is a common problem when a widely used convention is repurposed, so we distinguished them by recognizing something as an agent only when its email domain belonged to a known tool.

3.2 Put the Convention in the Tool, Not in People

This was the most unexpected part. Commit conventions are not usually followed consistently. Even when they are documented and explained during onboarding, they fade after a few weeks. That is generally what happens with rules that depend on people’s diligence. Agents, however, read the project’s convention file every time and follow it exactly. They do not rely on memory; because they reread it each time they work, it does not fade over time.

## 커밋 메시지 규약

[<백로그>] <type>(<scope>): <제목>

X-Agent: <역할>              # dev · review · test · fix · chore 중 하나
X-Agent-Session: <식별자>    # 세션 내내 같은 값을 쓴다

- 제목 한 줄이면 끝이다. 본문을 쓰지 않는다.
- 한 커밋에 역할 두 개를 담지 않는다.

The concept of a commit convention itself is not new. Everyone knew it was important, but it simply was not followed. What changed was not the convention, but the party enforcing it. That makes where to place the convention file important. If it is kept inside the repository, changing the convention goes through review just like a code change and is preserved in version history; if the code changes while the convention remains unchanged, that discrepancy becomes visible in the same commit. If it had been in a wiki, it could have fallen out of sync with the code without anyone noticing.

There is one rule that must be followed, however: do not add the reporting trailer to commits made directly by humans. If people start adding it for convenience, human work and agent work become mixed together, and at that moment the questions this data was meant to answer become entirely meaningless. When creating a convention, you must decide not only what to record, but also what not to record.

Limiting the roles to five and allowing only one role per commit follows the same principle. If you find yourself wanting to record two, that is a signal to split the commit. Values not included in the list are not discarded; they are counted as-is. The fact that the convention is not being followed is itself a signal worth observing, and its proportion becomes the metric for convention compliance.

3.3 Derive Classifications Instead of Storing Them

After finishing the implementation, we opened the dashboard and found that the numbers were much smaller than expected. The cause was not the data but the design. Commit entities were designed to be immutable: once loaded, their values could not be changed. Yet we were storing the classification result in a field of that entity. Commits ingested before the parser was added therefore had that field permanently empty.

We chose the third option. The classification rules will continue to change, and a structure that requires migrating the past every time the rules change will not hold up over the long term. If the result is derived, a single re-aggregation applies the new rules to the past as well. When we rechecked the parsing cost we had been concerned about, it did not make a meaningful difference to aggregation time. The cost we had hoped to save by storing the result was not worth saving in the first place.

4. Results

4.1 It Keeps Working As-Is When a New Repository Is Added

The choices above may look like separate decisions, but they come together around one question: what more needs to be done when a repository is newly registered as an analysis target?

As a result, there is nothing special to prepare when registering a new repository. As long as the team has been following the commit conventions, metrics appear from the moment it is registered, and past commits are calculated as well, so you can see the trend immediately on the first screen. Conversely, if the records had been accumulated in a separate repository, each repository would have required integration work, and if the determinations had been stored, the history of a newly connected repository would have remained empty forever.

4.2 Make the screen distinguish why a 0 was produced

We also had to decide whether to hide members with zero reported AI usage from the list or show them as they are. Hiding them makes the screen look cleaner. However, once they are hidden, “does not use it” and “uses it but does not report it” look identical on the screen. We had decided earlier to use the reporting rate itself as a metric, but if the screen conflates those two cases, that decision becomes meaningless. So instead of hiding zero, we made the screen state directly why it is zero.

No AI use was reported in this period.
Commits alone cannot tell not-used from not-declared.

We applied the same principle when the metric could not be loaded. If a query fails, we do not display 0; we state that the value could not be loaded. A thermometer showing 36.5 and one whose scale has stopped because it is broken represent completely different states, but they cannot be distinguished by the number alone. A failed query and a value of 0 are the same way.

5. Conclusion

Bringing an agent into a team is different from simply using one more tool. It means adding a new participant to the development process, which requires creating a new place to record what that participant did. Most of this work went into deciding what to record and what not to record, rather than into parsing or aggregation.

The most practical discovery was that conventions can be embedded in tools rather than left to people. People have said for a long time that conventions are important, but that idea has always relied on individual diligence, and as a result the conventions were usually not followed. In effect, the focus shifted from the conventions themselves to where those conventions are enforced. The one rule that people must still follow, however, is not to mix human work with agent work.

Ultimately, what we wanted was simple: even when the code was created by an agent, someone who arrives later should be able to take over immediately after receiving only the repository, and the moment that repository is connected to the analysis, the results and trends should be visible without any preparation. To make that possible, explanations must be located alongside the code, conventions must live inside the repository, and interpretation must be reproducible at any time outside the repository.

References

Git – git-interpret-trailers — https://git-scm.com/docs/git-interpret-trailers

GitHub Docs – Creating a commit with multiple authors — https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

Conventional Commits 1.0.0 — https://www.conventionalcommits.org/ko/v1.0.0/

toffeeman