[Devlime Development Log] Introducing Scrum AI
1. Introduction
When developing with Scrum, writing backlog items and user stories takes more time than expected. Even when a meeting concludes with, "Let's add a reservation feature in the next sprint," there is still work left for the developer. The title must be organized, the content written in the As-A / I-Want / So-That format, acceptance criteria added, and story points and value scores entered. Any one of these tasks seems quick enough, but repeating them around twenty times every sprint changes the picture.
So, while developing Devlime's Scrum module, we decided to reduce the effort involved in this process. The initial idea was simple: enter the requirements, generate a UserStory draft, and find similar past work to use as a reference for scoring. We expected it to take about three weeks.
In practice, development turned out differently than expected. Integrating the AI API itself was not difficult; most of the trial and error came from deciding which problems to delegate to AI. In this article, I summarize how we started the feature, where problems arose, and how we changed the architecture.
2. Background for the Technology Choices
2.1 Structured Output
The first feature we built was UserStory draft generation. We sent the requirements entered by the user to Claude and populated the screen with the result received in JSON format. It worked well at first, but not consistently. Sometimes explanatory sentences were added before or after the response, Markdown code blocks appeared around the JSON, or specific fields were missing.
Initially, we handled this by processing the response string directly. We added code piece by piece to remove backticks, find where the JSON began, fill in default values for missing fields, and throw an exception if the process still failed. Eventually, the code for recovering responses became longer than the code for calling AI. We concluded that refining the prompt alone had limitations and that we needed a way for the application to control the model's output format itself.
We therefore applied the structured output provided by the Claude Java SDK. Instead of receiving a string and converting it ourselves, we define the DTO used by the server directly as the response schema.
public class UserStoryDraftAiSchema {
public boolean supported;
public String rejectionReason;
public List<TitleOption> titleOptions;
public List<NarrativeOption> narrativeOptions;
}We were able to remove most of the parsing and correction code we had written previously. More important than reducing the amount of code was being able to manage the data structure used by the server and the output structure required from the model as a single type. This is why we chose structured output for this feature. It was not because the model performed better, but because it reduced the code the application had to process unnecessarily.
2.2 Model Selection
At first, we used claude-opus-5, prioritizing quality. The results themselves were good, but when we used it on the actual screen, response time became a problem. Each request took about 6–7 seconds, which was burdensome considering that the feature would be called multiple times during sprint planning.
So we switched to claude-haiku. We expected the quality to drop significantly, but when we showed the results from both models to the team and compared them, they could not clearly distinguish the difference. This feature required organizing entered requirements into a defined format rather than complex reasoning or creative generation. As a result, we reduced both response time and invocation costs, confirming that choosing a model suited to the feature is better than simply choosing a more capable model.
2.3 Scoring Recommendation Approach
This was the area that changed direction the most. Initially, we planned to ask Claude for story points in the same way we generated drafts. The results looked reasonable, but we could not explain why a task received 3 points. When we submitted the same requirements multiple times, the result might be 3 points one time and 5 points another. Even lowering the temperature to make the results consistent did not provide a basis for the score.
In retrospect, this was an obvious outcome. Claude does not know what work the Devlime team performed over the past few months or how many points the team assigned to that work. Story points are relative values judged against a team's past work, not values with a generally correct answer. A number presented without supporting evidence is not useful as a reference. Moreover, simply because it appears first on the screen, it could become the basis for discussion.
So we redefined the problem. Instead of generating a score, we changed the approach to finding similar past work and displaying the scores assigned to those items.
3. Implementation Process
3.1 Search Instead of Generation
We vectorized and stored the titles of completed past BacklogItems using Voyage AI's embedding model. When new requirements arrive, we embed them in the same way, compare their cosine similarity with the existing data, and use the three most similar items as recommendations.
[저장] 완료된 BacklogItem 제목 → Voyage Embedding → Vector 저장
[검색] 새로운 요구사항 → Voyage Embedding → Cosine Similarity → 유사 항목 Top 3The screen displays not only the recommended scores, but also the titles of the past items that provided the basis and the scores they received at the time.
추천 참고 항목
1. 예약 화면에 사용자 검색 기능 추가 (Story Point 3)
2. 예약 상세 조회 기능 추가 (Story Point 3)
3. 예약 상태 변경 기능 추가 (Story Point 5)In this structure, AI does not present a definitive answer. It only provides information that similar work existed in the past and was evaluated at a certain level; the user makes the final decision. The ability to show supporting evidence was the biggest reason we changed our approach.
3.2 Request Path
We could have called the Anthropic API directly from React. However, doing so would expose the API Key to the client and make it difficult to control the calls.
React → Scrum Backend → AI APIThe frontend passes only the values required for the request.
const draftUserStory = (params: {
rawInput: string;
productId: string;
epicId?: string;
}) => {
const query = { ...params };
return axios.post(url('/draft-user-story/fetch'), query);
};All AI-related policies, such as the prompt, model selection, output schema, and retries, are managed on the server. With this setup, even if we replace the model later, there will be almost no changes to the frontend.
3.3 Environments Without an API Key
Developers who do not use the AI features do not have an Anthropic API Key in their local environments. However, if the Anthropic Client is always created when the server starts, the application itself may fail to start in environments without the Key. We therefore registered the Bean conditionally.
@Bean
@ConditionalOnProperty(name = "ANTHROPIC_API_KEY")
public AnthropicClient anthropicClient() {
return AnthropicOkHttpClient.fromEnv();
}The Client is created only in environments where the API Key exists. In environments without it, only the AI features are disabled, while the rest of the functionality operates normally. We created this boundary so that a single AI feature would not affect the development environment of the entire team.
3.4 Schema Field Order
After switching to structured output, we encountered an unexpected issue: the results varied depending on the order of the schema fields. Initially, we declared that title candidates should be generated first. As a result, even inputs that were difficult to regard as requirements, such as "Choosing a team dinner location for next week," generated titles first. Since supported, which determines whether the input can be handled, was decided later, the system sometimes treated the input as supported to fit the result that had already been generated.
So we moved the fields required for the decision to the front.
public boolean supported;
public String rejectionReason;
public List<TitleOption> titleOptions;
public List<NarrativeOption> narrativeOptions;We changed the order so that the system first determines whether the input is a requirement that can be turned into a UserStory, and generates the remaining content only when the result is true. Through this process, we learned that a schema is not merely a DTO; it is also part of the prompt that controls the model's output process.
4. Problem Solving
Problems continued even after the feature was integrated. The following five issues were all identified through actual use after implementation was complete.
4.1 The Response Was Successful, but the Content Was Empty
There was a problem where narrativeOptions in the UserStory draft was returned as an empty array. The HTTP Status was 200, and no exception occurred on the server, so we initially thought it was a frontend problem. When we inspected the actual response, we found that it was being cut off during generation.
The cause was maxTokens. We had set it to 2048 based on BacklogItem, but UserStory had to generate multiple titles, descriptions for each title, and acceptance criteria, so it required considerably more output. Increasing the value to 4096 resolved the issue, but more importantly, we learned that an HTTP 200 does not necessarily mean that an AI response completed successfully. We now check the response's stop_reason and log the event when the maximum output token limit has been reached.
4.2 The Number of Recommendations Became Zero
After deploying the score recommendation feature, we received feedback that certain projects were not receiving any recommendations. We found that there were zero items in the similarity search target set. Initially, we limited the search to completed items with the same issue type as the currently selected issue. However, the actual data contained few completed items of certain types, so filtering left no data to compare against.
We therefore removed the type filter. We initially thought that referring to the scores of work with different types would be inaccurate, but the scores shown by this feature are references, not definitive values. Even if the types differ, similar titles and work content can still make an item a useful past example. In the end, we chose to ensure that search results existed, even at the cost of slightly reduced accuracy.
4.3 Waiting for the Response
Even after switching to Haiku, it took about 2–4 seconds from the request to the response. At first, we displayed only a Spinner, but in actual use, this time felt longer than expected. So we changed the status message step by step while generation was in progress.
요구사항을 분석하고 있습니다.
UserStory 초안을 작성하고 있습니다.
인수 조건을 정리하고 있습니다.These messages are not connected to the model's actual progress. However, they helped reduce the time users spent looking at nothing but a Spinner. The user experience of an AI feature is not determined solely by the quality of the model's response; deciding what users see while the model is responding is also part of the feature.
5. Final Architecture
The following architecture was ultimately created while building this feature.
사용자 → BFF → Scrum API
├─ Claude : UserStory 초안 생성
└─ Voyage : 과거 유사 항목 검색We did not group the two features together as the same AI problem. We separated the problem of generating sentences from the problem of finding supporting evidence in our data: Claude handles UserStory drafts, while Voyage embeddings handle searches for past work. The user makes the final decision. Once we divided the responsibilities, the architecture actually became simpler than it was initially.
6. Lessons Learned
The biggest lesson from developing this over three weeks was that the difficult part of an AI feature is not the API call itself. Code that calls the Claude API can be created more quickly than expected. The time-consuming work came afterward. We had to decide which tasks to delegate to a generative model and which tasks to find in our own data. We also had to verify that the actual content was valid even when we received a successful HTTP response, and consider how users would perceive the model's response time.
I felt this difference especially strongly with score recommendations. If you ask the model to make a judgment without any supporting evidence, it can produce a plausible answer, but whether that answer aligns with our team’s standards is a separate question. Conversely, if the team has historical data, it may be more appropriate to search that data and show it to the user rather than asking the model to generate an answer. Ultimately, it was important to first determine whether this was truly something AI should generate, rather than focusing only on what to make AI do.
7. Conclusion
At first, I thought it would be a simple feature. I assumed that all we needed to do was enter the requirements, have it create a UserStory, find similar tasks, and display their scores. However, once we actually built it, we changed direction significantly twice. The first change was approaching score recommendations as a generation problem, and the second was restricting the candidates too much in an attempt to improve search accuracy. Neither issue was caused by writing the code incorrectly; both resulted from defining the original problem in the wrong way.
Through this work, I plan to keep checking three things whenever we build AI features in the future. First, choose a model that fits the feature. Second, make sure to control any calls that incur costs on the server. Third, have team members use the feature themselves after it has been built.
I think the last point is especially important. Most of the problems we found this time could not be identified during development. The empty-array issue, the Korean input issue, and the problem of ending up with zero search results all became apparent only after we actually used the feature.
Ultimately, the thing that changed the most during this Scrum AI development was not the code, but the way we looked at the problem. When adding AI features to Devlime in the future, rather than choosing a model first, I plan to begin by defining what the problem is and determining whether generation, search, or rules are the most appropriate approach for it.
Thank you for reading this long post.
jun