Solving MSA Quality Control with CI Gates

1. The reason for placing the quality management system in CI

I am currently participating in an MSA consulting role as a DevOps engineer. When it comes to consulting, you usually think of design discussions such as service boundaries or domain separation, but the area I was in charge of was the quality management system. This article is a record of that work.

When thinking about quality management, items like code style, static analysis, test coverage, and image vulnerability scanning come to mind. The items themselves are not new. The real challenge was 'where and with what level of enforcement to run this'. In MSA, this question is much heavier than expected.

When services are broken down into smaller pieces, the number of deployment units increases accordingly. A deployment that was one until yesterday is now divided into ten or twenty parts, and each service operates on different cycles. In such a situation, if the quality items are merely set as recommendations of 'let's try to follow them,' the pace of service growth is overtaken by gaps. This was exactly the situation when I first joined.

So I started from the point of quality management. The conclusion was CI.

CI is a path that every change must pass through at least once. If quality standards are set as a gate in this path, whoever is working and whatever service they are working on will go through the same criteria. It becomes the pipeline that holds the standard, rather than human diligence. This was the most natural position to turn quality from 'it's good to adhere to' into 'you can't move on to the next unless you pass'.

Below is a story about how I incorporated quality items into CI from that perspective.

2. Agent coding and quality

Before diving into gate design, there is one premise to lay down. These days, the way we write code itself is changing rapidly. A significant part of development is done in collaboration with AI agents. I see this change as an opportunity rather than a threat from a quality perspective.

Test code is a prime example. In the past, creating sufficient tests was a cost in itself. To raise coverage to 80%, humans had to manually create boundary values and exception cases, and since that time was precious, they often just oversaw the core paths. Testing was the most postponed task.

Once agents are involved, this changes. It is now possible to quickly generate tests that include boundary conditions and exception flows, and the more repetitive and tedious tasks tend to be, the more effective the results are. This means we can achieve higher and denser coverage in the same amount of time. As the tools have improved, the quality benchmarks can be raised correspondingly.

However, there is one caveat. Whether the code or tests created by the agent are genuinely meaningful validations needs to be consistently judged by someone. If this judgement is left solely to human eyes, while production speed increases, the speed of validation cannot keep up. The volume produced increases, but the filtering remains unchanged.

Thus, 'making things faster and in greater quantity with agents' and 'consistently enforcing standards with CI gates' need to go hand in hand. As the speed of production increases, the value of the gate that makes mechanical judgments on whether to upload the results increases correspondingly. The rest of this article will detail how that gate was constructed.

3. CI Quality Gates

The design principle itself is simple. The earlier a defect is caught, the cheaper it is, and the later it is caught, the more expensive it becomes. That is all there is to embedding this fact into the pipeline structure.

So, instead of making the quality gate a single large inspection, we divided it into four stages with different responsibilities.

Early blockage → Baseline confirmation → Operational candidate validation → Promotion determination

The meaning of passing varies at each stage. If you pass early blockage, merging opens up; if you pass baseline confirmation, you are registered as the official quality baseline; and only by passing the operational candidate validation do you finally become eligible for promotion review.

Here, one thing has been made clear. Regardless of the branch strategy, the standards for quality control remain the same. Strategies can change depending on stages and situations, but if the quality level required by the gate fluctuates based on the strategy, then it can't be called a standard.

In this project, during the implementation (SI) phase, release control is important, so we based it on GitFlow, and in the operational (SM) phase, we outlined a path to switch to trunk-based once automation becomes sufficiently mature. Thus, the gate is defined as a common model, and we only mapped the execution points to fit the branch strategy. Even if the strategy changes, the inspection content will follow along.

Personally, I think that in an environment operating MSA as a monorepo, trunk-based ultimately fits better. In a monorepo, multiple services share a single repository, and if release, develop, and feature branches split off like in GitFlow, the number of integration points increases by the number of services. The costs of conflicts and merges also increase. Trunk-based allows for short tasks and frequent merges to the trunk, thereby reducing the situations where changes collide all at once from being far apart.

When selective CI, which will be discussed later, is added to this, it creates a pretty good synergy. Even with many services in the repository, we only validate the scope of changes, allowing us to integrate frequently while keeping CI costs manageable. The risks arising from short integration cycles are supported by the quality gate. Therefore, we also organized a recommendation that this project start with GitFlow in SI but move to trunk-based as operations stabilize.

[ Figure 1. Quality gate stages functioning independently of branch strategy ]

Once we separated stages and responsibilities like this, it became clear what 'this change needs to pass what at which stage' means. The structure switched from having one reviewer bear everything in their head to a structure where the pipeline serves as the first line of defense.

There was another practical issue we encountered while setting up the gates. In monorepos with many services, validating the entire system for each change quickly makes CI unmanageable. If fixing a single line of code means you have to wait 20 minutes, no one will like the gates.

So, we applied selective CI. We built only the changed modules and included only the services affected according to the dependency graph as test targets. However, we didn't just blindly narrow it down. When common libraries, event contracts, infrastructure change, or when creating a release candidate, we specified exceptions to run the whole without narrowing it down. The scariest thing is to lose sight of regression in the pursuit of speed, so we clearly defined the line as 'narrow down but broaden for risky changes'.

However, there are moments when even reducing time with selective CI is insufficient. This can happen when there is no room to go through all validations and we need to go out immediately. This is true for emergency hotfixes and also for days when we are pressed by demo or presentation schedules. This is not a matter of will but a commonplace scenario.

Therefore, we made it possible to enable and disable validation items within the quality gate.

Heavy validations like static analysis, integration testing, coverage checking, and vulnerability scanning are exposed as pipeline parameters so that some can be skipped depending on the situation. When the gate entirely hampers progress in an urgent situation, people ultimately seek more dangerous paths outside of CI. It seemed better to openly and evenly decide 'what to leave out this time' within the pipeline rather than that.

Of course, there was a cost to turning things off. The builds record which validations were skipped, and key blocks such as build failures or unit test failures were initially excluded from the option list so they couldn't be turned off in any situation. The purpose of making it optional is not to loosen control, but to incorporate 'even in situations where you need to go fast' into the gate design and make it visible.

4. Block Strength (Hard Block / Soft Block / Warning)

What I pondered the longest while creating the gate was surprisingly not 'what to block' but 'what not to block'.

If everything is strictly blocked, the pipeline quickly earns dislike. If deployment is halted over a formatting error, everyone starts devising ways to bypass the gate. Therefore, I divided the strength of blocking into three levels.

Hierarchy	Meaning	Representative Item
Hard Block	No merging or promotion without exception approval	Build failures, unit test failures, Critical/High vulnerabilities, unmet mandatory approvals
Soft Block	Conditional approval if a remediation plan is documented	Coverage shortfall, performance regression, Medium vulnerabilities, unstable integration tests
Warning	Not blocking but only managing trends	Formatting, low severity, missing documentation, style issues

By dividing this way, the gate became a system that reacts according to the magnitude of the risk rather than a "wall that indiscriminately blocks." It lets minor issues pass while definitely catching dangerous ones.

As the intensity is lowered, there are also opportunities to escape. Therefore, we put in complementary rules. If the same item fails consecutively more than twice, we consider raising it from Soft to Hard, and failures related to core business paths like payments or settlements are handled as Hard, regardless of the test layer. The weight of the same failure varies depending on where it occurs.

Ultimately, determining what "will not be blocked" and deciding on a "path that will never yield" are part of the same bundle. Dividing the strength into three was also a measure to balance these two.

5. Static Analysis and Testing

After establishing the framework of the gate, we had to fill in what each gate would actually inspect.

Since the codebase is in Kotlin, we deliberately split the static analysis tools. detekt handles Kotlin language-specific analysis, while ktlint manages formatting and linting, and coverage measurement is separately done by Kover.

The goal was not to increase the number of tools. detekt effectively captures Kotlin's unique idioms and meaning-based defects, while ktlint mechanically enforces style consistency. They excel in different areas. Keeping them separate clarifies which tool flagged the violation, making the criteria for the gate much cleaner.

Tests have different points in time and responsibilities at each layer. Running all tests at every stage would be cost-prohibitive. The basic framework is to place fast and cheap checks at the front, with slow and expensive checks at the back.

Testing Layers	Execution Point	Purpose
Unit Test	PR / Baseline Confirmation	Accuracy of Module Internal Logic
Integration (Testcontainers)	PR center	Early detection of service boundary and integration regression
E2E smoke	Baseline confirmation	Shallow regression verification before deployment
E2E expansion	after stage	Verification of essential flow based on release version

Coverage was not treated as an absolute score. It was seen more as an indicator reflecting risk areas not covered by tests. The baseline was set at 70% overall, with core domains and new code at 80%, but we treated the state of unverified core paths as a greater risk than the numbers themselves. If we focus only on the numbers, there will inevitably be empty tests just to pass.

So we established one operational rule. The coverage numbers are not to be treated as absolute cut-offs for gates. New code coverage is seen as a leading indicator that preemptively prevents regressions and is thus viewed relatively strictly, while the low numbers for existing code were raised gradually, starting from the core paths, rather than blocking them all at once. As mentioned earlier, we believe that by utilizing agents, we can realistically set this target line higher.

6. immutable artifact

No matter how well quality is inspected, if something different from what was inspected goes live, the validation becomes meaningless. Surprisingly, this is where leaks often occur.

Therefore, we have established the principle that operational promotion will not involve rebuilding. We take the exact output that passed verification in the stage and retag it with the same digest for operation. If the SHA of the verified commit is different from the SHA of the promotion target, we block the promotion itself. We have completely removed processes like "rebuild for production" from our path.

Even if it seems trivial, this principle holds significant weight in MSA. The more frequent the independent deployment of services, the more critical it is that "the byte that was verified goes live as is," as this guarantees the reliability of failure analysis and rollback. Without this guarantee, one has to question from the outset whether "the verified version and the live version are indeed the same" when a failure occurs.

[ Figure 2. Flow of promotion with the same digest without rebuilding ]

Vulnerability management has also been integrated into the same space. Trivy is run within the CI runner for image scanning, and Critical and High vulnerabilities have been excluded from the promotion targets even if the image is already in the registry. The results were collected in SARIF format to observe trends centrally. Security was also placed as part of the pass criteria rather than as a 'recommendation'.

7. AI Code Review

Around the time I was organizing the quality gates, I found out that AI could assist in code reviews at the CI stage, so I decided to implement it myself.

When a PR is raised, the AI scans the changes to highlight potential defects, regression risks, tests that may be missing, and possible rule violations in a preliminary manner. I integrated it into the pipeline, and honestly, its performance exceeded my expectations.

What was most impressive was that it did not stop at one or two perspectives. It followed almost exactly the order in which a human reviews and highlighted issues from multiple angles. When a PR is submitted, it organizes feedback in roughly this manner.

• Change Analysis— Summarize in one line what files were changed and why

• Potential Bugs/Improvements— Identify areas where documentation is vague, and configuration dependencies that might result in errors during deployment if left unaddressed

• Security Issues— Check for any chance that sensitive information like API keys or credentials may become mixed in

• Structure/Coupling— Suggest verification of changes in inheritance relationships or increased coupling with other modules, along with the corresponding impact range

• Duplication/Optimization— Mark similar logic repetition or unused code as candidates for refactoring

• Test proposal— Questioning whether the coverage is sufficient compared to the impact of changes, and recommending additional necessary cases

Even when I submitted a small PR that added a single line of a marker to the document, I followed through on "how does this change affect other code?" From the reviewer's perspective, it felt like the starting line for review was moved one step forward. By filtering out side issues first, the person was able to focus more on the key judgment.

However, this feature ultimately was not implemented in actual operation.

The reason was security. In an environment where the sensitivity of the data handled is high, there was inevitably a sensitivity to the paths through which code and the information involved could leak out, and the method of sending changes to external AI services was difficult to adopt as is. Whether a feature works well and whether it can actually be brought into this environment are separate issues.

So, I stopped at the point of saying, "Validation has been done and the effectiveness has been confirmed, but operation will proceed only after meeting security standards." If it were to be implemented, it would need to be based on a private LLM that does not export data, and in that case, the principle remains that AI is merely supportive and the final approval comes from human reviewers.

It was a way of measuring the distance between what is technically possible and what can be brought into this field, and I think leaving behind a standard to fill that distance is a result in itself.

8. Conclusion

In retrospect, this work did not involve grand tools. The key was ensuring that quality items went through the same standards in CI.

If I summarize what I particularly felt during the work, it is as follows.

• Most quality items are most naturally and consistently handled in CI, the gateway where code enters.

• The flow of coding as an agent is not a threat to quality but an opportunity. If used well, it can actually enhance coverage and validation density.

• Gates must be designed not only to block what should be stopped but also to determine what should not be stopped in order to survive without being disliked.

• If the immutable principle of uploading validated outputs as is is omitted, all the validations in front become powerless in an instant.

• What is technically possible and what can be brought into this environment are different. Cases where effectiveness was seen but held back for security reasons, like AI reviews, serve as examples.

In MSA consulting, noticeable deliverables like service design or deployment systems are first to attract attention. However, what supports these deliverables to maintain a consistent quality over time is what I believe to be the quietly running CI quality gate every day.

Tim