Designing and Managing Integration Test Scenarios

1. Introduction

The survey management service allows users to respond to surveys online, and staff and administrators to check and manage the results. While it seems simple on the surface, the permissions and screen flows differ by roles such as users, staff, and administrators, and the authentication methods vary depending on membership status. Additionally, it is entangled with data synchronization with external integration systems, making it difficult to guarantee the overall quality of the service with only the normal operation of individual functions.

In this project, I served in the QA role, where I wrote integration test scenarios based on the requirements definition document and executed and managed them through three stages: development (dev), staging (stg), and production (prod). In this process, I ultimately created and operated an integration test document consisting of 71 test scenarios (TS) and 376 detailed test cases (TC).

This article does not simply report that 'tests were conducted' but organizes my experiences centered on why I designed the scenarios in this structure, the criteria I used to derive exception cases, and how to manage the documents to ensure they remain a reliable asset over time.

2. Background for the Need for Integration Test Scenario Design

At the beginning of the project, we were validating the normal operation of each API and screen through unit tests at the functional level. However, in flow functionalities like the sequence from membership registration to identity verification, agreement to terms, and hospital information integration, problems existed where the stage could be individually normal, yet status values could be incorrectly passed in the overall flow or exception handling from previous stages affected the next stage.

Furthermore, even for the same screen, the information displayed and the possible actions varied according to roles such as users, staff, medical personnel, and administrators, making it impossible to replace verifications like 'Is this role restricted as intended when accessing this screen?' with functional unit tests. Therefore, it was determined that an integration test scenario restructured by user flow based on the individual functions in the requirements definition document was necessary.

There were three practical purposes for designing scenarios separately. To clearly define the regression test scope before deployment, to allow new personnel to understand the entire service flow through documentation alone, and to provide a reference point for tracking in which scenarios and cases bugs occurred when they arose.

3. How to Structure the Requirements Definition Document into Test Scenarios

The requirements definition document is usually organized by functional units. If this is directly transferred to test items, one case may become excessively vast, or conversely, it may be excessively divided, obscuring the overall flow. To solve this, I designed the scenario document in a two-tier structure.

The first tier is the test scenario (TS). TS encompasses the entire workflow that a user goes through to achieve a single purpose as a single unit. For example, a single scenario called 'Agreement to Terms and Identity Verification' is defined as one goal-oriented flow that encompasses various steps such as entering a name, phone number verification, and checking the verification number.

The second tier is the test case (TC). TC refers to individual conditions that need to be verified within a scenario. In the case of the aforementioned scenario of 'Agreement to Terms and Identity Verification,' there would be multiple cases underneath, such as handling when the name is not entered, handling when the phone number format is incorrect, and handling when the verification number does not match.

This structure, which separates scenarios and cases, provided practical advantages in the field. By quickly scanning the list of scenarios, one can grasp the overall workflow of the service at a glance, execution can be managed in detail by case unit, and when bugs occur, it’s possible to immediately identify which case of which scenario it relates to, significantly reducing communication costs with the development team.

4. Classifying by Two Axes: Target and Functional Area

As the number of scenarios and cases increases, managing them in a simple list format becomes difficult. To efficiently manage 71 scenarios and 376 cases, two classification axes were assigned to all cases.

The first axis is 'Target'. It has been classified based on the roles of actual users, staff, medical personnel, administrators, kiosks, etc. Even with the same functionality, the screen composition and access rights differ depending on the role, so if this axis is not separated, it is easy to miss issues that occur only in specific roles.

The second axis is 'Classification', which has been classified based on the functional areas of the service such as authentication, formatting, surveys, statistical visualization, my page, communication box, dashboard, menu, management, etc. This axis was designed to allow the development team to perform regression tests only on the cases that belong to the specific area when they modify the code in that area.

Crossing these two axes allows for immediate extraction of only the necessary scope without reviewing the entire document, such as 'revalidating only the authentication-related functions of the administrator screen' or 'comprehensively validating all functions used by the user'. Given the short release cycle, it was difficult to rerun all 376 cases each time, so this classification system practically contributed to reasonably narrowing the scope of regression testing.

5. Deriving exception cases that go beyond the Happy Path

The requirements specification generally outlines the normal flow, that is, the Happy Path. However, a considerable number of issues that arise during actual service operation occur not in the normal flow but in the surrounding exception situations. It is difficult to derive these exception cases thoroughly by simply transferring the requirements literally, so a method of repeatedly applying the following four verification patterns each time a scenario is written was used.

• Mandatory value verification: Attempting to proceed to the next step when a required input item is empty

• Format verification: Entering a value that deviates from the specified format (number of digits, allowed characters, etc.)

• State-based verification: Cases where there is already an existing or duplicate state, or where the values do not match each other

• Policy-based verification: Cases where the flow changes due to policies such as exceeding time limits or attempts limit

The table below summarizes examples of applying these four patterns to authentication-related scenarios.

For example, the requirement specification simply stated 'accept the mobile phone verification code and process the verification' in one line, but applying the above pattern actually derived about five cases such as no input for the verification code, format errors, mismatches, expiration of validity, and exceeding the number of attempts. In this way, the work of proactively deriving cases that are not explicitly stated in the requirement document from a QA perspective was the part that required the most time and attention in the scenario writing process.

6. Testing strategy for dev, stg, and prod environments

The written scenarios were executed in three stages: development (dev), staging (stg), and production (prod). Since the purposes of the three environments were different, even for the same case, the perspective on verification was approached differently depending on the environment.

In the dev environment, the functionality was quickly and repeatedly validated with controlled test data to ensure that it was implemented according to the requirements, that is, the consistency of the logic itself. In the stg environment, data synchronization with external integration systems and revalidation of the authority system were performed under conditions similar to production, and most issues related to the order of processing or response delays from external systems were revealed at this stage.

In the prod environment, the scope of verification was minimized with a smoke test that only checked the core flow and the areas changed in this deployment shortly after the release. This allowed us to efficiently allocate testing resources while reusing the same scenario document by setting different purposes and scopes for each environment.

7. Principles for recording test results

During testing, it is common for not all cases to be clearly divided into success or failure. There were many situations where verification could only be done if the response from an external system preceded it, or where cases had to be excluded from verification due to policy. If we simply recorded these situations as failures, subsequent viewers of this record might misunderstand the cause of the problem, so we established a principle to classify the results into four categories as follows.

For cases recorded as ABORTED and BLOCKED, we always left a specific reason in the remarks section. For example, it was confirmed that a specific notification feature could not guarantee accurate results if the notification reception and query API call from the external system were processed as a single transaction. Therefore, we decided to separate the logic and revalidate it afterward. By classifying it as BLOCKED instead of simply leaving it as 'failed' and recording the cause and revalidation conditions together, we were able to accurately find and rerun that case when the logic was separated later.

Along with this, there were features where the validation and correction subjects are not within the project itself, similar to areas where the customer’s system screen is embedded in the service. When an error is found in these areas, since we cannot correct it directly, we specify the person in charge of that screen and request corrections along with the reproduction conditions, and after confirming the reflection, we re-execute only that case to update the results. Cases recorded as FAIL were mostly communicated to the responsible developer, and after modification distribution, we repeated the flow of re-validation with the same TC_ID to eliminate them. In this process, we were able to confirm that the ID system and result classification principles organized earlier effectively served as a benchmark for communication between organizations.

8. Principles of Managing Identifiers (IDs) in Scenario Documents

I have experienced through practice that as the number of cases increases, the ID system for identifying each case affects the overall credibility of the document. In the scenarios, TS_ID was assigned, and TC_ID was assigned to the cases in sequence; this ID was not only used within the scenario document but was also repeatedly referenced in various deliverables such as bug reports, regression test result tables, and deployment checklists.

The most important principle I adhered to in this process was that 'once an ID is assigned, its contents should not be overwritten unless the conditions for verification itself change.' If the verification purpose of a specific case completely changes, instead of modifying the contents of the existing ID, we chose to discard the existing ID and issue a new ID, treating it as a separate case. Minor changes such as correcting typos or refining expressions were made while maintaining the same ID, but in cases where the verification conditions or expected results themselves changed, a new ID was issued without exception.

The reason for adhering to these principles is that an ID is not just a simple number, but a reference point that indicates the test history at a specific moment in time. If the content under the same ID is completely changed to a different case, it will lead to confusion as bug reports or execution histories that referenced that ID in the past will actually point to different content. This can be said to follow the same principle as database schema changes or API version management, where a new version is added while leaving the existing one intact. Additionally, we have established a principle that once an ID is discarded, it will not be reused in the future, ensuring the complete preservation of document history in chronological order.

9. Application Results and Reflection

Based on the above principles, we completed an integration test document consisting of 71 scenarios and 376 test cases, and executed them sequentially in the dev, stg, and prod environments. The majority of the cases passed normally, while some were classified as ABORTED or BLOCKED due to external system integration or environmental constraints, and remain under separate management.

The biggest lesson is that in writing integration test scenarios, the task that requires a lot of time is not reading the requirements document, but rather finding the exceptional cases that are not specified in the documentation. I have also confirmed through practice that the habit of mechanically applying four verification patterns and seemingly trivial rules, such as ID management or result classification, supports the credibility of the entire document from the point when cases number in the hundreds.

In the future, if I take on a project of a similar scale, I think it would be good to document the scenario design principles and document management rules established this time in a form that can be shared by the entire team from the beginning of the project, and to focus manual testing resources on areas where human judgment is important, like discovering exceptional cases, by selecting areas for automation during the repeatedly executed regression tests.

eunice