Subscription Design of a Platform Deploying Itself
Consistency Issues and Solutions Encountered in Vizend Kollex Subscription's Distributed Deployment
1. Background and Problem Definition
Vizend Showcase is a platform that provisions Kollex (a deployment unit bundling multiple microservices) on Kubernetes clusters at the Pavilion (tenant) level using a GitOps approach. Kollex consists of Episodes, which are the screen units exposed to users, and Drama (gate, metro, porto, showcase, etc.), which are the platform backend services supporting that Episode. When a user subscribes to Kollex, the Showcase publishes the subscription fact as a SubscriptionLifecycleEvent to Metro (the platform service managing authorization and subscription status), and simultaneously writes Kubernetes manifests to the GitOps repository and syncs with ArgoCD. In other words, the orchestrator of the deployment is not an external service but Showcase itself.
This pipeline is elegantly organized as a one-way asynchronous flow as long as the subscription target is an external workload. The issue arises from the fact that the Showcase service, which constitutes the Vizend platform, is packaged as a single Kollex. In other words, Showcase exposes itself as a subscribable deployment target, making the entity handling the subscription the same process that is replaced due to the subscription. This article addresses the consistency issues arising in this self-referential deployment structure and the process of resolving them through state machines and event partitioning.
2. Builtin Kollex and Self-Referential Deployment
2-1. Definition of Builtin Kollex
The Kollex entity distinguishes whether it is a platform composition bundle through the isBuiltin property. Kollex with builtin=true refers to the set of services constituting the Vizend platform itself. When a new Pavilion is created, Showcase automatically initiates this Builtin Kollex subscription during the bootstrap process, resulting in the deployment of Showcase and its dependent infrastructure to the corresponding Pavilion cluster.
The essential constraints distinguishing general Kollex subscriptions from Builtin Kollex subscriptions can be summarized as follows.
The process that must determine the completion of the deployment is the same as the process being replaced during the deployment.
While the existing Showcase process is terminated as a new version of Showcase is rolled out, the entity able to confirm the completion of the deployment is the newly instantiated process post-rollout, which does not share any in-memory state with the previous process. Therefore, the lifecycle of a Builtin subscription must have a state transition path that is separate from the general subscription, which assumes the continuous execution of a single process.
2-2. SubscribeLifecycleState Transition Path
The two subscription types share the same SubscribeLifecycleState enum but follow different paths. While a general subscription ends in a single transaction flow, the Builtin subscription is divided into before and after the process restart point.
// 일반 Kollex 구독
Preparing → SyncingMetro → Completed | Failed
// Builtin Kollex 구독 (Self-Deploy)
Preparing
→ SyncingMetro // 1차 구독 이벤트 발행 (Metro)
→ PreparingSelfDeploy // self-deploy 이벤트 발행 직전 단계
→ DeployingSelf // 자기 배포 진행 — 프로세스 종료 임박
──────── 프로세스 재기동 경계 ────────
→ SelfDeployVerified // 신규 프로세스의 이미지 버전 검증 통과
→ FinalizingMetro // 최종 배포 이벤트 발행 (Drama 포함)
→ Completed
// 실패 전이
DeployingSelf → SelfDeployFailed // 이미지 버전 불일치
FinalizingMetro → FinalizeFailed // 이벤트 발행 실패Even if the process terminates in DeployingSelf state, the SubscribeLifecycle record is persisted in the database. The restarted process takes this record as the single source of truth and continues the interrupted lifecycle. Keeping the state in a persistent layer, rather than in-memory, is the key design that withstands process replacement.
2-3. Two-Step Partitioning of Deployment Events
The built-in subscription divides the SubscriptionLifecycleEvent into two stages. This division is based on the distribution dependency order between Episode and Drama.
Showcase is both an Episode exposed to the user and a showcase Drama that refers to itself, which is a backend layer distributed along with gate·metro·porto. By subscribing to Builtin Kollex, this showcase Drama is included in the subscription target, resulting in the Showcase replacing itself with a new version. If a completion signal for an old version process is published while the showcase Drama has not yet converged into a new version, a false positive occurs where the subscription is marked as complete before the replacement has finished.
To prevent this, the primary subscription event published to Metro includes only the Episode and excludes the showcase Drama. The replacement of the showcase Drama is delegated within Showcase as a separate self-deploy event. Only after the new process has properly started and passed image version validation will the final event, including the Drama, be published to Metro. The payload of this final event is serialized in JSON format and stored in the lifecycle record for restoration after a restart.
// 1차 구독 이벤트 → Metro 발행. self-deploy 대상(showcase Drama)은 제외
eventProxy.produceEvent(subscriptionLifecycleEvent); // Episode 포함, Drama 비움
// self-deploy 트리거 — Showcase 내부 이벤트. showcase Drama만 포함
SubscriptionLifecycleEvent selfDeployEvent = event.toBuilder()
.episodes(List.of())
.dramas(galleryDramas)
.build();
applicationEventPublisher.publishEvent(selfDeployEvent);
// 최종 이벤트(Drama 포함)는 재기동 후 Metro 발행을 위해 직렬화 보관
lifecycle.setFinalEventPayloadJson(finalEvent.toJson());▲ Reason for persisting the final event payload — A restarted process cannot inherit the in-memory context of the previous process.
3. Verification of self-deploy after a restart
3-1. Validation at the startup moment — ApplicationReadyEvent
When the startup of the new process is complete, BuiltinSubscribeLifecycleCompletionTask receives the ApplicationReadyEvent and performs validation. It queries the lifecycle that was suspended in the DeployingSelf state, compares the expected image version recorded in the lifecycle (expectedImageVersion) with the observed image version of the currently running process (observedImageVersion). If both values match, it transitions to SelfDeployVerified, and if they do not match, it records as SelfDeployFailed.
@EventListener(ApplicationReadyEvent.class)
@Transactional
public void verifySelfDeployOnStartup() {
if (!enabled) return;
String observed = resolveObservedImageVersion();
List<SubscribeLifecycle> deploying =
subscribeLifecycleLogic.findByStateIn(List.of(DeployingSelf));
for (SubscribeLifecycle lifecycle : deploying) {
if (Objects.equals(lifecycle.getExpectedImageVersion(), observed)) {
lifecycle.moveTo(SelfDeployVerified);
} else {
lifecycle.recordFailure(SelfDeployFailed, "image version mismatch");
}
subscribeLifecycleLogic.modifySubscribeLifecycle(lifecycle);
}
}Through version match verification, it ensures whether 'the image intended by the subscription actually started'. It can identify failures in case the rollout is delayed and the old version process is temporarily alive or if another version has started due to a rollback.
3-2. Observed version interpretation and environment independence
The observed image version is injected through different paths depending on the execution environment. In a Kubernetes environment, the image tag is passed as an environment variable, which does not exist in the local development environment. resolveObservedImageVersion() sequentially explores candidate properties defined by priority, and if all are absent, it falls back to the Implementation-Version of the packaged JAR manifest.
private String resolveObservedImageVersion() {
List<String> candidates = List.of(
"vizend.gallery.self-deploy.observed-image-version",
"VIZEND_GALLERY_IMAGE_TAG",
"GALLERY_IMAGE_TAG",
"IMAGE_TAG",
"BUILD_VERSION"
);
for (String name : candidates) {
String value = environment.getProperty(name);
if (StringUtils.hasText(value)) return value.trim();
}
Package pkg = getClass().getPackage(); // 로컬 폴백
return pkg != null ? pkg.getImplementationVersion() : null;
}Thanks to this priority chain, the validation logic does not branch based on the execution environment. The same code operates as a Docker image tag in the CI/CD pipeline and as a build artifact version locally, with environmental branching not seeping into the code.
3-3. Final event publishing and idempotency
The lifecycle in the SelfDeployVerified state polls by a fixed cycle scheduler to publish the final event. The reason for separating validation and publishing is that a brief grace period is needed until the message broker connection and transaction manager are fully stabilized, even if bean initialization is complete at the ApplicationReadyEvent moment. Instead of synchronous publishing, polling-based asynchronous publishing is adopted to avoid temporary instability immediately after startup.
@Scheduled(fixedDelayString =
"${vizend.gallery.builtin-subscribe.lifecycle.finalize-interval-ms:10000}")
@Transactional
public void publishFinalMetroEvents() {
List<SubscribeLifecycle> verified =
subscribeLifecycleLogic.findByStateIn(List.of(SelfDeployVerified));
for (SubscribeLifecycle lifecycle : verified) {
if (lifecycle.isFinalEventPublished()) continue; // 멱등 가드
lifecycle.moveTo(FinalizingMetro);
SubscriptionLifecycleEvent event =
SubscriptionLifecycleEvent.fromJson(lifecycle.getFinalEventPayloadJson());
eventProxy.produceEvent(event);
lifecycle.markFinalEventPublished(LocalDateTime.now());
}
}The isFinalEventPublished() guard ensures idempotency. Under any circumstances, such as periodic re-execution of the scheduler, failure to update the state after publishing, or concurrent instance contention, the final event of the same lifecycle will not be published multiple times. In an environment where at-least-once delivery to the event consumer side (Metro) is assumed, suppressing duplication by the publisher simplifies the integrity of the entire pipeline.
4. External Drama — Avoiding conflicts with existing infrastructure.
The Pavilion cluster may already have common infrastructure in operation. If the Drama of Kollex being subscribed overlaps with this existing infrastructure, reinstallation poses a risk of disrupting the workloads in operation. Such externally managed Drama is classified as External Drama. Users specify certain Drama as External at the time of subscription, and that identifier is passed as SubscribeCommand.externalTargetIds.
The key design decision here is the separation of "exclude from deployment" and "exclude from events." External Drama should be excluded from the GitOps manifest generation targets, but should not be excluded from the subscription event payload. Metro requires role mapping for Drama — service name, port, role identifier — for authorization and service discovery, and this metadata must be transmitted regardless of whether the manifest is created or not.
// External Drama: 이벤트에는 포함, 매니페스트 생성만 skip 지시
List<SubscriptionTargetSpec> dramaTargets = dramas.stream()
.map(drama -> {
boolean external = command.getExternalTargetIds().contains(drama.getId());
return SubscriptionTargetSpec.of(drama).withExternal(external);
})
.toList();
// Showcase 배포 단계는 isExternal 플래그로 매니페스트 생성 여부를 분기The initial implementation excluded External Drama from the event list in bulk, resulting in Metro being unable to interpret the endpoints associated with that Drama, causing communication between services to fail. This is a case where the decision of "not deploying" was erroneously propagated as "not existing," leaving a lesson that the deployment policy and topology information should be expressed separately in different channels.
5. GID — Multi-tenant Identifier System
In Vizend, multiple Pavilions independently subscribe to the same Kollex. Each Pavilion has a separate cluster, and the Episodes and Dramas that make up the Kollex are registered with separate local identifiers for each Pavilion. In this structure, it is not possible to globally distinguish resources of different tenants with just local identifiers, leading to identifier conflicts.
GID (Global ID) secures global uniqueness by combining the cluster context identifier (CID) and the Pavilion local identifier.
// 로컬 식별자 — Pavilion 내에서만 유일 (NT:1 = Pavilion ID)
KOLLEX_ID = "NT:1-K0001"
EPISODE_ID = "NT:1-E0002"
DRAMA_ID = "NT:1-D000b"
// GID — 플랫폼 전역에서 유일 (X8FD = CID, 클러스터 컨텍스트)
KOLLEX_GID = "X8FD-NT:1-K0001"
EPISODE_GID = "X8FD-NT:1-E0002"
DRAMA_GID = "X8FD-NT:1-D000b"Since GID is also passed with the SubscriptionLifecycleEvent, the deployment stages of Metro and Showcase can clearly determine which resources belong to which cluster and tenant without ambiguity. The design guide also trusts the full GID of EpisodeKey, DramaKey, and KollexKey and stipulates that if GID is empty, it should fail without fallback. The E2E testing of the subscription logic explicitly verifies the configuration of GID in the event payload, as this identification system is a prerequisite for multi-tenant consistency.
// E2E 검증 — GID가 CID + 로컬 식별자 조합으로 구성되는지 확인
assertThat(event.getEpisodes().get(0)
.getEpisodeInfo().getEpisodeKey().getGid())
.isEqualTo(EPISODE_GID); // "X8FD-NT:1-E0002"
assertThat(event.getEpisodes().get(0)
.getEpisodeInfo().getEpisodeKey().getId())
.isEqualTo(EPISODE_ID); // "NT:1-E0002"6. Conclusion
The implementation difficulty of the Kollex subscription function was inversely proportional to the simplicity of the domain term "subscription." A single constraint of self-referential deployment led to a cascade of consistency issues within the distributed system, such as:
-
State persistence — maintaining the progress state of processes being replaced by deployment in the database to withstand restarts
-
Two-step event splitting — preventing false completions before infrastructure dependencies converge
-
Version validation — ensuring that the intended image is indeed running in an environment-independent manner
-
Idempotent publishing — suppressing duplicates on the publisher side in an at-least-once delivery environment
-
Global identifier — resolving multi-tenant local identifier conflicts with GID
If even one of these is missing, the subscription transaction will conclude successfully, but the actual service ends up with a partial failure where it does not operate. The key design experience gained from this functionality is that the distributed consistency issues underlying the simplicity on the surface of the domain could be resolved with structured patterns of persistent state machines and event splitting.
James