Introduction

When you integrate AI features into a service, you quickly encounter situations like this. User modification. Yes, I have modified it.
Check the admin text for admin modification

I want to use different ChatModels depending on the functionality, how should I manage it?

I encountered this problem while developing an AI service for recommending tourist attractions.

At first, I thought that a text-based conversation model would be sufficient. However, when a tourist image is input, it is necessary to analyze the image and extract its features and descriptions. In this case A text-only model cannot understand images.

Ultimately, two types of models became necessary as follows.

• Text Model: A conversation-specialized model suitable for fast responses and long context processing

• Multimodal Model: A vision-supported model capable of understanding content by receiving images as input

This is where the 고민 began.

How to manage multiple ChatModels in a Spring Boot + LangChain4J environment?

Limits of LangChain4J Automatic Configuration

LangChain4J supports Spring Boot automatic configuration. For example, if you set it in application.yml as follows, the ChatModel bean will be automatically registered.

langchain4j:
  ollama:
    chat-model:
      base-url: https://ollama.example.com
      model-name: model-name

However, this method has its limitations.

Automatic settings are basically A method to register a single ChatModel instanceIt is closer. Therefore, if different models are to be used for different purposes, it is difficult to rely solely on automatic configuration.

Ultimately, to explicitly manage multiple models, it is necessary to define @Bean directly.

Method comparison

Method 1. Distinguishing with @Qualifier

The first method that comes to mind is to use @Qualifier to name each model bean.

@Configuration
public class ModelConfig {

    @Primary
    @Bean
    @Qualifier("textChatModel")
    public ChatModel textChatModel() {
        return OllamaChatModel.builder()
                .baseUrl("https://ollama.example.com")
                .modelName("chat-model-name")
                .build();
    }

    @Bean
    @Qualifier("imageChatModel")
    public ChatModel imageChatModel() {
        return OllamaChatModel.builder()
                .baseUrl("https://ollama.example.com")
                .modelName("multi-modal-model-name")
                .build();
    }
}

The user side injects it as follows.

@Service
public class SomeService {

    private final ChatModel textModel;
    private final ChatModel imageModel;

    public SomeService(
            @Qualifier("textChatModel") ChatModel textModel,
            @Qualifier("imageChatModel") ChatModel imageModel
    ) {
        this.textModel = textModel;
        this.imageModel = imageModel;
    }
}

This method is simple and intuitive. However, as the number of models increases, the @Qualifier strings spread across multiple classes.

Because it is string-based, it is difficult to catch typos at compile time, and the scope of impact increases when replacing models or changing names.

Method 2. Using @Qualifier + Proxy class

The second method is to encapsulate the @Qualifier without exposing it directly in the business logic, using proxy classes for each model.

This method is currently being used in the project.

@Service
public class ChatModelProxy {

    private final ChatModel chatModel;
    private final ObjectMapper objectMapper;

    public ChatModelProxy(
            @Qualifier("textChatModel") ChatModel chatModel,
            ObjectMapper objectMapper
    ) {
        this.chatModel = chatModel;
        this.objectMapper = objectMapper;
    }

    public String chat(Prompt prompt) {
        // 텍스트 모델 호출
    }

    public  T chat(Prompt prompt, Class clazz) {
        // 응답 JSON 파싱
    }

    public  List chatAnswerInList(Prompt prompt, Class clazz) {
        // 리스트 형태 응답 파싱
    }
}

The image analysis dedicated model is separated by a separate Proxy.

@Service
public class ImageChatModelProxy {

    private final ChatModel chatModel;

    public ImageChatModelProxy(
            @Qualifier("imageChatModel") ChatModel chatModel
    ) {
        this.chatModel = chatModel;
    }

    public String chatWithImage(String imageUrl, String textPrompt) {
        String base64 = fetchAsResizedBase64(imageUrl);
        String mimeType = detectMimeType(imageUrl);

        UserMessage message = UserMessage.from(
                ImageContent.from(base64, mimeType, ImageContent.DetailLevel.AUTO),
                TextContent.from(textPrompt)
        );

        return chatModel.chat(message).aiMessage().text();
    }
}

This way, the business logic no longer needs to know about @Qualifier.

@Service
public class ChatTask {

    private final ChatModelProxy chatModelProxy;
    private final ImageChatModelProxy imageChatModelProxy;

    public ChatTask(
            ChatModelProxy chatModelProxy,
            ImageChatModelProxy imageChatModelProxy
    ) {
        this.chatModelProxy = chatModelProxy;
        this.imageChatModelProxy = imageChatModelProxy;
    }
}

The advantages of this method are clear.

The @Qualifier string only exists within the Proxy, and the service layer does not need to know the specific bean names of each model. Additionally, model-specific pre-processing and post-processing logic, such as image downloading, Base64 encoding, and JSON parsing, can be encapsulated within the Proxy.

Method 3. Using a Configuration-Based Model Registry

If there are three or more models or if models need to be added or replaced just through configuration, the registry pattern should also be considered.

@Configuration
public class ModelRegistryConfig {

    @Bean
    public Map chatModelRegistry(ModelProperties props) {
        Map registry = new HashMap<>();

        props.getModels().forEach((name, config) -> {
            ChatModel model = OllamaChatModel.builder()
                    .baseUrl(config.getBaseUrl())
                    .modelName(config.getModelName())
                    .build();

            registry.put(name, model);
        });

        return registry;
    }
}

The settings can be managed as follows.

my-service:
  models:
    text:
      base-url: https://ollama.example.com
      model-name: text-model-name
      timeout: 10m
    image:
      base-url: https://ollama.example.com
      model-name: multi-modal-model-name
      timeout: 30m

This method is useful when there are many models or when models need to be swapped based on settings.

However, there are drawbacks. If the preprocessing logic required varies by model, simply managing it with a Map is insufficient.

For example, an image model requires image downloading, resizing, MIME type determination, and Base64 encoding. This logic ultimately ends up being positioned in a separate service or Proxy layer outside of the registry.

Therefore, if the model invocation method is simple and the same, the registry is suitable, but if each model has a different usage, the Proxy method is more natural.

The method chosen in the current project

In the current project, @Qualifier + Proxy class methodselected.

The roles of the two models are clearly different.

The text model is used for tasks such as general conversation and parsing JSON responses. In contrast, the image model requires a separate preprocessing step to handle image inputs.

Therefore, rather than forcing the two models into a single interface, it is more natural to separate them into proxies, each with its own responsibilities.

Of course, if the number of models increases to more than three or if there is a requirement to select models at runtime, the registry pattern can be considered as well.

In conclusion

1. It is difficult to manage multiple ChatModels in detail with just automatic configuration of LangChain4J.

2. If you need to use multiple models, you can define them directly with @Bean and distinguish them with @Qualifier.

3. By using a Proxy, the preprocessing and postprocessing logic for each model can be encapsulated, and the business logic does not need to be aware of the model bean names.

4. If the number of models increases or configuration-based replacement becomes important, the registry pattern can be considered.

Personally, in this situation, The Proxy method is the most appropriate choiceIt is considered that.

The text model and the image model are not merely 'two ChatModels'; rather, the way they are used and their responsibilities differ. Therefore, instead of abstracting them with a common interface or registry, it is better from a maintenance perspective to have a Proxy that clearly outlines the responsibilities of each model.

Ted