跳到正文
DEV Community· Said Olano·· 14 小時前AI 評分58

Spring AI 並行工作流:如何在 Java 中扇出呼叫 LLM

Parallelization Workflow with Spring AI: Fan-Out LLM Calls in Java

AI 導讀

本文拆解 Spring AI 範例中的並行工作流,示範如何在 Java 中同時執行多個獨立 LLM 呼叫,並按輸入順序合併結果。範例支援把不同子任務分段處理,或對同一輸入多次投票。文章亦説明結果排序、部分失敗、逾時、速率限制及成本等實作考量,並指出並行可縮短批次等待時間,但不會降低呼叫費用。

正文

When people first start building with LLMs in Java, the usual shape of the code is simple: build a prompt, call the model, read the answer. That works well for a single question. It starts to hurt when you need to ask the same model the same kind of question a hundred times, or when one request is really five smaller questions that happen to be asked together. Each call waits for the network, the model generates tokens at its own pace, and your thread sits idle for most of that time.

Anthropic's article "Building Effective Agents" describes a set of workflow patterns for composing LLM calls. One of them is the Parallelization Workflow: instead of running model calls one after another, you run them at the same time and then combine the results. The Spring AI team has published a small, focused example of this pattern in the spring-ai-examples repository, under agentic-patterns/parallelization-workflow. It is a compact class built on top of Spring AI's ChatClient, and it is a good starting point for understanding how to fan out LLM work in a Spring application.

In this article I walk through the pattern, the two variations it supports (sectioning and voting), the code in the example, and the practical issues you will face when you use it for real workloads: ordering, failures, rate limits, cost, and testing. I will keep the code close to the example so you can follow along with the repository open.

Why parallelize LLM calls

A typical chat completion can take anywhere from a second to tens of seconds, depending on the model, the size of the prompt, and the length of the answer. Those numbers matter when you multiply them.

Imagine you need to analyze ten product reviews, one at a time, with a call that takes four seconds. That is forty seconds of wall-clock time, most of it spent waiting. If the calls are independent of one another, there is no reason to wait for review number three before starting review number four. You can send them all at once and collect the answers as they arrive.

Parallelization helps in three ways:

  • Throughput. You process more items in the same amount of time.
  • Latency. The total time for a batch is closer to the slowest single call than to the sum of all calls.
  • Quality, in some cases. When you ask the same question several times and combine the answers, you can reduce the impact of a single unlucky generation.

It also has costs. You pay for every call, not for every batch. You can overload a provider's rate limits. And your application has to handle partial failures, which are more complicated when several calls are in flight at once. The rest of this article is about getting the benefits without ignoring those costs.

The pattern: sectioning and voting

The example README names two variations of the pattern. They solve different problems, so it helps to keep them separate in your head.

Sectioning breaks a larger task into independent subtasks and processes each one in parallel. The subtasks do not depend on each other, so the order of execution does not matter. Each call gets a different input and, often, a slightly different instruction. A typical case is analyzing several stakeholder groups, several document sections, or several customer segments with the same analytical prompt.

Voting runs the same prompt several times, usually against the same input, and then combines the results. The goal is diversity of perspective or a consensus. If you ask a model whether a transaction looks fraudulent five times and four answers say "review," you have a more robust signal than a single answer. The aggregation step can be a majority vote, a rule that requires a minimum agreement, or another model call that summarizes the candidates.

The distinction is practical. Sectioning is about speed on a set of different problems. Voting is about confidence on one problem. You can combine them, but it is clearer to start with one and then add the other.

The Spring AI example

The core of the example is one class: ParallelizationlWorkflow, in the package com.example.agentic. It takes a ChatClient in its constructor and exposes a single method:

public List<String> parallel(String prompt, List<String> inputs, int nWorkers)

The method takes a prompt template, a list of inputs, and the number of worker threads. It returns a list of model responses in the same order as the inputs. Here is the heart of the implementation, lightly trimmed:

ExecutorService executor = Executors.newFixedThreadPool(nWorkers);
try {
    List<CompletableFuture<String>> futures = inputs.stream()
            .map(input -> CompletableFuture.supplyAsync(() -> {
                try {
                    return chatClient.prompt(prompt + "\nInput: " + input).call().content();
                } catch (Exception e) {
                    throw new RuntimeException("Failed to process input: " + input, e);
                }
            }, executor))
            .collect(Collectors.toList());

    CompletableFuture<Void> allFutures = CompletableFuture.allOf(
            futures.toArray(CompletableFuture[]::new));
    allFutures.join();

    return futures.stream()
            .map(CompletableFuture::join)
            .collect(Collectors.toList());
} finally {
    executor.shutdown();
}

There are a few design decisions worth noticing.

First, the class creates a fixed thread pool for each call. The pool size is the nWorkers argument, which becomes a hard cap on how many model requests are in flight at once. This is a deliberate choice: the number of concurrent calls is an explicit input, not something you discover by accident.

Second, each input becomes a CompletableFuture submitted to that executor. The futures run concurrently, but they are stored in a list in the same order as the inputs. That is how the method keeps ordered results. The fact that a later request finishes first does not change where its answer ends up in the returned list.

Third, the method waits for all futures with CompletableFuture.allOf(...).join() and then collects the individual results with join() again. Once allOf completes, each individual join() returns immediately.

Fourth, the input is appended to the prompt with a simple "\nInput: " separator. This is a minimal way to build a per-item prompt from a shared instruction. In a production system you would probably use a structured template, but the principle is the same.

Finally, the finally block shuts the executor down. Because the pool is created inside the method, this prevents thread leaks across calls.

Wiring it into a Spring Boot application

The example's Application class shows the most direct use. It injects a ChatClient.Builder, builds a ChatClient, and runs the workflow from a CommandLineRunner:

@Bean
public CommandLineRunner commandLineRunner(ChatClient.Builder chatClientBuilder) {
    return args -> {
        List<String> parallelResponse = new ParallelizationlWorkflow(chatClientBuilder.build())
                .parallel("""
                        Analyze how market changes will impact this stakeholder group.
                        Provide specific impacts and recommended actions.
                        Format with clear sections and priorities.
                        """,
                        List.of(
                                "Customers: price sensitive, want better tech, environmental concerns",
                                "Employees: job security worries, need new skills, want clear direction",
                                "Investors: expect growth, want cost control, risk concerns",
                                "Suppliers: capacity constraints, price pressures, tech transitions"),
                        4);

        System.out.println(parallelResponse);
    };
}

In a real application you would not print to the console. You would expose the workflow as a bean and call it from a service or controller. Here is one way to do that:

@Configuration
class WorkflowConfig {

    @Bean
    ParallelizationlWorkflow parallelizationWorkflow(ChatClient.Builder builder) {
        return new ParallelizationlWorkflow(builder.build());
    }
}

@Service
class StakeholderAnalysisService {

    private static final String PROMPT = """
            Analyze how market changes will impact this stakeholder group.
            Provide specific impacts and recommended actions.
            """;

    private final ParallelizationlWorkflow workflow;

    StakeholderAnalysisService(ParallelizationlWorkflow workflow) {
        this.workflow = workflow;
    }

    List<String> analyze(List<String> stakeholderDescriptions) {
        return workflow.parallel(PROMPT, stakeholderDescriptions, 4);
    }
}

Note that the example's application.properties configures Anthropic and Ollama properties, while the pom.xml currently includes the Anthropic model starter. If you run the example, make sure the model provider you actually use is on the classpath and configured. Spring AI's auto-configuration will pick up the ChatClient.Builder from whichever model starter is present.

Sectioning in practice

The stakeholder example is a classic sectioning use case. The four stakeholder groups are independent: the analysis of customers does not depend on the analysis of suppliers. Running them in parallel with four workers means the batch takes roughly as long as the slowest single analysis, not the sum of all four.

The shape of the input matters. Each string in the list should carry enough context for the model to work on it alone. In the example, each group has a short bullet list of concerns. In a production system, you might pass a full document section, a customer record with its recent activity, or a set of tickets sharing a common category.

Sectioning also works well for document processing. Suppose you have a long contract split into clauses. You can ask the model to extract obligations from each clause in parallel:

List<String> clauses = splitIntoClauses(contractText);

List<String> obligations = workflow.parallel(
        """
        Extract every obligation the parties take on in this clause.
        Return a short bullet list. If there are none, return "None".
        """,
        clauses,
        6);

The results come back in clause order, so you can zip them with the original clauses to build a report without any extra bookkeeping.

A common mistake is to treat sectioning as a way to split a task that actually depends on itself. If the second section needs the conclusions of the first, the sections are not independent, and parallelizing them will give you wrong or inconsistent answers. The README makes this point clearly: ensure tasks are truly independent. If you need a chain of dependent steps, that is a different workflow, not a faster version of this one.

Voting in practice

Voting uses the same method, but the inputs are the same or nearly the same. The class appends the input to the prompt, so passing identical inputs produces identical prompts. You can use Collections.nCopies for that:

String transactionSummary = """
        Amount: 4,900 USD
        Merchant category: electronics
        Country: different from the cardholder's usual country
        Time: 03:12 local time
        """;

List<String> votes = workflow.parallel(
        "Is this transaction likely fraudulent? Answer with exactly one word: APPROVE or REVIEW.",
        Collections.nCopies(5, transactionSummary),
        5);

long reviewCount = votes.stream()
        .filter(v -> v.trim().toUpperCase().startsWith("REVIEW"))
        .count();

boolean needsReview = reviewCount >= 3;

There is an important detail here. If the model is deterministic, five identical prompts will produce five identical answers, and voting adds nothing. Diversity comes from sampling. You need a non-zero temperature, or a varied prompt, to get meaningfully different answers. Check the defaults of the model you use and set temperature explicitly through the ChatClient options if you depend on variation. Otherwise, you are paying for five calls to get one answer.

You can also vary the prompt slightly across voters, asking for different perspectives. For example, one voter can focus on the amount and another on the timing. This makes the vote more like a panel of reviewers and less like a repeated sample, but you need to make sure the aggregation logic still makes sense.

Parsing the answers deserves care. The example above is strict: it expects a single word and checks the start of the string. In production, you should either ask for structured output and parse it into a type, or define a clear fallback for answers that do not match the expected format. A vote that cannot be counted should not silently count as "approve."

Ordering and why it matters

The method's promise that results come back in input order is one of its most useful properties. It means you can treat the output as a parallel array aligned with the input list. You do not need to attach IDs to each request or sort results afterward.

This is worth preserving when you extend the code. If you switch to a version that returns results as they complete, you lose the alignment unless you carry an identifier through the pipeline. A simple approach is to return a small record:

public record ItemResult(int index, String input, String output, Throwable error) {
    public boolean succeeded() {
        return error == null;
    }
}

Each item keeps its index and its input, and the result can say whether it succeeded. That leads directly to the next topic.

Failures and partial results

The example is strict about failures. If any single input fails, the join() calls throw, and the whole batch fails. The exception carries a message that names the input that failed, which is helpful for debugging, but it means one bad call can discard the results of the other ones.

Whether that is the right behavior depends on your use case. For a batch that must be complete, such as a compliance check where a missing answer is worse than no answer, failing fast is correct. For a batch where most items matter and a few failures are tolerable, you want partial results.

Here is a sketch of a tolerant variant that keeps the successful answers and records the errors:

public List<ItemResult> parallelTolerant(String prompt, List<String> inputs, int nWorkers) {
    ExecutorService executor = Executors.newFixedThreadPool(nWorkers);
    try {
        List<CompletableFuture<ItemResult>> futures = new ArrayList<>();
        for (int i = 0; i < inputs.size(); i++) {
            int index = i;
            String input = inputs.get(i);
            futures.add(CompletableFuture
                    .supplyAsync(() -> chatClient.prompt(prompt + "\nInput: " + input)
                            .call()
                            .content(), executor)
                    .handle((output, error) -> new ItemResult(index, input, output, error)));
        }
        return futures.stream()
                .map(CompletableFuture::join)
                .toList();
    } finally {
        executor.shutdown();
    }
}

The handle method runs whether the future succeeded or failed, so every item produces an ItemResult. The caller can then decide what to do with failed items: retry them, mark them for manual review, or skip them with a log entry.

Two practical notes. First, do not swallow errors silently. A result list full of nulls is much harder to debug than a list with explicit error records. Second, if you need retries, add them around the model call inside the supplier, with a bounded number of attempts and a backoff. Retrying an entire batch because one item failed wastes calls and can make rate limiting worse.

Concurrency, rate limits, and cost

The nWorkers argument is the most important tuning knob in this pattern. It determines how many requests can be in flight at once, and therefore how hard your application pushes the model provider.

A higher number of workers reduces latency for large batches, up to the point where you hit a rate limit. Once the provider starts returning rate-limit errors, you are no longer parallelizing usefully; you are generating retries. A lower number of workers is slower but more predictable.

There is no universal right value. Start with a small number, measure the throughput and the error rate, and increase gradually. The README's advice to consider API rate limits is the right instinct. Your limits depend on your provider, your account tier, the model, and the token counts per request, so treat any number as a starting point to be verified in your environment.

Also remember that parallelism does not reduce cost. Five calls cost five times as much as one, whether they run in sequence or in parallel. Parallelization buys you time, not money. If you are running voting or sectioning at scale, the cost model should be part of the design from the start: how many items per day, how many votes per item, how long the prompts are, and which model you use for each step.

One more consideration is the executor itself. The example creates a new fixed thread pool on every call. For a low-volume application that is fine. For a high-volume service, you might prefer a shared executor bean with a bounded queue, so that concurrent batches from different requests share the same capacity and cannot exhaust your resources together. In that case, you would also need to think about backpressure: what happens when the queue is full, and whether the caller should wait, fail, or degrade.

Timeouts

Every parallel call should have a timeout. An LLM request that hangs will hold a worker thread indefinitely, and with a fixed pool that means fewer workers available for the rest of the batch. A batch where one slow call blocks the join is not a parallel system; it is a sequential one with extra steps.

You can bound the wait with CompletableFuture.orTimeout or completeOnTimeout on each future, or with get(timeout, unit) when collecting results. Choose a value that reflects the real latency distribution of your prompts, not an optimistic guess. Log timeouts with the input that caused them, so you can see whether certain items are consistently expensive.

Testing the workflow

Because the class depends only on ChatClient, it is easy to test with a stub. You do not need a live model to check the important properties: that results come back in order, that inputs are passed through correctly, and that failures are handled as you expect.

Here is one way to structure a test with a fake client. The exact mocking approach depends on your test setup, so treat this as a sketch:

@Test
void resultsKeepInputOrderEvenWhenCallsFinishOutOfOrder() {
    ChatClient client = mock(ChatClient.class, RETURNS_DEEP_STUBS);

    when(client.prompt(anyString()).call().content()).thenAnswer(invocation -> {
        String prompt = invocation.getArgument(0);
        if (prompt.contains("Input: slow")) {
            Thread.sleep(200);
        }
        return prompt.substring(prompt.indexOf("Input: ") + 7).toUpperCase();
    });

    ParallelizationlWorkflow workflow = new ParallelizationlWorkflow(client);

    List<String> results = workflow.parallel("Echo in uppercase.", List.of("slow", "fast"), 2);

    assertEquals(List.of("SLOW", "FAST"), results);
}

The point of this test is the ordering guarantee. The "slow" input finishes after the "fast" one, but the output list still starts with "SLOW". If someone later changes the implementation to return results as they complete, this test will catch it.

You should also test the empty-input and zero-worker validation, and the failure path. These are cheap tests that protect the contract of the method.

For integration tests against a real model, keep the number of calls small, use a fixed seed or low temperature where the provider supports it, and assert on structure rather than exact wording. Model output varies, and tests that compare full paragraphs will be brittle.

Common pitfalls

A few mistakes come up repeatedly with this pattern.

Treating dependent tasks as independent. If one call needs the output of another, parallelizing them will produce answers that ignore each other. Use a sequential chain or a different orchestration pattern for those cases.

Assuming voting adds accuracy without diversity. Five identical deterministic answers are one answer, five times. Check sampling settings before relying on a vote.

Ignoring the cost of fan-out. A batch of five hundred items with five votes each is 2,500 calls. Calculate the cost before running it in production.

Letting one failure discard a batch. Decide whether your use case needs all-or-nothing behavior, and implement the tolerant variant if it does not.

Sizing the pool by instinct. Measure. The right number of workers depends on the provider, the prompt size, and your rate limits.

Parsing free text as if it were structured. When you aggregate votes or extract fields, use a defined output format and handle anything that does not match it.

Forgetting timeouts. A single hung request should not stall the batch.

Best practices

Here is a short checklist I would apply to any production use of this pattern.

  1. Confirm independence. Write down, in one sentence, why each item can be processed without the others. If you cannot, do not parallelize.
  2. Make concurrency explicit. Pass the worker count from configuration, not as a hardcoded constant, so you can tune it per environment.
  3. Keep order or identity. Either preserve input order, as the example does, or carry an index or ID through the pipeline.
  4. Record per-item outcomes. Store success, failure, and error details for each input so you can retry or audit them.
  5. Bound every call. Set timeouts on each request and on the overall batch.
  6. Budget the calls. Estimate the number of model calls and the token cost before running a batch at scale.
  7. Use structured outputs for aggregation. Ask for a format you can parse, and define what happens when parsing fails.
  8. Test the contract. Cover ordering, failure handling, and validation with fast tests that do not call a model.
  9. Observe the system. Log latency, error rates, and rate-limit responses per batch, so tuning decisions are based on data.

When not to use this pattern

Parallelization is not always the answer. If the input set is small, the overhead of managing a thread pool and aggregating results may not be worth it. A few sequential calls are easier to reason about and debug.

If your workflow involves a decision that routes work to different specialists, a routing pattern fits better. If the output of one step should be improved by a critic before moving on, an evaluator-optimizer loop is the right shape. If the subtasks are not known in advance and must be planned by a model, an orchestrator-workers design is more appropriate. The Parallelization Workflow is the right tool when you already know the work splits into independent pieces, or when you want several independent opinions on the same question.

Where to go next

The example is intentionally small, and there are several natural extensions.

  • Structured outputs. Replace the string results with typed records, using Spring AI's structured output support, so that aggregation becomes a type-safe operation.
  • Retries with backoff. Wrap the model call in a retry policy that distinguishes transient errors, such as rate limits, from permanent ones, such as invalid requests.
  • Shared executor. Move the thread pool to a bean with explicit bounds and metrics, so that concurrency is controlled across the application.
  • Aggregators. Add a second model call that summarizes the parallel results into a single report, which is a common final step in sectioning workflows.
  • Observability. Instrument each call with timing and token counts, and export them to your metrics system.

Each of these builds on the same core idea: independent calls, run concurrently, combined at the end.

Conclusion

The Parallelization Workflow is one of the simplest and most useful patterns for building with LLMs. It turns a series of independent model calls into a concurrent batch, with results returned in a predictable order. The Spring AI example gives you a clean, readable starting point: a fixed thread pool, a CompletableFuture per input, and an ordered list of answers.

The pattern comes in two flavors. Sectioning speeds up work on different subtasks. Voting strengthens a single decision by gathering several perspectives. Both depend on the same discipline: the calls must be independent, the concurrency must be bounded, failures must be visible, and cost must be planned.

If you take one idea from this article, let it be this: parallelism is a property you design, not a switch you flip. Choose the worker count deliberately, handle the failures you will certainly see, and measure the results. The code in the example is a good foundation. The judgment about how to use it is yours.

Key takeaways

  • The Parallelization Workflow runs independent LLM calls concurrently and combines the results.
  • Sectioning splits a task into different subtasks; voting runs the same question several times for consensus.
  • The Spring AI example uses a fixed thread pool, CompletableFuture, and returns results in input order.
  • Worker count controls concurrency and must be tuned against rate limits.
  • Parallelization saves time, not money: every call is still billed.
  • Decide up front whether a batch should fail completely or return partial results.
  • Set timeouts, record per-item outcomes, and parse outputs against a defined format.
  • Use the pattern only when the subtasks are genuinely independent.

References

來源:DEV Community · dev.to