client-v2: a self-referential collection in a Dynamic column hangs (array/List) or throws StackOverflowError (Map) instead of a clean exception

[claude]

Describe the bug

A value that contains itself (e.g. Object[] a = new Object[1]; a[0] = a;) is not rejected when the client infers the type for a Dynamic column. ClickHouse cannot store a cyclic value, so the insert must fail — but instead of a ClientException, the client either spins forever or throws a bare StackOverflowError:

Value written to Dynamic	Result
Object[] a; a[0] = a	hangs (infinite loop, allocating; no timeout, no exception)
List<Object> l; l.add(l)	hangs
Map<String,Object> m; m.put("self", m)	java.lang.StackOverflowError
new Object[]{new Object[]{1,2}} (nested, no cycle)	OK — inserted (contrast case)
Object[] x = {1}; new Object[]{x, x} (shared, no cycle)	OK — Array(Array(Int32)) (contrast case)

The array/List case is the worse of the two: the call never returns, the calling thread is stuck in a tight loop, and a StringBuilder grows by "Array()" on every iteration, so the JVM heads toward OutOfMemoryError. The insert cannot be bounded with a query timeout because the serialization happens before anything is sent.

ClickHouse server version

Verified against a live server: 26.9.7.9 (HTTP, client-v2). The hang/StackOverflowError happens client-side during serialization, so the server version is not relevant to the failure.

Reproduction

End-to-end through the public insert API (client-v2, reusing the existing test POJO com.clickhouse.client.insert.PojoWithDynamic, which has Object any/Object nullableAny mapped to Dynamic). Each insert runs on a watchdog thread so the hang does not stall the test run:

package com.clickhouse.client.internal;

import com.clickhouse.client.api.Client;
import com.clickhouse.client.api.command.CommandSettings;
import com.clickhouse.client.insert.PojoWithDynamic;
import org.testng.annotations.Test;

import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;

public class CyclicDynamicE2ETest {

    @Test(groups = {"integration"})
    public void insertCyclicDynamic() throws Exception {
        try (Client client = new Client.Builder()
                .addEndpoint("http://localhost:8123")
                .setUsername("default").setPassword("")
                .compressServerResponse(false).compressClientRequest(false)
                .build()) {

            CommandSettings cs = new CommandSettings();
            cs.serverSetting("allow_experimental_dynamic_type", "1");
            client.execute("DROP TABLE IF EXISTS cyclic_dyn_test", cs).get();
            client.execute("CREATE TABLE cyclic_dyn_test (rowId Int32, any Dynamic, nullableAny Dynamic) ENGINE = Memory", cs).get();
            client.register(PojoWithDynamic.class, client.getTableSchema("cyclic_dyn_test"));

            Object[] selfArray = new Object[1];
            selfArray[0] = selfArray;                 // the array contains itself

            Map<String, Object> selfMap = new HashMap<>();
            selfMap.put("self", selfMap);             // the map contains itself

            probe(client, "nested array, no cycle (contrast)",
                    new PojoWithDynamic(3, new Object[]{new Object[]{1, 2}}, null));
            probe(client, "self-referential Object[]", new PojoWithDynamic(1, selfArray, null));
            probe(client, "self-referential Map", new PojoWithDynamic(2, selfMap, null));
        }
    }

    private static void probe(Client client, String label, PojoWithDynamic pojo) throws Exception {
        AtomicReference<String> result = new AtomicReference<>("NO_RESULT");
        Thread t = new Thread(() -> {
            try {
                client.insert("cyclic_dyn_test", Arrays.asList(pojo)).get();
                result.set("OK - inserted");
            } catch (Throwable e) {
                Throwable root = e;
                while (root.getCause() != null) root = root.getCause();
                result.set("THREW " + e.getClass().getName() + " / root " + root.getClass().getName() + ": " + root.getMessage());
            }
        });
        t.setDaemon(true);
        t.start();
        t.join(15_000);
        System.out.println("### " + label + " -> " + (t.isAlive() ? "HUNG (still running after 15s)" : result.get()));
    }
}

Actual output:

### nested array, no cycle (contrast) -> OK - inserted
### self-referential Object[] -> HUNG (still running after 15s)
### self-referential Map -> THREW java.lang.StackOverflowError / root java.lang.StackOverflowError: null

Expected: all three cyclic values fail fast with a ClientException (or another declared client exception) describing the circular reference; the contrast case keeps working.

The same is visible without a server by calling the inference entry point directly — SerializerUtils.valueToColumnForDynamicType(value), each call on a 10s watchdog thread:

### Object[] containing itself  -> HUNG (still running after 10s)
### List containing itself      -> HUNG (still running after 10s)
### Map containing itself       -> THREW java.lang.StackOverflowError: null
### nested no cycle             -> RETURNED v Array(Array(Int32))
### shared no cycle             -> RETURNED v Array(Array(Int32))

Root cause

client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java — no code path that descends into a container value tracks the values it has already entered:

• valueToColumnForDynamicType (SerializerUtils.java:230): the Map branch (:279-283) calls itself on the first entry's key and value. For m.put("self", m) that is unbounded recursion → StackOverflowError.
• listValue2Column (SerializerUtils.java:311) walks arrays/lists with an explicit Stack instead of recursion, so it does not overflow — it never terminates. A nested container is pushed back with depth + 1 (:342-343); a self-referential array is pushed again on every pop, maxDepth keeps increasing, and typeStr.insert(insertPos, "Array()") (:331) grows the type string without bound. Net effect: a spinning thread plus steadily growing allocation.

This is the Java counterpart of ClickHouse/clickhouse-cs#651 — same defect class, different symptom: the C# client aborts the process with a stack overflow, here the array/List case hangs and the Map case throws a bare Error.

Suggested fix

Bound the descent into container values in SerializerUtils:

• Thread a depth counter (or a reference-identity "visited" set, as the C# client already does for POCO fields) through valueToColumnForDynamicType's Map branch and through listValue2Column's stack entries, and throw a ClientException when the limit is exceeded or a value is re-entered. System.Text.Json's limit of 64 is a reasonable reference point for a depth cap — far above any realistic nesting, so scalar and shallow values pay nothing.
• A visited set gives the better message ("circular reference detected"); a depth cap is cheaper on this hot path. Either must keep the contrast cases working: nested containers without a cycle (new Object[]{new Object[]{1,2}}) and one instance referenced more than once without a cycle (new Object[]{x, x}) — so a visited set has to drop an entry when its scope ends.

Note that listValue2Column needs its own guard even though it is iterative: a limit there is what turns the hang into an exception.

Link

Relayed from ClickHouse/clickhouse-cs#651 — ClickHouse/clickhouse-cs#651
Central tracking issue: ClickHouse/integrations-ai-playground#534