Skip to content

Request: Batch ordinary committed SQLite transactions and row metadata #1992

Description

@TimFL

Problem

Large committed transactions still persist rows individually unless they qualify for the full-replacement batching path introduced in #1916.

This is a follow-up to #1752. That fix helps full-snapshot replacements, but large inserts, updates, deletes and row-metadata changes also occur during normal synchronization.

Versions

  • @tanstack/db-sqlite-persistence-core: 0.4.0
  • @tanstack/browser-db-sqlite-persistence: 0.2.25
  • @tanstack/db: 0.11.0

The same sequential path is present in upstream main inspected on October 2, 2026.

Reproduction

With a SQLite-backed adapter initialized for collectionId, instrument database calls and apply a transaction without truncate: true:

const rows = Array.from({ length: 10_000 }, (_, i) => ({
  id: `row-${i}`,
  version: 1,
}))

await adapter.applyCommittedTx(collectionId, {
  txId: 'bulk-insert',
  term: 1,
  seq: 1,
  rowVersion: 1,
  mutations: rows.map(value => ({
    type: 'insert',
    key: value.id,
    value,
  })),
  rowMetadataMutations: rows.map(row => ({
    type: 'set',
    key: row.id,
    value: { version: 1 },
  })),
})

The ordinary path performs an existing-row read, an upsert and tombstone cleanup for each inserted/updated row. Row metadata adds another per-row statement. In browser persistence, these statements cross the worker boundary.

Requested behavior

Extend batching to ordinary committed transactions while preserving:

  • One atomic transaction containing rows, metadata and stream position.
  • Partial-update merge behavior and metadata overrides.
  • Sequential behavior when the same key appears multiple times.
  • Expected-key evidence, tombstones and replay bookkeeping.
  • Driver-specific parameter limits.

Downstream workaround

We patch the adapter to batch independent keys, fold matching metadata into row writes, and batch remaining metadata updates. Repeated keys retain the sequential path.

Existing downstream tests include a 100,000-row transaction with metadata and an assertion of fewer than 3,250 database calls, plus repeated-key ordering and rollback after a later batch fails.

This report is based on source inspection. Those tests were not rerun for this report, and the call-count assertion is not a browser latency measurement.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions