feat(events): persist events to disk so they survive the process (tier 3)

[abelonogov-ld]

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Related issues

Tier 3 of the Event Durability spec, "Tier 3 — persistence". Stacked on #403 (tier 2) and targets that branch, so this diff shows only tier 3. It will be retargeted once the branches below it merge. The iOS counterpart is launchdarkly/ios-client-sdk#538, and both write the same on-disk format.

Opened as a draft: a multi-threading and readability review turned up items still being worked through (see Known open items).

Describe the solution you've provided

Recorded events go to an append-only log on disk rather than only an in-memory buffer, so an application that dies shortly after recording an event still reports it: the next run delivers what the previous one left behind. That covers what tier 2 cannot: SIGKILL, ANR kills, native crashes, and the system reclaiming a backgrounded process.

• EventPersistence (new public enum) and EventProcessorBuilder.eventPersistence(...):
  • DISABLED (default): events stay in memory and the SDK does not touch the filesystem for events.
  • DEFERRED: writes happen on the SDK's commit thread, a moment after recording.
  • IMMEDIATE: track and identify write on the caller's thread before returning, so there is no window.
• EventStore: an append-only log under the no-backup files directory, one directory per mobile key. It handles framing, capacity, torn-tail recovery of a log interrupted by a crash, and a fallback to memory when a write fails (for example, a full disk), so a disk failure never takes the application down. Each process appends to its own open-<process> log; closed ready-<payloadId> batches are shared, so a batch left by a process that never runs again is still delivered.
• Recording and commits (DirectEventProcessor): an evaluation appends to a pending run and updates a counter, with no encoding on the caller's thread. A commit encodes the run and its summaries together and stages them in the store. track and identify are commit points. Evaluations commit once a threshold accumulates, on flush, and before delivery.
• Delivery reads batches oldest first. A batch keeps its payload ID across retries and process restarts, so LaunchDarkly discards a repeated delivery instead of double counting. Delivery stops at the first retryable failure, schedules a single retry instead of sleeping on the events thread, and removes a batch only once it has been accepted or permanently refused. A batch whose read fails transiently is kept for a later attempt.
• AnalyticsEventSender posts a pre-assembled body under an explicit payload ID.
• DEFAULT_CAPACITY goes from 100 to 1000, matching iOS, since capacity now also bounds what is held on disk.

Describe alternatives you've considered

SQLite. Measured as a store and rejected: it costs more per event than an append to a log the SDK already frames, for durability properties the log gets from write alone. See sqlite-event-persistence-findings.md in the spec folder.

Fsync on every commit. What durability is defending against is the process dying, not the device losing power. Bytes handed to the kernel survive the former, and an fsync per event would dominate the cost of track.

One shared log for all processes. That is simpler, but one process's startup recovery could rename a log another live process is still appending to. A log per process avoids that without file locks.

Additional context

Tests. 873 unit tests pass locally (testDebugUnitTest). New coverage includes EventStoreMultiProcessTest (several stores on one directory, recovery, transient read failures), DirectEventProcessorTest (commit points, retry, capacity across memory and disk, persistence failing partway through, oversized frames reported as losses), OutboundEventBufferSerializationTest, and SameContextPerContextEventSummarizerTest.

Benchmarks. EventPersistenceBenchmark (JVM and instrumented) measures recording cost per mode. It runs only when LD_EVENT_BENCH=1, so ordinary test runs are unaffected.

Test app. Now configures EventPersistence.IMMEDIATE, so the events recorded by Eval+Kill now, which tier 2 loses, are recovered and delivered on the next launch. It also gains Over-Refresh Eval, a main-thread stress loop that compares evaluation cost while filling capacity and after it is full.

Known open items (from the threading review, to be resolved before this leaves draft):

• Capacity briefly under-counts a batch while it is being written to disk.
• The "events were lost" flag behind flushAsync() can be consumed by a different flush than the one that lost them.
• close()'s fallback commit can wait on the commit lock past the two-second close budget if the disk stalls.
• Batches left by a previous run are not counted against capacity until the first listing, which an offline start postpones.
• Readability: about 200 lines of tier-2 whole-payload encoding in OutboundEventBuffer are now unused outside tests, and a few comments still describe tier 2.

Version. EventPersistence and EventProcessorBuilder.eventPersistence have no @since tag yet; one should be added once the release version is known.