Add database pressure observability

[TheSentinel454]

Why

Database pressure currently collapses several distinct delays into one symptom. This adds the evidence layer needed to distinguish pool acquisition wait, logical database operation time, advisory-lock wait, and selected transaction duration before changing timeout or retry policy.

This is the phase 2 Lane A observability bundle for #26, #28, and #33. It is stacked on #6668.

What

• Record explicit reader/writer checkout wait and acquisition outcomes with buzz_db_pool_acquire_wait_seconds and buzz_db_pool_acquisitions_total.
• Extend the compile-time #[datastore_span(name = "...")] seam with buzz_db_operation_duration_seconds, so operation labels remain static source literals instead of request data.
• Route correctness-critical replacement, membership, push-gate, deletion, and migration/schema-safety advisory locks through one observer without changing their SQL, order, scope, or blocking behavior.
• Measure six internally owned transaction lifetimes with buzz_db_transaction_duration_seconds, starting after BEGIN succeeds and ending after explicit commit/rollback or scope exit.
• Emit root slow-operation warnings at 500 ms, logging the first slow completion and then 1/100 per call site with only operation, outcome, and elapsed_ms.
• Document names, units, fixed label vocabularies, measurement boundaries, and blind spots in this PR description.

Fixed labels are deliberately small:

• pool_role: writer, reader
• lock_type: replacement, membership, push_gate, deletion, migration_schema_safety
• outcome: success, error, timeout where SQLx/PostgreSQL can distinguish it accurately
• operation: compile-time datastore names plus the six closed transaction operation names documented in the runbook

No metric or slow warning contains community IDs, event IDs, event kinds, coordinates, d-tags, SQL/query text, query IDs, returned errors, or event content.

Coverage boundaries

• Operation duration is the complete annotated logical function body, not pure SQL execution; it may include implicit checkout, lock wait, nested operations, and application work. Cancelled futures do not reach its completion hook.
• Pool timing covers explicit helper checkouts, including proved-reader routing and selected writer-owned transactions. Implicit SQLx checkout through &PgPool remains folded into operation duration.
• Lock timing covers application-side blocking locks in the five named families. Trigger/stored-procedure locks, channel-TTL locking, the usage try-lock, and the audit service session lock remain outside this slice.
• Transaction timing covers only the six wholly owned boundaries documented in the runbook. It excludes pool wait, BEGIN, asynchronous rollback cleanup after an early return, and caller-owned Db::begin_transaction lifetime.

Relationship to #6229

#6229 is the incident-driven timeout precursor. This PR does not add or change statement_timeout, lock_timeout, idle_in_transaction_session_timeout, retries, audit durability, or client-visible conflicts. It provides the missing distributions needed to evaluate those policies later and intentionally leaves #6229's open audit retry/durability finding untouched.

The branches overlap in crates/buzz-db/src/lib.rs and crates/buzz-db/src/migration.rs, so a later rebase may need textual conflict resolution, but the behavior is complementary rather than duplicated.

Risk assessment

Moderate-low. The primary risk is instrumentation overhead and added static series. Cardinality is source-bounded, slow logs are sampled/redacted root events, and the lock/transaction changes wrap existing awaits without changing policy or ordering.

Verification

Author workstation: buzz-tornquist-db-pressure-observability (2010927), exact head d7cf833e26c528adfcde3917ded80daf6f4ddac9, parent 6f50e6b2b2a996349149af61d35bdd6a355f77fd.

• cargo fmt --all --check — passed
• cargo clippy -p buzz-datastore-tracing -p buzz-db -p buzz-audit -p buzz-search -p buzz-relay --all-targets -- -D warnings — passed
• cargo test -p buzz-datastore-tracing --quiet — 4 passed
• cargo test -p buzz-db --quiet — 109 passed, 200 ignored
• cargo test -p buzz-audit -p buzz-search --quiet — 16 passed, 25 ignored
• cargo test -p buzz-relay --lib --quiet -- --test-threads=1 — 906 passed, 48 ignored
• Native PostgreSQL focused tests for pool success/timeout/error, lock success/contention/timeout/error, replacement, membership serialization, push ordering, deletion fencing, migration/schema exclusion, and reader fallback — 8 passed

The default-parallel relay run passed once; subsequent runs exposed the existing load-sensitive api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo 504 at the end of the suite. That test passes in isolation and the full relay suite passes serially.

Independent exact-head review workstation: buzz-tornquist-db-pressure-observability-review (2013067). Formatting, the same all-target clippy command, datastore instrumentation tests, DB unit tests, source privacy guards, and diff/non-goal audits passed; no review findings.

Generated with Codex

[TheSentinel454]

Looks like the CI failure is due to some other stuff outside of this PR. Work happening in this PR: #6618

[wpfleger96]

🤖 Combined review at b74e724 (two independent source reviews plus live verification against a release relay at this exact head). No blocking findings.

Source verification: every instrumented advisory-lock site wraps the identical SQL, bind values, lock order, and shared/exclusive mode — observe_advisory_lock only times the same awaited future, and the deletion.rs helper refactor doesn't move any lock relative to row locks. Label cardinality is closed at the source level (four enums plus compile-time LitStr operation names), with PoolTimedOut/55P03 correctly mapped to timeout. All early-return Ok paths in the six timed transactions call mark_success(), and the explicit timer drop before the post-commit mention indexing in the membership publisher keeps that work out of the measured window. Privacy is enforced both by the source-guard test and the runtime tracing-layer assertion (parent: None, exactly operation/outcome/elapsed_ms). The relay's Prometheus _seconds matcher already covers the four new duration families.

Live verification at this head: the new metric families appeared on a real /metrics scrape; under genuine advisory-lock contention a 525 ms wait recorded correctly and the ≥500 ms slow warning fired with only the fixed fields — no host, pubkey, SQL, or error content leaked. Both new ignored PostgreSQL pressure probes passed against a real database (pool saturation timeout, closed-pool error, blocked-waiter contention, 55P03, aborted-transaction error).

Two non-blocking notes:

• The red Unit Tests check is unrelated to this PR: it fails linking sherpa-onnx-c-api while compiling buzz-voice, and the identical failure reproduces on main tip alone (intermittent — main alternates green/red on it). Needs a rerun or main-side fix before merge, but nothing here caused it.
• MINOR for the ignored-test CI follow-up you announced: mark_success() is manual, so a future early-return Ok path added to one of the six timed transactions would silently record error. An Ok-driven success wrapper would remove that footgun; also the two new #[ignore] observability tests fall into the same unselected-by-CI bucket as the rest.

[wpfleger96]

🤖 Re-approved at exact head 6363c3aedf3930e11f88d71c96248e03e6fca272 after verifying the rebase.

git range-diff d7cf833e^..b74e724e 6f8ad98b^..6363c3ae reports both reviewed commits as exact patch-equivalent (=); their stable patch IDs also match individually. The current branch is directly based on 9d1e4b257657f382d3111ce748f3da8d063b7671, which is the current PR base and origin/main tip fetched for this check. All current required and optional CI checks are green or intentionally skipped. No new source changes to review.

[wpfleger96]

🤖 Re-reviewed and approved at exact head d75c4e5733b4bf22292703304a8bad26ea62789c.

The only new commit since the previously approved 6363c3aedf3930e11f88d71c96248e03e6fca272 is Harden transaction observability coverage. I traced all six converted transaction paths: the new TransactionTimer::observe wrapper preserves each SQL operation, lock order, commit/rollback decision, return value, and the intended transaction measurement boundary while deriving the metric outcome from the final Result. The membership path still excludes post-commit mention indexing and now correctly skips that indexing when no event was inserted. The added PostgreSQL CI step selects the two ignored real-database observability probes that were previously uncovered. No new findings.

All current CI checks at this head are green or intentionally skipped.