fix: in-container demo reseed stuck at Scenario 0 (idempotent resolver + Solr data-dir + Arq dedup-retry)

[SoundMindsAI]

Summary

Fixes the home-button demo reseed sticking at "Scenario 0 of 6" (operator-reported, live). The symptom had three stacked root causes in the in-container reseed path — all verified fixed live on a Solr stack:

1. Reseed couldn't run at all — _resolve_engine_base_url (demo_seeding.py) only mapped host-shell localhost URLs → Compose-DNS and raised on anything else. But the worker is always in-container, where seed_meaningful_demos.py's _INSIDE_CONTAINER branch sets each scenario's host_base_url to the already-Compose-DNS URL — so the reachability snapshot fed those back in and the run died with Unrecognized engine host URL: http://elasticsearch:9200. Latent because the reseed's integration tests mock the engine-probe layer. Fix: make the resolver idempotent (pass an already-resolved Compose-DNS target through unchanged; still raise on genuinely-unknown URLs).

2. Solr couldn't create collections — install.sh only mkdir'd ./secrets, never the engine data dirs. After make reset (rm -rf ./data) or a fresh clone, ./data/solr doesn't exist when Solr starts, so its /var/solr bind is a phantom dir UID-8983 can't write — every collection CREATE failed (Underlying core creation failed). Fix: pre-create ./data/solr (+ chown 8983 on Linux) before compose-up, gated on Solr in COMPOSE_PROFILES, mirroring the pr.yml smoke job.

3. Couldn't reseed twice within an hour — the POST enqueues a deterministic _job_id ("demo_reseed:singleton") for double-click protection, but Arq aborts a re-enqueue while the prior run's result is cached under arq:result:<id> — kept for keep_result (Arq default 1 hour, not the 60s the old comment claimed). So after any terminal run, the next Reset click was silently deduped and stuck on "enqueued — waiting for worker" with an empty step log. Fix: delete the stale result key before enqueue (the running-status 409 guard already prevents genuine concurrency; rapid double-clicks are still deduped by the first click's in-flight arq:job key). Resolves the previously-filed bug_reseed_failure_blocks_retry_arq_singleton_dedup (folder removed).

Verified live (Solr-only stack)

• Reseed advances past the reachability snapshot (was: stuck Scenario 0) and runs the scenario's full 50-trial Optuna study.
• Retry after a completed run: stale arq:result key present → POST clears it → worker immediately picks up the job (no stuck "enqueued").
• install.sh re-runs idempotently; ./data/solr preserved, Solr healthy.

Test plan

• 3 unit cases — _resolve_engine_base_url idempotent per Compose-DNS URL (15/15 demo_seeding unit tests pass)
• Integration case — POST clears a seeded stale arq:result:<id> then enqueues (not dedup-to-None); SpyArqPool gains a delete() double
• make fmt && make lint && make typecheck && ruff format --check clean; bash -n/shellcheck clean on install.sh

Notes

• "Scenario 0 of 6" not visibly advancing during the first scenario's multi-minute Optuna study is a separate UX issue (the counter only increments when a whole scenario completes) — addressed by the next change (a per-scenario live-state checklist).
• Unverified: whether ES/OS bind dirs share the Solr missing-dir failure mode (couldn't test — those engines aren't running in this Solr-only stack). Scoped to the proven Solr case.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request makes the engine base URL resolver idempotent to prevent failures when running inside a container, adds corresponding unit tests, and pre-creates the Solr data directory with correct ownership in the installation script. The reviewer feedback suggests normalizing URLs by stripping trailing slashes for robustness and using non-interactive sudo -n in the installation script to prevent hanging in automated environments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To make the URL resolution more robust and defensive against trailing slashes (which are common in URL configurations), consider stripping any trailing slashes from host_base_url before performing the lookup in _ENGINE_BASE_URL_MAPPING and _COMPOSE_DNS_TARGETS.

    normalized_url = host_base_url.rstrip("/")
    resolved = _ENGINE_BASE_URL_MAPPING.get(normalized_url)
    if resolved is not None:
        return resolved
    # Idempotent pass-through for already-resolved Compose-DNS targets.
    if normalized_url in _COMPOSE_DNS_TARGETS:
        return normalized_url

[gemini-code-assist]

Using sudo -n (non-interactive) is a best practice for scripts that may run in non-interactive environments (such as CI/CD pipelines or automated provisioning tasks). It prevents the script from hanging indefinitely if passwordless sudo is not configured for the user, failing immediately and allowing the warning message to be printed.

Suggested change

	chown 8983:8983 ./data/solr 2>/dev/null \
	|| sudo chown 8983:8983 ./data/solr 2>/dev/null \
	chown 8983:8983 ./data/solr 2>/dev/null \
	|| sudo -n chown 8983:8983 ./data/solr 2>/dev/null \

[SoundMindsAI]

Review adjudication (Gemini Code Assist)

Fix commit: latest push (sudo -n).

#	Sev	Location	Verdict	Notes
1	Medium	scripts/install.sh:339	Accepted	Fixed — sudo → sudo -n in the Solr data-dir chown fallback. A bare sudo can block on a password prompt in non-interactive installs; -n fails fast and lets the WARN fallback fire.
2	Medium	backend/app/services/demo_seeding.py:469	Deferred	Defensive against an input that doesn't occur. _resolve_engine_base_url only ever receives the demo SCENARIOS' hardcoded host_base_url values + the ES/OS/SOLR module constants — all trailing-slash-free (seed_meaningful_demos.py:93-105). Operator-registered cluster URLs never reach this function (it's demo-seeding-only). The rstrip("/") would also subtly change the returned value for the pass-through branch. Not a regression this PR introduces; declining the speculative hardening.

Outcomes

• Applied (1): sudo -n non-interactive chown.
• Deferred as non-regression (1): trailing-slash normalization (no live trailing-slash path into this resolver).

Ready for merge.