Skip to content

Consolidate duplicate bounded-concurrency map helpers in src/queue and src/signals #6602

Description

@JSONbored

Context

src/queue/map-with-concurrency.ts exports the canonical bounded-concurrency worker-pool helper:

export async function mapWithConcurrency<T, R>(items: T[], concurrency: number, mapper: (item: T) => Promise<R>): Promise<R[]>

It is already reused by src/queue/duplicate-detection.ts, src/queue/job-dispatch.ts, and src/queue/processors.ts (imported at src/queue/processors.ts:369 and re-exported at :370).

Two other files under this same subsystem independently re-implement the identical algorithm instead of importing it:

  • src/signals/focus-manifest-loader.ts:239 defines its own exported mapWithConcurrencyLimit<T, U>(items, limit, mapper) — same worker-pool loop, same behavior, different name.
  • src/queue/patchless-secret-scan.ts:157 defines its own private mapPatchLessSecretScanFilesWithConcurrency<T, R>(items, limit, mapper) — same loop again.

The drift is directly visible in src/queue/processors.ts, which imports both mapWithConcurrency (line 369, from ./map-with-concurrency) and mapWithConcurrencyLimit (line 509, from ../signals/focus-manifest-loader) side by side and uses each at different call sites (e.g. lines 888 and 1591 use mapWithConcurrencyLimit), even though they do the same thing.

Requirements

  • Replace the body of mapWithConcurrencyLimit in src/signals/focus-manifest-loader.ts (currently lines 239-249) with a thin delegation to mapWithConcurrency imported from ../queue/map-with-concurrency, preserving the exported name mapWithConcurrencyLimit and its existing (items, limit, mapper) call signature so every current call site (including the two in src/queue/processors.ts) keeps compiling and behaving unchanged.
  • Replace the body of mapPatchLessSecretScanFilesWithConcurrency in src/queue/patchless-secret-scan.ts (currently lines 157-173) with a call to mapWithConcurrency imported from ./map-with-concurrency, removing the hand-duplicated loop.
  • This is a pure internal consolidation: do not change the public behavior, call signatures, or export names of any existing caller of either function.
  • After this change, src/queue/map-with-concurrency.ts must be the only file under src/queue/ or src/signals/ that implements the bounded-concurrency worker-pool loop itself — every other consumer calls into it.

Deliverables

  • src/signals/focus-manifest-loader.ts's mapWithConcurrencyLimit delegates to src/queue/map-with-concurrency.ts's mapWithConcurrency
  • src/queue/patchless-secret-scan.ts's mapPatchLessSecretScanFilesWithConcurrency delegates to the same canonical helper
  • No remaining hand-duplicated bounded-concurrency loop in either file

Test Coverage Requirements

Touches src/queue/map-with-concurrency.ts, src/signals/focus-manifest-loader.ts, and src/queue/patchless-secret-scan.ts — all under src/**, so this repo's Codecov patch gate (99%+ on changed lines) applies. Both delegating wrapper functions are already exercised transitively by their existing callers' test suites (e.g. test/unit/patchless-secret-scan.test.ts, and the focus-manifest-loader/processors tests that already drive mapWithConcurrencyLimit), so no new test file should be required — just confirm the existing suites still pass green after the delegation.

Expected Outcome

mapWithConcurrency in src/queue/map-with-concurrency.ts is the single implementation of bounded-concurrency fan-out under src/queue/ and src/signals/; the two other hand-duplicated copies that could silently drift apart are gone.

Links & Resources

  • src/queue/map-with-concurrency.ts (canonical implementation)
  • src/signals/focus-manifest-loader.ts:239-249 (mapWithConcurrencyLimit)
  • src/queue/patchless-secret-scan.ts:157-173 (mapPatchLessSecretScanFilesWithConcurrency)
  • src/queue/processors.ts:369-370,509,888,1591 (imports/uses both today)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions