Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions migrations/0148_ams_signals.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- Gittensory AMS (#5681) — central telemetry collector store, mirroring orb_signals' pattern for the miner
-- product. Receives anonymized PR-outcome batches from opt-in AMS instances (orb-export.js). repo_hash and
-- pr_hash are HMAC-anonymized by the sender before this table ever sees them — no repo names, owner
-- identifiers, or PR content is stored here. A separate table from orb_signals rather than a shared
-- discriminator column: AMS has no gate_verdict/reversal_flag concept (a miner submission isn't gated the
-- way a reviewed PR is), so forcing both products into one row shape would mean a pile of always-null
-- Orb-only columns on every AMS row.
CREATE TABLE IF NOT EXISTS ams_signals (
id INTEGER PRIMARY KEY,
instance_id TEXT NOT NULL,
repo_hash TEXT NOT NULL,
pr_hash TEXT NOT NULL,
decision TEXT NOT NULL CHECK (decision IN ('merged', 'closed')),
reason_bucket TEXT,
closed_at TEXT,
received_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE (instance_id, pr_hash)
);
CREATE INDEX IF NOT EXISTS ams_signals_instance ON ams_signals (instance_id, received_at);
14 changes: 14 additions & 0 deletions migrations/0149_ams_instances.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- Gittensory AMS (#5681) — instance registration gate, mirroring orb_instances (see that table's own
-- migration for the full trust-model rationale). Every AMS instance that POSTs an anonymized batch to
-- /v1/ams/ingest is recorded here on first contact, but signals only count toward any future AMS-side
-- aggregate until an operator explicitly registers it (registered=1) — same das-github-mirror-modeled
-- trust anchor Orb already uses, so a stranger can't move an aggregate until a human opts them in.
CREATE TABLE IF NOT EXISTS ams_instances (
instance_id TEXT PRIMARY KEY NOT NULL,
registered INTEGER NOT NULL DEFAULT 0,
first_seen_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_seen_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
registered_at TEXT
);

CREATE INDEX IF NOT EXISTS ams_instances_registered_idx ON ams_instances(registered);
2 changes: 1 addition & 1 deletion packages/gittensory-miner/bin/gittensory-miner.js
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ if (cliArgs[0] === "queue") {
}

if (cliArgs[0] === "orb" && cliArgs[1] === "export") {
process.exit(runOrbExportCli(cliArgs.slice(2)));
process.exit(await runOrbExportCli(cliArgs.slice(2)));
}

if (cliArgs[0] === "claim") {
Expand Down
2 changes: 2 additions & 0 deletions packages/gittensory-miner/docs/env-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Generated by `npm run miner:env-reference`. Do not edit manually.

| Name | First reference | Default |
| --- | --- | --- |
| `GITTENSORY_MINER_AMS_COLLECTOR_TOKEN` | `lib/orb-export.js` | `""` |
| `GITTENSORY_MINER_AMS_COLLECTOR_URL` | `lib/orb-export.js` | `""` |
| `GITTENSORY_MINER_AMS_POLICY_PATH` | `lib/ams-policy.js` | (none) |
| `GITTENSORY_MINER_ATTEMPT_LOG_DB` | `lib/attempt-log.js` | (none) |
| `GITTENSORY_MINER_CLAIM_LEDGER_DB` | `lib/claim-ledger.js` | (none) |
Expand Down
29 changes: 29 additions & 0 deletions packages/gittensory-miner/docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,32 @@ Then point your own `prometheus.yml` at node_exporter as usual — no changes to
are needed. See [`prometheus/rules/alerts.yml`](../../../prometheus/rules/alerts.yml)'s
`gittensory-miner-prediction` / `gittensory-miner-portfolio-queue` / `gittensory-miner-governor` rule groups for
alert rules that already target these exact metric names.

## Anonymized central telemetry (opt-in, off by default)

Everything above stays entirely on your own machine. Separately, the miner can send a small, anonymized batch
of its own PR-outcome history to gittensory's hosted AMS collector — the same fleet-growth/usage telemetry Orb's
self-host collector already sends for maintainers, mirrored for contributors:

```sh
gittensory-miner orb export --enable --send
```

- **`--enable`** alone only builds and prints the anonymized batch locally — no network call, so you can inspect
exactly what would be sent before ever transmitting anything.
- **`--enable --send`** additionally POSTs that batch to the collector and advances a local cursor, so the next
run only sends events since the last successful send.

**What's sent:** for each of your own resolved PRs — an HMAC-anonymized repo hash and PR hash (a per-instance
secret generated once and kept only on your machine; the collector never holds it and can't reverse the hash), the
`merged`/`closed` decision, a fixed low-cardinality rejection-reason bucket, and the close timestamp. No repo
names, PR numbers, diffs, code, or free text ever leave your machine.

**Nothing is sent unless you explicitly opt in.** There is no default-on behavior here (unlike Orb's own
maintainer-side collector) — every invocation requires `--enable --send` explicitly.

| Variable | Purpose |
| --- | --- |
| `GITTENSORY_MINER_AMS_COLLECTOR_URL` | Override the collector endpoint (default: gittensory's hosted collector). |
| `GITTENSORY_MINER_AMS_COLLECTOR_TOKEN` | Optional bearer credential, only needed if your collector requires one. |
| `GITTENSORY_MINER_ORB_EXPORT_DB` | Override the local secret+cursor store path (default: `orb-export.sqlite3` under `GITTENSORY_MINER_CONFIG_DIR`). |
2 changes: 1 addition & 1 deletion packages/gittensory-miner/lib/cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ export function printHelp(input) {
" gittensory-miner hooks check --tool <name> --input <json> [--json]",
" gittensory-miner state get <owner/repo> [--json]",
" gittensory-miner state set <owner/repo> <idle|discovering|planning|preparing> [--dry-run] [--json]",
" gittensory-miner orb export [--enable] [--dry-run] [--json] Build the opt-in anonymized telemetry batch",
" gittensory-miner orb export [--enable] [--send] [--dry-run] [--json] Build (and optionally send) the opt-in anonymized telemetry batch",
" gittensory-miner purge --repo <owner/repo> [--dry-run] [--json]",
" Right-to-be-forgotten: delete a repo's rows from every local store",
"",
Expand Down
32 changes: 30 additions & 2 deletions packages/gittensory-miner/lib/orb-export.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ export interface OrbExportStore {
close(): void;
}

/** Result of sending a batch to the AMS collector — `error` present only on a non-2xx response, a network
* failure, or an empty batch (never thrown). */
export type AmsExportSendResult = { sent: number; error?: string };

/** A pr_outcome record as produced by `readPrOutcomes` (the local ledger's latest-per-PR reduction). */
export type OrbExportOutcome = NormalizedPrOutcomePayload & { repoFullName: string };

Expand All @@ -41,7 +45,25 @@ export function collectOrbExportBatch(options?: {
enabled?: boolean;
}): OrbExportRow[] | null;

export type ParsedOrbExportArgs = { json: boolean; enable: boolean; dryRun: boolean } | { error: string };
export function amsInstanceId(secret: string): string;

export function filterBatchSinceCursor(batch: OrbExportRow[], cursor: string | null): OrbExportRow[];

export function latestClosedAt(batch: OrbExportRow[]): string | null;

export const DEFAULT_AMS_COLLECTOR_URL: string;

export function resolveAmsCollectorUrl(env?: Record<string, string | undefined>): string;

export function sendAmsExportBatch(options: {
batch: OrbExportRow[];
secret: string;
collectorUrl?: string;
collectorToken?: string | undefined;
fetchFn?: typeof fetch;
}): Promise<AmsExportSendResult>;

export type ParsedOrbExportArgs = { json: boolean; enable: boolean; send: boolean; dryRun: boolean } | { error: string };

export function parseOrbExportArgs(args: string[]): ParsedOrbExportArgs;

Expand All @@ -50,5 +72,11 @@ export function runOrbExportCli(
options?: {
openOrbExportStore?: () => OrbExportStore;
initEventLedger?: () => PrOutcomeLedgerReader;
sendAmsExportBatch?: (options: {
batch: OrbExportRow[];
secret: string;
collectorToken?: string | undefined;
}) => Promise<AmsExportSendResult>;
env?: Record<string, string | undefined>;
},
): number;
): Promise<number>;
Loading
Loading