gh-pages deploy failing: artifact crossed Pages publish cap on Oct 26 #3231

Activity

1. leofang

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   P0High priority - Must do!High priority - Must do!

   CI/CDCI/CD infrastructureCI/CD infrastructure

   on Oct 27, 2026
2. leofang commented on Oct 27, 2026

   

   leofang

   on Oct 27, 2026

   Last edited by leofang

   MemberAuthor

   More actions

   Measuring where the growth is coming from, so each option can be sized.

   Per-preview breakdown (sample: pr-preview/pr-3186, 590 MB)

   Bucket	MB	%
   rendered HTML + assets	572.7	97.0%
   .doctrees/	15.9	2.7%
   _sources/	1.7	0.3%

   Published trees (docs/latest/, docs/cuda-bindings/latest/, etc.) already don't ship .doctrees or _sources — those two directories exist only inside PR previews.

   What recently inflated the previews

   Each preview's cuda-bindings/latest/module/generated/ has two cohorts across the 37 current previews:

   • baseline — 691 autosummary files, ~100 MB
   • inflated — 1603 autosummary files, ~488 MB

   11 of 37 previews are already in the inflated cohort; the remaining 26 are baseline because they branched before the inflection. Diffing the two sets, all 912 new pages are cuda.bindings._v2.driver.* autosummary pages (e.g. driver_v2/cuda.bindings._v2.driver.AccessPolicyWindow_v1.html). As in-flight PRs rebase past the v2-driver landing on main, each one will pick up ~390 MB extra. Any one-shot trim gets eaten back as rebases roll in.

   Published docs/cuda-bindings/latest/ is already inflated too, which is why the main cuda-bindings/ bucket sits at 2.25 GB.

   Options

   1. Drop .doctrees + _sources from preview uploads. Saves ~17.6 MB per preview (≈ 650 MB uncompressed across 37, ~100–150 MB compressed). Low effort, buys this week, doesn't address the growth driver.
   2. Skip the v2 autosummary expansion in preview builds only. Previews link to the published cuda-bindings/latest/module/. Saves the full ~390 MB per inflated preview — 100% of the recent growth. Published docs keep the full API reference.
   3. Only build the subproject the PR touches. The build workflow inspects changed paths and skips subprojects the PR doesn't touch. Many PRs drop to a cuda-core-only or cuda-pathfinder-only preview (~45 MB).
   4. Trigger cleanup-pr-previews.yml manually to sweep the 5 orphan closed-PR previews ahead of the 04:00 UTC nightly run. Small, immediate.
   5. Longer-term: move previews off gh-pages. Upload each per-PR docset as a workflow artifact (10 GB cap, 90-day retention) and post the link on the PR. Removes previews from the Pages site budget entirely. xref: CI: Reduce traffic and slow down size-bloating of gh-pages #3014

   Suggested sequencing: 4 now to recover the smallest increment, 1 + 2 together as the short-term durable fix, 5 as the long-term direction. Option 3 is complementary to any of these.

   CTK-tagged archive pruning is intentionally not on this list — all 26 tags combined are ~100 MB, well under 2% of the site, and a non-goal to touch.

   -- Leo's bot
3. leofang commented on Oct 27, 2026

   

   leofang

   on Oct 27, 2026

   MemberAuthor

   More actions

   • Skip the v2 autosummary expansion in preview builds only. Previews link to the published cuda-bindings/latest/module/. Saves the full ~390 MB per inflated preview — 100% of the recent growth. Published docs keep the full API reference.

   @mdboom WDYT?
4. leofang commented on Oct 27, 2026

   

   leofang

   on Oct 27, 2026

   MemberAuthor

   More actions

   • Trigger cleanup-pr-previews.yml manually to sweep the 5 orphan closed-PR previews ahead of the 04:00 UTC nightly run. Small, immediate.

   xref: https://github.com/NVIDIA/cuda-python/actions/runs/40424476295
5. leofang

   added a commit that references this issue on Oct 27, 2026

   Purge orphaned cuda-core pre-release doc trees

   ...

   30f5f0a
6. mdboom

   self-assigned this

   on Oct 27, 2026
7. mdboom commented on Oct 27, 2026

   

   mdboom

   on Oct 27, 2026

   Last edited by mdboom

   Contributor

   More actions

   • Skip the v2 autosummary expansion in preview builds only. Previews link to the published cuda-bindings/latest/module/. Saves the full ~390 MB per inflated preview — 100% of the recent growth. Published docs keep the full API reference.

   @mdboom WDYT?

   #3095 added the _v2 driver module and its associated docs. It's both (a) more extensively documented than the existing one and (b) our biggest module by far. But I definitely wasn't aware of just /how much/ of an impact it had.

   I think option #4 right now, (which @leofang already did), #2 today, with a plan to get to option #5 later is the right approach. #1 seems to be low benefit and #3 is very complex for the benefit we would get.
8. leofang commented on Oct 29, 2026

   

   leofang

   on Oct 29, 2026

   MemberAuthor

   More actions

   This still happens after #3244 was merged.
9. leofang

   added 4 commits that reference this issue on Oct 29, 2026

   Purge gh-pages trees with no matching git tag

   ...

   7d206dd

   Skip docs deploy on tag CI runs

   ...

   3fd6462

   Restore docs/cuda-bindings/13.4.4 to its v13.4.4 release content

   ...

   c6b0d90

   Empty docs/pr-preview to bring Pages site back under the 1 GB cap

   ...

   0e116d6
10. leofang commented on Oct 30, 2026

    

    leofang

    on Oct 30, 2026

    MemberAuthor

    More actions

    gh-pages corrective commits during this incident

    commit	summary	uncompressed delta
    30f5f0a	Purge orphaned cuda-core pre-release doc trees (26 X.Y.Z.devN dirs)	-664.5 MB
    7d206dd	Purge gh-pages trees with no matching git tag (3 top-level 13.1.2/13.2.1/13.3.2, 4 cuda-bindings/{13.1.2, 13.2.1, 13.3.2, 13.4.0b1}, cuda-pathfinder/1.5.7)	-413.5 MB
    c6b0d90	Restore docs/cuda-bindings/13.4.4/ to its real v13.4.4 release content, undoing the overwrite caused by the cuda-pathfinder-v1.8.5 tag CI deploy	-406 MB
    0e116d6	Empty docs/pr-preview/ (41 preview trees) to bring the site back under the 1 GB cap	-3.3+ GB

    Net: docs/ tree dropped from ~7 GB uncompressed to ~2.7 GB. At the ~5× sphinx-HTML compression ratio, the published artifact should land around ~550 MB.

    Side effects:

    • All 41 open PR preview links in sticky comments are temporarily dead. Each PR's next push will re-build its preview under docs/pr-preview/pr-N/ through the normal doc_preview action; the sticky comment will refresh at that point.
    • Historical doc trees under cuda-core, cuda-bindings, and cuda-pathfinder are untouched; version switcher behavior is unchanged.

    The underlying leak that caused the orphan accumulation and the Oct 29 cuda-bindings/13.4.4 overwrite is not fixed by these commits — it is addressed in #3232 (pending review). Until that lands, each subsequent tag push on any subproject will re-leak at least one directory via the same mechanism.

    -- Leo's bot
11. leofang

    added a commit that references this issue on Oct 30, 2026

    Skip docs deploy on tag CI runs (#3232)

    ...

    03b4153