Skip to content

fix: replace MinIO with RustFS so chart installs pass again - #848

Merged
claudiacodacy merged 5 commits into
masterfrom
fix/replace-minio-with-rustfs
Oct 30, 2026
Merged

claudiacodacy merged 5 commits into
masterfrom
fix/replace-minio-with-rustfs

Conversation

@claudiacodacy

Copy link
Copy Markdown
Contributor

Why

Every chart PR off master currently fails install-k8s-unstable: the bundled MinIO pod never becomes ready, Helm hits its 20m timeout and rolls back. Cause: the minio 5.0.33 subchart pins minio/minio:RELEASE.2020-06-14T18-32-17Z and minio/mc:RELEASE.2020-05-28T23-43-36Z, and MinIO's public images are no longer pullable from Docker Hub or Quay. install-k8s-unstable is a required check, so nothing can merge to master until this is fixed (e.g. #847).

release-17.0.0 (#833) already fixes this by switching to RustFS, and its installs pass on k8s 1.30–1.33. #833 is held up by its QA suites, so this PR takes only the object-store swap and lands it on master on its own.

What

Cherry-picked from release-17.0.0 (authorship kept, -x trailers point to the originals):

  • ce0c597 Replace MinIO with RustFS as the bundled object store
  • 92920b5 Register the rustfs Helm repo in setup_helm_repos
  • 942faec Upgrade the helm-push plugin to v0.10.4 / cm-push (the old plugin can't push a v1 chart that has a v2 subchart)
  • 7551652 Replace the old built-in Helm in the .do deploy jobs

Left out on purpose:

Checked locally (Helm v3.19.0)

  • helm dep build resolves rustfs-1.0.3; helm lint passes.
  • helm template with values.yaml, values-production.yaml and values-microk8s.yaml: no MinIO resources; RustFS Deployment, Service (codacy-rustfs-svc), PVCs, Secret and bucket Job render, and global.minio.location resolves to codacy-rustfs-svc.
  • rustfs/rustfs:1.0.3, amazon/aws-cli:2.17.0 and busybox:stable all pull from Docker Hub.

CI on this PR is the real install check.

For reviewers

  • Upgrades: existing installs keep their data on the codacy-minio PVC. This PR doesn't move that data to RustFS. That's the same as release: prepare 17.0.0 #833, but on master it reaches unstable sooner.
  • extract-codacy-logs.sh still targets MinIO's pod label and paths (noted in ce0c597).

🤖 Generated with Claude Code

Ngwerume and others added 4 commits October 30, 2026 10:59
MinIO's upstream images are no longer published on Docker Hub/Quay, so
the bundled minio dependency (charts.helm.sh/stable, 5.0.33) can no
longer pull. Swap it for RustFS (charts.rustfs.com, 1.0.3), an
S3-compatible, Apache-2.0 object store, as a first-class chart
dependency: one `helm install/upgrade codacy` still brings up
everything, no separate release and no more minio dependency at all.

- codacy/requirements.yaml, requirements-dev.yaml, requirements.lock:
  drop the minio dependency, add rustfs under the same
  global.minio.create condition so existing toggles keep working.
- codacy/values.yaml: global.minio.* keeps its name (every subchart -
  engine, api, listener, fluentd - reads global.minio.location/port/
  accessKey/secretKey, so renaming would mean touching those private
  repos too) but global.minio.location now points at codacy-rustfs-svc.
  The old minio: subchart block is replaced by a rustfs: block
  (standalone mode, single replica, cluster-default storage class,
  same 40Gi size, same example access/secret keys) including a
  post-install/post-upgrade bucket-creation Job that replaces MinIO's
  own bucket hook.
- codacy/values-production.yaml, .circleci/config.yml, .do/values.yaml:
  translate the old minio persistence/resources overrides to the
  equivalent rustfs storageclass/resources keys.
- .do/Makefile: the deployment filter used during CI teardown now
  excludes "rustfs" instead of "minio" (RustFS renders as a Deployment
  under that name).
- README.md, docs/: MinIO references in prose, parameter docs and
  example output updated to RustFS.

Verified with `helm dependency update` + `helm template` against
values-production.yaml: zero "# Source: .../minio/..." lines, RustFS
Service/Deployment/PVC/Secret/bucket-job all render, and the rendered
RustFS secret decodes to the same access/secret key values configured
under global.minio.*.

Not changed: docs/troubleshoot/extract-codacy-logs.sh still selects
`-l app=minio` and copies from /export/logs, which was MinIO's flat
filesystem layout. RustFS mounts object data under /data instead and
its on-disk bucket layout isn't confirmed, so this script is left
broken on purpose pending a follow-up pass (likely rewritten to pull
the logs bucket over the S3 API instead of kubectl cp).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit ce0c597)
helm dependency update/build failed with "no repository definition
for https://charts.rustfs.com" because setup_helm_repos only ever
added the now-unused legacy stable repo (charts.helm.sh/stable),
which nothing in requirements.yaml references anymore after the
MinIO -> RustFS swap. Swap that repo add for rustfs so CircleCI
(which calls `make setup_helm_repos`) and anyone else following the
normal flow can resolve the new dependency.

Verified with a clean `helm repo remove rustfs/stable` +
`make setup_helm_repos` + `helm dependency update codacy/`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit 92920b5)
Root cause confirmed via chartmuseum/helm-push#93: the bundled
helm-push 0.9.0 plugin loads an apiVersion v1 parent chart (ours)
using its own bundled Helm 2 library, which cannot read v2 subcharts
nested inside it (rustfs-1.0.3). It fails before anything reaches
ChartMuseum - matching the "error unpacking rustfs-1.0.3.tgz ...
apiVersion 'v2' is not valid" error from both prior CI runs. Neither
the system helm binary (confirmed v3.2.1) nor the plugin being merely
"old" (confirmed v0.9.0, already past the v0.8.0 Helm-3-client
threshold) was the actual issue - it's this specific, previously
unfixed bug.

Fixed upstream in helm-push v0.10.0 (Oct 2021), which also renamed the
plugin/command to cm-push (to avoid colliding with Helm 3.8+'s native
OCI `helm push`). Install v0.10.4 (the last release built against
Helm 3, matching this job's helm v3.2.1) in the helm_push job itself -
no codacy/ci-base image rebuild needed - and switch the push
invocation to `helm cm-push`. Shared by both helm_push_incubator and
helm_push_unstable via the helm_push step anchor.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit 942faec)
Error: UPGRADE FAILED: parse error at
(codacy/charts/rustfs/templates/_helpers.tpl:151): unclosed action

codacy/ci-do:2.8.0 - the image behind deploy_to_doks_release and every
other job using the default_doks_image anchor (validate/plan/create_
dev_cluster, setup_sandbox_db_values, setup_release_db_values, deploy_
to_doks_dev, deploy_to_doks_sandbox) - bakes in helm v3.2.1 (2020,
go1.13.10). That binary's Go template parser can't read the rustfs
chart's multi-line `{{- $mtls := dict ... }}` action in its
_helpers.tpl. Confirmed by pulling the actual image and running its
helm directly against the vendored rustfs-1.0.3.tgz: reproduces the
exact error. A current helm (v3.19.0 and v4.2.4 both tested) parses
the same chart with no error, so this is purely a client-version
limitation on a valid, modern chart - same root category as the
helm-push plugin issue already fixed, but this is a separate,
unrelated code path (a direct `helm upgrade --install`, no plugin
involved) that needed its own fix.

Added ensure_modern_helm, which replaces the binary with a current
v3.19.0 build (same release line already trusted elsewhere in this
pipeline, for the microk8s_install jobs) before anything is installed.
Made it a prerequisite of helm_install_infrastructure, helm_install_
from_local_dir and helm_install_from_chart_museum, so every one of the
.do/Makefile-driven deploy paths is covered, not just the release one.

Verified: curl/tar/mv are present in the real ci-do:2.8.0 image and
the download/extract steps succeed against it; a genuine v3.19.0
binary (tested natively, no emulation) renders the vendored rustfs
chart with exit 0; `make -n helm_install_from_chart_museum` shows the
new step correctly ordered before the helm upgrade call.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit 7551652)
@codacy-production

codacy-production Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

Codacy flagged the make-buckets container (3 high, Semgrep k8s rules):
it ran as root, with a writable root filesystem and privilege
escalation allowed. Run the pod as non-root (uid/gid 1000), drop all
capabilities, disallow privilege escalation and make the root
filesystem read-only. The AWS CLI gets a writable emptyDir at /tmp as
HOME.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claudiacodacy
claudiacodacy marked this pull request as ready for review October 30, 2026 11:17

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

⚠️ Resolve issues before merging

Committed object-store credentials must be removed before merging. The chart should obtain them from Kubernetes Secrets or installation-time values and reuse one source for all consumers. Core dependency, rendering, installation, lifecycle, and publishing scenarios are not covered by the submitted changes and should be validated.

Test suggestions

  • Build dependencies and lint/render the chart using the default, production, and microk8s values files.
  • Verify the rendered chart contains RustFS resources and no MinIO workload resources, with consumers pointing to the RustFS service.
  • Install or upgrade the chart and verify RustFS becomes ready, buckets are created, and lifecycle configuration is applied.
  • Verify the deployment path upgrades Helm before installing local or chart-museum releases.
  • Verify chart publishing installs the modern plugin and successfully pushes a v1 parent chart containing the RustFS v2 dependency.
  • Verify Fluentd can write logs to the RustFS S3-compatible endpoint using path-style addressing.
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Build dependencies and lint/render the chart using the default, production, and microk8s values files.
2. Verify the rendered chart contains RustFS resources and no MinIO workload resources, with consumers pointing to the RustFS service.
3. Install or upgrade the chart and verify RustFS becomes ready, buckets are created, and lifecycle configuration is applied.
4. Verify the deployment path upgrades Helm before installing local or chart-museum releases.
5. Verify chart publishing installs the modern plugin and successfully pushes a v1 parent chart containing the RustFS v2 dependency.
6. Verify Fluentd can write logs to the RustFS S3-compatible endpoint using path-style addressing.
Low confidence findings
  • Document and validate the upgrade behavior for existing MinIO data, since replacing the backend may leave prior data inaccessible and start RustFS with an empty store.

TIP How was this review? Give us feedback

Comment thread codacy/values.yaml Resolved
@claudiacodacy
claudiacodacy merged commit e5a31d2 into master Oct 30, 2026
9 checks passed
@claudiacodacy
claudiacodacy deleted the fix/replace-minio-with-rustfs branch October 30, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants