rpc error: code = Unknown desc = unexpected failure invoking barman-cloud-wal-archive: exit status 2

[alexeyshishkin-edb]

Hi, Team !

I have two setups:

1. native barman
2. plugin barman 0.6.0

Both of them use the same CNPG 1.27.0 and the same Minio configuration:

tee minio.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: minio-creds
data:
  ACCESS_KEY_ID: $ACCESS_KEY_ID
  ACCESS_SECRET_KEY: $ACCESS_SECRET_KEY

---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        image: minio/minio:latest
        args:
        - server
        - /data
        env:
        - name: MINIO_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: minio-creds
              key: ACCESS_KEY_ID
        - name: MINIO_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: minio-creds
              key: ACCESS_SECRET_KEY
        ports:
        - containerPort: 9000
        readinessProbe:
          httpGet:
            path: /minio/health/ready
            port: 9000
          initialDelaySeconds: 30
        livenessProbe:
          httpGet:
            path: /minio/health/live
            port: 9000
          initialDelaySeconds: 30
EOF

Setup 1) works fine, while setup 2) not - the following error is observed in the 'kubectl describe cluster $cluster_name' output:

'rpc error: code = Unknown desc = unexpected failure invoking barman-cloud-wal-archive: exit status 2'

ObjectStore configuration is:

tee minio-store.yaml << EOF
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: minio-store
spec:
  configuration:
    destinationPath: s3://cluster-backups/
    endpointURL: http://minio:9000
    s3Credentials:
      accessKeyId:
        name: minio-creds
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: minio-creds
        key: ACCESS_SECRET_KEY
    wal:
      compression: gzip
  retentionPolicy: "1d"
EOF

Cluster configuration is:

tee $cluster_name.yaml << EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: $cluster_name
spec:
  instances: 3
  imageName: $cnpg_image_name
  storage:
    size: 1Gi
  plugins:
  - name: barman-cloud.cloudnative-pg.io
    isWALArchiver: true
    parameters:
      barmanObjectName: minio-store
EOF

Report files of the 'kubectl cnpg report operator -n $oname-system --logs' and 'kubectl cnpg report cluster $cluster_name --logs' are attached.

report_cluster_cluster-barman-cloud-cnpg_2025-09-18T095613Z.zip
report_operator_2025-09-18T095559Z.zip

[dosubot]

Hi @alexeyshishkin-edb! I'm Dosu and I’m helping the plugin-barman-cloud team.

This error is a known issue in plugin-barman-cloud v0.6.0 when the ObjectStore and the credentials Secret have the same name. To fix it, rename either the ObjectStore or the Secret so their names are different (for example, use minio-store for the ObjectStore and minio-creds for the Secret). This resolves a cache conflict in the plugin that causes WAL archival to fail with a generic RPC error like the one you're seeing. Make sure both resources are in the same namespace as your cluster. More details here: plugin-barman-cloud issue #502.

If the issue persists after renaming, check pod resource limits and events for OOMKilled signals, and verify Minio health and network connectivity during backup attempts, as resource constraints or connectivity problems can also cause similar failures source.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[alexeyshishkin-edb]

@dosu, as you may see from above I already use minio-store for the ObjectStore and minio-creds for the Secret.

[dosubot]

You're right—the known cache bug with same-named ObjectStore and Secret doesn't apply here. In your setup, the most common causes for exit status 2 with plugin-barman-cloud v0.6.0 and MinIO are resource constraints (like OOMKilled pods), S3/MinIO compatibility issues, or subtle misconfigurations.

Here’s a focused checklist to help narrow it down:

• Double-check your ObjectStore and Secret config for typos, especially in destinationPath and endpointURL (even a small typo can break archiving) [example].
• For MinIO, set these environment variables in your ObjectStore’s instanceSidecarConfiguration:

  instanceSidecarConfiguration:
    env:
      - name: AWS_REQUEST_CHECKSUM_CALCULATION
        value: when_required
      - name: AWS_RESPONSE_CHECKSUM_VALIDATION
        value: when_required

  This avoids S3 header errors and silent failures [docs].
• Consider adding region: us-east-1 and usePathStyle: true to the ObjectStore config for better MinIO compatibility [docs].
• Check pod events for OOMKilled or resource limit signals (kubectl describe pod ... and kubectl top pod ...). If you see memory or CPU exhaustion, increase the sidecar’s resource limits [resource advice].
• Review the pod logs for errors like MissingContentLength, x-amz-content-sha256 must be UNSIGNED-PAYLOAD, or network/connectivity failures.
• If you recently changed ObjectStore or Secret, make sure the manifests were reapplied and the cluster reconciled.
• As a last resort, exec into the pod and manually run a barman-cloud-wal-archive or barman-cloud-backup command to see if you get a more specific error.

If you try these and still hit the same error, sharing the relevant pod log excerpts (especially from the plugin sidecar) will help pinpoint the root cause.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[alexeyshishkin-edb]

@dosu, as you may see from above I use the same configuration of CNPG and Minio for both setups and only barman plugin setup fails

[dosubot]

This is a common pain point with plugin-barman-cloud: even with identical MinIO and CNPG configs, the plugin can fail where native barman works because the plugin builds its environment variables and CLI args programmatically from the ObjectStore CRD and secrets, not from the pod or shell environment. This means S3/MinIO compatibility tweaks (like region, path style, or checksum envs) that work for native barman often need to be explicitly set in the ObjectStore spec for the plugin to see them details.

For MinIO, make sure your ObjectStore spec includes:

• instanceSidecarConfiguration.env with:

  - name: AWS_REQUEST_CHECKSUM_CALCULATION
    value: when_required
  - name: AWS_RESPONSE_CHECKSUM_VALIDATION
    value: when_required

• region: us-east-1 and usePathStyle: true in the configuration block if your MinIO setup needs them.
• The endpointURL should match the MinIO Tenant service (e.g., <tenant>-hl:9000), not a deprecated gateway see updated docs.

If you’ve already done all this, try running barman-cloud-wal-archive manually inside the plugin sidecar pod with the same env vars and see if it succeeds—if it does, the issue is almost certainly with how the plugin propagates env/config example.

You can also enable debug logging in the plugin sidecar by adding:

instanceSidecarConfiguration:
  additionalContainerArgs:
    - --log-level=debug

to your ObjectStore spec reference.

If you see OOMKilled or resource exhaustion in pod events, increase the sidecar’s memory/CPU limits resource advice.

If you share the plugin sidecar logs from a failed archive attempt, I can help pinpoint the root cause.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[alexeyshishkin-edb]

@dosu, I have provided particular files, but in return I have general advices, is it possible for you to use data-driven approach for your your answers ? if not, I will have to use the other way for troubleshooting

[dosubot]

I can't directly analyze attached files, but I can absolutely help you take a data-driven approach if you extract and share specific log excerpts or error messages from those reports. The most useful details are:

• Plugin sidecar logs around the time of the failure, especially lines mentioning barman-cloud-wal-archive, signal: killed, or S3/MinIO errors like MissingContentLength or x-amz-content-sha256 must be UNSIGNED-PAYLOAD (example).
• Pod events showing OOMKilled or resource exhaustion (example).
• The exact CLI invocation and environment variables used by the plugin (these are logged at info/debug level).
• Any stacktrace or error code details (e.g. exit status 2, 4, etc).

You can grep for keywords like error, OOMKilled, barman-cloud-wal-archive, or signal: killed in the logs inside your report files. If you paste those excerpts here, I’ll help you interpret them and pinpoint the root cause—whether it’s resource limits, S3/MinIO compatibility, or a plugin bug (reference, reference).

If you want to go deeper, enabling debug logging in the plugin sidecar (--log-level=debug) will surface even more actionable details (reference).

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[alexeyshishkin-edb]

the issue is solved by changing endpointURL value in minio-store.yaml file from http://minio:9000 to http://minio-service:9000

pls. close the ticket.