Mount pods fail with WIF (Workload Identity Federation) errors even though service account JSON credentials are configured in the secret

[agopalan87]

Running JuiceFS CSI Driver (v0.30.0) on GKE with GCS backend. Mount pods fail with WIF (Workload Identity Federation) errors even though service account JSON credentials are configured in the secret. What is that we are missing? For some reason the google-application-credentials file is not found in the jfs-mount container pod.

• GKE cluster
• JuiceFS CSI Driver v0.30.0
• JuiceFS CE v1.3.0
• Backend: Google Cloud Storage (gs://)
• Auth: Service Account JSON (not WIF)

Secret:

kubectl create secret generic juicefs-secret \
  --from-file=google-application-credentials=juicefs-gcs-writer-key.json \
  --from-literal=metaurl=redis://<redis_up>:6379/1 \
  --from-literal=storage=gs \
  --from-literal=bucket=gs://juicefs-demo \
  --from-literal=name=demo-fs

# juicefs-storage.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
provisioner: csi.juicefs.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
  csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: demo
  csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
  csi.storage.k8s.io/node-publish-secret-namespace: demo
  csi.storage.k8s.io/controller-publish-secret-name: juicefs-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: demo
  csi.storage.k8s.io/node-stage-secret-name: juicefs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: demo
  envs: |
    GOOGLE_APPLICATION_CREDENTIALS=/root/.juicefs/google-application-credentials
    GOOGLE_CLOUD_PROJECT=wsp-services

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: busybox-jfs-pvc
  namespace: demo
spec:
  accessModes:
    - ReadWriteMany  # Essential for shared access
  resources:
    requests:
      storage: 10Gi # This is just a placeholder; storage is virtually unlimited (GCS)
  storageClassName: juicefs-sc

apiVersion: v1
kind: Pod
metadata:
  name: busybox-juicefs-test
  namespace: demo
spec:
  containers:
  - name: busybox-client
    image: busybox:latest
    # Keep the container running indefinitely
    command: [ "sh", "-c", "while true; do sleep 3600; done" ]
    securityContext:
      privileged: true
    volumeMounts:
    - name: juicefs-volume
      mountPath: /mnt/jfs

  volumes:
  - name: juicefs-volume
    persistentVolumeClaim:
      claimName: busybox-jfs-pvc

The pod runs successfully but when I try to write to the /mnt/jfs it just waits indefinitely and gives the following error in jfs-mount container in kube-system namespace.

Error:
compute: Received 400 `Unable to fetch federated token; The Global STS returned 400 Bad Request: {"error":"invalid_target","error_description":"The target service indicated by the "audience" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}

[jiefenghuang]

@zxh326

[zxh326]

You need to create a separate secret containing credentials, for example, and then use config to mount it into the mountpod.

apiVersion: v1
kind: Secret
metadata:
  name:gc-secret
type: Opaque
stringData:
  google-application-credentials: |
   ...
---
apiVersion: v1
kind: Secret
metadata:
  name: juicefs-secret
type: Opaque
stringData:
  ...
  # Set Secret name and mount directory in configs. This mounts the whole Secret into the specified directory
  configs: "{gc-secret: /root/.config/gcloud}"

documentation:

https://juicefs.com/docs/csi/guide/configurations/#via-secret-1

[agopalan87]

Thanks much! That worked!