Onyxia Community Call

Vault improvement

Current behaviour

• Onyxia-ui ask a personal vault token for a user and inject the token in VAULT_TOKEN env var.

• /opt/onyxia-init.sh use this vault token to inject the content of a secret in environment variables.

• Some drawbacks:

  • vault token expired and pod can restart after expiration
  • in onyxia group, another user could steal the vault token and get access to non authorized secret (feature DISABLE_PERSONAL_INFOS_INJECTION_IN_GROUP).

Several solutions

• authenticate service account of the pod launched by the user to get a vault-token scoped to the namespace
• use vault-secrets-operator (that could synchronize a vault secret with a kubernetes secret)

Current work

• based on authentication of service account on vault with its jwt.
• onyxia-init.sh could ask a vault token with the service account of the pod and inject secret
• VAULT_TOKEN=$(vault write -field="token" auth/$VAULT_INJECTION_SA_AUTH_PATH/login role=$VAULT_INJECTION_SA_AUTH_ROLE jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token)
• no more personal informations, security scopes are ok and if pod restart new short vault token is issued

How to

vault auth enable --path=kubernetes-cluster jwt

cat <<EOF | vault write auth/kubernetes-cluster/role/kubernetes-role -
{
  "policies": ["kubernetes-role"],
  "user_claim": "/kubernetes.io/namespace",
  "user_claim_json_pointer": "true",
  "bound_audiences": "https://kubernetes.default.svc.cluster.local",
  "claim_mappings": {
    "/kubernetes.io/namespace": "namespace"
  },
  "role_type": "jwt",
  "ttl": "1h"
}
EOF

path "onyxia-kv/data/{{identity.entity.aliases.auth_jwt_9cdca6c6.name}}/*" {
        capabilities = ["read","list"]
    }

path "onyxia-kv/{{identity.entity.aliases.auth_jwt_9cdca6c6.name}}/*" {
        capabilities = ["read","list"]
    }

path "onyxia-kv/metadata/{{identity.entity.aliases.auth_jwt_9cdca6c6.name}}/*" {
        capabilities = ["read","list"]
    }

Shared s3 spaces with onyxia and minio

Current behaviour

• one bucket per onyxia namespace
• our needs: users need to share some s3 data in read or write groups of users

Number of buckets

• Minio recommendations : do not make a bucket by user.
• then onyxia region allow workingDirectory.bucketMode : shared
• today we have one bucket like s3/home/{username|projectName}

Policy of home space is

    {
      "Version": "2012-10-17",
      "Statement": [
          {
            "Sid": "S3ConsoleAccess",
            "Effect": "Allow",
            "Action": [
              "s3:GetBucketLocation",
              "s3:ListAllMyBuckets"
            ],
            "Resource": ["arn:aws:s3:::home"]
          },
          {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": "arn:aws:s3:::home",
            "Condition": {
                "StringLike": {
                  "s3:prefix": [
                    "user-${jwt:preferred_username}",
                    "user-${jwt:preferred_username}/",
                    "user-${jwt:preferred_username}/*"
                  ]
                }
            }
          },
          {
            "Sid": "ListMultipart",
            "Effect": "Allow",
            "Action": ["s3:ListMultipartUploadParts","s3:ListBucketMultipartUploads","s3:AbortMultipartUpload"],
            "Resource": "arn:aws:s3:::home"
          },
          {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
              "arn:aws:s3:::home/user-${jwt:preferred_username}/",
              "arn:aws:s3:::home/user-${jwt:preferred_username}/*"
            ]
          }
      ]
    }

A creation of a shared space s3:

• let say we want s3/shared/test-space to be opened in read/write to some users.
• 2 groups are created shared-test-r, and shared-test-w (in ldap)
• 1 policy is created shared-test
• the jwt issued by keycloak for onyxia-minio oidc client looks like: {preferred_username:'alexisdondon', groups : [shared-test-r], policy : [home, shared-test]}
• onyxia get an sts token with this jwt

the policy :

{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "shared-test-listobject",
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket"
                ],
                "Resource": "arn:aws:s3:::shared",
                "Condition": {
                    "ForAnyValue:StringEquals": {
                        "jwt:groups": [
                            "shared-test-r",
                            "shared-test-rw"
                        ]
                    },
                    "StringLike": {
                        "s3:prefix": [
                            "test/",
                            "test/*"
                        ]
                    }
                }
            },
            {
                "Sid": "shared-test-read",
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject"
                ],
                "Resource": [
                    "arn:aws:s3:::shared/test/*"
                ],
                "Condition": {
                    "ForAnyValue:StringEquals": {
                        "jwt:groups": [
                            "shared-test-r"
                        ]
                    }
                }
            },
            {
                "Sid": "shared-test-write",
                "Effect": "Allow",
                "Action": [
                    "s3:*"
                ],
                "Resource": [
                    "arn:aws:s3:::shared/test/*"
                ],
                "Condition": {
                    "ForAnyValue:StringEquals": {
                        "jwt:groups": [
                            "shared-test-w"
                        ]
                    }
                }
            }
        ]
    }

number of policies:

• we could have made two policies one for read, and, one for write but minio recommends to reduce number of policies.
• we ’ve tried with 2000 shared spaces and a user with 20 shared spaces average until 100 shared spaces