Storage Initializers¶

Note: Managing secrets directly in Kubernetes is still supported, but the suggested approach is to use Deploy’s secrets management tooling.

Some features in Seldon Deploy require access to external storage providers, for example, to run batch jobs.

A storage initializer mechanism similar to one used on the Pre Packaged model servers is used by Seldon Deploy. By default, Seldon Deploy uses the rclone-based storage initializer. Rclone offers compatibility with over 40 different cloud storage products and is therefore the default choice.

However, custom storage initializers may be used by modifying the storageInitializer values in deploy-values.yaml.

Configuration¶

Rclone storage initializers are configured using environmental variables passed as secrets in Seldon Deploy. These are configured per namespace.

Each remote storage provider needs to be configured using environment variables that follow the following pattern:

RCLONE_CONFIG_<remote name>_<config variable>: <config value>

Note

Multiple remotes can be configured simultaneously. e.g. Both s3 (e.g. AWS S3) and gs (e.g. Google Cloud Storage) remotes can be configured for the same storage initializer.

Once the remote is configured, the modelUri that is compatible with rclone takes the form:

modelUri: <remote>:<bucket name>

For example modelUri: minio:sklearn/iris, or modelUri: gs:seldon-models/cifar10. Rclone will remove the leading slashes for buckets so this is equivalent to minio://sklearn/iris or gs://seldon-models/cifar10.

Below you will find a few example configurations. For other storage providers, please consult the rclone documentation.

Please note that you need the labels secret-type: bucket and seldon-deploy: "true" on your secret for it to be correctly picked up by Deploy. Without these labels it will not show in the secrets dropdown in the deployment creation wizard.

Minio¶

This is an example of how to set up secrets so that Rclone may access minio as configured in the production installation.

Reference: rclone documentation

export MINIONAMESPACE=minio-system
export MINIOUSER=minioadmin
export MINIOPASSWORD=minioadmin
export NAMESPACE=seldon

cat << EOF > seldone-rclone-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: minio-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: minio
  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: ${MINIOUSER}
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: ${MINIOPASSWORD}
  RCLONE_CONFIG_S3_ENDPOINT: http://minio.${MINIONAMESPACE}.svc.cluster.local:9000
EOF

kubectl apply -n ${NAMESPACE} -f minio-bucket.yaml

Public GCS configuration¶

Note

This is configured by default in the seldonio/rclone-storage-initializer image.

Reference: rclone documentation.

apiVersion: v1
kind: Secret
metadata:
  name: gs-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_GS_TYPE: google cloud storage
  RCLONE_CONFIG_GS_ANONYMOUS: "true"

AWS S3 with access key and secret¶

Reference: rclone documentation

apiVersion: v1
kind: Secret
metadata:
  name: s3-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: aws
  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: "<your AWS_ACCESS_KEY_ID here>"
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: "<your AWS_SECRET_ACCESS_KEY here>"

Example AWS S3 with IAM roles configuration¶

Reference: rclone documentation

apiVersion: v1
kind: Secret
metadata:
  name: s3-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: aws
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: ""
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: ""
  RCLONE_CONFIG_S3_ENV_AUTH: "true"

GCP/GKE¶

Reference: rclone documentation

For GCP/GKE, you will need create a service-account key and save it as a local json file.

First make sure that you have a gcloud service account ([SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com) that has sufficient permissions to access the bucket with your models (i.e. Storage Object Admin). You can check this using the gcloud console.

Next, generate keys locally using the gcloud tool. This will create your service-account key file at gcloud-application-credentials.json:

gcloud iam service-accounts keys create gcloud-application-credentials.json --iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com

Use the content of the locally saved gcloud-application-credentials.json file to create the rclone secret:

apiVersion: v1
kind: Secret
metadata:
  name: gcs-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_GCS_TYPE: google cloud storage
  RCLONE_CONFIG_GCS_ANONYMOUS: "false"
  RCLONE_CONFIG_GCS_SERVICE_ACCOUNT_CREDENTIALS: '{"type":"service_account", ... <rest of gcloud-application-credentials.json>}'

Note

The remote name is gcs here so urls would take form similar to gcs:<your bucket>.

Custom Storage Initializer¶

If for some reason you would like to use a different storage initializer for batch jobs, e.g. kfserving storage initializer, you can set this by modifying the aforementioned deploy-values.yaml:

batchjobs:
  storageInitializer:
    image: gcr.io/kfserving/storage-initializer:v0.4.0

The corresponding secret for minio as configured in the production installation would also need to be created:

export MINIOUSER=minioadmin
export MINIOPASSWORD=minioadmin
export NAMESPACE=seldon

cat << EOF > seldon-kfserving-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: seldon-kfserving-secret
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: ${MINIOUSER}
  AWS_SECRET_ACCESS_KEY: ${MINIOPASSWORD}
  AWS_ENDPOINT_URL: http://minio.minio-system.svc.cluster.local:9000
  USE_SSL: "false"
EOF

kubectl apply -n ${NAMESPACE} -f seldon-kfserving-secret.yaml