Upgrading Seldon Deploy


When upgrading Seldon Deploy, please, always use matching version of installation resources that can be downloaded here. This is especially important for the Seldon Deploy helm charts as there may contain updates.


Seldon always recommend to first validate upgrade path in development cluster before rolling out changes to production.

Upgrading to 1.4.0

Seldon Core

  • Seldon Deploy 1.4.0 recommends Seldon Core in version 1.11.x. Upgrade should be straightforward following the related documentation page.

  • It is possible that model artifacts trained to work with pre-packaged model servers will require update. Model explainers and outlier detectors may be the components most likely at risk here. Reason for retraining are often related to the Python version change or library upgrades in the related server. Following workarounds are available:

    • Delay Seldon Core upgrade until you confirm in development environment that all your model work as expected and retrain them as required

    • Modify helm values (core-values.yaml) to specify previous version of the related server, for example:

        defaultImageVersion: 1.11.2
        image: seldonio/sklearnserver
  • In Seldon Core 1.11.2 the default version of Triton server is 21.08-py3. Currently gzip compressed payloads returned by it are not being logged, see more for details.

Authorization/Policy Management

  • Seldon Deploy 1.4.0 introduces a new API for managing OPA based policies. These new endpoints will be enabled by default if OPA authorization is enabled. If you do not want to manage your OPA policies via API you can disable them as described in the configuration guide. See the API documentation for instruction on how to use the new API endpoints.

Alerting Integration

  • In version 1.4.0 we introduce an alerting system. Metrics exposed on Deploy and Models can be used to alert when SLOs are breached. These alerts are configured to be sent to the frontend of Deploy, and can also be configured for external providers like Opsgenie and Pagerduty.

  • The default installation provides the configmaps and setup required for this to work out of the box, but changes to Prometheus and Alertmanager configuration will be required if not using the default installation. The production installation guide covers the setup required and also documents how to configure external notifications and custom alerts.

  • See the alerting integration demo to test the alerting flow.

  • Note: if your OIDC provider is using TLS and a self-signed certificate, alertmanager cannot send notifications to the frontend, see more for details.

Detector Configuration and Metrics

  • Seldon Deploy 1.4.0 updates the single-detector APIs for deployments to a multi-detector APIs using the SDK. This enables creation of multiple detectors i.e. drift detectors, outlier detectors and metrics servers. This is compatible with the current detector gitops structure and configuration so existing detectors work as-is but the detector APIs have been modified. See new swagger spec for changes.

  • The new APIs introduced also require a label for detector kservices in the non-gitops namespaces to appear on the UI. No change is needed for gitops enabled namespaces. Functionality of existing detectors remain intact.

  • To add the required labels, in a non-gitops namespace, this can be done individually to all existing detectors using the following command. Once this label is added, there is no change to the existing user journey in Seldon deploy UI to allow provisioning of per deployment detectors. For KServe Inference services, change SeldonDeployment to InferenceService in the following script.

kubectl label kservice -n ${namespace} ${kservice_name} seldon.io/deployment-name=${deployment_name}
kubectl label kservice -n ${namespace} ${kservice_name} seldon.io/deployment-kind=SeldonDeployment

Upgrading to 1.3.0

  • Seldon Core v1.8.0 changed the default storage initializer to be seldonio/rclone-storage-initializer. The secret format that holds credentials have changed and requires update.

    • If you do not wish to update them now, please, set following value in core-values.yaml used when installing Seldon Core with Helm

      image: gcr.io/kfserving/storage-initializer:v0.4.0

    This is documented in kfserving storage initializer documentation.

    • Following resources are available to help you upgrade to the rclone-compatible secret format:

      • Seldon Core 1.8 upgrading documentation

      • Handling Credentials documentation for pre-packaged model servers with rclone-based storage initializer

      • Tutorial on how to test new secret format without setting rclone-based storage initializer globally

      • An example upgrade path that illustrates steps required to smoothly upgrade AWS S3 / MinIO configured clusters from kfserving-based Secret and Service Account credentials to the new format.

  • SD 1.3.0 introduces improvement to the Batch Processing of requests - generated Argo workflows now use storage initializer mechanism for pulling in input file and uploading output file. The default storage initializer is seldonio/rclone-storage-initializer which requires adjustment of credential secret.

    • If you do want to keep using current format of secret without updating it now, set following value in deploy-values.yaml used when installing Seldon Deploy with Helm

        image: gcr.io/kfserving/storage-initializer:v0.4.0
    • To upgrade to the new format see links provided in the previous point. Also consult our documentation that contains example for MinIO-compatible secret.

  • SD 1.3.0 updates the request logger component’s image in the helm chart default values file to seldonio/seldon-request-logger:1.9.1. This version of request logger looks up model level metadata from Seldon Deploy Metadata service before creating predictions logging elasticsearch indexes. This feature is leveraged by new advanced monitoring features like distributions monitoring introduced in this SD version. Although, it is not a mandatory configuration and it is possible disable the metadata lookup from the request logger. New features like distributions monitoring will still work, you just won’t be able to enrich logged predictions with the model level predictions schema (e.g. feature category names).

    • As mentioned earlier, metadata lookup is not mandatory for the request logger. If you wish to disable this feature and would prefer the request logger to create the elasticsearch indexes without the model level metadata as in previous releases, follow the steps below.

      • If you have already enabled model level metadata feature from version v1.2.0 and want to continue using this feature without the request logger connection to Seldon Deploy, then use the default helm chart values in the request logger section

      • Alternatively, if you have disabled model level metadata feature by setting the metadata.pg.enabled to false also disables the request logger connection to Seldon Deploy as the model level metadata is not being stored.

    • To enable this metadata lookup feature, the request logger requires authentication configuration to connect with a Seldon Deploy instance. For configuring this feature, see the request logger page for details of the authentication setup needed to connect to Seldon Deploy. If you are using model metadata then adding auth from the request logger to Deploy enables the prediction schema for a model to be found and used to enrich prediction data and enrich monitoring. This will be enabled for all new elasticsearch logging indexes once setup. Existing indexes will work as before but will need to be reindexed to store additional enriched info. Enabling auth from request logger to Deploy currently requires a client credentials auth flow but can also be configured to work with the password grant flow if needed.

Upgrading to 1.2.0

  • 1.2.0 introduces the Model Metadata storage to Seldon Deploy. To enable/disable the model metadata storage functionality with all required prerequisites follow the instructions. The quick overview of the changes is:

    • If you do not want the model metadata functionality disable it by setting metadata.pg.enabled in the Seldon Deploy helm chart to false. Not recommended since all model metadata related features will be unavailable.

    • Install a self-hosted or managed PostgreSQL solution (instructions).

    • Configure the connection details to the PostgreSQL instance by adding a metadata-postgres secret in the Seldon Deploy namespace (instructions).

    • Enable the model metadata storage by setting metadata.pg.enabled in the Seldon Deploy helm chart to true (instructions).

  • Detector components for outlier detection, drift detection and metrics are now moving out of the seldon-logs namespace and into the user namespaces. To handle this

    • The easiest way to migrate is to delete any existing detectors from the UI and recreate them after upgrade.

    • To see what detectors you have, do kubectl get ksvc -n seldon-logs. Also check kubectl get trigger -n seldon-logs.

    • Detectors will have -outlier, -drift or -metrics in their names. Each will have a ksvc and a corresponding trigger. The name and namespace of the model they are for will also be in the name.

    • If you find any issues while carrying out this process, please contact Seldon.

  • The ElasticSearch integration will now enable basic authentication by default.

    • This is controlled by the elasticsearch.basicAuth flag in the Seldon Deploy Helm chart.

    • To disable it (and revert to the default behaviour of <= 1.1.0), you can set its value as elasticsearch.basicAuth=false.

    • To keep it enabled and configure ElasticSearch’s user / password credentials, please check the ElasticSearch integration setup guide.

  • The Seldon Core version is now 1.9.1. See the Getting Started > Production Installation section for installation. A helm upgrade --install can be used to upgrade.

  • The demo drift detector has changed from gs://seldon-models/alibi-detect/cd/ks/cifar10-0_4_3 to gs://seldon-models/alibi-detect/cd/ks/cifar10-0_4_4. This is because the alibi version in Seldon Core is upgraded.

Upgrading to 1.1.0

  • The helm values file now has an option to set user and password for basic auth to Elastic. For more information, see the request logger documentation.

  • The recommended prometheus settings have now changed to improve performance. We suggest uninstalling the prometheus helm chart and installing with the below changes.

    • See the metrics page for the full helm values.

    • The parts that have changed are:

      • prometheus.server.livenessProbePeriodSeconds is now 30

      • prometheus.server.extraArgs has been added with entries query.max-concurrency: 400 and storage.remote.read-concurrent-limit: 30

      • prometheus.server.resources.limits now has cpu: 2 and memory: 4Gi

      • prometheus.server.resources.requests now has cpu: 800m and memory: 1Gi

    • Remove analytics with helm delete -n seldon-system seldon-core-analytics.

    • Install again using the instructions

  • The recommended Seldon Deploy helm values have changed slightly to improve performance. Set the following:

    • resources.limits should now have cpu: 800m and memory: 800Mi

  • A further change to the Seldon Deploy helm values is the addition of a configuration for the runAsUser for Knative pods started by Deploy:

defaultUserID: "8888"

Upgrading to 1.0.0

  • If you are upgrading by more than one version, please see previous notes.

  • The self-install trial customers, the file dockercreds.txt is now called sdconfig.txt (anyone using Product Installation can ignore this).


  • New components have been introduced for Batch support. Argo and minio. See Getting Started > Production Installation

    • Note the new Batch component requires per-namespace config for any Batch-enabled namespaces. See Getting Started > Production Installation section on Argo and refer to references to .

  • For upgrading Seldon Deploy specifically

    • The requestLogger.image in the helm values file is now docker.io/seldonio/seldon-request-logger:1.5.0

    • The alibidetect.image in the helm values is now seldonio/alibi-detect-server:1.5.0.

    • There’s a new entry in the batchjobs section of the helm values - serviceAccount: "workflow"

    • The Seldon Core version is now 1.5.0. See the Getting Started > Production Installation section for installation.

    • The KFServing version is 0.4.1.

Upgrading to 0.9.0

  • Seldon Deploy 0.9.0 uses Knative 0.18. To upgrade this, see the Upgrading Knative section.

  • For upgrading Seldon Deploy specifically

    • A license is required at app install/startup time. Contact Seldon for a license.

    • Seldon Core Analytics or Prometheus needs to be upgraded. Prometheus recording rules have been introduced. See Getting Started > Production Installation > Metrics Monitoring

    • If Seldon Core Analytics is used, upgrade with using seldon-deploy-install/prerequisites-setup/prometheus/seldon-core-analytics.sh from the scripts extracted in Getting Started > Download Installation Resources.

    • Upgrading Seldon Core Analytics will reset prometheus data.

    • The requestLogger.image in the helm values file is now docker.io/seldonio/seldon-request-logger:1.3.0-json

    • The requestLogger.trigger in the helm values now has apiVersion “eventing.knative.dev/v1” and broker “default”.

    • The alibidetect.image in the helm values is now seldonio/alibi-detect-server:1.4.0.

    • There is a new batchjobs section in the helm values.

Upgrading to 0.8.2

Upgrading to 0.8.0

  • The sd-install script is now called sd-install-default. Or for a kubeflow install there’s sd-install-kubeflow.

  • The helm values file now has a single gitops section. This involves the following changes.

    • The recommended way to configure git credentials is in a git secret, though a token parameter to the chart can still be used.

    • The sd-install-default script contains an example of this. It is:

kubectl create secret generic git-creds -n seldon-system --from-file ~/.ssh/id_rsa --from-file ~/.ssh/known_hosts --from-literal=passphrase="$GIT_SSHKEY_PASSPHRASE" --from-literal=username="$GIT_USER" --from-literal=token="$GIT_TOKEN" --from-literal=email="$GIT_EMAIL"  --dry-run -o yaml | kubectl apply -f -
  • skipVerifyGit, gitwebhook, GITOPS_FORMAT and argocd config have all moved from github to the new gitops section in helm values file, see documentation.

  • if your ArgoCD Application does not follow naming pattern seldon-gitops-${namespace}, please, specify the ArgoCD application using namespace labels, see documentation.

  • The contents of the helm chart have also changed. In particular:

    • The istio virtual service no longer does path rewriting.

    • Permission is now required to watch argocd applications.

    • The liveness and readiness probes are now on /seldon-deploy/api/status

Upgrading to 0.7.0

  • The Deploy helm chart now contains an option to install a request logger in the seldon-logs namespace. This is on by default. Previously loggers were per-namespace. To use a single request logger installed through the chart, ensure the seldon-logs namespace is labelled for knative-evenint and that seldon core is v1.2.1 and has executor.requestLogger.defaultEndpoint http://default-broker.seldon-logs. For KfServing, if using it, the default broker will for now continue to be in the current namespace unless set at the point of creating the model deployment.

  • The helm values file has seen some changes.

    • There’s now a request logger section for installing the request logger to a dedicated namespace, if desired.

    • The istioIngress section has been renamed ingressGateway

    • There are extra options in the prometheus section for for using a secured prometheus (assumes not secure by default, as before).

    • There are extra options in the elasticsearch section for for using a secured prometheus (assumes not secure by default, as before).