Upgrading¶
Upgrading Seldon Deploy
Upgrading to 1.2.0¶
1.2.0 introduces the Model Metadata storage to Seldon Deploy. To enable/disable the model metadata storage functionality with all required prerequisites follow the instructions. The quick overview of the changes is:
If you do not want the model metadata functionality disable it by setting
metadata.pg.enabled
in the Seldon Deploy helm chart tofalse
. Not recommended since all model metadata related features will be unavailable.Install a self-hosted or managed PostgreSQL solution (instructions).
Configure the connection details to the PostgreSQL instance by adding a
metadata-postgres
secret in the Seldon Deploy namespace (instructions).Enable the model metadata storage by setting
metadata.pg.enabled
in the Seldon Deploy helm chart totrue
(instructions).
Detector components for outlier detection, drift detection and metrics are now moving out of the
seldon-logs
namespace and into the user namespaces. To handle thisThe easiest way to migrate is to delete any existing detectors from the UI and recreate them after upgrade.
To see what detectors you have, do
kubectl get ksvc -n seldon-logs
. Also checkkubectl get trigger -n seldon-logs
.Detectors will have
-outlier
,-drift
or-metrics
in their names. Each will have aksvc
and a correspondingtrigger
. The name and namespace of the model they are for will also be in the name.If you find any issues while carrying out this process, please contact Seldon.
The ElasticSearch integration will now enable basic authentication by default.
This is controlled by the
elasticsearch.basicAuth
flag in the Seldon Deploy Helm chart.To disable it (and revert to the default behaviour of
<= 1.1.0
), you can set its value aselasticsearch.basicAuth=false
.To keep it enabled and configure ElasticSearch’s user / password credentials, please check the ElasticSearch integration setup guide.
The Seldon Core version is now
1.7.0
. See theGetting Started > Production Installation
section for installation. Ahelm upgrade --install
can be used to upgrade.The demo drift detector has changed from
gs://seldon-models/alibi-detect/cd/ks/cifar10-0_4_3
togs://seldon-models/alibi-detect/cd/ks/cifar10-0_4_4
. This is because the alibi version in Seldon Core is upgraded.
Upgrading to 1.1.0¶
The helm values file now has an option to set user and password for basic auth to Elastic. For more information, see the request logger documentation.
The recommended prometheus settings have now changed to improve performance. We suggest uninstalling the prometheus helm chart and installing with the below changes.
See the metrics page for the full helm values.
The parts that have changed are:
prometheus.server.livenessProbePeriodSeconds
is now 30prometheus.server.extraArgs
has been added with entriesquery.max-concurrency: 400
andstorage.remote.read-concurrent-limit: 30
prometheus.server.resources.limits
now hascpu: 2
andmemory: 4Gi
prometheus.server.resources.requests
now hascpu: 800m
andmemory: 1Gi
Remove analytics with
helm delete -n seldon-system seldon-core-analytics
.Install again using the instructions
The recommended seldon deploy helm values have changed slightly to improve performance. Set the following:
resources.limits
should now havecpu: 800m
andmemory: 800Mi
A further change to the seldon deploy helm values is the addition of a configuration for the runAsUser for knative pods started by deploy:
defaultUserID: "8888"
Upgrading to 1.0.0¶
If you are upgrading by more than one version, please see previous notes.
The self-install trial customers, the file
dockercreds.txt
is now calledsdconfig.txt
(anyone usingProduct Installation
can ignore this).The variable
KUBEFLOW_USER_EMAIL
has becomeSD_USER_EMAIL
andKUBEFLOW_PASSWORD
has becomeSD_PASSWORD
New components have been introduced for Batch support. Argo and minio. See
Getting Started > Production Installation
Note the new Batch component requires per-namespace config for any Batch-enabled namespaces. See
Getting Started > Production Installation
section onArgo
and refer to references to .
For upgrading Seldon Deploy specifically
The
requestLogger.image
in the helm values file is nowdocker.io/seldonio/seldon-request-logger:1.5.0
The
alibidetect.image
in the helm values is nowseldonio/alibi-detect-server:1.5.0
.There’s a new entry in the
batchjobs
section of the helm values -serviceAccount: "workflow"
The Seldon Core version is now 1.5.0. See the
Getting Started > Production Installation
section for installation.The KFServing version is 0.4.1.
Upgrading to 0.9.0¶
Seldon Deploy 0.9.0 uses knative 0.18. To upgrade this, see the Upgrading Knative section.
For upgrading Seldon Deploy specifically
A license is required at app install/startup time. Contact Seldon for a license.
Seldon Core Analytics or Prometheus needs to be upgraded. Prometheus recording rules have been introduced. See
Getting Started > Production Installation > Metrics Monitoring
If Seldon Core Analytics is used, upgrade with using
seldon-deploy-install/prerequisites-setup/prometheus/seldon-core-analytics.sh
from the scripts extracted inGetting Started > Download Installation Resources
.Upgrading Seldon Core Analytics will reset prometheus data.
The
requestLogger.image
in the helm values file is nowdocker.io/seldonio/seldon-request-logger:1.3.0-json
The
requestLogger.trigger
in the helm values now hasapiVersion
“eventing.knative.dev/v1” andbroker
“default”.The
alibidetect.image
in the helm values is nowseldonio/alibi-detect-server:1.4.0
.There is a new
batchjobs
section in the helm values.
Upgrading to 0.8.2¶
In installations that have restricted external access (eg. air gapped), the following image will need to be downloaded:
Upgrading to 0.8.0¶
The sd-install script is now called sd-install-default. Or for a kubeflow install there’s sd-install-kubeflow.
The helm values file now has a single gitops section. This involves the following changes.
The recommended way to configure git credentials is in a git secret, though a token parameter to the chart can still be used.
The sd-install-default script contains an example of this. It is:
kubectl create secret generic git-creds -n seldon-system --from-file ~/.ssh/id_rsa --from-file ~/.ssh/known_hosts --from-literal=passphrase="$GIT_SSHKEY_PASSPHRASE" --from-literal=username="$GIT_USER" --from-literal=token="$GIT_TOKEN" --from-literal=email="$GIT_EMAIL" --dry-run -o yaml | kubectl apply -f -
skipVerifyGit, gitwebhook, GITOPS_FORMAT and argocd config have all moved from
github
to the newgitops
section in helm values file, see documentation.if your ArgoCD Application does not follow naming pattern
seldon-gitops-${namespace}
, please, specify the ArgoCD application using namespace labels, see documentation.The contents of the helm chart have also changed. In particular:
The istio virtual service no longer does path rewriting.
Permission is now required to watch argocd applications.
The liveness and readiness probes are now on
/seldon-deploy/api/status
Upgrading to 0.7.0¶
The deploy helm chart now contains an option to install a request logger in the seldon-logs namespace. This is on by default. Previously loggers were per-namespace. To use a single request logger installed through the chart, ensure the seldon-logs namespace is labelled for knative-evenint and that seldon core is v1.2.1 and has executor.requestLogger.defaultEndpoint http://default-broker.seldon-logs. For KfServing, if using it, the default broker will for now continue to be in the current namespace unless set at the point of creating the model deployment.
The helm values file has seen some changes.
There’s now a request logger section for installing the request logger to a dedicated namespace, if desired.
The istioIngress section has been renamed ingressGateway
There are extra options in the prometheus section for for using a secured prometheus (assumes not secure by default, as before).
There are extra options in the elasticsearch section for for using a secured prometheus (assumes not secure by default, as before).