Upgrading to 1.5.0¶
Secrets Management¶
Seldon Deploy 1.5.0 introduces the ability to manage secrets for registry and bucket access directly through the UI.
The default Minio secret used in some installations has been renamed from
seldon-rclone-secret
tominio-bucket
, and the URI remote has changed froms3://
tominio://
Secrets fields are now dropdowns rather than free text fields. Secrets created through the Deploy interface or API will automatically appear, but to allow your existing secrets to show in the dropdown please label your Kubernetes secret resources with the following:
seldon-deploy=true secret-type=<secret-type> # where <secret-type> is registry or bucket
ML Data Logging - Metronome¶
Seldon Deploy 1.5 introduces a new request logger developed by Seldon called
metronome
to replace the old request logger. It is available as a public docker image at https://hub.docker.com/r/seldonio/metronome, where tag1.0.1
is the default version being used by Seldon Deploy 1.5.Note that
metronome
is a required replacement for the old request logger as several new features introduced by Seldon Deploy 1.5 depend on it.If upgrading from an older version of Seldon Deploy to 1.5, please ensure that the following section has been updated to use
metronome
for request logging:requestLogger: create: true image: seldonio/metronome:1.0.1
Monitoring - Feature Distributions¶
Seldon deploy 1.5 allows node level prediction data logging and feature distributions. Because of that, the Elasticsearch index pattern used by the feature distributions dashboard has been updated to include the node name/model id of the predictor, which is the pattern that the new
metronome
request logger uses.This means that data from previous versions (<v1.5.0) of Seldon Deploy using the old request logger), will no longer be found on the monitoring - feature distributions dashboard since the index pattern is different.
This also applies to reference data.
New requests that are stored in this new index using
metronome
will be availableIn order to re-use requests logged using the old python request logger, users may manually reindex the old requests on Elasticsearch:
For example, given a Seldon Deployment with the following spec (this can be viewed on the Deployment Dashboard), the node name/model id of the model is
income-container
:predictors: - graph: implementation: SKLEARN_SERVER modelUri: 'gs://seldon-models/sklearn/income/model-0.23.2' name: income-container name: default
Port forward the Elasticsearch port to localhost:9200:
kubectl port-forward svc/elasticsearch-opendistro-es-client-service 9200:9200
For convenience, we export the address to the variable
ES_ADDR
:export ES_ADDR=http://localhost:9200
Get the names of the old and new index. The pattern of the old index should follow the pattern
inference-log-<serving engine>-<deployment namespace>-<deployment name>-<deployment endpoint>
while the new index would beinference-log-<serving engine>-<deployment namespace>-<deployment name>-<deployment endpoint>-<deployment node>
. If the above Seldon Deployment is in thedev
namespace, the old index would beinference-log-seldon-dev-income-default
, whereas the new index would beinference-log-seldon-dev-income-default-income-container
. The following command is useful:curl --location --request GET 'https://localhost:9200/_cat/indices/inference-log-*'
We export and refer to the old and new indices as follows:
export OLD_INDEX=inference-log-seldon-dev-income-default export NEW_INDEX=inference-log-seldon-dev-income-default-income-container
Get the mapping of the old index. Store the mapping in a shell variable
MAPPINGS
as follows:MAPPINGS=$(curl --request GET "${ES_ADDR}/${OLD_INDEX}?pretty" | jq ".\"${OLD_INDEX}\".mappings")
Create the new index with the same mappings as the old index.
curl --request PUT "${ES_ADDR}/${NEW_INDEX}" --header 'Content-Type: application/json' --data-raw "{\"mappings\": ${MAPPINGS}}"
Reindex the old requests to the new index with the following command:
curl --request POST "${ES_ADDR}/_reindex" \ --headertomised your Deploy installation, please be aware that the following Helm values have changed since v1.4.0:
Name |
Previous value |
Current value |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|