Seldon Deploy installs into standard kubernetes clusters and can be accessed by all modern browsers mentioned in the specifications.
Contact the Seldon team with the relevant issue and provide necessary details about your particular install of Seldon Deploy. The version details can be found at the about page of the tool that can be accessed from the user menu on the top-right corner.
This about page displays the Seldon Deploy version details, license details and your browser version. For more details on your specific browser, please visit https://www.whatsmybrowser.org/ and share the URL that can provide us with more information about your browser usage like resolution and other support needed.
We recommend using the latest browser version available to your operating system. See your browser’s documentation to learn more about checking and updating your version. The project supports all modern browsers based on global usage (> 0.2%). The full list of browser versions supported include stable versions of Chrome, Firefox, Safari, Microsoft Edge and has a global coverage of 91.09%. The full set is listed here
and_chr 86 and_ff 82 and_uc 12.12 chrome 86 chrome 85 chrome 84 chrome 83 chrome 80 chrome 49 edge 86 edge 85 edge 18 firefox 82 firefox 81 ios_saf 14 ios_saf 13.4-13.7 ios_saf 13.3 ios_saf 12.2-12.4 ios_saf 10.3 ios_saf 9.3 opera 71 safari 14 safari 13.1 safari 13 samsung 12.0
Important browser settings To log into and edit your site, check if these browser settings are enabled or disabled in your browser:
Browser add-ons or extensions - Browser add-ons might interfere with site editing. While disabling them isn’t always required, we may ask you to disable them when helping you troubleshoot.
Browser window sizes Your computer’s screen size determines the maximum browser window resolution. For the best experience editing your site, use a browser window at least 1280 pixels wide and 768 pixels tall.
Errors from Seldon Deploy¶
If Seldon Deploy crashes or returns an error the best first steps to get details are:
Turn on the network tab in the browser (via right-click and ‘Inspect’ in chrome), hit ‘Preserve Log’ and recreate the issue. Then find the failed call to see the full message.
Find the Seldon Deploy pod (usually in the seldon-system namespace) and inspect its logs.
This should help determine if the cause is Seldon Deploy or another component. If another component, see the other debugging sections from here. Either way the information obtained will be useful if an issue needs to be reported to Seldon.
Insufficient ephemeral storage in EKS clusters¶
eksctl, the volume size for each node will be of 20Gb by default.
However, with large images this may not be enough.
This is discussed at length on this
thread in the
When this happens, pods usually start to get evicted.
If you run
kubectl describe on any of these pods, you should be able to see
not enough ephemeral storage.
You should also be able to see some
DiskPressure events on the output of
kubectl describe nodes.
To fix it, it should be enough to increase the available space.
eksctl, you can do so by tweaking the
nodeGroups config and adding a
For instance, to change the volume to 100Gb you could do the following in your
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig ... nodeGroups: - volumeSize: 100 volumeType: gp2 ...
Elastic Queue Capacity¶
If request logging is used with a high throughput then it’s possible to hit a
rejected execution of processing error in the logger.
This comes with a
queue capacity message.
To address this the
thread_pool.write.queue_size needs to be increased.
For example, with the Elasticsearch Helm chart this could be:
esConfig: elasticsearch.yml: | thread_pool.write.queue_size: 2000
Request Logs Entries Missing¶
Sometimes requests fail to appear in the request logs. Often this is a problem with the request logger setup. If so see the request logging docs.
Sometimes we see this error in the request logger logs:
RequestError(400, 'illegal_argument_exception', 'mapper [request.instance] cannot be changed from type [long] to [float]')
What happens here is Elasticsearch has inferred the type of the fields in the request for the model’s index. This is inferred on the first request and if it changes or is inferred incorrectly this has to be addressed manually.
The best thing to do here is to delete the index.
First port-forward Elasticsearch. If the example OpenDistro Elasticsearch is used then this is
kubectl port-forward -n seldon-logs svc/opensearch-cluster-master 9200.
To delete the index we need to know its name. These follow a pattern. The pattern is:
default unless there’s a canary, and
modelid is usually
<modelname>-container if created via the Seldon Deploy UI.
You may use the Elasticsearch API to list the available indices in Elasticsearch with
curl -k --request GET 'https://localhost:9200/_cat/indices/inference-log-*', which should return something like:
yellow open inference-log-seldon-seldon-income-default-income-container xWEBE-xXQA65cEIitp6SHw 1 1 9646 0 3.4mb 3.4mb yellow open inference-log-seldon-seldon-income-classifier-default-income-classifier-container wUeX7fiISpSIGFCoHI05Cg 1 1 61 60 151.1kb 151.1kb yellow open inference-log-seldon-seldon-batch-demo-default-batch-demo-container 3oYtAwL1Q-2Gi3og91iGdg 1 1 1848 1093 1.3mb 1.3mb
Optionally, for easier debugging, you may also install Kibana to visualize and inspect indices in Elasticsearch. You may install Kibana either directly or its equivalent OpenSearch Dashboards via the OpenSearch installation. Our trial install docs also provide steps for you to access Kibana at
/kibana/, without port-forwarding, via a
Then delete the index with a curl command. If the auth is
admin/admin and there’s a
cifar10 seldon model in a namespace also called
seldon then it’s
curl -k -v -XDELETE https://admin:[email protected]:9200/inference-log-seldon-seldon-cifar10-default
Request Logs Entries Incorrect¶
Model names should not overlap as at the time of writing there is a bug with the request log page. Its searches can get confused between, say, ‘classifier’ and ‘classifier-one’. The logs in the backend are distinct but the searches can get muddled. This issue is being tracked - for now the workaround is to use distinct names. This affects both viewing the prediction logs and also the feature distributions monitoring feature. In the distributions monitoring dashboard, this error may manifest as missing statistics and distribution graphs.
See the auth section for debugging tips.
See the Knative install section for how verify Knative.
Argo and Batch¶
See the argo section for debugging batch and the minio section for minio.
See the metrics section for debugging prometheus.
For Seldon Core debugging, it is best to see their respective docs.
In our demos we load models from google storage buckets. In the model logs we sometimes see this:
Compute Engine Metadata server unavailable onattempt
This is a known google storage issue but does not cause failures. Treat as a warning.
Request logging for Tritonserver in Seldon Core 1.11.2¶
This version of
gzip value in
Accept-Encoding header and returns compressed payloads indicated by
'Content-Encoding': 'gzip' header in the response. This value is returned back to the client so the proper content decoding can occur - most api clients do this automatically.
The relevant header unfortunately is not passed to the request logger, see SC3690 issue. This means that
gzip compressed payloads will not get logged. To workaroud this issue one can send requests explicitly setting
'Accept-Encoding': 'ident' header.
This will instruct Tritonserver to send back uncompressed responses that can be logged.
Another possible workaround is to temporarily downgrade Triton server image tag to previous version, e.g.
20.08-py3 by setting in
core-values.yaml a following
predictor_servers: TRITON_SERVER: protocols: kfserving: defaultImageVersion: 20.08-py3 image: nvcr.io/nvidia/tritonserver