Model Outlier Detection¶
When ML models are deployed in production, it is an important to monitor the data that the model runs inference on. Changes in a data can adversely affect the performance of ML models and hence it is important to track this outlier instances of data. This demo is based on VAE outlier detection method in the Alibi Detect project for tabular datasets.
Here we will :
Launch an image classifier model trained on the CIFAR10 dataset. The data instances contains 32x32x3 pixels images that are classified into 10 classes including truck, frog, cat etc
Setup an VAE outlier detector for this particular model
Send a request to get an image classification
Send a perturbed request to get a positive outlier detection
Important
This demo requires Knative installation on the cluster as the drift detector will be installed as a kservice. See Knative intallation instructions for necessary setup required.
Launch a Seldon Core V1 Deployment¶
Create an image classifier model deployment into an appropriate namespace
Click
Create
on the deployments page to create a Seldon Deployment.Enter the deployment details in the deployment creation wizard, then click
Next
:Field
Value
Name
cifar10
Namespace
seldon
[1]Type
Seldon Deployment
Protocol
Tensorflow
Configure the default predictor as follows:
Field
Value
Runtime
Tensorflow
Model URI
gs://seldon-models/tfserving/cifar10/resnet32
Project
default
Storage Secret
(Leave blank)[2]
Model Name
(Leave blank)
Click
Next
for the remaining steps[3], then clickLaunch
.
seldonand
seldon-gitopsnamespaces are installed by default, which may not always be available. Please select a namespace which best describes your environment.
2. A secret may be required for private buckets.
3. Additional steps may be required for your specific model.
Add An Outlier Detector¶
From the overview page, select your deployment to enter the dashboard.
Inside the dashboard, add an outlier detector by clicking the
Add
button within theOutlier Detection
widget.Configure the detector with the following values:
Field
Value
Name
outlier-detect
Model URI
gs://seldon-models/alibi-detect/od/OutlierVAE/cifar10-0.9.0
Project
default
Storage Secret[1]
(Leave blank)
Reply URL[2]
http://seldon-request-logger.seldon-logs
Click
CREATE DETECTOR
to complete the setup.
2. By default, the Reply URL is set as
seldon-request-loggerin the logger's
defaultnamespace. If you are using a custom installation, please change this parameter according to your installation.
Make Predictions¶
Run a single prediction using the expected instance of the truck image in tensorflow payload format. Click the payload to download it from the following table. Also a perturbed image of the truck in the same format is available in the following table. Make a couple of these requests at random using the predict tool in the UI.
Payload type |
Image |
Tensorflow Payload |
---|---|---|
Expected Instance |
||
Outlier Instance |
View Outliers From Historical Requests¶
Go to the requests screen to view all the historical requests. Here, you may see the outlier score of each instance. You can also highlight outliers and filter them using the filters on the right hand side.
Real-Time Outlier Monitoring¶
You can view a timeline of outlier requests under the “Monitor” section for your deployment dashboard.
Congratulations! You have successfully deployed a Seldon Core V1 deployment with an outlier detector! 🥳 Ready to try another demo? Check them out here.
Troubleshooting¶
If you experience issues with this demo, see the troubleshooting docs and also the Knative or Elasticsearch sections.