Model Outlier Detection

In a production environment, it is critical to monitor the data your machine learning model runs inference on, as changes in data can adversely affect the performance of ML models.

This demo will illustrate how to identify outliers in your inference data, using Alibi Detect’s VAE outlier detection method for tabular datasets in the following steps.

• Launch an image classifier model trained on the CIFAR-10 dataset. The data instances contains 32x32x3 pixels images that are classified into 10 classes such as truck, frog, and cat etc.

• Set up an VAE outlier detector for this particular model

• Send a request to get an image classification

• Send a perturbed request to identify an outlier instance

Important

This demo requires Knative installation on the cluster as the drift detector will be installed as a kservice. See Knative installation instructions for necessary setup required.

Launch a Seldon Core V1 Deployment

Create an image classifier model deployment into an appropriate namespace

1. Click Create on the deployments page to create a Seldon Deployment.

2. Enter the deployment details in the deployment creation wizard, then click Next:

   Field	Value
   Name	cifar10
   Namespace	seldon[1]
   Type	Seldon Deployment
   Protocol	Tensorflow

3. Configure the default predictor as follows:

   Field	Value
   Runtime	Tensorflow
   Model URI	gs://seldon-models/tfserving/cifar10/resnet32
   Project	default
   Storage Secret	(leave blank)[2]
   Model Name	(leave blank)

4. Click Next for the remaining steps^[3], then click Launch.

1. The

seldon

and

seldon-gitops

namespaces are installed by default, which may not always be available. Please select a namespace which best describes your environment.
2. A secret may be required for private buckets.
3. Additional steps may be required for your specific model.

Expand to see default predictor details

Add An Outlier Detector

From the overview page, select your deployment to enter the dashboard.

1. Inside the dashboard, add an outlier detector by clicking the Add button within the Outlier Detection widget.

2. Configure the detector with the following values:

   Field	Value
   Name	outlier-detect
   Model URI	gs://seldon-models/alibi-detect/od/OutlierVAE/cifar10-0.9.0
   Project	default
   Storage Secret[1]	(leave blank)
   Reply URL[2]	http://seldon-request-logger.seldon-logs

3. Click CREATE DETECTOR to complete the setup.

1. A secret may be required for private buckets.
2. By default, the Reply URL is set, as above. If you are using a custom installation, please change this parameter according to your installation.

Expand to see outlier detector creation

Make Predictions

Run a single prediction using an image of a truck in the Tensorflow payload format. Click the payload to download it from the following table, as well as a perturbed instance of the same image. Make a couple of these requests from the Predict screen on the UI.

Payload type	Image	Tensorflow Payload
Expected Instance		Download Truck Payload
Outlier Instance		Download Pertubed Truck Payload

Expand to see running prediction request

Step	Description
1	Click on the Predict button on the left navigation menu.
2	Select the Upload JSON File tab.
3	Upload the cifar10_image.json file.
4	Click Predict to send the request.

Step	Description
1	Click on the Predict button on the left navigation menu.
2	Select the Upload JSON File tab.
3	Upload the outlier_image.json file.
4	Click Predict to send the request.

View Outliers From Request Logs

Go to the Requests screen to view request logs. Outlier scores will be available on the right side of each instance. You can also highlight outliers and filter them using the Highlight Outliers toggle on the top-right of the screen.

Real-Time Outlier Monitoring

You can view a timeline of outlier/inlier requests in the Outlier Detection tab of the Monitor screen.

Troubleshooting

If you experience issues with this demo, see the troubleshooting docs and also the Knative or Elasticsearch sections.