Model Outlier Detection¶

In a production environment, it is critical to monitor the data your machine learning model runs inference on, as changes in data can adversely affect the performance of ML models.

This demo will illustrate how to identify outliers in your inference data, using Alibi Detect’s VAE outlier detection method for tabular datasets in the following steps.

Launch an image classifier model trained on the CIFAR-10 dataset. The data instances contains 32x32x3 pixels images that are classified into 10 classes such as truck, frog, and cat etc.
Set up an VAE outlier detector for this particular model
Send a request to get an image classification
Send a perturbed request to identify an outlier instance

Launch a Seldon ML pipeline¶

Create an image classifier model deployment in an appropriate namespace

Click Create new deployment on the deployments page to create a Seldon ML Pipeline.
Enter the pipeline details in the deployment creation wizard and click Next:
- Name: cifar10
- Type: Seldon ML Pipeline
The predictor details should have the Tensorflow runtime to use the correct server and the following Model URI:
```
gs://seldon-models/triton/tf_cifar10
```
Click Next for the remaining steps, then click Launch.

Add an Outlier detector¶

From the Deployment Overview page, select your pipeline to enter the pipeline dashboard. Inside the Pipeline Dashboard, add an outlier detector by clicking the Add button within the Outlier Detection widget.

setup_detector

Enter the following parameters in the configuration wizard which appears to configure the detector:

Detector Name: cifar10-outlier.
Storage URI: (For public google buckets, secret field is optional)
```
gs://seldon-models/scv2/examples/cifar10/outlier-detector
```
Reply URL: Leave as the default value, shown below, unless you are using a custom installation, please change this parameter according to your installation
```
http://seldon-request-logger.seldon-logs
```

Then, click Create Detector to complete the setup.

Make Predictions¶

Run a single prediction using an image of a frog in the Tensorflow payload format. Click the payload to download it from the following table, as well as a perturbed instance of the same image. Make a couple of these requests from the Predict screen on the UI.

Payload type	Image	Tensorflow Payload
Expected Instance		`Download Frog Payload`
Outlier Instance		`Download Perturbed Frog Payload`

classif_images

View Outliers From Request Logs¶

Go to the Requests screen to view request logs. Outlier scores will be available on the right side of each instance. You can also highlight outliers and filter them using the Highlight Outliers toggle on the top-right of the screen.

outlier_detection

Real-Time Outlier Monitoring¶

You can view a timeline of outlier/inlier requests in the Outlier Detection tab of the Monitor screen.

outlier_detection

Troubleshooting¶

If you experience issues with this demo, see the troubleshooting docs or Elasticsearch sections.