Model Outlier Detection

When ML models are deployed in production, it is an important to monitor the data that the model runs inference on. Changes in a data can adversely affect the performance of ML models and hence it is important to track this outlier instances of data. This demo is based on VAE outlier detection method in the Alibi Detect project for tabular datasets.

Here we will :

• Launch an image classifier model trained on the CIFAR10 dataset. The data instances contains 32x32x3 pixels images that are classified into 10 classes including truck, frog, cat etc

• Setup an VAE outlier detector for this particular model

• Send a request to get an image classification

• Send a perturbed request to get a positive outlier detection

Important

This demo requires Knative installation on the cluster as the drift detector will be installed as a kservice. See Knative intallation instructions for necessary setup required.

Launch a Seldon Core deployment

Create an image classifier model deployment into an appropriate namespace

1. Click Create on the deployments page to create a Seldon Deployment.

2. Enter the deployment details in the deployment creation wizard and click Next:

   • Name: cifar10

   • Type: Seldon Deployment

   • Protocol: Tensorflow

   In the deployment creation wizard, enter a name for your new deployment (e.g. cifar10). Select the namespace you would like the deployment to reside in (e.g. seldon). From the protocol dropdown menu, select Tensorflow and click Next.

3. The predictor details should have the Tensorflow runtime to use the pre-packaged server and the following storage URI:

   gs://seldon-models/tfserving/cifar10/resnet32
   

4. Click Next for the remaining steps, then click Launch.

Expand to see default predictor details

Add an Outlier detector

From the deployment overview page, select your deployment to enter the deployment dashboard. Inside the deployment dashboard, add an outlier detector with by clicking the Create button within the Outlier Detection widget.

Expand to see outlier detector creation

Enter the following parameters in the modal popup which appears, to configure the detector:

• Model Name: cifar10.

• Model URI: (For public google buckets, secret field is optional)

  gs://seldon-models/alibi-detect/od/OutlierVAE/cifar10-0.9.0
  

• Reply URL: (By default, the Reply URL is set as seldon-request-logger in the logger’s default namespace. If you are using a custom installation, please change this parameter according to your installation.)

  http://seldon-request-logger.seldon-logs
  

Then, click Create Outlier-Detector to complete the setup.

Make Predictions

Run a single prediction using the expected instance of the truck image in tensorflow payload format. Click the payload to download it from the following table. Also a perturbed image of the truck in the same format is available in the following table. Make a couple of these requests at random using the predict tool in the UI.

Payload type	Image	Tensorflow Payload
Expected Instance		Download Truck Payload
Outlier Instance		Download Pertubed Truck Payload

Expand to see running prediction requests

View outliers from historical requests

Go to the requests screen to view all the historical requests. You can see the outlier score on each instance. Also you can highlight outliers based on this score and also use the filter to see only the outliers as needed.

Monitor outlier instances on a timeline

Under the ‘Monitor’ section you can see a timeline of outlier requests.

Troubleshooting

If you experience issues with this demo, see the troubleshooting docs and also the Knative or Elasticsearch sections.