Drift Detection with CIFAR10 Image Classifier
This demo is based on model distiliation drift detector in the alibi detect project.
The idea is that input data can change over time and become significantly different from the model training data. When the input data distribution shifts then prediction quality can drop. We want to know when that happens.
Here we will :
- Launch an image classifier model trianed on the CIFAR10 dataset.
- Setup a drift detector for this particular model.
- Send a request to get a image classification.
- Send a request to trigger a drift detection.
- View the drift in the dashboard.
NoteThis demo requires Knative installation on the cluster as the outlier detector will be installed as a kservice.
Create A Model
- Verify the namespace you will be deploying to.
- Deployment names must be unique and only contain alphanumeric characters.
From the deployment overview screen, click on the
Create button to create a new deployment.
- In the deployment creation wizard, enter a name for your new deployment.
- Select the namespace you would like the deployment to reside in (e.g.
- From the protocol dropdown menu, select
Tensorflowand click next.
For the deployment details, enter the following values, then click next:
Skip the remaining steps, then click
Add A Drift Detector
- Make sure you are in the correct namespace.
- By default, the Reply URL is set as
seldon-request-loggerin the logger’s default namespace. If you are using a custom installation, please change this parameter according to your installation.
- Increase this value to make the drift detector less sensitive.
From the deployment overview page, select your
cifar10 deployment to enter the deployment dashboard.
Inside the deployment dashboard, add a drift detector with by clicking the
Create button within the
Drift Detection widget.
Enter the following parameters in the modal popup which appears, using the default settings:
- Model Name:
Create Drift-Detector to complete the setup.
- From the deployment dashboard, click on
make new prediction.
- Run a single prediction using the tensorflow payload format of an image truck.
Next, run a single prediction using the tensorflow payload format of an image of an outlier truck image.
Monitor Drift On The Monitor Screen
Under the ‘Monitor’ section of your deployment, you can see a timeline of drift requests.
This becomes clearer if you make many requests over a period of time.