Model Accuracy Metrics¶

Iris is the genus of flower which contains 3 species: setosa, versicolor, and virginica. This demo is based on iris classification model based on flower properties like sepal length, sepal width, petal length, and petal width. The species are also the classes that will be used for the classification. Here we will:

Launch an iris classifier model
Set up a metrics server for this particular model
Send a request to get an iris classification
Send feedback requests to gather accuracy metrics

Important

This demo requires Knative installation on the cluster as the metrics server will be installed as a kservice. Also, the metrics server only works with classification models in this version.

Create Model¶

Deploy the iris classifier model from the catalogue into an appropriate namespace

From the Deployments page, select the Create new deployment button in the top right corner:

Expand to see deploy model
Enter the deployment details in the deployment creation wizard and click Next:
- Name: iris-accuracy
- Type: Seldon Deployment
- Protocol: Seldon
Expand to see deploy model
Fill in details about the default predictor:
- Runtime: SciKit Learn
- Model URI: gs://seldon-models/v1.18.1/sklearn/iris
Expand to see default predictor details
Click Next for the remaining steps, then click Launch.

Setup Metrics Server¶

From the Dashboard page, click on Add in the Metrics Server panel:
- Detector Name: multiclass
- Storage URI: adserver.cm_models.multiclass_numeric.MultiClassNumeric
- Storage Secret: Leave empty as we are using a public bucket
- Reply URL: Leave as default value, http://seldon-request-logger.seldon-logs
Expand to see metrics server

Make Predictions¶

Run a single prediction using the ndarray payload format. Make a couple of these requests at random using the predict tool in the UI.

Go to the predict page, using the link on the left side:

Paste in the following JSON data for the prediction, and click the Predict button:

Expand to see prediction request

{
  "data": {
    "names": [
      "Sepal length",
      "Sepal width",
      "Petal length",
      "Petal Width"
    ],
    "ndarray": [
      [
        6.8,
        2.8,
        4.8,
        1.4
      ]
    ]
  }
}

Inspect the response, to see that the class with the highest confidence is the second of the iris dataset, versicolor.

Expand to see prediction response

{
  "data": {
    "names": [
      "t:0",
      "t:1",
      "t:2"
    ],
    "ndarray": [
      [
         0.008074020139119268,
         0.7781601484223357,
         0.21376583143854502
      ]
    ]
  },
  "meta": {
    "requestPath": {
    "iris-accuracy-container": "seldonio/sklearnserver:1.18.1"
    }
  }
}

Send Feedback¶

As we saw the prediction response was versicolor (the second class of iris). In numeric form the response is,

{
  "data": {
    "ndarray": [
      1
    ]
  }
}

A feedback request consists of the returned value in a response (response.data on line 5) and a truth value (truth.data on line 12), for that data. For example, a true-positive feedback request looks like this:

{
  "response": {
    "data": {
      "ndarray": [
        1
      ]
    }
  },
  "truth": {
    "data": {
      "ndarray": [
        1
      ]
    }
  }
}

We’ll send several feedback requests to the metrics server, that we set-up earlier, for various scenarios.

Set-up the environment variable(s) for the request:

Expand to see the environment variable(s)

export CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

(Recommended) You may also need to set-up for the Authorization header:
Expand to see the authorization environment variable
1. Return to the predict page, and paste in the prediction, as you did above
2. Click the Copy as curl button, and copy the value that looks like Bearer <authorization header>
3. Set the environment variable:
```
export AUTH_TOKEN="<paste the authorization value>"
```

Scenario 1: True-positive:

Expand to see the cURL command

Request without Authorization header

curl -k \
    -H "Content-Type: application/json" \
    https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
    -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'

Request with Authorization header:

curl -k \
    -H "Content-Type: application/json" \
    -H "Authorization: $AUTH_TOKEN" \
    https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
    -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'

Response:

{
  "data": {
     "tensor": {
     "shape": [
         0
       ]
      }
    },
    "meta": {
     "requestPath": {
       "iris-accuracy-container": 1.18.1
    }
  }
}

Scenario 2: False-positive:

Expand to see the cURL command

The difference from the previous scenario is the in the truth value:

-   -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'
+   -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'

Request without Authorization header:

curl -k \
    -H "Content-Type: application/json" \
    https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
    -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'

Request with Authorization header:

curl -k \
    -H "Content-Type: application/json" \
    -H "Authorization: $AUTH_TOKEN" \
    https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
    -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'

Response:

{
  "data": {
     "tensor": {
     "shape": [
         0
       ]
      }
    },
    "meta": {
     "requestPath": {
       "iris-accuracy-container": 1.18.1
    }
  }
}

Monitor accuracy metrics on the Monitor Screen¶

Having done a prediction, metrics will begin to become available.

Go to the monitor screen’s Prediction Accuracy tab to view all the metrics.
Set the time range to view the metrics, using the “From Time” and “To Time” selectors.
You can see metrics like accuracy, precision, recall and specificity here. Notice the drop in accuracy metrics after the false feedback was received.

Submit batch feedback using Batch Processor component¶

Now we will submit a feedback as a batch using the Batch Processor component.

We will use two files, each containing 10k feedback instances:

We need to upload the files to MinIO’s data bucket. For details on interacting with MinIO UI please see Batch Demo.
Once the files are in the MinIO bucket, go to the Batch Jobs screen using either the navigation bar on the left side or with the button on the model dashboard. For each of feedback-input-90.txt and feedback-input-40.txt files, submit a batch job. Wait for the first to complete before proceeding to the next. Submit batch request using the following form values:
- Input Data Location: minio://data/feedback-input-40.txt
- Output Data Location: minio://data/output-data-40-{{workflow.name}}.txt
- Number of Workers: 15
- Number of Retries: 3
- Batch Size: 1
- Minimum Batch Wait Interval (sec): 0
- Method: Feedback
- Transport Protocol: REST
- Input Data Type: Raw Data
- Object Store Secret Name: minio-bucket-envvars
- Resources: Leave defaults
Expand to see batch job
Now go to the monitor view and observe how metrics value evolve over time.

Troubleshooting¶

If you experience issues with this demo, see the troubleshooting docs and also the Knative or Elasticsearch sections.