Model Accuracy Metrics

Iris is the genus of flower which contains several species such as the setosa, versicolor, virginica, etc. This demo is based on Iris classification model based on flower properties like Sepal length, Sepal width, Petal length, Petal width. Here we will:

• Launch an iris classifier model

• Set up a metrics server for this particular model

• Send a request to get an iris classification

• Send feedback requests to gather accuracy metrics

Important

This demo requires Knative installation on the cluster as the metrics server will be installed as a kservice. Also the metrics server only works with classification models in this version.

Create Model

Deploy the iris classifier model from the catalogue into an appropriate namespace

1. From the Deployments page, select the CREATE button in the top right corner:

   Expand to see deploy model

   

2. Enter the deployment details in the deployment creation wizard and click Next:

   • Name: iris-accuracy

   • Type: Seldon Deployment

   • Protocol: Seldon

   Expand to see deploy model

   In the deployment creation wizard, enter a name for your new deployment (e.g. iris-accuracy). Select the namespace you would like the deployment to reside in (e.g. seldon). From the protocol dropdown menu, select Seldon and click Next.

3. Fill in details about the default predictor:

   • Runtime: SciKit Learn

   • Model URI: gs://seldon-models/v1.15.0/sklearn/iris

   Expand to see default predictor details

   

4. Click Next for the remaining steps, then click Launch.

Setup Metrics Server

Set up a metrics server with the following values:

• Detector Name: multiclass

• Storage URI: (For public google buckets, secret field is optional)

adserver.cm_models.multiclass_numeric.MultiClassNumeric

• Storage Secret: Leave empty as we are using a public bucket. (Default)

• Reply URL: http://seldon-request-logger.seldon-logs (Default)

Expand to see create metrics server

Click the CREATE button in the Metrics Server box in the deployment dashboard:

Enter the details of the metrics server and click the CREATE METRICS-SERVER button:

Make Predictions

Run a single prediction using the ndarray payload format. Make a couple of these requests at random using the predict tool in the UI.

Note

You can access the predict UI by clicking on + Predict in the navigation on the left hand side.

{
	"data": {
		"names": ["Sepal length", "Sepal width", "Petal length", "Petal Width"],
		"ndarray": [
			[6.8, 2.8, 4.8, 1.4]
		]
	}
}

The prediction response is versicolor (the second class of Iris).

{
	"data": {
		"names": [
			"t:0",
			"t:1",
			"t:2"
		],
		"ndarray": [
			[
				0.008074020139119268,
				0.7781601484223357,
				0.21376583143854502
			]
		]
	},
	"meta": {
		"requestPath": {
			"iris-accuracy-container": "seldonio/sklearnserver:1.15.0"
		}
	}
}

Send Feedback

As we saw the prediction response was versicolor (the second class of Iris). In numeric form the response is,

{
  "data": {
    "ndarray": [
      1
    ]
  }
}

Now prepare a feedback with the prediction response and the truth for the numeric metrics server as follows:

{
	"response": {
		"data": {
			"ndarray": [
				1
			]
		}
	},
	"truth": {
		"data": {
			"ndarray": [
				1
			]
		}
	}
}

Now use curl to send the feedback request to the metrics server. This feedback represents a case of true positive.

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k -H "Content-Type: application/json" https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'

You may need to include the authorization header to send the request, which you can retrieve from the curl form in the predict tool.

Expand to see retrieving the auth token header

Copy the authorization header from the curl form and paste it into the export command below.

export AUTH_TOKEN_HEADER=<paste authorization header here>
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k -H "$AUTH_TOKEN_HEADER" -H "Content-Type: application/json" https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'

Also prepare a feedback with the prediction response and the truth for the numeric metrics server such that the truth is different from the prediction.

{
	"response": {
		"data": {
			"ndarray": [
				1
			]
		}
	},
	"truth": {
		"data": {
			"ndarray": [
				0
			]
		}
	}
}

Now send a feedback which represents a case of false positives and negatives. Similarly to above, add the authorization header if required.

CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k -H "X-Auth-Token: $AUTH_TOKEN" -H "Content-Type: application/json" https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'

Monitor accuracy metrics on the Monitor Screen

Go to the monitor screen’s Prediction Accuracy tab to view all the metrics. Set the time range to view the metrics. You can see metrics like accuracy, precision, recall and specificity here. Notice the drop in accuracy metrics after the false feedback was received.

Submit batch feedback using Batch Processor component

Now we will submit a feedback as a batch using the Batch Processor component.

We will use two files, each containing 10k feedback instances:

• 90% success rate feedback file

• 40% success rate feedback file

We need to upload the files to MinIO’s data bucket. For details on interacting with MinIO UI please see Batch Demo.

Once the files are in S3 bucket, we go to the Batch Jobs screen using either the navigation bar on the side or with the button seen below on the model dashboard.

and submit batch request using the following form values for both feedback-input-90.txt and feedback-input-40.txt files. Wait for the first to complete before proceeding to the next.

Input Data Location: minio://data/feedback-input-40.txt
Output Data Location: minio://data/output-data-{{workflow.name}}.txt
Number of Workers: 15
Number of Retries: 3
Batch Size: 1
Minimum Batch Wait Interval (sec): 0
Method: Feedback
Transport Protocol: REST
Input Data Type: Raw Data
Object Store Secret Name: minio-bucket

Now go to the monitor view and observe how metrics value evolve over time.

Troubleshooting

If you experience issues with this demo, see the troubleshooting docs and also the Knative or Elasticsearch sections.