ML data events logging¶

Important

Before starting the installation procedure, please download installation resources as explained here and make sure that all pre-requisites are satisfied.

This page also assumes that main Seldon Core and Seldon Deploy components are installed.

Install Knative Eventing¶

If you’ve installed a different ingress controller than Istio, you’ll not be able to use Knative Eventing and will need to configure Seldon Deploy accordingly. See Running without Istio (as an ingress controller) for more details.

Knative Eventing technology is utilized to create an event driven architecture for ML data events logging. See Knative section for installation steps.

Install Elasticsearch Stack¶

Elasticsearch is the main data storage for various ML data event payloads. OpenSearch is available as a fully open-source as an alternative to Elasticsearch. We provide installation guides for both.

Fluentd is used for collecting application logs. We use an in-house developed component called metronome (described below) for storing the event payloads and other data as Elasticsearch documents.

Seldon Core and Deploy Configuration¶

For Seldon Core add to your core-values.yaml following options

executor:
  requestLogger:
    defaultEndpoint: "http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default"

For Seldon Deploy add to your deploy-values.yaml following options

requestLogger:
  create: true

Metronome¶

The Seldon Deploy install with the above mentioned configuration deploys a component known as metronome, which is a request logger for ML data payload logging developed by Seldon. Metronome splits out batch payloads and enriches request payloads with additional metadata from the ML prediction schema. This component is also responsible for creating the correct database mappings, define the payload schema and storing inferred feature data.

Accessing ML Metadata from Seldon Deploy API¶

Metronome can optionally obtain a prediction schema from Seldon Deploy Metadata service if enabled. This can be used to enrich prediction logs (e.g. add category name for categorical features and identify probabilistic features) for better logging and advanced monitoring.

To enable this feature, it is necessary to ensure that the Seldon Deploy instance needs to be configured with a valid licence. And then follow the steps below,

Ensure model metadata storage feature is enabled i.e metadata.pg.enabled is set to true. For more details see postgres setup.
Have an auth provider that supports the client credentials grant flow and enable this on a client. In some OIDC providers, including Keycloak, setting up a service account is required for the client credentials auth flow. The password grant flow can be used for cases in which a client credentials flow is not supported.

Create an auth secret for the metronome configuration as below, setting the parameters as per your environment.

kubectl create secret generic {request-logger-auth-secret-name} -n seldon-logs \
  --from-literal=OIDC_PROVIDER="${OIDC_PROVIDER}" \
  --from-literal=CLIENT_ID="${CLIENT_ID}" \
  --from-literal=CLIENT_SECRET="${CLIENT_SECRET}" \
  --from-literal=OIDC_AUTH_METHOD="client_credentials" \
  --from-literal=OIDC_SCOPES="${OIDC_SCOPES}" \
  --dry-run=client -o yaml | kubectl apply -f -

Ensure requestLogger.deployHost and requestLogger.authSecret are set. Default Helm chart values should be fine.
```
requestLogger:
  authSecret: { request-logger-auth-secret-name }
```

If you are not using app-level auth, e.g. because you are using ingress-level authentication, skip all the above steps except (1) and set the below in deploy-values.yaml:

requestLogger:
  authSecret: ""

Authentication on Elasticsearch¶

The Seldon Deploy Helm values file has two options for connecting to a secured Elasticsearch database.

One is token-based authentication. Use this if you have an auth token. This is used for openshift cluster logging flavour of elastic .

The other option is basic authentication. Elasticsearch can be configured with basic auth. Similarly, this needs to be applied to Fluentd.

If Kibana has been installed, this requires an xpack feature and creds will need updating in the Kibana configs - note the env vars are not quite the same.

For Deploy this would need secrets in the namespaces seldon-logs (containing Elasticsearch and metronome) and seldon-system (containing Deploy) as Deploy would need to speak to Elasticsearch using the secret.

This could look like:

ELASTIC_USER=admin
ELASTIC_PASSWORD=admin

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Running without Knative¶

Disable Knative Triggers¶

If you run without Knative, for example because you use an ingress other than Istio, you need to disable Knative Triggers in your deploy-values.yaml file:

requestLogger:
  trigger:
    create: false

This will create a deployment of Metronome (seldon request logger) that can still log payloads from Core v2 pipelines.

Configure Seldon Core¶

If you just run without Knative but still use Istio and Seldon Core v1, you need to adjust the Seldon Core Helm installation to not use Knative Eventing for logging. You can do this by setting the following in your core-values.yaml:

executor:
  requestLogger:
    defaultEndpoint: "http://seldon-request-logger.seldon-logs.svc.cluster.local"

ML data payload parsing¶

Metronome will try to infer the request and response payloads, parse and insert them into Elasticsearch. If metadata is available in the Seldon Deploy model catalogue, then this will be used to enrich the payloads. Metronome supports Seldon v1, tensorflow and Kserve v2 payloads for prediction and alibi-detect protocol for outlier/drift detection use cases. In general, Metronome:

Assumes the first dimension of tensor based data is the batch dimension and attempt to split the data into individual request/response entries before storing them in Elasticsearch
Passes through seldon v1 jsonData, strData and binData unmodified.