ML data and events logging¶

Important

Before starting the installation procedure, please download installation resources as explained here and make sure that all pre-requisites are satisfied.

This page also assumes that main Seldon Core and Seldon Enterprise Platform components are installed.

Install Knative Eventing¶

If you have not installed Istio as your ingress controller, you will not be able to use Knative Eventing and will need to configure Seldon Enterprise Platform accordingly.

Knative Eventing technology is utilized to create an event driven architecture for ML data events logging and is required to log prediction requests for Core V1. See Knative section for installation steps.

Install Elasticsearch Stack¶

Elasticsearch is the main data storage for various ML data event payloads. OpenSearch is available as a fully open-source as an alternative to Elasticsearch. We provide installation guides for both.

Fluentd is used for collecting application logs. We use an in-house developed component called metronome (described below) for storing the event payloads and other data as Elasticsearch documents.

Seldon Core and Enterprise Platform Configuration¶

For Seldon Core, add the following options to your core-values.yaml file:

executor:
  requestLogger:
    defaultEndpoint: "http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default"

For Seldon Enterprise Platform, add the following options to your deploy-values.yaml file:

requestLogger:
  create: true

Metronome¶

The above configuration for Seldon Enterprise Platform creates a component called Metronome, developed by Seldon. Metronome is a request logger for ML events. Metronome splits out batch payloads and enriches request payloads with additional metadata from the ML prediction schema. This component is also responsible for creating the correct database mappings, define the payload schema and storing inferred feature data.

Note

Note that not only Metronome versions 1.7.1 and below are compatible with OpenSearch.

ML Metadata¶

Metronome can optionally obtain a prediction schema from the Seldon Enterprise Platform metadata service, if enabled. This can be used to enrich prediction logs (e.g. add category name for categorical features and identify probabilistic features) for better logging and advanced monitoring.

To enable this feature, it is necessary to ensure that the Seldon Enterprise Platform instance is configured with a suitable licence. Then, follow the steps below:

For more details see the PostgreSQL setup.

Have an OIDC provider configured that supports the client credentials grant flow and enable this on a client for Metronome to use. In some OIDC providers, including Keycloak, setting up a service account is required for the client credentials auth flow. The password grant flow can be used for cases in which a client credentials flow is not supported.

Create an auth secret for the Metronome configuration as below, setting the parameters as per your environment.

kubectl create secret generic -n seldon-logs request-logger-auth \
  --from-literal=OIDC_PROVIDER="${OIDC_PROVIDER}" \
  --from-literal=CLIENT_ID="${CLIENT_ID}" \
  --from-literal=CLIENT_SECRET="${CLIENT_SECRET}" \
  --from-literal=OIDC_AUTH_METHOD="client_credentials" \
  --from-literal=OIDC_SCOPES="${OIDC_SCOPES}" \
  --dry-run=client -o yaml | kubectl apply -f -

Ensure requestLogger.deployHost and requestLogger.authSecret are set, in deploy-values.
```
requestLogger:
  authSecret: request-logger-auth
```

If you are not using app-level auth, e.g. because you are using ingress-level authentication, skip all the above steps except (1) and set the below in deploy-values.yaml:

requestLogger:
  authSecret: ""

Authentication on Elasticsearch/OpenSearch¶

The Seldon Enterprise Platform Helm values file has two options for connecting to a secured Elasticsearch database.

One is token-based authentication, which you should use if you have an auth token. This is used for OpenShift cluster logging flavour of Elasticsearch/OpenSearch

The other option is basic authentication. This requires creating a secret with the username and password for the Elasticsearch/OpenSearch database. These steps are described in the Elasticsearch installation documentation.

For Enterprise Platform this would need secrets in the namespaces seldon-logs (containing Elasticsearch and Metronome) and seldon-system (containing the Plaform itself) as Enterprise Platform would need to speak to Elasticsearch using the secret.

This could look like:

ELASTIC_USER=replaceme
ELASTIC_PASSWORD=replaceme

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Running without Knative¶

Disable Knative Triggers¶

If you run without Knative, for example because you use an ingress other than Istio, you need to disable Knative Triggers in your deploy-values.yaml file:

requestLogger:
  trigger:
    create: false

This will create a deployment of Metronome (Seldon request logger) that can still log payloads from Core v2 pipelines.

Configure Seldon Core¶

If you just run without Knative, but still use Istio and Seldon Core v1, you need to adjust the Seldon Core Helm installation to not use Knative Eventing for logging. You can do this by setting the following in your core-values.yaml:

executor:
  requestLogger:
    defaultEndpoint: "http://seldon-request-logger.seldon-logs.svc.cluster.local"

ML data payload parsing¶

Metronome will try to infer the request and response payloads, parse and insert them into Elasticsearch. If metadata is available in the Seldon Enterprise Platform model catalogue, then this will be used to enrich the payloads. Metronome supports the Seldon v1, TensorFlow, and Open Inference Protocol payloads for the prediction and alibi-detect protocols for outlier/drift detection use cases. In general, Metronome:

Assumes the first dimension of tensor based data is the batch dimension and attempts to split the data into individual request/response entries before storing them in Elasticsearch.
Passes through seldon v1 jsonData, strData and binData unmodified.

If you’re sending image or text data, the default maximum payload of Metronome, which is 300 KB, may not be enough. To change this, you can add the following to your deploy-values.yaml:

requestLogger:
  env:
    MAX_PAYLOAD_BYTES: "300000" # 300 KB, change as needed