ML data events logging¶

Important

Before starting the installation procedure, please download installation resources as explained here and make sure that all pre-requisites are satisfied.

This page also assumes that main Seldon Core and Seldon Deploy components are installed.

Install Knative Eventing¶

Knative Eventing technology is utilized to create an event driven architecture for ML data events logging. See Knative section for installation steps. Knative Eventing is recommended but see Running without Knative below if this represents a major barrier for you.

Install Elasticsearch Stack¶

Elasticsearch is the main data storage for various ML data event payloads. We suggest using the OpenDistro version of Elasticsearch for a trial installation. Other flavours are also available.

Fluentd is used for collecting application logs. We use an in-house developed component called metronome (described below) for storing the event payloads and other data as Elasticsearch documents.

Seldon Core and Deploy Configuration¶

For Seldon Core add to your core-values.yaml following options

executor:
  requestLogger:
    defaultEndpoint: "http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default"

For Seldon Deploy add to your deploy-values.yaml following options

requestLogger:
  create: true

Metronome¶

The Seldon Deploy install with the above mentioned configuration deploys a component known as metronome, which is a request logger for ML data payload logging developed by Seldon. Metronome splits out batch payloads and enriches request payloads with additional metadata from the ML prediction schema. This component is also responsible for creating the correct database mappings, define the payload schema and storing inferred feature data.

Accessing ML Metadata from Seldon Deploy API¶

Metronome can optionally obtain a prediction schema from Seldon Deploy Metadata service if enabled. This can be used to enrich prediction logs (e.g. add category name for categorical features and identify probabilistic features) for better logging and advanced monitoring.

To enable this feature, it is necessary to ensure that the Seldon Deploy instance needs to be configured with a valid licence. And then follow the steps below,

Ensure model metadata storage feature is enabled i.e metadata.pg.enabled is set to true. For more details see postgres setup.
Have an auth provider that supports the client credentials grant flow and enable this on a client. This can also be done using a password grant flow for other cases where a client credentials flow is not supported.

Create an auth secret for the metronome configuration as below, setting the parameters as per your environment.

kubectl create secret generic {request-logger-auth-secret-name} -n seldon-logs \
  --from-literal=OIDC_PROVIDER="${OIDC_PROVIDER}" \
  --from-literal=CLIENT_ID="${CLIENT_ID}" \
  --from-literal=CLIENT_SECRET="${CLIENT_SECRET}" \
  --from-literal=OIDC_AUTH_METHOD="client_credentials" \
  --from-literal=OIDC_SCOPES="${OIDC_SCOPES}" \
  --dry-run=client -o yaml | kubectl apply -f -

Ensure requestLogger.deployHost and requestLogger.authSecret are set. Default Helm chart values should be fine.
```
requestLogger:
  authSecret: { request-logger-auth-secret-name }
```

If you are not using app-level auth,e.g. because you are using ingress-level authentication, skip all the above steps except (1) and set the below in deploy-values.yaml:

requestLogger:
  authSecret: ""

Trial Installation Setup¶

In the provided trial installation, a keycloak client is configured called ‘sd-api’ and a service account is also created. The trial installation scripts create the auth secret needed and fully configure metronome and metadata automatically. But if you are installing components yourself then you need to set this up specifically. Setting up a separate client is optional as the main Deploy client could be used. In keycloak, setting up a service account is required for the client credentials auth flow.

Custom Request Logger¶

It’s possible for you to add your own custom request logger, with any custom logic you’d like to add. In order to do this, you need to make sure each Seldon Deployment points to the endpoint of your custom request logger.

For this, you will need to make sure you enable the Seldon Operator Environment variable for the logger. Prior to v1.2.x this was "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT_PREFIX" and then became "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT". It can be enabled through the core Helm chart’s executor.requestLogger.defaultEndpoint.

Below we show an example of how you would do this for our non-Knative default request logger using metronome.

Running without Knative¶

It’s also possible to set up request logging without Knative dependency. For this, you will have to run a non-Knative request logger, which you can trigger by running the configuration below.

Make sure that you edit the Elasticsearch host variable below to point to the correct Elasticsearch service address.

Important: for a normal install use the Helm chart to install the logger. But that uses Knative Eventing. If you don’t want that then you can refer to the Helm chart to create your own logger spec like the below.

To do this, first disable the requestLogger that is installed by the Helm charts by setting in the deploy-values.yaml

requestLogger:
  create: false

then create the deployment with your custom logger:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: seldon-request-logger
  namespace: seldon-logs
  labels:
    app: seldon-request-logger
spec:
  replicas: 2
  selector:
    matchLabels:
      app: seldon-request-logger
  template:
    metadata:
      labels:
        app: seldon-request-logger
    spec:
      containers:
        - name: user-container
          image: docker.io/seldonio/metronome:1.0.1
          imagePullPolicy: Always
          env:
            - name: ELASTICSEARCH_HOST
              value: "opensearch-cluster-master.seldon-logs.svc.cluster.local"
            - name: ELASTICSEARCH_PORT
              value: "9200"
            - name: ELASTICSEARCH_PROTOCOL
              value: "https"
            - name: ELASTICSEARCH_USER
              valueFrom:
                secretKeyRef:
                  name: elastic-credentials
                  key: username
            - name: ELASTICSEARCH_PASS
              valueFrom:
                secretKeyRef:
                  name: elastic-credentials
                  key: password
---
apiVersion: v1
kind: Service
metadata:
  name: seldon-request-logger
  namespace: seldon-logs
spec:
  selector:
    app: seldon-request-logger
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

For a secured Elasticsearch instance you need to set ELASTICSEARCH_PROTOCOL, ELASTICSEARCH_USER and ELASTICSEARCH_PASS that in above example are sourced from the elastic-credentials secret.

Configure Seldon Core to use that endpoint¶

In order to make sure the Seldon Deployments are sending the requests to that endpoint, you will need to make sure you provide the request logger prefix. In this case you will need the following extra attributes in the Seldon Core values.yaml:

executor:
  requestLogger:
    defaultEndpoint: "http://seldon-request-logger.seldon-logs.svc.cluster.local"

It’s important that you make sure it’s on that format, which is http://<LOGGER_SERVICE>.<LOGGER_NAMESPACE>.svc.cluster.local.

If you prefer to have one request logger per kubernetes namespace set defaultEndpoint: http://<LOGGER_SERVICE>. (note the trailing dot) - the Seldon Service Orchestrator will then add the namespace where the Seldon Deployment is running as a suffix.

Overriding Request Logging Endpoint for specific Seldon Deployment¶

Once you have created the request logger, now you have to make sure your deployments are pointing to the correct custom request logger. You can set up the custom request logger address by adding the following configuration to every Seldon Core SeldonDeployment file:

      logger:
        mode: all
        url: http://seldon-request-logger.default

The mode configuration can be set to request, response or all. The url is where this should be logged to.

Authentication on Elasticsearch¶

The Seldon Deploy Helm values file has two options for connecting to a secured Elasticsearch database.

One is token-based authentication. Use this if you have an auth token. This is used for openshift cluster logging flavour of elastic .

The other option is basic authentication. Elasticsearch can be configured with basic auth. Similarly, this needs to be applied to Fluentd.

If Kibana has been installed, this requires an xpack feature and creds will need updating in the Kibana configs - note the env vars are not quite the same.

For Deploy this would need secrets in the namespaces seldon-logs (containing Elasticsearch and metronome) and seldon-system (containing Deploy) as Deploy would need to speak to Elasticsearch using the secret.

This could look like:

ELASTIC_USER=admin
ELASTIC_PASSWORD=admin

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

ML data payload parsing¶

Metronome will try to infer the request and response payloads, parse and insert them into Elasticsearch. If metadata is available in the Seldon Deploy model catalogue, then this will be used to enrich the payloads. Metronome supports Seldon v1, tensorflow and Kserve v2 payloads for prediction and alibi-detect protocol for outlier/drift detection use cases. In general, metronome:

Assumes the first dimension of tensor based data is the batch dimension and attempt to split the data into individual request/response entries before storing them in Elasticsearch
Passes through seldon v1 jsonData, strData and binData unmodified.