OpenShift Installation

Installations for OpenShift

Introduction

This document walks through the installation procedure for Seldon Deploy Advanced (SD) v2.0.0 onto Red Hat OpenShift (RHOCP) v4.11.

Note

These instructions have been tested on OpenShift 4.11. We currently do not support OpenShift 4.12. This instruction may work on OpenShift 4.8 - 4.10 but this have not been tested.

The prerequisites for the installation are aligned with usual requirements for installation / configuration of operators on the Red Hat OpenShift platform:

  • Access to the OpenShift Container Platform web console.

  • An account with the cluster-admin role.

  • Being logged in to the OpenShift Container Platform cluster as an administrator.

Note

Whenever a (*) symbol is present next to an operator version it indicates a version available in Operator Hub at the moment of writing this documentation. When following this guide the available / default versions of operators may be different. These versions are noted here for information purposes only and upstream OpenShift documentation should be consulted when in doubts.

Preparation

Creating the Seldon Namespaces

Seldon Deploy has a number of namespaces which it expects to be present, and associated with given labels.

The first namespace to create is where the Seldon Deploy controller pod will run. This is the main orchestrator of the Seldon technology stack, and expects to run in the seldon-system namespace.

oc create namespace seldon-logs
oc create namespace seldon-system

Next, create a namespace within which models can be deployed. In this documentation this is going to be called seldon, however can be configured to any name of your choosing. Create the namespace, and then add the seldon.restricted label so SD has access to it.

oc create namespace seldon
oc label namespace seldon seldon.restricted=false --overwrite=true

Dependencies

OpenShift Service Mesh

The first step within the installation process is to add OpenShift Service Mesh. This is required for the networking of all other pieces within the Seldon Deploy stack, as well as the ingress/egress for model endpoints.

Warning

Istio (OpenShift Service Mesh) is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the Istio installation used by Seldon.

Adding Operators

The initial action taken is to add the relevant operators required for the logging stack. Log into the RHOCP console and navigate to the OperatorHub, within the Operators tab.

Search for and install the following operators, with the default options:

  1. Red Hat OpenShift distributed tracing platform (provided by Red Hat) (1.39.0-3*)

  2. Kiali Operator (provided by Red Hat) (1.57.3*)

  3. Red Hat OpenShift Service Mesh (provided by Red Hat) (2.3.0*)

Configuring the ServiceMeshControlPlane

Next, configure the ServiceMeshControlPlane. Ensuring Control Plane in Version v2.0 is installed and the control plane is created within the istio-system namespace, as per the OpenShift documentation.

Add Namespaces to ServiceMeshMemberRoll

Navigate to the Red Hat OpenShift Service Mesh operator under your installed operators. Select the Istio Service Mesh Member Roll tab and create a new ServiceMeshMemberRoll in istio-system namespace with the following namespaces added to the member roll:

  • seldon

  • seldon-logs

  • seldon-kafka

  • seldon-system

Note: this is easiest done using the YAML editor.

Create Seldon’s Istio Gateway

Seldon then requires an Istio gateway to allow traffic to and from the SD controller pod as well as to enable advanced routing features like canary and shadow deployments. Create a following YAML manifest called in this example istio-gateway.yaml:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Apply it with

oc apply -f istio-gateway.yaml

Create SSL Secure Route

OpenShift clusters usually come with Let’s Encrypt certificates enabled for default ingress domain. One can create Route with the tls.termination: edge in order to re-use these certificates. Create the following YAML manifest called seldon-route.yaml:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: seldon-route
  namespace: istio-system
spec:
  host: seldon.<Ingress_Domain>
  port:
    targetPort: http2
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: edge
  to:
    kind: Service
    name: istio-ingressgateway
    weight: 100

Note

To display your default ingress domain, run the following command:

$ oc get ingresses.config/cluster -o jsonpath={.spec.domain}

After using it in the above manifest in place of <Ingress_Domain> your Istio Ingress will be exposed under

INGRESS_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath={.spec.domain})
echo https://seldon.$INGRESS_DOMAIN/

Note

The above setup terminates the SSL on edge and provides non-SSL in-cluster traffic.

Apply the above manifest with

oc apply -f seldon-route.yaml

OpenShift Serverless

Seldon Deploy uses OpenShift Serverless, in the form of Knative Serving and Knative Eventing, to power many of the advanced monitoring components associated with your deployments; namely outlier and drift detection. Without this component these features will fail to function within the platform.

Warning

Knative is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the Knative installation used by Seldon.

Serverless Operator

Once more, navigate to the Operator Hub and install the official Red Hat OpenShift Serverless (1.26.0*) operator. Install using the default options.

Install Knative Eventing

Using the Serverless Operator, and as per the OpenShift documentation, install an instance of Knative Eventing.

Install Knative Serving

Using the Serverless Operator, and as per the OpenShift documentation, install an instance of Knative Serving.

Logs Namespace and Broker

The Knative events which activate the outlier and drift detectors in the form of Knative served pods are reliant on events from the Seldon logging stack. The Seldon logging stack is installed into the seldon-logs namespace with a Knative Eventing Broker configured within it.

If you did not create this namespace yet create it now

oc create namespace seldon-logs

Create the Knative Eventing Broker, eventing-broker.yaml

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  name: default
  namespace: seldon-logs

and apply it with

oc apply -f eventing-broker.yaml

Add NetworkPolicy Resources

Finally, we need to add a couple of NetworkPolicy resources to ensure that traffic can flow from Knative Eventing and Serving to the different Seldon namespace:

Seldon Logs Namespace

To allow traffic from Knative Eventing into the Seldon Logs namespace we will create a NetworkPolicy resource. Create networkpolicy-seldon-logs.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: seldon-request-logs
  namespace: seldon-logs
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: knative-eventing

and apply it with

kubectl apply -f networkpolicy-seldon-logs.yaml
Model Namespaces

For each model namespace, we will need to create a couple of NetworkPolicy resources to ensure that traffic from both Knative Eventing and Serving can go into our model namespace. For this, first create a file named networkpolicy-detectors.yaml with the following resources:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: seldon-detectors
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: knative-eventing
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: seldon-detectors-serving
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: knative-serving

You can then apply these to your model namespace as:

oc apply -f networkpolicy-detectors.yaml -n <model-namespace>

Then apply it to the seldon namespace:

oc apply -f networkpolicy-detectors.yaml -n seldon

Elasticsearch

Elasticsearch is responsible for storing all requests and responses sent to the machine learning models hosted within Seldon Deploy. Requests and responses are forwarded to Elasticsearch by the Seldon request logging component, which also runs within the seldon-logs namespace.

Elasticsearch also stores the container logs of all running models and monitoring components hosted within Seldon Deploy. These are forwarded to Elasticsearch by Fluentd.

Warning

ElasticSearch is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the ElasticSearch instance used by Seldon.

Installing the ECK Operator

The first step to configure Elasticsearch is to add the Elasticsearch (ECK) Operator (2.5.0*) from within the Operator Hub. This operator should be installed with default options, with access to all namespaces.

Create the Elasticsearch Cluster

Navigate to the Elasticsearch (ECK) Operator operator under your installed operators. Select the Elasticsearch Cluster tab and create a new cluster in seldon-logs namespace called elasticsearch-seldon using 7.12.x version:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-seldon
  namespace: seldon-logs
spec:
  version: 7.12.1
  ...

Granting Access to Elasticsearch

In order for Seldon Deploy to access the Elasticsearch cluster there are two secrets which are required to be created. One secret in the seldon-logs namespace to allow access to the cluster for the request logger component. The other in the seldon-system namespace where the Seldon Deploy pod will be installed.

Grab the Elasticsearch password and assign it to a variable for later use.

ELASTIC_PASSWORD=$(kubectl get secret elasticsearch-seldon-es-elastic-user -n seldon-logs -o go-template='{{.data.elastic | base64decode}}')

Create the secret in the seldon-logs namespace:

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="elastic" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Create the secret in the seldon-system namespace:

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="elastic" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Container Logs Forwarding

To enable container logs visibility in Seldon Deploy we use OpenShift Logging.

Installing OpenShift Logging Operator

Follow OpenShift documentation to install Red Hat OpenShift Logging (5.5.5*) operator.

Installing ClusterLogging Component

Note

As we will be forwarding logs to the Elastic instance elasticsearch-seldon in seldon-logs namespace you can disable the internal Elasticsearch logStore and Kibana visualization components from the ClusterLogging custom resource (CR). Therefore installation of the OpenShift Elasticsearch Operator is not required.

Navigate to the Red Hat OpenShift Logging operator under your installed operators. Select the Cluster Logging tab and create an instance containing at minimum the fluentd logs collection:

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: fluentd
  managementState: Managed

Installing ClusterLogForwarder Component

First we need to create seldon-elasticsearch secret that fluentd will use to authenticate with our instance of Elastic.

Fetch elastic password

ELASTIC_PASSWORD=$(kubectl get secret elasticsearch-seldon-es-elastic-user -n seldon-logs -o go-template='{{.data.elastic | base64decode}}')

Fetch the required ca-bundle certificates

kubectl get secret -n seldon-logs elasticsearch-seldon-es-http-certs-public -o go-template='{{index .data "tls.crt" | base64decode }}' > ca-bundle.crt

Create the secret in the openshift-logging namespace:

kubectl create secret generic seldon-elasticsearch -n openshift-logging \
  --from-literal=username="elastic" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --from-file=./ca-bundle.crt \
  --dry-run=client -o yaml | kubectl apply -f -

Navigate to the Red Hat OpenShift Logging operator under your installed operators. Select the Cluster Log Forwarder tab and create an instance forwarding logs to our Elastic instance.

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  inputs:
  - name: seldon-pods
    application:
      selector:
        matchLabels:
          app.kubernetes.io/managed-by: seldon-core
  outputs:
  - name: seldon-elasticsearch
    secret:
      name: seldon-elasticsearch
    type: elasticsearch
    url: 'https://elasticsearch-seldon-es-http.seldon-logs:9200'
  pipelines:
  - name: seldon-container-logs
    inputRefs:
    - seldon-pods
    outputRefs:
    - seldon-elasticsearch

For further details refer to OpenShift documentation.

Finally, we need to add NetworkPolicy to allow traffic from openshift-logging into seldon-logs namespace. Create networkpolicy-seldoncontainerlogs.yaml with following resource:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: seldon-container-logs
  namespace: seldon-logs
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-logging

and apply it with

kubectl apply -f networkpolicy-seldoncontainerlogs.yaml

OpenShift Monitoring

OpenShift provides an out of the box monitoring stack consisting of Prometheus and Thanos, alongside the Prometheus AlertManager. This stack is configured to monitor the standard OpenShift workloads, but can be extended to collect the metrics which Seldon Deploy produces. This is done through adding a PodMonitor component to any of the namespaces where Seldon models are expected to be running.

Configuring Cluster Monitoring Stack

First, check that the OpenShift cluster has the correct configuration applied in order to monitor the standard workloads it expects to. This can be done by following the relevant OpenShift documentation.

Once cluster wide monitoring has been set up, the next configuration to add is that for user defined workloads - such as Seldon Deploy. The steps are very similar to cluster monitoring configuration, and can be completed by following the user defined workload monitoring documentation available here.

By following the OpenShift documentation you should now have these two configmaps created with enableUserWorkload enabled:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |

and that Prometheus pods in openshift-user-workload-monitoring namespace are running

$ kubectl get pods -n openshift-user-workload-monitoring
NAME                                  READY   STATUS    RESTARTS   AGE
prometheus-operator-f5b597fc4-7wzql   2/2     Running   0          17m
prometheus-user-workload-0            5/5     Running   0          17m
prometheus-user-workload-1            5/5     Running   0          17m
thanos-ruler-user-workload-0          3/3     Running   0          17m
thanos-ruler-user-workload-1          3/3     Running   0          17m

Providing Seldon Deploy Access to Prometheus

Seldon Deploy requires a token in order to access Prometheus. This can be configured by following the steps documented here.

Obtain the authentication token and save it to the text file:

SECRET=`oc get secret -n openshift-user-workload-monitoring | grep  prometheus-user-workload-token | head -n 1 | awk '{print $1 }'`
TOKEN=`echo $(oc get secret $SECRET -n openshift-user-workload-monitoring -o json | jq -r '.data.token') | base64 -d`
echo -n $TOKEN > jwt-seldon.txt

Finally, apply the token as a secret within the seldon-system namespace. This is the secret with which Seldon will authenticate itself against the Prometheus instance:

oc create secret generic jwt-seldon -n seldon-system --from-file=./jwt-seldon.txt -o yaml --dry-run=client | kubectl apply -f -

Note

Depending on the permissions of the token used above different metrics and alerts will be available for Seldon Deploy to use and display.

Adding Network Access Policies

The next step in configuring the monitoring services is to add NetworkPolicy that will allow for ingress from openshift-user-workload-monitoring namespace to any namespace containing seldon specific deployments.

Create networkpolicy-monitoring.yaml file:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: user-workload-monitoring
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-user-workload-monitoring

Apply this policy to seldon-system namespace:

oc apply -f networkpolicy-monitoring.yaml -n seldon-system

You also must apply this policy to all namespaces hosting Seldon models:

oc apply -f networkpolicy-monitoring.yaml -n <model-namespace>

Apply it to the seldon namespace now:

oc apply -f networkpolicy-monitoring.yaml -n seldon

Adding the Seldon Deploy PodMonitor

First, we add PodMonitor for metrics exposed directly on Seldon Deploy pod. Create deploy-podmonitor.yaml file:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: seldon-deploy-monitor
  namespace: seldon-system
  labels:
    app: seldon-deploy
spec:
  namespaceSelector:
    matchNames:
    - seldon-system
  selector:
    matchLabels:
      app.kubernetes.io/name: seldon-deploy
  podMetricsEndpoints:
  - port: metrics
    path: /metrics

And apply it with

oc apply -f deploy-podmonitor.yaml -n seldon-system

For any of the namespaces which Seldon deployed models are going to run within there needs to be a couple of PodMonitor resources created within that namespace. Create file seldon-podmonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: seldon-podmonitor
spec:
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: seldon-core
  podMetricsEndpoints:
  - port: metrics
    path: /prometheus
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: seldon-podmonitor-metrics-server
spec:
  selector:
    matchLabels:
      seldon.io/metrics: "true"
  podMetricsEndpoints:
  - path: /v1/metrics
    port: user-port

and apply it to all namespaces hosting Seldon models:

oc apply -f seldon-podmonitor.yaml -n <model-namespace>

Apply it to seldon namespace now

oc apply -f seldon-podmonitor.yaml -n seldon

Adding the PrometheusRules for Model Usage

For any of the namespaces which Seldon deployed models are going to run within there needs to be a PrometheusRule resource created within that namespace.

Create file model-usage-prometheus-rules.yaml which content you can find in the appendix at the end of this document. You must apply it to all namespaces that will host SeldonDeployment models

oc apply -f model-usage-prometheus-rules.yaml -n <model-namespace>

Apply it to seldon namespace now

oc apply -f model-usage-prometheus-rules.yaml -n seldon

Adding the PrometheusRules for Alerting

The alerting functionality can be configured through PrometheusRules resources.

Create files user-alerts.yaml and infra-alerts.yaml which contents you can find in the appendix at the end of this document.

Apply them to seldon-system namespace:

oc apply -n seldon-system -f user-alerts.yaml
oc apply -n seldon-system -f infra-alerts.yaml

Once OpenShift reconciles the relevant configuration changes you can verify in the Admin UI -> Observe -> Alerting -> Alerting rules that TestAlertNoActionRequired rule was created (you may need to disable Platform filter to find it).

Configuring Seldon Deploy as receiver of Alertmanager

The OpenShift documentation explains how to configure alert receivers. This can be done either:

  • using OpenShift Container Platform web console

  • using CLI to modify main alertmanager-main secret in openshift-monitoring namespace

  • creating AlertmanagerConfig custom resource (alpha preview of OpenShift feature)

Use a following configuration as an example

receivers:
- name: default-receiver
- name: deploy-webhook
  webhook_configs:
  - url: "http://seldon-deploy.seldon-system:80/seldon-deploy/api/v1alpha1/webhooks/firing-alert"
route:
  group_wait: 10s
  group_by: ['alertname']
  group_interval: 5m
  receiver: default-receiver
  repeat_interval: 3h
  routes:
  - receiver: deploy-webhook
    matchers:
    - severity =~ "warning|critical"
    - type =~ "user|infra"

And if you have OIDC provider configured

webhook_configs:
- url: "http://seldon-deploy.seldon-system:80/seldon-deploy/api/v1alpha1/webhooks/firing-alert"
  http_config:
    oauth2:
      client_id: "{{ keycloak_api_clientid }}"
      client_secret: "{{ keycloak_api_secret }}"
      scopes: [openid]
      token_url: "{{ external_protocol }}://{{ external_address }}/auth/realms/deploy-realm/protocol/openid-connect/token"

PostgreSQL for Model Catalogue

The Model Catalog acts as a registry for all models deployed onto the Seldon platform, where additional metadata can be added to allow for faster deployment, easier model re-use and provenance of metadata across your experimentation, deployment and monitoring tools. The Model Catalog persists this metadata within an instance of PostgreSQL.

Warning

PostgreSQL is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the PostgreSQL instance used by Seldon.

Our PostgreSQL documentation page contains extensive information how to configure connection to managed postgres solution. In the document here we will give an example using the built-in PostgreSQL application template provided by RHOCP.

Creating built-in PostgreSQL instance

Warning

These instructions will help you to quickly spin up a PostgreSQL instance. However, we don’t recommend using it in a production context, and should be treated as development-only.

To create postgres instance

oc new-app postgresql-persistent -n seldon-system \
    -p POSTGRESQL_USER=postgresql \
    -p POSTGRESQL_PASSWORD=postgresql \
    -p POSTGRESQL_DATABASE=metadata \
    -p DATABASE_SERVICE_NAME=metadata

Once the template is instantiated, the following Openshift/Kubernetes resources will be created to support the Model Catalog:

  • DeploymentConfiguration

  • ReplicationController

  • Postgresql pod

  • Service

  • PersistentVolumeClaim

Adding Secrets

Seldon Deploy needs to be able to authenticate to the PostgreSQL instance, and therefore a secret is created called metadata-postgres using the below command.

oc create secret generic -n seldon-system metadata-postgres \
  --from-literal=user=postgresql \
  --from-literal=password=postgresql \
  --from-literal=host=metadata.seldon-system.svc.cluster.local \
  --from-literal=port=5432 \
  --from-literal=dbname=metadata \
  --from-literal=sslmode=prefer

ArgoCD

Seldon Deploy leverages GitOps to ensure an up-to-date declarative representation of model deployments. GitOps enables changes in deployments to be tracked and deployments to be rolled back to previous states- via commits to a Git repository. The Git repository stores the SeldonDeployments which describe how to create the machine learning models on the Kubernetes cluster.

Red Hat OpenShift provides a GitOps operator, which is built on top of ArgoCD and provides an easy to install and maintain component for enabling GitOps workflows. This installation will leverage the OpenShift GitOps Operator to enable Seldon’s own GitOps functionality.

Prepare Seldon Namespace for GitOps

Each namespace in which Seldon models are meant to be deployed using GitOps needs to specially prepared. Here we will provide example for namespace called seldon-gitops:

oc create ns seldon-gitops

oc label ns seldon-gitops seldon.restricted=false --overwrite=true
oc label ns seldon-gitops seldon.gitops=enabled --overwrite=true
oc annotate ns seldon-gitops git-repo="https://github.com/<your-organization>/<private-repo>" --overwrite=true

The above configures the seldon-gitops namespace to be recognized as gitops-enabled by Seldon Deploy. Assuming that we install ArgoCD instance into the seldon-argocd namespace we need to allow seldon-gitops namespace to be managed by it:

oc label namespace seldon-gitops argocd.argoproj.io/managed-by=seldon-argocd

In addition, for every new namespace we need

  • add namespace to ServiceMeshMemberRoll (see Mesh configuration section)

  • apply NetworkPolicy, PodMonitoring and PrometheusRules (see Serverless and Monitoring configuration sections):

    oc apply -f networkpolicy-monitoring.yaml -n seldon-gitops
    oc apply -f networkpolicy-detectors.yaml -n seldon-gitops
    oc apply -f seldon-podmonitor.yaml -n seldon-gitops
    oc apply -f model-usage-prometheus-rules.yaml -n seldon-gitops
    

Installing the OpenShift GitOps Operator

The first step to configure GitOps is to add the Red Hat OpenShift GitOps (1.7.0*) operator from within the Operator Hub. This operator should be installed with default options. Please follow OpenShift documentation here.

The OpenShift GitOps Operator automatically creates an ArgoCD instance in the openshift-gitops namespace. You can use this ArgoCD instance or create a new one as we describe in the next section.

Creating ArgoCD Instance

For purpose of this documentation we will use a new ArgoCD instance. First create a new project/namespace:

oc create namespace seldon-argocd

Then, create a new Argo CD instance dedicated to Seldon following the OpenShift documentation. We recommend to make following changes to the Argo CD instance using the YAML editor:

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: seldon-argocd        # Change here
  namespace: seldon-argocd   # Change here
spec:
  server:
    route:
      enabled: true
      tls:                      # Change here
        termination: reencrypt  # Change here
  dex:
    openShiftOAuth: true
  rbac:
    policy: g, cluster-admins, role:admin  # Change here
    scopes: '[groups]'
  ...

Explanation:

  • spec.server.route.tls.termination: this can be set to re-use the SSL certificates as we did when setting seldon-route in the istio-system namespace

  • spec.server.rbac.policy: the default value there reads system:cluster-admins which in certain configurations does not provide expected admin access

Note: the above definition enables Dex OpenShift OAuth Connector that allows you to log into ArgoCD using OpenShift OAuth. OpenShift admin user (belonging to cluster-admins) group will have admin privileges in ArgoCD UI.

Your ArgoCD instance will now be available under

INGRESS_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath={.spec.domain})
echo https://seldon-argocd-server-seldon-argocd.$INGRESS_DOMAIN/

Configuring Git Repository (Seldon Deploy)

To configure our Git credentials in Seldon Deploy, we will follow these steps:

  1. Create a Kubernetes secret containing our credentials, either as a SSH key or a User / Password combination. This secret can have any arbitrary name, but must live in the same namespace as Seldon Deploy.

    If we assume that our private key is present under $GIT_SSH_PATH, we can create the credentials secret as:

    kubectl create secret generic git-creds -n seldon-system \
      --from-file=id_rsa=${GIT_SSH_PATH} \
      --from-file=known_hosts=${GIT_KNOWN_HOSTS_PATH} \
      --from-literal=passphrase="${GIT_SSHKEY_PASSPHRASE}" \
      --from-literal=username="${GIT_USER}" \
      --from-literal=email="${GIT_EMAIL}"  \
      --dry-run=client -o yaml | kubectl apply -f -
    

    Note that the passphrase field can be left empty if they SSH key doesn’t have a passphrase.

  2. Make sure that Seldon Deploy’s configuration point to our newly created secret. In particular, we verify the gitops section of the values of the Seldon Deploy Helm chart. Here, we need the gitops.argocd.enabled flag be to true, and the gitops.git.secret field to point to the right secret name. The Helm installation of Seldon Deploy is described in section further down in this document and the Helm values provided there already have GitOps enabled.

    gitops:
      git:
        secret: git-creds
      argocd:
        enabled: true
    

Configuring Git Repository (ArgoCD)

There are multiple ways in which git repository can be configured in ArgoCD. One of easiest way is to use ArgoCD UI logged in as an admin user.

Here, we provide example of configuring the repository using declarative approach assuming user/password authentication over HTTPS:

apiVersion: v1
kind: Secret
metadata:
  name: seldon-gitops-repository
  namespace: seldon-argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://github.com/<your-organization>/<private-repo>
  password: my-password
  username: my-username

For more examples refer to ArgoCD documentation.

ArgoCD Project

There are multiple ways in which AppProject can be created: OpenShift UI, ArgoCD UI or declaratively.

Create a following AppProject:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: seldon
  namespace: seldon-argocd
spec:
  description: Seldon Deploy Project
  sourceRepos:
  - https://github.com/<your-organization>/<private-repo>
  destinations:
  - namespace: seldon-gitops
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  roles:
  - name: seldon-admin
    policies:
    - p, proj:seldon:seldon-admin, applications, get, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, create, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, update, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, delete, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, sync, seldon/*, allow

ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: seldon-gitops-seldon-gitops
  namespace: seldon-argocd
spec:
  project: seldon
  destination:
    namespace: seldon-gitops
    server: https://kubernetes.default.svc
  source:
    directory:
      recurse: true
    path: seldon-gitops
    repoURL: https://github.com/<your-organization>/<private-repo>
  syncPolicy:
    automated: {}

Note

If your ArgoCD application does not follow the seldon-gitops-${namespace} naming convention, you can label the namespace accordingly:

kubectl label namespace $namespace argocdapp=${ARGO_APP_NAME} --overwrite=true

Kafka

Warning

Kafka is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the Kafka installation used by Seldon.

Install Kafka Operator

The first step to install Kafka is to install an operator that can manage Kafka cluster

Add the Red Hat Integration - AMQ Streams (2.2.0-4*) operator from within the Operator Hub. This operator should be installed with default options. AMQ Streams is based on Strimzi Operator and can read more about it in the Red Hat documentation here.

Create Kafka Cluster

Once we have Strimzi (Strimzi provided by Strimzi or AMQ Streams provided by Red Hat) operator up and running we need to create Kafka cluster.

Create seldon-kafka namespace for our Kafka cluster

oc create namespace seldon-kafka

Note

Make sure that seldon-kafka namespace is added to Service Mesh Member Roll as described in OpenShift Service Mesh section.

Select seldon-kafka project and navigate to your Kafka operator under your installed operators. Select the Kafka tab and create the Kafka cluster. Following is a minimal required configuration.

kind: Kafka
apiVersion: kafka.strimzi.io/v1beta2
metadata:
  name: seldon               # Change here
  namespace: seldon-kafka   # Change here
spec:
  kafka:
    version: 3.2.3
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      auto.create.topics.enable: true             # Change here
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 1                      # Change here
      inter.broker.protocol.version: '3.2'
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

Seldon Core V1

Seldon Core is used to serve machine learning models over REST and gRPC endpoints, using a variety of advanced deployment strategies (canaries, shadows, A/B, multi-armed bandits).

Seldon Core (v1.15.0) is available as an operator within the Operator Hub and can therefore be readily installed onto OpenShift.

Once the operator has been installed there are a number of configuration changes required to ensure smooth interaction with the wider environment of tools. This can be achieved by editing the operator’s ClusterServiceVersion in the YAML tab of newly installed operator.

The configuration parameters to edit are the deployment environment variables:

  • ISTIO_ENABLED set to true

  • EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT set to http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default

Seldon Core V2

Seldon Core V2 can be installed using published Helm charts. To add Helm charts run

helm repo add seldon-charts https://seldonio.github.io/helm-charts
helm repo update seldon-charts

We will be installing Seldon Core V2 into the seldon namespace. Opposite to how Seldon Core V1 worked, Seldon Core V2 is going to be installed in the namespace in which you want the inference multi-model servers to be deployed in.

CRDs

Install Seldon Core V2 CRDs with

helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
    --version 2.1.0 \
    --namespace seldon-system \
    --install

Note

We install CRDs into seldon-system namespace. In K8s CRDs are being installed globally so we install them only once. If you will want to install Seldon Core V2 in another namespace you do not need to re-install the CRDs.

Components

To install Seldon Core V2 Components we first need to create components-values.yaml that we will use to configure the installation

Create components-values.yaml file

components-values.yaml
envoy:
  service:
    type: ClusterIP
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

controller:
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

kafka:
  debug: all
  bootstrap: "seldon-kafka-bootstrap.seldon-kafka.svc:9092"
  topics:
    replicationFactor: 2
    numPartitions: 4

opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

dataflow:
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

modelgateway:
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

hodometer:
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

pipelinegateway:
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null

serverConfig:
  securityContext:
    runAsNonRoot: true
    runAsUser: null
    runAsGroup: null
    fsGroup: null


serviceGRPCPrefix: "http2-"

and conduct Helm installation with:

helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.1.0 \
    -f components-values.yaml \
    --namespace seldon \
    --install

Servers

To install Seldon Core V2 Servers we first need to create servers-values.yaml that we will use to configure the installation

Create servers-values.yaml file

servers-values.yaml
mlserver:
  replicas: 1

triton:
  replicas: 1

and conduct Helm installation with

helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
    --version 2.1.0 \
    -f server-values.yaml \
    --namespace seldon \
    --install

Seldon Mesh

Seldon Core V2 inference api is exposed via envoy on seldon-mesh service in seldon namespace

NAME          TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)                       AGE
seldon-mesh   LoadBalancer   10.24.3.84   <none>          80:31979/TCP,9003:31803/TCP   15h

To expose this service via Istio you need to create a following VirtualService. Create seldon-mesh-vs.yaml file

seldon-mesh-vs.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: seldon-mesh
  namespace: seldon
spec:
  gateways:
  - istio-system/seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - headers:
        namespace:
          exact: seldon
        inference:
          exact: seldon-mesh
    name: data-plane-seldon
    route:
    - destination:
        host: seldon-mesh.seldon.svc.cluster.local
        port:
          number: 80

and apply it with

kubectl apply -f seldon-mesh-vs.yaml

Note

To send HTTP requests to Seldon Mesh you will need to set two headers: -H "namespace:<namespace>" -H "inference:seldon-mesh" to reach Seldon Mesh in given namespace. For example to reach iris pipeline in seldon namespace:

INGRESS_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath={.spec.domain})

curl -k https://seldon.$INGRESS_DOMAIN/v2/pipelines/iris/infer -H "Content-Type: application/json" -H "namespace:seldon" -H "inference:seldon-mesh" -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

Seldon Deploy

Installation

To install Seldon Deploy, download the installation resources (including Helm charts) as explained here.

Seldon Deploy relies on Helm charts to perform the installation. There is a master configuration file which contains all of the relevant Helm values for the given installation, in the appendix of this document there is the recommended Helm values for the installation of Seldon Deploy on OpenShift. Save these values as values-openshift.yaml and then run the following helm command to install Seldon Deploy.

helm upgrade seldon-deploy ./seldon-deploy-install/helm-charts/seldon-deploy/ \
    -f values-openshift.yaml \
    --namespace=seldon-system \
    --install

Obtaining ingress URL

Once the Seldon Deploy pods have come up, the UI can be accessed by running the following command, and entering the resultant URI into the browser.:

INGRESS_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath={.spec.domain})
echo https://seldon.$INGRESS_DOMAIN/seldon-deploy/

Apendix

Adding new namespace for Seldon Deploy

  1. To add a new namespace for Seldon Deploy to use called my-new-namespace do

    oc create namespace my-new-namespace
    oc label namespace my-new-namespace seldon.restricted=false --overwrite=true
    
    oc apply -f networkpolicy-monitoring.yaml -n my-new-namespace
    oc apply -f networkpolicy-detectors.yaml -n my-new-namespace
    oc apply -f seldon-podmonitor.yaml -n my-new-namespace
    oc apply -f model-usage-prometheus-rules.yaml -n my-new-namespace
    
  2. Add namespace to ServiceMeshMemberRoll as described in OpenShift Service Mesh section.)

  3. If the new namespace is meant to be gitops-enabled (recommended) follow steps described in argocd section:

    • add seldon.gitops=enabled label

    • add git-repo annotation

    • add argocd.argoproj.io/managed-by label

    • update AppProject with new namespace entry

    • create new Application resource

  4. If you specified explicitly namespaces in the Cluster Log Forwarder config you need to add new namespace to the list.

Validating Installation

This subsection describes basic validation steps for the Seldon installation.

Validating Ingress

Verify that Istio Gateway and Route for Seldon is created with

$ oc get gateway -n istio-system seldon-gateway
NAME             AGE
seldon-gateway   7d20h

$ oc get route -n istio-system seldon-route
NAME           HOST/PORT                PATH   SERVICES               PORT    TERMINATION     WILDCARD
seldon-route   seldon.$INGRESS_DOMAIN          istio-ingressgateway   http2   edge/Redirect   None
Validating Serverless

Verify that Broker exist and is in READY state with

$ oc get broker -n seldon-logs
NAME      URL                                                                            AGE     READY   REASON
default   http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default   7d20h   True
Validating NetworkPolicy resources

Verify that following NetworkPolicy resources exist in seldon-system and seldon-logs namespaces:

$ oc get networkpolicy -n seldon-system
NAME                       POD-SELECTOR                   AGE
user-workload-monitoring   <none>                         7d19h

$ oc get networkpolicy -n seldon-logs
NAME                       POD-SELECTOR                   AGE
seldon-container-logs      <none>                         7d19h
seldon-request-logs        <none>                         7d20h

Verify that in every namespace with your Seldon models a following NetworkPolicy resources exist:

$ oc get networkpolicy -n <model namespace>
NAME                       POD-SELECTOR                   AGE
seldon-detectors           <none>                         7d20h
seldon-detectors-serving   <none>                         7d20h
user-workload-monitoring   <none>                         7d19h
Validating Monitoring Resources

Verify that following PodMonitor and PrometheusRules exists in seldon-system namespace:

$ oc get podmonitor,prometheusrules -n seldon-system
NAME                                                     AGE
podmonitor.monitoring.coreos.com/seldon-deploy-monitor   6d

NAME                                                       AGE
prometheusrule.monitoring.coreos.com/deploy-infra-alerts   2d21h
prometheusrule.monitoring.coreos.com/deploy-user-alerts    2d21h

Verify that following PodMonitor and PrometheusRules exist in every model namespace:

$ oc get podmonitor,prometheusrules -n <model namespace>
NAME                                                                AGE
podmonitor.monitoring.coreos.com/seldon-podmonitor                  7d20h
podmonitor.monitoring.coreos.com/seldon-podmonitor-metrics-server   7d20h

NAME                                                            AGE
prometheusrule.monitoring.coreos.com/seldon-model-usage-rules   7d20h
Validating Kafka

Verify that following pods are present in seldon-kafka namespace:

NAME                                      READY   STATUS    RESTARTS   AGE
seldon-entity-operator-5f5cc6f7ff-db9gb   3/3     Running   0          10m
seldon-kafka-0                            1/1     Running   0          11m
seldon-kafka-1                            1/1     Running   0          11m
seldon-kafka-2                            1/1     Running   0          11m
seldon-zookeeper-0                        1/1     Running   0          11m
seldon-zookeeper-1                        1/1     Running   0          11m
seldon-zookeeper-2                        1/1     Running   0          11m
Validating Seldon Core V2

Verify that following pods are present in your model namespace, e.g. seldon:

$ oc get pods -n seldon
NAME                                         READY   STATUS    RESTARTS   AGE
mlserver-0                                   3/3     Running   0             12h
seldon-controller-manager-5697d9f8bc-qwrmd   1/1     Running   0             12h
seldon-dataflow-engine-54bc74bd87-rhgs9      1/1     Running   0             12h
seldon-envoy-75b44947bd-q9hxm                1/1     Running   0             12h
seldon-hodometer-6d9dbf689c-lg8mw            1/1     Running   0             12h
seldon-modelgateway-7b9ddfc644-q2knl         1/1     Running   0             12h
seldon-pipelinegateway-7f6f4ffd6-fzhpk       1/1     Running   0             12h
seldon-scheduler-0                           1/1     Running   0             12h
triton-0                                     3/3     Running   0             12h

Seldon Deploy Helm Values

This set of helm values of Seldon Deploy is designed to work properly on OpenShift 4.9 platform with all dependencies installed and configured as described in this document.

image:
  image: seldonio/seldon-deploy-server:2.0.0

applicationLogs:
  elasticIndexPattern: "app-write"
  elasticNamespaceField: "kubernetes.namespace_name"
  elasticContainerNameField: "kubernetes.container_name"
  elasticPodNameField: "kubernetes.pod_name"

defaultUserID: ''

enableAppAuth: false
enableAppAnalytics: false

external:
  protocol: http

env:
  USERID_CLAIM_KEY: name
  ALERTMANAGER_URL: https://alertmanager-main.openshift-monitoring:9094/api/v1/alerts

gitops:
  argocd:
    enabled: true
    namespace: seldon-argocd

metadata:
 pg:
   enabled: true
   secret: "metadata-postgres"

prometheus:
  knative:
    url: http://prometheus-system-np.knative-monitoring.svc.cluster.local:8080/api/v1/
  seldon:
    # see https://github.com/openshift/cluster-monitoring-operator/issues/768
    namespaceMetricName: namespace
    serviceMetricName: exported_service

    url: https://thanos-querier.openshift-monitoring.svc:9091/api/v1/
    resourceMetricsUrl: https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/

    jwtSecretKey: jwt-seldon.txt
    jwtSecretName: jwt-seldon

elasticsearch:
  url: https://elasticsearch-seldon-es-http.seldon-logs:9200
  basicAuth: true
  secret:
    name: "elastic-credentials"
    userKey: "username"
    passwordKey: "password"

requestLogger:
  create: true
  elasticsearch:
    host: elasticsearch-seldon-es-http.seldon-logs
  kafka_consumer:
    enabled: true
    bootstrap_servers: "seldon-kafka-bootstrap.seldon-kafka.svc.cluster.local:9092"

seldon:
  enabled: true
  curlForm: |
    DOMAIN=$(oc get route -n istio-system seldon-route -o jsonpath={.spec.host})<br>
    curl -k -H "{{ .TokenHeader }}: {{ .Token }} " -H "Content-Type: application/json" https://$DOMAIN/seldon/{{ .Namespace }}/{{ .ModelName }}/api/v0.1/predictions -d '{{ .Payload }}'
  tensorFlowCurlForm: |
    DOMAIN=$(oc get route -n istio-system seldon-route -o jsonpath={.spec.host})<br>
    curl -k -H "{{ .TokenHeader }}: {{ .Token }} " -H "Content-Type: application/json" https://$DOMAIN/seldon/{{ .Namespace }}/{{ .ModelName }}/v1/models/:predict -d '{{ .Payload }}'
  kfservingV2CurlForm: |
    DOMAIN=$(oc get route -n istio-system seldon-route -o jsonpath={.spec.host})<br>
    curl -k -H "{{ .TokenHeader }}: {{ .Token }} " -H "Content-Type: application/json" https://$DOMAIN/seldon/{{ .Namespace }}/{{ .ModelName }}/v2/models/{{ .GraphModelName }}/infer -d '{{ .Payload }}'

seldonCoreV2:
  enabled: true

Prometheus Rules for Model Usage

Save file as model-usage-prometheus-rules.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: seldon-model-usage-rules
  labels:
    prometheus: k8s
    role: record-rules
spec:
  groups:
    - name: model-usage.rules
      interval: 3m
      rules:
        - record: model_containers_average
          expr: label_replace(sum by (label_seldon_deployment_id,namespace) ((sum_over_time(kube_pod_labels{label_app_kubernetes_io_managed_by=~"seldon-core"}[2m]) / scalar(max(sum_over_time(kube_pod_labels[2m])))) * on(pod,namespace) group_right(label_seldon_deployment_id) max by (namespace,pod,container,namespace) (avg_over_time(kube_pod_container_info[2m] ))), "name","$1","label_seldon_deployment_id", "(.+)")
          labels:
            type: "SeldonDeployment"
        - record: model_memory_usage_bytes
          expr: label_replace(sort_desc(sum by (label_seldon_deployment_id,namespace) ((sum_over_time(kube_pod_labels{label_app_kubernetes_io_managed_by=~"seldon-core"}[2m] ) / scalar(max(sum_over_time(kube_pod_labels[2m] )))) * on(pod,namespace) group_right(label_seldon_deployment_id) sum by (namespace,pod,container) (rate(container_memory_usage_bytes[2m] )))), "name","$1","label_seldon_deployment_id", "(.+)")
          labels:
            type: "SeldonDeployment"
        - record: model_cpu_usage_seconds_total
          expr: label_replace(sort_desc(sum by (label_seldon_deployment_id,namespace) ((sum_over_time(kube_pod_labels{label_app_kubernetes_io_managed_by=~"seldon-core"}[2m] ) / scalar(max(sum_over_time(kube_pod_labels[2m] )))) * on(pod,namespace) group_right(label_seldon_deployment_id) sum by (namespace,pod,container) (rate(container_cpu_usage_seconds_total[2m] )))), "name","$1","label_seldon_deployment_id", "(.+)")
          labels:
            type: "SeldonDeployment"
        - record: model_cpu_requests
          expr: label_replace(sort_desc(sum by (label_seldon_deployment_id,namespace) ((sum_over_time(kube_pod_labels{label_app_kubernetes_io_managed_by=~"seldon-core"}[2m] ) / scalar(max(sum_over_time(kube_pod_labels[2m] )))) * on(pod,namespace) group_right(label_seldon_deployment_id) sum by (namespace,pod,container) (kube_pod_container_resource_requests{resource="cpu",unit="core"} ))), "name","$1","label_seldon_deployment_id", "(.+)")
          labels:
            type: "SeldonDeployment"
        - record: model_cpu_limits
          expr: label_replace(sort_desc(sum by (label_seldon_deployment_id,namespace) ((sum_over_time(kube_pod_labels{label_app_kubernetes_io_managed_by=~"seldon-core"}[2m] ) / scalar(max(sum_over_time(kube_pod_labels[2m] )))) * on(pod,namespace) group_right(label_seldon_deployment_id) sum by (namespace,pod,container) (kube_pod_container_resource_limits{resource="cpu",unit="core"} ))), "name","$1","label_seldon_deployment_id", "(.+)")
          labels:
            type: "SeldonDeployment"
        - record: model_memory_requests_bytes
          expr: label_replace(sort_desc(sum by (label_seldon_deployment_id,namespace) ((sum_over_time(kube_pod_labels{label_app_kubernetes_io_managed_by=~"seldon-core"}[2m] ) / scalar(max(sum_over_time(kube_pod_labels[2m] )))) * on(pod,namespace) group_right(label_seldon_deployment_id) sum by (namespace,pod,container) (kube_pod_container_resource_requests{resource="memory",unit="byte"} ))), "name","$1","label_seldon_deployment_id", "(.+)")
          labels:
            type: "SeldonDeployment"
        - record: model_memory_limits_bytes
          expr: label_replace(sort_desc(sum by (label_seldon_deployment_id,namespace) ((sum_over_time(kube_pod_labels{label_app_kubernetes_io_managed_by=~"seldon-core"}[2m] ) / scalar(max(sum_over_time(kube_pod_labels[2m] )))) * on(pod,namespace) group_right(label_seldon_deployment_id) sum by (namespace,pod,container) (kube_pod_container_resource_limits{resource="memory",unit="byte"} ))), "name","$1","label_seldon_deployment_id", "(.+)")
          labels:
            type: "SeldonDeployment"

Prometheus Rules for Alerting

Save file as user-alerts.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: deploy-user-alerts
spec:
  groups:
  - name: deploy-user-alerts.rules
    rules:
    - alert: ModelHighErrorRate
      # This matches a regex for 5XX errors, and calculates the percentage over 5 minutes.
      # It requires a percentage higher than 30 for at least 100 requests to fire.
      expr: (sum(rate(seldon_api_executor_client_requests_seconds_count{code=~"5[0-9]{2}"}[5m])) by (seldon_deployment_id)
        /
        sum(rate(seldon_api_executor_client_requests_seconds_count[5m])) by (seldon_deployment_id) * 100.0) > 30
        and
        sum(increase(seldon_api_executor_client_requests_seconds_count[5m])) by (seldon_deployment_id) > 100
      for: 1m
      annotations:
        title: 'High error rate on deployed model.'
        description: 'Model {{ $labels.seldon_deployment_id }} has an internal error rate of greater than 30% for more than 100 requests total.'
      labels:
        severity: 'critical'
        type: 'user'
    - alert: TestAlertNoActionRequired
      expr: increase(deploy_alerting_trigger_test_alert[3m]) > 1
      for: 1m
      annotations:
        title: 'Test alert, safe to ignore.'
        description: 'This is a test alert, used to verify the alerting system is working correctly - it will resolve itself and no action is required.'
      labels:
        severity: 'warning'
        type: 'user'

Save file as infra-alerts.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: deploy-infra-alerts
spec:
  groups:
  - name: deploy-infra-alerts.rules
    rules:
    - alert: MetasyncerNotSyncing
      expr: sum(increase(deploy_metasyncer_processed_sync_total[11m])) < 1
      for: 11m
      # Sync window is 10 minutes, we measure increase over 11m and expect failure condition for 11m to prevent edge cases. Can't measure over 20m for 1m as it would alert at initial startup.
      annotations:
        title: 'The runtime metasyncer has not synced for two cycles.'
        description: 'Deployment information from Kubernetes and model metadata information from Postgres has not been synchronised for more than 20 minutes. Functionality depending on this, like project-based authorisation and the metadata API, might be affected.'
      labels:
        severity: 'critical'
        type: 'infra'
    - alert: DeployIsDown
      expr: absent(up{container="seldon-deploy"})
      for: 1m
      annotations:
        title: 'Deploy server is down.'
        description: 'Deploy is not running, check the pods in Kubernetes for status.'
      labels:
        severity: 'critical'
        type: 'infra'
    - alert: DeployApiHighErrorRate
      # This matches a regex for 5XX errors, and calculates the percentage over 5 minutes.
      # It requires a percentage higher than 30 for at least 100 requests to fire.
      expr: (sum(rate(http_request_duration_seconds_count{code=~"5[0-9]{2}"}[5m])) by (handler)
        /
        sum(rate(http_request_duration_seconds_count[5m])) by (handler) * 100.0) > 30
        and
        sum(increase(http_request_duration_seconds_count[5m])) by (handler) > 100
      for: 1m
      annotations:
        title: 'High error rate on Deploy api.'
        description: 'The Deploy api for handler {{ $labels.handler }} has an internal error rate of greater than 30% for more than 100 requests total.'
      labels:
        severity: 'critical'
        type: 'infra'
    - alert: DeployMetadataMigrationsFailed
      expr: increase(deploy_metadata_sql_migrations_total{status="failure"}[5m]) > 1
      for: 5m
      annotations:
        title: 'Deploy Metadata SQL Migrations Failed'
        description: 'Deploy has failed to perform SQL migrations on the metadata database {{ $value }} times in the last 5 minutes.'
      labels:
        severity: 'warning'
        type: 'infra'
    - alert: DeployOPADynamicPolicyUpdateFailed
      expr: increase(deploy_opa_policies_updates_total{status="failure"}[5m]) > 0
      for: 5m
      annotations:
        title: 'Deploy OPA Dynamic Policy Update Failed'
        description: 'Deploy has failed updating the OPA policies from the dynamic policy config {{ $value }} times in the last 5 minutes.'
      labels:
        severity: 'warning'
        type: 'infra'

Troubleshooting

Core V2 Pipelines

If you see an error from the producer in the Pipeline gateway complaining about not enough insync replicas then the replication factor Seldon is using is less than the cluster setting for min.insync.replicas which for a default AWS MSK cluster defaults to 2. Ensure this is equal to that of the cluster. This value can be set in the Helm chart with kafka.topics.replicationFactor.

Prometheus Metrics

If you do not see any metrics in Seldon Deploy first check if all NetworkPolicy, PodMonitor and kind: PrometheusRule resources are configured correctly. If you still do not see any metrics verify if JWT token given to Seldon Deploy is correct.

To verify token

oc run -it --rm ubuntu --image=ubuntu:latest --restart=Never -n seldon-system -- bash
apt update && apt install curl jq

$ token=...
$ curl -s -H "Authorization: Bearer $token" -k "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/targets" | jq .status
"success"
$ curl -s -H "Authorization: Bearer $token" -k "https://alertmanager-main.openshift-monitoring:9094/api/v1/alerts" | jq .status
"success"

Elasticsearch

To verify Elasticsearch credentials and if Seldon indices are being populated:

oc port-forward -n seldon-logs svc/elasticsearch-seldon-es-http 9200

Get token from Elastic secret (user is “elastic”)

oc get secret elasticsearch-seldon-es-elastic-user -n seldon-logs -o go-template='{{.data.elastic | base64decode}}'

Verify that secret matches

oc get secret -n seldon-logs elastic-credentials -o json | jq '.data | map_values(@base64d)'

and

oc get secret -n seldon-logs elastic-credentials -o json | jq '.data | map_values(@base64d)'

Verify credentials and indices

curl -k -u elastic:$token https://localhost:9200
curl -k -u elastic:$token https://localhost:9200/_cat/indices