Seldon Core v2

This page contains instructions and details for the installation of Seldon Core v2.

Prerequisite: Kafka

Kafka is required to run Seldon Core v2 Pipelines.

Warning

Kafka is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the Kafka instance used by Seldon.

For production installation we highly recommend using managed Kafka instance.

Strimzi Operator

Note

This page uses Strimzi Operator to install and configure Kafka integration. This will get you up and running quickly. For integrating with managed Kafka solutions for production see our Kafka Integration page.

Strimzi provides K8s Operator that can be used to managed Kafka installations in your K8s cluster. You can install Strimzi Operator using Helm:

kubectl create ns kafka || echo "Namespace kafka already exists"
helm repo add strimzi https://strimzi.io/charts/
helm install kafka-operator strimzi/strimzi-kafka-operator --namespace kafka --version 0.35.1

Once Strimzi Operator is up and running create Kafka cluster.

Create kafka.yaml file

kafka.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: seldon
  namespace: kafka
spec:
  kafka:
    replicas: 3
    version: 3.3.1
    config:
      auto.create.topics.enable: true
      default.replication.factor: 1
      inter.broker.protocol.version: 3.3
      min.insync.replicas: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral

and apply it with:

kubectl apply -f kafka.yaml

You should now see following pods running in kafka namespace

seldon-kafka-0                              1/1     Running   1 (80s ago)   2m2s
seldon-kafka-1                              1/1     Running   1 (80s ago)   2m2s
seldon-kafka-2                              1/1     Running   1 (80s ago)   2m2s
seldon-zookeeper-0                          1/1     Running   0             2m2s
strimzi-cluster-operator-566948f58c-lthbs   1/1     Running   0             11h

Core v2 Installation

Seldon Core v2 can be installed using published Helm charts. To add Helm charts run:

helm repo add seldon-charts https://seldonio.github.io/helm-charts
helm repo update seldon-charts

The Seldon Core v2 installation consists of a few different components, each of these having its own corresponding Helm chart.

Helm Chart

Description

Recommended Namespace

seldon-core-v2-crds

CRDs defining Core v2 resources

default or seldon-system

seldon-core-v2-setup

Core v2 configuration chart (operator, templates, RBAC)

seldon-system for cluster-wide installation, or each model namespace for namespaced installations, e.g. seldon

seldon-core-v2-runtime

Seldon Runtime defines core components required in each model namespace

each model namespace, e.g. seldon

seldon-core-v2-servers

Seldon Core v2 pre-configured servers to host your models (optional)

each model namespace, e.g. seldon

Installation Modes

Seldon Core v2 supports both cluster-wide and namespaced installations:

  • In cluster-wide mode, we recommend installing the seldon-core-v2-setup Helm Chart into the seldon-system namespace. The operator will then reconcile Core v2 resources like SeldonRuntime, Server, Model, and Pipeline in all namespaces.

  • In namespaced mode, you must install the seldon-core-v2-setup Helm chart into each model namespace. Each operator will then reconcile Core v2 resources only in the namespace it is installed in itself.

Note

Cluster-wide installation of Seldon Core v2 is only available from version 2.6.0 onwards. Installation of the Seldon Core operator into the seldon-system namespace (i.e. the same namespace as Core v1) is only available from version 2.7.0.

For a namespaced installation, we will use seldon as an exemplary namespace throughout this page to install Core v2. You’d have to repeat these steps for each namespace you want to use Core v2 in. In case of a cluster-wide installation, the seldon namespace is an exemplary namespace for only the Seldon Runtime and Servers.

CRDs

Install Seldon Core v2 CRDs with:

helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
    --version 2.8.2 \
    --namespace default \
    --install

Operator

The Seldon Core v2 operator (seldon-core-v2-setup Helm chart) can be installed either in cluster-wide or namespaced mode.

Prepare the required namespaces with:

kubectl create ns seldon || echo "Namespace seldon already exists"
kubectl create ns seldon-system || echo "Namespace seldon-system already exists"

Create components-values.yaml file that we will use to configure the installation. The values below are meant as a starting point and should be edited where necessary:

components-values.yaml
controller:
  clusterwide: true

dataflow:
  resources:
    cpu: 500m

envoy:
  service:
    type: ClusterIP

kafka:
  bootstrap: "seldon-kafka-bootstrap.kafka:9092"
  topics:
    numPartitions: 4

opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP

serverConfig:
  mlserver:
    resources:
      cpu: 1
      memory: 2Gi

  triton:
    resources:
      cpu: 1
      memory: 2Gi

serviceGRPCPrefix: "http2-"

and conduct Helm installation with:

helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.8.2 \
    -f components-values.yaml \
    --namespace seldon-system \
    --install

Hint

See our Kafka Integration page for information how to configure integration with managed Kafka solutions.

Seldon Runtime

Conduct Helm installation of Seldon Runtime for Seldon Core v2 with:

helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
    --version 2.8.2 \
    --namespace seldon \
    --install

Servers

In order to run models, you will need to provision a server or more. As a convenience for getting started, you can install pre-configured Seldon Core v2 Servers. To do this, we first need to create servers-values.yaml that we will use to configure the installation. Create servers-values.yaml file (below are just the default values, adjust them to your needs):

servers-values.yaml
mlserver:
  replicas: 1

triton:
  replicas: 1

and conduct Helm installation with:

helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
    --version 2.8.2 \
    -f servers-values.yaml \
    --namespace seldon \
    --install

Validation

You should see a pod like the following running in the seldon-system namespace:

seldon-v2-controller-manager-7db857ccc-s9xxn   1/1     Running   0          5h25m

And also pods like the following running in the seldon namespace:

mlserver-0                                   3/3     Running   0             7h49m
seldon-dataflow-engine-54bc74bd87-rhgs9      1/1     Running   1             7h49m
seldon-envoy-75b44947bd-q9hxm                1/1     Running   0             7h49m
seldon-hodometer-6d9dbf689c-lg8mw            1/1     Running   0             7h49m
seldon-modelgateway-7b9ddfc644-q2knl         1/1     Running   0             7h49m
seldon-pipelinegateway-7f6f4ffd6-fzhpk       1/1     Running   0             7h49m
seldon-scheduler-0                           1/1     Running   0             7h49m
triton-0                                     3/3     Running   0             7h49m

Note

In case of namespaced installation, the seldon-v2-controller-manager pod (with a hash suffix) will be found in seldon namespace.

Adding new namespaces

Note

Do not install the CRDs again as these are global resources.

To install Seldon Core v2 in additional namespaces, you need to do these steps:

  1. Create the new namespace

  2. Install the runtime and servers into new the namespace

Configuration

Enable Core v2 support in Seldon Enterprise Platform

To enable Core v2 support in Seldon Enterprise Platform, add or change the following variable in deploy-values.yaml file:

seldonCoreV2:
  enabled: true

Once you modify your deploy-values.yaml you need to apply it with:

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f deploy-values.yaml \
    --namespace=seldon-system \
    --version 2.3.1 \
    --install

And make sure that the Seldon Enterprise Platform deployment restarts to pick up the new configuration:

kubectl rollout status deployment/seldon-deploy -n seldon-system

Expose Seldon Mesh

The Seldon Core v2 inference API is exposed via the seldon-mesh service in the seldon namespace

NAME          TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)                       AGE
seldon-mesh   ClusterIP      10.24.3.84   <none>          80:31979/TCP,9003:31803/TCP   15h

If you install Core v2 into multiple namespaces, you may need to expose the seldon-mesh service in each of them.

For the example below, we will expose the service via Istio or NGINX Ingress (depending on your installation) in the seldon namespace.

To expose this service via Istio, you need to create a VirtualService. Create the file seldon-mesh-vs.yaml with the following:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: seldon-mesh
  namespace: seldon
spec:
  gateways:
    - istio-system/seldon-gateway
  hosts:
    - "*"
  http:
    - name: "data-plane-seldon"
      match:
        - authority:
            exact: "seldon.inference.seldon"
      route:
        - destination:
            host: "seldon-mesh.seldon.svc.cluster.local"
            port:
              number: 80
    - name: "control-plane-seldon"
      match:
        - authority:
            exact: "seldon.admin.seldon"
      route:
        - destination:
            host: "seldon-scheduler.seldon.svc.cluster.local"
            port:
              number: 9004

And apply it with:

kubectl apply -f seldon-mesh-vs.yaml

Enable Metrics Monitoring

Note

Make sure that you installed and configured Prometheus Operator according to our documentation.

To configure metrics collection create the following PodMonitor resources:

PODMONITOR_RESOURCE_LOCATION=https://raw.githubusercontent.com/SeldonIO/seldon-core/v2.8.2/prometheus/monitors

kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/agent-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/envoy-servicemonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/pipelinegateway-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/server-podmonitor.yaml

Enable ML Data Events Logging

Note

Make sure that you that you followed documentation on enabling ML Data Events Logging.

To enable ML Data Events Logging for Seldon Core v2 models you need to configure Metronome to connect to Kafka. To do it add or change a following variables in deploy-values.yaml file

requestLogger:
  kafka_consumer:
    enabled: true
    bootstrap_servers: "seldon-kafka-bootstrap.kafka.svc.cluster.local:9092"

Once you modify your deploy-values.yaml you need to apply it with

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f deploy-values.yaml \
    --namespace=seldon-system \
    --version 2.3.1 \
    --install

and make sure that Seldon Enterprise Platform deployment restarts to pick up the new configuration

kubectl rollout status deployment/seldon-deploy -n seldon-system

See our Kafka Integration page for information how to configure integration with managed Kafka solutions.