Seldon Core v2¶

This page contains instructions and details for the installation of Seldon Core v2.

Prerequisite: Kafka¶

Kafka is required to run Seldon Core v2 Pipelines.

Warning

Kafka is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the Kafka instance used by Seldon.

For production installation we highly recommend using managed Kafka instance.

Strimzi Operator¶

Note

This page uses Strimzi Operator to install and configure Kafka integration. This will get you up and running quickly. For integrating with managed Kafka solutions for production see our Kafka Integration page.

Strimzi provides K8s Operator that can be used to managed Kafka installations in your K8s cluster. You can install Strimzi Operator using Helm:

kubectl create ns kafka || echo "Namespace kafka already exists"
helm repo add strimzi https://strimzi.io/charts/
helm install kafka-operator strimzi/strimzi-kafka-operator --namespace kafka --version 0.35.1

Once Strimzi Operator is up and running create Kafka cluster.

Create kafka.yaml file

kafka.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: seldon
  namespace: kafka
spec:
  kafka:
    replicas: 3
    version: 3.3.1
    config:
      auto.create.topics.enable: true
      default.replication.factor: 1
      inter.broker.protocol.version: 3.3
      min.insync.replicas: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral

and apply it with:

kubectl apply -f kafka.yaml

You should now see following pods running in kafka namespace

seldon-kafka-0                              1/1     Running   1 (80s ago)   2m2s
seldon-kafka-1                              1/1     Running   1 (80s ago)   2m2s
seldon-kafka-2                              1/1     Running   1 (80s ago)   2m2s
seldon-zookeeper-0                          1/1     Running   0             2m2s
strimzi-cluster-operator-566948f58c-lthbs   1/1     Running   0             11h

Core v2 Installation¶

Seldon Core v2 can be installed using published Helm charts. To add Helm charts run:

helm repo add seldon-charts https://seldonio.github.io/helm-charts
helm repo update seldon-charts

The Seldon Core v2 installation consists of a few different components, each of these having its own corresponding Helm chart.

Helm Chart	Description	Recommended Namespace
seldon-core-v2-crds	CRDs defining Core v2 resources	`default` or `seldon-system`
seldon-core-v2-setup	Core v2 configuration chart (operator, templates, RBAC)	`seldon-system` for cluster-wide installation, or each model namespace for namespaced installations, e.g. `seldon`
seldon-core-v2-runtime	Seldon Runtime defines core components required in each model namespace	each model namespace, e.g. `seldon`
seldon-core-v2-servers	Seldon Core v2 pre-configured servers to host your models (optional)	each model namespace, e.g. `seldon`

Installation Modes¶

Seldon Core v2 supports both cluster-wide and namespaced installations:

In cluster-wide mode, we recommend installing the seldon-core-v2-setup Helm Chart into the seldon-system namespace. The operator will then reconcile Core v2 resources like SeldonRuntime, Server, Model, and Pipeline in all namespaces.
In namespaced mode, you must install the seldon-core-v2-setup Helm chart into each model namespace. Each operator will then reconcile Core v2 resources only in the namespace it is installed in itself.

Note

Cluster-wide installation of Seldon Core v2 is only available from version 2.6.0 onwards. Installation of the Seldon Core operator into the seldon-system namespace (i.e. the same namespace as Core v1) is only available from version 2.7.0.

For a namespaced installation, we will use seldon as an exemplary namespace throughout this page to install Core v2. You’d have to repeat these steps for each namespace you want to use Core v2 in. In case of a cluster-wide installation, the seldon namespace is an exemplary namespace for only the Seldon Runtime and Servers.

CRDs¶

Install Seldon Core v2 CRDs with:

helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
    --version 2.8.0 \
    --namespace default \
    --install

Operator¶

The Seldon Core v2 operator (seldon-core-v2-setup Helm chart) can be installed either in cluster-wide or namespaced mode.

Prepare the required namespaces with:

kubectl create ns seldon || echo "Namespace seldon already exists"
kubectl create ns seldon-system || echo "Namespace seldon-system already exists"

Create components-values.yaml file that we will use to configure the installation. The values below are meant as a starting point and should be edited where necessary:

components-values.yaml

controller:
  clusterwide: true

dataflow:
  resources:
    cpu: 500m

envoy:
  service:
    type: ClusterIP

kafka:
  bootstrap: "seldon-kafka-bootstrap.kafka:9092"
  topics:
    numPartitions: 4

opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP

serverConfig:
  mlserver:
    resources:
      cpu: 1
      memory: 2Gi

  triton:
    resources:
      cpu: 1
      memory: 2Gi

serviceGRPCPrefix: "http2-"

and conduct Helm installation with:

helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.8.0 \
    -f components-values.yaml \
    --namespace seldon-system \
    --install

Prepare the require namespaces with:

kubectl create ns seldon || echo "Namespace seldon already exists"

Create components-values.yaml file that we will use to configure the installation. The values below are meant as a starting point and should be edited where necessary:

components-values.yaml

controller:
  clusterwide: false

dataflow:
  resources:
    cpu: 500m

envoy:
  service:
    type: ClusterIP

kafka:
  bootstrap: "seldon-kafka-bootstrap.kafka:9092"
  topics:
    numPartitions: 4

opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP

serverConfig:
  mlserver:
    resources:
      cpu: 1
      memory: 2Gi

  triton:
    resources:
      cpu: 1
      memory: 2Gi

serviceGRPCPrefix: "http2-"

and conduct Helm installation with:

helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.8.0 \
    -f components-values.yaml \
    --namespace seldon \
    --install

Hint

See our Kafka Integration page for information how to configure integration with managed Kafka solutions.

Seldon Runtime¶

Conduct Helm installation of Seldon Runtime for Seldon Core v2 with:

helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
    --version 2.8.0 \
    --namespace seldon \
    --install

Servers¶

In order to run models, you will need to provision a server or more. As a convenience for getting started, you can install pre-configured Seldon Core v2 Servers. To do this, we first need to create servers-values.yaml that we will use to configure the installation. Create servers-values.yaml file (below are just the default values, adjust them to your needs):

servers-values.yaml

mlserver:
  replicas: 1

triton:
  replicas: 1

and conduct Helm installation with:

helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
    --version 2.8.0 \
    -f servers-values.yaml \
    --namespace seldon \
    --install

Validation¶

You should see a pod like the following running in the seldon-system namespace:

seldon-v2-controller-manager-7db857ccc-s9xxn   1/1     Running   0          5h25m

And also pods like the following running in the seldon namespace:

mlserver-0                                   3/3     Running   0             7h49m
seldon-dataflow-engine-54bc74bd87-rhgs9      1/1     Running   1             7h49m
seldon-envoy-75b44947bd-q9hxm                1/1     Running   0             7h49m
seldon-hodometer-6d9dbf689c-lg8mw            1/1     Running   0             7h49m
seldon-modelgateway-7b9ddfc644-q2knl         1/1     Running   0             7h49m
seldon-pipelinegateway-7f6f4ffd6-fzhpk       1/1     Running   0             7h49m
seldon-scheduler-0                           1/1     Running   0             7h49m
triton-0                                     3/3     Running   0             7h49m

Note

In case of namespaced installation, the seldon-v2-controller-manager pod (with a hash suffix) will be found in seldon namespace.

Adding new namespaces¶

Note

Do not install the CRDs again as these are global resources.

To install Seldon Core v2 in additional namespaces, you need to do these steps:

Create the new namespace
Install the runtime and servers into new the namespace

Create the new namespace
Install the seldon-core-v2-setup Helm chart, runtime and servers into new the namespace

Configuration¶

Enable Core v2 support in Seldon Enterprise Platform¶

To enable Core v2 support in Seldon Enterprise Platform, add or change the following variable in deploy-values.yaml file:

seldonCoreV2:
  enabled: true

Once you modify your deploy-values.yaml you need to apply it with:

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f deploy-values.yaml \
    --namespace=seldon-system \
    --version 2.2.1 \
    --install

And make sure that the Seldon Enterprise Platform deployment restarts to pick up the new configuration:

kubectl rollout status deployment/seldon-deploy -n seldon-system

Expose Seldon Mesh¶

The Seldon Core v2 inference API is exposed via the seldon-mesh service in the seldon namespace

NAME          TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)                       AGE
seldon-mesh   ClusterIP      10.24.3.84   <none>          80:31979/TCP,9003:31803/TCP   15h

If you install Core v2 into multiple namespaces, you may need to expose the seldon-mesh service in each of them.

For the example below, we will expose the service via Istio or NGINX Ingress (depending on your installation) in the seldon namespace.

To expose this service via Istio, you need to create a VirtualService. Create the file seldon-mesh-vs.yaml with the following:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: seldon-mesh
  namespace: seldon
spec:
  gateways:
    - istio-system/seldon-gateway
  hosts:
    - "*"
  http:
    - name: "data-plane-seldon"
      match:
        - authority:
            exact: "seldon.inference.seldon"
      route:
        - destination:
            host: "seldon-mesh.seldon.svc.cluster.local"
            port:
              number: 80
    - name: "control-plane-seldon"
      match:
        - authority:
            exact: "seldon.admin.seldon"
      route:
        - destination:
            host: "seldon-scheduler.seldon.svc.cluster.local"
            port:
              number: 9004

And apply it with:

kubectl apply -f seldon-mesh-vs.yaml

To expose this service via NGINX ingress, you need to create an Ingress. Create the file seldon-mesh-ingress.yaml with the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: seldon-mesh
  namespace: seldon
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
spec:
  tls:
  rules:
  - host: seldon.inference.seldon
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: seldon-mesh
            port:
              number: 80
  - host: seldon.admin.seldon
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: seldon-scheduler
            port:
              number: 9004
  ingressClassName: nginx

and apply it with

kubectl apply -f seldon-mesh-ingress.yaml

Enable Metrics Monitoring¶

Note

Make sure that you installed and configured Prometheus Operator according to our documentation.

To configure metrics collection create the following PodMonitor resources:

PODMONITOR_RESOURCE_LOCATION=https://raw.githubusercontent.com/SeldonIO/seldon-core/v2.8.0/prometheus/monitors

kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/agent-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/envoy-servicemonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/pipelinegateway-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/server-podmonitor.yaml

Enable ML Data Events Logging¶

Note

Make sure that you that you followed documentation on enabling ML Data Events Logging.

To enable ML Data Events Logging for Seldon Core v2 models you need to configure Metronome to connect to Kafka. To do it add or change a following variables in deploy-values.yaml file

requestLogger:
  kafka_consumer:
    enabled: true
    bootstrap_servers: "seldon-kafka-bootstrap.kafka.svc.cluster.local:9092"

Once you modify your deploy-values.yaml you need to apply it with

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f deploy-values.yaml \
    --namespace=seldon-system \
    --version 2.2.1 \
    --install

and make sure that Seldon Enterprise Platform deployment restarts to pick up the new configuration

kubectl rollout status deployment/seldon-deploy -n seldon-system

See our Kafka Integration page for information how to configure integration with managed Kafka solutions.