Seldon Core v2¶

This page contains instructions and details for the installation of Seldon Core v2.

Prerequisite: Kafka¶

Kafka is required to run Seldon Core v2 Pipelines.

Warning

Kafka is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the Kafka instance used by Seldon.

Warning

For production installation we highly recommend using managed Kafka instance.

Strimzi Operator¶

Note

This page uses Strimzi Operator to install and configure Kafka integration. This will get you up and running quickly. For integrating with managed Kafka solutions for production see our Kafka Integration page.

Strimzi provides K8s Operator that can be used to managed Kafka installations in your K8s cluster. You can install Strimzi Operator using Helm:

kubectl create ns kafka || echo "Namespace kafka already exists"
helm repo add strimzi https://strimzi.io/charts/
helm install kafka-operator strimzi/strimzi-kafka-operator -n kafka

Once Strimzi Operator is up and running create Kafka cluster.

Create kafka.yaml file

kafka.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: seldon
  namespace: kafka
spec:
  kafka:
    replicas: 3
    version: 3.3.1
    config:
      auto.create.topics.enable: true
      default.replication.factor: 1
      inter.broker.protocol.version: 3.3
      min.insync.replicas: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral

and apply it with

kubectl apply -f kafka.yaml

You should now see following pods running in kafka namespace

seldon-kafka-0                              1/1     Running   1 (80s ago)   2m2s
seldon-zookeeper-0                          1/1     Running   0             2m2s
strimzi-cluster-operator-566948f58c-lthbs   1/1     Running   0             11h

Core v2 Installation¶

Seldon Core v2 can be installed using published Helm charts. To add Helm charts run

helm repo add seldon-charts https://seldonio.github.io/helm-charts
helm repo update seldon-charts

We will be installing Seldon Core v2 into the seldon namespace. Unlike Seldon Core v1, which uses a global installation, Seldon Core v2 is installed in the namespace in which you want the inference multi-model servers to be deployed.

Important

Core v2 is installed per namespace. The namespace seldon is used here as an example.

Prepare the require namespace with:

kubectl create ns seldon || echo "Namespace seldon already exists"

CRDs¶

Install Seldon Core v2 CRDs with

helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
    --version 2.5.0 \
    --namespace seldon \
    --install

Components¶

To install Seldon Core v2 Components we first need to create components-values.yaml that we will use to configure the installation

Create components-values.yaml file

components-values.yaml

dataflow:
  resources:
    cpu: 500m

envoy:
  service:
    type: ClusterIP

kafka:
  bootstrap: "seldon-kafka-bootstrap.kafka:9092"
  topics:
    numPartitions: 4

opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP

serverConfig:
  mlserver:
    resources:
      cpu: 1
      memory: 2Gi

  triton:
    resources:
      cpu: 1
      memory: 2Gi

serviceGRPCPrefix: "http2-"

and conduct Helm installation with:

helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.5.0 \
    -f components-values.yaml \
    --namespace seldon \
    --install

See our Kafka Integration page for information how to configure integration with managed Kafka solutions.

Servers¶

To install Seldon Core v2 Servers we first need to create servers-values.yaml that we will use to configure the installation

Create servers-values.yaml file

servers-values.yaml

mlserver:
  replicas: 1

triton:
  replicas: 1

and conduct Helm installation with

helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
    --version 2.5.0 \
    -f servers-values.yaml \
    --namespace seldon \
    --install

Validation¶

You should see now following pods running in the seldon namespace

mlserver-0                                   3/3     Running   0             12h
seldon-controller-manager-5697d9f8bc-qwrmd   1/1     Running   1 (62m ago)   12h
seldon-dataflow-engine-54bc74bd87-rhgs9      1/1     Running   3 (62m ago)   12h
seldon-envoy-75b44947bd-q9hxm                1/1     Running   0             12h
seldon-hodometer-6d9dbf689c-lg8mw            1/1     Running   0             12h
seldon-modelgateway-7b9ddfc644-q2knl         1/1     Running   0             12h
seldon-pipelinegateway-7f6f4ffd6-fzhpk       1/1     Running   0             12h
seldon-scheduler-0                           1/1     Running   0             12h
triton-0                                     3/3     Running   0             12h

Adding new namespace¶

To install Seldon Core v2 in additional namespace:

create new namespace
install components and servers into new namspace

Note

Do not install CRDs again as these are global resources.

Configuration¶

Enable Core v2 support in Seldon Deploy¶

To Enable V2 support in Seldon Deploy add or change a following variable in deploy-values.yaml file

seldonCoreV2:
  enabled: true

Once you modify your deploy-values.yaml you need to apply it with

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f deploy-values.yaml \
    --namespace=seldon-system \
    --version 2.1.1 \
    --install

and make sure that Seldon Deploy deployment restarts to pick up the new configuration

kubectl rollout status deployment/seldon-deploy -n seldon-system

Expose Seldon Mesh¶

The Seldon Core v2 inference API is exposed via the seldon-mesh service in the seldon namespace

NAME          TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)                       AGE
seldon-mesh   ClusterIP      10.24.3.84   <none>          80:31979/TCP,9003:31803/TCP   15h

You can install Seldon Core v2 into multiple namespaces and would then expose the seldon-mesh service in each of them.

For the example below, we will expose the service via Istio or NGINX Ingress (depending on your installation) in the seldon namespace.

To expose this service via Istio, you need to create a VirtualService. Create the file seldon-mesh-vs.yaml with the following:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: seldon-mesh
  namespace: seldon
spec:
  gateways:
    - istio-system/seldon-gateway
  hosts:
    - "*"
  http:
    - name: "data-plane-seldon"
      match:
        - authority:
            exact: "seldon.inference.seldon"
      route:
        - destination:
            host: "seldon-mesh.seldon.svc.cluster.local"
            port:
              number: 80
    - name: "control-plane-seldon"
      match:
        - authority:
            exact: "seldon.admin.seldon"
      route:
        - destination:
            host: "seldon-scheduler.seldon.svc.cluster.local"
            port:
              number: 9004

and apply it with

kubectl apply -f seldon-mesh-vs.yaml

To expose this service via NGINX ingress, you need to create an Ingress. Create the file seldon-mesh-ingress.yaml with the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: seldon-mesh
  namespace: seldon
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
spec:
  tls:
  rules:
  - host: seldon.inference.seldon
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: seldon-mesh
            port:
              number: 80
  - host: seldon.admin.seldon
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: seldon-scheduler
            port:
              number: 9004
  ingressClassName: nginx

and apply it with

kubectl apply -f seldon-mesh-ingress.yaml

Enable Metrics Monitoring¶

Note

Make sure that you installed and configured Prometheus Operator according to our documentation.

To configure metrics collection create following PodMonitor resources

PODMONITOR_RESOURCE_LOCATION=https://raw.githubusercontent.com/SeldonIO/seldon-core/v2.5.0/prometheus/monitors

kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/agent-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/envoy-servicemonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/pipelinegateway-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/server-podmonitor.yaml

Enable ML Data Events Logging¶

Note

Make sure that you that you followed documentation on enabling ML Data Events Logging.

To enable ML Data Events Logging for Seldon Core v2 models you need to configure Metronome to connect to Kafka. To do it add or change a following variables in deploy-values.yaml file

requestLogger:
  kafka_consumer:
    enabled: true
    bootstrap_servers: "seldon-kafka-bootstrap.kafka.svc.cluster.local:9092"

Once you modify your deploy-values.yaml you need to apply it with

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f deploy-values.yaml \
    --namespace=seldon-system \
    --version 2.1.1 \
    --install

and make sure that Seldon Deploy deployment restarts to pick up the new configuration

kubectl rollout status deployment/seldon-deploy -n seldon-system

See our Kafka Integration page for information how to configure integration with managed Kafka solutions.