Istio Ingress

Configure Seldon ingress with Istio

Istio is a production-ready service mesh solution, which Seldon Deploy uses for routing of traffic. Similarly, traffic can also be managed by Istio for dynamically created Seldon models, which simplifies how users can send requests to a single endpoint.

Important

Before starting the installation procedure, please download installation resources as explained here and make sure that all pre-requisites are satisfied.

This page also assumes that main Seldon Core and Seldon Deploy components are installed.

You can use the Istio CLI to install the Istio containers into your cluster. Istio provides multiple installation configuration profiles, it is important to choose the most appropriate configuration, especially as Istio is often used for applications beyond just Seldon itself. You can read more about the profiles and configurations in the Istio documentation.

Installing Istio

Warning

This section is for users installing Istio for the first time.

If you already have Istio, but a newer version is required, please refer to the official documentation for details on upgrading an existing installation instead.

Either way, you will still need to configure Seldon for Istio after installing or upgrading.

First, obtain istioctl CLI that will help in installation of Istio. We will be installing version 1.17.1 which is supported with kubernetes clusters version 1.23 …1.26. The compatibility matrix between Istio and different version of kubernetes can be found here.

export ISTIO_VERSION=1.17.1
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} sh -

./istio-${ISTIO_VERSION}/bin/istioctl install --set profile=default -y
./istio-${ISTIO_VERSION}/bin/istioctl verify-install

It’s worth emphasising that Istio in itself is quite a feature-rich open source project, which is often leveraged as central service mesh for clusters. If the use-cases are more specific, there are quite a lot of different configurations that can be chosen, examples covered above include the different installation profiles, but also there are instructions like the documentation in the Istio section for Knative.

Installing Seldon with Istio Configuration Enabled

In order to install Seldon with Istio enabled there are a couple of requirements, namely:

  • An Istio gateway needs to be created

  • Seldon Core Helm Chart needs to be installed with the Istio values

  • Seldon Deploy Helm Chart needs to be installed with Istio values

Setting up Istio Gateway

Istio Gateway is required to access deployments and Seldon Deploy itself

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: http
        number: 80
        protocol: HTTP

To setup the https-enabled ingress, please, follow official Istio documentation.

The Istio gateway can be created in the Istio namespace, but it can also be created in a different namespace, you will need to make sure that the configuration of Seldon Deploy and Core are also aligned to the namespace and gateway name.

Install Seldon Core with Istio Enabled

You can now add the following values in your core-values.yaml file. You need to make sure that the value for the gateway is <namespace>/<gatewayname><namespace>.

istio:
  enabled: true
  gateway: "istio-system/seldon-gateway"

Install Seldon Deploy with Istio Enabled

Similarly you would add following entries to your deploy-values.yaml file. Remember to update {{ISTIO_INGRESS}} with the correct address, which can be found as described below.

ingressGateway:
  seldonIngressService: "istio-ingressgateway"
  ingressNamespace: "istio-system"

virtualService:
  create: true
  gateways:
    - istio-system/seldon-gateway

seldon:
  curlForm: |
    curl -k https://{{ISTIO_INGRESS}}/seldon/{{ .Namespace }}/{{ .ModelName }}/api/v0.1/predictions \<br/>
    &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
    &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
    &nbsp;&nbsp;-d '{{ .Payload }}'
  tensorFlowCurlForm: |
    curl -k https://{{ISTIO_INGRESS}}/seldon/{{ .Namespace }}/{{ .ModelName }}/v1/models/:predict \<br/>
    &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
    &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
    &nbsp;&nbsp;-d '{{ .Payload }}'
    
seldonCoreV2:
  curlForm: |
    curl -k https://{{ISTIO_INGRESS}}/v2/models/{{ .ModelName }}/infer \<br/>
    &nbsp;&nbsp;-H "Host: {{ .Namespace }}.inference.seldon" \<br/>
    &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
    &nbsp;&nbsp;-H "Seldon-Model: {{ .ModelName }}.pipeline" \<br/>
    &nbsp;&nbsp;-d '{{ .Payload }}'
  enabled: true
  requestForm: '{{ .SeldonProtocol }}://seldon-mesh.{{ .Namespace }}.svc.cluster.local/v2/pipelines/{{
    .ModelName }}/infer'

Find address of your Seldon Deploy

Following script can help you find address of your Seldon Deploy instance running with Istio:

ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "Seldon Deploy: http://$ISTIO_INGRESS/seldon-deploy/"