Meta-Operator (Preview Alpha)¶

Custom installation crafted to your needs using Seldon Meta Operator

Warning

This is alpha preview. The Seldon Meta Operator is currently in alpha phase and it is not production ready. This documentation is provided as preview only in order to gather feedback and encourage testing in development environments.

Seldon Meta-Operator Overview¶

The Seldon Meta-Operator is a cloud native installation method for the Seldon Enterprise Products that provides an alternative method to efficiently trial/evaluate different components of the Seldon Enterprise ecosystem, as well as a robust production configuration for the Seldon components.

The primary motivation for developing the meta-operator is to solve some of the complexities and overheads involved in the installation and maintenance of the Seldon Enterprise ecosystem. The objectives of the meta-operator include:

Providing a simple, unified method to configure a full Seldon Deploy environment with minimal complexity through built-in, operator-managed components.
Providing a modular way to adopt ecosystem components by enabling feature-specific workflows in a way that is directly related to specific new ecosystem frameworks (e.g. Postgres, Elasticsearch, etc).
Minimising the overheads of maintaining Seldon Enterprise products through the cloud native operator pattern, which ensures the consistency of configurations as well as the health of integrations.

The Meta-Operator has been designed in a modular way which allows for each feature to be programmatically enabled or disabled. The high-level conceptual design can be visualised in the diagram below, and is further explained in the following sections.

component overview diagram

Built-in components

These components include Seldon Core, Seldon Deploy and currently Seldon Core Analytics. They are installed and configured by Seldon Meta-Operator.

Built-in for trial only components

These components include trial instances of Elasticsearch and PostgreSQL. They can be easily installed via the Seldon Meta-Operator for the purpose of trial and testing. Deploying these component via the Meta-Operator can get you quickly up and running but they should be replaced by a proper managed services when moving to production.

Manually installed components

These components are not managed by the Seldon Meta-Operator. Manual installation of them may be required to enable certain features. Among others these include Istio, Knative, Keycloak, ArgoCD, etc.

See the following table for components summary and ecosystem dependencies details:

Component	Built-in Component?	Notes
Seldon Core	Yes	Required for Seldon ecosystem
Seldon Deploy	Yes	Required for advanced Seldon functionality
Seldon Core Analytics	Yes	Optional - metrics, monitoring, and alerting
Istio	No	Required for external routing and access
PostgreSQL	Yes (for trial)	Optional - model metadata
Elasticsearch	Yes (for trial)	Optional - request logging
Keycloak	No	Required for restricting access; authn and authz provider
Fluentd	No	Optional - application and audit logging
Knative	No	Optional - advance monitoring features
ArgoCD	No	Optional - required for gitops
Argo Workflows	No	Optional - required for batch

Prerequisites¶

Kubernetes cluster with kubectl access with cluster-admin privileges.

Meta Operator Installation¶

Installation¶

To install Seldon Meta Operator:

kubectl create namespace seldon-system || true
kubectl apply -f https://storage.googleapis.com/seldonoperator/seldonoperator_install.yaml

Verification¶

After a moment you should see a pod running in seldon-system namespace. Run following command to verify

$ kubectl get pods -n seldon-system
NAME                                                 READY   STATUS    RESTARTS   AGE
seldon-operator-controller-manager-9fbc785bc-hwh5s   2/2     Running   0          10m

Minimal Seldon Installation¶

The minimal installation will consist only of Seldon Core and Seldon Deploy. It is recommended to start with minimal installation and build on it.

Installation¶

To install Seldon create seldon-instance.yaml manifest

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
metadata:
  name: seldonoperator-sample
  namespace: seldon-system
spec: {}

and apply it with

kubectl apply -f seldon-instance.yaml

Note

In the above manifest the spec: {} section is empty. In successive parts of this document we will be modifying the spec section as we enable / configure features. For brevity of each section only the relevant changes will be given as

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  <relevant changes>
...

After each modification to the seldon-instance.yaml manifest execution of

kubectl apply -f seldon-instance.yaml

command is required.

It is recommended to keep seldon-instance.yaml manifest under VCS.

Verification¶

Two new pods should appear in seldon-system namespace. Run following command to verify

$ kubectl get pods -n seldon-system
NAME                                                 READY   STATUS    RESTARTS   AGE
seldon-controller-manager-8457b8b5c7-wh9ls           1/1     Running   0          26s
seldon-deploy-57d947f768-9vhx5                       1/1     Running   0          19s
seldon-operator-controller-manager-9fbc785bc-hwh5s   2/2     Running   0          12m

Getting Access¶

You can now get access to Seldon Deploy through svc/seldon-deploy created in seldon-system namespace

$ kubectl get svc -n seldon-system | grep seldon-deploy
seldon-deploy                                        ClusterIP   10.96.239.72   <none>        80/TCP     116s

You use the created svc as any other in your cluster by configuring ingress of your choice. Seldon Deploy officially only support Istio ingress that can be configured as described in routing section.

Port-forwarding allows to easily get access to the Seldon Deploy UI on developer’s local machine.

Execute a following port-forward command in terminal:

kubectl port-forward -n seldon-system svc/seldon-deploy 8080:80

Open following URL in your browser: http://localhost:8080/seldon-deploy/

By default created svc is of ClusterIP type but it can be changed to LoadBalancer by adding the following configuration to the spec section of the SeldonOperator resource

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  enterprise:
    values:
      service:
        type: LoadBalancer
...

Get the LoadBalancer URL with

IP=$(kubectl get svc -n seldon-system seldon-deploy -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "http://${IP}/seldon-deploy/

Warning

Seldon strongly advise to configure routing using Istio Gateway. It is also highly advisable to enable AppAuth and using VPC for enhanced security.

After getting access to the Seldon Deploy UI you will be welcomed by the license prompt. Please enter your license now to get access to the Seldon Deploy UI.

Built-In Trial Ecosystem Frameworks¶

The Seldon Meta Operator allows to easily deploy a built-in instances of:

Prometheus (via Seldon Core Analytics)
Elasticsearch
PostgreSQL

which enables advance features such as metrics/monitoring, payload logging and metadata store.

Warning

The built-in instances of Elasticsearch and PostgreSQL are meant for development and testing. For production deployments we encourage to use managed solutions as discussed in further sections of this document.

Configuration¶

To enable the built-in features apply a following configuration to the seldon-instance.yaml resource:

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  enable:
    analytics: true      # enables managed instance of Seldon Core Analytics
    catalogue: true      # enables built-in trial instance of PostgreSQL
    payloadstore: true   # enables built-in trial instance of Elasticsearch
...

Verification¶

You can verify successful deployment of built-in features with

$ kubectl get pods -n seldon-system | grep -P "internal|analytics"
seldon-core-analytics-kube-state-metrics-79b4fd4bf7-crg77       1/1     Running             0          55s
seldon-core-analytics-prometheus-alertmanager-9bc86c99d-pjhl2   2/2     Running             0          55s
seldon-core-analytics-prometheus-node-exporter-95ps5            1/1     Running             0          55s
seldon-core-analytics-prometheus-pushgateway-7c669fdb57-5q6hc   1/1     Running             0          55s
seldon-core-analytics-prometheus-seldon-6b7c7d4dd4-4pn62        2/2     Running             0          55s
seldon-internal-opensearch-cluster-master-0                     1/1     Running             0          106s
seldon-internal-opensearch-cluster-master-1                     1/1     Running             0          106s
seldon-internal-opensearch-cluster-master-2                     1/1     Running             0          106s
seldon-internal-postgres-postgresql-0                           1/1     Running             0          85s`

Enable Routing¶

Requirements

This feature requires Istio. The responsibility of installation and maintenance of Istio lies on customer.

Advance routing can be enabled given that Istio is installed in the Kubernetes cluster. Enabling routing allows to create canary and shadow deployments. It is also prerequisite for enabling detectors.

Installation¶

The routing component requires Istio. Assuming that istioctl CLI tool is present it can be installed with:

istioctl install -y

For further context on Istio installation please refer to our main documentation and Istio documentation.

Configuration¶

To enable routing functionality apply one of following modifications to the seldon-instance.yaml depending on your use case.

By default Seldon Meta-Operator creates a simple HTTP Gateway like the one from our main documentation.

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  enable:
    routing: true
...

The built-in Gateway will be created in CRD instance namespace - in our example it is seldon-system:

$ kubectl get gateway -n seldon-system
NAMESPACE       NAME             AGE
seldon-system   seldon-gateway   84s

Configuring custom Gateway allows to configure SSL and other advance ingress features.

Given that seldon-gateway Gateway in the istio-system namespace is created by user:

$ kubectl get gateway -n istio-system
NAMESPACE       NAME             AGE
istio-system    seldon-gateway   42h

Following configuration instructs Seldon Meta-Operator to use this Gateway:

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  enable:
    routing: true
  routing:
    create_gateway: false
  core:
    values:
      istio:
        gateway: "istio-system/seldon-gateway"
  enterprise:
    values:
      virtualService:
        gateways:
          - "istio-system/seldon-gateway"
...

Please refer to Istio documentation for more details on how configure custom Istio Gateway.

Getting Access¶

One routing is enabled you can access

ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "Seldon Deploy: http://$ISTIO_INGRESS/seldon-deploy/"

Enabling Advance Monitoring¶

Requirements

This feature requires Knative and Istio. The responsibility of installation and maintenance of these components lies on customer.

Installation¶

For details on installing Istio please refer to Istio section.

To install Knative Eventing:

kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.1.0/eventing-crds.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.1.0/eventing-core.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.1.0/in-memory-channel.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.1.0/mt-channel-broker.yaml

To install Knative Serving

kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.1.0/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.1.0/serving-core.yaml
kubectl apply -f https://github.com/knative/net-istio/releases/download/knative-v1.1.0/net-istio.yaml

For furthe rcontext on Knative installation please refer to our main documentation and Knative documentation.

Configuration¶

Following configuration instructs Seldon Meta-Operator to use enable detectors features:

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  enable:
    detectors: true
...

Verification¶

To verify if broker and triger isare created

kubectl get brokers,triggers -n seldon-system
NAME                                  URL                                                                              AGE   READY   REASON
broker.eventing.knative.dev/default   http://broker-ingress.knative-eventing.svc.cluster.local/seldon-system/default   11m   True

NAME                                                         BROKER    SUBSCRIBER_URI                                                 AGE   READY   REASON
trigger.eventing.knative.dev/seldon-request-logger-trigger   default   http://seldon-request-logger.seldon-system.svc.cluster.local   11m   True

Enabling External Payloadstore¶

Requirements

This feature requires Elasticsearch. The responsibility of installation and maintenance of Elasticsearch lies on customer.

Installation¶

The details on installing Elasticsearch are covered in our main documentation.

Configuration¶

Following configuration instructs Seldon Meta-Operator how to connect to external instance of Elastic:

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  enable:
    payloadstore: true
  payloadstore:
    internal_elastic:
      use_internal: false
  enterprise:
    values:
      elasticsearch:
        url: "https://opensearch-cluster-master.seldon-logs.svc.cluster.local:9200"
        basicAuth: true
        secret:
          name: elastic-credentials
          userKey: "username"
          passwordKey: "password"
      requestLogger:
        elasticsearch:
          host: "opensearch-cluster-master.seldon-logs.svc.cluster.local"
          port: "9200"
          protocol: "https"
...

Further Configuration¶

Compatibility with Production Installation

The above documentation covers only subset of possible configuration options that Seldon Meta-Operator offers.

More documentation of various Seldon Deploy features can be find in our main Production Installation documentation. To follow the main documentation with Seldon Meta-Operator simply provide all helm values in the relevant sections of SeldonOperator CRD instance.

Seldon Meta-Operator offer same configuration flexibility as the usual helm-based installation. When following the main production installation docs skip all helm upgrade ... commands for Seldon Core and Seldon Deploy and instead add the relevant helm values as described below.

To configure Seldon Core Helm Values modify the spec.core.values section of SeldonOperator CRD instance:

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  core:
    values:
      <Seldon Core Helm Values Here>
...

To configure Seldon Deploy Helm Values modify the spec.enterprise.values section of SeldonOperator CRD instance:

apiVersion: operator.seldon.io/v1alpha1
kind: SeldonOperator
...
spec:
  enterprise:
    values:
      <Seldon Deploy Helm Values Here>
...

Removing Meta Operator¶

To remove Seldon Meta-Operator first remove SeldonOperator CRD instance

kubectl delete -f seldon-instance.yaml

Once deletion of resources is completed remove the operator

kubectl delete -f https://storage.googleapis.com/seldonoperator/seldonoperator_install.yaml

Known Issues¶

Removing Meta Operator Before Removing CRDs¶

If you remove Seldon Meta-Operator before removing all seldonoperator instances the removal process will hang on finalizers. To workaround this kubectl edit -n seldon-system seldon-operator seldonoperator-sample and manually remove the finalizers section.

Outlier / Drift Detectors Reply URL¶

When creating Outlier or Drift Detectors the default reply_url will be http://seldon-request-logger.seldon-logs. The Seldon Meta-Operator creates logger in seldon-system namespace. As currently there is no way to configure default reply_url one need to customize it manually each time when creating a new detector.