Elasticsearch

Elasticsearch Installation

Warning

ElasticSearch is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the ElasticSearch instance used by Seldon.

Initial Configuration

Copy the default Fluentd config

cp ./seldon-deploy-install/reference-configuration/efk/values-fluentd.yaml values-fluentd.yaml

As the starting fluentd configuration is crafted for Elasticsearch by Opendistro one needs to modify the elasticsearch section in the values-fluentd.yaml file:

elasticsearch:
  hosts: ['elasticsearch-master.seldon-logs.svc.cluster.local']
  logstash:
    enabled: true
    prefix: 'kubernetes_cluster'
  auth:
    enabled: false
  scheme: "http"
  sslVerify: false

Elasticsearch

Install Elasticsearch using the following script

kubectl create namespace seldon-logs || echo "namespace seldon-logs exists"

helm upgrade --install elasticsearch elasticsearch \
    --version 7.6.0 \
    --namespace seldon-logs \
    --set service.type=ClusterIP \
    --set antiAffinity="soft" \
    --repo https://helm.elastic.co \
    --set image=docker.elastic.co/elasticsearch/elasticsearch-oss

kubectl rollout status statefulset/elasticsearch-master -n seldon-logs

Fluentd

If not using auth (not by default) then set elasticsearch.auth.enabled to false in the Fluentd values file.

Install Fluentd using the following script

helm upgrade --install fluentd fluentd-elasticsearch \
    --version 10.0.1 \
    --namespace seldon-logs -f values-fluentd.yaml \
    --repo https://kokuwaio.github.io/helm-charts

Kibana (optional)

Kibana is useful for creating visualizations and dashboards for Elasticsearch. It is not required for Seldon Deploy, however users may choose to install it for debugging purposes:

helm upgrade --install kibana kibana \
    --version 7.6.0 \
    --namespace seldon-logs \
    --set service.type=ClusterIP \
    --repo https://helm.elastic.co \
    --set image=docker.elastic.co/kibana/kibana-oss

kubectl rollout status deployment/kibana-kibana -n seldon-logs

Configure Seldon Deploy

The following Helm values need to be set in deploy-values.yaml.

If you did not install Knative Eventing, requestLogger.trigger.create has to be set to false.

requestLogger:
  create: true
  elasticsearch:
    host: elasticsearch-master.seldon-logs.svc.cluster.local
    port: "9200"
    protocol: http
  trigger:
    create: true # false if not using Knative

elasticsearch:
  basicAuth: false
  url: http://elasticsearch-master.seldon-logs.svc.cluster.local:9200

Auth is not required for the Elastic-provided version of Elasticsearch. The elasticsearch.basicAuth option in the Seldon Deploy helm chart can be set to false, as illustrated above.

Authentication

If authentication is required to access your ElasticSearch cluster, you will need to configure your credentials so that Seldon can access it. To do this, you can provide your ElasticSearch user and password through a secret. By default, Seldon will look for a secret named elastic-credentials.

As an example, if we assume that ElasticSearch can be accessed using the admin / admin credentials, we could create the relevant secrets as:

ELASTIC_USER=admin
ELASTIC_PASSWORD=admin

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Authorization

The Seldon Deploy setup needs authorization on the Elasticsearch cluster to create, manage and search indexes for prediction logging and other monitoring features. The following security privileges are mandatory for proper functioning of the current Seldon Deploy features. Read more about Elasticsearch security privileges here.

• Seldon Deploy User Security privileges

Elasticsearch Privileges	Privilege Level	Index-pattern
monitor	Cluster	NA
index	Index	inference-log-*
index	Index	reference-log-*
index	Index	drift-log-*
read	Index	inference-log-*
read	Index	reference-log-*
read	Index	drift-log-*
read	Index	kubernetes_cluster-*
read	Index	*

• Seldon Request Logger User Security privileges

Elasticsearch Privileges	Privilege Level	Index-patterns
monitor	Cluster	NA
create_index	Index	inference-log-*, reference-log-*, drift-log-*
index	Index	inference-log-*, reference-log-*, drift-log-*
read	Index	inference-log-*, reference-log-*, drift-log-*
write	Index	inference-log-*, reference-log-*, drift-log-*
manage	Index	inference-log-*, reference-log-*, drift-log-*
bulk	Index	inference-log-*, reference-log-*, drift-log-*

Configure EFK Ingress (Optional)

Kibana

It can be useful to access Kibana’s UI without having to port-forward.

To expose Kibana externally it needs to have its own path. This means a custom values file:

extraEnvs:
  - name: SERVER_BASEPATH
    value: "/kibana"

That should be referenced with -f as an additional parameter on the previous helm install command.

Next, configure either an Istio VirtualService or an NGINX Ingress.

IstioNGINX Ingress

The following VirtualService for Kibana, created in kibana-vs.yaml, enables its ingress:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kibana
  namespace: seldon-logs
spec:
  gateways:
  - istio-system/seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /kibana/
    rewrite:
      uri: /
    route:
    - destination:
        host: kibana-kibana
        port:
          number: 5601

Then execute:

kubectl apply -f kibana-vs.yaml

Afterwards you can access Kibana at <your-ingress>/kibana.