OpenSearch¶
Installation for OpenSearch, a fully open-source alternative to Elasticsearch.
OpenSearch Installation¶
Warning
OpenSearch is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator’s responsibility to administrate and manage the OpenSearch instance used by Seldon.
Compatibility¶
The below table summarises the compatibility between different components in the Seldon ecosystem and different release lines of OpenSearch.
Component |
Compatible with OpenSearch 1.x |
Compatible with OpenSearch 2.x |
|---|---|---|
Seldon Enterprise Platform 2.3.1 |
Yes |
Yes |
Metronome >=1.8.2 |
No |
No |
Metronome 1.1…1.8.1 |
Yes |
Yes |
Metronome 1.0 |
Yes |
No |
Fluentd |
Yes |
With a plugin |
Initial Configuration¶
Copy default Fluentd and OpenSearch Helm config files (and edit if desired)
cp ./seldon-deploy-install/reference-configuration/efk/values-fluentd.yaml values-fluentd.yaml
cp ./seldon-deploy-install/reference-configuration/efk/values-opensearch.yaml values-opensearch.yaml
cp ./seldon-deploy-install/reference-configuration/efk/values-opensearch-dashboards.yaml values-opensearch-dashboards.yaml
Ensure Required Namespaces Exist¶
We’ll be installing in the seldon-logs namespace.
We’ll also set up some config in the seldon-system namespace.
kubectl create namespace seldon-logs || echo "namespace seldon-logs exists"
kubectl create namespace seldon-system || echo "namespace seldon-system exists"
Authentication¶
Authentication is optional for OpenSearch, but enabled by default.
See the security plugin documentation for how to disable it.
Using authentication means components interacting with OpenSearch will need secrets, including Seldon Enterprise
Platform.
We’ll need secrets in the seldon-logs namespace (for the request logger) and the seldon-system namespace (for
Enterprise Platform).
Using the OpenSearch defaults, we can set:
ELASTIC_USER=admin
ELASTIC_PASSWORD=admin
kubectl create secret generic elastic-credentials -n seldon-logs \
--from-literal=username="${ELASTIC_USER}" \
--from-literal=password="${ELASTIC_PASSWORD}" \
--dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic elastic-credentials -n seldon-system \
--from-literal=username="${ELASTIC_USER}" \
--from-literal=password="${ELASTIC_PASSWORD}" \
--dry-run=client -o yaml | kubectl apply -f -
It is possible to set different credentials for the default user(s) in the internal user database. For further information on this, please refer to the Open Distro documentation, as the process is fundamentally the same; exact filesystem paths may differ.
Once updated, these credentials can be provided to Fluentd in its Helm values file.
If you have chosen to install OpenSearch Dashboards (equivalent to Kibana), you can configure the credentials to access OpenSearch using the Helm values.
OpenSearch¶
Install OpenSearch using the following script:
helm upgrade --install opensearch \
opensearch --repo https://opensearch-project.github.io/helm-charts/ \
--version 1.11.1 \
--namespace=seldon-logs \
--values=values-opensearch.yaml
You can wait for it to come up with:
kubectl rollout status -n seldon-logs statefulset/opensearch-cluster-master
OpenSearch Dashboards¶
Note
OpenSearch Dashboards is not a required dependency for Seldon Enterprise Platform and as such is optional.
OpenSearch Dashboards is equivalent to Kibana for Elasticsearch.
Install it using the following:
helm upgrade --install opensearch-dashboards \
opensearch-dashboards --repo https://opensearch-project.github.io/helm-charts/ \
--version 1.5.1 \
--namespace=seldon-logs \
--values=values-opensearch-dashboards.yaml
Fluentd¶
Install Fluentd with this:
helm upgrade --install fluentd \
fluentd-elasticsearch --repo https://kokuwaio.github.io/helm-charts \
--version 10.0.1 \
--namespace=seldon-logs \
--values=values-fluentd.yaml
Note that if alternative creds are used then these need to be set in the Fluentd Helm values file.
Configure Seldon Enterprise Platform¶
Note
Not all request logger versions are compatible with OpenSearch. Currently only metronome versions <=1.7.1 are compatible.
The following Helm values need to be set in deploy-values.yaml.
requestLogger:
create: true
image: seldonio/metronome:1.7.1 # cannot be higher than 1.7.1
elasticsearch:
host: opensearch-cluster-master.seldon-logs.svc.cluster.local
port: "9200"
protocol: https
trigger:
create: true # false if not using Knative
elasticsearch:
basicAuth: true
url: https://opensearch-cluster-master.seldon-logs.svc.cluster.local:9200
If you did not install Knative Eventing, requestLogger.trigger.create has to be set to false.
Verify Installation¶
You can access the OpenSearch cluster by port-forwarding the service and sending a basic API request:
kubectl port-forward -n seldon-logs svc/opensearch-cluster-master 9200
curl --insecure -X GET -u admin:admin https://localhost:9200
You should see output like the below:
{
"name": "opensearch-cluster-master-0",
"cluster_name": "opensearch-cluster",
"cluster_uuid": "IAbgl0dqSoiSyMuyvXpcHw",
"version": {
"distribution": "opensearch",
"number": "1.3.2",
"build_type": "tar",
"build_hash": "6febcf7b53ff189de767e460e905e9e5aeecc8cb",
"build_date": "2022-05-04T03:58:48.328641Z",
"build_snapshot": false,
"lucene_version": "8.10.1",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "The OpenSearch Project: https://opensearch.org/"
}