Audit logging

Audit logging is an optional part of the platform which can be enabled or disabled based on your requirements. With audit logging all requests to Seldon Enterprise Platform will be logged to a configurable place.

Configuration

Please refer to the production installation setup guide.

Log schema

Seldon Enterprise Platform uses a structured log schema. Each request to Enterprise Platform will produce a single JSON message containing the following fields:

Field	Type	Description
Host	string	The host which serves this request.
Method	string	The HTTP method (GET, POST, PUT, etc.).
Referer	string	The referring URL, if sent in the request.
RequestID	string	A randomly generated version 4 UUID.
ResponseBody	string	The response body that the server returns for this request.
ResponseSize	number	The size of the response body in bytes.
ResponseStatus	number	The HTTP response status code.
SourceAddress	string	The IP of the sender of the request.
Timestamp	string	UTC timestamp of the request.
URI	string	The endpoint called in the request.
UserAgent	string	The user agent of the caller.
UserInfo	object	The authentication information of the caller.

Where UserInfo has the following fields:

Field	Type	Description
Id	string	The ID of the caller.
Name	string	The Name of the caller.
Email	string	The Email of the caller.
Groups	array of strings	The Groups that the caller belongs to.

Example log entry:

{
  "Host": "5.6.7.8",
  "Method": "POST",
  "Referer": "https://5.6.7.8/seldon-deploy/deployments/dashboard?ns=seldon&name=iris&kind=SeldonDeployment",
  "RequestID": "ce5f10a2-c874-4115-881e-70e7c5e8eb3a",
  "ResponseBody": "{\"data\":{\"result\":[{\"metric\":{\"deployment_name\":\"iris\",\"model_image\":\"seldonio/sklearnserver\",\"model_name\":\"iris-container\",\"model_version\":\"1.10.0\",\"predictor_name\":\"default\"},\"values\":[[1632312180,\"NaN\"],[1632312210,\"NaN\"],[1632312240,\"NaN\"],[1632312270,\"NaN\"],[1632312300,\"NaN\"],[1632312330,\"NaN\"],[1632312360,\"NaN\"],[1632312390,\"NaN\"],[1632312420,\"NaN\"],[1632312450,\"NaN\"],[1632312480,\"NaN\"],[1632312510,\"NaN\"],[1632312540,\"NaN\"],[1632312570,\"NaN\"],[1632312600,\"NaN\"],[1632312630,\"NaN\"],[1632312660,\"NaN\"],[1632312690,\"NaN\"],[1632312720,\"NaN\"],[1632312750,\"NaN\"],[1632312780,\"NaN\"],[1632312810,\"NaN\"],[1632312840,\"NaN\"],[1632312870,\"NaN\"],[1632312900,\"NaN\"],[1632312930,\"NaN\"],[1632312960,\"NaN\"],[1632312990,\"NaN\"],[1632313020,\"NaN\"],[1632313050,\"NaN\"],[1632313080,\"NaN\"]]}],\"resultType\":\"matrix\"},\"status\":\"success\"}",
  "ResponseSize": 821,
  "ResponseStatus": 200,
  "SourceAddress": "1.2.3.4",
  "Timestamp": "2021-09-22T12:18:19.269456495Z",
  "URI": "/seldon-deploy/api/v1alpha1/namespaces/seldon/seldondeployments/iris/monitor",
  "UserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
  "UserInfo": {
    "Id": "seldon",
    "Name": "Hari Seldon",
    "Email": "[email protected]",
    "Groups": [
      "data-scientist"
    ]
  }
}

Operations

Enterprise Platform exports Prometheus metrics that can be used to diagnose the operational health of the audit logging pipeline.

The metric is a counter called deploy_audit_logs_total. It has 2 additional labels:

• output - one of writer (indicating writing to stdout) or fluentd (indicating forwarding to Fluentd).

• status - one of success or failure.

Additionally, Fluentd can be configured to export metrics itself to help observe the rest of the storage pipeline - how Fluentd stores the logs in your desired storage solution. Please refer to the official documentation for setting this up.

Limitations

Audit logging in Seldon Enterprise Platform covers all of its endpoints, including model metadata management and deployment management. Any requests outside of Seldon Enterprise Platform will not be present in the audit logs. This includes requests that are made directly to the ML deployments managed by Enterprise Platform.