ML data events monitoring

Parse and log machine learning data for advanced monitoring usecases

Seldon Enterprise Platform uses an event-driven architecture to manage all ML data payloads. All models deployed as Seldon Deployments, have the ability to forward the prediction payloads to a user defined URI as events, where a separate component can receive, parse and log them into a database for model monitoring features (more on this below). Alibi-detect powered outlier detection and drift detection features in Seldon Enterprise Platform may also log their detection payloads to the same URI. These events conform to the cloud-events specifications, which enables creating, parsing, sending, and receiving events in any supported programming language and protocol.

For managing these events, Knative Eventing is utilized and a Knative broker is set up to receive these events such that multiple consumers can subscribe to these events and can be processed asynchronously. A good example is the prediction requests event payload is sent for logging the event itself, but also sent to the outlier detection for data monitoring and alerting.

By default, Seldon Enterprise Platform uses its own event and payload logger component called metronome, which enriches request payloads and also splits out batch requests so that they are individually logged. This allows requests to be displayed in the UI and selected for running explanations. this component is also responsible for creating the correct database mappings, defining the payload schema and storing inferred feature data based on ML prediction schema.

The machine learning event payloads are parsed and stored in a format that can be then utilised by Seldon Enterprise Platform to create advanced monitoring capabilities. These features include prediction data monitoring, feature distributions monitoring, outlier and drift detection metrics monitoring.