Authorization¶
Concepts and configuration for authorization in Seldon Enterprise Platform
User Identity¶
Once a user has been authenticated, they have an identity in the form of an OIDC ID token. The token defines a user ID and groups to which the user belongs. This information can be used to authorize their actions.
The user ID is determined by a claim in the OIDC token.
This defaults to preferred_username, but can be configured through the environment variable USERID_CLAIM_KEY in deploy-values.yaml. This can be used for both human users and service accounts. If it is not possible to share claim keys for both, it is possible to use the environment variable SA_ID_CLAIM_KEY for service accounts. Note: the USERID_CLAIM_KEY is always used if present. SA_ID_CLAIM_KEY is used only if USERID_CLAIM_KEY does not exist in the token
env:
USERID_CLAIM_KEY: "preferred_username"
SA_ID_CLAIM_KEY: "preferred_username"
The groups are determined by a claim in the OIDC token.
This defaults to groups, but can be configured through the environment variable GROUPS_CLAIM_KEY in deploy-values.yaml.
env:
GROUPS_CLAIM_KEY: "groups"
Hint
Claim is a technical term in the OpenID specification. It means a piece of information about a user, or “entity”. You can read more about claims here.
Permission Evaluation¶
Every call to the Seldon Enterprise Platform API is protected by authorization checks. This happens regardless of whether you are interacting with it via the UI, via the Python SDK, or directly.
Seldon Enterprise Platform operates on a restrictive model. This means that a user is assumed to have no permissions except those which they have been explicitly granted.
The effective list of permissions that a user has is the union of:
Those that are granted to them by their user ID.
Those that are granted to all users.
Those that are granted to one or more of the groups to which the user belongs.
Note
There is no way to define negative permissions, i.e. that a user is denied access to a particular resource.
Resource Actions¶
There are three actions users can be granted for resources:
Read for listing available resources and viewing particular ones.
Write for creating, deleting, and modifying resources.
Grant for granting other users permissions to specific resources.
These actions do not imply one another.
For example, having write access does not also grant read access.
Each action is handled completely independently in the OPA policies, and each grant must be specified explicitly.
Resource Scopes¶
Resources in Enterprise Platform are grouped into three categories:
Namespace-scoped resources that belong to a specific Kubernetes namespace. This includes ML deployments and artifact secrets.
Project-scoped resources, including models in the Model Catalog and deployments using these. Projects are a Seldon Enterprise Platform concept, but not a Kubernetes one.
System resources, such as authorization policies themselves!
Resources can sometimes be both project-scoped and namespace-scoped. For example, an ML deployment is namespace-scoped because it exists in Kubernetes, and is also project-scoped based on models it uses. When this happens, a user needs access to both the namespace and all projects associated with that deployment to be able to view or interact with it.
Projects¶
A project defines a logical grouping of related resources.
At present, models are the only resources that are explicitly associated with projects.
Any model that does not explicitly declare its associated project will be implicitly assigned to the default project.
A project can have multiple models associated with it, but each model is only associated with a single project. Projects-to-models is a one-to-many relationship.
As deployments reference models, they are also protected by project-based policies. Deployments can reference multiple models, so can be associated with multiple different projects. A user must have suitable permissions for all projects associated with a deployment in order to interact with it.
Note
The term “model” here refers to inference models, explainers, and detectors.
Hint
It is advised to have a policy such that the default project is accessible to everyone.
Namespaces¶
A namespace defines a physical grouping of resources.
Any resources related to deployments will exist in a namespace, including deployed models, explainers, detectors, canaries, shadows, and artifact storage secrets.
A namespace can have many resources associated with it, but each resource is only associated with a single namespace. Namespaces-to-resources is a one-to-many relationship.
System¶
System resources are resources that relate to administrative concerns for the Seldon Enterprise Platform platform, instead of ML models and deployments.
Currently the only system-level resources that can be protected by access controls in the Seldon Enterprise Platform API are authorization policies themselves. As a result, the initial policies must be defined by someone with access to the Kubernetes cluster.
Associating Scopes With Kubernetes Resources¶
As namespaces are a native Kubernetes concept, there is a natural association between namespaced resources and their associated namespace.
In contrast, projects are not a native Kubernetes concept and so Seldon Enterprise Platform must record the association between a resource and its project somewhere. It does this via Kubernetes metadata: annotations and labels.
When creating deployments through the Seldon Enterprise Platform UI, you can use the pre-filled value or specify the appropriate project for your model.
When creating deployments via the Seldon Enterprise Platform API, whether directly or with the Seldon Enterprise Platform SDK, you must provide the appropriate project metadata in the resource definition.
In either case, when no project is explicitly specified then the default project will be used.
For Seldon Core v1, projects are defined via annotations at the predictor level.
Each predictor in a deployment’s spec.predictors has an annotations field that can contain annotations of the following form:
"project.seldon.io/<name of component in graph>": "<name of project>"
Example - Seldon Core v1
Given below is a heavily simplified example of a Seldon Core v1 deployment with a single-model predictor.
The key things to note are the annotations field and the metadata.namespace field.
The former corresponds to the mock-example-container model in the graph and associates it with project-a.
The latter defines the namespace to be seldon-deployments.
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: mock-deployment
namespace: seldon-deployments
spec:
name: mock-example
predictors:
- name: default
annotations:
project.seldon.io/mock-example-container: project-a
graph:
name: mock-example-container
type: MODEL
componentSpecs:
- spec:
containers:
- name: mock-example-container
image: "seldonio/mock_classifier:1.5.0"
Example - Seldon Core v2
Given below is an example of a Seldon Core v2 Model.
The key things to note here are the metadata.annotations field, which associates the model with project-a, and the metadata.namespace field, which defines the namespace to be seldon-deployments.
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: iris
namespace: seldon-deployments
annotations:
seldon.io/project: project-a
spec:
storageUri: "gs://seldon-models/mlserver/iris"
requirements:
- sklearn
memory: 100Ki
OPA Policies¶
Seldon Enterprise Platform can use Open Policy Agent (OPA) policies to determine if a user has access to a resource. OPA is popular, flexible open-source technology for defining policies. You can enable it by following the installation guide.
Enable OPA¶
To enable the use of OPA policies for the namespace and system scopes, set the following Helm variables in deploy-values.yaml:
rbac:
opa:
enabled: true
nsLabelsAuth:
enabled: false
As OPA policies cannot be used with label-based authorization, this should be disabled as shown above.
However, Seldon Enterprise Platform still requires the seldon.restricted label to be present on namespaces in order to detect them.
When using OPA policies, it does not matter whether the value of this label is true or false.
If you also want to enable OPA for projects, set the following configuration in deploy-values.yaml:
rbac:
opa:
projectAuthEnabled: true
Policy Management APIs¶
Seldon Enterprise Platform has API endpoints for managing OPA policies. These can be found in the API documentation.
By default, these are enabled and access can be granted to them on a per-user/group basis. If you want to disable these, set the following Helm value:
rbac:
opa:
permissionManagementAPIDisabled: true
Policy Schema¶
OPA policies are defined in a JSON document as resource-action pairs for users or groups. This JSON document exists in a ConfigMap, as discussed in the installation guide.
The structure of policies is as follows:
{
"role_grants": {
"<group name>": [
{
"resource": "<resource>",
"action": "<action>"
},
// More permissions for this group
],
// More groups
},
"user_grants": {
"<user name>": [
{
"resource": "<resource>",
"action": "<action>"
},
// More permissions for this user
],
// More users
}
}
The top-level fields of the document are the grant types: role_grants and user_grants.
These are mandatory fields and correspond to groups and user IDs respectively.
Each grant type defines a map from user/group name to a list of permissions. These user or group names are called grant targets.
Each permission is defined as a map, describing an action on a resource scope.
For example this permission allows read access to resources in a namespace called seldon:
{
"action": "read",
"resource": "namespace/seldon"
}
Hint
When you do not want to specify either user or role grants, use an empty JSON object as the value, e.g.
{
"user_grants": {},
"role_grants": {
// ...
}
}
{
"user_grants": {
// ...
},
"role_grants": {}
}
Granting Access For All Users¶
For user grants, there is a special grant target called the star user and denoted *.
This special target can be used to provide policies for all users.
To see how this can be used, please check the examples.
Note
This special star user is not a glob but rather a well-known value that Seldon Enterprise Platform understands.
Seldon Enterprise Platform does not support globbing for user and group names.
For example, a grant target like abc* would not be valid.
If you need to define permissions for a collection of users, please use role grants.
Resource Names¶
Resources are identified by the scope as a prefix and a name.
For the system scope, there is only one possible value:
system/iam
For the project scope, aside from the default project the names are determined by you:
project/defaultproject/foo…
For the namespace scope, the names should correspond to Kubernetes namespaces for deploying ML models:
namespace/seldonnamespace/seldon-gitops…
Seldon Enterprise Platform supports globbing of resource names with an asterisk (*).
It is valid to use globbing anywhere within the resource name, but not for the resource scope.
For example, these would all be allowed:
"resource": "namespace/*""resource": "namespace/seldon-*""resource": "namespace/*-gitops""resource": "project/dev*"
On the other hand, these would all be prohibited:
"resource": "*""resource": "*/*""resource": "*/seldon""resource": "name*/seldon"
Note
For more details on the specifics of each resource type, please see the namespace and project sections.
Examples¶
Giving everyone access to resources in a scope is possible using the special “star” user, *.
An example of giving all users access to the default project looks like this:
{
"role_grants": {},
"user_grants": {
"*": [
{
"action": "read",
"resource": "project/default"
},
{
"action": "write",
"resource": "project/default"
}
]
}
}
Giving a group access to resources in a scope is possible using role_grants.
An example of giving the data-scientist group access to the prod namespace looks like this:
{
"role_grants": {
"data-scientist": [
{
"action": "read",
"resource": "namespace/prod"
},
{
"action": "write",
"resource": "namespace/prod"
}
]
},
"user_grants": {}
}
Giving a user access to resources in a scope is possible using user_grants.
An example of giving the alice user access to the system/iam scope looks like this:
{
"role_grants": {},
"user_grants": {
"alice": [
{
"action": "read",
"resource": "system/iam"
},
{
"action": "write",
"resource": "system/iam"
}
]
}
}
An example of a complete policy file is given below.
In this example, users belonging to the data_scientist group will be able to create, modify, and read models from the iris project, as well as see deployments in the seldon namespace.
However, they do not have permission to create or modify deployments in that namespace.
Members of the ml_engineer group can create deployments from the iris project in the seldon namespace.
They are also allowed to interact with other resources in the seldon namespace, such as storage secrets.
Members of the ops group can view all resources in namespaces with the seldon prefix (including seldon itself), and can see the full Model Catalog.
The * user grant specifies that all users have read and write permissions for the default project and default namespace.
Only the user alice has access to the alice namespace.
{
"role_grants": {
"data_scientist": [
{
"action": "read",
"resource": "project/iris"
},
{
"action": "write",
"resource": "project/iris"
},
{
"action": "read",
"resource": "namespace/seldon"
}
],
"ml_engineer": [
{
"action": "read",
"resource": "project/iris"
},
{
"action": "write",
"resource": "project/iris"
},
{
"action": "read",
"resource": "namespace/seldon"
},
{
"action": "write",
"resource": "namespace/seldon"
}
],
"ops": [
{
"action": "read",
"resource": "namespace/seldon*"
},
{
"action": "read",
"resource": "project/*"
}
]
},
"user_grants": {
"*": [
{
"action": "read",
"resource": "project/default"
},
{
"action": "write",
"resource": "project/default"
},
{
"action": "read",
"resource": "namespace/default"
},
{
"action": "write",
"resource": "namespace/default"
}
],
"alice": [
{
"action": "read",
"resource": "namespace/alice"
},
{
"action": "write",
"resource": "namespace/alice"
}
]
}
}
Policy Operations¶
Interacting With Policies¶
The policy file can be modified in various ways:
Using
kubectlUsing Seldon’s
policy_clientcommand-line utilityUsing the permission management endpoints of the Seldon Enterprise Platform API
You can interact with the ConfigMap directly using kubectl.
If your policies are defined in a JSON file called policy.json, you can turn it into a Kubernetes manifest called policy.yaml using the following command:
kubectl -n seldon-system create configmap seldon-deploy-policies --from-file=data=policy.json --dry-run=client -o yaml > policy.yaml
This resulting policy.yaml file can be tracked under version control if you wish.
You can overwrite the existing policies by running the following:
kubectl apply -f policy.yaml
The policy_client command-line utility can be downloaded from the seldon-deploy-resources repository.
You will need to download the correct binary for your operating system and architecture, e.g. Linux AMD x64.
For full documentation of the tool you can run policy_client --help.
The permission management API is provided via the /iam/policy endpoints discussed in the API documentation.
Seldon Enterprise Platform will automatically reload the policies if it detects that they have changed. This is dependent on how the file is mounted and how quickly Kubernetes propagates changes. If you want to ensure that Seldon Enterprise Platform definitely has the latest changes, you can restart it using:
kubectl -n seldon-system rollout restart deployment seldon-deploy
Adding Policies¶
To add a new policy you can run:
policy_client addPolicy --resource=namespace/seldon --action=write --target_users=alice --target_groups=admin
The arguments target_groups and target_users can take multiple entries.
Deleting Policies¶
Similarly to adding, to remove an existing policy you can run:
policy_client removePolicy --resource=namespace/seldon --action=write --target_users=alice --target_groups=admin
Limitations¶
Using a ConfigMap for the policy file is flexible and easy to setup. However, it comes with a limitation which is true for all Kubernetes resources - it is limited to 1MB size, since that is the underlying etcd value limit.
If your use case requires a bigger size for the policy file consider mounting the file from somewhere else like an object store (S3, minio), or a database.
We are working on improving support for such use cases, which should be available in a future release.
Migrating From Namespace Labels¶
To migrate from using namespace labels for defining access requirements to a namespace you can use the following command:
policy_client sync
It will create a ConfigMap called seldon-deploy-policies in the seldon-system namespace if one is not present.
The name and namespace are configurable by flags.
It will then populate policies corresponding to the namespace labels in the cluster.
The command will not delete any existing policies in the ConfigMap.
To confirm the change has been successful you can check the ConfigMap by running:
kubectl get cm -n seldon-system seldon-deploy-policies -o jsonpath='{.data.data}' | jq
Namespace Labels¶
Warning
This approach has been deprecated and will be removed in the future. For a more complete and efficient authorization solution, please use Open Policy Agent policies.
Label-based and OPA policy-based authorization cannot be used at the same time. Attempts to do so will result in only OPA policies taking effect.
Seldon Enterprise Platform can use labels on Kubernetes namespaces to determine if a user has access to resources within that namespace. Note that namespace labels can only be used for authorizing access to namespaces, but not to projects.
To authorize a user or a group to access resources in a namespace, add labels in the following manner along with the associated permission.
seldon.user.<user name>: "write"
seldon.group.<group name>: "read"
As an example, setting permissions on a namespace for user alice, service account sd_api and group ml_engineer would involve adding labels as follows,
seldon.user.alice: "write"
seldon.user.sd_api: "read"
seldon.group.ml_engineer: "read"
Note
Service accounts are also treated as users for authorization purposes and the service account user identity can be configured using a separate token claim if needed.
Configuration Reference¶
The full list of Helm values for authorization is given below.
These can be set in deploy-values.yaml.
Please refer to the above documentation for how to configure each setting.
rbac:
opa:
enabled: true
projectAuthEnabled: true
permissionManagementAPIDisabled: false
istioPolicySyncInterval: ""
configMap: seldon-deploy-policies
nsLabelsAuth:
enabled: false