Introducing Hierarchical Namespaces

Author: Adrian Ludwin (Google)

Safely hosting large numbers of users on a single Kubernetes cluster has always been a troublesome task. One key reason for this is that different organizations use Kubernetes in different ways, and so no one tenancy model is likely to suit everyone. Instead, Kubernetes offers you building blocks to create your own tenancy solution, such as Role Based Access Control (RBAC) and NetworkPolicies; the better these building blocks, the easier it is to safely build a multitenant cluster.

Namespaces for tenancy

By far the most important of these building blocks is the namespace, which forms the backbone of almost all Kubernetes control plane security and sharing policies. For example, RBAC, NetworkPolicies and ResourceQuotas all respect namespaces by default, and objects such as Secrets, ServiceAccounts and Ingresses are freely usable within any one namespace, but fully segregated from other namespaces.

Namespaces have two key properties that make them ideal for policy enforcement. Firstly, they can be used to represent ownership. Most Kubernetes objects must be in a namespace, so if you use namespaces to represent ownership, you can always count on there being an owner.

Secondly, namespaces have authorized creation and use. Only highly-privileged users can create namespaces, and other users require explicit permission to use those namespaces - that is, create, view or modify objects in those namespaces. This allows them to be carefully created with appropriate policies, before unprivileged users can create “regular” objects like pods and services.

The limits of namespaces

However, in practice, namespaces are not flexible enough to meet some common use cases. For example, let’s say that one team owns several microservices with different secrets and quotas. Ideally, they should place these services into different namespaces in order to isolate them from each other, but this presents two problems.

Firstly, these namespaces have no common concept of ownership, even though they’re both owned by the same team. This means that if the team controls multiple namespaces, not only does Kubernetes not have any record of their common owner, but namespaced-scoped policies cannot be applied uniformly across them.

Secondly, teams generally work best if they can operate autonomously, but since namespace creation is highly privileged, it’s unlikely that any member of the dev team is allowed to create namespaces. This means that whenever a team wants a new namespace, they must raise a ticket to the cluster administrator. While this is probably acceptable for small organizations, it generates unnecessary toil as the organization grows.

Introducing hierarchical namespaces

Hierarchical namespaces are a new concept developed by the Kubernetes Working Group for Multi-Tenancy (wg-multitenancy) in order to solve these problems. In its simplest form, a hierarchical namespace is a regular Kubernetes namespace that contains a small custom resource that identifies a single, optional, parent namespace. This establishes the concept of ownership across namespaces, not just within them.

This concept of ownership enables two additional types of behaviours:

• Policy inheritance: if one namespace is a child of another, policy objects such as RBAC RoleBindings are copied from the parent to the child.
• Delegated creation: you usually need cluster-level privileges to create a namespace, but hierarchical namespaces adds an alternative: subnamespaces, which can be manipulated using only limited permissions in the parent namespace.

This solves both of the problems for our dev team. The cluster administrator can create a single “root” namespace for the team, along with all necessary policies, and then delegate permission to create subnamespaces to members of that team. Those team members can then create subnamespaces for their own use, without violating the policies that were imposed by the cluster administrators.

Hands-on with hierarchical namespaces

Hierarchical namespaces are provided by a Kubernetes extension known as the Hierarchical Namespace Controller, or HNC. The HNC consists of two components:

• The manager runs on your cluster, manages subnamespaces, propagates policy objects, ensures that your hierarchies are legal and manages extension points.
• The kubectl plugin, called kubectl-hns, makes it easy for users to interact with the manager.

Both can be easily installed from the releases page of our repo.

Let’s see HNC in action. Imagine that I do not have namespace creation privileges, but I can view the namespace team-a and create subnamespaces within it¹. Using the plugin, I can now say:

$ kubectl hns create svc1-team-a -n team-a

This creates a subnamespace called svc1-team-a. Note that since subnamespaces are just regular Kubernetes namespaces, all subnamespace names must still be unique.

I can view the structure of these namespaces by asking for a tree view:

$ kubectl hns tree team-a
# Output:
team-a
└── svc1-team-a

And if there were any policies in the parent namespace, these now appear in the child as well². For example, let’s say that team-a had an RBAC RoleBinding called sres. This rolebinding will also be present in the subnamespace:

$ kubectl describe rolebinding sres -n svc1-team-a
# Output:
Name:         sres
Labels:       hnc.x-k8s.io/inheritedFrom=team-a  # inserted by HNC
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  admin
Subjects: ...

Finally, HNC adds labels to these namespaces with useful information about the hierarchy which you can use to apply other policies. For example, you can create the following NetworkPolicy:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-team-a
  namespace: team-a
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchExpressions:
          - key: 'team-a.tree.hnc.x-k8s.io/depth' # Label created by HNC
            operator: Exists

This policy will both be propagated to all descendants of team-a, and will also allow ingress traffic between all of those namespaces. The “tree” label can only be applied by HNC, and is guaranteed to reflect the latest hierarchy.

You can learn all about the features of HNC from the user guide.

Next steps and getting involved

If you think that hierarchical namespaces can work for your organization, HNC v0.5.1 is available on GitHub. We’d love to know what you think of it, what problems you’re using it to solve and what features you’d most like to see added. As with all early software, you should be cautious about using HNC in production environments, but the more feedback we get, the sooner we’ll be able to drive to HNC 1.0.

We’re also open to additional contributors, whether it’s to fix or report bugs, or help prototype new features such as exceptions, improved monitoring, hierarchical resource quotas or fine-grained configuration.

Please get in touch with us via our repo, mailing list or on Slack - we look forward to hearing from you!

Adrian Ludwin is a software engineer and the tech lead for the Hierarchical Namespace Controller.

• ← Previous
Next →