Building a Platform Engineering API Layer with KCP

Baseline

  • Platform is automated and self service
  • We always have a bunch of consumers and service providers that get connected via an internal db plattform
graph TD
    subgraph Consume
        A
        B
    end
    subgraph Provider
        Cert
        DB
    end
    IDP
    A-->|discover available services|IDP
    A-->|order db|IDP
    IDP-->|Notify|DB
    DB-->|fulfill|A

Why KubeAPI

  • We have it all: Groups, Versions, Optionally Namespaced, …
  • It is extendable via CRDs
  • Challenges: CRDs are Cluster Scoped -> Everyone shares them across Namespaces
  • Idea: Everyone get’s their own cluster
  • Problem: Spinning up clusters is slow and resource intensive
  • Idea: “Lightweight clusters” aka Hosted Control Plane
  • Problem: Now we have to share CRDs Across Clusters

WTF is KCP?

  • Idea: What if we had seperate control planes but with a shared datastore
  • Goal: Horitontally scalable control plane for extenable APIs
  • You don’t need Kubernetes to run KCP (it’s a standalone binary)
  • It does not spin up a real api server but a workspace wit a low memory footprint
  • It does not implement all of the container related stuff (Pod, Deployment, …)

Access and setup

graph LR
User-->|Create APIServer Team A|KCP
KCP-->|Kubeconfig|User
subgraph KCP
    APIA(Workspace Team A)
    APIB(Workspace Team B)
    Datastore
end

User-->|Kubectl get ns|APIA
APIA-->|Return NS for Workspace A|User

Internal Organization

  • Workspaces are organized in a tree
  • Possibility of nested fun: /clusters/root:org-a:team-a
  • Sub-Workspaces can’t access ressources from the root workspace
graph TD
Root
Root-->OrgA
Root-->OrgB
OrgA-->TeamA

Sharing

  • KCP owns all Workspaces -> It can share stuff across clusters
  • To share: APIExport (Can Share multiple CRDs in one Package)
  • To use: APIBinding (Just reference the Exported API By Path to Workspace and name)

Order fulfillment

  • Classic Kubernetes: Controller -> But they are isolated, aren’t they?
  • Virtual Workspace: Provite a computed view of parts of a workspace -> Basicly a URL that you provide to the controller that can be used to watch objects accross workspaces
  • Part of KCPs magic -> You don’t create it, but it get’s managed for each APIExport

Notes from the demo

  • Spin up locally is near instant
  • Switching to the namespace can be achived with a simple api command or

But why do we even need a universal API layer

  • Service Providers should not be responsible to make things discoverable, the plattform should
  • The internal pülatform can be bought, customized or diyed but the api layer does not change -> Interchangeable backend switching
  • Kubernetes is already widespread and makes it easy to use different projects
  • Backed by the CNCF, flat learning curve

Q&A

  • Is OIDC Provided: Yes, r/n globally for all workspaces, per workspace oidc is WIP
  • What about KCPxCrossplane: Yes it is possible, more in septemeber with a talk during Container Days