API Overview

• 1: Kubernetes API Concepts
• 2: Server-Side Apply
• 3: Client Libraries
• 4: Kubernetes Deprecation Policy
• 5: Deprecated API Migration Guide
• 6: Kubernetes API health endpoints

1 - Kubernetes API Concepts

This page describes common concepts in the Kubernetes API.

The Kubernetes API is a resource-based (RESTful) programmatic interface provided via HTTP. It supports retrieving, creating, updating, and deleting primary resources via the standard HTTP verbs (POST, PUT, PATCH, DELETE, GET), includes additional subresources for many objects that allow fine grained authorization (such as binding a pod to a node), and can accept and serve those resources in different representations for convenience or efficiency. It also supports efficient change notifications on resources via "watches" and consistent lists to allow other components to effectively cache and synchronize the state of resources.

Standard API terminology

Most Kubernetes API resource types are objects: they represent a concrete instance of a concept on the cluster, like a pod or namespace. A smaller number of API resource types are "virtual" - they often represent operations rather than objects, such as a permission check (use a POST with a JSON-encoded body of SubjectAccessReview to the subjectaccessreviews resource). All objects will have a unique name to allow idempotent creation and retrieval, but virtual resource types may not have unique names if they are not retrievable or do not rely on idempotency.

Kubernetes generally leverages standard RESTful terminology to describe the API concepts:

• A resource type is the name used in the URL (pods, namespaces, services)
• All resource types have a concrete representation in JSON (their object schema) which is called a kind
• A list of instances of a resource type is known as a collection
• A single instance of the resource type is called a resource

All resource types are either scoped by the cluster (/apis/GROUP/VERSION/*) or to a namespace (/apis/GROUP/VERSION/namespaces/NAMESPACE/*). A namespace-scoped resource type will be deleted when its namespace is deleted and access to that resource type is controlled by authorization checks on the namespace scope. The following paths are used to retrieve collections and resources:

• Cluster-scoped resources:

  • GET /apis/GROUP/VERSION/RESOURCETYPE - return the collection of resources of the resource type
  • GET /apis/GROUP/VERSION/RESOURCETYPE/NAME - return the resource with NAME under the resource type

• Namespace-scoped resources:

  • GET /apis/GROUP/VERSION/RESOURCETYPE - return the collection of all instances of the resource type across all namespaces
  • GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE - return collection of all instances of the resource type in NAMESPACE
  • GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME - return the instance of the resource type with NAME in NAMESPACE

Since a namespace is a cluster-scoped resource type, you can retrieve the list of all namespaces with GET /api/v1/namespaces and details about a particular namespace with GET /api/v1/namespaces/NAME.

Almost all object resource types support the standard HTTP verbs - GET, POST, PUT, PATCH, and DELETE. Kubernetes uses the term list to describe returning a collection of resources to distinguish from retrieving a single resource which is usually called a get.

Some resource types will have one or more sub-resources, represented as sub paths below the resource:

• Cluster-scoped subresource: GET /apis/GROUP/VERSION/RESOURCETYPE/NAME/SUBRESOURCE
• Namespace-scoped subresource: GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME/SUBRESOURCE

The verbs supported for each subresource will differ depending on the object - see the API documentation for more information. It is not possible to access sub-resources across multiple resources - generally a new virtual resource type would be used if that becomes necessary.

Efficient detection of changes

To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a watch. Every Kubernetes object has a resourceVersion field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a resourceVersion value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied resourceVersion. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned resourceVersion, or perform a new collection request and begin again. See Resource Version Semantics for more detail.

For example:

1. List all of the pods in a given namespace.

   GET /api/v1/namespaces/test/pods
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {"resourceVersion":"10245"},
     "items": [...]
   }
   

2. Starting from resource version 10245, receive notifications of any creates, deletes, or updates as individual JSON objects.

   GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
   ---
   200 OK
   Transfer-Encoding: chunked
   Content-Type: application/json
   
   {
     "type": "ADDED",
     "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
   }
   {
     "type": "MODIFIED",
     "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...}
   }
   ...
   

A given Kubernetes server will only preserve a historical list of changes for a limited time. Clusters using etcd3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code 410 Gone, clearing their local cache, performing a list operation, and starting the watch from the resourceVersion returned by that new list operation. Most client libraries offer some form of standard tool for this logic. (In Go this is called a Reflector and is located in the k8s.io/client-go/cache package.)

Watch bookmarks

To mitigate the impact of short history window, we introduced a concept of bookmark watch event. It is a special kind of event to mark that all changes up to a given resourceVersion the client is requesting have already been sent. Object returned in that event is of the type requested by the request, but only resourceVersion field is set, e.g.:

GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245&allowWatchBookmarks=true
---
200 OK
Transfer-Encoding: chunked
Content-Type: application/json

{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
}
...
{
  "type": "BOOKMARK",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "12746"} }
}

Bookmark events can be requested by allowWatchBookmarks=true option in watch requests, but clients shouldn't assume bookmarks are returned at any specific interval, nor may they assume the server will send any bookmark event.

Retrieving large results sets in chunks

On large clusters, retrieving the collection of some resource types may result in very large responses that can impact the server and client. For instance, a cluster may have tens of thousands of pods, each of which is 1-2kb of encoded JSON. Retrieving all pods across all namespaces may result in a very large response (10-20MB) and consume a large amount of server resources. Starting in Kubernetes 1.9 the server supports the ability to break a single large collection request into many smaller chunks while preserving the consistency of the total request. Each chunk can be returned sequentially which reduces both the total size of the request and allows user-oriented clients to display results incrementally to improve responsiveness.

To retrieve a single list in chunks, two new parameters limit and continue are supported on collection requests and a new field continue is returned from all list operations in the list metadata field. A client should specify the maximum results they wish to receive in each chunk with limit and the server will return up to limit resources in the result and include a continue value if there are more resources in the collection. The client can then pass this continue value to the server on the next request to instruct the server to return the next chunk of results. By continuing until the server returns an empty continue value the client can consume the full set of results.

Like a watch operation, a continue token will expire after a short amount of time (by default 5 minutes) and return a 410 Gone if more results cannot be returned. In this case, the client will need to start from the beginning or omit the limit parameter.

For example, if there are 1,253 pods on the cluster and the client wants to receive chunks of 500 pods at a time, they would request those chunks as follows:

1. List all of the pods on a cluster, retrieving up to 500 pods each time.

   GET /api/v1/pods?limit=500
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {
       "resourceVersion":"10245",
       "continue": "ENCODED_CONTINUE_TOKEN",
       ...
     },
     "items": [...] // returns pods 1-500
   }
   

2. Continue the previous call, retrieving the next set of 500 pods.

   GET /api/v1/pods?limit=500&continue=ENCODED_CONTINUE_TOKEN
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {
       "resourceVersion":"10245",
       "continue": "ENCODED_CONTINUE_TOKEN_2",
       ...
     },
     "items": [...] // returns pods 501-1000
   }
   

3. Continue the previous call, retrieving the last 253 pods.

   GET /api/v1/pods?limit=500&continue=ENCODED_CONTINUE_TOKEN_2
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "PodList",
     "apiVersion": "v1",
     "metadata": {
       "resourceVersion":"10245",
       "continue": "", // continue token is empty because we have reached the end of the list
       ...
     },
     "items": [...] // returns pods 1001-1253
   }
   

Note that the resourceVersion of the list remains constant across each request, indicating the server is showing us a consistent snapshot of the pods. Pods that are created, updated, or deleted after version 10245 would not be shown unless the user makes a list request without the continue token. This allows clients to break large requests into smaller chunks and then perform a watch operation on the full set without missing any updates.

remainingItemCount is the number of subsequent items in the list which are not included in this list response. If the list request contained label or field selectors, then the number of remaining items is unknown and the API server does not include a remainingItemCount field in its response. If the list is complete (either because it is not chunking or because this is the last chunk), then there are no more remaining items and the API server does not include a remainingItemCount field in its response. The intended use of the remainingItemCount is estimating the size of a collection.

Lists

There are dozens of list types (such as PodList, ServiceList, and NodeList) defined in the Kubernetes API. You can get more information about each list type from the Kubernetes API documentation.

When you query the API for a particular type, all items returned by that query are of that type. For example, when you ask for a list of services, the list type is shown as kind: ServiceList and each item in that list represents a single Service. For example:

GET /api/v1/services
---
{
  "kind": "ServiceList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "2947301"
  },
  "items": [
    {
      "metadata": {
        "name": "kubernetes",
        "namespace": "default",
...
      "metadata": {
        "name": "kube-dns",
        "namespace": "kube-system",
...

Some tools, such as kubectl provide another way to query the Kubernetes API. Because the output of kubectl might include multiple list types, the list of items is represented as kind: List. For example:

$ kubectl get services -A -o yaml 

apiVersion: v1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
items:
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: "2021-06-03T14:54:12Z"
    labels:
      component: apiserver
      provider: kubernetes
    name: kubernetes
    namespace: default
...
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      prometheus.io/port: "9153"
      prometheus.io/scrape: "true"
    creationTimestamp: "2021-06-03T14:54:14Z"
    labels:
      k8s-app: kube-dns
      kubernetes.io/cluster-service: "true"
      kubernetes.io/name: CoreDNS
    name: kube-dns
    namespace: kube-system

Note: Keep in mind that the Kubernetes API does not have a kind: List type. kind: List is an internal mechanism type for lists of mixed resources and should not be depended upon.

Receiving resources as Tables

kubectl get is a simple tabular representation of one or more instances of a particular resource type. In the past, clients were required to reproduce the tabular and describe output implemented in kubectl to perform simple lists of objects. A few limitations of that approach include non-trivial logic when dealing with certain objects. Additionally, types provided by API aggregation or third party resources are not known at compile time. This means that generic implementations had to be in place for types unrecognized by a client.

In order to avoid potential limitations as described above, clients may request the Table representation of objects, delegating specific details of printing to the server. The Kubernetes API implements standard HTTP content type negotiation: passing an Accept header containing a value of application/json;as=Table;g=meta.k8s.io;v=v1beta1 with a GET call will request that the server return objects in the Table content type.

For example, list all of the pods on a cluster in the Table format.

GET /api/v1/pods
Accept: application/json;as=Table;g=meta.k8s.io;v=v1beta1
---
200 OK
Content-Type: application/json

{
    "kind": "Table",
    "apiVersion": "meta.k8s.io/v1beta1",
    ...
    "columnDefinitions": [
        ...
    ]
}

For API resource types that do not have a custom Table definition on the server, a default Table response is returned by the server, consisting of the resource's name and creationTimestamp fields.

GET /apis/crd.example.com/v1alpha1/namespaces/default/resources
---
200 OK
Content-Type: application/json
...

{
    "kind": "Table",
    "apiVersion": "meta.k8s.io/v1beta1",
    ...
    "columnDefinitions": [
        {
            "name": "Name",
            "type": "string",
            ...
        },
        {
            "name": "Created At",
            "type": "date",
            ...
        }
    ]
}

Table responses are available beginning in version 1.10 of the kube-apiserver. As such, not all API resource types will support a Table response, specifically when using a client against older clusters. Clients that must work against all resource types, or can potentially deal with older clusters, should specify multiple content types in their Accept header to support fallback to non-Tabular JSON:

Accept: application/json;as=Table;g=meta.k8s.io;v=v1beta1, application/json

Alternate representations of resources

By default, Kubernetes returns objects serialized to JSON with content type application/json. This is the default serialization format for the API. However, clients may request the more efficient Protobuf representation of these objects for better performance at scale. The Kubernetes API implements standard HTTP content type negotiation: passing an Accept header with a GET call will request that the server return objects in the provided content type, while sending an object in Protobuf to the server for a PUT or POST call takes the Content-Type header. The server will return a Content-Type header if the requested format is supported, or the 406 Not acceptable error if an invalid content type is provided.

See the API documentation for a list of supported content types for each API.

For example:

1. List all of the pods on a cluster in Protobuf format.

   GET /api/v1/pods
   Accept: application/vnd.kubernetes.protobuf
   ---
   200 OK
   Content-Type: application/vnd.kubernetes.protobuf
   
   ... binary encoded PodList object
   

2. Create a pod by sending Protobuf encoded data to the server, but request a response in JSON.

   POST /api/v1/namespaces/test/pods
   Content-Type: application/vnd.kubernetes.protobuf
   Accept: application/json
   ... binary encoded Pod object
   ---
   200 OK
   Content-Type: application/json
   
   {
     "kind": "Pod",
     "apiVersion": "v1",
     ...
   }
   

Not all API resource types will support Protobuf, specifically those defined via Custom Resource Definitions or those that are API extensions. Clients that must work against all resource types should specify multiple content types in their Accept header to support fallback to JSON:

Accept: application/vnd.kubernetes.protobuf, application/json

Protobuf encoding

Kubernetes uses an envelope wrapper to encode Protobuf responses. That wrapper starts with a 4 byte magic number to help identify content in disk or in etcd as Protobuf (as opposed to JSON), and then is followed by a Protobuf encoded wrapper message, which describes the encoding and type of the underlying object and then contains the object.

The wrapper format is:

A four byte magic number prefix:
  Bytes 0-3: "k8s\x00" [0x6b, 0x38, 0x73, 0x00]

An encoded Protobuf message with the following IDL:
  message Unknown {
    // typeMeta should have the string values for "kind" and "apiVersion" as set on the JSON object
    optional TypeMeta typeMeta = 1;

    // raw will hold the complete serialized object in protobuf. See the protobuf definitions in the client libraries for a given kind.
    optional bytes raw = 2;

    // contentEncoding is encoding used for the raw data. Unspecified means no encoding.
    optional string contentEncoding = 3;

    // contentType is the serialization method used to serialize 'raw'. Unspecified means application/vnd.kubernetes.protobuf and is usually
    // omitted.
    optional string contentType = 4;
  }

  message TypeMeta {
    // apiVersion is the group/version for this type
    optional string apiVersion = 1;
    // kind is the name of the object schema. A protobuf definition should exist for this object.
    optional string kind = 2;
  }

Clients that receive a response in application/vnd.kubernetes.protobuf that does not match the expected prefix should reject the response, as future versions may need to alter the serialization format in an incompatible way and will do so by changing the prefix.

Resource deletion

Resources are deleted in two phases: 1) finalization, and 2) removal.

{
  "kind": "ConfigMap",
  "apiVersion": "v1",
  "metadata": {
    "finalizers": {"url.io/neat-finalization", "other-url.io/my-finalizer"},
    "deletionTimestamp": nil,
  }
}

When a client first deletes a resource, the .metadata.deletionTimestamp is set to the current time. Once the .metadata.deletionTimestamp is set, external controllers that act on finalizers may start performing their cleanup work at any time, in any order. Order is NOT enforced because it introduces significant risk of stuck .metadata.finalizers. .metadata.finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list.

Once the last finalizer is removed, the resource is actually removed from etcd.

Single resource API

API verbs GET, CREATE, UPDATE, PATCH, DELETE and PROXY support single resources only. These verbs with single resource support have no support for submitting multiple resources together in an ordered or unordered list or transaction. Clients including kubectl will parse a list of resources and make single-resource API requests.

API verbs LIST and WATCH support getting multiple resources, and DELETECOLLECTION supports deleting multiple resources.

Dry-run

The modifying verbs (POST, PUT, PATCH, and DELETE) can accept requests in a dry run mode. Dry run mode helps to evaluate a request through the typical request stages (admission chain, validation, merge conflicts) up until persisting objects to storage. The response body for the request is as close as possible to a non-dry-run response. The system guarantees that dry-run requests will not be persisted in storage or have any other side effects.

Make a dry-run request

Dry-run is triggered by setting the dryRun query parameter. This parameter is a string, working as an enum, and the only accepted values are:

• All: Every stage runs as normal, except for the final storage stage. Admission controllers are run to check that the request is valid, mutating controllers mutate the request, merge is performed on PATCH, fields are defaulted, and schema validation occurs. The changes are not persisted to the underlying storage, but the final object which would have been persisted is still returned to the user, along with the normal status code. If the request would trigger an admission controller which would have side effects, the request will be failed rather than risk an unwanted side effect. All built in admission control plugins support dry-run. Additionally, admission webhooks can declare in their configuration object that they do not have side effects by setting the sideEffects field to "None". If a webhook actually does have side effects, then the sideEffects field should be set to "NoneOnDryRun", and the webhook should also be modified to understand the DryRun field in AdmissionReview, and prevent side effects on dry-run requests.
• Leave the value empty, which is also the default: Keep the default modifying behavior.

For example:

POST /api/v1/namespaces/test/pods?dryRun=All
Content-Type: application/json
Accept: application/json

The response would look the same as for non-dry-run request, but the values of some generated fields may differ.

Dry-run authorization

Authorization for dry-run and non-dry-run requests is identical. Thus, to make a dry-run request, the user must be authorized to make the non-dry-run request.

For example, to run a dry-run PATCH for Deployments, you must have the PATCH permission for Deployments, as in the example of the RBAC rule below.

rules:
- apiGroups: ["extensions", "apps"]
  resources: ["deployments"]
  verbs: ["patch"]

See Authorization Overview.

Generated values

Some values of an object are typically generated before the object is persisted. It is important not to rely upon the values of these fields set by a dry-run request, since these values will likely be different in dry-run mode from when the real request is made. Some of these fields are:

• name: if generateName is set, name will have a unique random name
• creationTimestamp/deletionTimestamp: records the time of creation/deletion
• UID: uniquely identifies the object and is randomly generated (non-deterministic)
• resourceVersion: tracks the persisted version of the object
• Any field set by a mutating admission controller
• For the Service resource: Ports or IPs that kube-apiserver assigns to v1.Service objects

Server Side Apply

Starting from Kubernetes v1.18, you can enable the Server Side Apply feature so that the control plane tracks managed fields for all newly created objects. Server Side Apply provides a clear pattern for managing field conflicts, offers server-side Apply and Update operations, and replaces the client-side functionality of kubectl apply. For more details about this feature, see the section on Server Side Apply.

Resource Versions

Resource versions are strings that identify the server's internal version of an object. Resource versions can be used by clients to determine when objects have changed, or to express data consistency requirements when getting, listing and watching resources. Resource versions must be treated as opaque by clients and passed unmodified back to the server. For example, clients must not assume resource versions are numeric, and may only compare two resource versions for equality (i.e. must not compare resource versions for greater-than or less-than relationships).

ResourceVersion in metadata

Clients find resource versions in resources, including the resources in watch events, and list responses returned from the server:

v1.meta/ObjectMeta - The metadata.resourceVersion of a resource instance identifies the resource version the instance was last modified at.

v1.meta/ListMeta - The metadata.resourceVersion of a resource collection (i.e. a list response) identifies the resource version at which the list response was constructed.

The ResourceVersion Parameter

The get, list, and watch operations support the resourceVersion parameter.

The exact meaning of this parameter differs depending on the operation and the value of resourceVersion.

For get and list, the semantics of resource version are:

Get:

List:

v1.19+ API servers support the resourceVersionMatch parameter, which determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set. If resourceVersion is unset, resourceVersionMatch is not allowed. For backward compatibility, clients must tolerate the server ignoring resourceVersionMatch:

• When using resourceVersionMatch=NotOlderThan and limit is set, clients must handle HTTP 410 "Gone" responses. For example, the client might retry with a newer resourceVersion or fall back to resourceVersion="".
• When using resourceVersionMatch=Exact and limit is unset, clients must verify that the resourceVersion in the ListMeta of the response matches the requested resourceVersion, and handle the case where it does not. For example, the client might fall back to a request with limit set.

Unless you have strong consistency requirements, using resourceVersionMatch=NotOlderThan and a known resourceVersion is preferable since it can achieve better performance and scalability of your cluster than leaving resourceVersion and resourceVersionMatch unset, which requires quorum read to be served.

Footnotes:

[1] If the server does not honor the resourceVersionMatch parameter, it is treated as if it is unset.

The meaning of the get and list semantics are:

• Most Recent: Return data at the most recent resource version. The returned data must be consistent (i.e. served from etcd via a quorum read).

• Any: Return data at any resource version. The newest available resource version is preferred, but strong consistency is not required; data at any resource version may be served. It is possible for the request to return data at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches. Clients that cannot tolerate this should not use this semantic.

• Not older than: Return data at least as new as the provided resourceVersion. The newest available data is preferred, but any data not older than the provided resourceVersion may be served. For list requests to servers that honor the resourceVersionMatch parameter, this guarantees that resourceVersion in the ListMeta is not older than the requested resourceVersion, but does not make any guarantee about the resourceVersion in the ObjectMeta of the list items since ObjectMeta.resourceVersion tracks when an object was last updated, not how up-to-date the object is when served.

• Exact: Return data at the exact resource version provided. If the provided resourceVersion is unavailable, the server responds with HTTP 410 "Gone". For list requests to servers that honor the resourceVersionMatch parameter, this guarantees that resourceVersion in the ListMeta is the same as the requested resourceVersion, but does not make any guarantee about the resourceVersion in the ObjectMeta of the list items since ObjectMeta.resourceVersion tracks when an object was last updated, not how up-to-date the object is when served.

• Continue Token, Exact: Return data at the resource version of the initial paginated list call. The returned Continue Tokens are responsible for keeping track of the initially provided resource version for all paginated list calls after the initial paginated list call.

For watch, the semantics of resource version are:

Watch:

The meaning of the watch semantics are:

• Get State and Start at Most Recent: Start a watch at the most recent resource version, which must be consistent (i.e. served from etcd via a quorum read). To establish initial state, the watch begins with synthetic "Added" events of all resources instances that exist at the starting resource version. All following watch events are for all changes that occurred after the resource version the watch started at.
• Get State and Start at Any: Warning: Watches initialize this way may return arbitrarily stale data! Please review this semantic before using it, and favor the other semantics where possible. Start a watch at any resource version, the most recent resource version available is preferred, but not required; any starting resource version is allowed. It is possible for the watch to start at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches. Clients that cannot tolerate this should not start a watch with this semantic. To establish initial state, the watch begins with synthetic "Added" events for all resources instances that exist at the starting resource version. All following watch events are for all changes that occurred after the resource version the watch started at.
• Start at Exact: Start a watch at an exact resource version. The watch events are for all changes after the provided resource version. Unlike "Get State and Start at Most Recent" and "Get State and Start at Any", the watch is not started with synthetic "Added" events for the provided resource version. The client is assumed to already have the initial state at the starting resource version since the client provided the resource version.

"410 Gone" responses

Servers are not required to serve all older resource versions and may return a HTTP 410 (Gone) status code if a client requests a resourceVersion older than the server has retained. Clients must be able to tolerate 410 (Gone) responses. See Efficient detection of changes for details on how to handle 410 (Gone) responses when watching resources.

If you request a a resourceVersion outside the applicable limit then, depending on whether a request is served from cache or not, the API server may reply with a 410 Gone HTTP response.

Unavailable resource versions

Servers are not required to serve unrecognized resource versions. List and Get requests for unrecognized resource versions may wait briefly for the resource version to become available, should timeout with a 504 (Gateway Timeout) if the provided resource versions does not become available in a reasonable amount of time, and may respond with a Retry-After response header indicating how many seconds a client should wait before retrying the request. Currently, the kube-apiserver also identifies these responses with a "Too large resource version" message. Watch requests for an unrecognized resource version may wait indefinitely (until the request timeout) for the resource version to become available.

2 - Server-Side Apply

Introduction

Server Side Apply helps users and controllers manage their resources through declarative configurations. Clients can create and modify their objects declaratively by sending their fully specified intent.

A fully specified intent is a partial object that only includes the fields and values for which the user has an opinion. That intent either creates a new object or is combined, by the server, with the existing object.

The system supports multiple appliers collaborating on a single object.

Changes to an object's fields are tracked through a "field management" mechanism. When a field's value changes, ownership moves from its current manager to the manager making the change. When trying to apply an object, fields that have a different value and are owned by another manager will result in a conflict. This is done in order to signal that the operation might undo another collaborator's changes. Conflicts can be forced, in which case the value will be overridden, and the ownership will be transferred.

If you remove a field from a configuration and apply the configuration, server side apply checks if there are any other field managers that also own the field. If the field is not owned by any other field managers, it is either deleted from the live object or reset to its default value, if it has one. The same rule applies to associative list or map items.

Server side apply is meant both as a replacement for the original kubectl apply and as a simpler mechanism for controllers to enact their changes.

If you have Server Side Apply enabled, the control plane tracks managed fields for all newly created objects.

Field Management

Compared to the last-applied annotation managed by kubectl, Server Side Apply uses a more declarative approach, which tracks a user's field management, rather than a user's last applied state. This means that as a side effect of using Server Side Apply, information about which field manager manages each field in an object also becomes available.

For a user to manage a field, in the Server Side Apply sense, means that the user relies on and expects the value of the field not to change. The user who last made an assertion about the value of a field will be recorded as the current field manager. This can be done either by changing the value with POST, PUT, or non-apply PATCH, or by including the field in a config sent to the Server Side Apply endpoint. When using Server-Side Apply, trying to change a field which is managed by someone else will result in a rejected request (if not forced, see Conflicts).

When two or more appliers set a field to the same value, they share ownership of that field. Any subsequent attempt to change the value of the shared field, by any of the appliers, results in a conflict. Shared field owners may give up ownership of a field by removing it from their configuration.

Field management is stored in amanagedFields field that is part of an object's metadata.

A simple example of an object created by Server Side Apply could look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-cm
  namespace: default
  labels:
    test-label: test
  managedFields:
  - manager: kubectl
    operation: Apply
    apiVersion: v1
    time: "2010-10-10T0:00:00Z"
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:test-label: {}
      f:data:
        f:key: {}
data:
  key: some value

The above object contains a single manager in metadata.managedFields. The manager consists of basic information about the managing entity itself, like operation type, API version, and the fields managed by it.

Note: This field is managed by the API server and should not be changed by the user.

Nevertheless it is possible to change metadata.managedFields through an Update operation. Doing so is highly discouraged, but might be a reasonable option to try if, for example, the managedFields get into an inconsistent state (which clearly should not happen).

The format of the managedFields is described in the API.

Conflicts

A conflict is a special status error that occurs when an Apply operation tries to change a field, which another user also claims to manage. This prevents an applier from unintentionally overwriting the value set by another user. When this occurs, the applier has 3 options to resolve the conflicts:

• Overwrite value, become sole manager: If overwriting the value was intentional (or if the applier is an automated process like a controller) the applier should set the force query parameter to true and make the request again. This forces the operation to succeed, changes the value of the field, and removes the field from all other managers' entries in managedFields.

• Don't overwrite value, give up management claim: If the applier doesn't care about the value of the field anymore, they can remove it from their config and make the request again. This leaves the value unchanged, and causes the field to be removed from the applier's entry in managedFields.

• Don't overwrite value, become shared manager: If the applier still cares about the value of the field, but doesn't want to overwrite it, they can change the value of the field in their config to match the value of the object on the server, and make the request again. This leaves the value unchanged, and causes the field's management to be shared by the applier and all other field managers that already claimed to manage it.

Managers

Managers identify distinct workflows that are modifying the object (especially useful on conflicts!), and can be specified through the fieldManager query parameter as part of a modifying request. It is required for the apply endpoint, though kubectl will default it to kubectl. For other updates, its default is computed from the user-agent.

Apply and Update

The two operation types considered by this feature are Apply (PATCH with content type application/apply-patch+yaml) and Update (all other operations which modify the object). Both operations update the managedFields, but behave a little differently.

Note:

Whether you are submitting JSON data or YAML data, use application/apply-patch+yaml as the Content-Type header value.

All JSON documents are valid YAML.

For instance, only the apply operation fails on conflicts while update does not. Also, apply operations are required to identify themselves by providing a fieldManager query parameter, while the query parameter is optional for update operations. Finally, when using the apply operation you cannot have managedFields in the object that is being applied.

An example object with multiple managers could look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-cm
  namespace: default
  labels:
    test-label: test
  managedFields:
  - manager: kubectl
    operation: Apply
    apiVersion: v1
    fields:
      f:metadata:
        f:labels:
          f:test-label: {}
  - manager: kube-controller-manager
    operation: Update
    apiVersion: v1
    time: '2019-03-30T16:00:00.000Z'
    fields:
      f:data:
        f:key: {}
data:
  key: new value

In this example, a second operation was run as an Update by the manager called kube-controller-manager. The update changed a value in the data field which caused the field's management to change to the kube-controller-manager.

If this update would have been an Apply operation, the operation would have failed due to conflicting ownership.

Merge strategy

The merging strategy, implemented with Server Side Apply, provides a generally more stable object lifecycle. Server Side Apply tries to merge fields based on the actor who manages them instead of overruling based on values. This way multiple actors can update the same object without causing unexpected interference.

When a user sends a "fully-specified intent" object to the Server Side Apply endpoint, the server merges it with the live object favoring the value in the applied config if it is specified in both places. If the set of items present in the applied config is not a superset of the items applied by the same user last time, each missing item not managed by any other appliers is removed. For more information about how an object's schema is used to make decisions when merging, see sigs.k8s.io/structured-merge-diff.

A number of markers were added in Kubernetes 1.16 and 1.17, to allow API developers to describe the merge strategy supported by lists, maps, and structs. These markers can be applied to objects of the respective type, in Go files or in the OpenAPI schema definition of the CRD:

If listType is missing, the API server interprets a patchMergeStrategy=merge marker as a listType=map and the corresponding patchMergeKey marker as a listMapKey.

The atomic list type is recursive.

These markers are specified as comments and don't have to be repeated as field tags.

Compatibility across topology changes

On rare occurrences, a CRD or built-in type author may want to change the specific topology of a field in their resource without incrementing its version. Changing the topology of types, by upgrading the cluster or updating the CRD, has different consequences when updating existing objects. There are two categories of changes: when a field goes from map/set/granular to atomic and the other way around.

When the listType, mapType, or structType changes from map/set/granular to atomic, the whole list, map, or struct of existing objects will end-up being owned by actors who owned an element of these types. This means that any further change to these objects would cause a conflict.

When a list, map, or struct changes from atomic to map/set/granular, the API server won't be able to infer the new ownership of these fields. Because of that, no conflict will be produced when objects have these fields updated. For that reason, it is not recommended to change a type from atomic to map/set/granular.

Take for example, the custom resource:

apiVersion: example.com/v1
kind: Foo
metadata:
  name: foo-sample
  managedFields:
  - manager: manager-one
    operation: Apply
    apiVersion: example.com/v1
    fields:
      f:spec:
        f:data: {}
spec:
  data:
    key1: val1
    key2: val2

Before spec.data gets changed from atomic to granular, manager-one owns the field spec.data, and all the fields within it (key1 and key2). When the CRD gets changed to make spec.data granular, manager-one continues to own the top-level field spec.data (meaning no other managers can delete the map called data without a conflict), but it no longer owns key1 and key2, so another manager can then modify or delete those fields without conflict.

Custom Resources

By default, Server Side Apply treats custom resources as unstructured data. All keys are treated the same as struct fields, and all lists are considered atomic.

If the Custom Resource Definition defines a schema that contains annotations as defined in the previous "Merge Strategy" section, these annotations will be used when merging objects of this type.

Using Server-Side Apply in a controller

As a developer of a controller, you can use server-side apply as a way to simplify the update logic of your controller. The main differences with a read-modify-write and/or patch are the following:

• the applied object must contain all the fields that the controller cares about.
• there is no way to remove fields that haven't been applied by the controller before (controller can still send a PATCH/UPDATE for these use-cases).
• the object doesn't have to be read beforehand, resourceVersion doesn't have to be specified.

It is strongly recommended for controllers to always "force" conflicts, since they might not be able to resolve or act on these conflicts.

Transferring Ownership

In addition to the concurrency controls provided by conflict resolution, Server Side Apply provides ways to perform coordinated field ownership transfers from users to controllers.

This is best explained by example. Let's look at how to safely transfer ownership of the replicas field from a user to a controller while enabling automatic horizontal scaling for a Deployment, using the HorizontalPodAutoscaler resource and its accompanying controller.

Say a user has defined deployment with replicas set to the desired value:

application/ssa/nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

And the user has created the deployment using server side apply like so:

kubectl apply -f https://k8s.io/examples/application/ssa/nginx-deployment.yaml --server-side

Then later, HPA is enabled for the deployment, e.g.:

kubectl autoscale deployment nginx-deployment --cpu-percent=50 --min=1 --max=10

Now, the user would like to remove replicas from their configuration, so they don't accidentally fight with the HPA controller. However, there is a race: it might take some time before HPA feels the need to adjust replicas, and if the user removes replicas before the HPA writes to the field and becomes its owner, then apiserver will set replicas to 1, its default value. This is not what the user wants to happen, even temporarily.

There are two solutions:

• (basic) Leave replicas in the configuration; when HPA eventually writes to that field, the system gives the user a conflict over it. At that point, it is safe to remove from the configuration.

• (more advanced) If, however, the user doesn't want to wait, for example because they want to keep the cluster legible to coworkers, then they can take the following steps to make it safe to remove replicas from their configuration:

First, the user defines a new configuration containing only the replicas field:

application/ssa/nginx-deployment-replicas-only.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3

The user applies that configuration using the field manager name handover-to-hpa:

kubectl apply -f https://k8s.io/examples/application/ssa/nginx-deployment-replicas-only.yaml \
  --server-side --field-manager=handover-to-hpa \
  --validate=false

If the apply results in a conflict with the HPA controller, then do nothing. The conflict indicates the controller has claimed the field earlier in the process than it sometimes does.

At this point the user may remove the replicas field from their configuration.

application/ssa/nginx-deployment-no-replicas.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2

Note that whenever the HPA controller sets the replicas field to a new value, the temporary field manager will no longer own any fields and will be automatically deleted. No clean up is required.

Transferring Ownership Between Users

Users can transfer ownership of a field between each other by setting the field to the same value in both of their applied configs, causing them to share ownership of the field. Once the users share ownership of the field, one of them can remove the field from their applied configuration to give up ownership and complete the transfer to the other user.

Comparison with Client Side Apply

A consequence of the conflict detection and resolution implemented by Server Side Apply is that an applier always has up to date field values in their local state. If they don't, they get a conflict the next time they apply. Any of the three options to resolve conflicts results in the applied configuration being an up to date subset of the object on the server's fields.

This is different from Client Side Apply, where outdated values which have been overwritten by other users are left in an applier's local config. These values only become accurate when the user updates that specific field, if ever, and an applier has no way of knowing whether their next apply will overwrite other users' changes.

Another difference is that an applier using Client Side Apply is unable to change the API version they are using, but Server Side Apply supports this use case.

Upgrading from client-side apply to server-side apply

Client-side apply users who manage a resource with kubectl apply can start using server-side apply with the following flag.

kubectl apply --server-side [--dry-run=server]

By default, field management of the object transfers from client-side apply to kubectl server-side apply without encountering conflicts.

Caution:

Keep the last-applied-configuration annotation up to date. The annotation infers client-side apply's managed fields. Any fields not managed by client-side apply raise conflicts.

For example, if you used kubectl scale to update the replicas field after client-side apply, then this field is not owned by client-side apply and creates conflicts on kubectl apply --server-side.

This behavior applies to server-side apply with the kubectl field manager. As an exception, you can opt-out of this behavior by specifying a different, non-default field manager, as seen in the following example. The default field manager for kubectl server-side apply is kubectl.

kubectl apply --server-side --field-manager=my-manager [--dry-run=server]

Downgrading from server-side apply to client-side apply

If you manage a resource with kubectl apply --server-side, you can downgrade to client-side apply directly with kubectl apply.

Downgrading works because kubectl server-side apply keeps the last-applied-configuration annotation up-to-date if you use kubectl apply.

This behavior applies to server-side apply with the kubectl field manager. As an exception, you can opt-out of this behavior by specifying a different, non-default field manager, as seen in the following example. The default field manager for kubectl server-side apply is kubectl.

kubectl apply --server-side --field-manager=my-manager [--dry-run=server]

API Endpoint

With the Server Side Apply feature enabled, the PATCH endpoint accepts the additional application/apply-patch+yaml content type. Users of Server Side Apply can send partially specified objects as YAML to this endpoint. When applying a configuration, one should always include all the fields that they have an opinion about.

Clearing ManagedFields

It is possible to strip all managedFields from an object by overwriting them using MergePatch, StrategicMergePatch, JSONPatch, or Update, so every non-apply operation. This can be done by overwriting the managedFields field with an empty entry. Two examples are:

PATCH /api/v1/namespaces/default/configmaps/example-cm
Content-Type: application/merge-patch+json
Accept: application/json
Data: {"metadata":{"managedFields": [{}]}}

PATCH /api/v1/namespaces/default/configmaps/example-cm
Content-Type: application/json-patch+json
Accept: application/json
Data: [{"op": "replace", "path": "/metadata/managedFields", "value": [{}]}]

This will overwrite the managedFields with a list containing a single empty entry that then results in the managedFields being stripped entirely from the object. Note that setting the managedFields to an empty list will not reset the field. This is on purpose, so managedFields never get stripped by clients not aware of the field.

In cases where the reset operation is combined with changes to other fields than the managedFields, this will result in the managedFields being reset first and the other changes being processed afterwards. As a result the applier takes ownership of any fields updated in the same request.

Caution: Server Side Apply does not correctly track ownership on sub-resources that don't receive the resource object type. If you are using Server Side Apply with such a sub-resource, the changed fields won't be tracked.

Disabling the feature

Server Side Apply is a beta feature, so it is enabled by default. To turn this feature gate off, you need to include the --feature-gates ServerSideApply=false flag when starting kube-apiserver. If you have multiple kube-apiserver replicas, all should have the same flag setting.

3 - Client Libraries

This page contains an overview of the client libraries for using the Kubernetes API from various programming languages.

To write applications using the Kubernetes REST API, you do not need to implement the API calls and request/response types yourself. You can use a client library for the programming language you are using.

Client libraries often handle common tasks such as authentication for you. Most client libraries can discover and use the Kubernetes Service Account to authenticate if the API client is running inside the Kubernetes cluster, or can understand the kubeconfig file format to read the credentials and the API Server address.

Officially-supported Kubernetes client libraries

The following client libraries are officially maintained by Kubernetes SIG API Machinery.

Community-maintained client libraries

Caution: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects. This page follows CNCF website guidelines by listing projects alphabetically. To add a project to this list, read the content guide before submitting a change.

The following Kubernetes API client libraries are provided and maintained by their authors, not the Kubernetes team.

4 - Kubernetes Deprecation Policy

This document details the deprecation policy for various facets of the system.

Kubernetes is a large system with many components and many contributors. As with any such software, the feature set naturally evolves over time, and sometimes a feature may need to be removed. This could include an API, a flag, or even an entire feature. To avoid breaking existing users, Kubernetes follows a deprecation policy for aspects of the system that are slated to be removed.

Deprecating parts of the API

Since Kubernetes is an API-driven system, the API has evolved over time to reflect the evolving understanding of the problem space. The Kubernetes API is actually a set of APIs, called "API groups", and each API group is independently versioned. API versions fall into 3 main tracks, each of which has different policies for deprecation:

A given release of Kubernetes can support any number of API groups and any number of versions of each.

The following rules govern the deprecation of elements of the API. This includes:

• REST resources (aka API objects)
• Fields of REST resources
• Annotations on REST resources, including "beta" annotations but not including "alpha" annotations.
• Enumerated or constant values
• Component config structures

These rules are enforced between official releases, not between arbitrary commits to master or release branches.

Rule #1: API elements may only be removed by incrementing the version of the API group.

Once an API element has been added to an API group at a particular version, it can not be removed from that version or have its behavior significantly changed, regardless of track.

Note: For historical reasons, there are 2 "monolithic" API groups - "core" (no group name) and "extensions". Resources will incrementally be moved from these legacy API groups into more domain-specific API groups.

Rule #2: API objects must be able to round-trip between API versions in a given release without information loss, with the exception of whole REST resources that do not exist in some versions.

For example, an object can be written as v1 and then read back as v2 and converted to v1, and the resulting v1 resource will be identical to the original. The representation in v2 might be different from v1, but the system knows how to convert between them in both directions. Additionally, any new field added in v2 must be able to round-trip to v1 and back, which means v1 might have to add an equivalent field or represent it as an annotation.

Rule #3: An API version in a given track may not be deprecated until a new API version at least as stable is released.

GA API versions can replace GA API versions as well as beta and alpha API versions. Beta API versions may not replace GA API versions.

Rule #4a: Other than the most recent API versions in each track, older API versions must be supported after their announced deprecation for a duration of no less than:

• GA: 12 months or 3 releases (whichever is longer)
• Beta: 9 months or 3 releases (whichever is longer)
• Alpha: 0 releases

This covers the maximum supported version skew of 2 releases.

Note: Until #52185 is resolved, no API versions that have been persisted to storage may be removed. Serving REST endpoints for those versions may be disabled (subject to the deprecation timelines in this document), but the API server must remain capable of decoding/converting previously persisted data from storage.

Rule #4b: The "preferred" API version and the "storage version" for a given group may not advance until after a release has been made that supports both the new version and the previous version

Users must be able to upgrade to a new release of Kubernetes and then roll back to a previous release, without converting anything to the new API version or suffering breakages (unless they explicitly used features only available in the newer version). This is particularly evident in the stored representation of objects.

All of this is best illustrated by examples. Imagine a Kubernetes release, version X, which introduces a new API group. A new Kubernetes release is made every approximately 3 months (4 per year). The following table describes which API versions are supported in a series of subsequent releases.

REST resources (aka API objects)

Consider a hypothetical REST resource named Widget, which was present in API v1 in the above timeline, and which needs to be deprecated. We document and announce the deprecation in sync with release X+1. The Widget resource still exists in API version v1 (deprecated) but not in v2alpha1. The Widget resource continues to exist and function in releases up to and including X+8. Only in release X+9, when API v1 has aged out, does the Widget resource cease to exist, and the behavior get removed.

Starting in Kubernetes v1.19, making an API request to a deprecated REST API endpoint:

1. Returns a Warning header (as defined in RFC7234, Section 5.5) in the API response.

2. Adds a "k8s.io/deprecated":"true" annotation to the audit event recorded for the request.

3. Sets an apiserver_requested_deprecated_apis gauge metric to 1 in the kube-apiserver process. The metric has labels for group, version, resource, subresource that can be joined to the apiserver_request_total metric, and a removed_release label that indicates the Kubernetes release in which the API will no longer be served. The following Prometheus query returns information about requests made to deprecated APIs which will be removed in v1.22:

   apiserver_requested_deprecated_apis{removed_release="1.22"} * on(group,version,resource,subresource) group_right() apiserver_request_total
   

Fields of REST resources

As with whole REST resources, an individual field which was present in API v1 must exist and function until API v1 is removed. Unlike whole resources, the v2 APIs may choose a different representation for the field, as long as it can be round-tripped. For example a v1 field named "magnitude" which was deprecated might be named "deprecatedMagnitude" in API v2. When v1 is eventually removed, the deprecated field can be removed from v2.

Enumerated or constant values

As with whole REST resources and fields thereof, a constant value which was supported in API v1 must exist and function until API v1 is removed.

Component config structures

Component configs are versioned and managed similar to REST resources.

Future work

Over time, Kubernetes will introduce more fine-grained API versions, at which point these rules will be adjusted as needed.

Deprecating a flag or CLI

The Kubernetes system is comprised of several different programs cooperating. Sometimes, a Kubernetes release might remove flags or CLI commands (collectively "CLI elements") in these programs. The individual programs naturally sort into two main groups - user-facing and admin-facing programs, which vary slightly in their deprecation policies. Unless a flag is explicitly prefixed or documented as "alpha" or "beta", it is considered GA.

CLI elements are effectively part of the API to the system, but since they are not versioned in the same way as the REST API, the rules for deprecation are as follows:

Rule #5a: CLI elements of user-facing components (e.g. kubectl) must function after their announced deprecation for no less than:

• GA: 12 months or 2 releases (whichever is longer)
• Beta: 3 months or 1 release (whichever is longer)
• Alpha: 0 releases

Rule #5b: CLI elements of admin-facing components (e.g. kubelet) must function after their announced deprecation for no less than:

• GA: 6 months or 1 release (whichever is longer)
• Beta: 3 months or 1 release (whichever is longer)
• Alpha: 0 releases

Rule #6: Deprecated CLI elements must emit warnings (optionally disable) when used.

Deprecating a feature or behavior

Occasionally a Kubernetes release needs to deprecate some feature or behavior of the system that is not controlled by the API or CLI. In this case, the rules for deprecation are as follows:

Rule #7: Deprecated behaviors must function for no less than 1 year after their announced deprecation.

This does not imply that all changes to the system are governed by this policy. This applies only to significant, user-visible behaviors which impact the correctness of applications running on Kubernetes or that impact the administration of Kubernetes clusters, and which are being removed entirely.

An exception to the above rule is feature gates. Feature gates are key=value pairs that allow for users to enable/disable experimental features.

Feature gates are intended to cover the development life cycle of a feature - they are not intended to be long-term APIs. As such, they are expected to be deprecated and removed after a feature becomes GA or is dropped.

As a feature moves through the stages, the associated feature gate evolves. The feature life cycle matched to its corresponding feature gate is:

• Alpha: the feature gate is disabled by default and can be enabled by the user.
• Beta: the feature gate is enabled by default and can be disabled by the user.
• GA: the feature gate is deprecated (see "Deprecation") and becomes non-operational.
• GA, deprecation window complete: the feature gate is removed and calls to it are no longer accepted.

Deprecation

Features can be removed at any point in the life cycle prior to GA. When features are removed prior to GA, their associated feature gates are also deprecated.

When an invocation tries to disable a non-operational feature gate, the call fails in order to avoid unsupported scenarios that might otherwise run silently.

In some cases, removing pre-GA features requires considerable time. Feature gates can remain operational until their associated feature is fully removed, at which point the feature gate itself can be deprecated.

When removing a feature gate for a GA feature also requires considerable time, calls to feature gates may remain operational if the feature gate has no effect on the feature, and if the feature gate causes no errors.

Features intended to be disabled by users should include a mechanism for disabling the feature in the associated feature gate.

Versioning for feature gates is different from the previously discussed components, therefore the rules for deprecation are as follows:

Rule #8: Feature gates must be deprecated when the corresponding feature they control transitions a lifecycle stage as follows. Feature gates must function for no less than:

• Beta feature to GA: 6 months or 2 releases (whichever is longer)
• Beta feature to EOL: 3 months or 1 release (whichever is longer)
• Alpha feature to EOL: 0 releases

Rule #9: Deprecated feature gates must respond with a warning when used. When a feature gate is deprecated it must be documented in both in the release notes and the corresponding CLI help. Both warnings and documentation must indicate whether a feature gate is non-operational.

Deprecating a metric

Each component of the Kubernetes control-plane exposes metrics (usually the /metrics endpoint), which are typically ingested by cluster administrators. Not all metrics are the same: some metrics are commonly used as SLIs or used to determine SLOs, these tend to have greater import. Other metrics are more experimental in nature or are used primarily in the Kubernetes development process.

Accordingly, metrics fall under two stability classes (ALPHA and STABLE); this impacts removal of a metric during a Kubernetes release. These classes are determined by the perceived importance of the metric. The rules for deprecating and removing a metric are as follows:

Rule #9a: Metrics, for the corresponding stability class, must function for no less than:

• STABLE: 4 releases or 12 months (whichever is longer)
• ALPHA: 0 releases

Rule #9b: Metrics, after their announced deprecation, must function for no less than:

• STABLE: 3 releases or 9 months (whichever is longer)
• ALPHA: 0 releases

Deprecated metrics will have their description text prefixed with a deprecation notice string '(Deprecated from x.y)' and a warning log will be emitted during metric registration. Like their stable undeprecated counterparts, deprecated metrics will be automatically registered to the metrics endpoint and therefore visible.

On a subsequent release (when the metric's deprecatedVersion is equal to current_kubernetes_version - 3)), a deprecated metric will become a hidden metric. Unlike their deprecated counterparts, hidden metrics will no longer be automatically registered to the metrics endpoint (hence hidden). However, they can be explicitly enabled through a command line flag on the binary (--show-hidden-metrics-for-version=). This provides cluster admins an escape hatch to properly migrate off of a deprecated metric, if they were not able to react to the earlier deprecation warnings. Hidden metrics should be deleted after one release.

Exceptions

No policy can cover every possible situation. This policy is a living document, and will evolve over time. In practice, there will be situations that do not fit neatly into this policy, or for which this policy becomes a serious impediment. Such situations should be discussed with SIGs and project leaders to find the best solutions for those specific cases, always bearing in mind that Kubernetes is committed to being a stable system that, as much as possible, never breaks users. Exceptions will always be announced in all relevant release notes.

5 - Deprecated API Migration Guide

As the Kubernetes API evolves, APIs are periodically reorganized or upgraded. When APIs evolve, the old API is deprecated and eventually removed. This page contains information you need to know when migrating from deprecated API versions to newer and more stable API versions.

Removed APIs by release

v1.25

The v1.25 release will stop serving the following deprecated API versions:

CronJob

The batch/v1beta1 API version of CronJob will no longer be served in v1.25.

• Migrate manifests and API clients to use the batch/v1 API version, available since v1.21.
• All existing persisted objects are accessible via the new API
• No notable changes

EndpointSlice

The discovery.k8s.io/v1beta1 API version of EndpointSlice will no longer be served in v1.25.

• Migrate manifests and API clients to use the discovery.k8s.io/v1 API version, available since v1.21.
• All existing persisted objects are accessible via the new API
• Notable changes in discovery.k8s.io/v1:
  • use per Endpoint nodeName field instead of deprecated topology["kubernetes.io/hostname"] field
  • use per Endpoint zone field instead of deprecated topology["topology.kubernetes.io/zone"] field
  • topology is replaced with the deprecatedTopology field which is not writable in v1

Event

The events.k8s.io/v1beta1 API version of Event will no longer be served in v1.25.

• Migrate manifests and API clients to use the events.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• Notable changes in events.k8s.io/v1:
  • type is limited to Normal and Warning
  • involvedObject is renamed to regarding
  • action, reason, reportingController, and reportingInstance are required when creating new events.k8s.io/v1 Events
  • use eventTime instead of the deprecated firstTimestamp field (which is renamed to deprecatedFirstTimestamp and not permitted in new events.k8s.io/v1 Events)
  • use series.lastObservedTime instead of the deprecated lastTimestamp field (which is renamed to deprecatedLastTimestamp and not permitted in new events.k8s.io/v1 Events)
  • use series.count instead of the deprecated count field (which is renamed to deprecatedCount and not permitted in new events.k8s.io/v1 Events)
  • use reportingController instead of the deprecated source.component field (which is renamed to deprecatedSource.component and not permitted in new events.k8s.io/v1 Events)
  • use reportingInstance instead of the deprecated source.host field (which is renamed to deprecatedSource.host and not permitted in new events.k8s.io/v1 Events)

PodDisruptionBudget

The policy/v1beta1 API version of PodDisruptionBudget will no longer be served in v1.25.

• Migrate manifests and API clients to use the policy/v1 API version, available since v1.21.
• All existing persisted objects are accessible via the new API
• Notable changes in policy/v1:
  • an empty spec.selector ({}) written to a policy/v1 PodDisruptionBudget selects all pods in the namespace (in policy/v1beta1 an empty spec.selector selected no pods). An unset spec.selector selects no pods in either API version.

PodSecurityPolicy

PodSecurityPolicy in the policy/v1beta1 API version will no longer be served in v1.25, and the PodSecurityPolicy admission controller will be removed.

PodSecurityPolicy replacements are still under discussion, but current use can be migrated to 3rd-party admission webhooks now.

RuntimeClass

RuntimeClass in the node.k8s.io/v1beta1 API version will no longer be served in v1.25.

• Migrate manifests and API clients to use the node.k8s.io/v1 API version, available since v1.20.
• All existing persisted objects are accessible via the new API
• No notable changes

v1.22

The v1.22 release will stop serving the following deprecated API versions:

Webhook resources

The admissionregistration.k8s.io/v1beta1 API version of MutatingWebhookConfiguration and ValidatingWebhookConfiguration will no longer be served in v1.22.

• Migrate manifests and API clients to use the admissionregistration.k8s.io/v1 API version, available since v1.16.
• All existing persisted objects are accessible via the new APIs
• Notable changes:
  • webhooks[*].failurePolicy default changed from Ignore to Fail for v1
  • webhooks[*].matchPolicy default changed from Exact to Equivalent for v1
  • webhooks[*].timeoutSeconds default changed from 30s to 10s for v1
  • webhooks[*].sideEffects default value is removed, and the field made required, and only None and NoneOnDryRun are permitted for v1
  • webhooks[*].admissionReviewVersions default value is removed and the field made required for v1 (supported versions for AdmissionReview are v1 and v1beta1)
  • webhooks[*].name must be unique in the list for objects created via admissionregistration.k8s.io/v1

CustomResourceDefinition

The apiextensions.k8s.io/v1beta1 API version of CustomResourceDefinition will no longer be served in v1.22.

• Migrate manifests and API clients to use the apiextensions.k8s.io/v1 API version, available since v1.16.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.scope is no longer defaulted to Namespaced and must be explicitly specified
  • spec.version is removed in v1; use spec.versions instead
  • spec.validation is removed in v1; use spec.versions[*].schema instead
  • spec.subresources is removed in v1; use spec.versions[*].subresources instead
  • spec.additionalPrinterColumns is removed in v1; use spec.versions[*].additionalPrinterColumns instead
  • spec.conversion.webhookClientConfig is moved to spec.conversion.webhook.clientConfig in v1
  • spec.conversion.conversionReviewVersions is moved to spec.conversion.webhook.conversionReviewVersions in v1
  • spec.versions[*].schema.openAPIV3Schema is now required when creating v1 CustomResourceDefinition objects, and must be a structural schema
  • spec.preserveUnknownFields: true is disallowed when creating v1 CustomResourceDefinition objects; it must be specified within schema definitions as x-kubernetes-preserve-unknown-fields: true
  • In additionalPrinterColumns items, the JSONPath field was renamed to jsonPath in v1 (fixes #66531)

APIService

The apiregistration.k8s.io/v1beta1 API version of APIService will no longer be served in v1.22.

• Migrate manifests and API clients to use the apiregistration.k8s.io/v1 API version, available since v1.10.
• All existing persisted objects are accessible via the new API
• No notable changes

TokenReview

The authentication.k8s.io/v1beta1 API version of TokenReview will no longer be served in v1.22.

• Migrate manifests and API clients to use the authentication.k8s.io/v1 API version, available since v1.6.
• No notable changes

SubjectAccessReview resources

The authorization.k8s.io/v1beta1 API version of LocalSubjectAccessReview, SelfSubjectAccessReview, and SubjectAccessReview will no longer be served in v1.22.

• Migrate manifests and API clients to use the authorization.k8s.io/v1 API version, available since v1.6.
• Notable changes:
  • spec.group was renamed to spec.groups in v1 (fixes #32709)

CertificateSigningRequest

The certificates.k8s.io/v1beta1 API version of CertificateSigningRequest will no longer be served in v1.22.

• Migrate manifests and API clients to use the certificates.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• Notable changes in certificates.k8s.io/v1:
  • For API clients requesting certificates:
    • spec.signerName is now required (see known Kubernetes signers), and requests for kubernetes.io/legacy-unknown are not allowed to be created via the certificates.k8s.io/v1 API
    • spec.usages is now required, may not contain duplicate values, and must only contain known usages
  • For API clients approving or signing certificates:
    • status.conditions may not contain duplicate types
    • status.conditions[*].status is now required
    • status.certificate must be PEM-encoded, and contain only CERTIFICATE blocks

Lease

The coordination.k8s.io/v1beta1 API version of Lease will no longer be served in v1.22.

• Migrate manifests and API clients to use the coordination.k8s.io/v1 API version, available since v1.14.
• All existing persisted objects are accessible via the new API
• No notable changes

Ingress

The extensions/v1beta1 and networking.k8s.io/v1beta1 API versions of Ingress will no longer be served in v1.22.

• Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.backend is renamed to spec.defaultBackend
  • The backend serviceName field is renamed to service.name
  • Numeric backend servicePort fields are renamed to service.port.number
  • String backend servicePort fields are renamed to service.port.name
  • pathType is now required for each specified path. Options are Prefix, Exact, and ImplementationSpecific. To match the undefined v1beta1 behavior, use ImplementationSpecific.

IngressClass

The networking.k8s.io/v1beta1 API version of IngressClass will no longer be served in v1.22.

• Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.19.
• All existing persisted objects are accessible via the new API
• No notable changes

RBAC resources

The rbac.authorization.k8s.io/v1beta1 API version of ClusterRole, ClusterRoleBinding, Role, and RoleBinding will no longer be served in v1.22.

• Migrate manifests and API clients to use the rbac.authorization.k8s.io/v1 API version, available since v1.8.
• All existing persisted objects are accessible via the new APIs
• No notable changes

PriorityClass

The scheduling.k8s.io/v1beta1 API version of PriorityClass will no longer be served in v1.22.

• Migrate manifests and API clients to use the scheduling.k8s.io/v1 API version, available since v1.14.
• All existing persisted objects are accessible via the new API
• No notable changes

Storage resources

The storage.k8s.io/v1beta1 API version of CSIDriver, CSINode, StorageClass, and VolumeAttachment will no longer be served in v1.22.

• Migrate manifests and API clients to use the storage.k8s.io/v1 API version
  • CSIDriver is available in storage.k8s.io/v1 since v1.19.
  • CSINode is available in storage.k8s.io/v1 since v1.17
  • StorageClass is available in storage.k8s.io/v1 since v1.6
  • VolumeAttachment is available in storage.k8s.io/v1 v1.13
• All existing persisted objects are accessible via the new APIs
• No notable changes

v1.16

The v1.16 release stopped serving the following deprecated API versions:

NetworkPolicy

The extensions/v1beta1 API version of NetworkPolicy is no longer served as of v1.16.

• Migrate manifests and API clients to use the networking.k8s.io/v1 API version, available since v1.8.
• All existing persisted objects are accessible via the new API

DaemonSet

The extensions/v1beta1 and apps/v1beta2 API versions of DaemonSet are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.templateGeneration is removed
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  • spec.updateStrategy.type now defaults to RollingUpdate (the default in extensions/v1beta1 was OnDelete)

Deployment

The extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions of Deployment are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.rollbackTo is removed
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  • spec.progressDeadlineSeconds now defaults to 600 seconds (the default in extensions/v1beta1 was no deadline)
  • spec.revisionHistoryLimit now defaults to 10 (the default in apps/v1beta1 was 2, the default in extensions/v1beta1 was to retain all)
  • maxSurge and maxUnavailable now default to 25% (the default in extensions/v1beta1 was 1)

StatefulSet

The apps/v1beta1 and apps/v1beta2 API versions of StatefulSet are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  • spec.updateStrategy.type now defaults to RollingUpdate (the default in apps/v1beta1 was OnDelete)

ReplicaSet

The extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions of ReplicaSet are no longer served as of v1.16.

• Migrate manifests and API clients to use the apps/v1 API version, available since v1.9.
• All existing persisted objects are accessible via the new API
• Notable changes:
  • spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades

PodSecurityPolicy

The extensions/v1beta1 API version of PodSecurityPolicy is no longer served as of v1.16.

• Migrate manifests and API client to use the policy/v1beta1 API version, available since v1.10.
• Note that the policy/v1beta1 API version of PodSecurityPolicy will be removed in v1.25.

What to do

Test with deprecated APIs disabled

You can test your clusters by starting an API server with specific API versions disabled to simulate upcoming removals. Add the following flag to the API server startup arguments:

--runtime-config=<group>/<version>=false

For example:

--runtime-config=admissionregistration.k8s.io/v1beta1=false,apiextensions.k8s.io/v1beta1,...

Locate use of deprecated APIs

Use client warnings, metrics, and audit information available in 1.19+ to locate use of deprecated APIs.

Migrate to non-deprecated APIs

• Update custom integrations and controllers to call the non-deprecated APIs

• Change YAML files to reference the non-deprecated APIs

  You can use the kubectl-convert command (kubectl convert prior to v1.20) to automatically convert an existing object:

  kubectl-convert -f <file> --output-version <group>/<version>.

  For example, to convert an older Deployment to apps/v1, you can run:

  kubectl-convert -f ./my-deployment.yaml --output-version apps/v1

  Note that this may use non-ideal default values. To learn more about a specific resource, check the Kubernetes API reference.

6 - Kubernetes API health endpoints

The Kubernetes API server provides API endpoints to indicate the current status of the API server. This page describes these API endpoints and explains how you can use them.

API endpoints for health

The Kubernetes API server provides 3 API endpoints (healthz, livez and readyz) to indicate the current status of the API server. The healthz endpoint is deprecated (since Kubernetes v1.16), and you should use the more specific livez and readyz endpoints instead. The livez endpoint can be used with the --livez-grace-period flag to specify the startup duration. For a graceful shutdown you can specify the --shutdown-delay-duration flag with the /readyz endpoint. Machines that check the health/livez/readyz of the API server should rely on the HTTP status code. A status code 200 indicates the API server is healthy/live/ready, depending of the called endpoint. The more verbose options shown below are intended to be used by human operators to debug their cluster or specially the state of the API server.

The following examples will show how you can interact with the health API endpoints.

For all endpoints you can use the verbose parameter to print out the checks and their status. This can be useful for a human operator to debug the current status of the API server, it is not intended to be consumed by a machine:

curl -k https://localhost:6443/livez?verbose

or from a remote host with authentication:

kubectl get --raw='/readyz?verbose'

The output will look like this:

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
healthz check passed

The Kubernetes API server also supports to exclude specific checks. The query parameters can also be combined like in this example:

curl -k 'https://localhost:6443/readyz?verbose&exclude=etcd'

The output show that the etcd check is excluded:

[+]ping ok
[+]log ok
[+]etcd excluded: ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]shutdown ok
healthz check passed

Individual health checks

Each individual health check exposes an HTTP endpoint and could can be checked individually. The schema for the individual health checks is /livez/<healthcheck-name> where livez and readyz and be used to indicate if you want to check the liveness or the readiness of the API server. The <healthcheck-name> path can be discovered using the verbose flag from above and take the path between [+] and ok. These individual health checks should not be consumed by machines but can be helpful for a human operator to debug a system:

curl -k https://localhost:6443/livez/etcd