What is drain?

kubectl drain is a command used in Kubernetes to safely evict all pods from a node in preparation for maintenance or updates, ensuring minimal disruption to services.

It marks a node as unschedulable (cordon) and then it gracefully evicts all running pods, allowing them to be reschedulable on other nodes. This command ensures preventing service interruptions, and handles everything smoothly during maintenance.

kubectl drain NODE_NAME

Detailed Explanation

The drain command marks the node unschedulable to prevent new pods from arriving. It evicts the pods if the API server supports eviction, else it will delete the pods.

The Daemon set-managed pods cannot be drain’ed without using --ignore-daemonsets, because those pods would immediately be replaced by the daemon set controller, which ignores unschedulable markings.

kubectl drain NODE_NAME --ignore-daemonsets

If there are any pods that are neither mirror pods nor managed by a:

  • Replication Controller
  • Replica Set
  • Daemon Set
  • Stateful Set
  • Job

then, the drain will not delete any pods unless you use --force, which will allow deletion to proceed if the managing resource of one or more pods is missing.

# drain even if they are not managed by (above mentioned)
kubectl drain NODE_NAME --force

# abort drain if they are not managed by (above mentioned)
kubectl drain NODE_NAME --grace-period=900

Essential Commands

  • Local data used by pods are deleted when the node is drained.
kubectl drain NODE_NAME --delete-emptydir-data
  • DELETE pods even if eviction is supported. (Bypasses checking PodDisruptionBudgets)
kubectl drain NODE_NAME --disable-eviction
  • Terminating pods gracefully within the given seconds/time
kubectl drain NODE_NAME --grade-period=8 #-1 uses the default value specified in pod
  • Duration before giving up, (0 = infinite)
kubectl drain NODE_NAME --timeout-duration

The above mentioned commands are the one you’ll most probably use everyday, but checkout the official documentation to learn more about other available options.

Example

Using drain command to rename the nodes, Checkout this post to learn more.

In order to rename the node:

  1. Drain the node (On Control Plane)
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data
  1. Stop the systemd service and rename the hostname (On Worker Node)
sudo systemctl stop k3s-agent.service
sudo hostnamectl set-hostname NEW_HOSTNAME
  1. Update the records in /etc/host(On Worker Node)
#  i.e: rename `OLD_HOSTNAME` to `NEW_HOSTNAME` 
sudoedit /etc/hosts
  1. Delete the node (On Control Plane)
kubectl delete node NODE_NAME
  1. Reinstall k3s-agent (On Worker Node)
  • Copy node-token
sudo cat /var/lib/rancher/k3s/server/node-token
  • Execute the installation script
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.31.11:6443 K3S_TOKEN=<token> K3S_NODE_NAME=<NEW_HOSTNAME> sh -

This is one of the way I use drain often times to re-register k3s-agents. But there are other ways to use it as well, this is just an example to give you an overall idea of how drain command plays a vital role in evicting pods without breaking stuff.