How to Download Logs from a Kubernetes Node for investigation

Recently I encountered an issue with a Kubernetes Cluster not routing to the Internet properly. It was first discovered by the Product Team during the execution of a Production Release. Pods were repeating the CrashLoopBackoff cycle. The logs revealed one of the containers was unable to establish a connection to Azure Service Bus. This resulted in a 10 hour marathon working with Azure Support to identify and resolve the issue. Thankfully we were able to initiate our Business Continuity Disaster Recovery Plan and rolled all customer workloads to an alternate Region.

During this event Microsof Azure Rapid Response Technical Support requested copies of IP Tables dumps etc to allow them to investigate. For experienced Linux Administrators this would seem simple enough - ssh admin@cluster -c 'iptables -L' > iptables.out, right? Not so quick on a Managed Kubernetes Cluster. One of seveal solutions to this problem is to use Node Shell to access the Kubernetes node.

The following demonstration is based on Azure Kubernetes Service. The kubectl commands are universal to any Kubernetes Cluster.

List the clusters in the Subscription.

# List the Clusters
az aks list

Obtain the credentials for the Cluster

# Get the Credentials
az aks get-cedentials --name <name of the cluster> --resource-group <resource group name>

List the nodes

kubect get nodes --output wide

Enter the Node. Essentially we are SSHing into the node via the Node Shell. As this demo is based on a "managed" deployment, I do not have the ability to directly SSH into the node. Albeit, yes - I could have created a pod with the proper private SSH key to accomplish the same, but in the interest of time ...

kubectl node-shell aks-cluster1-33854278-vmss000000

What has happened at this point is that a nsenter pod has been created for you on the node. When you exit the node-shell session the pod will be removed automatically.

Package the messages and syslog files up in a tar ball. Here I am using GZip compression. Note that for distinction I am using the pattern of <node name>.<log file type/name>.tar.gz. This identifies that we are creating a tar ball using gzip compression and which type of files for which machine.

tar czf aks-cluster1-33854278-vmss000000.messages.tar.gz /var/log/messages*
tar czf aks-cluster1-33854278-vmss000000.syslog.tar.gz /var/log/syslog*

Do not exit this terminal session yet.

Now we need to leverage the pod that was created so we can transfer off the archived files. To do this we need to get the name of the pod. So here we will get a list of pods in the default namespace. We need to identify the pod having a name starting with nsenter.

kubectl get pods --namespace default

NAME             READY   STATUS    RESTARTS   AGE
nsenter-9knomx   1/1     Running   0          14m

Now that we have the name of the pod we can copy the archives to your local workstation. Here we use the kubectl cp command. Note the path to the files created ealier are referenced following the : following the pod name. This is the same methodology used in Rsync and SCP.

kubectl cp nsenter-9knomx:/tmp/aks-cluster1-33854278-vmss000000.syslog.tar.gz aks-cluster1-33854278-vmss000000.syslog.tar.gz

Now you can list the files you downloaded.

#Linux
ls -l *.gz
-rw-r--r--  1 srhodes  staff  15280652 Apr 16 20:26 aks-cluster1-33854278-vmss000000.messages.tar.gz
-rw-r--r--  1 srhodes  staff  14671049 Apr 16 20:27 aks-cluster1-33854278-vmss000000.syslog.tar.gz

For non-*nix users, you can use dir in place of ls.