In Oct 2019, Ahmet Alp Balkan wrote this blog post that explains how to use mitmproxy to observe the requests made by kubectl. But I couldn’t use the tutorial for two reasons:
- I use
kindto create local clusters which means I hit the Gonet/httplimitation (skips proxying for hostslocalhostand127.0.0.1, see this blog post) - I use client certs authentication, which can’t work with the method presented by Ahmet; it can only work for header-based authentication (e.g. token) but not for client certs.
In the following, I detail how I managed to make all that work.
# Let's use a separate kubeconfig file to avoid messing with ~/.kube/config.
export KUBECONFIG=~/.kube/kind-kind
# Now, let's create a cluster.
kind create cluster
# Let's see what the content of $KUECONFIG is:
cat $KUBECONFIG
We can see that the kubeconfig uses a client cert as a way to authenticate to the apiserver.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EWXlPVEE1TURJeU4xb1hEVE13TURZeU56QTVNREl5TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS25qCmhmRzBvenZVb05jMXY1STkvYm13dFBqb2QvK0RyczF4TFZOcWgxQjhFcTY1S3lnQjBSbDdQcTJueUhyRVRnTmsKZ2VPQ2RzRURObGpKOE12SC9GbE9nRkUvS0dqK2hwN2hwc0dHRExReWFUOExUY25JN0FNL2t3KzY5R0hidlN3Nwo2V0Y1VERTMDU1RkRqdnRveGdHcERycmZQZTk5bXN5Zmk3aWtteDk5MmRyMHFQd0xxanJpZHNkWU52MUZqU1Y1CndkRlFISGxBS2hBcmlUWmpQMnhNL3poOENBOFhndjF0UUxVVk5IS1hrSG5UYWlkeFY1MkduaUVaQmd0d2tSK3oKQ3hQempVZFAxQ1JRYzU0YmxDYW9FQVdTc0NYTUVPREhTSnowSi9CWHJCU2JaeGZIakd0Y0k0bEhBUmx2aURNawp3U3lOLy9qdE9tbWhDT29BTzBzQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFIMDBrYnZpaWNNT3IxdFJoTkQweVpGa3ZkT1AKUzFGOEZKK1BFd1o1WExUTVVyVG1yekVlZmgza21xWkxYUnlyK3c0Snk1a0grK1o3enBpdlp6Q3BGOEtwclJaWAp5N2Z6TkJOeWMrOHFKN3dCek0xZ21BdTRha3BlNVBYbkw3akZIcU9aUmNXTmZKcUpTci9BS05sK2c2SjAzV2pnCmJDc0NQNTNrZ1czSDZYMkRoS1JzZFRWQlc1UGVlek1YRVlON2VYbGpnWE4xcElMdEU1VE5oWVZIcTZaUDVCaDkKMmU2Y1U4bEFuNlhYQ0Ira2RsTkhudGp1cTlSL2lPQVJRaWpVSVlWSFdCT1NyWGp6TnhkTklEakRRd3lKUnViMApDVEhrcDg4eUlNaEV1Nk1OV1VCUU82ZnVqbURHbUJJbTlzdzJpSnI1UlArSUVUVnQyWFc0Q2dHT1cwVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://127.0.0.1:53744
name: kind-helix
contexts:
- context:
cluster: kind-helix
user: kind-helix
name: kind-helix
current-context: kind-helix
kind: Config
preferences: {}
users:
- name: kind-helix
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJYTZXc25HT0RGZW93RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TURBMk1qa3dPVEF5TWpkYUZ3MHlNVEEyTWprd09UQXlNekJhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXRkbkxsb0V1cnFoSmRrMGgKME5VcXpFSHhUbmVHS040QTNtWDBobmFLcE1TT3hISlBQcTB6V0t4WEVxNTZPTkdhSkhvS0VTRUM2Yjh2MGcwTQo5dXVGOXJyQ1VORkpXMmdHcFArRG96TEpVS3RtN2F0WFZRYXNVQ2tjbFRtQ003aHlqOGZTTlRDd2lxT3RNOUFvCnFVVTJYd1k0a2xhL0RuMkRBc1V1VmNiUTdITDA1N0tFbjZvVzlFVy9mQ2hMVE5OTWhmelpkNzFISGs3MFNOZDMKTG5jTXNjNmp2eS9kN2MwbjM4ek45SjVzWVZOTjNsS1YvOU5MT2FaTEQ0bSsyVU5XV1FVazRuci8rZjlaZk1IMwphVlFibDZSWEZwaW1jYWg4UjRJTmhRNkhYbmVEbUI2dUl3RGdjQnhhQUtoNFVPVlZwODB1Uy9pYUVLV1VSWE1PCkg4YXBZUUlEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFJaWtmbis5TkJQc2FNSzNVMU5pWmtBaWxsM1pWVXJDdzNsWgpIeGRGbm9MZGxZYmtPeFVEN25EK3lrNXhZWW4yaDc3WUU4NlU5czJ3aitkdFJTR2Ezc3p4WVJQbkt5MDN3L1M4CkhqZzhwSkZOZFhHWlNPUWJDQTBmT1BONXl5MlpYRlVWd0JwZ3VTMC9nTkNUUkRPUDl3QmNjQXhiSGRmSjMxQzQKa2hCemF4QXI5UEliMVBzUlhqS2ZSRnkvcllwYWRBYVhhYmZMOFRvbTJ4cGpLVDYreGxoL3lJb2tZbVhxSnlXRgp1SGFmWG1qUEdyRDFoZUo2UnR5U0xRM0dKUXQxWmVPK3BaYlJlRjcxU0ZRSjRQdFhtTHhGMjZQL3dlV3F0NWFJClhTK0ZnanRYbzZqUG1mWWRweDZ3bWp3UHRaZ1pRdXhSL2tsejVvQnErNnFaYUZDWXg4RT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdGRuTGxvRXVycWhKZGswaDBOVXF6RUh4VG5lR0tONEEzbVgwaG5hS3BNU094SEpQClBxMHpXS3hYRXE1Nk9OR2FKSG9LRVNFQzZiOHYwZzBNOXV1RjlyckNVTkZKVzJnR3BQK0RvekxKVUt0bTdhdFgKVlFhc1VDa2NsVG1DTTdoeWo4ZlNOVEN3aXFPdE05QW9xVVUyWHdZNGtsYS9EbjJEQXNVdVZjYlE3SEwwNTdLRQpuNm9XOUVXL2ZDaExUTk5NaGZ6WmQ3MUhIazcwU05kM0xuY01zYzZqdnkvZDdjMG4zOHpOOUo1c1lWTk4zbEtWCi85TkxPYVpMRDRtKzJVTldXUVVrNG5yLytmOVpmTUgzYVZRYmw2UlhGcGltY2FoOFI0SU5oUTZIWG5lRG1CNnUKSXdEZ2NCeGFBS2g0VU9WVnA4MHVTL2lhRUtXVVJYTU9IOGFwWVFJREFRQUJBb0lCQVFDVkNJOXZJeVB0QkFKZwpyOG44NmhhUEc2UDFtTU1jandUTFAyZHRJNDF3aDU0eHBUVUl1czJQNkgzYjA1NWJIbnhqVkprWGZLUjBpTGxhClBsUFhzU0l6R00vVGlCSEVsYmFNVnRPOVZndml6dllsNWZ4R3RKZFhncm5vR2g5NDM3c1Qxc0dSMGZ0OVE3TFkKK2NtNUgvMzFWcFhhYUxsZjJNRWI3aG1STnNWV1lXWm9MUHJ2QUJPemVmUnpKU0RONHU0M3M0eVpnNHJoL3IzWQo3YUhYOS9CSHpJRk93WTRNL1BRQzdYaFhDMXBIeWNPY2lSWVhFTkpuMTdQN2NOSkdoMnJ4dnc2OVQ1QW9rdXY5Ckcxd2lUNmVrVmZuaVYrSlVnL1lwcTNSRGxNdVFETDRZdCtGNG1zVGJtN3NFZk5yVVMzWGZFMFdmcjB2Wk1YN2sKMnE3dXkxNEJBb0dCQU5jb1BxVHNBa3N5YzR6UWtmQmNzb0Y1ekdJM3NQTUFGTHRSMGF6S2x1Q1V3VGMzYVk5dgpiTE05OTBVK3VpVzdtRHF0VHpZMVNVSzZGNW0yMzJoSW5hTnBCTUVJcDZHQ0I3dis5WWZIQURVSkljaVdodmJhCmpIY0M5Qjl4SG5uUTBydU5aNUErdWVtRE9EZkRKUllIVUhrMEh3MlJpTE1qRFhVRmhBanJBUGJ4QW9HQkFOaGUKL3BHb2FWOWtUREtNOWZwTTZUQkZwMFVtb2lSb3crVG9TWjhWeVBmMkl1VlNpa0dtaXUydHU3MzZCZkJtVzNtQgpGWmFRc01rNkMydExkK1NBQ0lSbktQTUN4czYzR05WNkJ0aXRIRFlBaGNtRCtvU3VpaFRrd3l1WXlpQml4aFovCms1UDY3UHFKVkQ2YnhkNDV6YlFsS2VNaUVtMUhnQUVGSEF0UFBqbHhBb0dBYzNnaHhwankwakNOV3ZGRW9WN2UKWGlaanpnSmRjTXlHVTlHaFdiNlFJbzh5OHROR1Q3aFkrZ2t6ZjNJZXJNbDA5V2kxcmo0Q3gxRGdBWnJuWXl3MQpqZEY2djY1SmFLQkVUbHlTb1AvbjJJN0NGc2pTUGdFa2lXcUlZYWR2MTZoK3NERS9kMlp5bUNQWU0vVURIa05tCnFPV1VGTkFhTVNtS3UxYnVlV3JGNWNFQ2dZQnJvNTVySWVnQjM2aVVnVkdoU24rN1Z2dG15RmhqV29jUnFvbHQKamUzamhWeEl6eTRlaU5hV2RSWnY1U0R0UGs2RmZMVWJxVEY1ZWRuU2I4SGVOOStFMXJrbFk1MDVteGJNcEo4aApUY1U2RERxQ1RKamxSdHRFbDZXTVc3ODZLMGsyU2hORnk4LzJ0empreUtPLzhPdW5rZEZyd0RpQWl0QmdNWVdKCkRzdjYwUUtCZ1FDR2htYnZrODRaYm1sdERxYW9hOVR6YU5UT3FrN3dtb0tDby9FVEg5NEtNVVZMVmk3bHhpN1kKcXFQYUdiSm9aYkZGd0NvWW12Rk5jU21nYk9SNjdCVVBkdUEzMHo0VythV1pmQVkwZGpUR08yTC8zVVhoWHdHegpVUGpZTXZZQU5sdzFmZTVmZmk3UExJSGJvZXFhTmhhaDliRVJXdFNRbm95NnYzV2hlNTh6VkE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
Let’s run mitmproxy using the above client cert and keys:
mitmproxy -p 9000 --ssl-insecure --set client_certs=<(kubectl config view --minify --flatten -o=go-template='{{(index (index .users 0).user "client-key-data")}}' | base64 -d && kubectl config view --minify --flatten -o=go-template='{{(index (index .users 0).user "client-certificate-data")}}' | base64 -d)
Now, let’s make sure that we have some DNS alias to 127.0.0.1 to work around the Go proxy limitation which skips proxying for 127.0.0.1 and localhost:
grep "^127.0.0.1.*local$" /etc/hosts || sudo bash -c 'echo "127.0.0.1 local" >> /etc/hosts'
Finally you can run kubectl to see the logs of kube-scheduler:
% HTTPS_PROXY=:9000 kubectl --kubeconfig=<(sed "s|127.0.0.1|local|" $KUBECONFIG) --insecure-skip-tls-verify logs -n kube-system -l component=kube-scheduler,tier=control-plane
I0701 11:11:11.585249 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0701 11:11:11.585329 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0701 11:11:12.976581 1 serving.go:313] Generated self-signed cert in-memory
W0701 11:11:17.407371 1 authentication.go:349] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0701 11:11:17.407465 1 authentication.go:297] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0701 11:11:17.407483 1 authentication.go:298] Continuing without authentication configuration. This may treat all requests as anonymous.
W0701 11:11:17.407513 1 authentication.go:299] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0701 11:11:17.478349 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0701 11:11:17.478483 1 registry.go:150] Registering EvenPodsSpread predicate and priority function
W0701 11:11:17.491616 1 authorization.go:47] Authorization is disabled
W0701 11:11:17.491729 1 authentication.go:40] Authentication is disabled
I0701 11:11:17.491883 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0701 11:11:17.500576 1 secure_serving.go:178] Serving securely on 127.0.0.1:10259
I0701 11:11:17.500678 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0701 11:11:17.500919 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0701 11:11:17.500931 1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0701 11:11:17.718085 1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0701 11:11:17.801300 1 leaderelection.go:242] attempting to acquire leader lease kube-system/kube-scheduler...
I0701 11:11:34.659176 1 leaderelection.go:252] successfully acquired lease kube-system/kube-scheduler
It works!! Here is what mitmproxy is showing:
18:17:42 GET HTTPS local /api/v1/namespaces/kube-system/pods/kube-scheduler-helix-control-plane 200 …plication/json 5.2k 119ms
18:17:42 GET HTTPS local /api/v1/namespaces/kube-system/pods/kube-scheduler-helix-control-plane/log 200 text/plain 2.46k 227ms