Kubernetes — Debugging NetworkPolicy (Part 3)

What should you do if you don’t know exactly what egress traffic is being blocked?

A word of advice: don’t install debug tools in your application image or application containers

Example

FROM alpine:latestRUN apk update && \
apk --no-cache add \
bash \
curl
FROM alpine:latestRUN apk update && \
apk --no-cache add \
bash \
tcpdump
CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait"
> docker build --no-cache --progress=plain -f docker\Dockerfile.app -t do-wget:1.0.0 docker\> docker build --no-cache --progress=plain -f docker\Dockerfile.debug -t debug-tools:1.0.0 docker\
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
namespace: local-demo-debugnetworkpolicy-ns
spec:
selector:
matchLabels:
app.kubernetes.io/name: my-deployment
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/name: my-deployment
spec:
containers:
- name: do-wget
image: do-wget:1.0.0
imagePullPolicy: IfNotPresent
command:
- bash
- -c
- |
while true; do
curl example.com --connect-timeout 1 > file.txt
sleep 10
done
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 50m
memory: 64Mi

You might be able to use debugging tools in an ephemeral container

You may be able to use a temporary sidecar container

apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
namespace: local-demo-debugnetworkpolicy-ns
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: my-deployment
template:
metadata:
labels:
app.kubernetes.io/name: my-deployment
spec:
containers:
- command:
- bash
- -c
- |
while true; do
curl example.com --connect-timeout 1 > file.txt
sleep 10
done
image: do-wget:1.0.0
imagePullPolicy: IfNotPresent
name: do-wget
resources:
limits:
cpu: 50m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
- image: debug-tools:1.0.0
imagePullPolicy: IfNotPresent
name: debugtools-sidecar
resources:
limits:
cpu: 50m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
bases:
- ../local
patchesJson6902:
- target:
version: v1
group: apps
kind: Deployment
name: my-deployment
path: inject-debugtools-sidecar-patch.yaml
- op: add
path: "/spec/template/spec/containers/-"
value:
name: debugtools-sidecar
image: debug-tools:1.0.0
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 50m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
>kubectl kustomize k8s\overlays\debug | kubectl apply -f -
namespace/local-demo-debugnetworkpolicy-ns unchanged
deployment.apps/my-deployment configured
networkpolicy.networking.k8s.io/allow-allpods-to-dns unchanged
networkpolicy.networking.k8s.io/deny-all unchanged
>kubectl -n local-demo-debugnetworkpolicy-ns get pod
NAME READY STATUS RESTARTS AGE
my-deployment-5cccff8466-h4bws 2/2 Running 0 29s
>kubectl -n local-demo-debugnetworkpolicy-ns exec -it my-deployment-5cccff8466-h4bws -c debugtools-sidecar -- bash
bash-5.1# tcpdump -i eth0

Interpreting tcpdump output

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
19:36:41.508244 IP my-deployment-5cccff8466-h4bws.59813 > kube-dns.kube-system.svc.cluster.local.53: 47110+ A? example.com.local-demo-debugnetworkpolicy-ns.svc.cluster.local. (80)
19:36:41.508427 IP my-deployment-5cccff8466-h4bws.59813 > kube-dns.kube-system.svc.cluster.local.53: 47910+ AAAA? example.com.local-demo-debugnetworkpolicy-ns.svc.cluster.local. (80)
<snip>
19:36:41.510322 IP my-deployment-5cccff8466-h4bws.49736 > 93.184.216.34.80: Flags [S], seq 3928263212, win 64800, options [mss 1440,sackOK,TS val 1896154571 ecr 0,nop,wscale 7], length 0
19:36:41.518392 IP my-deployment-5cccff8466-h4bws.43159 > kube-dns.kube-system.svc.cluster.local.53: 42685+ PTR? 10.0.96.10.in-addr.arpa. (41)
19:36:41.525286 IP kube-dns.kube-system.svc.cluster.local.53 > my-deployment-5cccff8466-h4bws.43159: 42685*- 1/0/0 PTR kube-dns.kube-system.svc.cluster.local. (116)
19:36:41.526170 IP my-deployment-5cccff8466-h4bws.55320 > kube-dns.kube-system.svc.cluster.local.53: 18320+ PTR? 34.216.184.93.in-addr.arpa. (44)
19:36:41.673054 IP kube-dns.kube-system.svc.cluster.local.53 > my-deployment-5cccff8466-h4bws.55320: 18320 NXDomain 0/1/0 (138)
19:36:46.517367 ARP, Request who-has my-deployment-5cccff8466-h4bws tell ip-10-244-120-64.eu-west-2.compute.internal, length 28
19:54:30.534146 IP my-deployment-5cccff8466-x5kh9.46734 > 93.184.216.34.80: Flags [S], seq 4131307425, win 64800, options [mss 1440,sackOK,TS val 1393289043 ecr 0,nop,wscale 7], length 0
19:54:30.627479 IP my-deployment-5cccff8466-x5kh9.51298 > kube-dns.kube-system.svc.cluster.local.53: 13910+ PTR? 34.216.184.93.in-addr.arpa. (44)
19:54:30.636325 IP 93.184.216.34.80 > my-deployment-5cccff8466-x5kh9.46734: Flags [S.], seq 452881824, ack 4131307426, win 65535, options [mss 1460,wscale 2,eol], length 0
19:54:30.636388 IP my-deployment-5cccff8466-x5kh9.46734 > 93.184.216.34.80: Flags [.], ack 1, win 507, length 0

Security settings on some clusters may by default prevent tcpdump from running

Distinguished Architect at Sun Life Financial. Focused on containers & Kubernetes. Views & opinions expressed here are my own, not necessarily those of Sun Life

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Stop Words in NLP

How to install Python locally

Build an Interactive, Modern Dashboard With Dash

Using Deployment Manager to automate the creation of GCP’s Shared VPC

What I Learned at Work this Week: Friendly IFrames and Debounce

Claiming your Stargaze Airdrop with your Legder

Web Accessibility

Software Development Life Cycle (SDLC) explained

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Paul Dally

Paul Dally

Distinguished Architect at Sun Life Financial. Focused on containers & Kubernetes. Views & opinions expressed here are my own, not necessarily those of Sun Life

More from Medium

Kubernetes — Debugging NetworkPolicy (Part 2)

Let’s talk about Container Security

Kubernetes Admission Controllers Examples

Newsletter of Carlos Santana — Issue #34