eBPF for Backend Engineers — Zero-Instrumentation Observability

Introduction#

eBPF (extended Berkeley Packet Filter) runs sandboxed programs inside the Linux kernel, intercepting syscalls, network packets, and function calls without touching your application code. For backend engineers, this means observing CPU usage, network flows, and latency at microsecond resolution without restarting services or adding instrumentation libraries. This post covers Cilium, Hubble, Parca, and bpftrace for production observability.

What eBPF Is and How It Works#

eBPF programs run in a privileged kernel context but are sandboxed — the kernel verifier checks every program before loading to prevent infinite loops, invalid memory access, and kernel crashes. Unlike kernel modules, eBPF does not require recompiling the kernel, rebooting, or matching kernel versions.

eBPF programs attach to hook points: syscall entry/exit, network ingress/egress, kernel function calls (kprobes), and user-space function calls (uprobes). When the hook fires, the eBPF program runs synchronously in the kernel, collecting data into ring buffers that user-space tools read via memory-mapped regions.

Key capabilities for observability:

• Trace every syscall without any application-level instrumentation
• Capture network packets at line rate with zero copies to user space
• Profile CPU usage at function granularity across all languages and runtimes
• Measure latency at any kernel boundary — disk I/O, network, scheduler

Requirements: Linux 5.10 or later, kernel compiled with BPF support (standard on most distributions since 2021).

Cilium for Kubernetes Network Observability#

Cilium uses eBPF to replace iptables-based networking in Kubernetes. It implements network policies, service load balancing, and observability entirely in the kernel — no sidecar proxies, no userspace packet processing.

# Install Cilium with Hubble observability layer
helm repo add cilium https://helm.cilium.io
helm repo update
 
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set prometheus.enabled=true

Cilium observes:

• Pod-to-pod TCP/UDP flows with source and destination labels
• DNS queries from every pod, with resolution results
• HTTP requests at L7 with method, path, and status codes
• Network policy verdicts (allowed, denied, dropped)

# Enforce and observe L7 HTTP policy
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-ingress
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "3000"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /api/.*
  denyLog: true

Hubble for Service-to-Service Flow Visibility#

Hubble is Cilium's observability layer. It exports network flows as structured JSON, queryable in real time via CLI or the Hubble UI.

# Port-forward to Hubble relay
kubectl port-forward -n kube-system svc/hubble-relay 4245:4245
 
# Observe flows from a specific pod selector
hubble observe --pod-selector k8s:app=frontend --output json
 
# Find dropped packets with reasons
hubble observe --verdict DROPPED --last 1000 \
  --output json | jq '.flow | {src: .source, dst: .destination, reason: .drop_reason_desc}'
 
# Identify top-talking pods by connection count
hubble observe --last 5000 --output json \
  | jq -r '.flow.source.pod_name' \
  | sort | uniq -c | sort -rn | head 10

Export Hubble metrics to Prometheus for long-term storage and alerting:

apiVersion: v1
kind: ConfigMap
metadata:
  name: hubble-metrics-config
  namespace: kube-system
data:
  config.yaml: |
    metrics:
      enabled:
        - dns:query;ignoreAAAA
        - drop
        - tcp
        - flow
        - icmp
        - http

Parca for Continuous CPU and Memory Profiling#

Parca captures CPU and memory profiles from every process on every node using eBPF frame pointers — without any code changes. Deploy as a DaemonSet.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: parca-agent
  namespace: observability
spec:
  selector:
    matchLabels:
      app: parca-agent
  template:
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: parca-agent
        image: ghcr.io/parca-dev/parca-agent:latest
        securityContext:
          privileged: true
        env:
        - name: PARCA_AGENT_STORE_ADDRESS
          value: parca-server.observability:7070
        - name: PARCA_AGENT_NODE
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: debugfs
          mountPath: /sys/kernel/debug
        - name: proc
          mountPath: /host/proc
      volumes:
      - name: debugfs
        hostPath:
          path: /sys/kernel/debug
      - name: proc
        hostPath:
          path: /proc

Once deployed, access the Parca UI at localhost:7070 (after port-forwarding) to view flame graphs for any service, compare CPU usage before and after a deploy, and identify hot functions without reading any code.

bpftrace for Ad-Hoc Investigation#

bpftrace is a high-level tracing language that compiles to eBPF. Use it for point-in-time investigations without deploying new infrastructure.

# Trace all file opens by nginx
bpftrace -e 'tracepoint:syscalls:sys_enter_openat /comm == "nginx"/ {
  printf("%s opened %s\n", comm, str(args->filename));
}'
 
# Find syscalls taking longer than 10ms
bpftrace -e '
  tracepoint:raw_syscalls:sys_enter { @start[tid] = nsecs; }
  tracepoint:raw_syscalls:sys_exit /@start[tid]/ {
    $latency = (nsecs - @start[tid]) / 1000000;
    if ($latency > 10) {
      printf("Slow syscall: %d ms on pid %d\n", $latency, pid);
    }
    delete(@start[tid]);
  }
'
 
# Count memory allocations by size bucket
bpftrace -e '
  kprobe:kmalloc { @allocs = hist(args->size); }
  END { print(@allocs); }
'
 
# Monitor TCP retransmits in real time
bpftrace -e '
  kprobe:tcp_retransmit_skb {
    @retransmits[pid, comm] = count();
  }
  interval:s:5 { print(@retransmits); clear(@retransmits); }
'

BCC Tools for Common Investigations#

BCC (BPF Compiler Collection) ships with pre-built tools for the most common investigation patterns.

# See every process spawned on the system
sudo execsnoop
 
# Monitor TCP connections and byte counts
sudo tcptracer
 
# Track TCP retransmits with source/destination
sudo tcpretrans
 
# Profile CPU by function stack across all languages
sudo profile -F 99 30  # 99 Hz sampling for 30 seconds
 
# Trace slow disk I/O (above 10ms)
sudo biolatency -m 10

These tools require bpftrace and linux-headers matching your kernel version:

# Ubuntu/Debian
apt-get install -y bpftrace linux-headers-$(uname -r)

eBPF vs Sidecar Overhead#

eBPF wins on efficiency for network and system-level visibility. APM SDKs win for business transaction tracing and application-level context. The recommended production stack uses both: eBPF for network flows and profiling, APM for distributed tracing and error tracking.

Production Safety Checklist#

eBPF runs in the kernel. The verifier prevents most mistakes, but established tools are safer than custom programs.

# Verify kernel requirements
uname -r          # Need 5.10+
cat /boot/config-$(uname -r) | grep CONFIG_BPF  # Should be y=yes
 
# Check eBPF program memory usage
bpftool prog show
bpftool map show
 
# Monitor Cilium agent CPU usage
kubectl top pod -n kube-system -l k8s-app=cilium
 
# Graceful shutdown: unload all eBPF programs
kubectl delete -n kube-system daemonset cilium

Test eBPF tools in staging before production. Parca and Cilium are safe defaults — they are heavily tested by large production deployments. Custom bpftrace scripts warrant more caution on production kernel versions.

Key Takeaways#

• eBPF programs run in a kernel sandbox verified before loading — they cannot crash the kernel or access arbitrary memory.
• Cilium replaces iptables with eBPF-based networking, providing L3-L7 observability for every pod-to-pod flow in Kubernetes with no sidecars.
• Hubble exports structured network flow data to CLI, UI, and Prometheus — use it to find top talkers, dropped packets, and DNS failures.
• Parca deploys as a DaemonSet and profiles CPU and memory for every process on every node with zero code instrumentation.
• bpftrace one-liners answer specific investigation questions: slow syscalls, large memory allocations, TCP retransmits, file access patterns.
• eBPF observability adds 2-5% CPU overhead versus 10-20% for sidecar proxies and 5-15% for APM SDKs.
• Combine eBPF (network, profiling) with APM (traces, errors) for complete production observability coverage.
• Minimum kernel version for full eBPF observability features is Linux 5.10, available on all major cloud providers.