What eBPF Actually Sees, and What It Costs You

eBPF programs run in the kernel, attached to hook points — kprobes, tracepoints, LSM hooks, socket filters, XDP — and are checked by a verifier that rejects anything with unbounded loops, unchecked pointer arithmetic, or the potential to run too long. They cannot crash the kernel, which is the property that makes the whole thing viable. They emit data to userspace through ring buffers and maps.

For security monitoring, this means you can observe every process execution, every file open, every network connection, every privilege change, across every container on a node, without modifying a single application or injecting anything into a container image. That is a genuinely different capability from what came before, and the enthusiasm around it is largely justified. The part that is oversold is the overhead, and specifically the way overhead is quoted as a single number when it varies by two orders of magnitude depending on what you subscribe to.

What the syscall boundary actually reveals

Every meaningful action a process takes crosses into the kernel. Reading a file, opening a socket, spawning a child, loading a module, changing UID. A container is a process tree with namespaces and cgroups; there is no action it can take that does not appear at this boundary. So the visibility claim is real and the coverage is genuinely complete in a way that log-based or agent-based approaches are not.

What you get, concretely:

• Process execution with full lineage. Not just that curl ran, but that it was spawned by sh, spawned by the JVM, in this pod, on this node, with these arguments and this working directory. The ancestry is what turns a benign-looking exec into a detection.
• File access with resolved paths. Reads of /etc/shadow, writes to /usr/bin, access to service account tokens at /var/run/secrets/kubernetes.io/.
• Network flows with process attribution. Which binary in which pod opened a connection to which destination. Conventional network monitoring sees the flow; eBPF sees the flow and the process that caused it.
• Privilege transitions. setuid, capability changes, namespace escapes, writes to /proc/self/exe — the runc container-escape class.
• Enforcement, not just observation. Via LSM BPF hooks, a program can return -EPERM and block the operation rather than reporting it afterwards.

What it does not see is equally important. TLS payloads are encrypted before the syscall — you see a write() of ciphertext unless you additionally attach uprobes to the TLS library, which is possible, version-fragile, and considerably more expensive. Anything happening entirely in userspace — a JIT-compiled expression, an interpreted script that never forks, in-process memory manipulation — produces no syscalls and is invisible. And in a managed control plane, you cannot instrument the API server nodes at all, so kernel-level visibility is a data-plane story only.

The overhead question, honestly

The marketing number is "less than 1 percent." That is achievable and it is also not a property of eBPF — it is a property of a specific, narrow event subscription. The actual cost is roughly proportional to event rate multiplied by per-event work.

These are ranges from clusters I have measured, not benchmarks, and the variance within each row is large. The structural point stands regardless: the difference between a well-scoped and a default configuration is roughly an order of magnitude, and defaults in commercial eBPF security products tend toward the comprehensive end because comprehensive demos better.

Three things drive the cost, and all three are controllable:

Filter in the kernel, not in userspace. An eBPF program that emits every openat() to userspace for filtering there is paying a ring buffer write and a context switch per event. A program that checks the path prefix in-kernel and emits only matches pays almost nothing for the events it drops. Any tool that does not let you push filters into the BPF program is going to be expensive.

Prefer ring buffers to perf buffers. BPF ring buffers (kernel 5.8+) are shared across CPUs with proper ordering and better memory efficiency than the older per-CPU perf buffers. Most modern tooling uses them; if you are on something older, this alone is a meaningful difference.

Watch the userspace agent, not just the BPF program. The kernel-side cost is often the smaller half. The agent that enriches events with Kubernetes metadata, evaluates rules, and ships to a backend is a normal process doing normal work, and on a node generating 50k events per second it will be the thing consuming a core. Measure it separately.

# kernel-side cost per program, if BPF stats are enabled
sysctl -w kernel.bpf_stats_enabled=1
bpftool prog show | grep -A2 run_time_ns
# run_time_ns / run_cnt = average nanoseconds per invocation
# above ~2000ns on a hot hook, investigate

# userspace side
kubectl top pods -n falco --containers

Kernel version determines what you can do

This is the constraint that derails eBPF rollouts, usually after tool selection.

Two traps. CO-RE (Compile Once, Run Everywhere) requires the kernel to be built with BTF debug info — CONFIG_DEBUG_INFO_BTF=y. Amazon Linux 2023, recent Ubuntu, Bottlerocket, and COS all ship it. Older Amazon Linux 2 kernels and some hardened enterprise distributions do not, and without BTF your tool falls back to compiling a module against kernel headers on each node, which is slow, fragile, and occasionally impossible.

Enforcement via LSM BPF additionally needs bpf in the kernel's LSM list, which is a boot parameter. On a managed node group that means a custom launch template and a node replacement. Verify before promising anyone blocking capability:

ls /sys/kernel/btf/vmlinux              # BTF present?
cat /sys/kernel/security/lsm            # is 'bpf' in the list?
uname -r

Detection quality is a rules problem, not a kernel problem

The kernel gives you a firehose of true facts. Turning that into alerts someone will act on is the actual work, and it is where most deployments stall — typically at a state where the tool is installed, generating several hundred alerts a day, and everyone has muted the channel.

Two things make the difference.

Start with a small number of high-precision rules. Not the vendor's default ruleset. Six to ten detections with near-zero false positives, each of which would genuinely warrant investigation: a shell spawned inside a container that has no business having one, a write to a service account token path by anything other than the kubelet, an outbound connection from a pod to an IP outside your egress allowlist, a process executing from /tmp or /dev/shm, a container starting with CAP_SYS_ADMIN that was not on the approved list.

Scope by workload identity, not by node. The same syscall is benign in a CI runner and alarming in a payments pod. Rules that do not incorporate the Kubernetes context produce noise proportional to the diversity of your cluster.

- rule: Shell in production service container
  desc: Interactive shell spawned in a pod not permitted one
  condition: >
    spawned_process
    and container
    and shell_procs
    and k8s.ns.name in (payments, checkout, identity)
    and not k8s.pod.label[security.acme/shell-allowed] = "true"
  output: >
    Shell in prod container
    (pod=%k8s.pod.name ns=%k8s.ns.name cmd=%proc.cmdline
     parent=%proc.pname user=%user.name image=%container.image.repository)
  priority: CRITICAL
  tags: [container, shell, mitre_execution]

Every rule needs a documented response. A CRITICAL alert with no runbook trains people to ignore CRITICAL alerts, which is worse than not having the rule.

Where it sits relative to everything else

eBPF runtime monitoring is a detection and response control. It tells you what happened and, with LSM hooks, can stop some of it. It is complementary to — not a substitute for — the preventive controls that decide what is allowed to run in the first place. A cluster that verifies image provenance at admission has eliminated a whole category of things eBPF would otherwise have to detect, and the combination is much stronger than either half.

On the observability side, there is a real efficiency argument. eBPF aggregates in the kernel, which means network flow metrics, latency histograms, and connection maps can be produced without emitting a log line per event. For the specific job of service-to-service visibility, this generates dramatically less data than application-level logging for equivalent insight, and it is one of the few ways to reduce telemetry spend without reducing what you can see.

It also overlaps meaningfully with what a sidecar mesh provides. Cilium's kernel-level identity-based policy and Istio's ambient mode both push L4 enforcement into shared infrastructure rather than per-pod proxies, and the cost profile of that choice is worth working out before committing to either.

A realistic starting position: process execution and network connection events only, kernel-side filtering on, a curated ten-rule set, measured overhead under 3 percent, alerts routed to a team that has runbooks. Expand from there based on what you failed to detect, not based on what the product can do.