The XDP(eXpress Data Path) is a powerful new network feature in Linux based on eBPF, that enables high-performance programmable access to networking packets before they enter the networking stack. But it also has a high start curve to learn, many developers have written introduction blogs for this feature. e.g. Paolo Abeni’s Using XDP in RHEL8 and Toke’s Using XDP in RHEL8.
As the XDP is a new technology and is still in fast-moving. The eBPF/XDP coding format and style are also changing. So developers are trying to make it easy to write. Some tools/frameworks are created to help develop BPF applications. e.g. libbpf, xdp-tools which we will use in this blog.
In this blog, I will introduce how to start with XDP program with 4 steps:
- Write a small program to drop everything to show what a simple XDP program looks like
- What a samll program looks like
- How to build and dump bpf object
- How to load bpf object
- How to show bpf programs
- How to unload bpf object
- Extend the program to let you know how to deal with specific packets
- Use a packet counter to show how to use BPF maps
- Add a userspace tool to show how to load the BPF program with customer tools
The reader needs to be familiar with C code and IP header structures. All examples are tested with RHEL-8.3
The article should take you about 30m.
As a preparation of the developing environment. We should install the following packages.
$ sudo dnf install clang llvm gcc libbpf libbpf-devel libxdp libxdp-devel xdp-tools bpftool kernel-headers
Task 1: XDP: Drop everything
Task 1.1: Write a Hello World program
First, let’s start with a simple XDP program in C:
linux/bpf.h is provided by the kernel-header package, which defines
all the supported bpf helpers and xdp_actions, like
XDP_DROP in this example.
bpf/bpf_helpers.h is provided by libbpf-devel, which provided some
useful eBPF macros, like
SEC in this example. The
SEC, short for section,
is used to place a fragment of the compiled object in different ELF sections,
which we could see in the later
xdp_drop_prog() we have a parameter
struct xdp_md *ctx, which is
not used yet. I will talk about it later. This function returns
means we will drop all incoming packets. There are also some other XDP actions
Finally, the last line formally specifies the license associated with this program. Some eBPF helpers are accessible only by GPL licensed programs, and the verifier will use this info to enforce such a restriction.
Task 1.2: Build and dump the bpf object
After the programming, let’s build it with clang:
$ clang -O2 -g -Wall -target bpf -c xdp_drop.c -o xdp_drop.o
-O Specify which optimization level to use,
-g could generate debug
You can use
llvm-objdump to show the ELF format after build. With
can see the sections in the object. With
-S it will display the source
interleaved with the disassembly.
The llvm-objdump is very useful if you want to know what the object does without source code.
Task 1.3: load the bpf object
Warn: do not load it on default interface, please use veth interface for testing.
After building the object, there are multiple ways to load the object.
The easiest way to load it is using
ip cmd, like
$ sudo ip link set veth1 xdpgeneric obj xdp_drop.o sec xdp_drop
ip cmd doesn’t support the BTF(BPF Type Format)
type map that we will talk later. Although the latest ip-next
has fixed this issue by adding libbpf support, it’s not backport to main line yet.
The recommand way to load XDP object on RHEL is using xdp-loader. Not only because it depends on libbpf which has full BTF support, but also that’s the only way we can get multiple programs on one interface.
To get Red Hat support for XDP feature, please use
libxdp based on document
XDP is conditionally supported in RHEL 8.3 release note
Now let’s load it the object on interface
-m sbk means to use skb mode.
There are some other modes like native, offload. As these modes are not
supported on all NIC driver, we just use skb mode in this article.
-s xdp_drop specify to use the section xdp_drop.
$ sudo xdp-loader load -m skb -s xdp_drop veth1 xdp_drop.o
Task 1.4: show the XDP programs
There are also multi ways to show the XDP program we just load:
If you load with
ip cmd, there will only 1 XDP program load. If you load with
xdp-loader, there will be 2 programs loaded by default. One is
created by xdp-loader, another is
xdp_drop_prog, write by us.
From the above result, we can see the
xdp_dispatcher with id 15 and
xdp_drop_prog with id 20.
Task 1.5: unload the XDP programs
If you use
ip cmd to load the program, you can use
$ sudo ip link set veth1 xdpgeneric off to unload the program. Note you need
to use Correspond xdp flag to unload, e.g. use
xdpgeneric off if you load it by
ip link set veth1 xdpgeneric obj ... or
xdp off if you load it by
ip link set veth1 xdp obj ...
Or we can just unload all the programs by
-a means to unload all
the programs on the interface:
$ sudo xdp-loader unload -a veth1
Task 2: XDP: Drop specific packets
The first example will drop every packet. It is just an intro example and doesn’t have any real meaning. Now let’s do some real stuff. Let’s try to drop some specific packets. Like IPv6 packets.
Compare with the first example. We added two more header files
arpa/inet.h for struct ethhdr and function htons().
struct xdp_md is also used in this example. It’s defined(in linux 5.10) like
The packet contents are between
And the data starts with ethernet header, so we assign ethhdr like
When accessing the data in
struct ethhdr, we must make sure we don’t access
invalid areas by checking
data + sizeof(struct ethhdr) > data_end.
This check is compulsory by the BPF verifer.
Then if the protocol in ether header is IPv6 with checking
h_proto == htons(ETH_P_IPV6),
we will drop it by returning
XDP_DROP. For others we just return
Task 3: MAP: Count the processed packets
In the previous example we dropped IPv6 packets, but how to know how many packets are dropped? Here we got another BPF feature, MAPS. BPF maps are used to share data between kernel and userspace. We can update the map data in kernel and read it from userspace, or vice versa.
Here is an example of new
BTF-defined map(introduced by upstream commit abd29c931459).
The map is named as
rxcnt with type
BPF_MAP_TYPE_PERCPU_ARRAY. This type
indicates that we will have one instance of the map per CPU core (if you have
4 cores, you will have 4 instances of the map). It will be used to count how
many packets are processed per core. Then we defined the key/value type and
limit the max entries to 1, as we only need to count one number(the received IPv6 packet).
In C code it would look like we defined an array on each cpu with a size of one.
unsigned int rxcnt.
BPF also supports more map types like BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_ARRAY, etc.
Then we lookup the value with function
bpf_map_lookup_elem() and update it.
Here is the full code.
Now let’s name the new program to xdp_drop_ipv6_count.c and build it to xdp_drop_ipv6_count.o.
After loading the object, then send some IPv6 packets to this interface.
bpftool map show we can see the map
rxcnt's id is 13. Then
bpftool map dump id 13 we could see there are 13 packets
are processed on cpu 0 and 7 packets are processed on cpu 1.
Task 4: Loader: load XDP objects with custom loader
We can load the XDP objects with
ip and show the map number with
But if we want more advanced features(create, read, write maps, attach xdp
programs to interfaces, etc), we need to write the loader ourselves.
Here is an example of how to show the total packets and PPS we dropped. In this example I hard code a lot of stuff like kernel object name, section name, etc. These all can be set via parameters in your own code.
The purpose of each function has commented in the code.
ulimit to unlimited and build the program with flag
then run the program and we could see the pkt count output.
With this article we know what a XDP program looks like, how to add a BPF map and how to write customer loader. If you want to learn more about XDP programming, you can reference to xdp-tutorial.
Author Hangbin Liu
LastMod 2021-07-15 (132542b)