What is EVL
EVL is the latest version of the Xenomai project also called Xenomai 4. Linux is a General Purpose Operating System (GPOS) and as a result makes no guarantees about meeting real-time requirements. The EVL project attempts to convert the Linux kernel to a dual kernel via a technique called interrupt pipelining which allows a real-time core (EVL Core) to reside alongside the Linux kernel and guarantee real-time performance. The EVL project does this while making the user-space application still remain a normal thread for ease of us. Of course the user-space application must limit its interactions to the EVL Core (via
libevl) while expecting to maintain real-time guarantees.
The EVL project overview provides a great introduction to the project.
There are two components to the EVL project:
- a modified Linux kernel that has the Dovetail interface as well as the EVL Core
- a library called
libevlthat allows user-space interactions with the EVL Core
The EVL project instructions for building EVL suggest downloading the EVL patched kernel and building it. This is not a very simple task and can be further complicated if we want to cross-compile the kernel and boot into a useful user-space. There are a few challenges:
- build a cross-compilation toolchain which is a complicated task
- optionally create a device tree binary depending on the system we are compiling for
- construct a root filesystem (rootfs) so we have a useful system
Let’s acquire the latest version of
Buildroot which is
2022.08 at the time of writing:
ekohandel@ekohandel-server:~/sandbox$ mkdir evl ekohandel@ekohandel-server:~/sandbox$ cd evl ekohandel@ekohandel-server:~/sandbox/evl$ git clone -b 2022.08 https://git.buildroot.net/buildroot Cloning into 'buildroot'... remote: Enumerating objects: 34346, done. remote: Counting objects: 100% (34346/34346), done. remote: Compressing objects: 100% (16518/16518), done. remote: Total 478912 (delta 20181), reused 30438 (delta 17725), pack-reused 444566 Receiving objects: 100% (478912/478912), 104.74 MiB | 31.31 MiB/s, done. Resolving deltas: 100% (332909/332909), done.
We will also need the latest version of
libevl which is
r38 at the time of writing:
ekohandel@ekohandel-server:~/sandbox/evl$ git clone -b r38 https://git.xenomai.org/xenomai4/libevl.git Cloning into 'libevl'... warning: redirecting to https://source.denx.de/Xenomai/xenomai4/libevl.git/ remote: Enumerating objects: 3041, done. remote: Counting objects: 100% (23/23), done. remote: Compressing objects: 100% (23/23), done. remote: Total 3041 (delta 6), reused 0 (delta 0), pack-reused 3018 Receiving objects: 100% (3041/3041), 550.41 KiB | 1.47 MiB/s, done. Resolving deltas: 100% (2126/2126), done.git clone -b r38 https://git.xenomai.org/xenomai4/libevl.git
Buildroot comes with a predefined configuration for the QEMU AArch64 virt machine we are interested in so we can use that as a basis for our configuration. You can see all the predefined
Buildroot configurations in the
In order to create a working configuration based on the default configuration run:
ekohandel@ekohandel-server:~/sandbox/evl/buildroot$ make BR2_DEFCONFIG=$(pwd)/configs/qemu_aarch64_virt_defconfig defconfig
or the shorthand equivalent:
ekohandel@ekohandel-server:~/sandbox/evl/buildroot$ make qemu_aarch64_virt_defconfig
You should see a message like below appear at the end of the output indicating that a
.config file is written to the
Buildroot base directory:
# # configuration written to /home/ekohandel/sandbox/evl/buildroot/.config #
We have created a configuration for
Buildroot to build a kernel for the QEMU AArch64 virt machine. It is whatever default kernel
Buildroot decided to use and not our EVL patched kernel that we want. So we need to modify the
Buildroot configuration to use the EVL patched kernel.
We need to make some other modifications to the
Buildroot configuration as well, so we will do these all at the same time:
- point to
v5.15.y-evl-rebaseas the kernel
glibcinstead of the default
uClibc-ngas the C library
<sys/auxv.h>for compilation and
uClibc-ngdoes not provide support for this
- enable C++ support for the toolchain
libevlneeds a C++ compiler so we need to enable C++ support for our toolchain
- include the
libevlartifacts into the root filesystem
Buildrootallows for including root filesystem overlay directories which are copied into the final root filesystem created
We have configured
Buildroot appropriately now but we are still using a vanilla Linux kernel configuration. We need to modify that configuration to enable the EVL Core.
Just like the
Buildroot configuration we need to do a few changes to the Linux kernel configuration so we will do them all at the same time:
- Enable EVL real-time core
- Note that this automatically enabled the Dovetail feature as well
- Enable EVL quota-base scheduling and temporal partitioning policy
- Not necessary to do this but some EVL checks will fail if we don’t
- Enable EVL debug support
.configsupport and access to the configuration via
- EVL can check the kernel configuration for us to ensure no known issues are detected
The following is recorded after running
menu linux-nconfig for the first time.
Buildroot does a lot of work the first time you run this command and I have skipped over that as it is not very valuable for the demonstration purposes.
What is important to mention is that
Buildroot builds the toolchain prior to launching the Linux kernel configuration menu. If you go back after this step and make any changes to the toolchain configuration in
Buildroot you will have to do a full rebuild. Note the this will blow away your Linux kernel configuration and you will have to repeat this step.
Building and Staging
Before we build the kernel and create the root filesystem, we need to build the
libevl library so it can be included in the root filesystem:
Let’s create a build directory to work in:
ekohandel@ekohandel-server:~/sandbox/evl/libevl$ mkdir build ekohandel@ekohandel-server:~/sandbox/evl/libevl$ cd build
meson as a build system and to cross-compile the library for our target, we need to provide it a
Buildroot has already built the cross-compile toolchain for us and placed it inside of the
buildroot/output/host/bin directory. There are sample
cross-files in the
libevl/meson directory and luckily the
aarch64-none-linux-gnu file is almost exactly what we need. We need to modify the
vendor part of the target triples from
buildroot which can be done by
ekohandel@ekohandel-server:~/sandbox/evl/libevl/build$ sed 's/-none-/-buildroot-/g' ../meson/aarch64-none-linux-gnu > aarch64-buildroot-linux-gnu
Now we are ready to cross-compile
libevl and install it to our staging
sysroot directory. This is the same directory we already told
Buildroot to find a filesystem overlay in. This means when
Buildroot creates the root filesystem it will look here and include these files in the root filesystem it creates. We need to also make it so
meson can find our cross-compile toolchain and we can do that by augmenting the
PATH environment variable when we call
Setup the project:
ekohandel@ekohandel-server:~/sandbox/evl/libevl/build$ PATH="$(pwd)/../../buildroot/output/host/bin:$PATH" meson setup --cross-file ./aarch64-buildroot-linux-gnu -Dbuildtype=debug -Dprefix=/usr -Duapi=$(pwd)/../../buildroot/dl/linux/git . .. The Meson build system Version: 0.61.2 -------------------------------------->8-------------------------------------- Build targets in project: 100 libevl 0.38 User defined options Cross files: ./aarch64-buildroot-linux-gnu buildtype : debug prefix : /usr uapi : /home/ekohandel/sandbox/evl/libevl/build/../../buildroot/dl/linux/git Found ninja-1.10.1 at /usr/bin/ninja Cleaning... 0 files.
ekohandel@ekohandel-server:~/sandbox/evl/libevl/build$ PATH="$(pwd)/../../buildroot/output/host/bin:$PATH" meson compile [227/227] Linking target tidbits/oob-net-icmp
ekohandel@ekohandel-server:~/sandbox/evl/libevl/build$ DESTDIR=$(pwd)/sysroot meson install ninja: Entering directory `/home/ekohandel/sandbox/evl/libevl/build' [2/19] Generating eshi/git_stamp.h with a custom command Installing lib/libevl.so.1.1.0 to /home/ekohandel/sandbox/evl/libevl/build/sysroot/usr/lib Installing symlink pointing to libevl.so.1.1.0 to /home/ekohandel/sandbox/evl/libevl/build/sysroot/usr/lib/libevl.so.1 -------------------------------------->8-------------------------------------- Installing /home/ekohandel/sandbox/evl/libevl/utils/trace.timer to /home/ekohandel/sandbox/evl/libevl/build/sysroot/usr/libexec/evl Installing /home/ekohandel/sandbox/evl/libevl/utils/kconf-checklist.evl to /home/ekohandel/sandbox/evl/libevl/build/sysroot/usr/libexec/evl Running custom install script '/bin/sh /home/ekohandel/sandbox/evl/libevl/meson/post-install.sh' Running custom install script '/bin/sh /home/ekohandel/sandbox/evl/libevl/meson/copy-uapi.sh /home/ekohandel/sandbox/evl/libevl/build/include/uapi /usr/include'
Building the final image
We are finally ready to build the Linux kernel and create the root filesystem to boot into:
ekohandel@ekohandel-server:~/sandbox/evl/buildroot$ make -------------------------------------->8-------------------------------------- Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): done Copying files into the device: done Writing superblocks and filesystem accounting information: done ln -sf rootfs.ext2 /home/ekohandel/sandbox/evl/buildroot/output/images/rootfs.ext4 ln -snf /home/ekohandel/sandbox/evl/buildroot/output/host/aarch64-buildroot-linux-gnu/sysroot /home/ekohandel/sandbox/evl/buildroot/output/staging >>> Executing post-image script board/qemu/post-image.sh
If you have gotten this far, it means we have built a Linux kernel and a root filesystem. You can find both of them here:
ekohandel@ekohandel-server:~/sandbox/evl/buildroot/output/images$ ls -l total 23744 -rw-r--r-- 1 ekohandel ekohandel 11037184 Sep 12 00:10 Image -rw-r--r-- 1 ekohandel ekohandel 62914560 Sep 12 00:10 rootfs.ext2 lrwxrwxrwx 1 ekohandel ekohandel 11 Sep 12 00:10 rootfs.ext4 -> rootfs.ext2 -rwxr-xr-x 1 ekohandel ekohandel 515 Sep 12 00:10 start-qemu.sh
There is also a
start-qemu.sh helper script generated that simplifies launching QEMU with the built Linux kernel and root filesystem. We do not want a GUI so we call the script with the
ekohandel@ekohandel-server:~/sandbox/evl/buildroot/output/images$ ./start-qemu.sh serial-only Booting Linux on physical CPU 0x0000000000 [0x410fd034] Linux version 5.15.64 (ekohandel@ekohandel-server) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2022.08) 11.3.0, GNU ld (GNU Binutils) 2.37) #1 SMP IRQPIPE Mon Sep 12 00:07:46 UTC 2022 Machine model: linux,dummy-virt -------------------------------------->8-------------------------------------- EVL: core started [DEBUG] -------------------------------------->8-------------------------------------- Welcome to Buildroot buildroot login: root #
You can see in the kernel boot log above that the EVL core has started:
EVL: core started [DEBUG].
Testing EVL Linux
We can run the EVL tests to ensure everything is configured correctly:
# evl test basic-xbuf: OK clock-timer-periodic: OK clone-fork-exec: OK detach-self: OK duplicate-element: OK element-visibility: OK fault: OK fpu-preload: OK fpu-stress: OK heap-torture: OK mapfd: OK monitor-deadlock: OK monitor-event: OK monitor-flags: OK monitor-pi: OK monitor-pi-deadlock: OK monitor-pp-dynamic: OK monitor-pp-lower: OK monitor-pp-nested: OK monitor-pp-pi: OK monitor-pp-raise: OK monitor-pp-tryenter: OK monitor-pp-weak: OK monitor-steal: OK monitor-trylock: OK monitor-wait-multiple: OK observable-hm: OK observable-inband: OK observable-onchange: OK observable-oob: OK observable-race: OK observable-thread: OK observable-unicast: OK poll-close: OK poll-flags: OK poll-many: OK poll-multiple: OK poll-nested: OK poll-observable-inband: OK poll-observable-oob: OK poll-sem: OK poll-xbuf: OK proxy-echo: OK proxy-eventfd: OK proxy-pipe: OK proxy-poll: OK ring-spray: OK ** sched-quota-accuracy: BROKEN
We do have one failure but that is ok, afterall we are running on a virtual system, we can worry about this when we try to run this on a real system.
We can also run the EVL check utility which verifies the Linux kernel is indeed configured correctly:
# evl check # echo $? 0
The utility is awfully quiet so we can make sure it did indeed exit successfully
Build EVL for a Beagle Bone Black and do some latency measurements!