Build Maneuverings with External Linux Kernel Modules
Posted December 12, 2014 ‐ 7 min read
Much of the material relating to writing Linux kernel modules does not discuss the scenario where you would like to replace existing kernel code or a driver with a wrapping interface, or a whole new implementation of the same component, or another scenario where you have one external kernel module that depends on another. Our Linux kernel is a standalone component and it doesn't like these sort of tricks, but its build system is advanced enough to allow to implement them cleanly.
First, just a plain-old module
To demonstrate the topics I am about to discuss, let's create a simple kernel module that we can work with:
foo-objs += obj-m +=
The following .gitignore file can be used:
*.o *.mod.o *.ko .* modules.order Module.symvers
The C file can be very minimal for now:
Building it for the currently running kernel is quite simple:
$ make -C /lib/modules/`uname -r`/source M=`pwd`/foo make: Entering directory `/usr/src/kernels/3.17.3-200.fc20.x86_64' CC [M] /home/dan/module/foo/foo-main.o LD [M] /home/dan/module/foo/foo.o Building modules, stage 2. MODPOST 1 modules LD [M] /home/dan/module/foo/foo.ko make: Leaving directory `/usr/src/kernels/3.17.3-200.fc20.x86_64'
We should be able to load it via
insmod foo/foo.ko. But, since the module does not do anything, and does not register on any subsystem, it is effectively just a library. A novice reader can add call to
foo_export on module init. For now we will continue to focus on the building scriptology in this post.
Prepare for external access
We would like for another module to use our kernel code. But first, we need to export it via the standard C means, because
EXPORT_MODULE is not enough - this just tells the kernel that it is okay to link against
foo_export in module load time.
So we add this header under
We could have added a
foo.h directly under
foo, but that is not good because we would like to separate the interface from implementation. We expect other users to include the location of the header file via
-I. Let's take care of our internal user
foo-main.c too, by modifying the Makefile:
# If we being invoked from kbuild, prepend the proper include paths LINUXINCLUDE := endif foo-objs += obj-m +=
We can now proceed to also add
#include <foo/foo.h> in
LINUXINCLUDE directive in kbuild is very useful - it allows to insert new include paths before or after other paths or the kernel headers themselves. Here, we needed abstraction for our private module include paths. The module's code is now agnostic to where headers are located, which is a good preparation for any future point in time where the headers might move, and perhaps these headers can move into the kernel itself if our kernel code is really dandy and useful - who knows.
A second, dependent kernel module
foo, we have created
bar. However, in bar we insert a run-time dependency over foo:
But we would like to avoid the following error:
/home/dan/test/bar/bar-main.c:2:21: fatal error: foo/foo.h: No such file or directory #include <foo/foo.h>
Let's take the first step. We need to depend on
bar. If we would like to be completely flexible, we can support the modes where
foo arrives from the kernel itself or from another external kernel module.
FOO_INCLUDE= endif # If we being invoked from kbuild, prepend the proper include paths LINUXINCLUDE := endif bar-objs += obj-m +=
LINUXINCLUDE we can now have a working build, if we point
FOO_PATH to the correct place when building
make -C /lib/modules/`uname -r`/source M=`pwd`/bar FOO_PATH=`pwd`/foo
But is it not over yet, because we get an error from
modpost (although the build is successful):
WARNING: "foo_export" [/home/dan/test/bar/bar.ko] undefined!
The kernel keeps track of which modules are exported by which binary code. By default the program
modpost that runs during the build process performs a lookup on all undefined symbols via the kernel's own
Module.symvers located under
/lib/modules in our case.
foo is not there. Let's extend our kernel module's Makefile. Luckily we can pass
KBUILD_EXTRA_SYMBOLS to kbuild, but it is ineffectual to do so from within the Makefile itself when it is invoked by kbuild.
# If we being invoked from kbuild, prepend the proper include paths FOO_INCLUDE= endif LINUXINCLUDE := else export FOO_PATH export KBUILD_EXTRA_SYMBOLS= : bar-objs += obj-m +=
In order to not make the command line any longer, we have inserted a proxy target 'all'. Also,
FOO_PATH are forwarded to kbuild, and
KDIR receives the path of the kernel tree from the top level invocation.
$ make -C bar KDIR=/lib/modules/`uname -r`/source FOO_PATH=`pwd`/foo make: Entering directory `/home/dan/test/bar' make -C /lib/modules/3.17.3-200.fc20.x86_64/source M=/home/dan/test/bar make: Entering directory `/usr/src/kernels/3.17.3-200.fc20.x86_64' LD /home/dan/test/bar/built-in.o CC [M] /home/dan/test/bar/bar-main.o LD [M] /home/dan/test/bar/bar.o Building modules, stage 2. MODPOST 1 modules CC /home/dan/test/bar/bar.mod.o LD [M] /home/dan/test/bar/bar.ko make: Leaving directory `/usr/src/kernels/3.17.3-200.fc20.x86_64' make: Leaving directory `/home/dan/test/bar'
There are several things consider.
foo must already be built when
bar is built, otherwise
Module.symvers is missing, and you would get a warning. The other thing to keep in mind is that the Makefile is evaluated twice - once in our direct execution and a second time when kbuild evaluates it. The current directory (`pwd`) in the second evaluation is the kernel tree, and it already contains a lots of useful bits such as
LINUXINCLUDE. We should be careful about kbuild's own evaluation so that we don't accidentally override stuff that we did not intend or insert makefile targets that don't belong under the kbuild environment.
Note that the use of
LINUXINCLUDE to replace existing kernel headers should be used with care. It is powerful enough to allow re-packaging of whole stacks of code that overlap with existing kernel software stack. Some kernel subsystems are spreading their headers over paths that don't necessarily start with
linux/, for example,
uapi/linux (user space headers), and
asm/ (architecture specific headers, actually located under
arch/*/include). Ordering of paths when extending
LINUXINCLUDE is key in that case.
One should mind including the proper headers so that the API are validated in compile time and match in run-time. For example, it is possible for an external package X to replace some modular existing kernel subsystem X with modified APIs, but when any other piece of code - user-space and kernel space - tries to compile against X, it should refer to the intended headers.