CS380L: Advanced Operating Systems
The goal of this assignment is to be able to compile and boot the
Linux kernel on the KVM-qemu virtual machine. If you have never
built or installed a
kernel before then please start this assignment early. We will use the Ubuntu distribution of Linux.
There are three parts to this system: the KVM-qemu virtual machine, the virtual machine's disk image, and your own build of the Linux kernel. Modern versions of Linux allow you to boot another version of Linux inside a virtual machine. In order to boot Linux you need something that looks like a disk that has all of the programs and the file system that Linux needs in order to run. Finally, we want to play with the kernel, so we want to compile it ourselves.
Unfortunately, the CS machines do not support KVM, so you will need to find a machine that does. If you can't find one, let us know.
The instructions for this lab are deliberately non-explicit. One of
the main goals is giving you the opportunity to read some
documentation and figure things out for yourself, and almost
all of the steps here can be accomplished many different ways.
That said, if you're stuck on something for a long time, do ask
questions. The failure modes of many of these tools can be cryptic.
Getting a VM running in KVM
- Get a VM with a copy of the latest LTS release of Ubuntu running in KVM on your machine
- In previous years, the easiest way to
create a virtual machine has been to use VMBuilder (documentation
available online). While I encourage you to try it, it relies on
some deprecated components, so your mileage may vary.
You may find simply booting from an ISO image and
installing onto a new virtual disk is simpler. If you use an installer,
be careful of Ubuntu installer defaults, and don't let it partition
your virtual disk in ways that don't make sense for a virtual
- Regardless of your VM construction method, make sure you
include the openssh-client package if you build a VM
You should be able to get your VM running inside KVM by specifying the
location of the VM with -drive or -hda. You may want to append
"--snapshot" after the file name of your VM. This ensures that no
are made to your image during an execution so you can do something
dangerous and have the original image file preserved.
Once you've been able to get your VM running inside KVM, try installing a package using aptitude to verify your VM has network access.
Obtaining and building the kernel
Download Linux kernel release from
kernel.org. Which version you choose is up to you, but linux kernel versions
newer than 4.9 have a number of default configuration options that can complicate later steps in this lab.
The simplest option is to use version 4.4.85. For extra credit, use any kernel version starting with 4.10 or
later, but be aware you'll need to do some sleuthing in later sections of the lab.
Installing and Copying Kernel Modules
In this step you will install the kernel modules locally and then copy them to your image. Depending on how you copy your modules, it can take a while. Feel free to build a kernel that does not need modules. Just document what options you changed from default in your write up.
NOTE. For this step of the process you will need to have the guestmount package installed.
After your copy your kernel modules, depending on how you boot,
you might need to run
grub-related utility (the
bootloader) to notify it that you are booting a new kernel from this
disk. Hint: if you are using the "-kernel" option below, you shouldn't
For the curious, other methods for manipulating your virtual disk and file system
from the host include:
- You can also use nbd to mount a qcow2 image, though I have found it terribly slow.
- You can convert the qcow2 to a raw disk image
convert -O raw test.qcow2 test.raw, and then do a loopback
mount -o loop,offset=32256 /path/to/image.img
/mnt/mountpoint. Note, your offset might be different from
mine (maybe fdisk can help you), but loopback mounts are fast.
Here is a good reference on qemu images.
Booting KVM with your new Kernel
Now that you have compiled the kernel and copied the new kernel modules to your VM, you should be able to boot your VM with the new kernel.
- You will need to add -kernel "<kbuild>/arch/x86/boot/bzImage" and -append "root=/dev/sda1 console=ttyS0,115200n8" to the parameters you pass to KVM. (The root parameter specifies where the root partition of the hard disk is and the console parameter adds a serial console at boot so you can see boot messages.)
- You can also specify -nographic option, and use your current console to log in to the VM, but you need to create file /etc/init/ttyS0.conf in the VM image, which contains
If you have trouble booting your VM, you might want to explore some of the other command line options to KVM such as -serial or -gdb.
Once the VM has booted completely, you should be able to login and try installing another package from aptitude.
# ttyS0 - getty
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
start on stopped rc or RUNLEVEL=
stop on runlevel [!2345]
exec /sbin/getty -L 115200 ttyS0 vt102
Please complete a short writeup that you will print out and bring to class.
Identify completely your hardware and software.
Explain any changes you made to the kernel build process and why.
Report your qemu command line.
Booting, kernel modules, and discovering devices
Boot the Kernel, and run
dmesg to capture the kernel output from the
boot. In your writeup, report the elapsed wall clock time for your boot, and compare that with the time reported by the kernel. Are they the same? Why or why not?
For each device, quote the parts of the dmesg
output that show the kernel discovering that device, and then report
what kind of device it is, and how you made your determination. You may also use
lspci.Please only quote a maximum of 3 lines.
Tracing the kernel
- Compile the following code, transfer the program to your simulated disk and run the program.
Attach gdb to the kernel. You will find that you need to compile your kernel with debug symbols, and depending on your
kernel version, support for kvm, and proper CONFIG_DEBUG* options. See these instructions.
Set a conditional breakpoint in spin_lock in kernel code that will only stop execution if the above process is running. (Note: spin_lock might be an inline function or macro; please check the kernel source of your version to find out the actual symbol name where you need to set the breakpoint.)
int fd = open("/dev/urandom", O_RDONLY);
read(fd, &data, 4096);
fd = open("/dev/null", O_WRONLY);
write(fd, &data, 4096);
Find three instances of spin_lock being called from three substantially different contexts in the kernel, where you define substantially different. Report exactly how you set the breakpoints, in what functions, how you made sure your process was running, etc. For each of the three calls to spin_lock that you choose, put the minimum number of lines of the kernel backtrace into your report and then summarize what is going on in the kernel in AT MOST two sentences.
In your write up, briefly explain the difference between /dev/random and /dev/urandom.
- To find the PID of the currently executing process from GDB, you will need to inspect (condition on) the value stored in the task_struct of the current process.
- To find out the address of the task_struct of the current process, you can use the thread_info struct of the process, which has a task field pointing to task_struct.
- The thread_info struct is located at the bottom of the process' kernel stack, so you can use register $rsp and mask off the lower bits. You need to figure out the size of the kernel stack by looking at arch/x86/include/asm/page_64_types.h or arch/x86/include/asm/page_32_types.h.
Building a KVM+GDB-friendly kernel
- Backup your .config in kbuild dir.
- In linux kernel source, arch/x86/Kconfig: change the DEBUG_RODATA from y to n. (https://www.kernel.org/pub/linux/kernel/people/jwessel/kdb/CompilingAKernel.html)
- cd into kbuild
- make -C kernel-src O=$(pwd) x86_64_defconfig
- make -C kernel-src O=$(pwd) kvmconfig
- make -C kernel-src O=$(pwd) menuconfig
- The "menuconfig" target gives you a (ncurses-based) GUI for managing kernel config options. You care about the following:
- Kernel hacking / Compile-time checks and compiler options / Compile the kernel with debug info (hit space)
- In the sub-option of Compile the kernel with debug info
- Generate dwarf4 debuginfo (hit space)
- Provide GDB scripts for kernel debugging (hit space)
- KGDB: kernel debugger (hit space)
- General setup / Configure standard kernel features (expert users) (hit space and hit enter) / Include all symbols in kallsyms (hit space)
- If you see something like curses.h not found, install libncurses5-dev.
Please think of your report like the papers we are reading for the
class. Organize the information that you gathered and present it
logically. For example, early in your report specify your
experimental platform, including the hardware, the OS, whether you are
running qemu or a hypervisor, etc. Include anything relevent to
understand the results, but keep your description concise. Of course
balancing completeness and concision is difficult, but that balance is
generally well accomplished by the (modern) papers we read.
Some people use
virsh to manage their VM. This can be sort of useful, but it obscures exactly what is going on, for example with your exact command line to qemu. I would suggest not using it.
Please report how much time you spent on the lab.