Background

Recently, I am preparing to study the PVM solution proposed by the Linux kernel expert Lai Jiangshan. After a brief review of the paper and the patch, I found that it needs a deep understanding of paravirtualization to understand it. When I entered the field of virtualization, KVM had already dominated the virtualization field, so I did not study the implementation of paravirtualization solutions like lguest/xen from a code perspective at that time. In order to learn PVM, I must gain a more thorough understanding of lguest and xen.

lguest is the simplest paravirtualization solution, which is very suitable for learning. It was integrated into the Linux kernel in version 2.6.23 and removed in version 4.14. With the spirit of “true engineers get their hands dirty,” I am ready to run lguest right away. Of course, it is not surprising that, following the documentation, I began to set up the environment and then encountered failures, which is typical for open-source projects. This article records the problems I encountered and how I resolved them. I hope it can provide some help to the people in the field of virtualization.

The issue

I create a VirtualBox VM and install an Ubuntu 16.04 OS in it. I choose Ubuntu 16.04 because it uses 4.4 kernel version and is a LTS version. In order to run lguest, we need prepare following:

• Prepare the host kernel(the lg module) and the guest kernel
• The initrd file and the rootfs file
• build lguest userspace tool(like qemu)

Build kernel

Then I download Linux kernel 4.4 source code as 4.4 is the Ubuntu 16.04 shiped with 4.4 kernel. Following the instruction. I build the same kernel as guest and host. Some of the configuration:

            ## CONFIG_EXPERIMENTAL=y // no available
            CONFIG_PARAVIRT=y
            CONFIG_LGUEST_GUEST=y
            CONFIG_HIGHMEM64G=n
            CONFIG_PHYSICAL_ALIGN=0x100000
            CONFIG_VIRTIO_BLK=m
            CONFIG_VIRTIO_NET=m
            CONFIG_TUN=m
            CONFIG_LGUEST=m

Download initrd and rootfs file

I download initrd from here I download rootfs from here

Run it

In the Linux kerne soruce tree tools/lguest, type ‘make’ to build lguest userspace tool. Using following command to run lguest.

            modprobe lg
            ./lguest 64m /home/test/linux-4.4/vmlinux --tunnet=192.168.19.1 --initrd=/home/test/lguest/initrd-1.1-i386.img --block=/home/test/lguest/CentOS6.x-x86-root_fs  root=/etc/vda

As we can see we get an error “lguest: Reinjecting trap 13 for fault at 0x1afaeaa: Invalid argument”.

The solution

First issue: general protection fault

After reading the code, I know this is the gpf error casued by the guest. When dispatched to lguest, it can’t emulate so report this error. Let’s first print the instruction. Add following printf to lguest.c file.

            default:
                    /* OK, we don't know what this is, can't emulate. */
                    printf("can't emulate:%x %x %x\n", insn[0], insn[1], insn[2]);
                    goto no_emulate;
            }

Let’s disassemble vmlinux binary and find where this instruction comes.

            root@test-VirtualBox:~/lguest# objdump  -S /home/test/linux-4.4/vmlinux > vmlinux1.S
            root@test-VirtualBox:~/lguest# cat vmlinux1.S  | grep "65 a1 14"  | grep afaeaa
            c1afaeaa:	65 a1 14 00 00 00    	mov    %gs:0x14,%eax

It’s the prologue of function ‘load_ucode_intel_bsp’. The fault instruction is ‘move %gs:0x14,%eax’.

After some investigation, I know this instruction is introduced about ‘stack protector’. So I just build another kernel without stack canary.

            make "KCFLAGS=-fno-stack-protector" -j6

After building, let’s try.

Second issue: rdmsr

Another issue. Let’s just go to 0x1034c25 to see what it is.

It’s ‘rdmsr’ has 2 lenght instruction. Let’s just ignore this instruction. Add following code to lguest.c.

            if (insn[insnlen] == 0x0f) {
                    insnlen = 2;
                    printf("ignore readmsr\n");
                    goto skip_insn;
            }

After this patch, we finally run lguest successfully.

The rdmsr instruction is also called from ‘load_ucode_bsp’ call chain.

Analysis

There are two questions I don’t understand currently.

• why ‘move %gs:0x14,%eax’ instruction cause gpf
• why the guest uses native_rdmsr instead of pv rdmsr as ‘rdmsr’ is privileged instruction

For the first issue I read the SDM and found the clue at Volume 2 Chapter 4.3 Instruction(M-U). At the MOV-Move part:

And in the function ‘lguest_arch_setup_regs’: Only the ‘cs/ds/es/ss’ is initialized and the ‘gs’ is not initialized.

For the second issue after look at the ‘load_ucode_bsp’ I know the reason.

Here ‘call load_ucode_bsp’ is called by the kernel entrypoint(startup_32). And this function is called before the lguest initialization(lguest_init). When ‘load_ucode_bsp’ is called, the gs segment is not initialized and then cause a gpf. Aslo this function call invoke the ‘native_rdmsr’ directly and which execute the instruction ‘rdmsr’ and causes the second issue.

We can notice we can eliminate this function by set CONFIG_MICROCODE=n. I have tried this, it can work without modifying any lguest code.