Linux Livepatching

### Livepatching

The Good, The Bad, and The Ugly

---

#### What does it look like today?

Let's take a look at Canonical's offering: Livepatch service

----

#### Getting started

- Need to sign up for Ubuntu Pro (and attach your key to the machine using `sudo pro attach`)
- Run the command `sudo pro enable livepatch`

----

#### Caveats

- Only [certain kernels](https://ubuntu.com/security/livepatch/docs/livepatch/reference/kernels) are supported

---

#### What is a kernel module

- Code that can be plugged into a running system
- Usually written for drivers (unless compiled directly into the kernel) 
    - `uvcvideo` $\rightarrow$ Webcam drivers
- Runs in kernel mode (obviously)

----

#### How are live patches different?

- Patches are just a special kernel modules that are able to dynamically replace calls to functions
- Has additional consistency guarantees

---

### Origins

- First ideas of live patching came ~2008-ish with KSplice
    - Developed within MIT as a master's thesis
    - Spun off into KSplice Inc.

----

- Acquired by Oracle
    - Closed source in 2011 
- Offered for free on Ubuntu desktop around 2015 
- Currently available for RHEL and Oracle Linux

---

### Red Hat

- Released in 2014, Red Hat created kPatch under GPL2
- Submitted for inclusion in the kernel in 2015

----

![](ritlug_w7_livepatching/kpatch_consistency.png)

Notes:
- Stops all tasks
- examines all stack traces, if no refs found it is applied
- if found, tasks are restarted, wait for some time, and try again

---

### SUSE

- Released in 2014, SUSE created kGraft
    - Kernel part GPL2
    - Userspace part GPL3

----

![](ritlug_w7_livepatching/kgraft_consistency.png)

Notes:
- Complex. Implements a "universe view". Patches are applied as soon as possible
- Calls are evaluated to see which universe they are from, routing calls to the old/new version as required.
    - trampoline
- Once it is determined it has applied, trampoline is removed and call is done directly

---

### Linux 4.0

- A common livepatching core was implemented in Linux 4.0
    - Donated by both Red Hat and SUSE engineers
    - Only had the basic stuff (injection into functions pre-execution)

Notes:
- Both companies "secret sauce" was the consistency models and patches themselves

----

### Consistency Model

- Uses a combination of kpatch and kGraft
    - Uses per-task consistency
    - Uses kpatch's stack trace switching model

Notes:
- Added 5.12
- As per kpatch, all sleeping tasks stacks are checked, if no refs found, task is patched
- Running tasks use trampoline model
- Long tasks (IO) need a SIGSTOP and SIGCONT to switch fully

---

#### What is the purpose of this?

- Uptime, mainly 
    - "New CVE has been released but we cannot reboot this machine!"

----

#### What are the alternatives?

- Regular reboot
    - Slow, requires downtime 
- Kexec 
    - Faster, still requires downtime 
    - Can leave the system in a weird state

Notes:
- show kexec demo

----

# Demo

Notes:
- ./watch_patch.sh

----

#### What just happened?

- The `cmdline_proc_show` function was dynamically replaced with our version
- ...But how?

----

#### Ftrace

- Facility in the kernel that allows developers to trace kernel function calls
- Used by Kprobes 
    - Utility that allows developers to add/remove code in existing functions

Notes:
Requires compiler support to add this in

----

- Adds `nop`'s to the beginning of every (supported) function call
    - These nops usually do nothing, but can be hooked into

Notes:
- The stack is still clean when the hook occurs, so call integrity is preserved
- /sys/kernel/tracing
- echo {nop,function,function_graph} > trace
- echo ext4_* > set_ftrace_filter
- less +F trace

----

![](ritlug_w7_livepatching/function_trace.png)

Notes:
The nop is there as the placeholder, but ftrace allows the kernel to replace that instruction with a call instruction

----

![](ritlug_w7_livepatching/mcount_showcase.jpg)

----

#### What does a patch look like?

```c
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/livepatch.h>
#include <linux/seq_file.h>

static int livepatch_cmdline_proc_show(struct seq_file *m, void *v)
{
	seq_printf(m, "%s\n", "hello RITLUG!");
	return 0;
}

static struct klp_func funcs[] = {
	{
		.old_name = "cmdline_proc_show",
		.new_func = livepatch_cmdline_proc_show,
	}, { }
};

static struct klp_object objs[] = {
	{
		/* name being NULL means vmlinux */
		.funcs = funcs,
	}, { }
};

static struct klp_patch patch = {
	.mod = THIS_MODULE,
	.objs = objs,
};

static int livepatch_init(void)
{
	return klp_enable_patch(&patch);
}

static void livepatch_exit(void)
{
}

module_init(livepatch_init);
module_exit(livepatch_exit);
MODULE_LICENSE("GPL");
MODULE_INFO(livepatch, "Y");
```

----

#### Replacements and additions

- Patches can be stacked on top of one another
    - Not recommended
- Patches can also be a cumulative replacement 
    - `.replace` property in module

Notes:
Show module replacement

---

### Going further

- This is difficult
    - The function we patched before is remarkably simple. Short, no data changes, no internal calls.

----

### /proc/meminfo

- Patches are usually built with modified function only
- This handler contains multiple calls to internal (not exported) functions 
- When building, the linker doesn't know where these exist!

----

- Solution?
    - Use the kernel's module loader infrastructure 
    - Apply relocations to the module ELF so functions can be linked at runtime

Notes:
- This is all as of 5.8, before then this was all done by livepatches themselves
- This made everything more arch agnostic

----

```
$ readelf --relocs livepatch-meminfo-string.ko
...
Relocation section '.klp.rela.vmlinux..text.meminfo_proc_show' at offset 0xac308 contains 52 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000003f  005400000004 R_X86_64_PLT32    0000000000000000 .klp.sym.vmlinux.si_sw - 4
000000000046  005500000002 R_X86_64_PC32     0000000000000000 .klp.sym.vmlinux.vm_co + 4
...
```

----

- This is why tools such as [kpatch](https://github.com/dynup/kpatch) exist
- Given a patch file and kernel source tree: 
    - Build original kernel 
    - Apply patch 
    - Build patched object files 
    - Run diffs on the ELF objects generated to find address of relocation 
    - Generate patch with generated relocation offsets

Notes:
- This is the part that's still the most unclear to me
- Relocations allow local function calls referenced in NEW function to be located in original vmlinux
- PLT -> position relative addressing
    - offset in module = 3f, 000000 placeholder, resolved symbol name - 4
- PC -> instruction pointer offset calc
    = offset in mod = 46

----

#### Data changes

- Not necessarily recommended to livepatch at this point, but possible
    - Kernel **Shadow Variables**
    - Allows you to store *new* data without modifying kernel structures, which is unsafe/impossible.

----

```
#define KPATCH_SHADOW_NEWPID 0

...
newpid = klp_shadow_get_or_alloc(p, KPATCH_SHADOW_NEWPID,
        sizeof(*newpid), GFP_KERNEL,
        NULL, NULL);
...
```

Notes:
- Allocation is global (added to global hashtable), IDs must be unique
- Returns pointer that canbe written to/dereferenced like normal

----

```
#define KPATCH_SHADOW_NEWPID 0

newpid = klp_shadow_get(p, KPATCH_SHADOW_NEWPID);
if (newpid)
    seq_printf(m, "newpid:\t%d\n", *newpid);
...
```

----

![](ritlug_w7_livepatching/struct_shadowvar.png)

----

#### Other functions

- `klp_shadow_free()`
- `klp_shadow_alloc()`
- `klp_shadow_free_all()`

----

#### Other Limitations

- Cannot patch `notrace` functions
    - This includes any functions with the `inline` annotation
- Very limited support for patching syscalls
    - Most are implemented with inline, but workarounds can be done
- Cannot really change the function declaration
    - No parameter changes, only additional variables and such

---

#### Current State of Livepatching