Exercise 1 from chapter 4

[Vinícius Gajo]

Description:

Hello, hope you're good.

When I was working in the 1st exercise of the 4th chapter, I noticed that the values are different.

So, my thought process was:

First, use the command specified in the chapter:

cd chapter-4/
sudo strace -e bpf ./hello-buffer-config.py
# ...
# bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=44, insns=0x7ff4c652ebe8, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 15, 99), prog_flags=0, prog_name="hello", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=4, func_info_rec_size=8, func_info=0x55855b344ae0, func_info_cnt=1, line_info_rec_size=16, line_info=0x55855b3ac890, line_info_cnt=21, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 128) = 7
# ...

Notice that the insn_cnt value is 44.

Then, for checking the number of instructions from the dumped eBPF program I used:

sudo bpftool prog dump xlated name hello
# int hello(void * ctx):
# ; int hello(void *ctx) {
#    0: (bf) r6 = r1
#    1: (b7) r1 = 0
# ; struct data_t data = {};
#    2: (63) *(u32 *)(r10 -8) = r1
#    3: (7b) *(u64 *)(r10 -16) = r1
#    4: (7b) *(u64 *)(r10 -24) = r1
#    5: (7b) *(u64 *)(r10 -32) = r1
#    6: (b7) r1 = 6581362
# ; char message[12] = "Hello World";
#    7: (63) *(u32 *)(r10 -48) = r1
#    8: (18) r1 = 0x6f57206f6c6c6548
#   10: (7b) *(u64 *)(r10 -56) = r1
# ; data.pid = bpf_get_current_pid_tgid() >> 32;
#   11: (85) call bpf_get_current_pid_tgid#164096
# ; data.pid = bpf_get_current_pid_tgid() >> 32;
#   12: (77) r0 >>= 32
# ; data.pid = bpf_get_current_pid_tgid() >> 32;
#   13: (63) *(u32 *)(r10 -40) = r0
# ; data.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
#   14: (85) call bpf_get_current_uid_gid#164432
# ; data.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
#   15: (63) *(u32 *)(r10 -36) = r0
# ; struct data_t data = {};
#   16: (bf) r1 = r10
#   17: (07) r1 += -32
# ; bpf_get_current_comm(&data.command, sizeof(data.command));
#   18: (b7) r2 = 16
#   19: (85) call bpf_get_current_comm#164544
# ; p = bpf_map_lookup_elem((void *)bpf_pseudo_fd(1, -1), &data.uid);
#   20: (18) r1 = map[id:6]
# ; data.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
#   22: (bf) r2 = r10
#   23: (07) r2 += -36
# ; p = bpf_map_lookup_elem((void *)bpf_pseudo_fd(1, -1), &data.uid);
#   24: (85) call __htab_map_lookup_elem#181488
#   25: (15) if r0 == 0x0 goto pc+1
#   26: (07) r0 += 56
#   27: (bf) r3 = r10
# ; 
#   28: (07) r3 += -56
# ; if (p != 0) {
#   29: (15) if r0 == 0x0 goto pc+1
#   30: (bf) r3 = r0
# ; struct data_t data = {};
#   31: (bf) r1 = r10
#   32: (07) r1 += -16
# ; 
#   33: (b7) r2 = 12
#   34: (85) call bpf_probe_read_kernel#-68352
# ; bpf_perf_event_output(ctx, bpf_pseudo_fd(1, -2), CUR_CPU_IDENTIFIER, &data, sizeof(data));
#   35: (18) r2 = map[id:5]
#   37: (bf) r4 = r10
# ; struct data_t data = {};
#   38: (07) r4 += -40
# ; bpf_perf_event_output(ctx, bpf_pseudo_fd(1, -2), CUR_CPU_IDENTIFIER, &data, sizeof(data));
#   39: (bf) r1 = r6
#   40: (18) r3 = 0xffffffff
#   42: (b7) r5 = 36
#   43: (85) call bpf_perf_event_output#-67376
# ; return 0;
#   44: (b7) r0 = 0
#   45: (95) exit

As I'm understanding it, the result is 45, which is different than 44.

Am I doing something wrong in this case?

[Quentin Monnet]

Everything looks in order here. The eBPF backend in LLVM is evolving, and different versions of clang may generate slightly different versions of the eBPF bytecode for a given program - this is nothing unexpected.

If you're really curious about it, you could try different versions of clang, or even compile it to bisect and pin down what change in the LLVM repo caused the difference in the bytecode :).

Scratch the above, I didn't read correctly and replied too fast, sorry.

[Quentin Monnet]

One thing that was valid from my previous comment: Everything is still in order :), you've done nothing wrong here.

So your loaded program has two more instructions (46, given numbering starts from 0) than the initial bytecode. Let's compare the instructions from the program you load with what's in the kernel. We can make BCC print the raw bytecode:

iff --git a/chapter4/hello-buffer-config.py b/chapter4/hello-buffer-config.py
index 59c9fd131a68..7f0fb2abcd31 100755
--- a/chapter4/hello-buffer-config.py
+++ b/chapter4/hello-buffer-config.py
@@ -45,6 +45,8 @@ int hello(void *ctx) {
 b = BPF(text=program) 
 syscall = b.get_syscall_fnname("execve")
 b.attach_kprobe(event=syscall, fn_name="hello")
+print(b.dump_func("hello").hex())
+exit()
 b["config"][ct.c_int(0)] = ct.create_string_buffer(b"Hey root!")
 b["config"][ct.c_int(501)] = ct.create_string_buffer(b"Hi user 501!"

Run it:

# ./hello-buffer-config.py | fold -w 16
bf16000000000000
b701000000000000
631af8ff00000000
7b1af0ff00000000
7b1ae8ff00000000
7b1ae0ff00000000
b7010000726c6400
631ad0ff00000000
1801000048656c6c
000000006f20576f
7b1ac8ff00000000
850000000e000000
7700000020000000
630ad8ff00000000
850000000f000000
630adcff00000000
bfa1000000000000
07010000e0ffffff
b702000010000000
8500000010000000
1811000005000000
0000000000000000
bfa2000000000000
07020000dcffffff
8500000001000000

<2 missing instructions here, number 25 and 26 from your output>

bfa3000000000000
07030000c8ffffff
1500010000000000
bf03000000000000
bfa1000000000000
07010000f0ffffff
b70200000c000000
8500000071000000
1812000004000000
0000000000000000
bfa4000000000000
07040000d8ffffff
bf61000000000000
18030000ffffffff
0000000000000000
b705000024000000
8500000019000000
b700000000000000
9500000000000000

(I added the note on missing instructions to this output, obviously.)

So the kernel adds two instructions after your map lookup:

   24: (85) call __htab_map_lookup_elem#181488
   25: (15) if r0 == 0x0 goto pc+1
   26: (07) r0 += 56

Where do these instructions come from? Note how __htab_map_lookup_elem has replaced bpf_map_lookup_elem. This is a hint: the verifier has inlined the map lookup. As explained in the comments for the relevant function, this means that it replaces the following call sequence:

 * bpf_prog
 *   bpf_map_lookup_elem
 *     map->ops->map_lookup_elem
 *       htab_map_lookup_elem
 *         __htab_map_lookup_elem

with the following, less costly at runtime:

 * bpf_prog
 *   __htab_map_lookup_elem

But to do that, it must add a few additional instructions, that would otherwise be called from the function we're now skipping. That's where you get the additional check from.

[Vinícius Gajo]

Great, thanks for the detailed answer @qmonnet! Now it's clearer to me.