Daniel Borkmann
a493a87f38
bpf, x64: implement retpoline for tail call
...
Implement a retpoline [0] for the BPF tail call JIT'ing that converts
the indirect jump via jmp %rax that is used to make the long jump into
another JITed BPF image. Since this is subject to speculative execution,
we need to control the transient instruction sequence here as well
when CONFIG_RETPOLINE is set, and direct it into a pause + lfence loop.
The latter aligns also with what gcc / clang emits (e.g. [1]).
JIT dump after patch:
# bpftool p d x i 1
0: (18) r2 = map[id:1]
2: (b7) r3 = 0
3: (85) call bpf_tail_call#12
4: (b7) r0 = 2
5: (95) exit
With CONFIG_RETPOLINE:
# bpftool p d j i 1
[...]
33: cmp %edx,0x24(%rsi)
36: jbe 0x0000000000000072 |*
38: mov 0x24(%rbp),%eax
3e: cmp $0x20,%eax
41: ja 0x0000000000000072 |
43: add $0x1,%eax
46: mov %eax,0x24(%rbp)
4c: mov 0x90(%rsi,%rdx,8),%rax
54: test %rax,%rax
57: je 0x0000000000000072 |
59: mov 0x28(%rax),%rax
5d: add $0x25,%rax
61: callq 0x000000000000006d |+
66: pause |
68: lfence |
6b: jmp 0x0000000000000066 |
6d: mov %rax,(%rsp) |
71: retq |
72: mov $0x2,%eax
[...]
* relative fall-through jumps in error case
+ retpoline for indirect jump
Without CONFIG_RETPOLINE:
# bpftool p d j i 1
[...]
33: cmp %edx,0x24(%rsi)
36: jbe 0x0000000000000063 |*
38: mov 0x24(%rbp),%eax
3e: cmp $0x20,%eax
41: ja 0x0000000000000063 |
43: add $0x1,%eax
46: mov %eax,0x24(%rbp)
4c: mov 0x90(%rsi,%rdx,8),%rax
54: test %rax,%rax
57: je 0x0000000000000063 |
59: mov 0x28(%rax),%rax
5d: add $0x25,%rax
61: jmpq *%rax |-
63: mov $0x2,%eax
[...]
* relative fall-through jumps in error case
- plain indirect jump as before
[0] https://support.google.com/faqs/answer/7625886
[1] a31e654fa1
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net >
Signed-off-by: Alexei Starovoitov <ast@kernel.org >
2018-02-22 15:31:42 -08:00
..
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-15 19:56:20 -06:00
2017-12-29 21:13:04 +01:00
2018-01-16 03:58:38 +01:00
2018-01-14 09:51:25 -08:00
2017-03-02 08:42:37 +01:00
2018-02-14 17:02:15 -08:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-05 14:01:15 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-14 12:14:51 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-09-24 11:19:01 +02:00
2018-01-27 19:10:45 +01:00
2017-12-18 11:56:22 -03:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-15 01:15:50 +01:00
2017-11-02 11:10:55 +01:00
2017-12-17 12:58:53 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-15 01:15:50 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2015-11-29 08:50:58 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-15 19:56:20 -06:00
2017-11-02 11:10:55 +01:00
2017-12-23 21:13:00 +01:00
2017-11-02 11:10:55 +01:00
2018-02-06 10:29:42 +01:00
2018-02-01 15:04:17 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-12-29 17:02:49 -08:00
2017-11-02 11:10:55 +01:00
2018-01-30 15:08:27 +01:00
2017-11-02 11:10:55 +01:00
2018-01-10 16:40:54 +01:00
2018-01-15 09:34:59 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-16 11:43:11 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-12 17:33:38 -08:00
2017-12-22 20:13:05 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-30 15:30:35 +01:00
2016-12-18 09:25:38 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-30 23:55:33 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-23 20:18:18 +01:00
2018-01-14 21:11:54 +01:00
2018-01-14 20:18:23 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2016-02-23 07:37:23 +01:00
2017-11-01 21:50:11 +01:00
2017-11-02 11:10:55 +01:00
2017-11-23 20:17:59 +01:00
2016-07-21 09:37:11 -03:00
2017-11-02 11:10:55 +01:00
2017-12-23 21:13:00 +01:00
2017-11-02 11:10:55 +01:00
2017-11-27 13:39:11 +02:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2016-05-05 13:58:55 -07:00
2018-02-04 15:55:52 +02:00
2017-01-09 23:13:36 +01:00
2017-12-22 20:13:04 +01:00
2017-11-07 10:51:10 +01:00
2017-11-16 12:49:48 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-12-04 10:56:53 +10:00
2016-03-08 12:26:41 +01:00
2017-11-02 11:10:55 +01:00
2017-06-22 18:21:11 +02:00
2018-01-30 23:55:32 +01:00
2017-11-02 11:10:55 +01:00
2017-11-07 10:51:10 +01:00
2017-12-29 21:13:04 +01:00
2017-12-17 13:59:52 +01:00
2018-01-14 21:11:54 +01:00
2017-11-02 11:10:55 +01:00
2018-02-06 18:32:43 -08:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-03 09:02:11 -08:00
2017-12-17 13:59:52 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-12 17:33:37 -08:00
2017-12-05 21:20:24 +01:00
2017-11-02 11:10:55 +01:00
2018-01-31 18:25:35 +01:00
2017-11-02 11:10:55 +01:00
2017-11-10 10:03:13 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2016-07-14 15:07:00 +02:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-12-18 12:58:29 +01:00
2018-01-16 01:50:59 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-12-23 21:13:00 +01:00
2017-12-22 20:13:01 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-12-17 12:58:53 +01:00
2018-01-14 21:11:54 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-10 13:16:35 -08:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-01 15:04:17 +01:00
2017-11-02 11:10:55 +01:00
2018-01-30 21:54:29 +01:00
2016-03-29 12:23:26 +02:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-22 15:31:42 -08:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-07-26 13:18:20 +02:00
2017-07-26 13:18:20 +02:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-13 16:25:06 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-15 01:15:52 +01:00
2018-02-15 01:15:52 +01:00
2017-11-02 11:10:55 +01:00
2018-01-16 16:40:09 +01:00
2017-11-02 11:10:55 +01:00
2018-01-11 11:22:39 -06:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-15 15:01:28 -08:00
2017-12-17 12:58:53 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-12-23 21:13:00 +01:00
2018-01-30 15:30:35 +01:00
2018-02-15 01:15:52 +01:00
2018-01-05 00:39:57 +01:00
2017-12-23 21:12:59 +01:00
2017-11-15 18:21:05 -08:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-31 17:18:38 -08:00
2018-01-31 17:18:38 -08:00
2017-11-02 11:10:55 +01:00
2015-12-19 11:49:56 +00:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-01-14 10:45:53 +01:00
2018-02-14 17:02:15 -08:00
2017-11-02 11:10:55 +01:00
2017-12-23 21:12:59 +01:00
2017-12-17 12:58:53 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-16 13:06:27 -08:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-07 10:32:44 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-09-28 09:45:05 +02:00
2018-01-26 15:53:16 +01:00
2017-11-02 11:10:55 +01:00
2017-12-17 12:58:53 +01:00
2017-11-07 10:32:44 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-28 00:28:56 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2016-01-30 11:22:17 +01:00
2018-02-17 09:40:45 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-07 10:32:44 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-12-22 20:13:02 +01:00
2017-11-15 18:21:05 -08:00
2017-11-15 18:21:05 -08:00
2017-11-02 11:10:55 +01:00
2017-12-15 12:21:38 +01:00
2017-12-15 12:21:38 +01:00
2017-11-02 11:10:55 +01:00
2017-12-04 10:57:24 -06:00
2018-01-10 16:41:12 +01:00
2017-12-31 13:13:56 -08:00
2017-11-02 11:10:55 +01:00
2018-02-05 21:34:57 +01:00
2018-01-30 15:30:36 +01:00
2017-10-20 10:37:33 +02:00
2017-11-02 11:10:55 +01:00
2018-02-04 11:45:55 -08:00
2017-11-02 11:10:55 +01:00
2017-11-13 19:07:38 -08:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2018-02-15 01:15:52 +01:00
2017-07-27 10:53:00 +02:00
2017-11-02 11:10:55 +01:00
2018-01-19 16:31:28 +01:00
2017-11-13 19:07:38 -08:00
2018-01-30 21:54:31 +01:00
2018-01-30 21:54:31 +01:00
2018-01-30 21:54:31 +01:00
2017-11-08 11:16:22 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-08-10 14:16:19 +02:00
2018-01-03 16:14:46 +01:00
2017-12-11 18:42:11 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-02 11:10:55 +01:00
2017-11-07 10:32:44 +01:00
2016-06-16 00:04:31 +02:00
2017-11-02 11:10:55 +01:00
2017-10-12 14:01:53 +02:00
2017-12-23 21:13:00 +01:00
2017-11-02 11:10:55 +01:00
2018-01-14 21:11:53 +01:00
2016-04-13 11:37:40 +02:00
2017-11-02 11:10:55 +01:00
2016-04-13 11:37:40 +02:00
2017-11-15 18:21:04 -08:00