With CONFIG_BPF_JIT, the kernel makes indirect calls to dynamically
generated code, which the compile-time Control-Flow Integrity (CFI)
checking cannot validate. This change adds basic sanity checking to
ensure we are jumping to a valid location, which narrows down the
attack surface on the stored pointer.
In addition, this change adds a weak arch_bpf_jit_check_func function,
which architectures that implement BPF JIT can override to perform
additional validation, such as verifying that the pointer points to
the correct memory region.
Bug: 145210207
Change-Id: I1a90c70cdcef25673a870d3c4f2586a829c0d32e
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change adds the CONFIG_CFI_CLANG option, CFI error handling,
and a faster look-up table for cross module CFI checks.
Bug: 145210207
Change-Id: I118303de50114ca6f85d89a7d69c5cbc47e2f5c0
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Allow CONFIG_LTO_CLANG and CONFIG_THINLTO to be enabled.
Bug: 145210207
Change-Id: If0d2cf24eabd3720576489cc74410681ef722784
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
LLVM's integrated assembler fails with the following error when
building KVM:
<inline asm>:12:6: error: expected absolute expression
.if kvm_update_va_mask == 0
^
<inline asm>:21:6: error: expected absolute expression
.if kvm_update_va_mask == 0
^
<inline asm>:24:2: error: unrecognized instruction mnemonic
NOT_AN_INSTRUCTION
^
LLVM ERROR: Error parsing inline asm
These errors come from ALTERNATIVE_CB and __ALTERNATIVE_CFG,
which test for the existence of the callback parameter in inline
assembly using the following expression:
" .if " __stringify(cb) " == 0\n"
This works with GNU as, but isn't supported by LLVM. This change
splits __ALTERNATIVE_CFG and ALTINSTR_ENTRY into separate macros
to fix the LLVM build.
Bug: 145210207
Change-Id: I3f80fca8aafdac4e185f79ce5a4eee9ba367bb33
(am from https://lore.kernel.org/patchwork/patch/1146950/)
Link: https://github.com/ClangBuiltLinux/linux/issues/472
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Unlike gcc, clang considers each inline assembly block to be independent
and therefore, when using the integrated assembler for inline assembly,
any preambles that enable features must be repeated in each block.
This change defines __LSE_PREAMBLE and adds it to each inline assembly
block that has LSE instructions, which allows them to be compiled also
with clang's assembler.
Bug: 145210207
Change-Id: Ifdcb160ddb074bea62a52239fffb0590f409df46
(am from https://lore.kernel.org/patchwork/patch/1146951/)
Link: https://github.com/ClangBuiltLinux/linux/issues/671
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With LTO, the compiler doesn't necessarily obey link order for
initcalls, and the initcall variables need to be globally unique
to avoid naming collisions.
In order to preserve the intended order, this change moves each
initcall variable into its own section and generates a linker
script (in scripts/link-vmlinux.sh) to define the correct order
for these sections. We also add a __COUNTER__ prefix to the name,
so we can retain the order of initcalls within each compilation
unit, and __LINE__ to help ensure uniqueness.
Bug: 145210207
Change-Id: I602038783853497790c5a2941343c546e380c525
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Disable LTO for rodata.o to allow objcopy to be used to
manipulate sections.
Bug: 145210207
Change-Id: I387a37fd2dd13a877e9e66e9f99c9c4b10b0e963
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With CONFIG_LTO_CLANG, we produce LLVM IR instead of object files. Since LTO
is not really needed here and the Makefile assumes we produce an object file,
disable LTO for libstub.
Bug: 145210207
Change-Id: I7f1f9af7430164ebbcb0e85f66abae5cb9feee6a
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With CONFIG_LTO_CLANG, clang generates LLVM IR instead of ELF object
files. As empty.o is used for probing target properties, disable LTO
for it to produce an object file instead.
Bug: 145210207
Change-Id: I618d8b86ed88ad048abdee3c541ced19d12982c0
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With CONFIG_LTO_CLANG enabled, LLVM IR won't be compiled into object
files until modpost_link. This change postpones calls to recordmcount
until after this step.
In order to exclude ftrace_process_locs from inspection, we add a new
code section .text..ftrace, which we tell recordmcount to ignore, and
a __norecordmcount attribute for moving functions to this section.
Bug: 145210207
Change-Id: Ib77f7c431fce54243c46d584b55761ed2342965c
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change adds the configuration option CONFIG_LTO_CLANG, and
build system support for Clang's Link Time Optimization (LTO). In
preparation for LTO support with other compilers, potentially common
parts of the changes are gated behind CONFIG_LTO instead.
With -flto, instead of object files, Clang produces LLVM bitcode,
which is compiled into a native object at link time, allowing the
final binary to be optimized globally. For more details, see:
https://llvm.org/docs/LinkTimeOptimization.html
While the kernel normally uses GNU ld for linking, LLVM supports LTO
only with LLD or GNU gold linkers. This change assumes LLD is used.
Bug: 145210207
Change-Id: If1164ff33d073358ee7d4bba84cbb06c349c4a88
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Similarly to the CC_IS_CLANG config, add LD_IS_LLD to simplify feature
selection based on the linker.
Bug: 145210207
Change-Id: I097c52899dcf9829eb0e1ea89211b17972301c1a
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
platform_find_device_by_driver calls bus_find_device and passes
platform_match as the callback function. Casting the function to a
mismatching type trips indirect call Control-Flow Integrity (CFI) checking.
This change adds a callback function with the correct type and instead
of casting the function, explicitly casts the second parameter to struct
device_driver* as expected by platform_match.
Bug: 145210207
Change-Id: Idef667974d3c54ebd79f0813531cf2523d651dfe
(cherry picked from commit 492c88720d
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git
driver-core-next)
Fixes: 36f3313d6b ("platform: Add platform_find_device_by_driver() helper")
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Cc: stable <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/20191112214156.3430-1-samitolvanen@google.com
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
If we detect a corrupted x18 and SCS is enabled, restore the register
before jumping back to instrumented code. This is safe, because the
wrapper is called with preemption disabled and a separate shadow stack
is used for interrupt handling.
Bug: 145210207
Change-Id: Idb75117e38c895231a14f5573261861e722c1264
(am from https://lore.kernel.org/patchwork/patch/1149060/)
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reserve the x18 register from general allocation when SCS is enabled,
because the compiler uses the register to store the current task's
shadow stack pointer. Note that all external kernel modules must also be
compiled with -ffixed-x18 if the kernel has SCS enabled.
Bug: 145210207
Change-Id: I0407d38a0a5ecb6852b3f281d52f6601c565157e
(am from https://lore.kernel.org/patchwork/patch/1149058/)
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
The graph tracer hooks returns by modifying frame records on the
(regular) stack, but with SCS the return address is taken from the
shadow stack, and the value in the frame record has no effect. As we
don't currently have a mechanism to determine the corresponding slot
on the shadow stack (and to pass this through the ftrace
infrastructure), for now let's disable the graph tracer when SCS is
enabled.
Bug: 145210207
Change-Id: I65dd098be827121ecf0c08538b11e3b98f5eacde
(am from https://lore.kernel.org/patchwork/patch/1149057/)
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change adds generic support for Clang's Shadow Call Stack,
which uses a shadow stack to protect return addresses from being
overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that security guarantees in the kernel differ from the
ones documented for user space. The kernel must store addresses
of shadow stacks used by other tasks and interrupt handlers in
memory, which means an attacker capable reading and writing
arbitrary memory may be able to locate them and hijack control
flow by modifying shadow stacks that are not currently in use.
Bug: 145210207
Change-Id: I2a8ba6a3decac50c169731c3121c9dcab96621d2
(am from https://lore.kernel.org/patchwork/patch/1149054/)
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
In preparation of reserving x18, stop treating it as caller save in
the KVM guest entry/exit code. Currently, the code assumes there is
no need to preserve it for the host, given that it would have been
assumed clobbered anyway by the function call to __guest_enter().
Instead, preserve its value and restore it upon return.
Bug: 145210207
Change-Id: I341bcb10b615999a59a8413a6b98cb2ce1c62e02
(am from https://lore.kernel.org/patchwork/patch/1149065/)
Link: https://patchwork.kernel.org/patch/9836891/
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
[Sami: updated commit message, switched from x18 to x29 for the guest context]
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Because period and duty cycle are defined as ints with units of
nanoseconds, the maximum time duration that can be set is limited to
~2.147 seconds. Change their definitions to u64 in the structs of the
PWM framework so that higher durations may be set.
Also make the relevant fixes to those drivers that use the period and
duty_cycle struct members in division operations, viz. replacing the
division operations with 64-bit division macros as appropriate.
Bug: 140290586
Change-Id: Ibb8c7f007f4c11ebf8cf01e7a468cc9c29aa8b23
Link: https://www.spinics.net/lists/linux-pwm/msg11133.html
Signed-off-by: Guru Das Srinagesh <gurus@codeaurora.org>
Reported-by: kbuild test robot <lkp@intel.com>
Signed-off-by: Todd Kjos <tkjos@google.com>
Set the default for FW_CACHE to n to match Android expectations.
Signed-off-by: Mark Salyzyn <salyzyn@google.com>
Bug: 32180327
Bug: 137566496
Change-Id: I86a28040be07fb2f30125d28f7f379798727785c
By conditionally dropping support for the legacy binding and exporting
the newly introduced 'arm_smmu_impl_init()' function we can allow the
ARM SMMU driver to be built as a module.
Signed-off-by: Will Deacon <will@kernel.org>
Bug: 140290589
Link: https://lore.kernel.org/lkml/20191121114918.2293-14-will@kernel.org/
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id42da5829cb180966a0d051ec636be0c9f90efed
When removing the SMMU driver, we need to clear any state that we
registered during probe. This includes our bus ops, sysfs entries and
the IOMMU device registered for early firmware probing of masters.
Signed-off-by: Will Deacon <will@kernel.org>
Bug: 140290589
Link: https://lore.kernel.org/lkml/20191121114918.2293-13-will@kernel.org/
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5dd6e68a609ac60bc704f9f916acc0f31dc1bd67
When removing the SMMUv3 driver, we need to clear any state that we
registered during probe. This includes our bus ops, sysfs entries and
the IOMMU device registered for early firmware probing of masters.
Signed-off-by: Will Deacon <will@kernel.org>
Bug: 140290589
Link: https://lore.kernel.org/lkml/20191121114918.2293-11-will@kernel.org/
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I54372d917c871231ced86250498fe2e45cf39246
Forcefully unbinding the Arm SMMU drivers is a pretty dangerous operation,
since it will likely lead to catastrophic failure for any DMA devices
mastering through the SMMU being unbound. When the driver then attempts
to "handle" the fatal faults, it's very easy to trip over dead data
structures, leading to use-after-free.
On John's machine, he reports that the machine was "unusable" due to
loss of the storage controller following a forced unbind of the SMMUv3
driver:
| # cd ./bus/platform/drivers/arm-smmu-v3
| # echo arm-smmu-v3.0.auto > unbind
| hisi_sas_v2_hw HISI0162:01: CQE_AXI_W_ERR (0x800) found!
| platform arm-smmu-v3.0.auto: CMD_SYNC timeout at 0x00000146
| [hwprod 0x00000146, hwcons 0x00000000]
Prevent this forced unbinding of the drivers by setting "suppress_bind_attrs"
to true.
Link: https://lore.kernel.org/lkml/06dfd385-1af0-3106-4cc5-6a5b8e864759@huawei.com
Reported-by: John Garry <john.garry@huawei.com>
Signed-off-by: Will Deacon <will@kernel.org>
Bug: 140290589
Link: https://lore.kernel.org/lkml/20191121114918.2293-10-will@kernel.org/
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I8b1c439f7ba11d6fc38cdf686948ef5909871dfb
'bus_set_iommu()' allows IOMMU drivers to register their ops for a given
bus type. Unfortunately, it then doesn't allow them to be removed, which
is necessary for modular drivers to shutdown cleanly so that they can be
reloaded later on.
Allow 'bus_set_iommu()' to take a NULL 'ops' argument, which clear the
ops pointer for the selected bus_type.
Signed-off-by: Will Deacon <will@kernel.org>
Bug: 140290589
Link: https://lore.kernel.org/lkml/20191121114918.2293-7-will@kernel.org/
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ia89fa2df61d2a9ad04901d4c2e71e293c4c6171d
