asm/alternative.h contains both the macros needed to use alternatives,
as well the type definitions and function prototypes for applying them.
Split the header in two, so that alternatives can be used from core
header files such as linux/compiler.h without the risk of circular
includes
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit 7cda23da52)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 145210207
Change-Id: Idf276e7af1fa5e2d5e5a2565c1d88738355f199b
The uao_* alternative asm macros are only used by the uaccess assembly
routines in arch/arm64/lib/, where they are included indirectly via
asm-uaccess.h. Since they're specific to the uaccess assembly (and will
lose the alternatives in subsequent patches), let's move them into
asm-uaccess.h.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
[will: update #include in mte.S to pull in uao asm macros]
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit e2a2190a80)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 145210207
Change-Id: I92b592f07f5bf5a5286f40df912876b697541860
Prepare to enable LTO, which requires options that depends on objtool.
Bug: 175705438
Signed-off-by: Alistair Delva <adelva@google.com>
Change-Id: I830614180286b3e2222a5a445fb4db82658f6670
When `pkg-config --libs' wasn't used or didn't return anything useful,
the test for libelf would fail even if the headers and libraries were
available to kbuild via KBUILD_CFLAGS and KBUILD_LDFLAGS.
This makes the check in the core Makefile match the check that is done
in tools/objtool/Makefile. Specifically, the C compiler is passed the
full KBUILD_CFLAGS and KBUILD_LDFLAGS, which may have set up additional
sysroot/include/library search paths to find libelf.
Signed-off-by: Alistair Delva <adelva@google.com>
Link: https://lore.kernel.org/lkml/20210112190010.1582888-1-adelva@google.com/
Bug: 175705438
Change-Id: I06b0997c2df8545877d8b2d29b7a3045551c92e2
CONFIG_CFI_CLANG breaks cross-module function address equality, which
breaks virtio_dma_buf as it compares a locally taken function address to
a one passed from a different module. Remove these sanity checks for now
to allow Cuttlefish to boot with CFI.
Bug: 178495907
Change-Id: I508e2fa9e3ddd888dc2fdb743cec20a400eaf1f5
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
During runtime-suspend of ufs host, the scsi devices are
already suspended and so are the queues associated with them.
But the ufs host sends SSU to wlun during its runtime-suspend.
During the process blk_queue_enter checks if the queue is not in
suspended state. If so, it waits for the queue to resume, and never
comes out of it.
The commit
(d55d15a33: scsi: block: Do not accept any requests while suspended)
adds the check if the queue is in suspended state in blk_queue_enter().
Fix this, by decoupling wlun scsi devices from block layer pm.
The runtime-pm for these devices would be managed by bsg and sg drivers.
Call trace:
__switch_to+0x174/0x2c4
__schedule+0x478/0x764
schedule+0x9c/0xe0
blk_queue_enter+0x158/0x228
blk_mq_alloc_request+0x40/0xa4
blk_get_request+0x2c/0x70
__scsi_execute+0x60/0x1c4
ufshcd_set_dev_pwr_mode+0x124/0x1e4
ufshcd_suspend+0x208/0x83c
ufshcd_runtime_suspend+0x40/0x154
ufshcd_pltfrm_runtime_suspend+0x14/0x20
pm_generic_runtime_suspend+0x28/0x3c
__rpm_callback+0x80/0x2a4
rpm_suspend+0x308/0x614
rpm_idle+0x158/0x228
pm_runtime_work+0x84/0xac
process_one_work+0x1f0/0x470
worker_thread+0x26c/0x4c8
kthread+0x13c/0x320
ret_from_fork+0x10/0x18
Bug: 178653131
Link: https://lore.kernel.org/linux-arm-msm/7929cc67311133f8dbdfe5e627cf6a2f1daa486e.1611719814.git.asutoshd@codeaurora.org/T/#u
Change-Id: I78e49e0dd7d4d16c1d6691ea04f9aabab2ef8f43
Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
Signed-off-by: Can Guo <cang@codeaurora.org>
Signed-off-by: Bao D. Nguyen <nguyenb@codeaurora.org>
During checkpoint=disable period, f2fs bypasses all the synchronous IOs such as
sync and fsync. So, when enabling it back, we must flush all of them in order
to keep the data persistent. Otherwise, suddern power-cut right after enabling
checkpoint will cause data loss.
Bug: 171063590
Fixes: 4354994f09 ("f2fs: checkpoint disabling")
Cc: stable@vger.kernel.org
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
(cherry picked from commit 8d52dbb373579b48f5758dd0cdd2ac0fb4e5be7f git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git dev)
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Change-Id: Iaca2d6fc1841fffa8677d5d592732c94241fb3fb
arm_smccc_1_1_hvc() only adds write contraints for x0-3 in the inline
assembly for the HVC instruction so make sure those are the only
registers that change when __do_hyp_init is called.
Tested-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Andrew Scull <ascull@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210125145415.122439-3-ascull@google.com
(cherry picked from commit e500b805c3
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4c3c3b9308bb2ab9ac088fe3d2429a359755298e
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The use of a tagged address could be pretty confusing for the
whole memslot infrastructure as well as the MMU notifiers.
Forbid it altogether, as it never quite worked the first place.
Cc: stable@vger.kernel.org
Reported-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 139bc8a614
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5a04c24948e1fc78c4d34cca514f348a0e7d9bc3
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
When running on v8.0 HW, make sure we don't try to advertise
events in the 0x4000-0x403f range.
Cc: stable@vger.kernel.org
Fixes: 88865beca9 ("KVM: arm64: Mask out filtered events in PCMEID{0,1}_EL1")
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210121105636.1478491-1-maz@kernel.org
(cherry picked from commit 9529aaa056
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5f4dbfd1d8be740f01d9d297e4af15e61ff86617
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KASAN in HW_TAGS mode will store MTE tags in the top byte of the
pointer. When computing the offset for TPIDR_EL2 we don't want anything
in the top byte, so remove the tag to ensure the computation is correct
no matter what the tag.
Fixes: 94ab5b61ee ("kasan, arm64: enable CONFIG_KASAN_HW_TAGS")
Signed-off-by: Steven Price <steven.price@arm.com>
[maz: added comment]
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210108161254.53674-1-steven.price@arm.com
(cherry picked from commit e1663372d5
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I9b2d2b32b4c3028e35674fd8ea9e283b8bf6e463
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The reg_to_encoding() macro is a wrapper over sys_reg() and conveniently
takes a sys_reg_desc or a sys_reg_params argument and returns the 32 bit
register encoding. Use it instead of calling sys_reg() directly.
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210106144218.110665-1-alexandru.elisei@arm.com
(cherry picked from commit 7ba8b4380a
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I766371bee6a5879c69c318f9bcdcbda8b336e9ab
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The KVM/arm64 PSCI relay assumes that SYSTEM_OFF and SYSTEM_RESET should
not return, as dictated by the PSCI spec. However, there is firmware out
there which breaks this assumption, leading to a hyp panic. Make KVM
more robust to broken firmware by allowing these to return.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201229160059.64135-1-dbrazdil@google.com
(cherry picked from commit 2c91ef3921
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Idf6dae27320dc7030249831f4f8b1b72a7c89ad7
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Now that all PMU registers are gated behind a .visibility callback,
remove the other checks against an absent PMU.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 7ded92e25c
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I425f8288d73f7ec6f4daa8c99729be3caadaa595
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
It appears that while we are now able to properly hide PMU
registers from the guest when a PMU isn't available (either
because none has been configured, the host doesn't have
the PMU support compiled in, or that the HW doesn't have
one at all), we are still exposing more than we should to
userspace.
Introduce a visibility callback gating all the PMU registers,
which covers both usrespace and guest.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 11663111cd
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fixes)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I3c2dfb7f35494bfd4141d001cb67e20be773e5b6
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KVM_ARM_PMU only existed for the benefit of 32bit ARM hosts,
and makes no sense now that we are 64bit only. Get rid of it.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 8cbebc4118)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5a364cc363dae36bb57d5ca2c3b8b2e77a7458d1
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Although not a problem right now, it flared up while working
on some other aspects of the code-base. Remove the useless
semicolon.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 957cbca731)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I50238656ebb9c34f308e9c326a24df3bdb0eaed1
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The __init annotations on hyp_cpu_pm_{init,exit} are obviously incorrect,
and the build system shouts at you if you enable DEBUG_SECTION_MISMATCH.
Nothing really bad happens as we never execute that code outside of the
init context, but we can't label the callers as __int either, as kvm_init
isn't __init itself. Oh well.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Nathan Chancellor <natechancellor@gmail.com>
Link: https://lore.kernel.org/r/20201223120854.255347-1-maz@kernel.org
(cherry picked from commit 44362a3c35)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I151425e9da2aa45a27c60987e7a746c234066e49
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
dist->ready setting is pointlessly spread across the two vgic
backends, while it could be consolidated in kvm_vgic_map_resources().
Move it there, and slightly simplify the flows in both backends.
Suggested-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 101068b566)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I03eba0098367c67ae8f82c9a0029b89f12e999da
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KVM_ARM_VCPU_INIT ioctl calls kvm_reset_vcpu(), which in turn resets the
PMU with a call to kvm_pmu_vcpu_reset(). The function zeroes the PMU
chained counters bitmap and stops all the counters with a perf event
attached. Because it is called before the VCPU has had the chance to run,
no perf events are in use and none are released.
kvm_arm_pmu_v3_enable(), called by kvm_vcpu_first_run_init() only if the
VCPU has been initialized, also resets the PMU. kvm_pmu_vcpu_reset() in
this case does the exact same thing as the previous call, so remove it.
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201201150157.223625-6-alexandru.elisei@arm.com
(cherry picked from commit 282ff80135)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I16372319868421ce2359727e69d52fd7158fd979
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
kvm_vgic_map_resources() is called when a VCPU if first run and it maps all
the VGIC MMIO regions. To prevent double-initialization, the VGIC uses the
ready variable to keep track of the state of resources and the global KVM
mutex to protect against concurrent accesses. After the lock is taken, the
variable is checked again in case another VCPU took the lock between the
current VCPU reading ready equals false and taking the lock.
The double-checked lock pattern is spread across four different functions:
in kvm_vcpu_first_run_init(), in kvm_vgic_map_resource() and in
vgic_{v2,v3}_map_resources(), which makes it hard to reason about and
introduces minor code duplication. Consolidate the checks in
kvm_vgic_map_resources(), where the lock is taken.
No functional change intended.
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201201150157.223625-4-alexandru.elisei@arm.com
(cherry picked from commit 1c91f06d29)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ib8b7c4529a0823726bcce60a461fd369526347bd
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
kvm_timer_enable() is called in kvm_vcpu_first_run_init() after
kvm_vgic_map_resources() if the VGIC wasn't ready. kvm_vgic_map_resources()
is the only place where kvm->arch.vgic.ready is set to true.
For a v2 VGIC, kvm_vgic_map_resources() will attempt to initialize the VGIC
and set the initialized flag.
For a v3 VGIC, kvm_vgic_map_resources() will return an error code if the
VGIC isn't already initialized.
The end result is that if we've reached kvm_timer_enable(), the VGIC is
initialzed and ready and vgic_initialized() will always be true, so remove
this check.
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
[maz: added comment about vgic initialisation, as suggested by Eric]
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201201150157.223625-3-alexandru.elisei@arm.com
(cherry picked from commit f16570ba47)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I6c5d3a27b02600b88955f2aab6619b49c56e906b
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The API documentation states that general error codes are not detailed, but
errors with specific meanings are. On arm64, KVM_RUN can return error
numbers with a different meaning than what is described by POSIX or the C99
standard (as taken from man 3 errno).
Absent from the newly documented error codes is ERANGE which can be
returned when making a change to the EL2 stage 1 tables if the address is
larger than the largest supported input address. Assuming no bugs in the
implementation, that is not possible because the input addresses which are
mapped are the result of applying the macro kern_hyp_va() on kernel virtual
addresses.
CC: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201201150157.223625-2-alexandru.elisei@arm.com
(cherry picked from commit 3557ae187c)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I895cb9efefde65ffdab3c89ea69ae6bba606dc1e
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Although there is nothing wrong with the current host PSCI relay
implementation, we can clean it up and remove some of the helpers
that do not improve the overall readability of the legacy PSCI 0.1
handling.
Opportunity is taken to turn the bitmap into a set of booleans,
and creative use of preprocessor macros make init and check
more concise/readable.
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 767c973f2e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I31a9f62da7ab63c2765bcb079e2426fcbd0d6f38
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Small cleanup moving declarations of hyp-exported variables to
kvm_host.h and using macros to avoid having to refer to them with
kvm_nvhe_sym() in host.
No functional change intended.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-5-dbrazdil@google.com
(cherry picked from commit 61fe0c37af)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I92685a9ce7a24c7d58eda5192c5df56856a525c4
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Computing the hyp VA layout is redundant when the kernel runs in EL2 and
hyp shares its VA mappings. Make calling kvm_compute_layout()
conditional on not just CONFIG_KVM but also !is_kernel_in_hyp_mode().
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-4-dbrazdil@google.com
(cherry picked from commit c3e181aec9)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I8fa9babb88700502afccb87ff66ebf9a0b30fcf2
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
init_hyp_physvirt_offset() computes PA from a kernel VA. Conversion to
kernel linear-map is required first but the code used kvm_ksym_ref() for
this purpose. Under VHE that is a NOP and resulted in a runtime warning.
Replace kvm_ksym_ref with lm_alias.
Reported-by: Qian Cai <qcai@redhat.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-3-dbrazdil@google.com
(cherry picked from commit 7a96a0687b)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I67988a954f3a253e8f97102c8b3e0c081de33c63
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
PSCI driver exposes a struct containing the PSCI v0.1 function IDs
configured in the DT. However, the struct does not convey the
information whether these were set from DT or contain the default value
zero. This could be a problem for PSCI proxy in KVM protected mode.
Extend config passed to KVM with a bit mask with individual bits set
depending on whether the corresponding function pointer in psci_ops is
set, eg. set bit for PSCI_CPU_SUSPEND if psci_ops.cpu_suspend != NULL.
Previously config was split into multiple global variables. Put
everything into a single struct for convenience.
Reported-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-2-dbrazdil@google.com
(cherry picked from commit ff367fe473)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I9118766dc065d3664a28f022a96082c5335060f7
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Conflict resolution gone astray results in the kernel not booting
on VHE-capable HW when VHE support is disabled. Thankfully spotted
by David.
Reported-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 0cc519f85a)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I98572e477ba1dd1769c3e931563035a92d2a81c5
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
With protected nVHE hyp code interception host's PSCI SMCs, the host
starts seeing new CPUs boot in EL1 instead of EL2. The kernel logic
that keeps track of the boot mode needs to be adjusted.
Add a static key enabled if KVM protected mode initialization is
successful.
When the key is enabled, is_hyp_mode_available continues to report
`true` because its users either treat it as a check whether KVM will be
/ was initialized, or whether stub HVCs can be made (eg. hibernate).
is_hyp_mode_mismatched is changed to report `false` when the key is
enabled. That's because all cores' modes matched at the point of KVM
init and KVM will not allow cores not present at init to boot. That
said, the function is never used after KVM is initialized.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-27-dbrazdil@google.com
(cherry picked from commit f19f6644a5)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5c14d42952672b2756b731663b9406463af2b5dd
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
While protected KVM is installed, start trapping all host SMCs.
For now these are simply forwarded to EL3, except PSCI
CPU_ON/CPU_SUSPEND/SYSTEM_SUSPEND which are intercepted and the
hypervisor installed on newly booted cores.
Create new constant HCR_HOST_NVHE_PROTECTED_FLAGS with the new set of HCR
flags to use while the nVHE vector is installed when the kernel was
booted with the protected flag enabled. Switch back to the default HCR
flags when switching back to the stub vector.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-26-dbrazdil@google.com
(cherry picked from commit b93c17c418)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4ca55f5d2bfaa0259b03f0193ffb8860f0b3c7ca
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KVM by default keeps the stub vector installed and installs the nVHE
vector only briefly for init and later on demand. Change this policy
to install the vector at init and then never uninstall it if the kernel
was given the protected KVM command line parameter.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-25-dbrazdil@google.com
(cherry picked from commit fa8c3d6553)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5cf077021d5135924ff9e9c38706908421603c9f
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of SYSTEM_SUSPEND host PSCI SMCs. The semantics are
equivalent to CPU_SUSPEND, typically called on the last online CPU.
Reuse the same entry point and boot args struct as CPU_SUSPEND.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-24-dbrazdil@google.com
(cherry picked from commit d945f8d9ec)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I81610430eb84ac3d09ac0caa74ff743a97d004fe
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of CPU_SUSPEND host PSCI SMCs. The SMC can either enter
a sleep state indistinguishable from a WFI or a deeper sleep state that
behaves like a CPU_OFF+CPU_ON except that the core is still considered
online while asleep.
The handler saves r0,pc of the host and makes the same call to EL3 with
the hyp CPU entry point. It either returns back to the handler and then
back to the host, or wakes up into the entry point and initializes EL2
state before dropping back to EL1. No EL2 state needs to be
saved/restored for this purpose.
CPU_ON and CPU_SUSPEND are both implemented using struct psci_boot_args
to store the state upon powerup, with each CPU having separate structs
for CPU_ON and CPU_SUSPEND so that CPU_SUSPEND can operate locklessly
and so that a CPU_ON call targeting a CPU cannot interfere with
a concurrent CPU_SUSPEND call on that CPU.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-23-dbrazdil@google.com
(cherry picked from commit abf16336dd)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I94dd625f594a981480741e93408d5605fc1a4194
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of the CPU_ON PSCI call from host. When invoked, it looks
up the logical CPU ID corresponding to the provided MPIDR and populates
the state struct of the target CPU with the provided x0, pc. It then
calls CPU_ON itself, with an entry point in hyp that initializes EL2
state before returning ERET to the provided PC in EL1.
There is a simple atomic lock around the boot args struct. If it is
already locked, CPU_ON will return PENDING_ON error code.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-22-dbrazdil@google.com
(cherry picked from commit cdf3671927)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ib11e8bf3d0a1c4070460e7aa82c9c70481958731
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
All nVHE hyp code is currently executed as handlers of host's HVCs. This
will change as nVHE starts intercepting host's PSCI CPU_ON SMCs. The
newly booted CPU will need to initialize EL2 state and then enter the
host. Add __host_enter function that branches into the existing
host state-restoring code after the trap handler would have returned.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-21-dbrazdil@google.com
(cherry picked from commit 04e05f057a)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ia0b355b82ee01eb6693e2c10327f55a743cc3011
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
In preparation for adding a CPU entry point in nVHE hyp code, extract
most of __do_hyp_init hypervisor initialization code into a common
helper function. This will be invoked by the entry point to install KVM
on the newly booted CPU.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-20-dbrazdil@google.com
(cherry picked from commit f74e1e2128)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I9c295736ba41cb5bfba2760a53bc9b06940e4f87
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Forward the following PSCI SMCs issued by host to EL3 as they do not
require the hypervisor's intervention. This assumes that EL3 correctly
implements the PSCI specification.
Only function IDs implemented in Linux are included.
Where both 32-bit and 64-bit variants exist, it is assumed that the host
will always use the 64-bit variant.
* SMCs that only return information about the system
* PSCI_VERSION - PSCI version implemented by EL3
* PSCI_FEATURES - optional features supported by EL3
* AFFINITY_INFO - power state of core/cluster
* MIGRATE_INFO_TYPE - whether Trusted OS can be migrated
* MIGRATE_INFO_UP_CPU - resident core of Trusted OS
* operations which do not affect the hypervisor
* MIGRATE - migrate Trusted OS to a different core
* SET_SUSPEND_MODE - toggle OS-initiated mode
* system shutdown/reset
* SYSTEM_OFF
* SYSTEM_RESET
* SYSTEM_RESET2
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-19-dbrazdil@google.com
(cherry picked from commit 1fd12b7e4d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I7c9f48888d56b3b234a0a2e96bd90563801c526a
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a host-initialized constant to KVM nVHE hyp code for converting
between EL2 linear map virtual addresses and physical addresses.
Also add `__hyp_pa` macro that performs the conversion.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-18-dbrazdil@google.com
(cherry picked from commit d084ecc5c7)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I2120af630ebfadbbd8bb5733829c09e85eb82485
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of PSCI SMCs in nVHE hyp code. The handler is initialized
with the version used by the host's PSCI driver and the function IDs it
was configured with. If the SMC function ID matches one of the
configured PSCI calls (for v0.1) or falls into the PSCI function ID
range (for v0.2+), the SMC is handled by the PSCI handler. For now, all
SMCs return PSCI_RET_NOT_SUPPORTED.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-17-dbrazdil@google.com
(cherry picked from commit eeeee7193d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I16c0218312af4c2d37be84d4b35548f161f13a14
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add handler of host SMCs in KVM nVHE trap handler. Forward all SMCs to
EL3 and propagate the result back to EL1. This is done in preparation
for validating host SMCs in KVM protected mode.
The implementation assumes that firmware uses SMCCC v1.2 or older. That
means x0-x17 can be used both for arguments and results, other GPRs are
preserved.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-16-dbrazdil@google.com
(cherry picked from commit a805e1fb30)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id25a36b810c4809811f14813b4366e55c3aaed9e
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
When KVM starts validating host's PSCI requests, it will need to map
MPIDR back to the CPU ID. To this end, copy cpu_logical_map into nVHE
hyp memory when KVM is initialized.
Only copy the information for CPUs that are online at the point of KVM
initialization so that KVM rejects CPUs whose features were not checked
against the finalized capabilities.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-15-dbrazdil@google.com
(cherry picked from commit 94f5e8a464)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I43164f3d58d96bc996a536c106e4ff0a2c60c851
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
When compiling with __KVM_NVHE_HYPERVISOR__, redefine per_cpu_offset()
to __hyp_per_cpu_offset() which looks up the base of the nVHE per-CPU
region of the given cpu and computes its offset from the
.hyp.data..percpu section.
This enables use of per_cpu_ptr() helpers in nVHE hyp code. Until now
only this_cpu_ptr() was supported by setting TPIDR_EL2.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-14-dbrazdil@google.com
(cherry picked from commit 687413d34d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I394fce2211bc39b30aa78611243b9944e7b5ddac
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add rules for renaming the .data..ro_after_init ELF section in KVM nVHE
object files to .hyp.data..ro_after_init, linking it into the kernel
and mapping it in hyp at runtime.
The section is RW to the host, then mapped RO in hyp. The expectation is
that the host populates the variables in the section and they are never
changed by hyp afterwards.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-13-dbrazdil@google.com
(cherry picked from commit 2d7bf218ca)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Iaf6d437a0ed50a71dc0669e29f0f86195cdcd7f8
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
MAIR_EL2 and TCR_EL2 are currently initialized from their _EL1 values.
This will not work once KVM starts intercepting PSCI ON/SUSPEND SMCs
and initializing EL2 state before EL1 state.
Obtain the EL1 values during KVM init and store them in the init params
struct. The struct will stay in memory and can be used when booting new
cores.
Take the opportunity to move copying the T0SZ value from idmap_t0sz in
KVM init rather than in .hyp.idmap.text. This avoids the need for the
idmap_t0sz symbol alias.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-12-dbrazdil@google.com
(cherry picked from commit d3e1086c64)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4c69afd11266bf53581cdad3a4112c736babcd4a
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
