Small cleanup moving declarations of hyp-exported variables to
kvm_host.h and using macros to avoid having to refer to them with
kvm_nvhe_sym() in host.
No functional change intended.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-5-dbrazdil@google.com
(cherry picked from commit 61fe0c37af)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I92685a9ce7a24c7d58eda5192c5df56856a525c4
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Computing the hyp VA layout is redundant when the kernel runs in EL2 and
hyp shares its VA mappings. Make calling kvm_compute_layout()
conditional on not just CONFIG_KVM but also !is_kernel_in_hyp_mode().
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-4-dbrazdil@google.com
(cherry picked from commit c3e181aec9)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I8fa9babb88700502afccb87ff66ebf9a0b30fcf2
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
init_hyp_physvirt_offset() computes PA from a kernel VA. Conversion to
kernel linear-map is required first but the code used kvm_ksym_ref() for
this purpose. Under VHE that is a NOP and resulted in a runtime warning.
Replace kvm_ksym_ref with lm_alias.
Reported-by: Qian Cai <qcai@redhat.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-3-dbrazdil@google.com
(cherry picked from commit 7a96a0687b)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I67988a954f3a253e8f97102c8b3e0c081de33c63
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
PSCI driver exposes a struct containing the PSCI v0.1 function IDs
configured in the DT. However, the struct does not convey the
information whether these were set from DT or contain the default value
zero. This could be a problem for PSCI proxy in KVM protected mode.
Extend config passed to KVM with a bit mask with individual bits set
depending on whether the corresponding function pointer in psci_ops is
set, eg. set bit for PSCI_CPU_SUSPEND if psci_ops.cpu_suspend != NULL.
Previously config was split into multiple global variables. Put
everything into a single struct for convenience.
Reported-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201208142452.87237-2-dbrazdil@google.com
(cherry picked from commit ff367fe473)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I9118766dc065d3664a28f022a96082c5335060f7
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Conflict resolution gone astray results in the kernel not booting
on VHE-capable HW when VHE support is disabled. Thankfully spotted
by David.
Reported-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 0cc519f85a)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I98572e477ba1dd1769c3e931563035a92d2a81c5
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
With protected nVHE hyp code interception host's PSCI SMCs, the host
starts seeing new CPUs boot in EL1 instead of EL2. The kernel logic
that keeps track of the boot mode needs to be adjusted.
Add a static key enabled if KVM protected mode initialization is
successful.
When the key is enabled, is_hyp_mode_available continues to report
`true` because its users either treat it as a check whether KVM will be
/ was initialized, or whether stub HVCs can be made (eg. hibernate).
is_hyp_mode_mismatched is changed to report `false` when the key is
enabled. That's because all cores' modes matched at the point of KVM
init and KVM will not allow cores not present at init to boot. That
said, the function is never used after KVM is initialized.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-27-dbrazdil@google.com
(cherry picked from commit f19f6644a5)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5c14d42952672b2756b731663b9406463af2b5dd
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
While protected KVM is installed, start trapping all host SMCs.
For now these are simply forwarded to EL3, except PSCI
CPU_ON/CPU_SUSPEND/SYSTEM_SUSPEND which are intercepted and the
hypervisor installed on newly booted cores.
Create new constant HCR_HOST_NVHE_PROTECTED_FLAGS with the new set of HCR
flags to use while the nVHE vector is installed when the kernel was
booted with the protected flag enabled. Switch back to the default HCR
flags when switching back to the stub vector.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-26-dbrazdil@google.com
(cherry picked from commit b93c17c418)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4ca55f5d2bfaa0259b03f0193ffb8860f0b3c7ca
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KVM by default keeps the stub vector installed and installs the nVHE
vector only briefly for init and later on demand. Change this policy
to install the vector at init and then never uninstall it if the kernel
was given the protected KVM command line parameter.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-25-dbrazdil@google.com
(cherry picked from commit fa8c3d6553)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5cf077021d5135924ff9e9c38706908421603c9f
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of SYSTEM_SUSPEND host PSCI SMCs. The semantics are
equivalent to CPU_SUSPEND, typically called on the last online CPU.
Reuse the same entry point and boot args struct as CPU_SUSPEND.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-24-dbrazdil@google.com
(cherry picked from commit d945f8d9ec)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I81610430eb84ac3d09ac0caa74ff743a97d004fe
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of CPU_SUSPEND host PSCI SMCs. The SMC can either enter
a sleep state indistinguishable from a WFI or a deeper sleep state that
behaves like a CPU_OFF+CPU_ON except that the core is still considered
online while asleep.
The handler saves r0,pc of the host and makes the same call to EL3 with
the hyp CPU entry point. It either returns back to the handler and then
back to the host, or wakes up into the entry point and initializes EL2
state before dropping back to EL1. No EL2 state needs to be
saved/restored for this purpose.
CPU_ON and CPU_SUSPEND are both implemented using struct psci_boot_args
to store the state upon powerup, with each CPU having separate structs
for CPU_ON and CPU_SUSPEND so that CPU_SUSPEND can operate locklessly
and so that a CPU_ON call targeting a CPU cannot interfere with
a concurrent CPU_SUSPEND call on that CPU.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-23-dbrazdil@google.com
(cherry picked from commit abf16336dd)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I94dd625f594a981480741e93408d5605fc1a4194
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of the CPU_ON PSCI call from host. When invoked, it looks
up the logical CPU ID corresponding to the provided MPIDR and populates
the state struct of the target CPU with the provided x0, pc. It then
calls CPU_ON itself, with an entry point in hyp that initializes EL2
state before returning ERET to the provided PC in EL1.
There is a simple atomic lock around the boot args struct. If it is
already locked, CPU_ON will return PENDING_ON error code.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-22-dbrazdil@google.com
(cherry picked from commit cdf3671927)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ib11e8bf3d0a1c4070460e7aa82c9c70481958731
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
All nVHE hyp code is currently executed as handlers of host's HVCs. This
will change as nVHE starts intercepting host's PSCI CPU_ON SMCs. The
newly booted CPU will need to initialize EL2 state and then enter the
host. Add __host_enter function that branches into the existing
host state-restoring code after the trap handler would have returned.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-21-dbrazdil@google.com
(cherry picked from commit 04e05f057a)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ia0b355b82ee01eb6693e2c10327f55a743cc3011
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
In preparation for adding a CPU entry point in nVHE hyp code, extract
most of __do_hyp_init hypervisor initialization code into a common
helper function. This will be invoked by the entry point to install KVM
on the newly booted CPU.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-20-dbrazdil@google.com
(cherry picked from commit f74e1e2128)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I9c295736ba41cb5bfba2760a53bc9b06940e4f87
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Forward the following PSCI SMCs issued by host to EL3 as they do not
require the hypervisor's intervention. This assumes that EL3 correctly
implements the PSCI specification.
Only function IDs implemented in Linux are included.
Where both 32-bit and 64-bit variants exist, it is assumed that the host
will always use the 64-bit variant.
* SMCs that only return information about the system
* PSCI_VERSION - PSCI version implemented by EL3
* PSCI_FEATURES - optional features supported by EL3
* AFFINITY_INFO - power state of core/cluster
* MIGRATE_INFO_TYPE - whether Trusted OS can be migrated
* MIGRATE_INFO_UP_CPU - resident core of Trusted OS
* operations which do not affect the hypervisor
* MIGRATE - migrate Trusted OS to a different core
* SET_SUSPEND_MODE - toggle OS-initiated mode
* system shutdown/reset
* SYSTEM_OFF
* SYSTEM_RESET
* SYSTEM_RESET2
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-19-dbrazdil@google.com
(cherry picked from commit 1fd12b7e4d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I7c9f48888d56b3b234a0a2e96bd90563801c526a
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a host-initialized constant to KVM nVHE hyp code for converting
between EL2 linear map virtual addresses and physical addresses.
Also add `__hyp_pa` macro that performs the conversion.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-18-dbrazdil@google.com
(cherry picked from commit d084ecc5c7)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I2120af630ebfadbbd8bb5733829c09e85eb82485
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add a handler of PSCI SMCs in nVHE hyp code. The handler is initialized
with the version used by the host's PSCI driver and the function IDs it
was configured with. If the SMC function ID matches one of the
configured PSCI calls (for v0.1) or falls into the PSCI function ID
range (for v0.2+), the SMC is handled by the PSCI handler. For now, all
SMCs return PSCI_RET_NOT_SUPPORTED.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-17-dbrazdil@google.com
(cherry picked from commit eeeee7193d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I16c0218312af4c2d37be84d4b35548f161f13a14
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add handler of host SMCs in KVM nVHE trap handler. Forward all SMCs to
EL3 and propagate the result back to EL1. This is done in preparation
for validating host SMCs in KVM protected mode.
The implementation assumes that firmware uses SMCCC v1.2 or older. That
means x0-x17 can be used both for arguments and results, other GPRs are
preserved.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-16-dbrazdil@google.com
(cherry picked from commit a805e1fb30)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id25a36b810c4809811f14813b4366e55c3aaed9e
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
When KVM starts validating host's PSCI requests, it will need to map
MPIDR back to the CPU ID. To this end, copy cpu_logical_map into nVHE
hyp memory when KVM is initialized.
Only copy the information for CPUs that are online at the point of KVM
initialization so that KVM rejects CPUs whose features were not checked
against the finalized capabilities.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-15-dbrazdil@google.com
(cherry picked from commit 94f5e8a464)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I43164f3d58d96bc996a536c106e4ff0a2c60c851
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
When compiling with __KVM_NVHE_HYPERVISOR__, redefine per_cpu_offset()
to __hyp_per_cpu_offset() which looks up the base of the nVHE per-CPU
region of the given cpu and computes its offset from the
.hyp.data..percpu section.
This enables use of per_cpu_ptr() helpers in nVHE hyp code. Until now
only this_cpu_ptr() was supported by setting TPIDR_EL2.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-14-dbrazdil@google.com
(cherry picked from commit 687413d34d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I394fce2211bc39b30aa78611243b9944e7b5ddac
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add rules for renaming the .data..ro_after_init ELF section in KVM nVHE
object files to .hyp.data..ro_after_init, linking it into the kernel
and mapping it in hyp at runtime.
The section is RW to the host, then mapped RO in hyp. The expectation is
that the host populates the variables in the section and they are never
changed by hyp afterwards.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-13-dbrazdil@google.com
(cherry picked from commit 2d7bf218ca)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Iaf6d437a0ed50a71dc0669e29f0f86195cdcd7f8
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
MAIR_EL2 and TCR_EL2 are currently initialized from their _EL1 values.
This will not work once KVM starts intercepting PSCI ON/SUSPEND SMCs
and initializing EL2 state before EL1 state.
Obtain the EL1 values during KVM init and store them in the init params
struct. The struct will stay in memory and can be used when booting new
cores.
Take the opportunity to move copying the T0SZ value from idmap_t0sz in
KVM init rather than in .hyp.idmap.text. This avoids the need for the
idmap_t0sz symbol alias.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-12-dbrazdil@google.com
(cherry picked from commit d3e1086c64)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4c69afd11266bf53581cdad3a4112c736babcd4a
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Once we start initializing KVM on newly booted cores before the rest of
the kernel, parameters to __do_hyp_init will need to be provided by EL2
rather than EL1. At that point it will not be possible to pass its three
arguments directly because PSCI_CPU_ON only supports one context
argument.
Refactor __do_hyp_init to accept its parameters in a struct. This
prepares the code for KVM booting cores as well as removes any limits on
the number of __do_hyp_init arguments.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-11-dbrazdil@google.com
(cherry picked from commit 63fec24351)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Iea8e093168699921bce18cc3a6d85b3ef7879cd8
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KVM precomputes the hyp VA of __kvm_hyp_host_vector, essentially a
constant (minus ASLR), before passing it to __kvm_hyp_init.
Now that we have alternatives for converting kimg VA to hyp VA, replace
this with computing the constant inside __kvm_hyp_init, thus removing
the need for an argument.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-10-dbrazdil@google.com
(cherry picked from commit 5be1d6226d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Iab5e7d288cd6d32ac282d97a674664961caf2aa3
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
When a CPU is booted in EL2, the kernel checks for VHE support and
initializes the CPU core accordingly. For nVHE it also installs the stub
vectors and drops down to EL1.
Once KVM gains the ability to boot cores without going through the
kernel entry point, it will need to initialize the CPU the same way.
Extract the relevant bits of el2_setup into an init_el2_state macro
with an argument specifying whether to initialize for VHE or nVHE.
The following ifdefs are removed:
* CONFIG_ARM_GIC_V3 - always selected on arm64
* CONFIG_COMPAT - hstr_el2 can be set even without 32-bit support
No functional change intended. Size of el2_setup increased by
148 bytes due to duplication.
Signed-off-by: David Brazdil <dbrazdil@google.com>
[maz: reworked to fit the new PSTATE initial setup code]
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-9-dbrazdil@google.com
(cherry picked from commit 78869f0f05)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ifba6a6d458016d2c7e92d7ae6f9d0cbc3a5a7ebf
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
CPU index should never be negative. Change the signature of
(set_)cpu_logical_map to take an unsigned int.
This still works even if the users treat the CPU index as an int,
and will allow the hypervisor's implementation to check that the index
is valid with a single upper-bound check.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-8-dbrazdil@google.com
(cherry picked from commit c1f45f4eb6)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I7d88c41944da6c285fe24ec793a5b65b2f7a006d
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Small refactor that replaces array of v0.1 function IDs indexed by an
enum of function-name constants with a struct of function IDs "indexed"
by field names. This is done in preparation for exposing the IDs to
other parts of the kernel. Exposing a struct avoids the need for
bounds checking.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20201202184122.26046-6-dbrazdil@google.com
(cherry picked from commit 82ac62d165)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I647ba9d5de29a20fea51d57752dbaacc72102690
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Refactor implementation of v0.1+ functions (CPU_SUSPEND, CPU_OFF,
CPU_ON, MIGRATE) to have two functions psci_0_1_foo / psci_0_2_foo that
select the function ID and call a common helper __psci_foo.
This is a small cleanup so that the function ID array is only used for
v0.1 configurations.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20201202184122.26046-5-dbrazdil@google.com
(cherry picked from commit 0bc7474fb7)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I831b00e9c48bcca9a9fff66d613b99229ee91dee
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KVM's host PSCI SMC filter needs to be aware of the PSCI version of the
system but currently it is impossible to distinguish between v0.1 and
PSCI disabled because both have get_version == NULL.
Populate get_version for v0.1 with a function that returns a constant.
psci_opt.get_version is currently unused so this has no effect on
existing functionality.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20201202184122.26046-4-dbrazdil@google.com
(cherry picked from commit e6dd9d89a6)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ied02571cdf87ec7b86dc1e0f6605a136d9fe26bd
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Expose the boolean value whether the system is running with KVM in
protected mode (nVHE + kernel param). CPU capability was selected over
a global variable to allow use in alternatives.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-3-dbrazdil@google.com
(cherry picked from commit 3eb681fba2)
[will: Fix conflict in cpucaps definition, as ARM64_HAS_LDAPR is not present]
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I8bc16ffe7c024855ad1ae282749d660c7d48de90
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add an early parameter that allows users to select the mode of operation
for KVM/arm64.
For now, the only supported value is "protected". By passing this flag
users opt into the hypervisor placing additional restrictions on the
host kernel. These allow the hypervisor to spawn guests whose state is
kept private from the host. Restrictions will include stage-2 address
translation to prevent host from accessing guest memory, filtering its
SMC calls, etc.
Without this parameter, the default behaviour remains selecting VHE/nVHE
based on hardware support and CONFIG_ARM64_VHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-2-dbrazdil@google.com
(cherry picked from commit d8b369c4e3)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I1fe46bb18b40d0a1df41a600a07b848f82988ac6
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
As with SCTLR_ELx and other control registers, some PSTATE bits are
UNKNOWN out-of-reset, and we may not be able to rely on hardware or
firmware to initialize them to our liking prior to entry to the kernel,
e.g. in the primary/secondary boot paths and return from idle/suspend.
It would be more robust (and easier to reason about) if we consistently
initialized PSTATE to a default value, as we do with control registers.
This will ensure that the kernel is not adversely affected by bits it is
not aware of, e.g. when support for a feature such as PAN/UAO is
disabled.
This patch ensures that PSTATE is consistently initialized at boot time
via an ERET. This is not intended to relax the existing requirements
(e.g. DAIF bits must still be set prior to entering the kernel). For
features detected dynamically (which may require system-wide support),
it is still necessary to subsequently modify PSTATE.
As ERET is not always a Context Synchronization Event, an ISB is placed
before each exception return to ensure updates to control registers have
taken effect. This handles the kernel being entered with SCTLR_ELx.EOS
clear (or any future control bits being in an UNKNOWN state).
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-6-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit d87a8e65b5)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ib189dca3244847e2f500fb7b4ca4ee1643480765
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Let's make SCTLR_ELx initialization a bit clearer by using meaningful
names for the initialization values, following the same scheme for
SCTLR_EL1 and SCTLR_EL2.
These definitions will be used more widely in subsequent patches.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-5-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 2ffac9e3fd)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Icc3118148e8b0bb93ce221e64d5aaa95f6f76195
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
For a while now el2_setup has performed some basic initialization of EL1
even when the kernel is booted at EL1, so the name is a little
misleading. Further, some comments are stale as with VHE it doesn't drop
the CPU to EL1.
To clarify things, rename el2_setup to init_kernel_el, and update
comments to be clearer as to the function's purpose.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-4-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit ecbb11ab3e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Idcff6f2e73aafe79a8e3605463639326dd313848
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
To make callsites easier to read, add trivial C wrappers for the
SET_PSTATE_*() helpers, and convert trivial uses over to these. The new
wrappers will be used further in subsequent patches.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-3-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 515d5c8a13)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I0b45ca0bab8610eaf9b649d04fdb101e05ce3ada
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
For consistency, all tasks have a pt_regs reserved at the highest
portion of their task stack. Among other things, this ensures that a
task's SP is always pointing within its stack rather than pointing
immediately past the end.
While it is never legitimate to ERET from a kthread, we take pains to
initialize pt_regs for kthreads as if this were legitimate. As this is
never legitimate, the effects of an erroneous return are rarely tested.
Let's simplify things by initializing a kthread's pt_regs such that an
ERET is caught as an illegal exception return, and removing the explicit
initialization of other exception context. Note that as
spectre_v4_enable_task_mitigation() only manipulates the PSTATE within
the unused regs this is safe to remove.
As user tasks will have their exception context initialized via
start_thread() or start_compat_thread(), this should only impact cases
where something has gone very wrong and we'd like that to be clearly
indicated.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-2-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit f80d034086)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id6fca836e7830efb4df34350c01d44e7317b2dbd
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Cores that predate the introduction of ID_AA64PFR0_EL1.CSV3 to
the ARMv8 architecture have this field set to 0, even of some of
them are not affected by the vulnerability.
The kernel maintains a list of unaffected cores (A53, A55 and a few
others) so that it doesn't impose an expensive mitigation uncessarily.
As we do for CSV2, let's expose the CSV3 property to guests that run
on HW that is effectively not vulnerable. This can be reset to zero
by writing to the ID register from userspace, ensuring that VMs can
be migrated despite the new property being set.
Reported-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 4f1df628d4)
[will: Fix conflict with asymmetric 32-bit code in read_id_reg()]
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ifed7427d58bd57d98dce35d0abacdd2c25b98911
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
In order to reduce the impact of the VPT parsing happening on the GIC,
we can split the vcpu reseidency in two phases:
- programming GICR_VPENDBASER: this still happens in vcpu_load()
- checking for the VPT parsing to be complete: this can happen
on vcpu entry (in kvm_vgic_flush_hwstate())
This allows the GIC and the CPU to work in parallel, rewmoving some
of the entry overhead.
Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Shenming Lu <lushenming@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201128141857.983-3-lushenming@huawei.com
(cherry picked from commit 57e3cebd02)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5d3b5db2db817e99780e009caf8efe0f6692e0b1
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Our Meltdown mitigation state isn't exposed outside of cpufeature.c,
contrary to the rest of the Spectre mitigation state. As we are going
to use it in KVM, expose a arm64_get_meltdown_state() helper which
returns the same possible values as arm64_get_spectre_v?_state().
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 7f43c2014f)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: If31ecb6978a131696dd1cb1aaf8dfbb4b2cf93f9
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
DEMUX register presence depends on the host's hardware (the
CLIDR_EL1 register to be precise). This means there's no set
of them that we can bless and that it's possible to encounter
new ones when running on different hardware (which would
generate "Consider adding them ..." messages, but we'll never
want to add them.)
Remove the ones we have in the blessed list and filter them
out of the new list, but also provide a new command line switch
to list them if one so desires.
Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201126134641.35231-3-drjones@redhat.com
(cherry picked from commit c6232bd40b)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4b4a088b986837f7ef50451d616e6090a07e5032
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Not counting TnD, which KVM doesn't currently consider, CSSELR_EL1
can have a maximum value of 0b1101 (13), which corresponds to an
instruction cache at level 7. With CSSELR_MAX set to 12 we can
only select up to cache level 6. Change it to 14.
Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201126134641.35231-2-drjones@redhat.com
(cherry picked from commit c73a441617)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I3030c34b51fc7e596d723fe071459a3719924455
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
kvm_arch_vm_ioctl_check_extension() is only called from
kvm_vm_ioctl_check_extension(), so we can inline it and remove the extra
function.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201118194402.2892-3-will@kernel.org
(cherry picked from commit 36fb4cd55f)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I41b82f4316871a4366d568eb9ec0a8fd4de6615f
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
'struct kvm_arch_memory_slot' isn't part of the user ABI, so move it out
of the uapi/ headers in case we start using it in future and accidentally
back ourselves into a corner.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201118194402.2892-2-will@kernel.org
(cherry picked from commit 8d14797b53)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id24327a02b210ed8efa65086bd9173558111e3a1
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The PMU ready state has no user left. Goodbye.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 7521c3a9e6)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ic2ac0745d8e41aa439f560e5b385df8493f34620
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
We currently gate the update of the PMU state on the PMU being "ready".
The "ready" state is only set to true when the first vcpu run is
successful, and if it isn't, we never reach the update code.
So the "ready" state is never the right thing to check for, and it
should instead be the presence of the PMU feature, which makes
a bit more sense.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 46acf89de4)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: If51c248249d72b1256339c864a3eff0b1681e9f2
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The handling of traps in access_pmu_evcntr() has a couple of
omminous "else return false;" statements that don't make any sense:
the decoding tree coverse all the registers that trap to this handler,
and returning false implies that we change PC, which we don't.
Get rid of what is evidently dead code.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit a3da935802)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I70e709ef540eb13ff7b3f79d2ab507619c0797d4
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
