In preparation for using some of the pKVM fixed configuration register
definitions to filter the available VM CAPs in the host, split the
nvhe/fixed_config.h header so that the definitions can be shared
with the host, while keeping the hypervisor function prototypes in
the nvhe/ namespace.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I33894868e7652f7b79caa91a007dccad997ef4ab
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for supporting protected guests, where guest memory
defaults to being inaccessible to the host, extend our memory protection
mechanisms to support donation of pages from the host to a specific
guest.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ic397b6fd0f7b5f0911ddd8f457e40c6e6689673c
Signed-off-by: Quentin Perret <qperret@google.com>
Now that EL2 is able to manage guest stage-2 page-tables, avoid
allocating a separate MMU structure in the host and instead introduce a
new fault handler which responds to guest stage-2 faults by sharing
GUP-pinned pages with the guest via a hypercall. These pages are
recovered (and unpinned) on guest teardown via the page reclaim
hypercall.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ibbddc97cee322bf2db258b4f0848733e2efb1126
Signed-off-by: Quentin Perret <qperret@google.com>
The current implementation of pKVM doesn't support dirty logging or
read-only memslots. Although support for these features is desirable,
this will require future work, so let's cleanly report the limitations
to userspace by failing the ioctls until then.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ifc434c234ac58b46a244fdd44114bc9a51f53e19
Signed-off-by: Quentin Perret <qperret@google.com>
As the guest stage-2 page-tables will soon be managed entirely by EL2
when pKVM is enabled, guest memory will be pinned and the MMU notifiers
in the host will be unable to reconfigure mappings at EL2 other than
destrroying the guest and reclaiming all of the memory.
Forbid memslot move/delete operations for VMs that have run under pKVM,
returning -EPERM to userspace if such an operation is requested.
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I73650c1ac79d8c116a3f31d17ef2a4ef1b30a844
Signed-off-by: Quentin Perret <qperret@google.com>
Don't blindly assume that the PTE is valid when checking whether
it describes an executable or cacheable mapping.
This makes sure that we don't issue CMOs for invalid mappings.
Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I6cbcdb97033ec7b2ed2c9dce0cfc91491e573908
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for handling guest stage-2 mappings at EL2, extend our
memory protection mechanisms to support sharing of pages from the host
to a specific guest.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I8e1d7cf4db70ad55a29d935f60e6335fc83490eb
Signed-off-by: Quentin Perret <qperret@google.com>
Implement a new hypercall, __pkvm_host_reclaim_page(), so that the host
at EL1 can reclaim pages that were previously donated to EL2. This
allows EL2 to defer clearing of guest memory on teardown and allows
preemption in the host after reclaiming each page.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ifbeafc5ed3e930307f9a9ae04d05ee06cb4451ac
Signed-off-by: Quentin Perret <qperret@google.com>
In order to deal with PC updates (such as INCREMENT_PC and the
collection of flags that come with PENDING_EXCEPTION), add a single
mask that covers them all.
This will be used to manipulate these flags as a single entity.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Id24f79f482911efe3374abbead8a70e46cf12725
Signed-off-by: Quentin Perret <qperret@google.com>
Contrary to vanilla KVM, pKVM not only deals with flags in a vcpu,
but also synchronises them across host and hypervisor views of the same
vcpu.
Most of the time, this is about copying flags from one vcpu structure
to another, so let's offer a primitive that does this.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Icd67b617c1cd69706ccd99739756458864b422bb
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for poisoning guest memory pages in the pKVM hypervisor
when being reclaimed by the host, introduce a new 'flags' field in
'struct hyp_page' so that we will be able to track on a per-page basis
whether or not poisoning is required.
Rather than increase the total size of the structure, shrink the 16-bit
'order' field to a single byte and use the recovered space for the new
field.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I8eb1f7ed8da0374b878a315eb1a6f867d0f379a9
Signed-off-by: Quentin Perret <qperret@google.com>
As a stepping stone towards deprivileging the host's access to the
guest's vCPU structures, introduce some naive flush/sync routines to
copy most of the host vCPU into the hyp vCPU on vCPU run and back
again on return to EL1.
This allows us to run using the pKVM hyp structures when KVM is
initialised in protected mode.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Co-developed-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-26-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Iaf1c07cbf58eaff8a2968e9dc6457d36dcef83cf
Signed-off-by: Quentin Perret <qperret@google.com>
The pkvm hypervisor at EL2 may need to read the 'kvm_vgic_global_state'
variable from the host, for example when saving and restoring the state
of the virtual GIC.
Explicitly map 'kvm_vgic_global_state' in the stage-1 page-table of the
pKVM hypervisor rather than relying on mapping all of the host '.rodata'
section.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-24-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I13a190d4164a4e0fd68dd5ec88ab5647dd4e73fc
Signed-off-by: Quentin Perret <qperret@google.com>
Sharing 'kvm_arm_vmid_bits' between EL1 and EL2 allows the host to
modify the variable arbitrarily, potentially leading to all sorts of
shenanians as this is used to configure the VTTBR register for the
guest stage-2.
In preparation for unmapping host sections entirely from EL2, maintain
a copy of 'kvm_arm_vmid_bits' in the pKVM hypervisor and initialise it
from the host value while it is still trusted.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-23-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I613e7c0ef747324e73cf4d4f543354a3641c7505
Signed-off-by: Quentin Perret <qperret@google.com>
When pKVM is enabled, the hypervisor at EL2 does not trust the host at
EL1 and must therefore prevent it from having unrestricted access to
internal hypervisor state.
The 'kvm_arm_hyp_percpu_base' array holds the offsets for hypervisor
per-cpu allocations, so move this this into the nVHE code where it
cannot be modified by the untrusted host at EL1.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-22-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I8d67b2905ac97e15f4252d45c36b97e53d3072ce
Signed-off-by: Quentin Perret <qperret@google.com>
Rather than relying on the host to free the previously-donated pKVM
hypervisor VM pages explicitly on teardown, introduce a dedicated
teardown memcache which allows the host to reclaim guest memory
resources without having to keep track of all of the allocations made by
the pKVM hypervisor at EL2.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Co-developed-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-21-will@kernel.org
[willdeacon@: Fix GCC compat error due to variable declaration in for loop
initializer prior to C99]
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ib43b68f36fdaf5aac578f177ab8260c72acc6ed5
Signed-off-by: Quentin Perret <qperret@google.com>
Extend the initialisation of guest data structures within the pKVM
hypervisor at EL2 so that we instantiate a memory pool and a full
'struct kvm_s2_mmu' structure for each VM, with a stage-2 page-table
entirely independent from the one managed by the host at EL1.
The 'struct kvm_pgtable_mm_ops' used by the page-table code is populated
with a set of callbacks that can manage guest pages in the hypervisor
without any direct intervention from the host, allocating page-table
pages from the provided pool and returning these to the host on VM
teardown. To keep things simple, the stage-2 MMU for the guest is
configured identically to the host stage-2 in the VTCR register and so
the IPA size of the guest must match the PA size of the host.
For now, the new page-table is unused as there is no way for the host
to map anything into it. Yet.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-20-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I2fc61f4f16d16b4786a50f4e2e4c5b06ed357b22
Signed-off-by: Quentin Perret <qperret@google.com>
The initialisation of guest stage-2 page-tables is currently split
across two functions: kvm_init_stage2_mmu() and kvm_arm_setup_stage2().
That is presumably for historical reasons as kvm_arm_setup_stage2()
originates from the (now defunct) KVM port for 32-bit Arm.
Simplify this code path by merging both functions into one, taking care
to map the 'struct kvm' into the hypervisor stage-1 early on in order to
simplify the failure path.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Co-developed-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-19-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I47433ed294fb5329b17b5e8b290e41b8e6d5b4a9
Signed-off-by: Quentin Perret <qperret@google.com>
The host at EL1 and the pKVM hypervisor at EL2 will soon need to
exchange memory pages dynamically for creating and destroying VM state.
Indeed, the hypervisor will rely on the host to donate memory pages it
can use to create guest stage-2 page-tables and to store VM and vCPU
metadata. In order to ease this process, introduce a
'struct hyp_memcache' which is essentially a linked list of available
pages, indexed by physical addresses so that it can be passed
meaningfully between the different virtual address spaces configured at
EL1 and EL2.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-18-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I69996a4018e61473f25c5079f9ebcd82754914f4
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for handling cache maintenance of guest pages from within
the pKVM hypervisor at EL2, introduce an EL2 copy of icache_inval_pou()
which will later be plumbed into the stage-2 page-table cache
maintenance callbacks, ensuring that the initial contents of pages
mapped as executable into the guest stage-2 page-table is visible to the
instruction fetcher.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-17-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I0a1fb40d49bbc19eb53d59805acb93ebd7702b8b
Signed-off-by: Quentin Perret <qperret@google.com>
The nVHE object at EL2 maintains its own copies of some host variables
so that, when pKVM is enabled, the host cannot directly modify the
hypervisor state. When running in normal nVHE mode, however, these
variables are still mirrored at EL2 but are not initialised.
Initialise the hypervisor symbols from the host copies regardless of
pKVM, ensuring that any reference to this data at EL2 with normal nVHE
will return a sensibly initialised value.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-16-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I1e490d403e85ae5dbc984c1f0b8447c93cdf8700
Signed-off-by: Quentin Perret <qperret@google.com>
Mapping pages in a guest page-table from within the pKVM hypervisor at
EL2 may require cache maintenance to ensure that the initialised page
contents is visible even to non-cacheable (e.g. MMU-off) accesses from
the guest.
In preparation for performing this maintenance at EL2, introduce a
per-vCPU fixmap which allows the pKVM hypervisor to map guest pages
temporarily into its stage-1 page-table for the purposes of cache
maintenance and, in future, poisoning on the reclaim path. The use of a
fixmap avoids the need for memory allocation or locking on the map()
path.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Co-developed-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-15-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I8054dac551f4fd5c4afb5b497816589f1fae5285
Signed-off-by: Quentin Perret <qperret@google.com>
With the pKVM hypervisor at EL2 now offering hypercalls to the host for
creating and destroying VM and vCPU structures, plumb these in to the
existing arm64 KVM backend to ensure that the hypervisor data structures
are allocated and initialised on first vCPU run for a pKVM guest.
In the host, 'struct kvm_protected_vm' is introduced to hold the handle
of the pKVM VM instance as well as to track references to the memory
donated to the hypervisor so that it can be freed back to the host
allocator following VM teardown. The stage-2 page-table, hypervisor VM
and vCPU structures are allocated separately so as to avoid the need for
a large physically-contiguous allocation in the host at run-time.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-14-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I2823f936b69f756d867bf584c645b29357c6abe9
Signed-off-by: Quentin Perret <qperret@google.com>
Introduce a global table (and lock) to track pKVM instances at EL2, and
provide hypercalls that can be used by the untrusted host to create and
destroy pKVM VMs and their vCPUs. pKVM VM/vCPU state is directly
accessible only by the trusted hypervisor (EL2).
Each pKVM VM is directly associated with an untrusted host KVM instance,
and is referenced by the host using an opaque handle. Future patches
will provide hypercalls to allow the host to initialize/set/get pKVM
VM/vCPU state using the opaque handle.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Co-developed-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-13-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ibe946e86f9a773b6024bf74faeccbb6929548a1f
Signed-off-by: Quentin Perret <qperret@google.com>
In preparation for introducing VM and vCPU state at EL2, rename the
existing 'struct host_kvm' and its singleton 'host_kvm' instance to
'host_mmu' so as to avoid confusion between the structure tracking the
host stage-2 MMU state and the host instance of a 'struct kvm' for a
protected guest.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-12-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Iaf93fa2f37f1e87e56182b8494be808ce0dfcf95
Signed-off-by: Quentin Perret <qperret@google.com>
nvhe/mem_protect.h refers to __load_stage2() in the definition of
__load_host_stage2() but doesn't include the relevant header.
Include asm/kvm_mmu.h in nvhe/mem_protect.h so that users of the latter
don't have to do this themselves.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-10-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I9f1eea7b57d6ca5cc5ac172ce5175272f166164a
Signed-off-by: Quentin Perret <qperret@google.com>
Add helpers allowing the hypervisor to check whether a range of pages
are currently shared by the host, and 'pin' them if so by blocking host
unshare operations until the memory has been unpinned.
This will allow the hypervisor to take references on host-provided
data-structures (e.g. 'struct kvm') with the guarantee that these pages
will remain in a stable state until the hypervisor decides to release
them, for example during guest teardown.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-9-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I2918af9b4986b3d5d19ed1c6168b20b59b6a563e
Signed-off-by: Quentin Perret <qperret@google.com>
Memory regions marked as "no-map" in the host device-tree routinely
include TrustZone carev-outs and DMA pools. Although donating such pages
to the hypervisor may not breach confidentiality, it could be used to
corrupt its state in uncontrollable ways. To prevent this, let's block
host-initiated memory transitions targeting "no-map" pages altogether in
nVHE protected mode as there should be no valid reason to do this in
current operation.
Thankfully, the pKVM EL2 hypervisor has a full copy of the host's list
of memblock regions, so we can easily check for the presence of the
MEMBLOCK_NOMAP flag on a region containing pages being donated from the
host.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-8-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I21198f93a6d9c727b70c504ddd31345329eabb8f
Signed-off-by: Quentin Perret <qperret@google.com>
Transferring ownership information of a memory region from one component
to another can be achieved using a "donate" operation, which results
in the previous owner losing access to the underlying pages entirely
and the new owner having exclusive access to the page.
Implement a do_donate() helper, along the same lines as do_{un,}share,
and provide this functionality for the host-{to,from}-hyp cases as this
will later be used to donate/reclaim memory pages to store VM metadata
at EL2.
In a similar manner to the sharing transitions, permission checks are
performed by the hypervisor to ensure that the component initiating the
transition really is the owner of the page and also that the completer
does not currently have a page mapped at the target address.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Co-developed-by: Quentin Perret <qperret@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-7-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I19c349d40f63c1bd5eae0baf9acd21697e1e8762
Signed-off-by: Quentin Perret <qperret@google.com>
The 'pkvm_component_id' enum type provides constants to refer to the
host and the hypervisor, yet this information is duplicated by the
'pkvm_hyp_id' constant.
Remove the definition of 'pkvm_hyp_id' and move the 'pkvm_component_id'
type definition to 'mem_protect.h' so that it can be used outside of
the memory protection code, for example when initialising the owner for
hypervisor-owned pages.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-6-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I78454dc315a0993808a741ebd7fabf9dbcaf09bc
Signed-off-by: Quentin Perret <qperret@google.com>
In order to allow unmapping arbitrary memory pages from the hypervisor
stage-1 page-table, fix-up the initial refcount for pages that have been
mapped before the 'vmemmap' array was up and running so that it
accurately accounts for all existing hypervisor mappings.
This is achieved by traversing the entire hypervisor stage-1 page-table
during initialisation of EL2 and updating the corresponding
'struct hyp_page' for each valid mapping.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-5-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: I10df07bab97198ac4fd9477091b2e9340cc441b0
Signed-off-by: Quentin Perret <qperret@google.com>
The EL2 'vmemmap' array in nVHE Protected mode is currently very sparse:
only memory pages owned by the hypervisor itself have a matching 'struct
hyp_page'. However, as the size of this struct has been reduced
significantly since its introduction, it appears that we can now afford
to back the vmemmap for all of memory.
Having an easily accessible 'struct hyp_page' for every physical page in
memory provides the hypervisor with a simple mechanism to store metadata
(e.g. a refcount) that wouldn't otherwise fit in the very limited number
of software bits available in the host stage-2 page-table entries. This
will be used in subsequent patches when pinning host memory pages for
use by the hypervisor at EL2.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-4-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ifced0fb12faf2af6b757d68e18ba7cb74fdd0871
Signed-off-by: Quentin Perret <qperret@google.com>
All the contiguous pages used to initialize a 'struct hyp_pool' are
considered coalescable, which means that the hyp page allocator will
actively try to merge them with their buddies on the hyp_put_page() path.
However, using hyp_put_page() on a page that is not part of the inital
memory range given to a hyp_pool() is currently unsupported.
In order to allow dynamically extending hyp pools at run-time, add a
check to __hyp_attach_page() to allow inserting 'external' pages into
the free-list of order 0. This will be necessary to allow lazy donation
of pages from the host to the hypervisor when allocating guest stage-2
page-table pages at EL2.
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20221020133827.5541-3-will@kernel.org
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 233587962
Change-Id: Ie417f839ae6d30705847ab3e27e2b3c3ac6ee8dc
Signed-off-by: Quentin Perret <qperret@google.com>
On initialising the MMIO guard infrastructure, register the
earlycon mapping if present.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Bug: 233587962
Change-Id: I379387253d08e2414fa386a3360a45391da7d90d
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
In order to transfer the early mapping state into KVM's MMIO
guard infrastructure, provide a small helper that will retrieve
the associated PTE.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Bug: 233587962
Change-Id: Iefc1c57d5e9476b718a8a68f60e562a57b09fb6a
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Should a guest desire to enroll into the MMIO guard, allow it to
do so with a command-line option.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Bug: 233587962
Change-Id: Ia9a77f693531740500739693c52b4959abacafd4
[willdeacon@: Add hypercall IDs]
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Add a pair of hooks (ioremap_phys_range_hook/iounmap_phys_range_hook)
that can be implemented by an architecture. Contrary to the existing
arch_sync_kernel_mappings(), this one tracks things at the physical
address level.
This is specially useful in these virtualised environments where
the guest has to tell the host whether (and how) it intends to use
a MMIO device.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Bug: 233587962
Change-Id: I970c2e632cb2b01060d5e66e4194fa9248188f43
Signed-off-by: Will Deacon <willdeacon@google.com>
[ qperret: Fixed conflict in vmalloc.c due to call to
kmsan_ioremap_page_range ]
Signed-off-by: Quentin Perret <qperret@google.com>
When running as a protected guest, the KVM host does not have access to
any pages mapped into the guest. Consequently, KVM exposes hypercalls to
the guest so that pages can be shared back with the host for the purposes
of shared memory communication such as virtio.
Detect the presence of these hypercalls when running as a guest and use
them to implement the memory encryption interfaces gated by
CONFIG_ARCH_HAS_MEM_ENCRYPT which are called from the DMA layer to share
SWIOTLB bounce buffers for virtio.
Although no encryption is actually performed, "sharing" a page is akin
to decryption, whereas "unsharing" a page maps to encryption, albeit
without destruction of the underlying page contents.
Signed-off-by: Will Deacon <will@kernel.org>
[willdeacon@: Use asm/mem_encrypt.h instead of asm/set_memory.h;
Implement mem_encrypt_active(); Add hypercall IDs;
Drop unneeded GIC change]
[qperret@: Export set_memory_{en,de}crypted() to fix allmodconfig
modpost failures]
Bug: 233587962
Change-Id: I5955ff0dca65561183f9a60e94be87f28fbf14ec
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
In order to prepare the ground for providing set_mem_{en,de}crypted()
for arm64, make sure to include the right header to avoid allmodconfig
build failures.
Bug: 233587962
Change-Id: I6fb5a9e469a6fe8496dff81a8cda50b6c6c1ed7f
Signed-off-by: Quentin Perret <qperret@google.com>
Pull iommu fix from Joerg Roedel:
- Fix device mask to catch all affected devices in the recently added
quirk for QAT devices in the Intel VT-d driver.
* tag 'iommu-fix-v6.1-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu:
iommu/vt-d: Fix buggy QAT device mask
Pull misc fixes from Andrew Morton:
"Nine hotfixes.
Six for MM, three for other areas. Four of these patches address
post-6.0 issues"
* tag 'mm-hotfixes-stable-2022-12-10-1' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
memcg: fix possible use-after-free in memcg_write_event_control()
MAINTAINERS: update Muchun Song's email
mm/gup: fix gup_pud_range() for dax
mmap: fix do_brk_flags() modifying obviously incorrect VMAs
mm/swap: fix SWP_PFN_BITS with CONFIG_PHYS_ADDR_T_64BIT on 32bit
tmpfs: fix data loss from failed fallocate
kselftests: cgroup: update kmem test precision tolerance
mm: do not BUG_ON missing brk mapping, because userspace can unmap it
mailmap: update Matti Vaittinen's email address
Pull ARM fix from Russell King:
"One further ARM fix for 6.1 from Wang Kefeng, fixing up the handling
for kfence faults"
* tag 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm:
ARM: 9278/1: kfence: only handle translation faults
