Commit Graph

1143022 Commits

Author SHA1 Message Date
David Brazdil
cf3b8fa669 ANDROID: KVM: arm64: Reprogram S2MPUs in 'host_smc_handler'
Intercept SMCs known to be used by the host to inform EL3 about power
events, either powering SoC blocks on or off.

Test: builds, boots
Bug: 190463801
Change-Id: I306433c8c1b712df24569cbd4dc346f72b4c9650
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 8ca0b34fe4)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:10 +00:00
David Brazdil
563bfc1ade ANDROID: KVM: arm64: Enable S2MPUs in __pkvm_init_stage2_iommu
Initialize the S2MPU driver in __pkvm_init_stage2_iommu if requested by
the host. The driver sets kvm_iommu_ops and configures all S2MPUs which
are powered on at that point (ie. all S2MPUs on currently supported
devices).

The S2MPU L1ENTRY registers are set to 1G granularity and R/W access.
CTRL0/CTRL1/CFG as set to reasonable defaults, though the code relies on
the reset state blocking all traffic as well.

On fault the S2MPUs are configured to return SLVERR/DECERR (v8/9) to the
master. Interrupts are enabled for all VIDs and trigger an IRQ handler
if EL1 init registered a handler as a result of a DT interrupts entry.

Because the host can configure the SSMTs freely, all permission bits are
configured for all VIDs. For v9 CONTEXT_CFG_VALID_VIDS is set to the
value precomputed at EL1, allocating a context ID to each VID.

Test: builds, boots
Bug: 190463801
Change-Id: I4a824e90b5d474dd83c97ef53e4df3c8b68da6ba
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 8aa6c440da)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:10 +00:00
David Brazdil
8930f3af83 ANDROID: KVM: arm64: Copy S2MPU configuration to hyp
Create variables in hyp that will hold the DT information about S2MPUs
to use by hyp at runtime. Copy the information from EL1 to EL2.

The EL1 code computes the size of the data and allocates a sufficient
number of pages, which hyp will later take ownership of.

Test: builds, boots
Bug: 190463801
Change-Id: Ic3d4bfa3ec11f7c2e1b4474910e2f57a62139a75
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit bc80f81582)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:09 +00:00
David Brazdil
8877952284 ANDROID: KVM: arm64: Implement IRQ handler for S2MPU faults
The S2MPU can be configured to trigger an interrupt on faults: access
permission (both regular and during page table walks) and if no matching
context ID is found for request's VID (v9 only).

When interrupt information is provided in the S2MPU's DT node, parse the
information and enable an IRQ handler. Later patch will enable the
functionality in the S2MPU.

Test: builds, boots
Bug: 190463801
Change-Id: I11d1a896406011cff1506ee1bd124bfc66ffa914
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 2517c4e5f0)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:09 +00:00
David Brazdil
529308f73d ANDROID: KVM: arm64: Allocate context IDs for valid VIDs
S2MPU_CONTEXT_CFG_VALID_VID register must be configured on v9,
allocating a context ID in range 0 to S2MPU_NUM_CONTEXT to each valid
VID. For now assume that all 8 VIDs are valid. This will change once
the hypervisor takes control over SSMT configuration as well.

If there are more VIDs than available context IDs, the driver prints
a warning that DMA may be blocked and continues.

Test: builds, boots
Bug: 190463801
Change-Id: I0c9e0a5c9470b27debaade2c4e02e16c6577fbfe
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 923353be1e)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:09 +00:00
David Brazdil
ed7f0fcd8f ANDROID: KVM: arm64: Read and check S2MPU_VERSION
Read S2MPU_VERSION during driver init and check it against list of
supported versions. The register fields are as follows:
  - MAJOR_ARCH_VER,
  - MINOR_ARCH_VER,
  - REV_ARCH_VER,
  - RTL_VER.
Their exact use is not documented. For now, we mask out RTL_VER and
expect a match on MAJOR_, MINOR_ and REV_ARCH_VER. This may be tweaked
in the future.

Test: builds, boots
Bug: 190463801
Change-Id: I9709fde5f4d3ca4c23f84919c37b081302846917
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 4a7da93bdb)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:09 +00:00
David Brazdil
3c1759f91d ANDROID: KVM: arm64: Parse S2MPU MMIO region
Start EL1 portion of the S2MPU driver with an init function which
probes the Device tree for nodes compatible with 'google,s2mpu'.
Parse and check the base, size and power domain ID.

Test: builds, boots
Bug: 190463801
Change-Id: I5f0b32febb4e922fdfdfe10a9a9c823e20b8e26f
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 4e91a00153)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:09 +00:00
David Brazdil
c4d2c4f644 ANDROID: KVM: arm64: Create empty S2MPU driver
Create a skeleton driver for the S2MPU - an EL1 portion called during
KVM init which will parse the DT and configure the kernel, and an EL2
portion which will program the S2MPUs later at runtime. The code is
behind CONFIG_KVM_S2MPU.

Test: builds, boots
Bug: 190463801
Change-Id: I58206535f3493e1d989576a9db2112d370a1cb4d
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit b2de5483b7)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:09 +00:00
David Brazdil
2bc6495fed ANDROID: KVM: arm64: Add 'host_stage2_adjust_mmio_range' to kvm_iommu_ops
Add a new kvm_iommu_ops hook to the lower-EL instruction/data abort
handler, which allows the IOMMU driver to restrict the region of device
memory that is about to be mapped in the host stage-2.

This can be used by the IOMMU driver to restrict access to the MMIO
registers of the IOMMU itself.

Test: builds, boots
Bug: 190463801
Change-Id: I51cf3cfd84c889627e290d74579657447964ca16
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit cc1ad46fb2)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:09 +00:00
David Brazdil
59d406c88d ANDROID: KVM: arm64: Add 'host_mmio_dabt_handler' to kvm_iommu_ops
Add a new kvm_iommu_ops hook which allows the IOMMU driver to handle
data aborts in unmapped device memory regions. If the abort is handled
by the driver, the global abort handler will not attempt to map in the
page.

For example, this enables the IOMMU driver to virtualize access to
the underlying IOMMU hardware, or to allow access to a subset of the
functionality, eg. performance counters.

Test: builds, boots
Bug: 190463801
Change-Id: I84adbc992e577ac6ceb09f4856e1c648df580f76
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 25f81ec77b)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:08 +00:00
David Brazdil
6cf8566b50 ANDROID: KVM: arm64: Add 'host_stage2_set_owner' to kvm_iommu_ops
Add a new hook to kvm_iommu_ops that is invoked whenever a range of
pages changes their owner in the host stage2. This is currently limited
to finalize_host_mappings, which changes the owner of EL2-mapped pages
from host to hyp.

The driver is expected to apply corresponding changes in the IOMMU it
controls, so that only the new owner can access the page range.

Test: builds, boots
Bug: 190463801
Change-Id: I0809f4859a9117d1a37506b7aa9e19c6bd25ffdb
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 3cd8b5b00b)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:08 +00:00
David Brazdil
90473cbd4b ANDROID: KVM: arm64: Add 'host_smc_handler' to kvm_iommu_ops
IOMMU drivers need to intercept power management SMCs between the host
and EL3. Add a hook to hyp's 'handle_host_smc'.

Test: builds, boots
Bug: 190463801
Change-Id: Ied34b60d4bb0e5ae0fbf03f8ce1dc22a09679e37
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit d2efcdcb2b)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:08 +00:00
David Brazdil
dd5b25ca1f ANDROID: KVM: arm64: Introduce IOMMU driver infrastructure
Bootstrap infrastructure for IOMMU drivers by introducing kvm_iommu_ops
struct in EL2 that is populated based on a iommu_driver parameter to
__pkvm_init hypercall and selected in EL1 early init.

An 'init' operation is called in __pkvm_init_finalise, giving the driver
an opportunity to initialize itself in EL2 and create any EL2 mappings
that it will need. 'init' is specifically called before
'finalize_host_mappings' so that:
  (a) pages mapped by the driver change owner to hyp,
  (b) ownership changes in 'finalize_host_mappings' get reflected in
      IOMMU mappings (added in a future patch).

Test: builds, boots
Bug: 190463801
Change-Id: I04c9f32c6eda846e6e377cb3d23330eb143b6242
Signed-off-by: David Brazdil <dbrazdil@google.com>
(cherry picked from commit 79775d0225)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:08 +00:00
Will Deacon
a596e6423e ANDROID: KVM: arm64: Update pKVM hyp state series to v6
aosp/2257747 merged v5 of the pKVM hypervisor state series as FROMLIST.
Since then, version 6 was posted and queued by the upstream maintainer:

  https://lore.kernel.org/r/166819337067.3836113.13147674500457473286.b4-ty@kernel.org

Rather than revert v5 from android (and the dozens of dependent patches),
snap to v6 so that we're in-sync with upstream.

Bug: 233587962
[willdeacon@: Fix conflicts with 'stage2_mc' introduced by accounting work]
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I137bbd611c180cbe03e63a55705150f8f9c2ae31
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:08 +00:00
Vincent Donnefort
b97d00514a ANDROID: KVM: arm64: Add protected_shared_mem statistic
When using nVHE in protected mode, protected memory can be between
host and a guest. Tracking this value is interesting from a debug
perspective, to identify potential leaks.

Keeping the count of memory sharing is easy, each share/unshare will return
to the host where the accounting will take place.

Bug: 222044477
Change-Id: I43dcd258789f79dbfe489e5bf721e606c5e6e022
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:08 +00:00
Vincent Donnefort
781b6882ba ANDROID: KVM: arm64: count KVM s2 mmu usage in nVHE protected mode
When using the nVHE protected mode, the stage-2 page tables are handled by
the hypervisor, but are backed by memory donated by the host. That memory
is accounted during the donation (add to the vCPUs hyp_memcache) under
secondary pagetable stats.

On VM teardown, those pages are mixed with others in the teardown_mc, so use
a separated teardown_stage2_mc to deduct them from accounting after
reclaim.

Bug: 222044477
Change-Id: I2a45ce65c5ce9cf96aabd1b66d6f83ffe4808a0c
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:08 +00:00
Vincent Donnefort
b75ae68d19 ANDROID: KVM: arm64: Add protected_hyp_mem VM statistic
When using nVHE in protected mode, the host allocates memory for the
hypervisor to store shadow structures and the stage-2 page tables. This has
been proven to be an interesting value to follow, for debug and health
purpose. Account for those allocations in bytes, in a newly created VM
statistic "protected_hyp_mem".

It is expected, on VM teardown to reclaim all that memory. Raise a warning
if not all the donations are recovered.

Bug: 222044477
Change-Id: I18657d275f2ced67ceb6d0e4bd5ce41cf1d41dc8
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:07 +00:00
Fuad Tabba
059a19c4ef ANDROID: KVM: arm64: Fix sparse __percpu warning
Force the cast to silence the warning.

We don't have a proper way to dynamically allocate memory at EL2,
and hence no proper way to dynamically allocate percpu fields.
Instead, we rely on memory donated from the host and index it by
hyp_smp_processor_id().

Bug: 258616809
Reported-by: Todd Kjos <tkjos@google.com>
Change-Id: I3f7efd4a41294a6696992ce0a49270c4f9468bb5
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:07 +00:00
Will Deacon
8eaec3fe4a ANDROID: KVM: arm64: Relax SMCCC version check during FF-A proxy init
Although FF-A claims to require version v1.2 of SMCCC, in reality the
current set of calls work just fine with v1.1 and some devices ship with
EL3 firmware that advertises this configuration.

Allow pKVM to proxy FF-A calls for these devices by relaxing our SMCCC
version check to permit SMCCC v1.1+

Reported-by: Alan Stokes <alanstokes@google.com>
Bug: 222663556
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I41e9ff35f169df3609acee7bbc67999c1d11c9d1
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:07 +00:00
Quentin Perret
9402f3997c ANDROID: KVM: arm64: Increase size of FF-A buffer
As it turns out, the kernel's DMA code doesn't enforce the
SG_MAX_SEGMENTS limit on the number of elements in an sglist, which can
confuse the pKVM FF-A proxy which has a buffer sized to contain a
descriptor of at most SG_MAX_SEGMENTS constituents.

As the number of elements in an sglist doesn't seem to have an actual
upper bound, let's paper over the issue for now by increasing the size
of the pKVM buffer based on empirical 'measurements'. Longer term we
might need to make this value configurable on the kernel's cmdline, or
to rework the FF-A proxy to sanely handle large descriptors, although
this is not clear how at the time of writing.

Bug: 221256863
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: If252f01bec8ae71c0fe1f7007a3ca7b037924c84
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:07 +00:00
Quentin Perret
bfd7a52151 BACKPORT: FROMLIST: KVM: arm64: pkvm: Add support for fragmented FF-A descriptors
FF-A memory descriptors may need to be sent in fragments when they don't
fit in the mailboxes. Doing so involves using the FRAG_TX and FRAG_RX
primitives defined in the FF-A protocol.

Add support in the pKVM FF-A relayer for fragmented descriptors by
monitoring outgoing FRAG_TX transactions and by buffering large
descriptors on the reclaim path.

[ qperret: BACKPORT because I removed the erroneous ANDROID tag from the
  patch title posted upstream ]

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: I701f279cd4820abb0b6d7c2572ee28e0f943edad
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-13-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:07 +00:00
Will Deacon
cb335ce6d0 FROMLIST: KVM: arm64: Handle FFA_MEM_LEND calls from the host
Handle FFA_MEM_LEND calls from the host by treating them identically to
FFA_MEM_SHARE calls for the purposes of the host stage-2 page-table, but
forwarding on the original request to EL3.

Bug: 254811097
Change-Id: I8f53bca6f0865fabd9938eefd8427fa0e78016ed
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-12-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:07 +00:00
Will Deacon
2e47f0ecdc FROMLIST: KVM: arm64: Handle FFA_MEM_RECLAIM calls from the host
Intecept FFA_MEM_RECLAIM calls from the host and transition the host
stage-2 page-table entries from the SHARED_OWNED state back to the OWNED
state once EL3 has confirmed that the secure mapping has been reclaimed.

Bug: 254811097
Change-Id: I58365e1b3fafa47f290a292fe57f6d2ed7f9091b
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-11-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:07 +00:00
Will Deacon
8453f39a4a FROMLIST: KVM: arm64: Handle FFA_MEM_SHARE calls from the host
Intercept FFA_MEM_SHARE/FFA_FN64_MEM_SHARE calls from the host and
transition the host stage-2 page-table entries from the OWNED state to
the SHARED_OWNED state prior to forwarding the call onto EL3.

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: Ic75a943c67e6cb96794c250dccf2a59362bf857e
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-10-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:06 +00:00
Will Deacon
3399bf007b BACKPORT: FROMLIST: KVM: arm64: Add FF-A helpers to share/unshare memory with secure world
Extend pKVM's memory protection code so that we can update the host's
stage-2 page-table to track pages shared with secure world by the host
using FF-A and prevent those pages from being mapped into a guest.

[ qperret: BACKPORT due to context conflicts in mem_protect.c caused by
  the presense of guest-related memory transition in the android kernel
  (host_donate_guest and friends) ]

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: Ib4d404cd1d4fa11d7bf8c1d0b8ec00838a8038a0
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-9-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:06 +00:00
Will Deacon
e7e5e9120f FROMLIST: KVM: arm64: Handle FFA_RXTX_MAP and FFA_RXTX_UNMAP calls from the host
Handle FFA_RXTX_MAP and FFA_RXTX_UNMAP calls from the host by sharing
the host's mailbox memory with the hypervisor and establishing a
separate pair of mailboxes between the hypervisor and the SPMD at EL3.

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: Ib5fa89e9b01aa20f7c1b5b41df79d66e98d07f55
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-8-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:06 +00:00
Will Deacon
508713769a FROMLIST: KVM: arm64: Allocate pages for hypervisor FF-A mailboxes
The FF-A proxy code needs to allocate its own buffer pair for
communication with EL3 and for forwarding calls from the host at EL1.

Reserve a couple of pages for this purpose and use them to initialise
the hypervisor's FF-A buffer structure.

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: Id72cd7f59be20eb6d1faa6f1c5e64ecc8debf929
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-7-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:06 +00:00
Fuad Tabba
a805114bc5 FROMLIST: KVM: arm64: Handle FFA_FEATURES call from the host
Filter out advertising unsupported features, and only advertise
features and properties that are supported by the hypervisor proxy.

Bug: 254811097
Change-Id: I071766d6d241f4bdd00b8f80e6b237c184a1e59a
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-6-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:06 +00:00
Will Deacon
376236b858 BACKPORT: FROMLIST: KVM: arm64: Probe FF-A version and host/hyp partition ID during init
Probe FF-A during pKVM initialisation so that we can detect any
inconsistencies in the version or partition ID early on.

[ qperret: BACKPORT due to trivial conflict with header includes in
  setup.c ]

Bug: 254811097
Change-Id: I7def4c2c497017ba86621bc98298bc65ffdeefae
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-5-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:06 +00:00
Will Deacon
1e8c7d7216 FROMLIST: KVM: arm64: Block unsafe FF-A calls from the host
When KVM is initialised in protected mode, we must take care to filter
certain FFA calls from the host kernel so that the integrity of guest
and hypervisor memory is maintained and is not made available to the
secure world.

As a first step, intercept and block all memory-related FF-A SMC calls
from the host to EL3. This puts the framework in place for handling them
properly.

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: I5279bce56956c590862a68e8c4803dd2205e3f81
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-4-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:06 +00:00
Will Deacon
dba12d9059 FROMLIST: firmware: arm_ffa: Move comment before the field it is documenting
This is consistent with the other comments in the struct.

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: I10e9014a0d505fe5e132fb1cd6105b95a3f5f2bf
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-3-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:05 +00:00
Will Deacon
40e897b246 FROMLIST: firmware: arm_ffa: Move constants to header file
FF-A function IDs and error codes will be needed in the hypervisor too,
so move to them to the header file where they can be shared. Rename the
version constants with an "FFA_" prefix so that they are less likely
to clash with other code in the tree.

Bug: 254811097
Co-developed-by: Andrew Walbran <qwandor@google.com>
Change-Id: I00ed487279fdfb61ea34ae99140c6fac8ee89187
Signed-off-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20221116170335.2341003-2-qperret@google.com
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:05 +00:00
Quentin Perret
3c8afbbfa4 ANDROID: KVM: arm64: Issue CMOs when tearing down shadow pages
On the guest teardown path, pKVM will zero the pages used to back the
guest shadow data structures before returning them to the host as they
may contain secrets (e.g. in the vCPU registers). However, the zeroing
is done using a cacheable alias, and CMOs are missing, hence giving the
host a potential opportunity to read the original content of the shadow
structs from memory.

Fix this by issuing CMOs after zeroing the pages.

[ qperret@: moved the CMOs to __unmap_donated_memory() to cover all
  callers, including the __pkvm_init_vm() error path ]

Bug: 259551298
Change-Id: Id696d47d16e4c3fd870cb70b792eeb7f2282fc78
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:05 +00:00
Will Deacon
ffe9d28274 ANDROID: KVM: arm64: Use PSCI MEM_PROTECT to zap guest pages on reset
If a malicious/compromised host issues a PSCI SYSTEM_RESET call in the
presence of guest-owned pages then the contents of those pages may be
susceptible to cold-reboot attacks.

Use the PSCI MEM_PROTECT call to ensure that volatile memory is wiped by
the firmware if a SYSTEM_RESET occurs while unpoisoned guest pages exist
in the system. Since this call does not offer protection for a "warm"
reset initiated by SYSTEM_RESET2, detect this case in the PSCI relay and
repaint the call to a standard SYSTEM_RESET instead.

Signed-off-by: Will Deacon <will@kernel.org>
Bug: 254821051
Change-Id: I5c3dd93bc83ebcd0b6cea2ec734f6e3a77f0064e
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:05 +00:00
Will Deacon
22c8a338c7 ANDROID: KVM: arm64: Check IPA range for pvmfw during guest donation
When donating pages to the guest, we only check the first IPA in the
range against the pvmfw loading range. Although this is fine for the
page-at-a-time faulting path, it doesn't fit with the rest of the mem
protection logic, which deals with the possibility of an arbitrarily
sized contiguous address range.

Rework the logic so that we check the whole IPA range during guest
donation and trigger the pvmfw loading path if any of the pages
intersect with the pvmfw region.

Signed-off-by: Will Deacon <will@kernel.org>
Bug: 254819795
Change-Id: I6fef9f1898e65a95cab7f6a0ffa8aa422a8d5a91
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:05 +00:00
Will Deacon
677980a696 ANDROID: KVM: arm64: Use fixmap when poisoning pvmfw pages
When poisoning the pvmfw pages during system reset at EL2, ensure that we
use a writable fixmap mapping rather than the persistent read-only mapping
of the region.

Signed-off-by: Will Deacon <will@kernel.org>
Bug: 254819795
Change-Id: I4c8be092d3c822695afd7d03d0d64163664a9f64
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:05 +00:00
Will Deacon
a542a1202a ANDROID: KVM: arm64: Rename pkvm_clear_pvmfw_pages()
pkvm_clear_pvmfw_pages() is used to poison the pvmfw pages during reset,
so rename it to pkvm_poison_pvmfw_pages() instead.

Signed-off-by: Will Deacon <will@kernel.org>
Bug: 254819795
Change-Id: Ie5b9c90f0707fa81d9099425cff35383bfb0d009
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:05 +00:00
Will Deacon
8b744c24f0 ANDROID: KVM: arm64: Rename hyp_zero_page() and make available as helper
hyp_zero_page() is used for poisoning memory, so rename it to
hyp_poison_page() to avoid confusing with the concept of a "zero page"
and make it available outside of mem_protect.c as it will be used to
poison the pvmfw memory in a subsequent patch.

Signed-off-by: Will Deacon <will@kernel.org>
Bug: 254819795
Change-Id: Ia4aec46437db3ffe466ae09bd180392fa06c0b46
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:04 +00:00
Will Deacon
8f9c7f7acb ANDROID: KVM: arm64: Don't check for hyp_fixmap_map() returning NULL
hyp_fixmap_map() never returns NULL, so remove the redundant checks for
it and simplify the error handling in the callers.

Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 254819795
Change-Id: Ie73a97cc3d9bded3750abe6e243003827393ee5e
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:04 +00:00
Keir Fraser
6fe4c366af ANDROID: virtio_balloon: Do not clear VIRTIO_F_ACCESS_PLATFORM
This essentially reverts commit e41b135550
"virtio_balloon: disable VIOMMU support".

Although the virtio_balloon driver does not translate through a
VIOMMU (or bounce buffer) the pages that it sends to the device,
it *does* need to perform these translations on the virtio rings
themselves.

This fixes virtio_balloon initialisation inside a PKVM/ARM64
protected virtual machine.

Bug: 240239989
Change-Id: I2a84eec870fd638223b231e5c4d1c27216dc40a2
Signed-off-by: Keir Fraser <keirf@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:04 +00:00
Keir Fraser
a12648bd1e ANDROID: virtio_balloon: New module parameter "pkvm"
This specifies that the driver is running on a PKVM hypervisor
and must use the memrelinquish service to cooperatively release
memory. If this service is unavailable, virtio_balloon cannot be
used.

Bug: 240239989
Change-Id: I8800c4435d8fae9df6f1ab108cc61c8f93020773
Signed-off-by: Keir Fraser <keirf@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:04 +00:00
Keir Fraser
e210ad7cd9 ANDROID: KVM: arm64: Introduce kvm_has_memrelinquish_services
Used to determine whether memrelinquish services have been
initialised.

Bug: 240239989
Change-Id: I81dd23d8122ea54924d52b3fdc1fc4a8cdb28ea5
Signed-off-by: Keir Fraser <keirf@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:04 +00:00
Vincent Donnefort
bf76ea0a21 ANDROID: KVM: arm64: Flush nVHE hyp_vcpu memcache
When using nVHE in protected mode, the host donates pages through an arch
specific memcache the hyp can then pours in its local vcpu copy. The latter
should be flushed on VM teardown.

Bug: 237506543
Change-Id: Ic37d794ac33e9f844fa6ae1b4943febcdad5b033
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:04 +00:00
Keir Fraser
4a477ac0a9 ANDROID: KVM: arm64: Avoid unnecessary unmap walk in MEM_RELINQUISH hypercall
If the mapping is determined to be not present in an earlier walk,
attempting the unmap is pointless.

Bug: 259217067
Change-Id: I6fd939556b80d7a9a0731cab36166a652f7a7c6d
Signed-off-by: Keir Fraser <keirf@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:04 +00:00
Keir Fraser
15a4df5ad4 ANDROID: KVM: arm64: Strictly check page type in MEM_RELINQUISH hypercall
The VM should only relinquish "normal" pages. For a protected VM, this
means PAGE_OWNED; For a normal VM, this means PAGE_SHARED_BORROWED. All
other page types are rejected and failure is reported to the caller.

Bug: 259217067
Change-Id: Icff3474dc2c975a6c5befe546c5521a05b3bd575
Signed-off-by: Keir Fraser <keirf@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:03 +00:00
Keir Fraser
cafd6d7893 ANDROID: KVM: Include prototype for page_relinquish before definition
Fixes build failure on -Werror=missing-prototypes.

At the same time, make the header file more resilient to ordering by
declaring 'struct page'.

Bug: 240239989
Change-Id: I84d069bde5ff03d1afa55d25c01448b0d43042da
Signed-off-by: Keir Fraser <keirf@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:03 +00:00
Fuad Tabba
eb0f5f7486 ANDROID: arm64: ioremap/iounmap use stage-2 granule size
When MMIO guard is queried, it advertises the guard granule size
it uses. Use that value.

Fixes: arm64: Implement ioremap/iounmap hooks calling into KVM's MMIO guard

Bug: 251432016
Change-Id: Iff4dcb6229bf89aef832a29a98fecc041a1aec1b
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:03 +00:00
Fuad Tabba
9720e4d374 ANDROID: arm64: Check if pfn is valid for all ioremap loop iterations
The size could cover pfns that are valid.

Fixes:
arm64: Implement ioremap/iounmap hooks calling into KVM's MMIO guard

Bug: 251432016
Change-Id: Ie3b678d40d629ed610b0ee484b5007cb64437435
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:03 +00:00
Sebastian Ene
437fc98ccb ANDROID: arm64: Auto-enroll MMIO guard on protected vms
Set the MMIO guard flag for protected vms prior to entering the guest
for the first time.

Bug: 216798684
Change-Id: I1448102ae85176d495ae7f8d6d20de4092049f0d
Signed-off-by: Sebastian Ene <sebastianene@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:03 +00:00
Marc Zyngier
0dbb5f89f5 ANDROID: KVM: arm64: Add some documentation for the MMIO guard feature
Document the hypercalls user for the MMIO guard infrastructure.

Bug: 209580772
Change-Id: I927bcd6c5e3ef932265d817288ff2b46b0e0db66
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Quentin Perret <qperret@google.com>
2022-12-15 16:13:03 +00:00