A vendor hook is added in post_init_entity_util_avg before
a new cfs task's util is attached to cfs_rq's util so that
vendors can gather and modify se's information to modify
scheduling behavior and DVFS as they want.
trace_android_rvh_new_task_stats is not a proper hook because
it is called after the task's util is attached to cfs_rq's util,
which means updating cfs_rq's sched_avg and DVFS request are done.
Bug: 184219858
Signed-off-by: Choonghoon Park <choong.park@samsung.com>
Change-Id: I2deaa93297f8464895978496c9838cdffaa35b7f
Vendor scheduler may not want to rebalance between domains in some case.
New vendor hook is added for vendor to decide whether rebalance domains.
Bug: 176952463
Change-Id: Ie2edbd4b11679786096dd0170496904ae180583c
Signed-off-by: Park Bumgyu <bumgyu.park@samsung.com>
Device kernels merge gki_defconfig for modules build;
forcing them to sign the modules with no certificates
from the gki build available causing breakage.
GKI module signing support will be enabled via config
fragment; so partner builds don't have to disable support
in their fragments.
Fixes: 9ab6a24225 ("ANDROID: GKI: Add module load time protected symbol lookup")
Bug: 200082547
Test: TreeHugger, local builds
Signed-off-by: Ramji Jiyani <ramjiyani@google.com>
Change-Id: Ia90f0acd57ba9a8795da3f85c2cbe21758f5f4fb
Signed-off-by: Will McVicker <willmcvicker@google.com>
Fix the following warning issued when CONFIG_PERF_EVENTS is not
defined:
kernel/sysctl.c:124:12: error: ‘one_thousand’ defined but not used [-Werror=unused-variable]
These definitions in upstream has been changed [1] and therefore
the issue does not exist there.
[1] https://lore.kernel.org/all/20211124220801.ip01WsWPQ%25akpm@linux-foundation.org/
Fixes: 0f047989b0 ("FROMGIT: mm/pagealloc: sysctl: change watermark_scale_factor max limit to 30%")
Bug: 194652782
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I5539a2d0d27a126f7405455a8cf08c23b80d2e0b
This patch turns on CONFIG_DMABUF_SYSFS_STATS to enable the DMA-BUF
sysfs statistics.
Bug: 167709539
Change-Id: Idc4cb231edfedcdf672474119238e5d7e545002d
Signed-off-by: Hridya Valsaraju <hridya@google.com>
kvm_arm_init_arch_resources() was renamed to kvm_arm_init_sve() in
commit a3be836df7 ("KVM: arm/arm64: Demote
kvm_arm_init_arch_resources() to just set up SVE"). Fix the function
name in comment of kvm_vcpu_finalize_sve().
Signed-off-by: Zenghui Yu <yuzenghui@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211230141535.1389-1-yuzenghui@huawei.com
(cherry picked from commit e938eddbeb
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209777660
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ia4dd418253ee40891559a61430583c3c3188f47a
Ganapatrao reported that the kvm_pgtable->mmu pointer is more or
less hardcoded to the main S2 mmu structure, while the nested
code needs it to point to other instances (as we have one instance
per nested context).
Rework the initialisation of the kvm_pgtable structure so that
this assumtion doesn't hold true anymore. This requires some
minor changes to the order in which things are initialised
(the mmu->arch pointer being the critical one).
Reported-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com>
Reviewed-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211129200150.351436-5-maz@kernel.org
(cherry picked from commit 9d8604b285
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209777660
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: If942bf4a01a975a60c37ad8483eafc52185234d5
Running the KVM selftests results in these messages being dumped
in the kernel console:
[ 188.051073] kvm [469]: VGIC redist and dist frames overlap
[ 188.056820] kvm [469]: VGIC redist and dist frames overlap
[ 188.076199] kvm [469]: VGIC redist and dist frames overlap
Being amle to trigger this from userspace is definitely not on,
so demote these warnings to kvm_debug().
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211216104507.1482017-1-maz@kernel.org
(cherry picked from commit 440523b92b
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209777660
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I0c6c94795a86705925dc1c9519bb0dec831b641c
When handling an error at the point where we try and register
all the redistributors, we unregister all the previously
registered frames by counting down from the failing index.
However, the way the code is written relies on that index
being a signed value. Which won't be true once we switch to
an xarray-based vcpu set.
Since this code is pretty awkward the first place, and that the
failure mode is hard to spot, rewrite this loop to iterate
over the vcpus upwards rather than downwards.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211216104526.1482124-1-maz@kernel.org
(cherry picked from commit c95b1d7ca7
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209777660
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I9df3874b4a6542e5d20125d436108d8c1235b8e5
workaround_flags is a leftover from our earlier Spectre-v4 workaround
implementation, and now serves no purpose.
Get rid of the field and the corresponding asm-offset definition.
Fixes: 29e8910a56 ("KVM: arm64: Simplify handling of ARCH_WORKAROUND_2")
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 142ff9bddb
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209777660
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ie059d34e32f35718abc13b321fbf9f762a5585c1
Update symbol list after making the DMA-BUF heap page-pool helper
library built-in.
Bug: 183902174
Change-Id: I855e034942e2cbaf935e855d0db8882ccefb0395
Signed-off-by: Hridya Valsaraju <hridya@google.com>
Since vendors might depend on them for their system heap
implementations, make the page-pool library built-in to
freeze its KMI.
Bug: 183902174
Bug: 212210831
Change-Id: If633619ec1f78d0fbd73c43c48b19d98db7807af
Signed-off-by: Hridya Valsaraju <hridya@google.com>
this critical region should be protected by pool->mutex.
Bug: 207658347
Fixes: e7dac4c323 ("ANDROID: dma-buf: heaps: Add a shrinker controlled page pool")
Signed-off-by: liuhailong <liuhailong@oppo.com>
Signed-off-by: xieliujie <xieliujie@oppo.com>
Change-Id: I6f129926c96176258a965964c24602fc647db61e
dmabuf_page_pool_init_shrinker needs to be static to prevent a warning
when compiling with -Wmissing-prototypes. Change it to be static.
Fixes: e7dac4c323 ("ANDROID: dma-buf: heaps: Add a shrinker controlled page pool")
Bug: 168742043
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I64184cf4062e33c14a60b9c3d505db922f2b9c0b
To allow process_mrelease to reap targeted mm in parallel with exit_mmap
mark the victim with MMF_OOM_VICTIM flag.
Bug: 189803002
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I89cf5f8fbeeb18b93a340b9ebe7f200837ebe846
With exit_mmap holding mmap_write_lock during free_pgtables call,
process_mrelease does not need to elevate mm->mm_users in order to
prevent exit_mmap from destrying pagetables while __oom_reap_task_mm
is walking the VMA tree. The change prevents process_mrelease from
calling the last mmput, which can lead to waiting for IO completion
in exit_aio.
Fixes: 337546e83f ("mm/oom_kill.c: prevent a race between process_mrelease and exit_mmap")
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Link: https://lore.kernel.org/all/20211124235906.14437-2-surenb@google.com/
Bug: 130172058
Bug: 189803002
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I1e2728e0c477af9cc20e9e0b715ee67dee760618
oom-reaper and process_mrelease system call should protect against
races with exit_mmap which can destroy page tables while they
walk the VMA tree. oom-reaper protects from that race by setting
MMF_OOM_VICTIM and by relying on exit_mmap to set MMF_OOM_SKIP
before taking and releasing mmap_write_lock. process_mrelease has
to elevate mm->mm_users to prevent such race. Both oom-reaper and
process_mrelease hold mmap_read_lock when walking the VMA tree.
The locking rules and mechanisms could be simpler if exit_mmap takes
mmap_write_lock while executing destructive operations such as
free_pgtables.
Change exit_mmap to hold the mmap_write_lock when calling
free_pgtables. Operations like unmap_vmas() and unlock_range() are not
destructive and could run under mmap_read_lock but for simplicity we
take one mmap_write_lock during almost the entire operation. Note
also that because oom-reaper checks VM_LOCKED flag, unlock_range()
should not be allowed to race with it.
In most cases this lock should be uncontended. Previously, Kirill
reported ~4% regression caused by a similar change [1]. We reran the
same test and although the individual results are quite noisy, the
percentiles show lower regression with 1.6% being the worst case [2].
The change allows oom-reaper and process_mrelease to execute safely
under mmap_read_lock without worries that exit_mmap might destroy page
tables from under them.
[1] https://lore.kernel.org/all/20170725141723.ivukwhddk2voyhuc@node.shutemov.name/
[2] https://lore.kernel.org/all/CAJuCfpGC9-c9P40x7oy=jy5SphMcd0o0G_6U1-+JAziGKG6dGA@mail.gmail.com/
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Link: https://lore.kernel.org/all/20211124235906.14437-1-surenb@google.com/
Bug: 130172058
Bug: 189803002
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: Ic87272d09a0b68a1b0e968e8f1a1510fd6fc776a
Make use of the newly introduced unshare hypercall during guest teardown
to unmap guest-related data structures from the hyp stage-1.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-15-qperret@google.com
(cherry picked from commit 52b28657eb
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: Ic8ec2c74d3e803e4d5f0df37d144d2c2eb0e9ad3
Introduce an unshare hypercall which can be used to unmap memory from
the hypervisor stage-1 in nVHE protected mode. This will be useful to
update the EL2 ownership state of pages during guest teardown, and
avoids keeping dangling mappings to unreferenced portions of memory.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-14-qperret@google.com
(cherry picked from commit b8cc6eb5bd
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I0b2839046914a0fd9a6d0077383d53dc84a8ac3d
Tearing down a previously shared memory region results in the borrower
losing access to the underlying pages and returning them to the "owned"
state in the owner.
Implement a do_unshare() helper, along the same lines as do_share(), to
provide this functionality for the host-to-hyp case.
Reviewed-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-13-qperret@google.com
(cherry picked from commit 376a240f03
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I07aa6194099325c955f48ab608c4b601664419e5
__pkvm_host_share_hyp() shares memory between the host and the
hypervisor so implement it as an invocation of the new do_share()
mechanism.
Note that double-sharing is no longer permitted (as this allows us to
reduce the number of page-table walks significantly), but is thankfully
no longer relied upon by the host.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-12-qperret@google.com
(cherry picked from commit 1ee32109fd
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I9f5877b6ea7b9bf735a90492fbca16f485e73110
By default, protected KVM isolates memory pages so that they are
accessible only to their owner: be it the host kernel, the hypervisor
at EL2 or (in future) the guest. Establishing shared-memory regions
between these components therefore involves a transition for each page
so that the owner can share memory with a borrower under a certain set
of permissions.
Introduce a do_share() helper for safely sharing a memory region between
two components. Currently, only host-to-hyp sharing is implemented, but
the code is easily extended to handle other combinations and the
permission checks for each component are reusable.
Reviewed-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-11-qperret@google.com
(cherry picked from commit e82edcc75c
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I3ee4486ab08faa24015ac608979e3596b27806a2
In preparation for adding additional locked sections for manipulating
page-tables at EL2, introduce some simple wrappers around the host and
hypervisor locks so that it's a bit easier to read and bit more difficult
to take the wrong lock (or even take them in the wrong order).
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-10-qperret@google.com
(cherry picked from commit 61d99e33e7
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I413e2d7f514db507a1aca9a21f04115e97f72c00
Explicitly name the combination of SW0 | SW1 as reserved in the pte and
introduce a new PKVM_NOPAGE meta-state which, although not directly
stored in the software bits of the pte, can be used to represent an
entry for which there is no underlying page. This is distinct from an
invalid pte, as stage-2 identity mappings for the host are created
lazily and so an invalid pte there is the same as a valid mapping for
the purposes of ownership information.
This state will be used for permission checking during page transitions
in later patches.
Reviewed-by: Andrew Walbran <qwandor@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-9-qperret@google.com
(cherry picked from commit 3d467f7b8c
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I2e803a6ae83694be178a3d3eaf75fe918c881659
In order to simplify the page tracking infrastructure at EL2 in nVHE
protected mode, move the responsibility of refcounting pages that are
shared multiple times on the host. In order to do so, let's create a
red-black tree tracking all the PFNs that have been shared, along with
a refcount.
Acked-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-8-qperret@google.com
(cherry picked from commit a83e2191b7
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I58f39d139c84acb545676c99144547cd080cdda3
The create_hyp_mappings() function can currently be called at any point
in time. However, its behaviour in protected mode changes widely
depending on when it is being called. Prior to KVM init, it is used to
create the temporary page-table used to bring-up the hypervisor, and
later on it is transparently turned into a 'share' hypercall when the
kernel has lost control over the hypervisor stage-1. In order to prepare
the ground for also unsharing pages with the hypervisor during guest
teardown, introduce a kvm_share_hyp() function to make it clear in which
places a share hypercall should be expected, as we will soon need a
matching unshare hypercall in all those places.
[ qperret: BACKPORT due to resolution in kvm_arch_vcpu_run_map_fp() in
merge commit 2d761dbf7f ("Merge branch kvm-arm64/fpsimd-tracking
into kvmarm-master/next") ]
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-7-qperret@google.com
(cherry picked from commit 3f868e142c
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: Id3ba033a51550e5aada862d6af06773d0be833fb
kvm_pgtable_hyp_unmap() relies on the ->page_count() function callback
being provided by the memory-management operations for the page-table.
Wire up this callback for the hypervisor stage-1 page-table.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-5-qperret@google.com
(cherry picked from commit 34ec7cbf1e
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I481b1f722f4dddda72e35481aca725a937a52090
In nVHE-protected mode, the hyp stage-1 page-table refcount is broken
due to the lack of refcount support in the early allocator. Fix-up the
refcount in the finalize walker, once the 'hyp_vmemmap' is up and running.
Acked-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-4-qperret@google.com
(cherry picked from commit d6b4bd3f48
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I6c48e4e5a0ae969e1d1ac6be0ce601faf138cfd3
To prepare the ground for allowing hyp stage-1 mappings to be removed at
run-time, update the KVM page-table code to maintain a correct refcount
using the ->{get,put}_page() function callbacks.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-3-qperret@google.com
(cherry picked from commit 2ea2ff91e8
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: Ic4ce9177a4381a01e1f2c3b00e9f376e14015718
In nVHE protected mode, the EL2 code uses a temporary allocator during
boot while re-creating its stage-1 page-table. Unfortunately, the
hyp_vmmemap is not ready to use at this stage, so refcounting pages
is not possible. That is not currently a problem because hyp stage-1
mappings are never removed, which implies refcounting of page-table
pages is unnecessary.
In preparation for allowing hypervisor stage-1 mappings to be removed,
provide stub implementations for {get,put}_page() in the early allocator.
Acked-by: Will Deacon <will@kernel.org>
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20211215161232.1480836-2-qperret@google.com
(cherry picked from commit 1fac3cfb9c
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next)
Bug: 209599700
Change-Id: I9602cb4db9cef6f52e8beefa52e53cf71f2e0cd1
This reverts commit 2183d4b524.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: I1c7256d8992d29d38f122f5ad889ba904eda9793
This reverts commit b91c0e5d96.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: I34ad2c837d3d37cfe85b6a6e0fdba1368298fce6
This reverts commit 59b96bd25e.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: Icde87ddf560b90c55195f3c1250e6079b0795389
This reverts commit 75ed01242c.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: If7628be6ada702db6aa1326734fd70349aa1f3e4
This reverts commit d6d544e4af.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: Ie7351e6764d468a5b8a5c0ed663e50d8875914b3
This reverts commit 7c979b8271.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: I710c31254ce80a5b1de5ced791ccaec1b0bebd64
This reverts commit e41cc0f83a.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: Id418beae1a0d2d418c9e5167ae25105f231bdd68
This reverts commit 1d5d8f2493.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: I51fb6724d3a305800aa9d9226406c5d1b2e94331
This reverts commit 0a1b52003b.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: I9776a0847f6ccd15cf6ad5d62f0e81dd38aa0598
This reverts commit a8e6e3763a.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: Ibd16a2427b2de139bd7b2651a563238540949ecc
This reverts commit a39c93198b.
This will be replaced by a FROMGIT patch shortly.
Signed-off-by: Quentin Perret <qperret@google.com>
Change-Id: Idf31ec997871d56bcb21b464e73b18a5cb50d49a
