Setting a page flag without holding a reference to the page
is living dangerously. In the tag-writing path, we drop the
reference to the page by calling kvm_release_pfn_dirty(),
and only then set the PG_mte_tagged bit.
It would be safer to do it the other way round.
Fixes: f0376edb1d ("KVM: arm64: Add ioctl to fetch/store tags in a guest")
Cc: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Steven Price <steven.price@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/87k0mjidwb.wl-maz@kernel.org
(cherry picked from commit 98db7259fa)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: Iea5ad082147127c19a580c98629ac60c696092f8
A new capability (KVM_CAP_ARM_MTE) identifies that the kernel supports
granting a guest access to the tags, and provides a mechanism for the
VMM to enable it.
A new ioctl (KVM_ARM_MTE_COPY_TAGS) provides a simple way for a VMM to
access the tags of a guest without having to maintain a PROT_MTE mapping
in userspace. The above capability gates access to the ioctl.
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210621111716.37157-7-steven.price@arm.com
(cherry picked from commit 04c02c201d)
[willdeacon@: Dropped missing context]
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I489a4cc12cad2b4664b131d4d65c9d12024eaa48
The VMM may not wish to have it's own mapping of guest memory mapped
with PROT_MTE because this causes problems if the VMM has tag checking
enabled (the guest controls the tags in physical RAM and it's unlikely
the tags are correct for the VMM).
Instead add a new ioctl which allows the VMM to easily read/write the
tags from guest memory, allowing the VMM's mapping to be non-PROT_MTE
while the VMM can still read/write the tags for the purpose of
migration.
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210621111716.37157-6-steven.price@arm.com
(cherry picked from commit f0376edb1d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I2f1ea19d9f2bdbdd3025934b770468b1404020c6
Add a new VM feature 'KVM_ARM_CAP_MTE' which enables memory tagging
for a VM. This will expose the feature to the guest and automatically
tag memory pages touched by the VM as PG_mte_tagged (and clear the tag
storage) to ensure that the guest cannot see stale tags, and so that
the tags are correctly saved/restored across swap.
Actually exposing the new capability to user space happens in a later
patch.
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
[maz: move VM_SHARED sampling into the critical section]
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210621111716.37157-3-steven.price@arm.com
(cherry picked from commit ea7fc1bb1c)
[willdeacon@: Torrid time resolving mmu.c, as upstream merge commit made
significant changes to the original patch. Fixed UAPI
definitions in kvm.h]
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: If584166e765413456575f642f87003a4c9e41280
A KVM guest could store tags in a page even if the VMM hasn't mapped
the page with PROT_MTE. So when restoring pages from swap we will
need to check to see if there are any saved tags even if !pte_tagged().
However don't check pages for which pte_access_permitted() returns false
as these will not have been swapped out.
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Steven Price <steven.price@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210621111716.37157-2-steven.price@arm.com
(cherry picked from commit 69e3b846d8)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I2232daa13de3354acc2b487373977950847b0b41
According to ARM DDI 0487G.a, page D13-3895, setting the PMCR_EL0.P bit to
1 has the following effect:
"Reset all event counters accessible in the current Exception level, not
including PMCCNTR_EL0, to zero."
Similar behaviour is described for AArch32 on page G8-7022. Make it so.
Fixes: c01d6a1802 ("KVM: arm64: pmu: Only handle supported event counters")
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210618105139.83795-1-alexandru.elisei@arm.com
(cherry picked from commit 2a71fabf6a)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I47d26a718844563278a5f820452d85a773452f58
We currently uniformly perform CMOs of D-cache and I-cache in function
user_mem_abort before calling the fault handlers. If we get concurrent
guest faults(e.g. translation faults, permission faults) or some really
unnecessary guest faults caused by BBM, CMOs for the first vcpu are
necessary while the others later are not.
By moving CMOs to the fault handlers, we can easily identify conditions
where they are really needed and avoid the unnecessary ones. As it's a
time consuming process to perform CMOs especially when flushing a block
range, so this solution reduces much load of kvm and improve efficiency
of the stage-2 page table code.
We can imagine two specific scenarios which will gain much benefit:
1) In a normal VM startup, this solution will improve the efficiency of
handling guest page faults incurred by vCPUs, when initially populating
stage-2 page tables.
2) After live migration, the heavy workload will be resumed on the
destination VM, however all the stage-2 page tables need to be rebuilt
at the moment. So this solution will ease the performance drop during
resuming stage.
Reviewed-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Yanan Wang <wangyanan55@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210617105824.31752-5-wangyanan55@huawei.com
(cherry picked from commit 25aa28691b)
[willdeacon@: Fix conflicts due to lack of gfn-based spte notifiers]
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I8fb827c24f57e2c1bcb0a016729bda9a72eaa389
Adjust the parameter "kvm_pfn_t pfn" of __clean_dcache_guest_page
and __invalidate_icache_guest_page to "void *va", which paves the
way for converting these two guest CMO functions into callbacks in
structure kvm_pgtable_mm_ops. No functional change.
Reviewed-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Yanan Wang <wangyanan55@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210617105824.31752-4-wangyanan55@huawei.com
(cherry picked from commit 378e6a9c78)
[willdeacon@: Fix conflicts with cache maintenance rework and lack of gfn-based
spte notifiers]
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: Ic6c4de33e005ec6fdefd91985485ed2c22b0271c
To prepare for performing CMOs for guest stage-2 in the fault handlers
in pgtable.c, here introduce two cache maintenance callbacks in struct
kvm_pgtable_mm_ops. We also adjust the comment alignment for the
existing part but make no real content change at all.
Reviewed-by: Fuad Tabba <tabba@google.com>
Signed-off-by: Yanan Wang <wangyanan55@huawei.com>
[maz: fixed up comments and renamed callbacks]
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210617105824.31752-2-wangyanan55@huawei.com
(cherry picked from commit 6204004de3)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: If5bb0706b081c16b3e9853954866c4c81030b8a9
The hyp_page refcount is currently encoded on 4 bytes even though we
never need to count that many objects in a page. Make it 2 bytes to save
some space in the vmemmap.
As overflows are more likely to happen as well, make sure to catch those
with a BUG in the increment function.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210608114518.748712-8-qperret@google.com
(cherry picked from commit 6929586d8e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I1e36b015fc65b45b811a886607e9aa3bce873e27
Each struct hyp_page currently contains a pointer to a hyp_pool struct
where the page should be freed if its refcount reaches 0. However, this
information can always be inferred from the context in the EL2 code, so
drop the pointer to save a few bytes in the vmemmap.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210608114518.748712-6-qperret@google.com
(cherry picked from commit d978b9cfe6)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: Iadcbf118f2c9d7651e631ffb46393e2dbfbfc58b
We currently maintain two separate memory pools for the host stage-2,
one for pages used in the page-table when mapping memory regions, and
the other to map MMIO regions. The former is large enough to map all of
memory with page granularity and the latter can cover an arbitrary
portion of IPA space, but allows to 'recycle' pages.
However, this split makes accounting difficult to manage as pages at
intermediate levels of the page-table may be used to map both memory and
MMIO regions. Simplify the scheme by merging both pools into one. This
means we can now hit the -ENOMEM case in the memory abort path, but
we're still guaranteed forward-progress in the worst case by unmapping
MMIO regions. On the plus side this also means we can usually map a lot
more MMIO space at once if memory ranges happen to be mapped with block
mappings.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210608114518.748712-5-qperret@google.com
(cherry picked from commit 7c350ea39e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I0fbab059f0c01287f2074392391259092e55cf48
The list_head member of struct hyp_page is only needed when the page is
attached to a free-list, which by definition implies the page is free.
As such, nothing prevents us from using the page itself to store the
list_head, hence reducing the size of the vmemmap.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210608114518.748712-4-qperret@google.com
(cherry picked from commit 914cde58a0)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I6055802715da653caab20da16f9a33d777de3acf
The hyp buddy allocator currently checks the struct hyp_page list node
to see if a page is available for allocation or not when trying to
coalesce memory. Now that decrementing the refcount and attaching to
the buddy tree is done in the same critical section, we can rely on the
refcount of the buddy page to be in sync, which allows to replace the
list node check by a refcount check. This will ease removing the list
node from struct hyp_page later on.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210608114518.748712-3-qperret@google.com
(cherry picked from commit 581982decc)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I7a985eaf0665da27fd35e55701098241b63de7c9
The hyp_page refcount helpers currently rely on the hyp_pool lock for
serialization. However, this means the refcounts can't be changed from
the buddy allocator core as it already holds the lock, which means pages
have to go through odd transient states.
For example, when a page is freed, its refcount is set to 0, and the
lock is transiently released before the page can be attached to a free
list in the buddy tree. This is currently harmless as the allocator
checks the list node of each page to see if it is available for
allocation or not, but it means the page refcount can't be trusted to
represent the state of the page even if the pool lock is held.
In order to fix this, remove the pool locking from the refcount helpers,
and move all the logic to the buddy allocator. This will simplify the
removal of the list node from struct hyp_page in a later patch.
Signed-off-by: Quentin Perret <qperret@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210608114518.748712-2-qperret@google.com
(cherry picked from commit 6cbf874e51)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I3fc3296d2cce2c20bed530dbc3ee856f7b27efe3
The MMIO region of a device maybe huge (GB level), try to use
block mapping in stage2 to speedup both map and unmap.
Compared to normal memory mapping, we should consider two more
points when try block mapping for MMIO region:
1. For normal memory mapping, the PA(host physical address) and
HVA have same alignment within PUD_SIZE or PMD_SIZE when we use
the HVA to request hugepage, so we don't need to consider PA
alignment when verifing block mapping. But for device memory
mapping, the PA and HVA may have different alignment.
2. For normal memory mapping, we are sure hugepage size properly
fit into vma, so we don't check whether the mapping size exceeds
the boundary of vma. But for device memory mapping, we should pay
attention to this.
This adds get_vma_page_shift() to get page shift for both normal
memory and device MMIO region, and check these two points when
selecting block mapping size for MMIO region.
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210507110322.23348-3-zhukeqian1@huawei.com
(cherry picked from commit 2aa53d68ce)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I1af0ba01c521d3a3c66301f32fdc3f780f10fe6f
The MMIO regions may be unmapped for many reasons and can be remapped
by stage2 fault path. Map MMIO regions at creation time becomes a
minor optimization and makes these two mapping path hard to sync.
Remove the mapping code while keep the useful sanity check.
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210507110322.23348-2-zhukeqian1@huawei.com
(cherry picked from commit fd6f17bade)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I428bd02d7cb83f4f48d5e294c0be9f09fb15fae4
In order to deal with the lack of active state, we need to use
the mask/unmask primitives (after all, the active state is just an
additional mask on top of the normal one).
To avoid adding a bunch of ugly conditionals in the timer and vgic
code, let's use a timer-specific irqdomain to deal with the state
conversion. Yes, this is an unexpected use of irqdomains, but
there is no reason not to be just as creative as the designers
of the HW...
This involves overloading the vcpu_affinity, set_irqchip_state
and eoi callbacks so that the rest of the KVM code can continue
ignoring the oddities of the underlying platform.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 5f59229680)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I53ec1b772615e0edc1dfec20350b0a330be2f59a
As we are about to add some more things to the timer IRQ
configuration, move this code out of the main timer init code
into its own set of functions.
No functional changes.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 2f2f7e39db)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: Ifa99e1f81cff43356f5e21d583d08cabff7f0ba7
In order to deal with these systems that do not offer HW-based
deactivation of interrupts, let implement a SW-based approach:
- When the irq is queued into a LR, treat it as a pure virtual
interrupt and set the EOI flag in the LR.
- When the interrupt state is read back from the LR, force a
deactivation when the state is invalid (neither active nor
pending)
Interrupts requiring such treatment get the VGIC_SW_RESAMPLE flag.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 354920e794)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: Ic6d9c96f989e90cd6b7e55f0919f3e914520faaa
We already have the option to attach a callback to an interrupt
to retrieve its pending state. As we are planning to expand this
facility, move this callback into its own data structure.
This will limit the size of individual interrupts as the ops
structures can be shared across multiple interrupts.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit db75f1a33f)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I6176af86588b144158cf62f5a10201e9ff5cd6c4
The vGIC, as architected by ARM, allows a virtual interrupt to
trigger the deactivation of a physical interrupt. This allows
the following interrupt to be delivered without requiring an exit.
However, some implementations have choosen not to implement this,
meaning that we will need some unsavoury workarounds to deal with this.
On detecting such a case, taint the kernel and spit a nastygram.
We'll deal with this in later patches.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit f6c3e24fb7)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I00244ba04b9671eb27be0b1d566969e3dc921da6
As it turns out, not all the interrupt controllers are able to
expose a vGIC maintenance interrupt that can be independently
enabled/disabled.
And to be fair, it doesn't really matter as all we require is
for the interrupt to kick us out of guest mode out way or another.
To that effect, add gic_kvm_info.no_maint_irq_mask for an interrupt
controller to advertise the lack of masking.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 669062d2a1)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I06c247347faf9899aedc131e95b6f764aa474f64
As we we now entertain the possibility of FIQ being used on the host,
treat the signalling of a FIQ while running a guest as an IRQ,
causing an exit instead of a HYP panic.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 74501499d4)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: If83a70cff22464133f516ccc03ee13483121b802
The vGIC advertising code is unsurprisingly very much tied to
the GIC implementations. However, we are about to extend the
support to lesser implementations.
Let's dissociate the vgic registration from the GIC code and
move it into KVM, where it makes a bit more sense. This also
allows us to mark the gic_kvm_info structures as __initdata.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 0e5cb77706)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I7d20bad2ca51bc6166f1cffa88ac4d53c6dc4ef4
Although naming across the codebase isn't that consistent, it
tends to follow certain patterns. Moreover, the term "flush"
isn't defined in the Arm Architecture reference manual, and might
be interpreted to mean clean, invalidate, or both for a cache.
Rename arm64-internal functions to make the naming internally
consistent, as well as making it consistent with the Arm ARM, by
specifying whether it applies to the instruction, data, or both
caches, whether the operation is a clean, invalidate, or both.
Also specify which point the operation applies to, i.e., to the
point of unification (PoU), coherency (PoC), or persistence
(PoP).
This commit applies the following sed transformation to all files
under arch/arm64:
"s/\b__flush_cache_range\b/caches_clean_inval_pou_macro/g;"\
"s/\b__flush_icache_range\b/caches_clean_inval_pou/g;"\
"s/\binvalidate_icache_range\b/icache_inval_pou/g;"\
"s/\b__flush_dcache_area\b/dcache_clean_inval_poc/g;"\
"s/\b__inval_dcache_area\b/dcache_inval_poc/g;"\
"s/__clean_dcache_area_poc\b/dcache_clean_poc/g;"\
"s/\b__clean_dcache_area_pop\b/dcache_clean_pop/g;"\
"s/\b__clean_dcache_area_pou\b/dcache_clean_pou/g;"\
"s/\b__flush_cache_user_range\b/caches_clean_inval_user_pou/g;"\
"s/\b__flush_icache_all\b/icache_inval_all_pou/g;"
Note that __clean_dcache_area_poc is deliberately missing a word
boundary check at the beginning in order to match the efistub
symbols in image-vars.h.
Also note that, despite its name, __flush_icache_range operates
on both instruction and data caches. The name change here
reflects that.
No functional change intended.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-19-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit fade9c2c6e)
[willdeacon@: Dropped EFI change; fixed conflicts in hibernate.c and
smp_spin_table.c]
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I219bebf7e6740af1a8d34f68588d32a4497ba957
Fix and expand comments for the cache maintenance functions in
cacheflush.h. Adds comments to functions that weren't described
before. Explains what the functions do using Arm Architecture
Reference Manual terminology.
No functional change intended.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-18-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit 393239be1b)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: Idfae63fb707e8f99783951d4c59a36aedf2eff22
To be consistent with other functions with similar names and
functionality in cacheflush.h, cache.S, and cachetlb.rst, change
to specify the range in terms of start and end, as opposed to
start and size.
Because the code is shared with __dma_clean_area, it changes the
parameters for that as well. However, __dma_clean_area is local to
cache.S, so no other users are affected.
No functional change intended.
Reported-by: Will Deacon <will@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-14-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit 1f42faf1d2)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I92ac7c3a1dbe3c445ec3dbd059869edfdb2a93f7
To be consistent with other functions with similar names and
functionality in cacheflush.h, cache.S, and cachetlb.rst, change
to specify the range in terms of start and end, as opposed to
start and size.
No functional change intended.
Reported-by: Will Deacon <will@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-13-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit 814b186079)
[willdeacon@: Dropped EFI change; fixed context conflict in smp_spin_table.c]
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I0a858403b52485341aa96f116bc8d4769f961802
To be consistent with other functions with similar names and
functionality in cacheflush.h, cache.S, and cachetlb.rst, change
to specify the range in terms of start and end, as opposed to
start and size.
Because the code is shared with __dma_inv_area, it changes the
parameters for that as well. However, __dma_inv_area is local to
cache.S, so no other users are affected.
No functional change intended.
Reported-by: Will Deacon <will@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-11-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit e3974adb4e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: Ide5024737d08a221f3e7d55caab7e488afd2177c
Many comments refer to the function flush_icache_range, where the
intent is in fact __flush_icache_range. Fix these comments to
refer to the intended function.
That's probably due to commit 3b8c9f1cdf ("arm64: IPI each
CPU after invalidating the I-cache for kernel mappings"), which
renamed flush_icache_range() to __flush_icache_range() and added
a wrapper.
No functional change intended.
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-10-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit d044f81418)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: I3953e798bc36760eab29677b2ad798cac57c7984
Currently, kernel relocation function is configured in machine_kexec()
at the time of kexec reboot by using control_code_page.
This operation, however, is more logical to be done during kexec_load,
and thus remove from reboot time. Move, setup of this function to
newly added machine_kexec_post_load().
Because once MMU is enabled, kexec control page will contain more than
relocation kernel, but also vector table, add pointer to the actual
function within this page arch.kern_reloc. Currently, it equals to the
beginning of page, we will add offsets later, when vector table is
added.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
Link: https://lore.kernel.org/r/20210125191923.1060122-10-pasha.tatashin@soleen.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit 4c3c31230c)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: If8a871fda92bc7f1da9da38622f5de8ea8dcbb93
It would be helpful if we could use both `dcache_by_line_op` and
`invalidate_icache_by_line` for user memory without accidentally fixing
up unexpected faults when performing maintenance on kernel addresses.
Let's make this possible by having both macros take an optional fixup
label, and only generating an extable entry if a label is provided.
At the same time, let's clean up the labels used to be globally unique
using \@ as we do for other macros.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Cc: Ard Biesheuvel <aedb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Fuad Tabba <tabba@google.com>
Cc: Will Deacon <will@kernel.org>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20210524083001.2586635-3-tabba@google.com
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit d11b187760)
Signed-off-by: Will Deacon <willdeacon@google.com>
Bug: 192636784
Change-Id: If2f4ea856724039297715d543b598757759636e2
