shinys000114/linux
1
0
Fork 0
mirror of https://github.com/hardkernel/linux.git synced 2026-06-05 10:31:46 +09:00
Code Issues Packages Projects Releases Wiki Activity

UPSTREAM: workingset: memcg: sleep when flushing stats in workingset_refault()

Browse Source 
In workingset_refault(), we call
mem_cgroup_flush_stats_atomic_ratelimited() to read accurate stats within
an RCU read section and with sleeping disallowed.  Move the call above the
RCU read section to make it non-atomic.

Flushing is an expensive operation that scales with the number of cpus and
the number of cgroups in the system, so avoid doing it atomically where
possible.

Since workingset_refault() is the only caller of
mem_cgroup_flush_stats_atomic_ratelimited(), just make it non-atomic, and
rename it to mem_cgroup_flush_stats_ratelimited().

Link: https://lkml.kernel.org/r/20230330191801.1967435-7-yosryahmed@google.com
Change-Id: Ia073b9bcef12dfced053ebea7a5da96ce942596c
Signed-off-by: Yosry Ahmed <yosryahmed@google.com>
Acked-by: Shakeel Butt <shakeelb@google.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Josef Bacik <josef@toxicpanda.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vasily Averin <vasily.averin@linux.dev>
Cc: Zefan Li <lizefan.x@bytedance.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 4009b2f188)
Bug: 322544714
Signed-off-by: T.J. Mercier <tjmercier@google.com>
This commit is contained in:
Yosry Ahmed
2023-03-30 19:17:59 +00:00
committed by T.J. Mercier
parent 0448f92c8d
commit e3e3beed82
3 changed files with 7 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
include/linux/memcontrol.h
View File

@@ -1034,7 +1034,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,
void mem_cgroup_flush_stats(void);
void mem_cgroup_flush_stats_atomic(void);
void mem_cgroup_flush_stats_atomic_ratelimited(void);
void mem_cgroup_flush_stats_ratelimited(void);
void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,
int val);
@@ -1524,7 +1524,7 @@ static inline void mem_cgroup_flush_stats_atomic(void)
{
}
static inline void mem_cgroup_flush_stats_atomic_ratelimited(void)
static inline void mem_cgroup_flush_stats_ratelimited(void)
{
}

4
mm/memcontrol.c
View File

@@ -671,10 +671,10 @@ void mem_cgroup_flush_stats_atomic(void)
do_flush_stats(true);
}
void mem_cgroup_flush_stats_atomic_ratelimited(void)
void mem_cgroup_flush_stats_ratelimited(void)
{
if (time_after64(jiffies_64, READ_ONCE(flush_next_time)))
mem_cgroup_flush_stats_atomic();
mem_cgroup_flush_stats();
}
static void flush_memcg_stats_dwork(struct work_struct *w)

5
mm/workingset.c
View File

@@ -409,6 +409,9 @@ void workingset_refault(struct folio *folio, void *shadow)
unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset);
eviction <<= bucket_order;
/* Flush stats (and potentially sleep) before holding RCU read lock */
mem_cgroup_flush_stats_ratelimited();
rcu_read_lock();
/*
* Look up the memcg associated with the stored ID. It might
@@ -463,8 +466,6 @@ void workingset_refault(struct folio *folio, void *shadow)
lruvec = mem_cgroup_lruvec(memcg, pgdat);
mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr);
mem_cgroup_flush_stats_atomic_ratelimited();
/*
* Compare the distance to the existing workingset size. We
* don't activate pages that couldn't stay resident even if