Calvin Owens 1f649f0368 ksoftirqd: Enable IRQs and call cond_resched() before poking RCU ...

commit 28423ad283 upstream.

While debugging an issue with excessive softirq usage, I encountered the
following note in commit 3e339b5dae ("softirq: Use hotplug thread
infrastructure"):

    [ paulmck: Call rcu_note_context_switch() with interrupts enabled. ]

...but despite this note, the patch still calls RCU with IRQs disabled.

This seemingly innocuous change caused a significant regression in softirq
CPU usage on the sending side of a large TCP transfer (~1 GB/s): when
introducing 0.01% packet loss, the softirq usage would jump to around 25%,
spiking as high as 50%. Before the change, the usage would never exceed 5%.

Moving the call to rcu_note_context_switch() after the cond_sched() call,
as it was originally before the hotplug patch, completely eliminated this
problem.

Signed-off-by: Calvin Owens <calvinowens@fb.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mike Galbraith <mgalbraith@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

2015-05-06 21:59:20 +02:00

..

cpu

sched/preempt: Fix up missed PREEMPT_NEED_RESCHED folding

2014-01-13 17:38:55 +01:00

debug

kdb: fix incorrect counts in KDB summary command output

2015-03-06 14:43:30 -08:00

events

perf: Fix irq_work 'tail' recursion

2015-04-13 14:03:01 +02:00

gcov

gcov: reuse kbasename helper

2013-11-13 12:09:34 +09:00

irq

genirq: Prevent proc race against freeing of irq descriptors

2015-01-27 08:18:55 -08:00

locking

rtmutex: Plug slow unlock race

2014-06-30 20:11:58 -07:00

power

PM / Sleep: fix recovery during resuming from hibernation

2014-11-14 09:00:10 -08:00

printk

console: Fix console name size mismatch

2015-03-26 15:06:53 +01:00

rcu

rcu: Use rcu_gp_kthread_wake() to wake up grace period kthreads

2014-11-21 09:23:05 -08:00

sched

sched: Fix RLIMIT_RTTIME when PI-boosting to RT

2015-04-19 10:11:06 +02:00

time

ntp: Fixup adjtimex freq validation on 32-bit systems

2015-03-06 14:43:30 -08:00

trace

ring-buffer: Replace this_cpu_*() with __this_cpu_*()

2015-05-06 21:59:10 +02:00

.gitignore

Ignore generated file kernel/x509_certificate_list

2013-12-10 18:21:34 +00:00

acct.c

fs: Fix hang with BSD accounting on frozen filesystem

2013-05-04 14:57:58 -04:00

async.c

async: rename and redefine async_func_ptr

2013-03-12 13:59:14 -07:00

audit_tree.c

audit: keep inode pinned

2014-11-21 09:23:01 -08:00

audit_watch.c

inotify: Fix reporting of cookies for inotify events

2014-02-18 11:17:17 +01:00

audit.c

audit: AUDIT_FEATURE_CHANGE message format missing delimiting space

2014-11-21 09:23:01 -08:00

audit.h

audit: Use struct net not pid_t to remember the network namespce to reply in

2014-02-28 04:04:33 -08:00

auditfilter.c

audit: restore AUDIT_LOGINUID unset ABI

2015-01-08 10:00:50 -08:00

auditsc.c

audit: remove superfluous new- prefix in AUDIT_LOGIN messages

2014-07-09 11:18:28 -07:00

backtracetest.c

…

bounds.c

mm: do not allocate page->ptl dynamically, if spinlock_t fits to long

2013-12-20 12:25:45 -08:00

capability.c

CAPABILITIES: remove undefined caps from all processes

2014-09-17 09:19:09 -07:00

cgroup_freezer.c

cgroup: replace cftype->read_seq_string() with cftype->seq_show()

2013-12-05 12:28:04 -05:00

cgroup.c

move d_rcu from overlapping d_child to overlapping d_alias

2015-04-29 10:31:54 +02:00

compat.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal

2013-05-01 07:21:43 -07:00

configs.c

proc: Supply PDE attribute setting accessor functions

2013-05-01 17:29:18 -04:00

context_tracking.c

context_tracking: Wrap static key check into more intuitive function name

2013-12-02 20:43:14 +01:00

cpu_pm.c

kernel/cpu_pm.c: fix various typos

2012-05-31 17:49:27 -07:00

cpu.c

sched: Fix hotplug vs. set_cpus_allowed_ptr()

2014-06-11 11:54:11 -07:00

cpuset.c

cpuset: Fix cpuset sched_relax_domain_level

2015-03-26 15:06:54 +01:00

crash_dump.c

Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux

2011-11-06 19:44:47 -08:00

cred.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace

2012-12-18 10:55:28 -08:00

delayacct.c

kernel/delayacct.c: remove redundant checking in __delayacct_add_tsk()

2013-11-13 12:09:12 +09:00

dma.c

Remove all #inclusions of asm/system.h

2012-03-28 18:30:03 +01:00

elfcore.c

switch elf_core_write_extra_phdrs() to dump_emit()

2013-11-09 00:16:23 -05:00

exec_domain.c

…

exit.c

exit: call disassociate_ctty() before exit_task_namespaces()

2014-04-26 17:19:07 -07:00

extable.c

kernel/extable: fix address-checks for core_kernel and init areas

2013-11-28 09:49:41 -08:00

fork.c

mm: per-thread vma caching

2014-10-09 12:21:29 -07:00

freezer.c

freezer: Do not freeze tasks killed by OOM killer

2014-11-14 09:00:01 -08:00

futex_compat.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal

2013-02-23 18:50:11 -08:00

futex.c

futex: Ensure get_futex_key_refs() always implies a barrier

2014-10-30 09:38:24 -07:00

groups.c

userns: Don't allow setgroups until a gid mapping has been setablished

2015-01-08 10:00:50 -08:00

hrtimer.c

hrtimer: Set expiry time before switch_hrtimer_base()

2014-06-07 10:28:12 -07:00

hung_task.c

hung_task: Display every hung task warning

2014-01-25 12:13:33 +01:00

irq_work.c

Merge branch 'nohz/printk-v8' into irq/core

2013-02-05 00:48:46 +01:00

itimer.c

itimer: Use printk_once instead of WARN_ONCE

2012-04-10 11:00:30 +02:00

jump_label.c

static_key: WARN on usage before jump_label_init was called

2013-10-19 19:45:35 -04:00

kallsyms.c

kernel: kallsyms: memory override issue, need check destination buffer length

2013-04-15 15:17:26 +09:30

kcmp.c

kcmp: fix standard comparison bug

2014-10-05 14:52:20 -07:00

Kconfig.freezer

…

Kconfig.hz

kernel: remove CONFIG_USE_GENERIC_SMP_HELPERS

2013-11-15 09:32:22 +09:00

Kconfig.locks

locking/mutex: Disable optimistic spinning on some architectures

2014-07-28 08:06:03 -07:00

Kconfig.preempt

locking/kconfig: Simplify INLINE_SPIN_UNLOCK usage

2012-03-23 13:18:57 +01:00

kexec.c

powerpc, kexec: Fix "Processor X is stuck" issue during kexec from ST mode

2014-06-07 10:28:28 -07:00

kmod.c

execve: use 'struct filename *' for executable name passing

2014-02-05 12:54:53 -08:00

kprobes.c

kprobes: use KSYM_NAME_LEN to size identifier buffers

2013-11-13 12:09:26 +09:00

ksysfs.c

kdump: fix exported size of vmcoreinfo note

2014-01-23 16:37:03 -08:00

kthread.c

kthread: fix return value of kthread_create() upon SIGKILL.

2014-06-30 20:11:53 -07:00

latencytop.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

Makefile

KEYS: Remove files generated when SYSTEM_TRUSTED_KEYRING=y

2013-12-13 15:59:11 +00:00

module_signing.c

keys: change asymmetric keys to use common hash definitions

2013-10-25 17:15:18 -04:00

module-internal.h

KEYS: Separate the kernel signature checking keyring from module signing

2013-09-25 17:17:01 +01:00

module.c

modules, lock around setting of MODULE_STATE_UNFORMED

2014-11-14 08:59:52 -08:00

notifier.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

nsproxy.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace

2013-09-07 14:35:32 -07:00

padata.c

padata: Fix wrong usage of rcu_dereference()

2013-12-05 21:28:42 +08:00

panic.c

panic: Make panic_timeout configurable

2013-11-26 12:12:26 +01:00

params.c

params: improve standard definitions

2013-12-04 14:09:46 +10:30

pid_namespace.c

pid_namespace: pidns_get() should check task_active_pid_ns() != NULL

2014-04-26 17:19:04 -07:00

pid.c

exit: pidns: alloc_pid() leaks pid_namespace if child_reaper is exiting

2015-01-08 10:00:50 -08:00

posix-cpu-timers.c

posix-timers: Convert abuses of BUG_ON to WARN_ON

2013-12-09 16:56:29 +01:00

posix-timers.c

posix-timers: Fix stack info leak in timer_create()

2014-11-14 09:00:10 -08:00

profile.c

mm: fix GFP_THISNODE callers and clarify

2014-03-10 17:26:19 -07:00

ptrace.c

ptrace: fix race between ptrace_resume() and wait_task_stopped()

2015-05-06 21:59:14 +02:00

range.c

range: Do not add new blank slot with add_range_with_merge

2013-06-18 11:32:10 -05:00

reboot.c

kexec: migrate to reboot cpu

2013-12-18 19:04:50 -08:00

relay.c

kernel: delete __cpuinit usage from all core kernel files

2013-07-14 19:36:59 -04:00

res_counter.c

memcg: reduce function dereference

2013-09-12 15:38:02 -07:00

resource.c

kernel/resource.c: remove the unneeded assignment in function __find_resource

2013-07-03 16:08:06 -07:00

seccomp.c

seccomp: allow BPF_XOR based ALU instructions.

2013-03-26 11:07:19 +11:00

signal.c

kernel/signal.c: change do_signal_stop/do_sigaction to use while_each_thread()

2014-01-23 16:37:02 -08:00

smp.c

kernel/smp.c:on_each_cpu_cond(): fix warning in fallback path

2014-09-17 09:19:09 -07:00

smpboot.c

smpboot: Add missing get_online_cpus() in smpboot_register_percpu_thread()

2015-02-11 14:54:49 +08:00

smpboot.h

smpboot: Provide infrastructure for percpu hotplug threads

2012-08-13 17:01:07 +02:00

softirq.c

ksoftirqd: Enable IRQs and call cond_resched() before poking RCU

2015-05-06 21:59:20 +02:00

stacktrace.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

stop_machine.c

stop_machine: Fix^2 race between stop_two_cpus() and stop_cpus()

2014-03-11 11:33:47 +01:00

sys_ni.c

unify compat fanotify_mark(2), switch to COMPAT_SYSCALL_DEFINE

2013-05-09 13:46:38 -04:00

sys.c

kernel/sys.c: k_getrusage() can use while_each_thread()

2014-01-23 16:37:02 -08:00

sysctl_binary.c

kernel/sysctl_binary.c: use scnprintf() instead of snprintf()

2013-11-13 12:09:33 +09:00

sysctl.c

mm, pcp: allow restoring percpu_pagelist_fraction default

2014-07-09 11:18:26 -07:00

system_certificates.S

KEYS: correct alignment of system_certificate_list content in assembly file

2013-12-10 18:25:28 +00:00

system_keyring.c

KEYS: correct alignment of system_certificate_list content in assembly file

2013-12-10 18:25:28 +00:00

task_work.c

task_work: documentation

2013-09-11 15:58:27 -07:00

taskstats.c

genetlink: only pass array to genl_register_family_with_ops()

2013-11-19 16:39:05 -05:00

test_kprobes.c

kernel/: rename random32() to prandom_u32()

2013-04-29 18:28:42 -07:00

time.c

time: settimeofday: Validate the values of tv from user

2015-01-29 17:40:49 -08:00

timeconst.bc

kernel: Replace timeconst.pl with a bc script

2013-02-16 23:17:25 +01:00

timer.c

timer: Prevent overflow in apply_slack

2014-06-07 10:28:09 -07:00

tracepoint.c

tracepoint: Do not waste memory on mods with no tracepoints

2014-05-31 13:20:28 -07:00

tsacct.c

cputime: Use accessors to read task cputime stats

2013-01-27 19:23:31 +01:00

uid16.c

groups: Consolidate the setgroups permission checks

2015-01-08 10:00:49 -08:00

up.c

kernel: provide a __smp_call_function_single stub for !CONFIG_SMP

2013-11-15 09:32:22 +09:00

user_namespace.c

userns: Allow setting gid_maps without privilege when setgroups is disabled

2015-01-08 10:00:50 -08:00

user-return-notifier.c

hlist: drop the node parameter from iterators

2013-02-27 19:10:24 -08:00

user.c

userns: Add a knob to disable setgroups on a per user namespace basis

2015-01-08 10:00:50 -08:00

utsname_sysctl.c

kernel/utsname_sysctl.c: put get/get_uts() into CONFIG_PROC_SYSCTL code block

2013-02-27 19:10:22 -08:00

utsname.c

userns: Kill nsown_capable it makes the wrong thing easy

2013-08-30 23:44:11 -07:00

watchdog.c

kernel/watchdog.c: remove preemption restrictions when restarting lockup detector

2014-07-06 18:57:27 -07:00

workqueue_internal.h

sched: Rename sched.c as sched/core.c in comments and Documentation

2013-06-19 12:58:42 +02:00

workqueue.c

workqueue: fix hang involving racing cancel[_delayed]_work_sync()'s for PREEMPT_NONE

2015-03-26 15:06:54 +01:00