Calvin Owens 61ea92b948 ksoftirqd: Enable IRQs and call cond_resched() before poking RCU ...

commit 28423ad283 upstream.

While debugging an issue with excessive softirq usage, I encountered the
following note in commit 3e339b5dae ("softirq: Use hotplug thread
infrastructure"):

    [ paulmck: Call rcu_note_context_switch() with interrupts enabled. ]

...but despite this note, the patch still calls RCU with IRQs disabled.

This seemingly innocuous change caused a significant regression in softirq
CPU usage on the sending side of a large TCP transfer (~1 GB/s): when
introducing 0.01% packet loss, the softirq usage would jump to around 25%,
spiking as high as 50%. Before the change, the usage would never exceed 5%.

Moving the call to rcu_note_context_switch() after the cond_sched() call,
as it was originally before the hotplug patch, completely eliminated this
problem.

Signed-off-by: Calvin Owens <calvinowens@fb.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mike Galbraith <mgalbraith@suse.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

2015-05-06 21:56:27 +02:00

..

cpu

sched, idle: Fix the idle polling state logic

2013-11-29 11:11:42 -08:00

debug

kdb: fix incorrect counts in KDB summary command output

2015-03-06 14:40:52 -08:00

events

perf: Fix irq_work 'tail' recursion

2015-04-13 14:02:12 +02:00

gcov

kernel/gcov: remove depends on CONFIG_EXPERIMENTAL

2013-01-11 11:39:33 -08:00

irq

genirq: Sanitize spurious interrupt detection of threaded irqs

2014-06-30 20:09:45 -07:00

power

PM / QoS: remove duplicate call to pm_qos_update_target

2015-03-18 13:22:28 +01:00

sched

printk: rename printk_sched to printk_deferred

2014-08-07 14:30:26 -07:00

time

ntp: Fixup adjtimex freq validation on 32-bit systems

2015-03-06 14:40:52 -08:00

trace

ring-buffer: Replace this_cpu_*() with __this_cpu_*()

2015-05-06 21:56:21 +02:00

.gitignore

kernel/hz.bc: ignore.

2013-04-22 07:09:06 -07:00

acct.c

fs: Fix hang with BSD accounting on frozen filesystem

2013-05-04 14:57:58 -04:00

async.c

async: rename and redefine async_func_ptr

2013-03-12 13:59:14 -07:00

audit_tree.c

audit: keep inode pinned

2014-11-21 09:22:52 -08:00

audit_watch.c

audit: catch possible NULL audit buffers

2013-01-11 14:54:55 -08:00

audit.c

CAPABILITIES: remove undefined caps from all processes

2014-09-17 09:03:57 -07:00

audit.h

audit: fix mq_open and mq_unlink to add the MQ root as a hidden parent audit_names record

2013-12-04 10:57:03 -08:00

auditfilter.c

auditfilter.c: fix kernel-doc warnings

2013-05-24 16:22:52 -07:00

auditsc.c

auditsc: audit_krule mask accesses need bounds checking

2014-06-16 13:42:53 -07:00

backtracetest.c

…

bounds.c

memcg: remove direct page_cgroup-to-page pointer

2011-03-23 19:46:28 -07:00

capability.c

CAPABILITIES: remove undefined caps from all processes

2014-09-17 09:03:57 -07:00

cgroup_freezer.c

cgroup: rename ->create/post_create/pre_destroy/destroy() to ->css_alloc/online/offline/free()

2012-11-19 08:13:38 -08:00

cgroup.c

move d_rcu from overlapping d_child to overlapping d_alias

2015-04-29 10:34:00 +02:00

compat.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal

2013-05-01 07:21:43 -07:00

configs.c

proc: Supply PDE attribute setting accessor functions

2013-05-01 17:29:18 -04:00

context_tracking.c

Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2013-06-20 08:18:35 -10:00

cpu_pm.c

kernel/cpu_pm.c: fix various typos

2012-05-31 17:49:27 -07:00

cpu.c

sched: Fix hotplug vs. set_cpus_allowed_ptr()

2014-06-11 12:03:24 -07:00

cpuset.c

cpuset,mempolicy: fix sleeping function called from invalid context

2014-07-17 15:58:00 -07:00

crash_dump.c

Merge branch 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux

2011-11-06 19:44:47 -08:00

cred.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace

2012-12-18 10:55:28 -08:00

delayacct.c

cputime: Use accessors to read task cputime stats

2013-01-27 19:23:31 +01:00

dma.c

Remove all #inclusions of asm/system.h

2012-03-28 18:30:03 +01:00

elfcore.c

…

exec_domain.c

…

exit.c

introduce for_each_thread() to replace the buggy while_each_thread()

2014-10-05 14:54:15 -07:00

extable.c

extable: Flip the sorting message

2013-04-15 13:25:16 +02:00

fork.c

perf: fix perf bug in fork()

2014-10-09 12:18:42 -07:00

freezer.c

freezer: Do not freeze tasks killed by OOM killer

2014-11-14 08:47:58 -08:00

futex_compat.c

Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/signal

2013-02-23 18:50:11 -08:00

futex.c

futex: Make lookup_pi_state more robust

2014-06-07 13:25:41 -07:00

groups.c

userns: Don't allow setgroups until a gid mapping has been setablished

2015-01-08 09:58:16 -08:00

hrtimer.c

hrtimer: Set expiry time before switch_hrtimer_base()

2014-06-07 13:25:31 -07:00

hung_task.c

hung task debugging: Inject NMI when hung and going to panic

2012-04-25 12:39:25 +02:00

irq_work.c

Merge branch 'nohz/printk-v8' into irq/core

2013-02-05 00:48:46 +01:00

itimer.c

itimer: Use printk_once instead of WARN_ONCE

2012-04-10 11:00:30 +02:00

jump_label.c

jump_label: Export jump_label_rate_limit()

2012-08-06 19:00:35 +03:00

kallsyms.c

kernel: kallsyms: memory override issue, need check destination buffer length

2013-04-15 15:17:26 +09:30

kcmp.c

kcmp: fix standard comparison bug

2014-10-05 14:54:13 -07:00

Kconfig.freezer

…

Kconfig.hz

…

Kconfig.locks

locking/mutex: Disable optimistic spinning on some architectures

2014-07-28 08:00:07 -07:00

Kconfig.preempt

locking/kconfig: Simplify INLINE_SPIN_UNLOCK usage

2012-03-23 13:18:57 +01:00

kexec.c

PCI: Disable Bus Master only on kexec reboot

2013-12-20 07:45:08 -08:00

kmod.c

usermodehelper: check subprocess_info->path != NULL

2013-05-16 12:01:11 -07:00

kprobes.c

kprobes: Fix to free gone and unused optprobes

2013-05-28 10:37:59 +02:00

ksysfs.c

Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2012-12-11 18:10:49 -08:00

kthread.c

kthread: implement probe_kthread_data()

2013-04-30 17:04:02 -07:00

latencytop.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

lglock.c

brlocks/lglocks: turn into functions

2012-05-29 23:28:41 -04:00

lockdep_internals.h

…

lockdep_proc.c

lockdep: Use KSYM_NAME_LEN'ed buffer for __get_key_name()

2012-10-24 12:39:09 +02:00

lockdep_states.h

…

lockdep.c

Merge branch 'for-3.10/drivers' of git://git.kernel.dk/linux-block

2013-05-08 11:51:05 -07:00

Makefile

Merge tag 'modules-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux

2013-05-05 10:58:06 -07:00

modsign_certificate.S

CONFIG_SYMBOL_PREFIX: cleanup.

2013-03-15 15:09:43 +10:30

modsign_pubkey.c

keys: use keyring_alloc() to create module signing keyring

2012-12-20 17:40:21 -08:00

module_signing.c

MODSIGN: Don't use enum-type bitfields in module signature info block

2012-12-05 11:27:24 +10:30

module-internal.h

MODSIGN: Move the magic string to the end of a module and eliminate the search

2012-10-19 17:30:40 -07:00

module.c

modules, lock around setting of MODULE_STATE_UNFORMED

2014-11-14 08:47:55 -08:00

mutex-debug.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

mutex-debug.h

mutex: Use p->on_cpu for the adaptive spin

2011-04-14 08:52:33 +02:00

mutex.c

mutex: Back out architecture specific check for negative mutex count

2013-04-19 09:33:36 +02:00

mutex.h

mutex: Use p->on_cpu for the adaptive spin

2011-04-14 08:52:33 +02:00

notifier.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

nsproxy.c

proc: Split the namespace stuff out into linux/proc_ns.h

2013-05-01 17:29:39 -04:00

padata.c

padata: use __this_cpu_read per-cpu helper

2012-12-06 17:16:23 +08:00

panic.c

dump_stack: implement arch-specific hardware description in task dumps

2013-04-30 17:04:02 -07:00

params.c

params: Fix potential memory leak in add_sysfs_param()

2013-03-18 11:40:21 +00:00

pid_namespace.c

pid_namespace: pidns_get() should check task_active_pid_ns() != NULL

2014-04-26 17:15:34 -07:00

pid.c

exit: pidns: alloc_pid() leaks pid_namespace if child_reaper is exiting

2015-01-08 09:58:17 -08:00

posix-cpu-timers.c

posix_timers: Fix pre-condition to stop the tick on full dynticks

2013-04-22 19:59:25 +02:00

posix-timers.c

posix-timers: Fix stack info leak in timer_create()

2014-11-14 08:48:00 -08:00

printk.c

console: Fix console name size mismatch

2015-04-19 10:10:51 +02:00

profile.c

proc: Supply PDE attribute setting accessor functions

2013-05-01 17:29:18 -04:00

ptrace.c

ptrace: fix race between ptrace_resume() and wait_task_stopped()

2015-05-06 21:56:24 +02:00

range.c

range: Do not add new blank slot with add_range_with_merge

2013-06-18 11:32:10 -05:00

rcu.h

rcu: Provide RCU CPU stall warnings for tiny RCU

2013-01-28 22:06:21 -08:00

rcupdate.c

Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD

2013-01-28 22:25:21 -08:00

rcutiny_plugin.h

rcu: Provide RCU CPU stall warnings for tiny RCU

2013-01-28 22:06:21 -08:00

rcutiny.c

Merge branches 'doctorture.2013.01.29a', 'fixes.2013.01.26a', 'tagcb.2013.01.24a' and 'tiny.2013.01.29b' into HEAD

2013-01-28 22:25:21 -08:00

rcutorture.c

rcu: Allow rcutorture to be built at low optimization levels

2013-02-04 12:18:20 -08:00

rcutree_plugin.h

rcu: Don't allocate bootmem from rcu_init()

2013-05-15 10:41:12 -07:00

rcutree_trace.c

rcutrace: single_open() leaks

2013-05-05 00:16:35 -04:00

rcutree.c

rcu: Fix deadlock with CPU hotplug, RCU GP init, and timer migration

2013-06-10 13:37:12 -07:00

rcutree.h

rcu: Don't call wakeup() with rcu_node structure ->lock held

2013-06-10 13:37:11 -07:00

relay.c

Merge branch 'for-3.10/core' of git://git.kernel.dk/linux-block

2013-05-08 10:13:35 -07:00

res_counter.c

res_counter: return amount of charges after res_counter_uncharge()

2012-12-18 15:02:12 -08:00

resource.c

mem hotunplug: fix kfree() of bootmem memory

2013-04-29 15:54:40 -07:00

rtmutex_common.h

rtmutex: Simplify PI algorithm and make highest prio task get lock

2011-01-27 21:13:51 -05:00

rtmutex-debug.c

sched/rt: Move rt specific bits into new header file

2013-02-07 20:51:08 +01:00

rtmutex-debug.h

rtmutex: Handle deadlock detection smarter

2014-07-17 15:58:04 -07:00

rtmutex-tester.c

locking/rtmutex/tester: Set correct permissions on sysfs files

2013-04-10 14:48:37 +02:00

rtmutex.c

rtmutex: Plug slow unlock race

2014-07-17 15:58:04 -07:00

rtmutex.h

rtmutex: Handle deadlock detection smarter

2014-07-17 15:58:04 -07:00

rwsem.c

Revert "rw_semaphore: remove up/down_read_non_owner"

2013-03-23 15:53:52 -07:00

seccomp.c

seccomp: allow BPF_XOR based ALU instructions.

2013-03-26 11:07:19 +11:00

semaphore.c

semaphore: use `bool' type for semaphore_waiter's up

2013-04-30 17:04:08 -07:00

signal.c

Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next

2013-05-01 14:08:52 -07:00

smp.c

kernel/smp.c:on_each_cpu_cond(): fix warning in fallback path

2014-09-17 09:03:57 -07:00

smpboot.c

smpboot: Add missing get_online_cpus() in smpboot_register_percpu_thread()

2015-02-11 14:48:17 +08:00

smpboot.h

smpboot: Provide infrastructure for percpu hotplug threads

2012-08-13 17:01:07 +02:00

softirq.c

ksoftirqd: Enable IRQs and call cond_resched() before poking RCU

2015-05-06 21:56:27 +02:00

spinlock.c

locking/kconfig: Simplify INLINE_SPIN_UNLOCK usage

2012-03-23 13:18:57 +01:00

srcu.c

srcu: use ACCESS_ONCE() to access sp->completed in srcu_read_lock()

2013-02-07 15:19:36 -08:00

stacktrace.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

stop_machine.c

stop_machine: Mark per cpu stopper enabled early

2013-02-26 22:25:17 +01:00

sys_ni.c

unify compat fanotify_mark(2), switch to COMPAT_SYSCALL_DEFINE

2013-05-09 13:46:38 -04:00

sys.c

reboot: rigrate shutdown/reboot to boot cpu

2013-06-12 16:29:44 -07:00

sysctl_binary.c

switch compat_sys_sysctl to COMPAT_SYSCALL_DEFINE

2013-05-09 14:53:20 -04:00

sysctl.c

perf: Enforce 1 as lower limit for perf_event_max_sample_rate

2014-06-11 12:03:27 -07:00

task_work.c

task_work: task_work_add() should not succeed after exit_task_work()

2012-09-13 16:47:34 +02:00

taskstats.c

taskstats: cgroupstats_user_cmd() may leak on error

2012-10-06 03:05:31 +09:00

test_kprobes.c

kernel/: rename random32() to prandom_u32()

2013-04-29 18:28:42 -07:00

time.c

time: settimeofday: Validate the values of tv from user

2015-01-29 17:40:56 -08:00

timeconst.bc

kernel: Replace timeconst.pl with a bc script

2013-02-16 23:17:25 +01:00

timer.c

timer: Prevent overflow in apply_slack

2014-06-07 13:25:30 -07:00

tracepoint.c

tracepoint: Do not waste memory on mods with no tracepoints

2014-05-30 21:52:11 -07:00

tsacct.c

cputime: Use accessors to read task cputime stats

2013-01-27 19:23:31 +01:00

uid16.c

groups: Consolidate the setgroups permission checks

2015-01-08 09:58:16 -08:00

up.c

kernel: Map most files to use export.h instead of module.h

2011-10-31 09:20:12 -04:00

user_namespace.c

userns: Allow setting gid_maps without privilege when setgroups is disabled

2015-01-08 09:58:17 -08:00

user-return-notifier.c

hlist: drop the node parameter from iterators

2013-02-27 19:10:24 -08:00

user.c

userns: Add a knob to disable setgroups on a per user namespace basis

2015-01-08 09:58:16 -08:00

utsname_sysctl.c

kernel/utsname_sysctl.c: put get/get_uts() into CONFIG_PROC_SYSCTL code block

2013-02-27 19:10:22 -08:00

utsname.c

proc: Split the namespace stuff out into linux/proc_ns.h

2013-05-01 17:29:39 -04:00

wait.c

propagate name change to comments in kernel source

2012-12-06 10:39:54 +01:00

watchdog.c

watchdog: Add comments to explain the watchdog_disabled variable

2013-03-14 08:24:05 +01:00

workqueue_internal.h

workqueue: include workqueue info when printing debug dump of a worker task

2013-04-30 17:04:02 -07:00

workqueue.c

workqueue: fix hang involving racing cancel[_delayed]_work_sync()'s for PREEMPT_NONE

2015-03-26 15:00:58 +01:00