Skip to content

Commit 3e5cfe4

Browse files
committed
Add cooperative task yielding
A single call to `poll` on a top-level task may potentially do a lot of work before it returns `Poll::Pending`. If a task runs for a long period of time without yielding back to the executor, it can starve other tasks waiting on that executor to execute them, or drive underlying resources. See for example rust-lang/futures-rs#2047, rust-lang/futures-rs#1957, and rust-lang/futures-rs#869. Since Rust does not have a runtime, it is difficult to forcibly preempt a long-running task. Consider a future like this one: ```rust use tokio::stream::StreamExt; async fn drop_all<I: Stream>(input: I) { while let Some(_) = input.next().await {} } ``` It may look harmless, but consider what happens under heavy load if the input stream is _always_ ready. If we spawn `drop_all`, the task will never yield, and will starve other tasks and resources on the same executor. This patch adds a `coop` module that provides an opt-in mechanism for futures to cooperate with the executor to avoid starvation. This alleviates the problem above: ``` use tokio::stream::StreamExt; async fn drop_all<I: Stream>(input: I) { while let Some(_) = input.next().await { tokio::coop::proceed().await; } } ``` The call to [`proceed`] will coordinate with the executor to make sure that every so often control is yielded back to the executor so it can run other tasks. The implementation uses a thread-local counter that simply counts how many "cooperation points" we have passed since the task was first polled. Once the "budget" has been spent, any subsequent points will return `Poll::Pending`, eventually making the top-level task yield. When it finally does yield, the executor resets the budget before running the next task. The budget per task poll is currently hard-coded to 128. Eventually, we may want to make it dynamic as more cooperation points are added. The number 128 was chosen more or less arbitrarily to balance the cost of yielding unnecessarily against the time an executor may be "held up". At the moment, all the tokio leaf futures ("resources") call into coop, but external futures have no way of doing so. We probably want to continue limiting coop points to leaf futures in the future, but may want to also enable third-party leaf futures to cooperate to benefit the ecosystem as a whole. This is reflected in the methods marked as `pub` in `mod coop` (even though the module is only `pub(crate)`). We will likely also eventually want to expose `coop::limit`, which enables sub-executors and manual `impl Future` blocks to avoid one sub-task spending all of their poll budget. Benchmarks (see #2160) suggest that the overhead of `coop` is marginal.
1 parent fce6845 commit 3e5cfe4

File tree

17 files changed

+373
-16
lines changed

17 files changed

+373
-16
lines changed

tokio/src/coop.rs

+302
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,302 @@
1+
//! Opt-in yield points for improved cooperative scheduling.
2+
//!
3+
//! A single call to [`poll`] on a top-level task may potentially do a lot of work before it
4+
//! returns `Poll::Pending`. If a task runs for a long period of time without yielding back to the
5+
//! executor, it can starve other tasks waiting on that executor to execute them, or drive
6+
//! underlying resources. Since Rust does not have a runtime, it is difficult to forcibly preempt a
7+
//! long-running task. Instead, this module provides an opt-in mechanism for futures to collaborate
8+
//! with the executor to avoid starvation.
9+
//!
10+
//! Consider a future like this one:
11+
//!
12+
//! ```
13+
//! # use tokio::stream::{Stream, StreamExt};
14+
//! async fn drop_all<I: Stream + Unpin>(mut input: I) {
15+
//! while let Some(_) = input.next().await {}
16+
//! }
17+
//! ```
18+
//!
19+
//! It may look harmless, but consider what happens under heavy load if the input stream is
20+
//! _always_ ready. If we spawn `drop_all`, the task will never yield, and will starve other tasks
21+
//! and resources on the same executor. With opt-in yield points, this problem is alleviated:
22+
//!
23+
//! ```ignore
24+
//! # use tokio::stream::{Stream, StreamExt};
25+
//! async fn drop_all<I: Stream + Unpin>(mut input: I) {
26+
//! while let Some(_) = input.next().await {
27+
//! tokio::coop::proceed().await;
28+
//! }
29+
//! }
30+
//! ```
31+
//!
32+
//! The `proceed` future will coordinate with the executor to make sure that every so often control
33+
//! is yielded back to the executor so it can run other tasks.
34+
//!
35+
//! # Placing yield points
36+
//!
37+
//! Voluntary yield points should be placed _after_ at least some work has been done. If they are
38+
//! not, a future sufficiently deep in the task hierarchy may end up _never_ getting to run because
39+
//! of the number of yield points that inevitably appear before it is reached. In general, you will
40+
//! want yield points to only appear in "leaf" futures -- those that do not themselves poll other
41+
//! futures. By doing this, you avoid double-counting each iteration of the outer future against
42+
//! the cooperating budget.
43+
//!
44+
//! [`poll`]: https://doc.rust-lang.org/std/future/trait.Future.html#tymethod.poll
45+
46+
// NOTE: The doctests in this module are ignored since the whole module is (currently) private.
47+
48+
use std::cell::Cell;
49+
use std::task::{Context, Poll};
50+
51+
/// Constant used to determine how much "work" a task is allowed to do without yielding.
52+
///
53+
/// The value itself is chosen somewhat arbitrarily. It needs to be high enough to amortize wakeup
54+
/// and scheduling costs, but low enough that we do not starve other tasks for too long. The value
55+
/// also needs to be high enough that particularly deep tasks are able to do at least some useful
56+
/// work at all.
57+
///
58+
/// Note that as more yield points are added in the ecosystem, this value will probably also have
59+
/// to be raised.
60+
const BUDGET: usize = 128;
61+
62+
/// Constant used to determine if budgeting has been disabled.
63+
const UNCONSTRAINED: usize = usize::max_value();
64+
65+
thread_local! {
66+
static HITS: Cell<usize> = Cell::new(UNCONSTRAINED);
67+
}
68+
69+
/// Run the given closure with a cooperative task budget.
70+
///
71+
/// Enabling budgeting when it is already enabled is a no-op.
72+
#[inline(always)]
73+
pub(crate) fn budget<F, R>(f: F) -> R
74+
where
75+
F: FnOnce() -> R,
76+
{
77+
HITS.with(move |hits| {
78+
if hits.get() != UNCONSTRAINED {
79+
// We are already being budgeted.
80+
//
81+
// Arguably this should be an error, but it can happen "correctly"
82+
// such as with block_on + LocalSet, so we make it a no-op.
83+
return f();
84+
}
85+
86+
struct Guard<'a>(&'a Cell<usize>);
87+
impl<'a> Drop for Guard<'a> {
88+
fn drop(&mut self) {
89+
self.0.set(UNCONSTRAINED);
90+
}
91+
}
92+
93+
hits.set(BUDGET);
94+
let _guard = Guard(hits);
95+
f()
96+
})
97+
}
98+
99+
cfg_blocking_impl! {
100+
/// Forcibly remove the budgeting constraints early.
101+
pub(crate) fn stop() {
102+
HITS.with(|hits| {
103+
hits.set(UNCONSTRAINED);
104+
});
105+
}
106+
}
107+
108+
/// Invoke `f` with a subset of the remaining budget.
109+
///
110+
/// This is useful if you have sub-futures that you need to poll, but that you want to restrict
111+
/// from using up your entire budget. For example, imagine the following future:
112+
///
113+
/// ```rust
114+
/// # use std::{future::Future, pin::Pin, task::{Context, Poll}};
115+
/// use futures::stream::FuturesUnordered;
116+
/// struct MyFuture<F1, F2> {
117+
/// big: FuturesUnordered<F1>,
118+
/// small: F2,
119+
/// }
120+
///
121+
/// use tokio::stream::Stream;
122+
/// impl<F1, F2> Future for MyFuture<F1, F2>
123+
/// where F1: Future, F2: Future
124+
/// # , F1: Unpin, F2: Unpin
125+
/// {
126+
/// type Output = F2::Output;
127+
///
128+
/// // fn poll(...)
129+
/// # fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<F2::Output> {
130+
/// # let this = &mut *self;
131+
/// let mut big = // something to pin self.big
132+
/// # Pin::new(&mut this.big);
133+
/// let small = // something to pin self.small
134+
/// # Pin::new(&mut this.small);
135+
///
136+
/// // see if any of the big futures have finished
137+
/// while let Some(e) = futures::ready!(big.as_mut().poll_next(cx)) {
138+
/// // do something with e
139+
/// # let _ = e;
140+
/// }
141+
///
142+
/// // see if the small future has finished
143+
/// small.poll(cx)
144+
/// }
145+
/// # }
146+
/// ```
147+
///
148+
/// It could be that every time `poll` gets called, `big` ends up spending the entire budget, and
149+
/// `small` never gets polled. That would be sad. If you want to stick up for the little future,
150+
/// that's what `limit` is for. It lets you portion out a smaller part of the yield budget to a
151+
/// particular segment of your code. In the code above, you would write
152+
///
153+
/// ```rust,ignore
154+
/// # use std::{future::Future, pin::Pin, task::{Context, Poll}};
155+
/// # use futures::stream::FuturesUnordered;
156+
/// # struct MyFuture<F1, F2> {
157+
/// # big: FuturesUnordered<F1>,
158+
/// # small: F2,
159+
/// # }
160+
/// #
161+
/// # use tokio::stream::Stream;
162+
/// # impl<F1, F2> Future for MyFuture<F1, F2>
163+
/// # where F1: Future, F2: Future
164+
/// # , F1: Unpin, F2: Unpin
165+
/// # {
166+
/// # type Output = F2::Output;
167+
/// # fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<F2::Output> {
168+
/// # let this = &mut *self;
169+
/// # let mut big = Pin::new(&mut this.big);
170+
/// # let small = Pin::new(&mut this.small);
171+
/// #
172+
/// // see if any of the big futures have finished
173+
/// while let Some(e) = futures::ready!(tokio::coop::limit(64, || big.as_mut().poll_next(cx))) {
174+
/// # // do something with e
175+
/// # let _ = e;
176+
/// # }
177+
/// # small.poll(cx)
178+
/// # }
179+
/// # }
180+
/// ```
181+
///
182+
/// Now, even if `big` spends its entire budget, `small` will likely be left with some budget left
183+
/// to also do useful work. In particular, if the remaining budget was `N` at the start of `poll`,
184+
/// `small` will have at least a budget of `N - 64`. It may be more if `big` did not spend its
185+
/// entire budget.
186+
///
187+
/// Note that you cannot _increase_ your budget by calling `limit`. The budget provided to the code
188+
/// inside the buget is the _minimum_ of the _current_ budget and the bound.
189+
///
190+
#[allow(unreachable_pub, dead_code)]
191+
pub fn limit<R>(bound: usize, f: impl FnOnce() -> R) -> R {
192+
HITS.with(|hits| {
193+
let budget = hits.get();
194+
// with_bound cannot _increase_ the remaining budget
195+
let bound = std::cmp::min(budget, bound);
196+
// When f() exits, how much should we add to what is left?
197+
let floor = budget.saturating_sub(bound);
198+
// Make sure we restore the remaining budget even on panic
199+
struct RestoreBudget<'a>(&'a Cell<usize>, usize);
200+
impl<'a> Drop for RestoreBudget<'a> {
201+
fn drop(&mut self) {
202+
let left = self.0.get();
203+
self.0.set(self.1 + left);
204+
}
205+
}
206+
// Time to restrict!
207+
hits.set(bound);
208+
let _restore = RestoreBudget(&hits, floor);
209+
f()
210+
})
211+
}
212+
213+
/// Returns `Poll::Pending` if the current task has exceeded its budget and should yield.
214+
#[allow(unreachable_pub, dead_code)]
215+
#[inline]
216+
pub fn poll_proceed(cx: &mut Context<'_>) -> Poll<()> {
217+
HITS.with(|hits| {
218+
let n = hits.get();
219+
if n == UNCONSTRAINED {
220+
// opted out of budgeting
221+
Poll::Ready(())
222+
} else if n == 0 {
223+
cx.waker().wake_by_ref();
224+
Poll::Pending
225+
} else {
226+
hits.set(n.saturating_sub(1));
227+
Poll::Ready(())
228+
}
229+
})
230+
}
231+
232+
/// Resolves immediately unless the current task has already exceeded its budget.
233+
///
234+
/// This should be placed after at least some work has been done. Otherwise a future sufficiently
235+
/// deep in the task hierarchy may end up never getting to run because of the number of yield
236+
/// points that inevitably appear before it is even reached. For example:
237+
///
238+
/// ```ignore
239+
/// # use tokio::stream::{Stream, StreamExt};
240+
/// async fn drop_all<I: Stream + Unpin>(mut input: I) {
241+
/// while let Some(_) = input.next().await {
242+
/// tokio::coop::proceed().await;
243+
/// }
244+
/// }
245+
/// ```
246+
#[allow(unreachable_pub, dead_code)]
247+
#[inline]
248+
pub async fn proceed() {
249+
use crate::future::poll_fn;
250+
poll_fn(|cx| poll_proceed(cx)).await;
251+
}
252+
253+
#[cfg(all(test, not(loom)))]
254+
mod test {
255+
use super::*;
256+
257+
fn get() -> usize {
258+
HITS.with(|hits| hits.get())
259+
}
260+
261+
#[test]
262+
fn bugeting() {
263+
use tokio_test::*;
264+
265+
assert_eq!(get(), UNCONSTRAINED);
266+
assert_ready!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
267+
assert_eq!(get(), UNCONSTRAINED);
268+
budget(|| {
269+
assert_eq!(get(), BUDGET);
270+
assert_ready!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
271+
assert_eq!(get(), BUDGET - 1);
272+
assert_ready!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
273+
assert_eq!(get(), BUDGET - 2);
274+
});
275+
assert_eq!(get(), UNCONSTRAINED);
276+
277+
budget(|| {
278+
limit(3, || {
279+
assert_eq!(get(), 3);
280+
assert_ready!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
281+
assert_eq!(get(), 2);
282+
limit(4, || {
283+
assert_eq!(get(), 2);
284+
assert_ready!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
285+
assert_eq!(get(), 1);
286+
});
287+
assert_eq!(get(), 1);
288+
assert_ready!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
289+
assert_eq!(get(), 0);
290+
assert_pending!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
291+
assert_eq!(get(), 0);
292+
assert_pending!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
293+
assert_eq!(get(), 0);
294+
});
295+
assert_eq!(get(), BUDGET - 3);
296+
assert_ready!(task::spawn(()).enter(|cx, _| poll_proceed(cx)));
297+
assert_eq!(get(), BUDGET - 4);
298+
assert_ready!(task::spawn(proceed()).poll());
299+
assert_eq!(get(), BUDGET - 5);
300+
});
301+
}
302+
}

tokio/src/io/registration.rs

+6
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,9 @@ impl Registration {
139139
///
140140
/// This function will panic if called from outside of a task context.
141141
pub fn poll_read_ready(&self, cx: &mut Context<'_>) -> Poll<io::Result<mio::Ready>> {
142+
// Keep track of task budget
143+
ready!(crate::coop::poll_proceed(cx));
144+
142145
let v = self.poll_ready(Direction::Read, Some(cx))?;
143146
match v {
144147
Some(v) => Poll::Ready(Ok(v)),
@@ -190,6 +193,9 @@ impl Registration {
190193
///
191194
/// This function will panic if called from outside of a task context.
192195
pub fn poll_write_ready(&self, cx: &mut Context<'_>) -> Poll<io::Result<mio::Ready>> {
196+
// Keep track of task budget
197+
ready!(crate::coop::poll_proceed(cx));
198+
193199
let v = self.poll_ready(Direction::Write, Some(cx))?;
194200
match v {
195201
Some(v) => Poll::Ready(Ok(v)),

tokio/src/lib.rs

+2
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,8 @@ cfg_process! {
321321

322322
pub mod runtime;
323323

324+
pub(crate) mod coop;
325+
324326
cfg_signal! {
325327
pub mod signal;
326328
}

tokio/src/process/mod.rs

+3
Original file line numberDiff line numberDiff line change
@@ -700,6 +700,9 @@ where
700700
type Output = Result<T, E>;
701701

702702
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
703+
// Keep track of task budget
704+
ready!(crate::coop::poll_proceed(cx));
705+
703706
let ret = Pin::new(&mut self.inner).poll(cx);
704707

705708
if let Poll::Ready(Ok(_)) = ret {

tokio/src/runtime/basic_scheduler.rs

+1-1
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,7 @@ where
128128
pin!(future);
129129

130130
'outer: loop {
131-
if let Ready(v) = future.as_mut().poll(&mut cx) {
131+
if let Ready(v) = crate::coop::budget(|| future.as_mut().poll(&mut cx)) {
132132
return v;
133133
}
134134

tokio/src/runtime/blocking/task.rs

+8
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ where
2727
.take()
2828
.expect("[internal exception] blocking task ran twice.");
2929

30+
// This is a little subtle:
31+
// For convenience, we'd like _every_ call tokio ever makes to Task::poll() to be budgeted
32+
// using coop. However, the way things are currently modeled, even running a blocking task
33+
// currently goes through Task::poll(), and so is subject to budgeting. That isn't really
34+
// what we want; a blocking task may itself want to run tasks (it might be a Worker!), so
35+
// we want it to start without any budgeting.
36+
crate::coop::stop();
37+
3038
Poll::Ready(func())
3139
}
3240
}

tokio/src/runtime/enter.rs

+2-2
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,7 @@ cfg_blocking_impl! {
9898
let mut f = unsafe { Pin::new_unchecked(&mut f) };
9999

100100
loop {
101-
if let Ready(v) = f.as_mut().poll(&mut cx) {
101+
if let Ready(v) = crate::coop::budget(|| f.as_mut().poll(&mut cx)) {
102102
return Ok(v);
103103
}
104104

@@ -130,7 +130,7 @@ cfg_blocking_impl! {
130130
let when = Instant::now() + timeout;
131131

132132
loop {
133-
if let Ready(v) = f.as_mut().poll(&mut cx) {
133+
if let Ready(v) = crate::coop::budget(|| f.as_mut().poll(&mut cx)) {
134134
return Ok(v);
135135
}
136136

tokio/src/runtime/shell.rs

+1-1
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ impl Shell {
4646
let mut cx = Context::from_waker(&self.waker);
4747

4848
loop {
49-
if let Ready(v) = f.as_mut().poll(&mut cx) {
49+
if let Ready(v) = crate::coop::budget(|| f.as_mut().poll(&mut cx)) {
5050
return v;
5151
}
5252

0 commit comments

Comments
 (0)