Review and convert long running methods to `TimedCancellable`

[tegefaulkes]

Specification

With the TimedCancellable and Cancellable patterns we have a structured way of handling cancellation of long running async tasks. But we've been a bit lax applying it to the code base. Only updating code when doing refactoring or conversions. It's become evident that further review and work needs to be done to apply it to the whole code base.

So what does it need to be applied to. Well the following could take a long time to compelete.

• A promise doing something complex or waiting for something complex to complete.
• An AsyncGenerator returning a potentially unbounded stream of information.
• A normal generator doing the same
• Iterators generally but for the most part these will be bounded by memory limits so we will run into memory problems before time problems generally.

So based on this criteria we need to look through the code and apply cancelability to promises and generators. Futher more the following domains that do general background processing that needs to stop quickly need to be reviewed.

1. Nodes domain.
2. Tasks domain.
3. Discovery domain.
4. notifications domain.

Each of these maintain some degree of background state that can take time to clean up when stopping. In the worst case they will block stopping altogether if proper cancellation isn't supported.

Additional context

• Previous issue tracking similar work Apply timedCancellable across the board to control how long side-effects are allowed to complete #450

Tasks

1. Review code for methods that can have proper cancellation applied to them. Emphasis on promises and generators.
2. Review the above listed domains for general flow of cancellation since these will be the most affected by the changes.

[linear]

ENG-433 Review and convert long running methods to `TimedCancellable`