Run immediately in the same process

[chaporgin]

Hi, thank you for the library. I have been kicking tires here and there for my idea: provide an ability to execute the inserted job immediately in the same process that has just closed the transition scheduling the job. Yes, it's tricky because the transaction might span further after the job was inserted, so it's impossible to start it in the Insert{,Many} family of methods. But perhaps a dedicated method like Client.Start(ctx, jobId int64) could do the job if a user explicitly calls it after committing the transition. How hard would it be to implement, WDYT? The motivation is to reduce latencies and not rely too much on NOTIFY/LISTEN and polling. The worst thing (correct me if I'm wrong here) that could happen is that the client learns that the job written in the database has already been locked. It could return a dedicated error to the user saying that "the row has been locked and the job is being executed by some other process already", which the client should be ready for.

[brandur]

@chaporgin OOC: It seems like if you're trying to guarantee an immediate run, the more obvious path would be to just take the work and run it right in band with the operation that would've enqueued it. i.e. Bypassing the job queue completely. Is the purpose of the job queue in this instance to get tracking on it in case of failure?

It does seem like this would be doable one way or the other, although it'd need a separate locking query, and those are always a little gnarly.

Start's already the name for starting the client, so we'd want to find a more suitable name for the new function, probably involving the word "job".

[bgentry]

Can you provide some context on what job start latencies you're looking to achieve?

[chaporgin]

I want to start the job execution within 10 milliseconds.

[raymondregrello]

Out of curiosity, what's the specific use case you have in mind?

Assuming you can make the operation idempotent (which is desirable anyways if you want durable execution), you could do something like this in your caller:

1. Enqueue the job (include an idempotency key in the payload if necessary)
2. Start doing the work

When the job eventually runs, it will just do the same work again.

[chaporgin]

Out of curiosity, what's the specific use case you have in mind?

I want to avoid inter-process communication to start background tasks as soon as possible. Currently, an in-house queue implementation starts the async task in the process that persists the task into the database. That gives close-to-zero latency between scheduling and execution.

So it would be nice to be able to execute immediately with the River too. When thinking of Postgres's "listen/notify", I perceive it as a signaling mechanism. It would be nice if something like a local-process signaling mechanism existed, being the fastest one if one is choosing from 1) this one, 2) "listen/notify" and 3) polling. It could be a golang channel notifying about, "Hey, try to SELECT FOR UPDATE SKIP LOCKED this ID" which, if it fails, is not an error, following the standard path that river workers do now - just skip it.

Does it make sense to you? I am not suggesting implementing that because it is not an everyday use case, but I would like your opinions.

Because otherwise, the client gets FetchCooldown delay between scheduling and execution in the worst case scenario.

Trying to lock per-row would significantly reduce the throughput because currently, SELECT FOR UPDATE is done in a batching manner.

[bgentry]

@chaporgin I'm wondering if you've considered tuning your settings so that FetchCooldown is very low, but FetchPollInterval remains higher. The FetchCooldown controls the minimum duration between fetches, and it's also used to set a debounce period on emitting insert notifications. If you keep it super low, your inserted jobs would be picked up very quickly by an available worker that receives the insert notification. There are some tradeoffs here in terms of load but it's a thing you could try out today without any changes to the library.

The broader idea here seems feasible to a degree, but it would be complex to implement because it would require coordination between goroutines that currently are not designed to coordinate (the "producer" that fetches and spawns jobs to be worked, and the insertion routine which is not otherwise connected to the client's other internal goroutines.