[DEPR]: XQueue

[ormsbee]

Proposal Date

2025-01-29

Target Ticket Acceptance Date

2025-02-12

Earliest Open edX Named Release Without This Functionality

Ulmo

Rationale

The XQueue service adds a lot of operational complexity for the capability that it provides. While it did originally do more (e.g. push grading), that functionality is no longer used because it was prone to operational issues. The current form of XQueue mostly serves as a holding area for submissions, waiting for xqueue-watcher to poll for them.

A larger deprecation was proposed in #214, but we concluded that the off-site grading functionality that XQueue represented was still valuable to preserve.

Removal

The XQueue service will be discontinued and the xqueue repo will be archived.

The following settings will be removed from edx-platform:

• XQUEUE_INTERFACE
• XQUEUE_WAITTIME_BETWEEN_REQUESTS

We will maintain support for xqueue-watcher and ProblemBlock use of code graders.

When xqueue references are eventually completely removed from edx-platform (post-Teak), we will also execute #192 (MATLAB removal). The two systems are tightly coupled enough where it makes sense to remove them at the same time.

Replacement

Instead of having a separate XQueue service, the edx-submissions repo will be expanded to cover the same use case while running in-process in the LMS. The LMS will also re-implement the REST API that xqueue-watcher interacts with.

The LonCapaSystem service XQueueService will be replaced with a generic XBlock runtime service that is not ProblemBlock-specific.

Deprecation

The LMS will emit warning log messages in Teak whenever the old xqueue-callback URL for ProblemBlocks is hit.

Migration

• No content changes will be required for authored problems. Existing content is intended to work exactly as it always has.
• No code changes will be required for xqueue-watcher.
• Configuration changes will be required for xqueue-watcher, to point it at the LMS instead of at the XQueue service.

In the Teak release, all code grader submissions will be written to the new backend in edx-submissions, and will be available for xqueue-watcher to pull directly from the LMS. If XQUEUE_INTERFACE is also configured for edx-platform, the LMS will additionally post the request to the XQueue service. If the LMS gets a response from the XQueue service, it will delete the queued submission in edx-submissions.

This should allow us a smooth transition path where operators could make their transition by deploying updated configuration for their xqueue-watcher. Operators that are very downtime sensitive could run multiple xqueue-watcher instances concurrently during the transition, falling back to the old behavior if necessary.

Open question:

1. Should duplicate grading attempts for the same submission be "first one wins" or "last one wins"?
2. Do we need to preserve an alias for the old Loncapa XQueueService? This would only need to be done if people have extended their instance ProblemBlocks with custom input/response-types that use that service... which seems unlikely?

Support for posting to the XQueue service would be removed after 2025-08-12, and all code and tests in edx-platform related to xqueue will be removed at that point.

Additional Info

We ultimately want to iterate on this and create a much more flexible off-platform grading capability that can:

1. Make better use of other communications mechanisms, like the message queue backend that openedx-events can utilize.
2. Allows for multiple interfaces, and so accepts things like LTI's grade passbacks, and possibly bulk grading.
3. Understands team-related grading, like ORA does.
4. Remove some of the contortions we make to allow LTI blocks to authenticate non-user requests for grade passback purposes.

But the first step is to preserve the existing functionality while reducing operational complexity of running this system, which this DEPR represents.

Task List

No response

[ormsbee]

Posted to forum: https://discuss.openedx.org/t/deprecation-xqueue-again/14864