queue: initial queue service

[delan]

currently our self-hosted runner system falls back to github-hosted runners if there’s no available capacity at the exact moment of the select runner request. this is suboptimal, because if the job would take 5x as long on github-hosted runners, then you could wait up to 80% of that time for a self-hosted runner and still win.

this patch implements a new global queue service that allows self-hosted runner jobs to wait for available capacity. the service will run on one server for now, as a single queue that dispatches to all available servers, like any efficient supermarket. queueing a job works like this:

1. POST /profile/<profile_key>/enqueue?<unique_id>&<qualified_repo>&<run_id> (tokenful)
   or POST /enqueue?<unique_id>&<qualified_repo>&<run_id> (tokenless) to enqueue a job.

   • both endpoints return a random token that is used to authenticate the client in the next step.
   • the tokenless endpoint validates that the request came from an authorised job, using an artifact.
   • the request is rejected if no servers are configured to target non-zero runners for the requested profile, because we may never be able to satisfy it.
   • there are no limits to queue depth (at least not yet), but clients probably have better knowledge of the nature of their job anyway, and in theory, they could use that knowledge to decide how long to wait (see below).

2. POST /take/<unique_id>?<token> to try to take the runner for the enqueued job. once capacity is available, this endpoint is effectively proxied to POST /profile/<profile_key>/take on one of the underlying servers.

   • if the client failed to provide the correct token from the previous step, the response is HTTP 403.
   • if the unique id is unknown, because it expired or the queue service restarted, the response is HTTP 404.
   • if there’s no capacity yet, the response is HTTP 503. repeat after waiting for ‘Retry-After’ seconds.
   • if taking the runner was successful, the response is HTTP 200, with the runner details as JSON.
   • if taking the runner was somehow unsuccessful (bug), the response is HTTP 200, with null as JSON. this sucks, to be honest, but it was also true for the underlying monitor API.
     • when we fix this, we should be careful about curl --retry.
   • clients are free to abandon a queued job without actually taking it, by doing nothing for 30 seconds. for now, the runner-select action client abandons a queued job if it has been waiting for one hour.

i’ve added a “self-test” workflow that can be manually dispatched to test the new flow (e.g. ok 1, ok 2, ok 3, unsatisfiable, unauthorised). you can also play around with this locally by spinning up a monitor and a queue on your own machine, then sending the requests by hand (so three separate terminals):

• $ cargo build && sudo IMAGE_DEPS_DIR=$(nix eval --raw .\#image-deps) LIB_MONITOR_DIR=. $CARGO_TARGET_DIR/debug/monitor
• $ cargo build && sudo IMAGE_DEPS_DIR=$(nix eval --raw .\#image-deps) LIB_MONITOR_DIR=. $CARGO_TARGET_DIR/debug/queue
• $ unique_id=$RANDOM; curl --fail-with-body -sSX POST --retry-max-time 3600 --retry 3600 --retry-delay 1 'http://192.168.100.1:8002/take/'"$unique_id"'?token='"$(curl --fail-with-body -sSX POST --oauth2-bearer "$SERVO_CI_MONITOR_API_TOKEN" 'http://192.168.100.1:8002/profile/servo-windows10/enqueue?unique_id='"$unique_id"'&qualified_repo=delan/servo&run_id=123')"

[delan]

for this to work reliably and not prematurely dequeue jobs, the monitors should only accept requests from the queue, and stop accepting requests from workflows. i think we can do that by disabling the tokenless select endpoint in the monitor.

[delan]

the object/null comments belong on the take request, not on the enqueue request

[delan]

this should be ready for review! @sagudev @jschwe