A worker has twenty free slots. Ten thousand jobs are waiting. How many should it take?
Twenty sounds reasonable, until those jobs all belong to one customer, call an API whose shared budget is exhausted, or contain the same payload that killed the last three workers.
Queue depth tells you work exists. Worker capacity tells you there is room. Neither tells you which work is permitted to run.
I built headgate around that question. It is a distributed job queue for Go and Rust, with PostgreSQL, Redis, and MySQL backends. Its central design decision is to evaluate admission policy inside the store, in the same atomic operation that claims the work.
Capacity is only half the question
Imagine a service that processes customer imports. One customer uploads thousands of records. Two others upload a handful. Each record needs a call to the same external API.
A worker can limit its own concurrency. But if every worker independently permits ten requests, adding workers also increases the pressure on the upstream service. The budget belongs to the fleet; a process-local counter cannot enforce it.
Putting a shared limiter inside the handler addresses the budget, but it leaves another question: why claim the job before discovering that it cannot run? That job now occupies a lease, and potentially worker capacity, while other work could make progress.
headgate asks the store a different question:
Given the fleet's policy state and my available capacity, what may I run?
The store evaluates the applicable budgets, concurrency ceilings, tenant fairness, and quarantine state. For admitted work, policy accounting and the lease commit together. Work held back by a waiting policy remains unleased.
That boundary matters. If checking a budget and spending it are separate operations, two workers can both see the last available token. If claiming a job and creating its lease are separate operations, a crash between them can leave work with no recovery deadline.
PostgreSQL expresses the gate as an atomic statement, Redis as a Lua script, and MySQL through a transaction. The implementation differs; the contract must survive those differences.
Fairness starts with the candidates
One of the most useful failures came before the fairness calculation.
An early candidate query drew a bounded window from the queue, then applied fairness to the rows it found. That sounds sensible. It also means a sufficiently large tenant can fill the entire window. The quiet tenants never become candidates, so the scheduler never gets a chance to serve them.
The regression fixture uses 5,000 jobs from one tenant. In the documented failing case, the flat window returned three jobs from one tenant where the intended result was nine across three tenants. A small fixture had made the design look correct.
The fix was to draw candidates per partition: lateral queries in PostgreSQL and per-partition sorted sets in Redis. Fairness has to shape the work the gate considers, not just the order in which it examines an already biased sample.
It must also be work-conserving. Once other tenants have had their opportunity, a busy tenant should be able to use spare capacity. Leaving workers idle to punish a customer would turn isolation into a throughput bug.
The clock is part of the policy
Another early mistake was accepting the current time from the caller.
A token bucket refills according to elapsed time. If a worker reports a time sixty seconds ahead, it can manufacture sixty seconds of refill. In the recorded regression, a bucket with a limit of five admitted ten jobs in the same real second.
The same mistake affects leases. A worker's clock can push expiry too far into the future or make it arrive too soon. Either way, recovery becomes dependent on which machine happened to claim the job.
The gate now uses store time. Workers already share the store's policy state; they should share its clock for those decisions too.
A crash needs a different answer from an error
A handler returning an error and a process disappearing mid-job are different signals.
An ordinary error may mean a dependency is temporarily unavailable. Repeated crashes around the same payload may mean that retrying it keeps killing workers. Combining both into one attempt counter loses the information needed to distinguish those cases.
headgate tracks returned failures and crashes separately. Quarantine correlates crashes by fingerprint, derived from the task kind and payload, so another copy of the same work cannot simply restart the crash cycle under a new job ID. This is exact fingerprint correlation, not a classifier that discovers arbitrary families of similar bad payloads.
The distinction also applies to rate limiting. A handler that receives an upstream limit response can return a rate-limited outcome, requeueing without consuming an ordinary failure attempt. Waiting for permission is normal queue behavior.
The operator should be able to ask why
Once dequeue is an admission decision, a dashboard that only shows queue depth is incomplete.
A waiting job could be delayed by a paused queue, exhausted budget, or concurrency ceiling. Those require different interventions. Adding workers will not create more upstream tokens.
headgate exposes GET /jobs/{id}/admission so the operator can inspect the policy explanation. It is a view of current state, not a reservation or a promise that the next worker will claim that job. The fleet can change between inspection and admission.
This endpoint belongs alongside the gate because explaining a decision is part of operating it. Payloads stay excluded from ordinary inspection unless explicitly requested.
The test suite had to earn its claims too
Having a conformance suite did not automatically make the guarantees real.
During one mutation-testing round, removing the fencing token from the acknowledgement identity check still left all 462 assertions green. A stale acknowledgement could complete a job, and the suite did not notice. The tests exercised successful completion without forcing the displaced-holder case that fencing exists to reject.
That changed how I treated evidence. The capability register now ties claims to named tests or executed scenarios. A test inventory catches disappearing tests. Mutation checks ask a harder question: does a faithful violation of the guarantee make the suite fail?
Those mechanisms improve the evidence; they do not establish that every policy combination or failure mode has been covered. The evidence ledger records those limits as well.
What this design costs
Admission does more work than fetching rows. It reads shared policy state and updates accounting on the hot path. Fairness also changes observable ordering: a quiet tenant can get service ahead of older work from a flooding tenant.
The performance comparison has to be fair too. A baseline that updates rows but returns no job envelope cannot execute the work it claims. headgate's admission benchmark compares against a functional dequeue that returns and decodes the same envelope. PostgreSQL also has a narrow policy-free fast path, with an atomic applicability check and fallback to the full gate.
I am not claiming that fleet-wide limits, fairness, or quarantine originated here. The project's prior-art audit corrected those claims early. The reason I kept building was the combination: shared admission policy, inspectable decisions, and Go and Rust workers measured against a common contract across backend choices.
The question I want headgate to answer reliably is simple: there is work waiting and capacity available, so what may run now?
The source, architecture, and capability evidence are available together. The regression numbers above are recorded development results, not new measurements from this article.