What’s a task queue, anyway?
The traditional task queue is a service, which accepts tasks — pieces of work to be executed, together with the computational resources they demand. It schedules the jobs according to one of a myriad of heuristics, like fair-share scheduling and executes them on the appropriate hardware.
What does Bistro offer?
Bistro’s core strength lies in scheduling data-parallel jobs (consisting of many related tasks) with constraints on both computational resources (which are usually interchangeable) and on data resources (which are either unique, or have limited replication).
A traditional task queue is a simple reduction of the above problem:
- Each job consists of just 1 task.
- Computational resources are treated normally.
- There are no data resources.
At Facebook, a few deployments have used this very setup, with good success, and thousands of concurrent jobs. In Bistro jargon:
- All jobs use 1 node to produce their 1 task.
- The node has no data resources.
- Any worker setup (remote or local) is appropriate.
When does this make sense? Any of the below options might be good reasons.
- Your deployment needs a flexible resource model. In the aforementioned deployments Bistro resources were used to model GPUs, region locality, entitlements, etc.
- Only some of your jobs are data-parallel. This already works in a kludgy way, but really clean support is coming soon.
- Other aspects of Bistro are a good fit – its APIs or UIs, worker pool, execution model, task log & status handling.
One caveat is that Bistro’s current public release only supports loading job
configurations from a
Our high-performance MySQL-based
ConfigLoader has too many unreleased
dependencies to make it into the initial release. However, we would be glad
to guide you in implementing a high-performance, read-write
— describe your needs in an
issue, and we’ll help you