Why use Bistro?
Bistro is a toolkit for making services that schedule and execute tasks. It is an engineer’s tool — your clients need to do large amounts of computation, and your goal is to make a system that handles them easily, perfomantly, reliably.
You should consider using Bistro instead of rolling your own solution, because in our experience at Facebook, every heavily used task-execution service eventually has to build out a very similar set of layers:
- Execution shell: Start, wait for, kill tasks
- Worker pool: Maintain the distributed state of what tasks are running where
- Data model: Currently available tasks; resource accounting
- Configuration: Add/remove jobs; tune parameters
- Persistence: If the service crashes, don’t forget task completions
- Scheduling logic: Priorities; dependencies; resource constraints; host placement
- Task logs: Make logs accessible, while controlling disk usage
- Command-line tools: Scriptable, yet friendly
- Web UI: Minimal learning curve; great for visualizing data
- RPC API: When command-line scripting is too slow or clunky
Bistro provides all of the above, and improves continuously, so instead of re-inventing, you can focus on innovating.
