Getting started

Introduction

QueueFlow is a PostgreSQL-native distributed job queue and workflow engine written in Rust. Durable background jobs and DAG workflows on the database you already run.

QueueFlow runs durable background jobs and real DAG workflows directly on PostgreSQL. Any plain PostgreSQL 13 or newer works, with no extensions: RDS, Cloud SQL, Azure, or the Postgres on your laptop. There is no Redis, no broker, and no separate state store. The jobs table is the queue.

Workers claim rows with FOR UPDATE SKIP LOCKED and own them through lease tokens. A retry, a scheduled job, and a cron firing are all just rows whose scheduled_at lies in the future, so delays survive restarts. Handlers can be compiled into the server in Rust, or run in any language over an HTTP worker protocol with the same retry, dead-letter, and workflow semantics.

#What you get

• Durable jobs with priority, per-job retry policies, timeouts, future scheduling with run_at, and idempotent creation through an Idempotency-Key header.
• Workflows as DAGs. Declare steps and depends_on edges. The engine gates each step on its dependencies, threads results through a shared context, applies halt, skip, or continue failure policies, and rejects cycles at creation.
• Durable retries with fixed, linear, or exponential backoff, a delay cap, and jitter. Retries are rows scheduled in the future, not timers in memory.
• A dead-letter queue for exhausted, non-retryable, and unhandled jobs, with an API to inspect and replay them.
• Cron schedules from standard crontab expressions in UTC, deduplicated across servers, with pause and resume.
• Polyglot workers. A lease token protocol over HTTP, with first-party clients for TypeScript, Python, Go, and Rust.
• Multi-tenancy. Tenant isolation on every job, workflow, cron, and dead-letter endpoint, with API keys or HS256 JWTs.
• Observability. Prometheus metrics, structured JSON logs, health and readiness probes, and a Server-Sent Events stream of job status changes.
• A code-generated OpenAPI 3.1 spec. The document is derived from the handlers and ships with every release, so clients never drift from the server.

#How it fits together

                 ┌───────────────────────────────────────────────┐
  your app ──►   │  queueflow serve                              │
  (SDK / curl)   │   REST API :8000   ·   metrics :9090          │
                 │   in-process workers (optional, Rust)         │
                 │   scheduler · janitor · cron                  │
                 └──────────────────────┬────────────────────────┘
                                        │ SQL (FOR UPDATE SKIP LOCKED, LISTEN/NOTIFY)
                                        ▼
                              PostgreSQL 13+  (queueflow.jobs is the queue)
                                        ▲
                                        │ HTTP lease / heartbeat / complete / fail
  remote workers ───────────────────────┘
  (TypeScript, Python, Go, Rust, anything)

One binary, queueflow, runs the API, the in-process workers, or both (--mode api|worker|all). It applies its own migrations on startup. Clients talk to the REST API; remote workers lease jobs from the same API. Everything durable lives in Postgres.

#Components and versions

QueueFlow is MIT licensed. Source for everything is under the elision-labs GitHub organization.

#Where to go next

• New to QueueFlow: follow the quick start, then read Jobs and Workflows.
• Writing a worker in your language: read Remote workers and your SDK page.
• Putting it in production: read Configuration, Authentication and tenants, and Production deployment.
• Curious how it works: read How it works.