> For the complete documentation index, see [llms.txt](https://gyrinx.gitbook.io/gyrinx/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://gyrinx.gitbook.io/gyrinx/explanation/task-framework.md). # Task Framework Architecture This document explains the design decisions, message flow, and infrastructure provisioning strategy for Gyrinx's background task system. The basic concepts from Tasks are new in Django 6.0. We have a Pub/Sub backend for asynchronous execution, with support for scheduled tasks via Cloud Scheduler. The [`TaskRoute`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/route.py) is custom to Gyrinx, providing configuration options for retries and scheduling. ## Why Pub/Sub? We like using Google Cloud Platform built-ins in general. Pub/Sub is fully managed, and works well with Cloud Run (pushing to an endpoint), Cloud Scheduler, and IAM. It's fairly cost-effective per message, with no idle infrastructure costs, and we don't have separate worker processes to manage. ## How we use Pub/Sub * Fire-and-forget publishing: We don't wait for Pub/Sub confirmation to keep request latency low. This means message loss is theoretically possible on network failures. We could introduce silent retries in the future. See [`PubSubBackend.enqueue()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/backend.py). * No result storage: For now, tasks that need to report results should write to the database directly. * No task priorities: All tasks use the same delivery mechanism. For true priority support, we'd need separate topics. ## Message Flow ### On-Demand Task Execution ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Python Code │ │ Pub/Sub │ │ Cloud Run │ │ │ │ │ │ │ │ task.enqueue() ├────►│ Topic ├────►│ /tasks/pubsub/ │ │ │ │ │ │ │ │ │ └─────────────────┘ │ ▼ │ │ ▼ │ │ Subscription │ │ Task Function │ │ (push) │ │ │ └─────────────────┘ └─────────────────┘ ``` 1. Enqueue: Python code calls `task.enqueue(...)` with arguments 2. Publish: [`PubSubBackend.enqueue()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/backend.py) serialises arguments to JSON and publishes to the task-specific topic 3. Deliver: The push subscription sends an HTTP POST to `/tasks/pubsub/` with the message 4. Verify: The handler verifies the OIDC token to ensure the request came from Pub/Sub (see [`_verify_oidc_token()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/views.py)) 5. Route: The handler looks up the task by `task_name` in the [registry](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/registry.py) 6. Execute: The task function runs with the deserialised arguments 7. Acknowledge: HTTP 200 acknowledges the message; other codes trigger retry The push handler is [`pubsub_push_handler()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/views.py). ### Scheduled Task Execution ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Cloud Scheduler │ │ Pub/Sub │ │ Cloud Run │ │ │ │ │ │ │ │ Cron trigger ├────►│ Topic ├────►│ /tasks/pubsub/ │ │ │ │ │ │ │ │ │ └─────────────────┘ │ ▼ │ │ ▼ │ │ Subscription │ │ Task Function │ │ (push) │ │ │ └─────────────────┘ └─────────────────┘ ``` Scheduled tasks work identically to on-demand tasks, except: 1. Cloud Scheduler triggers based on the cron expression 2. Scheduler publishes a pre-configured message to the same topic 3. The rest of the flow is identical This design means scheduled and on-demand invocations use the same code path. ## Infrastructure Provisioning ### When Does Provisioning Run? Provisioning runs automatically during Cloud Run startup when the `K_SERVICE` environment variable is present. Cloud Run sets this automatically. See [`TasksConfig.ready()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/apps.py). ```python # In gyrinx/tasks/apps.py def ready(self): if not os.getenv("K_SERVICE"): return # Skip outside Cloud Run provision_task_infrastructure() ``` Provisioning is skipped during: * Local development (no `K_SERVICE`) * Migration commands * Test runs ### What Gets Provisioned? For each task in the [registry](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/registry.py), [`provision_task_infrastructure()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/provisioning.py) creates: 1. Pub/Sub Topic: `{env}--gyrinx.tasks--{module.path.task_name}` 2. Push Subscription: Points to `/tasks/pubsub/` with OIDC authentication 3. Cloud Scheduler Job (if scheduled): Publishes to the topic on the cron schedule ### Idempotency Provisioning is (in theory!) idempotent—it's safe to run on every startup: * Topics: Created if missing, skipped if exists * Subscriptions: Created if missing, **updated** if exists (to apply config changes) * Scheduler jobs: Created if missing, **updated** if exists This means you can change retry policies or schedules, and the new configuration is applied on the next deployment. ### Orphan Cleanup When you remove a scheduled task from the registry, the system automatically deletes the corresponding Cloud Scheduler job. See [`_cleanup_orphaned_scheduler_jobs()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/provisioning.py): 1. List all jobs with our naming prefix 2. Compare against current scheduled tasks 3. Delete jobs that no longer have a matching task This prevents stale scheduled jobs from continuing to run after a task is removed. ## Security Model ### OIDC Authentication Push subscriptions use OIDC tokens to authenticate requests. See [`_verify_oidc_token()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/views.py): ``` Pub/Sub → POST /tasks/pubsub/ Authorization: Bearer ``` The handler verifies: 1. The token signature (signed by Google) 2. The audience matches `CLOUD_RUN_SERVICE_URL` 3. The service account email matches `TASKS_SERVICE_ACCOUNT` (optional) In development (`DEBUG=True`), OIDC verification is skipped for convenience. ### Why Not Use Pub/Sub Pull? Pull subscriptions require the application to poll for messages. This doesn't work well with Cloud Run's scale-to-zero model: * Cloud Run instances can be terminated when idle * No background polling possible * Would require always-on worker instances Push subscriptions solve this by having Pub/Sub make HTTP requests, which naturally triggers Cloud Run to scale up. ## Error Handling and Retries ### Response Code Semantics | Code | Meaning | Retry? | | ---- | ----------------- | ---------------------------- | | 200 | Success | No (message acknowledged) | | 400 | Permanent failure | No (prevents infinite retry) | | 403 | Auth failure | No | | 429 | Capacity exceeded | Yes (exponential backoff) | | 500 | Transient failure | Yes (exponential backoff) | ### Backpressure When the database connection pool is exhausted, returning 500 would cause rapid retries that worsen the problem. Instead, we [return 429](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/views.py): ```python try: connection.ensure_connection() except OperationalError as e: if "connection slots" in str(e): return HttpResponse("Database at capacity", status=429) raise ``` Pub/Sub treats 429 like other retriable errors, applying exponential backoff. This gives the database time to recover. ### Retry Policy Each task can configure via [`TaskRoute`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/route.py): * `ack_deadline`: How long before Pub/Sub assumes the task failed (10–600 seconds) * `min_retry_delay`: Minimum wait before retry * `max_retry_delay`: Maximum wait (caps exponential backoff) Pub/Sub uses exponential backoff between min and max delay, doubling the wait time on each retry up to the maximum. ## Development vs Production | Aspect | Development (dev server) | Tests | Production | | ----------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | Backend | [`DatabaseBackend`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/local_backend.py) `worker` mode | `DatabaseBackend` `eager` mode | [`PubSubBackend`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/backend.py) | | Execution | In-process worker threads (async) | Inline on enqueue (sync), or driven by the `task_queue` fixture | Pub/Sub push → Cloud Run (async) | | OIDC verification | Skipped | Skipped | Enforced | | Provisioning | Skipped | Skipped | Automatic on startup | | Topic naming | n/a (no Pub/Sub) | n/a | `prod--gyrinx.tasks--...` | Production stays on `PubSubBackend`; `DatabaseBackend` is a dev/test-only replacement and is never selected in production. ## Local execution (development and tests) Locally there is no Pub/Sub. Instead, [`DatabaseBackend`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/local_backend.py) provides an in-process, durable, Pub/Sub-like queue — no extra process, no GCP dependency. It exists only so that async behaviour (and the codebase's at-least-once/idempotency handling) can be exercised in dev and in tests. ### The shared executor Both the production push handler and `DatabaseBackend` run a task's underlying function through the same [`run_task()`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/executor.py) helper, which fires the identical `task_started` / `task_finished` signals and drives the same [`TaskExecution`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/models.py) bookkeeping. This is deliberate: the local and production delivery paths cannot quietly diverge, so idempotency behaviour exercised locally matches what production does. ### Modes `DatabaseBackend` runs in one of three modes (set via `OPTIONS["mode"]`): * **`eager`** (base default; the test default) — runs the task inline inside `enqueue()`, exactly like Django's `ImmediateBackend`. No queue row is written, so the whole test suite stays synchronous and fast. * **`worker`** (the dev server) — persists the task to the `QueuedTask` table and lets an in-process pool of daemon threads deliver it, with retries, exponential backoff, and visibility leases. `scripts/dev.sh` selects this automatically because `runserver` is on the command line. * **`manual`** (opt-in in tests) — persists the task but delivers nothing automatically; the `task_queue` pytest fixture drives delivery explicitly and can script duplicates, failures, drops, and latency. ### Durability and redelivery In `worker`/`manual` mode a task is a real `QueuedTask` row claimed with `SELECT … FOR UPDATE SKIP LOCKED` under a visibility lease. If a delivery is interrupted (e.g. a `runserver` autoreload kills the daemon threads), the lease lapses and the row is redelivered rather than lost — mirroring Pub/Sub's at-least-once guarantee. ### Redelivery semantics (idempotency) Because delivery is at-least-once, the signal handlers treat the `TaskExecution` record carefully (see [`signals.py`](https://github.com/gyrinx-app/gyrinx/tree/main/gyrinx/tasks/signals.py)): * A redelivery of an already-**SUCCESSFUL** task re-runs the function (idempotency is the task's own responsibility) but leaves the record `SUCCESSFUL`. * A redelivery of a **FAILED** task is treated as a retry: the record is reset so the fresh attempt can record its own outcome. * Re-marking a task that is already terminal as `RUNNING` would be an illegal state transition — the handlers guard against it, which also prevents a production redelivery storm (a raised handler would return 500 and Pub/Sub would retry forever). ### Fault injection The dev server can inject Pub/Sub-like chaos — duplicate delivery, transient failure, message drop, latency — via `TASKS_FAULT_*` environment variables, off by default. See the [reference](/gyrinx/technical-reference/task-framework.md#fault-injection-environment-variables) and the [how-to guide](/gyrinx/how-to-guides/task-framework.md#run-the-dev-server-with-async-tasks-and-fault-injection). ## Future Considerations ### Deferred Execution Pub/Sub supports scheduled message delivery. We could add: ```python task.enqueue(..., run_after=timedelta(hours=1)) ``` This would set the `publishTime` attribute on the message. ### Task Results Currently tasks are fire-and-forget. To support `task.get_result()`: 1. Store results in the database or Cloud Storage 2. Implement `get_result()` in the backend 3. Consider result expiry/cleanup ### Dead Letter Queues Pub/Sub supports dead letter topics for messages that exceed retry limits. This would: 1. Create a dead letter topic per task 2. Configure max delivery attempts 3. Allow inspection/replay of failed messages --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://gyrinx.gitbook.io/gyrinx/explanation/task-framework.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.