Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 47 additions & 1 deletion apps/api/src/locales/@vitnode/core/en.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
"@vitnode/core:system": "System",
"@vitnode/core:system:can_view": "View integrations",
"@vitnode/core:system:can_send_test_email": "Send test email",
"@vitnode/core:queue": "Queue Tasks",
"@vitnode/core:queue:can_view": "View queue tasks",
"@vitnode/core:staff_moderators": "Staff: Moderators",
"@vitnode/core:staff_moderators:can_view": "View moderators list",
"@vitnode/core:staff_moderators:can_create": "Create moderators",
Expand Down Expand Up @@ -274,7 +276,8 @@
},
"advanced": {
"title": "Advanced",
"cron": "Cron Jobs"
"cron": "Cron Jobs",
"queue": "Queue Tasks"
}
}
},
Expand Down Expand Up @@ -302,6 +305,26 @@
}
}
}
},
"queue": {
"title": "Queue Tasks",
"desc": "Monitor background tasks queued for processing by the cron worker.",
"list": {
"name": "Task",
"queue": "Queue",
"status": "Status",
"attempts": "Attempts",
"availableAt": "Available At",
"createdAt": "Created At",
"lastError": "Last Error",
"statusFilter": "Status"
},
"status": {
"pending": "Pending",
"processing": "Processing",
"completed": "Completed",
"failed": "Failed"
}
}
},
"user": {
Expand Down Expand Up @@ -544,12 +567,35 @@
"insecure": "Using the default CRON_SECRET — set a secure one in production.",
"not_configured": "No cron adapter configured — jobs won't run automatically.",
"jobs": "{count, plural, =0 {No jobs scheduled} one {# job scheduled} other {# jobs scheduled}}"
},
"queue": {
"title": "Queue Tasks",
"desc": "Background jobs enqueued in the database and drained by the cron worker.",
"tasks": "{count, plural, =0 {No tasks registered} one {# task registered} other {# tasks registered}}",
"queued": "{pending} pending · {processing} running"
}
}
},
"debug": {
"title": "Debug Panel",
"desc": "Check logs, errors, and other debug information.",
"queue": {
"title": "Queue Tasks",
"empty": "No active queue tasks.",
"counts": {
"pending": "Pending",
"processing": "Processing",
"completed": "Completed",
"failed": "Failed"
},
"list": {
"name": "Task",
"queue": "Queue",
"status": "Status",
"attempts": "Attempts",
"availableAt": "Available At"
}
},
"actions": {
"clear_cache": {
"label": "Clear Cache",
Expand Down
180 changes: 180 additions & 0 deletions apps/docs/content/docs/dev/advanced/queue.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
---
title: Queue Tasks
description: Run one-off background work through a database-backed task queue, drained by cron, with retries and optional Redis coordination.
---

Cron jobs handle **recurring** work on a schedule. Queue tasks handle **one-off**
background work you enqueue on demand — "send this email later", "process this
import", "retry this webhook". Tasks are stored in the database (`core_queue`)
and drained every minute by a built-in cron job, so a slow task never blocks the
request that scheduled it.

Queue tasks build on the [cron system](/docs/dev/cron): the `process-queue` job
runs on the same per-minute tick, so **you need a cron adapter configured** (for
example [Node CRON](/docs/dev/cron/node-cron)) for tasks to be processed
automatically.

## How it works

```text
c.get("queue").dispatch({ name, payload }) ─► INSERT core_queue (status = pending)

cron tick "* * * * *" ─► process-queue job
1. optional Redis lock — one instance drains per tick when Redis is configured
2. claim a batch (UPDATE … status = processing … FOR UPDATE SKIP LOCKED)
3. run each task's handler, matched by pluginId:name
4. success → completed · error → retry with backoff → failed after maxAttempts
5. prune completed/failed rows older than 7 days
```

Concurrency is safe by default: batches are claimed with Postgres
`FOR UPDATE SKIP LOCKED`, so many instances (or many ticks) can drain the queue
at once without ever running the same task twice.

## Defining a task

A task is a named handler, registered like a cron job. The handler receives the
request context and the JSON `payload` the task was dispatched with.

<Steps>
<Step>
### Create the task file

```ts title="modules/newsletter/tasks/send-digest.task.ts"
import { buildQueueTask } from "@vitnode/core/api/lib/queue";

export const sendDigestTask = buildQueueTask({
name: "send-digest",
description: "Send the weekly digest email to a user",
// Optional — defaults to 3
maxAttempts: 5,
handler: async (c, payload) => {
const { userId } = payload as { userId: number };

// ...do the work; throwing schedules a retry with backoff
await c.get("log").debug(`Sending digest to user ${userId}`);
},
});
```

<Callout type="info" title="Payloads are plain JSON">
`payload` is typed as `Record<string, unknown>` because it round-trips through
the database as JSON. Narrow it inside the handler (a cast, or validate it with
zod) before use.
</Callout>

</Step>
<Step>
### Register it in a module

Pass your tasks to `queueTasks` — the same module builder that carries
`cronJobs`.

```ts title="modules/newsletter/newsletter.module.ts"
import { buildModule } from "@vitnode/core/api/lib/module";
import { CONFIG_PLUGIN } from "@/config";

// [!code ++]
import { sendDigestTask } from "./tasks/send-digest.task";

export const newsletterModule = buildModule({
pluginId: CONFIG_PLUGIN.pluginId,
name: "newsletter",
routes: [],
// [!code ++]
queueTasks: [sendDigestTask],
});
```

</Step>
</Steps>

## Enqueuing work

Dispatch a task from any route or model with `c.get("queue")`. The `name` must
match a registered task; the row is picked up on the next cron tick.

```ts title="modules/newsletter/routes/subscribe.route.ts"
export const subscribeRoute = buildRoute({
pluginId: CONFIG_PLUGIN.pluginId,
route: {},
handler: async c => {
await c.get("queue").dispatch({
name: "send-digest",
payload: { userId: c.get("user")?.id },
});

return c.json({ queued: true });
},
});
```

### dispatch options

import { TypeTable } from "fumadocs-ui/components/type-table";

<TypeTable
type={{
name: {
description: "Name of a registered queue task to run.",
type: "string",
},
payload: {
description: "JSON data passed to the handler.",
type: "Record<string, unknown>",
default: "{}",
},
queue: {
description: "Logical queue/channel name, for grouping and filtering.",
type: "string",
default: '"default"',
},
priority: {
description: "Higher values are claimed first within a batch.",
type: "number",
default: "0",
},
maxAttempts: {
description: "How many times to try before marking the task failed.",
type: "number",
default: "3",
},
availableAt: {
description:
"Earliest time the task may run — set it in the future to delay work.",
type: "Date",
default: "now",
},
}}
/>

## Retries and backoff

If a handler throws, the task is retried with an exponential backoff
(`10s`, `20s`, `40s`, … capped at one hour) until `maxAttempts` is reached, after
which its status becomes `failed` and the error is stored in `lastError`. A task
with no registered handler fails immediately.

Completed and failed rows are pruned automatically after 7 days, so the table
does not grow without bound.

## Optional: Redis coordination

The queue works with just Postgres. When you run **multiple instances**, adding
[Redis](/docs/dev/advanced/redis) lets a single instance drain the queue per
tick (an advisory lock), which reduces database contention. Correctness never
depends on it — `FOR UPDATE SKIP LOCKED` already prevents double processing.

There is nothing extra to enable: set `REDIS_URL` (see the
[Redis guide](/docs/dev/advanced/redis)) and the queue uses it automatically.
Without Redis, every instance simply claims its own disjoint batch.

## Monitoring in the AdminCP

- **Core → Advanced → Queue Tasks** — the full, paginated list of tasks with a
status filter, attempt counts, and the last error. Requires the
`queue:can_view` staff permission.
- **Debug Panel** — a live view of currently active (pending/processing) tasks
plus per-status counts.
- **Core → System → Integrations** — a card summarizing registered task handlers
and the number of pending/running tasks.
20 changes: 20 additions & 0 deletions apps/docs/migrations/0006_greedy_apocalypse.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
CREATE TABLE "core_queue" (
"id" serial PRIMARY KEY NOT NULL,
"pluginId" varchar(100) NOT NULL,
"name" varchar(100) NOT NULL,
"queue" varchar(100) DEFAULT 'default' NOT NULL,
"status" varchar(20) DEFAULT 'pending' NOT NULL,
"payload" jsonb DEFAULT '{}'::jsonb NOT NULL,
"priority" integer DEFAULT 0 NOT NULL,
"attempts" integer DEFAULT 0 NOT NULL,
"maxAttempts" integer DEFAULT 3 NOT NULL,
"availableAt" timestamp DEFAULT now() NOT NULL,
"reservedAt" timestamp,
"lastError" text,
"createdAt" timestamp DEFAULT now() NOT NULL,
"updatedAt" timestamp NOT NULL,
"completedAt" timestamp
);
--> statement-breakpoint
ALTER TABLE "core_queue" ENABLE ROW LEVEL SECURITY;--> statement-breakpoint
CREATE INDEX "core_queue_status_available_at_idx" ON "core_queue" USING btree ("status","availableAt");
Loading
Loading