A saved schedule is a promise about later. The user is gone, the HTTP request is over, and the process that accepted the form may have restarted twice before the next run is due.
For an AI agent, the promise is heavier than "call a function every hour." The scheduled work creates product state: a chat session, a stream, a queued model run, and eventually a persisted answer. If that handoff duplicates, disappears, or runs under the wrong user's scope, the bug is visible in the product.
Limerence handles that promise by splitting the lifecycle. The application database stores the automation. Pg-boss stores the cron registration. A short queue turns cron ticks into chat sessions. A second queue runs the long LLM job. The split is useful, but it also leaves exact failure windows the system has to name honestly.
A Cron String Becomes a Row and a Schedule
An automation starts as five fields: agent, name, prompt, schedule, and enabled flag. The API validates the cron as a 5-field expression, verifies that the target agent belongs to the caller's team, writes an Automation row, and only then registers Pg-boss when enabled is true.
That gives the system two sources of state with different jobs. The Automation row is the product record. It remembers the agent, creator, prompt, schedule, enabled flag, timestamps, and links to chat sessions created by the automation. Pg-boss holds the delivery registration that will later emit work.
The dual-write matters. Creating an automation writes the row and then schedules Pg-boss in a separate await. Updating an automation first unschedules the old key, updates the row, and schedules again if the resulting row is enabled. Those operations are simple in the current implementation, and they are not one database transaction.
The Schedule Key Is the Automation Id
The scheduling function is small enough to show in full:
export async function scheduleAutomation(
automationId: string,
cron: string,
): Promise<void> {
await boss.schedule(
AUTOMATION_RUN_QUEUE,
cron,
{ automationId } satisfies AutomationRunJobData,
{ key: automationId, singletonKey: automationId, tz: 'UTC' },
);
}export async function scheduleAutomation(
automationId: string,
cron: string,
): Promise<void> {
await boss.schedule(
AUTOMATION_RUN_QUEUE,
cron,
{ automationId } satisfies AutomationRunJobData,
{ key: automationId, singletonKey: automationId, tz: 'UTC' },
);
}The key is what makes schedule replacement and removal addressable. When the route disables, updates, or deletes an automation, it can call unschedule with the same automation id instead of searching by cron string or prompt. When registration succeeds, one saved automation maps to one Pg-boss schedule identity.
The singletonKey belongs to queue execution, not schedule lookup. If Pg-boss tries to enqueue another automation-run job while a job with the same automation id is already live, the singleton boundary is the guard. The same id appears twice because schedule identity and active-job identity both need to be stable across process restarts.
The Cron Worker Re-reads the Row
A scheduled job payload carries only { automationId }. That is deliberate. The cron registration does not freeze the prompt, enabled flag, agent id, or user id into every future tick.
When the worker receives the tick, it loads the current automation row. Missing rows return. Disabled rows return. Rows without a schedule return. The run uses the latest row state, but the tick itself may have been emitted before a schedule change.
That re-read closes a real race. A user can disable an automation after Pg-boss has already emitted a job but before the worker has picked it up. Route-time unscheduling cannot erase a job that already exists, so the worker repeats the enabled check at execution time.
runs.begin Turns a Tick Into a New Chat Session
The worker does not run the model. It calls the same run facade used by other entry points, passing the automation's agent, creator, prompt, and session attribution.
await runs.begin({
agentId: automation.agentId,
userId: automation.createdByUserId,
context: 'work',
session: { automationId: automation.id },
prompt: automation.prompt,
});await runs.begin({
agentId: automation.agentId,
userId: automation.createdByUserId,
context: 'work',
session: { automationId: automation.id },
prompt: automation.prompt,
});runs.begin generates a fresh chatId, creates a ChatSession, registers the stream, and sends a chat job. The scheduled tick has become a normal chat run with automation attribution.
◆Key Takeaway
A scheduled tick starts a chat by enqueueing one. The LLM work does not happen
inside the cron worker, so reliability depends on the handoff between
automation-run and chat-run.
- 1Pg-boss emits an
automation-runjob with{automationId}. - 2The worker reads the current
Automationrow by id. - 3
runs.begincreates a freshChatSessionandchatId. - 4The stream is registered before model work begins.
- 5
chat-runis enqueued with the prompt, user id, chat id, and stream id.
This path corrects a stale mental model. The cron worker is not occupied for the lifetime of the agent answer. It finishes after the chat job is queued, while the separate chat worker later calls the agent runner and persists streamed output.
automation-run Retries the Handoff, chat-run Runs the Model
The two queues have different policies because they protect different work.
automation-run
upsertQueue(AUTOMATION_RUN_QUEUE, {
retryLimit: 2,
expireInSeconds: 1800,
policy: 'exclusive',
});upsertQueue(AUTOMATION_RUN_QUEUE, {
retryLimit: 2,
expireInSeconds: 1800,
policy: 'exclusive',
});Short handoff job. It re-reads the automation and asks runs.begin to
create and enqueue a chat. A transient failure can be retried twice.
chat-run
upsertQueue(CHAT_RUN_QUEUE, {
retryLimit: 0,
expireInSeconds: 1800,
heartbeatSeconds: 30,
policy: 'exclusive',
});upsertQueue(CHAT_RUN_QUEUE, {
retryLimit: 0,
expireInSeconds: 1800,
heartbeatSeconds: 30,
policy: 'exclusive',
});Long LLM job. It owns model execution and stream persistence. It has a heartbeat, but it does not retry failed model execution.
The retry boundary is narrow on purpose. The scheduler queue is retryable because its expected work is enqueueing, not generating an answer. The chat queue is not retryable because a model stream is visible durable output, and replaying it can create a worse user experience than surfacing failure.
runs.begin also has cleanup for the enqueue boundary. If it creates a chat session but cannot enqueue the chat job, it deletes the new session and discards the stream. A failed handoff should not leave a history row that points to work no worker will ever run.
The scheduled path makes that failure operationally quiet. An EnqueueFailedError from an automation run is logged and swallowed by the worker; the manual run endpoint maps the same failure class to a 503. Both paths use the same run facade, but only the manual request has a caller waiting for an HTTP response.
The Chat Singleton Protects One Chat, Not One Automation
The second queue has its own singleton behavior, but it keys on chatId. Since runs.begin creates a new UUID for every scheduled tick, two different ticks from the same automation do not collide on the chat singleton.The singleton is still useful. It blocks duplicate sends for one chat session, which is a different invariant from "only one occurrence of this automation may run."
That distinction is easy to miss. The automation id is stable across the saved schedule. The chat id is stable only inside one created chat session. Once a cron tick becomes a new chat, the overlap question has moved from "is this automation already running?" to "is this exact chat already running?"
The current design allows overlap at the automation level. An every-minute automation can enqueue another chat while the prior chat is still answering, because the long work lives on chat-run and the singleton key there changes every tick. That is a reasonable throughput choice, but it is not per-automation mutual exclusion.
Create and PATCH Can Drift Away From Pg-boss
Create has one drift window. The API writes the Automation row, then calls the scheduler. A crash or Pg-boss error between those operations can leave an enabled automation row with no corresponding schedule.
PATCH has the reverse shape. The route unschedules the old automation id, updates the row, then schedules again if the new row is enabled. If the process dies after unschedule and before the final schedule call, the row can show the new enabled state while Pg-boss has no live cron registration. If it dies after the row update but before scheduling, Pg-boss can be behind the product state.
This is the cost of keeping the product record and scheduler registration separate. The design gets a clean product model and native Pg-boss cron delivery, but it does not currently get atomicity across both writes.
A Crash After Chat Enqueue Can Create Two Scheduled Chats
The sharpest failure window sits after the handoff succeeds but before the scheduler job is recorded as complete. Suppose runs.begin creates a session and enqueues chat-run. Then the worker dies before Pg-boss marks the automation-run job done.
Pg-boss can retry the automation job because automation-run has retryLimit: 2. The retry re-reads the automation and calls runs.begin again. Since runs.begin generates a fresh chat id every time, the second attempt can create a second scheduled chat for the same cron occurrence.
The missing object is a scheduled occurrence id. AutomationRunJobData has automationId. ChatRunJobData has chat state, user state, stream state, and message state. Neither payload carries "the 2026-05-19 09:00 UTC occurrence of automation X" as a durable idempotency key. Without that key, the retry cannot naturally prove that the first handoff already succeeded.
Startup Recovery Fails Streams, But It Does Not Rebuild Schedules
Startup recovery exists for the chat side of the handoff. On backend boot, stale active chat-run jobs are failed, and orphaned queued or running streams are marked failed unless a recoverable Pg-boss job still exists.
That recovery is about honest stream state. A chat that no worker can finish should not remain forever queued or running in the UI. Marking it failed is less magical than replaying model execution, but it matches the queue policy: long chat execution is failure-handled, not replayed.
Schedule recovery is a separate problem, and the findings did not find it. Startup creates queues, schedules staging cleanup, runs stream recovery, and registers workers. It does not scan enabled automations and re-register missing Pg-boss schedules.
UTC Is the Only Scheduling Timezone
Time is another explicit boundary. The schedule validator accepts a raw 5-field cron expression. The scheduler passes tz: 'UTC'. The automation model has no timezone column.
nextRunAt follows the same assumption. It is computed when automations are read, using the stored cron expression under UTC behavior, rather than stored as a reconciled scheduler fact.
That makes the product behavior simple and predictable for UTC schedules. It also means "9am every weekday" means 9am UTC, not 9am in the user's locale. A per-automation timezone would need schema, validation, UI, scheduling, and display changes. The current mechanism has no place to store it.
Run History Is Chat History
The run ledger is a chat-session query. A ChatSession can point back to an automation id, and the automation runs endpoint reads sessions by that link, ordered newest-first.
There is no separate AutomationRun model in the current schema. There is no per-run status enum attached to an automation occurrence. There is also no delivery target model for Slack, email, or another destination in this automation record. The observable history is "which chats were created by this automation?"
That model is enough for the current product surface: manual runs return chatId and sessionId, and automation history can list created chats. It is thinner than an operations ledger. ChatSession.automationId is optional and is nulled when the automation is deleted, so the link is attribution, not a durable occurrence record. A skipped cron tick, a schedule drift, or a failed handoff before session creation does not naturally appear as its own run record.
The Honest Boundary Is No Reconciliation and No Occurrence Id
The design gets several things right by giving each boundary one job. Team access is enforced through the agent relation before rows are read or written. Disabled automations are guarded at route time and worker time. Pg-boss schedule identity is stable because the automation id is the key. Long model execution stays out of the cron worker.
The unresolved edges are specific. Enabled automation rows are not reconciled against Pg-boss schedules at boot. Scheduled occurrences do not have durable idempotency keys. Backend lifecycle coverage was found at lower queue and run layers, but not as direct route or automation-worker tests. Run history is inferred from chats rather than represented as its own automation-run state machine.
Those are the next engineering boundaries, not mysteries. A reconciliation pass would make the database row and Pg-boss schedule converge after crashes. An occurrence id would make the retry window dedupe-able. A real run ledger would let the product show skipped, failed, queued, and completed scheduled work without pretending chat history is the whole story.