Queues and Idempotency for Long-Running AI Work

Safely handle long-running AI tasks

In this article, we outline how to safely handle long-running AI tasks with queues and idempotence.

There are many tasks in LLM services that do not end immediately upon request. Parsing a PDF, splitting the document into chunks, creating embeddings, storing them in a vector DB, and summarizing large documents can take anywhere from seconds to minutes.

If these tasks are processed synchronously within an API request, users will wait a long time, the server will easily become saturated, and recovery in case of failure will be difficult. So we need a queue.

But the moment you use a cue, another problem arises. Messages can be duplicated, retried, and workers can die while processing. So idempotency is needed.

Analysis Base Date: 2026-05-12 Lab Base Environment: FastAPI, PostgreSQL, Redis Queue, or SQS/Kafka Concepts Key References: AWS Builders Library, Amazon SQS Docs, Kafka Docs

Key takeaways

• Long-running AI tasks should be separated from API requests.
• Queues absorb failures and load, but create the possibility of redundant processing.
• The idempotency key prevents the results from being corrupted even if the same task is executed multiple times.
• Task state tables increase observability of asynchronous processing.
• Retry and DLQ are not devices that hide failures, but leave them in a state that can be analyzed.

1. Why AI tasks take so long

Long-running operations in the AI ​​backend include:

These tasks are difficult to tie to the same lifecycle as user requests.

2. Problems with synchronous processing

Synchronous processing is simple at first.

Copy1# This is an example.2POST /documents/{id}/index3to PDF parsing4→ Chunking5→ Embedding6Save to Vector7to reply

But a problem arises.

Copy1# This is an example.2[ ] The request time becomes longer.3[ ] You may experience a reverse proxy timeout.4[ ] The user cannot know the progress status.5[ ] In case of failure, it is difficult to know how far it has been dealt with.6[ ] If the same request comes in again, duplicate indexes may occur.

So we need to change it to asynchronous processing.

3. Queue-based processing flow

Copy1# This is an example.2Client3→ POST /documents/{id}/index4→ API server creates job5→ enqueue message6→ return job_id7→ Worker consumes message8→ process indexing9→ update job status10→ Client polls GET /jobs/{job_id}

The advantage of this structure is that the API server does not grab long operations directly. The user receivesjob_idand queries the status.

4. idempotency key Design

Indexing the same document with the same version must be done only once.

Copy1# This is an example.2idempotency_key = document_id + document_version + pipeline_version

example:

Copy1# This is an example.2doc_123:v7:rag-pipeline-v2

If a request comes in again with the same key, the existing job is returned without creating a new job.

Copy1# This is example code.2async def create_index_job(document_id: str, document_version: int):3    key = f"{document_id}:v{document_version}:rag-pipeline-v2"4 5    existing = await job_repo.find_by_idempotency_key(key)6    if existing:7        return existing8 9    job = await job_repo.create(10        type="DOCUMENT_INDEX",11        idempotency_key=key,12        status="PENDING",13    )14    await queue.publish({"job_id": job.id})15    return job

5. Task status table design

Asynchronous operations must store state.

Without a state table, it is impossible to know “how far things have gone” when a worker fails.

6. Retry and Backoff

Retries are necessary, but infinite retries are dangerous. Especially in an external model API failure situation, if all workers retry immediately, the failure may increase.

Recommended Policy:

Copy1# This is an example.21st failure: Retry in 5 seconds3Second failure: Retry in 30 seconds43rd failure: Retry in 2 minutes54th failure: DLQ move

You should also distinguish between retriable and non-retriable errors.

7. DLQ design

DLQ is not a place to dump failure messages. This is where it is isolated so that it can be analyzed.

Messages entering the DLQ must contain the following information:

Copy1// This is an example JSON structure.2{3  "job_id": "job_123",4  "idempotency_key": "doc_123:v7:rag-pipeline-v2",5  "attempt_count": 4,6  "last_error_code": "EMBEDDING_RATE_LIMIT",7  "last_error_message": "Rate limit exceeded",8  "trace_id": "0af..."9}

DLQ messages should be associated with notifications, dashboards, and reprocessing tools.

8. Document index example

Copy1# This is an example.21. User uploads document32. API creates document_version43. Create index job54. queue publish65. Worker parses document76. Chunk creation87. Create embedding98. Save to pgvector109. Change job status to SUCCEEDED

On failure:

Copy1# This is an example.2worker timeout3to message visibility timeout expires4to be reprocessed by another worker5to check the same idempotency key6to skip chunks that have already been processed7DLQ movement when to repeat failure

9. Operational indicators

10. Practical checklist

Copy1# This is an example.2[ ] Have you separated long-running tasks from API requests?3[ ] Can I check the status by job_id?4[ ] Is idempotency_key designed for the unit of work?5[ ] Can processing continue even after restarting the worker?6[ ] retry Did you distinguish between possible and impossible errors?7[ ] Is there a backoff and maximum number of retries?8[ ] Does the DLQ message include trace_id and error_code?9[ ] Is there a DLQ reprocessing procedure?

11. Q&A

Q1. Should I use Redis Queue, SQS, or Kafka?

For initial learning projects, Redis Queue is sufficient. In production environments, consider SQS or Kafka depending on durability, retries, DLQ, and throughput requirements.

Q2. Is DB unique constraint sufficient for idempotency?

In many cases, unique constraints are a good starting point. However, a separate job table is needed to handle job status, retries, and partial successes together.

Q3. Can we assume that messages will only be processed once?

It's safer not to assume that. In a queue-based system, it is realistic to design it to be idempotent, assuming the possibility of redundancy.

12. References and uncertainty

References

• AWS Builders Library — Making retries safe with idempotent APIs:https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
• Amazon SQS Visibility Timeout:https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
• Amazon SQS Dead-letter Queues:https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• Apache Kafka Documentation:https://kafka.apache.org/documentation/

uncertainty

• Queue selection will depend on traffic, durability, and team operating experience.
• Since exactly-once processing semantics are difficult to guarantee throughout the system, idempotency in business outcome criteria must be designed first.

---

Practical example: Avoid duplicate execution of document indexing operations

Document embedding operations can easily be duplicated when a user presses a button twice or a worker retries after a timeout. The idempotency key is usually created by combiningtenant_id,document_id,document_version, andembedding_model. If the document version and model are the same, a job already running is returned. If the document or embedding model changes, a new job must be created.

A failure case is simply doing too many retries. There are failures that cannot be immediately resolved by retrying, such as provider rate limits or document parsing errors. The retry count, last error, and next retry time must be left in the job table, and sent to DLQ after a certain number of times. A queue is not a device to postpone slow work, but rather an operational layer that classifies failures and makes them reprocessable.

In the task screen, it is better to distinguish betweenqueued,running,retry_scheduled,failed, andcompletedrather than just showing "Processing" alone. When a user uploads the same document again, existing jobs must be displayed to reduce duplication costs.