Batch PDF Verification at Scale: Async Queue Patterns for the HTPBE? API

A single PDF check is a request/response call. A thousand of them is an architecture problem. The moment your platform processes a stream of documents — nightly batches of loan applications, a queue of insurance claims, every payslip uploaded during an onboarding window — the question stops being “how do I call the API” and becomes “how do I call it ten thousand times without losing documents, double-spending credits, or hammering the service.”

This is the architecture piece of the developer cluster. The language guides cover the single-call surface: Node.js, Python, Go, Laravel and PHP, Ruby on Rails, and the programmatic detection overview. The Next.js guide covers per-request integration, and the event-driven guide covers reacting to a single upload. This one is about volume: a queue and worker design around the PDF tamper detection API, sized against your credit pool, safe to re-run, and honest about which failures are worth retrying. The code is TypeScript with BullMQ and Redis; the primitives map onto SQS, Cloud Tasks or any other queue.

The contract you are building on top of

Fix three facts about the API before designing anything.

It is synchronous. POST /analyze blocks until the verdict is computed, then returns 201 Created with { "id": "..." } and a Location header pointing at GET /result/{id}. There is no job to poll and no completion callback to register — HTPBE? does not push webhooks. A typical PDF takes 2–5 seconds, a complex one up to 15. The queue, the workers and the retries are yours; the API is one slow blocking step inside each job.

Capacity is server-wide. When the service is already running its maximum number of simultaneous analyses it answers 429 with code server_at_capacity and a Retry-After header. This is not per-key rate limiting, and it is shared by everyone calling the service — so your throughput is something you measure, not something you are promised.

One credit per document, from one pool. Every live-key call debits the same pool: the monthly subscription quota first, then any one-time credit packs. When the pool is empty the call returns 402 with code payment_required. The API has no balance endpoint, so 402 is your authoritative “stop” signal and any counter you keep is only an estimate.

Everything else follows from those three.

Why a naive loop fails

The first thing most teams write is a for loop over URLs with await client.verify(url) inside it. It works for ten documents and breaks at ten thousand for three reasons.

It is serial. Ten thousand documents at four seconds each is over eleven hours of wall-clock time. You need bounded parallelism.

It has no recovery. One thrown error — a transient 500, a 429, a malformed PDF — and the loop dies or silently skips. A batch that loses documents is worse than one that runs slowly.

It ignores backpressure. Fire a hundred analyze calls with Promise.all and you collect a wall of 429 responses. The server is telling you it is full; hitting it harder only prolongs the problem.

The queue topology

1. A producer enqueues one job per document, with a deterministic job id.
2. A Redis-backed queue (BullMQ) holds pending jobs.
3. A worker pool with bounded concurrency runs the synchronous analyze → result round-trip.
4. A verdict router pushes the result into the right lane — continue your standard checks, an issuer re-request, or human review. A structural verdict routes the case; it is never the sole basis for an automatic rejection.

producer ──> [ Redis queue ] ──> worker pool (concurrency = N)
                                       │
                                       ├─ dedup by file hash, then analyze + result
                                       │
                                       └─> verdict router ──> continue / re-request / review

Step 1: The job payload and the queue

Each job carries the document URL, a SHA-256 of the file bytes, your own entity id (application, claim, candidate) and a filename.

import { Queue } from 'bullmq';
import IORedis from 'ioredis';

export const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: null, // required by BullMQ workers
});

export interface VerifyJobData {
  /** URL the API will fetch the PDF from. Stays fixed for the life of the job. */
  pdfUrl: string;
  /** SHA-256 of the file bytes, computed once when the file entered your storage. */
  contentSha256: string;
  /** Your domain id — loan application, claim, candidate. */
  entityId: string;
  /** Stored as `filename` on the check; use it to find your documents later. */
  originalFilename: string;
}

export const verifyQueue = new Queue<VerifyJobData>('pdf-verify', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2_000 },
    removeOnComplete: { age: 86_400 },
    removeOnFail: false, // keep failures for triage
  },
});

attempts: 5 retries a throwing job up to four more times, spaced roughly 2s, 4s, 8s, 16s — the right shape for a transient 500. 429 is handled differently, in Step 4, because it is a statement about the whole service rather than about one job.

Step 2: Idempotency, in two layers

A batch that runs twice — a deploy restarted mid-run, a CSV re-uploaded, a producer retry — must not buy the same answer twice. Two different mechanisms cover two different failures, and they are not interchangeable.

Layer one: the Idempotency-Key header, for retries

The dangerous moment is the ambiguous failure: your request timed out, or the connection dropped, and you cannot tell whether the server already analysed the document. Retrying blindly can spend a credit twice. The API’s answer is the Idempotency-Key header (any string you generate, up to 255 characters):

• the original request already completed → the retry returns 201 with the same check id, spends nothing, and does not re-analyse;
• the original is still running → the retry returns 409 (idempotency_key_in_progress); wait and retry with the same key;
• the original failed and nothing was kept against your credits → the retry proceeds as a normal new request;
• the same key with a different url → 422 (idempotency_key_fingerprint_mismatch).

So the worker sends the job id as the key on every attempt of every job. Two consequences are easy to miss. First, the url must stay byte-identical across attempts: a presigned link that you re-mint on retry changes the URL and turns a safe retry into a 422, so issue a presigned URL whose lifetime covers the whole retry window. Second, the key is a retry guard, not a database. The documentation does not state how long a key is remembered, so never rely on a replay for a document you submitted last month.

Layer two: content-hash dedup, for the same file arriving twice

Nothing in the API deduplicates by file content. The same bytes sent under two different keys are two checks and two credits. That is your job, before the job exists:

import { createHash } from 'node:crypto';
import { verifyQueue, VerifyJobData } from './queue';

/** Same entity + same bytes = same job. A new file for the same entity is a new job. */
export function deterministicJobId(d: VerifyJobData): string {
  return createHash('sha256').update(`${d.entityId}:${d.contentSha256}`).digest('hex').slice(0, 32);
}

export async function enqueueBatch(documents: VerifyJobData[], priority = 1): Promise<void> {
  const CHUNK = 500; // one Redis pipeline per chunk, not 50k parallel ops
  for (let i = 0; i < documents.length; i += CHUNK) {
    await verifyQueue.addBulk(
      documents.slice(i, i + CHUNK).map((data) => ({
        name: 'verify',
        data,
        opts: { jobId: deterministicJobId(data), priority },
      }))
    );
  }
}

BullMQ refuses a second job with an existing id, so re-running the producer is a no-op for documents already queued or finished. The worker adds the cross-entity layer: before spending a credit it looks up contentSha256 in your own verdict table, and if another entity already carried the identical file it reuses that verdict. Compute the hash once, when the file lands in your storage (most object stores already return a checksum), not by re-downloading it in the producer.

Step 3: The API call

This function sends the idempotency key, bounds both calls with timeouts, and turns every response into one of four outcomes the worker can act on. The timeouts come from the documented limits: the API allows 30 seconds to download your file and 20 seconds to analyse it, so 60 seconds on analyze and 30 on result cover a legitimate request with margin.

const BASE = 'https://api.htpbe.tech/v1';

type Kind = 'retry' | 'backpressure' | 'out_of_credits' | 'permanent';

export class HTPBEError extends Error {
  constructor(
    message: string,
    public readonly status: number,
    public readonly code: string,
    public readonly retryAfterMs?: number
  ) {
    super(message);
    this.name = 'HTPBEError';
  }
  get kind(): Kind {
    if (this.status === 429) return 'backpressure';
    if (this.status === 402) return 'out_of_credits';
    if (this.status === 409 || this.status >= 500) return 'retry';
    if (this.code === 'download_failed') return 'retry'; // your storage may have blinked
    return 'permanent';
  }
}

export interface HTPBEResult {
  id: string;
  status: 'intact' | 'modified' | 'inconclusive';
  status_reason?: string;
  modification_confidence: 'certain' | 'high' | 'none' | null;
  modification_markers: string[];
  algorithm_version: string;
  usage_caution: { safe_for_automated_adverse_decision: false; recommended_action: string };
}

const auth = () => ({ Authorization: `Bearer ${process.env.HTPBE_API_KEY}` });

async function timed(url: string, init: RequestInit, ms: number): Promise<Response> {
  const ctl = new AbortController();
  const t = setTimeout(() => ctl.abort(), ms);
  try {
    return await fetch(url, { ...init, signal: ctl.signal });
  } finally {
    clearTimeout(t);
  }
}

export async function analyzeDocument(
  pdfUrl: string,
  originalFilename: string,
  idempotencyKey: string
): Promise<HTPBEResult> {
  const submit = await timed(
    `${BASE}/analyze`,
    {
      method: 'POST',
      headers: { ...auth(), 'Content-Type': 'application/json', 'Idempotency-Key': idempotencyKey },
      body: JSON.stringify({ url: pdfUrl, original_filename: originalFilename }),
    },
    60_000
  );
  if (!submit.ok) throw await toError(submit);
  const { id } = (await submit.json()) as { id: string };

  const fetched = await timed(`${BASE}/result/${id}`, { headers: auth() }, 30_000);
  if (!fetched.ok) throw await toError(fetched);
  return (await fetched.json()) as HTPBEResult;
}

async function toError(res: Response): Promise<HTPBEError> {
  let body: { error?: string; code?: string } = {};
  try {
    body = await res.json();
  } catch {
    /* non-JSON body, e.g. from a proxy */
  }
  const retryAfter = Number(res.headers.get('Retry-After'));
  return new HTPBEError(
    body.error ?? res.statusText,
    res.status,
    body.code ?? 'unknown',
    Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1_000 : undefined
  );
}

A timeout or dropped connection surfaces as an AbortError or TypeError, not an HTPBEError. Treat those as retry too — and because every attempt carries the same Idempotency-Key, retrying them is exactly the case the header exists for. The documentation lists 429, 500 and 409 as retryable; retrying other 5xx responses (a gateway in front of the service, say) is our recommendation, not a documented guarantee.

What each outcome means in practice:

Step 4: The worker

The worker’s concurrency is the most important dial in the system. Too low and the batch crawls; too high and you spend your time in 429s. Start at 2–4 per process and raise it only while you are not seeing capacity signals.

import { Worker, Job, UnrecoverableError } from 'bullmq';
import { connection, verifyQueue, VerifyJobData } from './queue';
import { analyzeDocument, HTPBEError } from './htpbe';
import { routeVerdict } from './router';

const CONCURRENCY = Number(process.env.VERIFY_CONCURRENCY ?? 3);

export const worker = new Worker<VerifyJobData>(
  'pdf-verify',
  async (job: Job<VerifyJobData>) => {
    const { pdfUrl, contentSha256, entityId, originalFilename } = job.data;

    // Layer two: this exact file already has a verdict on file — no credit spent.
    const known = await findVerdictByHash(contentSha256);
    if (known) {
      await attachVerdict(entityId, known);
      return { checkId: known.checkId, status: known.status, reused: true };
    }

    let result;
    try {
      // Layer one: the job id doubles as the Idempotency-Key on every attempt.
      result = await analyzeDocument(pdfUrl, originalFilename, job.id!);
    } catch (err) {
      if (err instanceof HTPBEError) {
        if (err.kind === 'backpressure') {
          // Server-wide: stop every worker on this queue, not just this job,
          // and hand the job back without consuming an attempt.
          await worker.rateLimit(err.retryAfterMs ?? 5_000);
          throw Worker.RateLimitError();
        }
        if (err.kind === 'out_of_credits') {
          await verifyQueue.pause();
          await alertBillingOwner('HTPBE credits exhausted — verification queue paused');
          throw new UnrecoverableError(`${err.code}: credit pool empty`);
        }
        if (err.kind === 'permanent') {
          throw new UnrecoverableError(`${err.code}: ${err.message}`);
        }
      }
      throw err; // retry / network / timeout: BullMQ applies the backoff from Step 1
    }

    await storeVerdict({
      entityId,
      contentSha256,
      checkId: result.id,
      status: result.status,
      statusReason: result.status_reason ?? null,
      markers: result.modification_markers,
      algorithmVersion: result.algorithm_version,
    });
    await routeVerdict(entityId, result);
    return { checkId: result.id, status: result.status };
  },
  { connection, concurrency: CONCURRENCY, limiter: { max: 30, duration: 10_000 } } // optional ceiling on job starts; also what `rateLimit` hooks into
);

// Replace these with your own persistence layer.
declare function findVerdictByHash(sha: string): Promise<{ checkId: string; status: string } | null>;
declare function attachVerdict(entityId: string, v: { checkId: string; status: string }): Promise<void>;
declare function storeVerdict(v: {
  entityId: string;
  contentSha256: string;
  checkId: string;
  status: string;
  statusReason: string | null;
  markers: string[];
  algorithmVersion: string;
}): Promise<void>;
declare function alertBillingOwner(message: string): Promise<void>;

Three decisions in that function are the point of the article.

429 does not consume an attempt. A capacity response says nothing about the document, so charging it against the job’s five attempts would send healthy documents to the dead-letter queue during a busy hour. worker.rateLimit() plus Worker.RateLimitError() is BullMQ’s manual rate-limit pattern: the worker stops pulling jobs for exactly the time the server asked for, and the job returns to the queue untouched. Check the call against the BullMQ version you pin; the snippets here follow its documented API and have not been run against a production queue.

402 stops the batch. It does not mean this document failed; it means every remaining job will fail identically. Pause the queue the instant it appears, tell whoever owns billing, and stop. After the top-up, retry only the jobs that failed for that reason — re-queueing every failed job would also resubmit documents that were rejected as unreadable:

export async function resumeAfterTopUp(): Promise<void> {
  const failed = await verifyQueue.getFailed(0, 999); // page through if you have more
  for (const job of failed) {
    if (job.failedReason?.startsWith('payment_required')) await job.retry();
  }
  await verifyQueue.resume();
}

Because a 402 charged nothing, retrying with the same Idempotency-Key runs as a normal new request, so nothing is charged twice. A long pause also stretches the retry window: if the presigned URL has expired by then, re-minting it under the same key is rejected with 422, so make the URL outlive the longest pause you are prepared to tolerate, or re-submit the document as a new job with a new key.

Permanent input errors do not retry. UnrecoverableError is BullMQ’s explicit signal to skip the remaining attempts. An oversized or unreadable PDF will fail the same way on attempt five as on attempt one.

Step 5: Routing the three verdicts

Detection only matters if the verdict lands somewhere that acts on it. The most common mistake at scale is collapsing inconclusive into a failure bucket. It is not a failure; it is a finding that the document was produced by consumer software, an online editor, a scanner or a similar origin, so there is no institutional original to check integrity against.

import { HTPBEResult } from './htpbe';

export async function routeVerdict(entityId: string, result: HTPBEResult): Promise<void> {
  switch (result.status) {
    case 'intact':
      await advanceEntity(entityId, 'no_findings'); // continue your normal checks; not proof of authenticity
      return;
    case 'modified':
      await advanceEntity(
        entityId,
        result.modification_confidence === 'certain' ? 'request_reissue' : 'manual_review',
        result.modification_markers
      );
      return;
    case 'inconclusive':
      // Not an error. For a document that claims institutional origin
      // (a bank statement, a payslip) send it to a reviewer.
      await advanceEntity(entityId, 'manual_review', [result.status_reason ?? 'inconclusive']);
      return;
  }
}

declare function advanceEntity(
  entityId: string,
  state: 'no_findings' | 'request_reissue' | 'manual_review',
  markers?: string[]
): Promise<void>;

An intact verdict means no edit was detected. It is not proof that a document is authentic, so that lane continues your standard checks rather than approving anything. Branch on the stable HTPBE_* marker ids in modification_markers — for example HTPBE_SIGNATURE_REMOVED or HTPBE_DATES_DISAGREE — not on human-readable labels. The ids never change once shipped; the labels live in how PDF tamper detection works. modification_confidence decides how firmly to escalate: certain supports holding the document and asking the issuer for a fresh copy, high supports a human look. Both lanes end at a person or an issuer. Every result also carries a usage_caution object as a standing reminder that a structural verdict is never the sole basis for an automatic adverse decision.

Step 6: Sizing, backfill and steady-state

Two very different workloads hit the same pipeline, and they should not share a budget.

Throughput is arithmetic you then verify. With W concurrent workers and a median call time of t seconds, you finish roughly W / t documents per second. Three workers at four seconds is about 45 documents a minute, so ten thousand documents take close to four hours — an illustration, not a promise, because the service’s capacity is shared. Measure your real median, watch the 429 rate, and let that decide the worker count.

Credits are the real ceiling. One document is one credit. Before a backfill, compare its size with your monthly quota and any credit packs on the pricing page; since there is no balance endpoint, count documents sent in your own metrics and treat the first 402 as the truth.

Steady-state is documents arriving all day: a producer triggered on upload, low concurrency, small queue depth, and an alert if depth keeps growing. Latency matters here, because a person is often waiting on the verdict.

Backfill is a historical archive pushed through once. It should be lower priority than live traffic and sized to a budget:

• enqueue it with a larger priority number than live jobs (BullMQ treats lower numbers as more urgent), so a nightly import never starves documents people are waiting on;
• release it in tranches no bigger than the credits you are willing to spend, leaving a reserve for live traffic, instead of enqueueing the whole archive and discovering the limit via 402 at 3 a.m.;
• run it off-peak and raise concurrency only while 429s stay rare.

Reconcile instead of trusting Redis. If a queue is lost or a deploy goes wrong, you can find out what was actually checked. Set original_filename to something you can map back to your entity (for example <entityId>__<document-name>.pdf, so one entity can carry several documents). GET /checks returns your own checks, filterable by status and by from_date / to_date on check_date, up to 500 per page, each with its filename. Page through the window, parse the entity key out of each filename, subtract those from the backlog, and re-enqueue only the rest. Because the key-replay window is undocumented, this list, together with your own hash table, is the durable dedup — not the header.

Step 7: Storing results

Store the minimum that lets you act and audit: your entity id, the file hash, the check id, the status, status_reason, the marker ids, and algorithm_version. The check id is the pointer to everything else — GET /result/{id} returns the full payload again whenever you need it, without re-analysing, so there is no reason to copy the whole response into your database. Keeping algorithm_version lets you tell later which verdicts predate a newer algorithm. Results are readable only by the account that created them.

Step 8: Testing the pipeline without spending credits

Every account has one test key alongside its live keys. Test keys accept a fixed set of mock URLs, return deterministic verdicts, download nothing and debit nothing, so the whole pipeline — producer, concurrency, retry, routing — runs in CI for free. The documented trigger-402.pdf mock returns a 402 on demand, which is the only way to exercise your queue-pause path with a test key.

import { describe, it, expect, afterAll } from 'vitest';
import { enqueueBatch } from './producer';
import { worker } from './worker';
import { verifyQueue } from './queue';

const TEST = 'https://api.htpbe.tech/v1/test';
const doc = (name: string, entityId: string) => ({
  pdfUrl: `${TEST}/${name}`,
  contentSha256: entityId.padEnd(64, '0'),
  entityId,
  originalFilename: name,
});

describe('verification pipeline', () => {
  it('completes a modified document', async () => {
    const done = new Promise<{ status: string }>((resolve) =>
      worker.once('completed', (_job, value) => resolve(value))
    );
    await enqueueBatch([doc('modified-high.pdf', 'loan-9001')]);
    expect((await done).status).toBe('modified');
  });

  it('pauses the queue when credits run out', async () => {
    await enqueueBatch([doc('trigger-402.pdf', 'loan-9002')]);
    await new Promise((r) => worker.once('failed', r));
    expect(await verifyQueue.isPaused()).toBe(true);
    await verifyQueue.resume();
  });

  afterAll(async () => {
    await worker.close();
  });
});

Test keys only take mock URLs of the form https://api.htpbe.tech/v1/test/{name}.pdf, such as clean.pdf and modified-high.pdf. They cannot exercise Idempotency-Key replays, since nothing is charged on a test key; test that layer against a live key with one throwaway document. A live key accepts any public PDF URL and debits one credit per call. Keep the two keys in separate environment files and never commit either.

Step 9: Operating the pipeline

Correctness at volume is half the job; the rest is being able to see and survive the system.

• Metrics. Count completed (tagged by verdict), failed (tagged by error code) and stalled, and sample queue depth, active and failed counts on a timer. Alert on three things: depth that keeps rising, a sustained failure rate above a few percent, and any payment_required.
• Dead-letter queue. removeOnFail: false keeps failures, but a separate queue that normal workers never consume, drained into a ticket or a channel by a daily job, makes sure no document vanishes silently. Everything that exhausts its attempts or throws UnrecoverableError belongs there with its error code.
• Graceful shutdown. On SIGTERM, call worker.close(), which waits for active jobs, then close the queue and the Redis connection. Give your orchestrator a termination grace period well above your worst-case call time. A killed in-flight call is re-queued as stalled, and with a stable Idempotency-Key the re-run replays rather than double-charges.
• A producer cap. A stuck worker pool and an unbounded producer fill Redis and hurt every other queue on it. Refuse to enqueue past a ceiling on waiting + delayed jobs and slow the producer instead; treat it as load-shedding.
• A dashboard. bull-board shows queues, errors and payloads. Put it behind internal auth: payloads contain document URLs.

What this architecture does not solve

The queue makes the API call reliable and cheap at volume. It does not change what the analysis can see. HTPBE? looks for changes made after a file was created; it says nothing about whether the content of a document was true when it was produced. That is a content-truth question, answered by other layers of a defence. For where this fits, see KYC vs. document forensics.

What the queue does buy is control over the failure modes. You bound the rate hitting a shared service, keep to one credit per document in normal operation by combining Idempotency-Key with your own file-hash dedup, retry the failures that deserve it without burning attempts on the ones that do not, stop cleanly when the credit pool is dry, and route everything unrecoverable somewhere a person will look. Failures still happen; the design makes them visible and survivable.

To build it, generate a test key, run the Step 8 tests, then point a live key at a small tranche before the full batch. The API reference has every field and error code, and the pricing page shows how monthly quota and one-time credit packs map onto the volume you plan to push.