PDF Tamper Detection in Go: net/http Client, errgroup Worker Pool and httptest

This article is a snapshot – content was accurate as of July 2026 (code examples tested against the API as of October 2026). The product evolves actively; specific counts, examples, and detection rules may have changed since publication – see the changelog for the current state.
A Go service that accepts PDFs from users has one job before it trusts them: find out whether the file was edited after it was created. The forged bank statement, the altered invoice and the doctored payslip all arrive through the same upload handler, and by the time a row is written to the database their claims are already part of your business logic. Ingress is the cheapest place to check.
This guide builds a small Go package around the PDF tamper detection API: a net/http client with per-attempt context timeouts, a retry policy that does not double-bill within a call, one typed error per documented error code, routing for each of the three verdicts, a worker pool that cancels itself when credits run out or the key is wrong, and tests that run entirely on test keys. The client is standard-library only; Step 9 adds errgroup, rate and singleflight from the golang.org/x modules. It does not repeat what the sibling guides cover. For the language-agnostic flow start with How to Detect PDF Tampering Programmatically; for other stacks see the Node.js, Python and Laravel / PHP guides. For queue-based throughput see batch verification with a job queue. What is specific to Go here is how errors, contexts, goroutines and httptest fit the API contract.
TL;DR
- Two calls:
POST /analyzereturns{"id": ...},GET /result/{id}returns the flat result withstatusofintact,modifiedorinconclusive. POST /analyzespends a credit. Send anIdempotency-Keyheader and reuse the same value on every retry of that request, so a timeout never turns into a second charge.- Retry only what can succeed later:
429,5xx, transport errors, and409 idempotency_key_in_progress. Everything else is permanent. - Do not auto-reject on
modified. Every result carriesusage_caution.safe_for_automated_adverse_decision: false; route to a human. - Tests run on
htpbe_test_*keys against the documented mock URLs, plus anhttptest.Serverfor retry logic. - This is structural PDF tamper detection. It is not KYC, not OCR and not AI-text detection.
Prerequisites
- Go 1.22 or newer (the handler below uses method-qualified
ServeMuxpatterns;log/slogneeds 1.21,math/rand/v2needs 1.22) - An HTPBE? API key from the dashboard; the same page shows your test key
- No third-party modules for the client itself (the optional storage snippet in Step 6 uses
aws-sdk-go-v2)
Step 1: Try the Flow with curl and a Test Key
Test keys (prefix htpbe_test_) accept only the mock URLs under https://api.htpbe.tech/v1/test/, return deterministic results and consume no credits.
curl -X POST https://api.htpbe.tech/v1/analyze \
-H "Authorization: Bearer $HTPBE_TEST_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://api.htpbe.tech/v1/test/clean.pdf"}'
# {"id":"00000000-0000-4000-8000-000000000001"}
curl https://api.htpbe.tech/v1/result/00000000-0000-4000-8000-000000000001 \
-H "Authorization: Bearer $HTPBE_TEST_API_KEY"POST /analyze is synchronous: when it returns 201 with the id, the result is already available. A live key accepts any public PDF URL up to 10 MB; the URL must be downloadable without authentication.
Step 2: The Result Type
The struct mirrors the documented GET /result/{id} response. encoding/json ignores unknown fields, so new fields never break decoding. Fields the API documents as nullable are pointers, so ‘absent’ stays distinguishable from zero.
package htpbe
const (
StatusIntact = "intact"
StatusModified = "modified"
StatusInconclusive = "inconclusive"
)
// Result mirrors the GET /result/{id} response.
type Result struct {
ID string `json:"id"`
Filename string `json:"filename"`
FileSize int64 `json:"file_size"`
// Unix seconds; null for test-key results and legacy records.
CheckDate *int64 `json:"check_date"`
AlgorithmVersion string `json:"algorithm_version"`
CurrentAlgorithmVersion string `json:"current_algorithm_version"`
// Present only when the check ran on an older algorithm version.
OutdatedWarning string `json:"outdated_warning"`
// "intact" | "modified" | "inconclusive"
Status string `json:"status"`
// Present only when Status is "inconclusive":
// consumer_software_origin | online_editor_origin | scanned_document |
// unverifiable_metadata | filled_form_origin | fill_sign_origin
StatusReason string `json:"status_reason"`
// "certain" | "high" | "none"; null on legacy records.
ModificationConfidence *string `json:"modification_confidence"`
CreationDate *int64 `json:"creation_date"`
ModificationDate *int64 `json:"modification_date"`
Creator *string `json:"creator"`
Producer *string `json:"producer"`
PDFVersion *string `json:"pdf_version"`
DateSequenceValid bool `json:"date_sequence_valid"`
MetadataCompletenessScore int `json:"metadata_completeness_score"`
XrefCount int `json:"xref_count"`
HasIncrementalUpdates bool `json:"has_incremental_updates"`
UpdateChainLength int `json:"update_chain_length"`
HasDigitalSignature bool `json:"has_digital_signature"`
SignatureCount int `json:"signature_count"`
SignatureRemoved bool `json:"signature_removed"`
ModificationsAfterSignature bool `json:"modifications_after_signature"`
PageCount int `json:"page_count"`
ObjectCount int `json:"object_count"`
HasJavascript bool `json:"has_javascript"`
HasEmbeddedFiles bool `json:"has_embedded_files"`
// Stable HTPBE_* ids, strongest first; empty unless Status is "modified".
ModificationMarkers []string `json:"modification_markers"`
UsageCaution UsageCaution `json:"usage_caution"`
}
// UsageCaution is the responsible-use guardrail present on every result.
type UsageCaution struct {
SafeForAutomatedAdverseDecision bool `json:"safe_for_automated_adverse_decision"` // always false
// "route_to_human_review" | "request_from_issuer" | "no_action"
RecommendedAction string `json:"recommended_action"`
Message string `json:"message"`
}Three fields deserve a closer look:
ModificationMarkersholds stable ids prefixedHTPBE_, for exampleHTPBE_SIGNATURE_REMOVEDorHTPBE_DATES_DISAGREE. They never change once shipped, so branch on the id and render the label from the dictionary on the how-it-works page. Never parse labels.ModificationConfidenceiscertainorhighwheneverStatusismodified. Use it to prioritise the review queue, not to decide whether a human looks.UsageCautionis part of the payload on purpose: a structural verdict describes the file, not the person who sent it.
Step 3: One Typed Error per Documented Code
Every non-2xx response has the shape {"error": "...", "code": "...", "details": "..."}. Wrap it in one type that carries the HTTP status, the machine-readable code and the Retry-After hint, so callers branch with errors.As instead of string matching.
package htpbe
import (
"fmt"
"time"
)
// Error codes documented for the API.
const (
CodeInvalidRequest = "invalid_request"
CodeInvalidURLFormat = "invalid_url_format"
CodeDownloadFailed = "download_failed"
CodeMissingAPIKey = "missing_api_key"
CodeInvalidAPIKey = "invalid_api_key"
CodePaymentRequired = "payment_required"
CodeInactiveClient = "inactive_client"
CodeTestURLRequired = "test_url_required"
CodeAccountRestricted = "account_restricted"
CodeIdempotencyInProgress = "idempotency_key_in_progress"
CodeFileTooLarge = "file_too_large"
CodeInvalidPDF = "invalid_pdf"
CodeIdempotencyMismatch = "idempotency_key_fingerprint_mismatch"
CodeServerAtCapacity = "server_at_capacity"
CodeInternalError = "internal_error"
CodeNotFound = "not_found"
)
// APIError is returned for every non-2xx response.
type APIError struct {
StatusCode int
Code string
Message string
Details string
RetryAfter time.Duration // from the Retry-After header; 0 when absent
}
func (e *APIError) Error() string {
return fmt.Sprintf("htpbe: HTTP %d %s: %s", e.StatusCode, e.Code, e.Message)
}
// Retryable reports whether sending the same request again can succeed.
func (e *APIError) Retryable() bool {
switch {
case e.StatusCode == 429, e.StatusCode >= 500:
return true
case e.StatusCode == 409 && e.Code == CodeIdempotencyInProgress:
return true
}
return false
}What each code means for your service:
| Status | code | Meaning | What to do |
|---|---|---|---|
| 400 | invalid_request, invalid_url_format | Malformed JSON, missing url, a non-HTTP(S) URL, or an oversized Idempotency-Key | Bug in your code or input. Fix it; do not retry |
| 400 | download_failed | The API could not fetch your URL (404, 403, auth required, 30 s download timeout, DNS) | Check that the URL is public and live. The cause is on the URL side, so do not retry blindly |
| 401 | missing_api_key, invalid_api_key | Key absent or not recognised | Configuration error; alert, do not retry |
| 402 | payment_required | No credit source left for this request | Billing event: top up on the pricing page; never retry |
| 403 | inactive_client, account_restricted | Key deactivated or account restricted | Contact support; do not retry |
| 403 | test_url_required | A test key was used with a real URL | You shipped a test key to production, or a real URL to a test run |
| 409 | idempotency_key_in_progress | A request with this Idempotency-Key is still being processed | Wait, retry with the same key |
| 413 | file_too_large | The PDF is over 10 MB | Reject the upload to the user |
| 422 | invalid_pdf | Not a PDF, corrupted, password-protected, or analysis could not complete | Ask for a fresh, unprotected copy; retrying the same file is not expected to help |
| 422 | idempotency_key_fingerprint_mismatch | The same key was sent with a different url | Use one key per document |
| 429 | server_at_capacity | Server-wide concurrency limit, not a per-key rate limit | Wait for Retry-After, retry |
| 5xx | internal_error | Transient server error | Retry with backoff |
GET /result/{id} adds 400 invalid_request for a malformed id and 404 not_found. The 404 also covers an id that belongs to another client, and a test key asking for anything other than the synthetic test ids.
Step 4: The Client
One Client is safe for concurrent use. The design decisions that matter:
- Per-attempt timeouts, caller-owned deadline. Each HTTP attempt gets its own
context.WithTimeout; the context you pass toVerifybounds the whole operation. The documented worst case for one analyze call is a 30 second download plus a 20 second analysis, so the default attempt budget is 55 seconds. - The POST is the billable step, so it carries an
Idempotency-Key. A client-side timeout does not tell you whether the server finished. With the header, a retry of a request that already completed returns201with the same check id and spends no second credit. Without it, a blind retry can bill you twice. - The key is generated once per
Verifycall and reused on every attempt. You can pass your own to make retries safe across process restarts (see the caveat after the code). - Transport errors are retried, context cancellation is not.
package htpbe
import (
"bytes"
"context"
"crypto/rand"
"encoding/json"
"errors"
"fmt"
"io"
mrand "math/rand/v2"
"net/http"
"net/url"
"strconv"
"strings"
"time"
)
const defaultBaseURL = "https://api.htpbe.tech/v1"
type Client struct {
apiKey string
baseURL string
hc *http.Client
maxRetries int
attemptTimeout time.Duration
}
type Option func(*Client)
func WithBaseURL(u string) Option {
return func(c *Client) { c.baseURL = strings.TrimRight(u, "/") }
}
func WithMaxRetries(n int) Option {
return func(c *Client) { c.maxRetries = n }
}
func WithAttemptTimeout(d time.Duration) Option {
return func(c *Client) { c.attemptTimeout = d }
}
func WithHTTPClient(hc *http.Client) Option {
return func(c *Client) {
if hc != nil {
c.hc = hc
}
}
}
func New(apiKey string, opts ...Option) (*Client, error) {
if apiKey == "" {
return nil, errors.New("htpbe: API key is required")
}
c := &Client{
apiKey: apiKey,
baseURL: defaultBaseURL,
hc: &http.Client{}, // timeouts come from contexts, not http.Client.Timeout
maxRetries: 3,
attemptTimeout: 55 * time.Second,
}
for _, opt := range opts {
opt(c)
}
return c, nil
}
type VerifyOptions struct {
// OriginalFilename is stored and returned as Result.Filename.
OriginalFilename string
// IdempotencyKey identifies this logical request. Empty means "generate one
// for this call". At most 255 characters; one key per document.
IdempotencyKey string
}
// Verify submits pdfURL and returns the full result.
func (c *Client) Verify(ctx context.Context, pdfURL string, opts VerifyOptions) (*Result, error) {
if pdfURL == "" {
return nil, errors.New("htpbe: pdfURL is required")
}
key := opts.IdempotencyKey
if key == "" {
var err error
if key, err = newIdempotencyKey(); err != nil {
return nil, err
}
}
body, err := json.Marshal(struct {
URL string `json:"url"`
OriginalFilename string `json:"original_filename,omitempty"`
}{pdfURL, opts.OriginalFilename})
if err != nil {
return nil, err
}
var created struct {
ID string `json:"id"`
}
err = c.do(ctx, func(ctx context.Context) (*http.Request, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/analyze", bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key)
return req, nil
}, &created)
if err != nil {
return nil, err
}
if created.ID == "" {
return nil, errors.New("htpbe: analyze response has no id")
}
var res Result
err = c.do(ctx, func(ctx context.Context) (*http.Request, error) {
return http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/result/"+url.PathEscape(created.ID), nil)
}, &res)
if err != nil {
return nil, err
}
return &res, nil
}
// do runs one logical request: build, send, decode, retry when it makes sense.
func (c *Client) do(ctx context.Context, build func(context.Context) (*http.Request, error), out any) error {
for attempt := 0; ; attempt++ {
err := c.once(ctx, build, out)
if err == nil {
return nil
}
if ctx.Err() != nil || !isRetryable(err) || attempt >= c.maxRetries {
return err
}
timer := time.NewTimer(backoff(attempt, err))
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
}
func (c *Client) once(ctx context.Context, build func(context.Context) (*http.Request, error), out any) error {
ctx, cancel := context.WithTimeout(ctx, c.attemptTimeout)
defer cancel()
req, err := build(ctx)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+c.apiKey)
resp, err := c.hc.Do(req)
if err != nil {
return fmt.Errorf("htpbe: %s %s: %w", req.Method, req.URL.Path, err)
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
return parseError(resp)
}
return json.NewDecoder(resp.Body).Decode(out)
}
// isRetryable: API errors follow Retryable(); anything else (connection reset,
// attempt timeout, truncated body) is a transport failure. Retrying is safe
// because the POST carries the same Idempotency-Key on every attempt.
func isRetryable(err error) bool {
var apiErr *APIError
if errors.As(err, &apiErr) {
return apiErr.Retryable()
}
return true
}
func backoff(attempt int, err error) time.Duration {
var apiErr *APIError
if errors.As(err, &apiErr) && apiErr.RetryAfter > 0 {
return apiErr.RetryAfter
}
d := min(time.Second<<attempt, 30*time.Second)
return d + mrand.N(time.Second) // jitter
}
func parseError(resp *http.Response) error {
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<16))
var eb struct {
Error string `json:"error"`
Code string `json:"code"`
Details string `json:"details"`
}
_ = json.Unmarshal(raw, &eb) // the body may not be JSON (proxy error pages)
e := &APIError{StatusCode: resp.StatusCode, Code: eb.Code, Message: eb.Error, Details: eb.Details}
if e.Message == "" {
e.Message = http.StatusText(resp.StatusCode)
}
if s, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && s > 0 {
e.RetryAfter = min(time.Duration(s)*time.Second, 5*time.Minute)
}
return e
}
// newIdempotencyKey returns a random UUID v4.
func newIdempotencyKey() (string, error) {
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
return "", err
}
b[6] = b[6]&0x0f | 0x40
b[8] = b[8]&0x3f | 0x80
return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:]), nil
}A caveat on caller-supplied keys: the server fingerprints the key against the url. If you reuse a key with a different URL you get 422 idempotency_key_fingerprint_mismatch. That matters with presigned URLs (Step 6): minting a fresh presigned URL for the same upload changes the url. When you retry across process restarts, persist the key together with the exact URL you sent, and replay both.
Step 5: Handle Each Verdict
The client returns facts; policy is yours. Two rules come straight from the contract. First, usage_caution.safe_for_automated_adverse_decision is always false, so no verdict is a licence to reject a person automatically. Second, inconclusive is neither a failure nor a pass: the integrity check does not apply to that kind of file.
package htpbe
// Action is where a document goes next.
type Action string
const (
ActionAccept Action = "accept" // continue the automated path
ActionHumanReview Action = "human_review" // a person looks before any decision
ActionRequestFromIssuer Action = "request_from_issuer" // confirm with the issuing organisation
)
// Route maps a verdict to an action. An unknown status (for example one added
// to the API after you shipped) fails closed to human review.
func Route(r *Result) Action {
switch r.Status {
case StatusIntact:
return ActionAccept
case StatusModified:
return ActionHumanReview
case StatusInconclusive:
return ActionRequestFromIssuer
default:
return ActionHumanReview
}
}
// ReviewPriority orders the human queue; it never decides whether a human looks.
func ReviewPriority(r *Result) int {
if r.ModificationConfidence != nil && *r.ModificationConfidence == "certain" {
return 0 // first in line
}
return 1
}How to treat each branch:
intactmeans no structural evidence of modification. It is not a guarantee of authenticity, so keep your own downstream checks (amounts, account lookups, sender authentication). LogOutdatedWarningif it is non-empty: the check ran on an older algorithm version and a re-analysis may give a different answer.modifiedcarries at least one marker inModificationMarkers. Store the list with the case so the reviewer sees why. ConsiderModificationsAfterSignatureandSignatureRemovedas shortcuts for the signature-specific cases.inconclusivehas aStatusReason. Log it as a metric dimension, but route every reason the same way: the test documentation advises giving allinconclusivereasons one fallback path rather than branching on the exact string. For a document that claims institutional origin (a bank statement, a payslip),consumer_software_originis itself worth a second look, since a real bank or payroll system does not distribute files that way. What ‘inconclusive’ means goes deeper.
Mapping errors in an HTTP handler
A handler also has to translate *APIError into something sensible for its own caller, without leaking billing or key problems.
package main
import (
"context"
"encoding/json"
"errors"
"log/slog"
"net/http"
"os"
"time"
"yourapp/htpbe"
)
type verifyRequest struct {
DocumentURL string `json:"document_url"`
OriginalFilename string `json:"original_filename"`
}
func main() {
client, err := htpbe.New(os.Getenv("HTPBE_API_KEY"))
if err != nil {
slog.Error("htpbe init failed", "err", err)
os.Exit(1)
}
mux := http.NewServeMux()
mux.HandleFunc("POST /documents", func(w http.ResponseWriter, r *http.Request) {
var in verifyRequest
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<16)).Decode(&in); err != nil || in.DocumentURL == "" {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "document_url is required"})
return
}
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()
res, err := client.Verify(ctx, in.DocumentURL, htpbe.VerifyOptions{OriginalFilename: in.OriginalFilename})
if err != nil {
status, msg := statusFor(err)
writeJSON(w, status, map[string]string{"error": msg})
return
}
switch htpbe.Route(res) {
case htpbe.ActionAccept:
writeJSON(w, http.StatusOK, map[string]any{"decision": "accept", "check_id": res.ID})
case htpbe.ActionRequestFromIssuer:
slog.Info("inconclusive", "check_id", res.ID, "reason", res.StatusReason)
writeJSON(w, http.StatusAccepted, map[string]any{"decision": "request_from_issuer", "check_id": res.ID})
default:
slog.Info("modified", "check_id", res.ID, "markers", res.ModificationMarkers)
writeJSON(w, http.StatusAccepted, map[string]any{"decision": "human_review", "check_id": res.ID})
}
})
srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
slog.Error("server stopped", "err", srv.ListenAndServe())
}
// statusFor translates a client error into our own response. Key, billing and
// test-key problems are our configuration, so the caller only sees 503.
func statusFor(err error) (int, string) {
var apiErr *htpbe.APIError
if !errors.As(err, &apiErr) {
slog.Error("htpbe verify failed", "err", err)
return http.StatusBadGateway, "verification failed"
}
switch apiErr.Code {
case htpbe.CodeMissingAPIKey, htpbe.CodeInvalidAPIKey, htpbe.CodePaymentRequired,
htpbe.CodeInactiveClient, htpbe.CodeAccountRestricted, htpbe.CodeTestURLRequired:
slog.Error("htpbe unavailable: needs operator attention", "code", apiErr.Code)
return http.StatusServiceUnavailable, "verification temporarily unavailable"
case htpbe.CodeInvalidURLFormat, htpbe.CodeDownloadFailed:
return http.StatusUnprocessableEntity, "the document could not be downloaded"
case htpbe.CodeFileTooLarge:
return http.StatusRequestEntityTooLarge, "PDF must be under 10 MB"
case htpbe.CodeInvalidPDF:
return http.StatusUnprocessableEntity, "the file could not be read as a PDF"
}
slog.Error("htpbe verify failed", "code", apiErr.Code, "status", apiErr.StatusCode)
return http.StatusBadGateway, "verification failed"
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}Verification inside the request handler blocks for the analysis time (typically 2 to 5 seconds, longer for large or complex files). That suits low-volume onboarding; for high-volume intake, return 202 immediately and verify from a worker, as in the queue guide.
Step 6: Giving the API a Reachable URL
The API downloads the file itself; it does not accept uploads. Store the upload privately, mint a short-lived presigned GET URL and pass that. With an S3-compatible store the shape is:
presigned, err := presigner.PresignGetObject(ctx, &s3.GetObjectInput{
Bucket: &bucket, Key: &key,
}, s3.WithPresignExpires(5*time.Minute))
if err != nil {
return nil, err
}
return client.Verify(ctx, presigned.URL, htpbe.VerifyOptions{OriginalFilename: name})Pass OriginalFilename, otherwise Result.Filename is the opaque object key. The same pattern works for Google Cloud Storage signed URLs and Cloudflare R2. One safety note on your side: every Verify call spends a credit, so never pass a caller-supplied URL straight through. Accept only URLs your own service minted (such as the presigned URLs above) or hosts on an allowlist you control.
Step 7: Bounded Concurrency for a Backlog
For a backlog, run a fixed pool of workers over one shared Client (it reuses connections). Indexing the output slice by job position keeps results ordered without a mutex.
type Outcome struct {
URL string
Result *htpbe.Result
Err error
}
func VerifyAll(ctx context.Context, c *htpbe.Client, urls []string, workers int) []Outcome {
workers = max(workers, 1)
out := make([]Outcome, len(urls))
jobs := make(chan int)
var wg sync.WaitGroup
for range workers {
wg.Add(1)
go func() {
defer wg.Done()
for i := range jobs {
res, err := c.Verify(ctx, urls[i], htpbe.VerifyOptions{})
out[i] = Outcome{URL: urls[i], Result: res, Err: err}
}
}()
}
for i := range urls {
jobs <- i
}
close(jobs)
wg.Wait()
return out
}429 server_at_capacity is server-wide, not per key, so a small worker count is the first defence and the Retry-After-aware retry in the client is the second. Check each Outcome.Err with errors.As: a payment_required mid-batch means the rest will fail the same way, so cancel the context you passed in (for example from a context.WithCancel wrapper that fires on that code) rather than collecting hundreds of identical errors.
Step 8: Tests on Test Keys
Two layers, both without spending a credit.
Retry logic with httptest. No key or network is needed. The server fails the first POST, then succeeds; the test asserts the retry reused the same Idempotency-Key and that the GET ran exactly once.
package htpbe_test
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
"yourapp/htpbe"
)
func TestVerifyRetriesPOSTWithSameIdempotencyKey(t *testing.T) {
var posts, gets atomic.Int32
keys := make(chan string, 4)
mux := http.NewServeMux()
mux.HandleFunc("POST /analyze", func(w http.ResponseWriter, r *http.Request) {
keys <- r.Header.Get("Idempotency-Key")
if posts.Add(1) == 1 {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusInternalServerError)
_, _ = w.Write([]byte(`{"error":"Failed to analyze PDF","code":"internal_error"}`))
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
_, _ = w.Write([]byte(`{"id":"00000000-0000-4000-8000-000000000001"}`))
})
mux.HandleFunc("GET /result/{id}", func(w http.ResponseWriter, r *http.Request) {
gets.Add(1)
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"` + r.PathValue("id") + `","status":"intact"}`))
})
srv := httptest.NewServer(mux)
defer srv.Close()
c, _ := htpbe.New("htpbe_test_dummy", htpbe.WithBaseURL(srv.URL), htpbe.WithMaxRetries(2))
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
res, err := c.Verify(ctx, "https://api.htpbe.tech/v1/test/clean.pdf", htpbe.VerifyOptions{})
if err != nil {
t.Fatal(err)
}
if res.Status != htpbe.StatusIntact {
t.Fatalf("status = %q", res.Status)
}
if posts.Load() != 2 || gets.Load() != 1 {
t.Fatalf("posts=%d gets=%d, want 2 and 1", posts.Load(), gets.Load())
}
first, second := <-keys, <-keys
if first == "" || first != second {
t.Fatalf("Idempotency-Key changed between attempts: %q vs %q", first, second)
}
}
func TestPermanentErrorsAreNotRetried(t *testing.T) {
var posts atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
posts.Add(1)
w.WriteHeader(http.StatusPaymentRequired)
_, _ = w.Write([]byte(`{"error":"No credits remaining.","code":"payment_required"}`))
}))
defer srv.Close()
c, _ := htpbe.New("htpbe_test_dummy", htpbe.WithBaseURL(srv.URL))
_, err := c.Verify(context.Background(), "https://example.com/a.pdf", htpbe.VerifyOptions{})
var apiErr *htpbe.APIError
if !errors.As(err, &apiErr) || apiErr.Code != htpbe.CodePaymentRequired {
t.Fatalf("err = %v", err)
}
if posts.Load() != 1 {
t.Fatalf("posts = %d, want exactly 1", posts.Load())
}
}The real API on a test key. Add "os" to the imports of the same test file, and gate these on an environment variable so CI without a key skips them. Test keys only accept the documented mock URLs, return deterministic results, and never debit credits.
func TestLiveTestKey(t *testing.T) {
key := os.Getenv("HTPBE_TEST_API_KEY")
if key == "" {
t.Skip("HTPBE_TEST_API_KEY not set")
}
c, _ := htpbe.New(key)
const base = "https://api.htpbe.tech/v1/test/"
cases := []struct {
file string
status string
action htpbe.Action
}{
{"clean.pdf", htpbe.StatusIntact, htpbe.ActionAccept},
{"signature-removed.pdf", htpbe.StatusModified, htpbe.ActionHumanReview},
{"dates-mismatch.pdf", htpbe.StatusModified, htpbe.ActionHumanReview},
{"inconclusive.pdf", htpbe.StatusInconclusive, htpbe.ActionRequestFromIssuer},
{"scanned-document.pdf", htpbe.StatusInconclusive, htpbe.ActionRequestFromIssuer},
}
for _, tc := range cases {
t.Run(tc.file, func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
res, err := c.Verify(ctx, base+tc.file, htpbe.VerifyOptions{})
if err != nil {
t.Fatal(err)
}
if res.Status != tc.status || htpbe.Route(res) != tc.action {
t.Fatalf("status=%q action=%q", res.Status, htpbe.Route(res))
}
})
}
t.Run("out of credits", func(t *testing.T) {
_, err := c.Verify(context.Background(), base+"trigger-402.pdf", htpbe.VerifyOptions{})
var apiErr *htpbe.APIError
if !errors.As(err, &apiErr) || apiErr.Code != htpbe.CodePaymentRequired {
t.Fatalf("err = %v", err)
}
})
}Notes that save debugging time:
trigger-402.pdfis the only way to exercise the402path with a test key, because a test key never debits credits. It has no check id and cannot be passed toGET /result/{id}.- Do not assert on
CheckDate: it isnullfor synthetic results. - A test key resolves only the synthetic ids. Asking for any other id, even a real check from the same account, returns
404 not_found. - Every mock URL and its expected verdict are listed in the public API documentation. Keep test and live keys in separate environment files and never commit either.
Step 9: A Worker Pool That Stops When the Money Does
VerifyAll in Step 7 is fine for a script. A long-running service needs three more behaviours, and each has a Go-idiomatic home: a client-side pace, a pool that cancels itself on a configuration or billing failure, and a clean stop on SIGTERM so a deploy never abandons a request mid-flight. errgroup, rate.Limiter and singleflight cover all three without a queue library.
package main
import (
"context"
"errors"
"log/slog"
"os"
"os/signal"
"syscall"
"golang.org/x/sync/errgroup"
"golang.org/x/sync/singleflight"
"golang.org/x/time/rate"
"example.com/app/htpbe"
)
// Job is one stored document. URL is a presigned GET minted for this run.
type Job struct {
Hash string // SHA-256 of the file bytes
URL string
}
type Checker struct {
client *htpbe.Client
limiter *rate.Limiter
flight singleflight.Group
}
// Check collapses concurrent calls for the same content hash into one request,
// so a file that lands twice in the same minute is analysed, and charged, once.
func (k *Checker) Check(ctx context.Context, j Job) (*htpbe.Result, error) {
v, err, _ := k.flight.Do(j.Hash, func() (any, error) {
if err := k.limiter.Wait(ctx); err != nil {
return nil, err
}
return k.client.Verify(ctx, j.URL, htpbe.VerifyOptions{})
})
if err != nil {
return nil, err
}
return v.(*htpbe.Result), nil
}
// fatal reports errors after which every remaining job would fail the same way.
func fatal(err error) bool {
var apiErr *htpbe.APIError
if !errors.As(err, &apiErr) {
return false
}
switch apiErr.Code {
case htpbe.CodePaymentRequired, htpbe.CodeInvalidAPIKey,
htpbe.CodeInactiveClient, htpbe.CodeAccountRestricted,
htpbe.CodeTestURLRequired:
return true
}
return false
}
func Run(ctx context.Context, k *Checker, jobs []Job, workers int) error {
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(workers)
for _, j := range jobs {
if ctx.Err() != nil {
break // a fatal error or SIGTERM: stop handing out work
}
g.Go(func() error {
res, err := k.Check(ctx, j)
switch {
case err == nil:
slog.Info("checked", "hash", j.Hash, "status", res.Status,
"action", htpbe.Route(res))
return nil
case fatal(err):
return err // errgroup cancels ctx for everyone
case ctx.Err() != nil:
return nil // shutting down; the job stays pending
default:
slog.Warn("check failed", "hash", j.Hash, "err", err)
return nil // a per-document failure must not stop the batch
}
})
}
return g.Wait()
}
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
client, err := htpbe.New(os.Getenv("HTPBE_API_KEY"))
if err != nil {
slog.Error("client", "err", err)
os.Exit(1)
}
// 5 requests per second with a burst of 5 is a pace you choose, not an API limit.
k := &Checker{client: client, limiter: rate.NewLimiter(5, 5)}
if err := Run(ctx, k, loadPending(), 4); err != nil {
slog.Error("batch stopped", "err", err)
os.Exit(1)
}
}Three details are worth the lines they take:
- Why
fatalexists.402 payment_requiredmeans no credit source is left, and401,403andtest_url_requiredmean configuration. Retrying or continuing is wasted effort and, for402, noise in your logs. Returning the error from theerrgroupgoroutine cancels the shared context, so the in-flight workers unwind and the loop stops handing out work. Per-document errors (an unreachable URL, a file that is not a valid PDF) returnnilso one bad file never stops the batch. - What cancellation does to a request in flight.
Verifytakes the context, so a cancelled context aborts the HTTP attempt. If the request had already reached the API, theIdempotency-Keythe client generated for it is lost with the process. Treat a job interrupted that way as pending and run it again with a new key on the next start; the contract allows a re-submission to spend another credit, which is the price of not persisting keys. If you cannot accept that, store a key on the job row before the call and pass it asVerifyOptions.IdempotencyKey. singleflightis per process. It shares the first caller’s context with the waiters, so cancel with care, and it does not deduplicate across replicas. Cross-instance dedup needs the hash in a database with a unique constraint, the pattern in the queue guide.
The pool is deliberately small. 429 server_at_capacity is server-wide, so four workers and a modest limiter beat sixty workers that spend their time in backoff.
Retry Policy at a Glance
| Situation | Retry? | Same Idempotency-Key? |
|---|---|---|
| Connection error, attempt timeout, truncated body | Yes, with backoff | Yes |
429 server_at_capacity | Yes, after Retry-After | Yes |
409 idempotency_key_in_progress | Yes, after Retry-After | Yes, never a new key |
5xx | Yes, with backoff | Yes |
| Parent context cancelled or past its deadline | No | n/a |
400, 401, 402, 403, 413, 422 | No | n/a |
The one rule to keep: a new key means a new, separately charged request. Never mint a fresh key just to get past a 409.
What This Does Not Catch
Structural analysis has honest limits, and a service that automates decisions should encode them:
- Content that was false from the start. The check finds evidence that a file was changed after it was created. A document whose figures were wrong when it was first produced carries no such evidence, because nothing was changed afterwards. That is why a file produced by consumer software comes back
inconclusiverather thanintact. - Skilled, deliberate fabrication. A well-resourced forger can sometimes produce a file that shows no structural sign of editing. It’s rare and takes effort, but
intactmeans no modification was detected, not that the document is genuine. Confirmation with the issuer remains the stronger check. - Password-protected PDFs. They come back as
422 invalid_pdf; the password has to be removed before submitting. - Cryptographic signature validity. The API reports whether a signature is present, was removed, or was followed by edits. It does not validate the certificate chain.
Pair the structural verdict with domain checks, such as amount validation, account lookups and sender authentication, and your KYC or OCR provider. See PDF Fraud Prevention Best Practices.
Before You Ship
- Set
HTPBE_API_KEYfrom the environment and fail fast at startup when it is missing. - Alert on
payment_required,invalid_api_keyandtest_url_required: all three mean configuration, not user input. - Decide where verification runs (inline for low volume, a worker for high volume) and give every call a deadline.
- Put
human_reviewandrequest_from_issuerbehind real queues with owners; the integration is only as good as what happens to those cases.
New accounts get five checks to try, then pay-per-check credits or a subscription (see current pricing). Copy your test key, run the curl call from Step 1, then the httptest suite above. The API reference documents every response field and error code the client branches on.