PiiBlur

'How to Build a Webhook-Based Redaction Pipeline'

'PiiBlur Team'5 min read

Polling is fine for a prototype. A production redaction workflow usually needs webhooks.

The reason is simple: image and video redaction is asynchronous. Your app uploads a file, PiiBlur queues the job, and the result becomes available later. If workers sit around polling every job, they waste capacity and make failure handling harder. A webhook pipeline lets upload workers submit jobs quickly, then lets a separate handler deal with completed files.

This article describes the shape of that pipeline. It is not tied to a specific queue provider; the same pattern works with SQS, Redis, RabbitMQ, Sidekiq, Laravel queues, or a database-backed job table.

The pipeline shape

A reliable redaction pipeline has five pieces:

  1. Source storage - where the original image or video currently lives.
  2. Submission worker - sends the file to /api/v1/media/redact.
  3. Job table - records your internal object ID, the PiiBlur media ID, status, selected categories, and retry count.
  4. Webhook endpoint - receives media.processed and media.failed events.
  5. Download worker - fetches the redacted output and stores it in your final bucket.

Keep submission and downloading separate. Uploading 10,000 images should not wait on the first 10 to finish.

Submit jobs with idempotency keys

Use an idempotency key for each source asset. If a network timeout happens after PiiBlur receives the upload, your retry can safely repeat the request.

curl -X POST https://piiblur.com/api/v1/media/redact \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Idempotency-Key: claim-photo-78391-v1" \
  -F "[email protected]" \
  -F "categories[]=heads" \
  -F "categories[]=license_plates" \
  -F "redaction_method=blur"

Store the idempotency key next to your internal asset record. Do not generate a random key on every retry; that defeats the point.

A useful key format includes the source asset ID and a version number:

  • listing-photo-44822-v1
  • dashcam-segment-20260608-0915-v2
  • claim-photo-78391-v1

Increment the version only when the file or category selection changes.

Record enough state to debug failures

Your job table should not be an afterthought. At minimum, store:

  • Internal asset ID
  • PiiBlur media ID
  • Original filename
  • Media type
  • Selected PII categories
  • Redaction method
  • Submission status
  • Processing status
  • Retry count
  • Last error code and request ID
  • Timestamps for submitted, processed, failed, and downloaded

The request_id from API errors is especially useful. Log it with your own job ID so support can trace the request if something fails.

Verify webhook signatures

Treat webhook requests as untrusted until the signature checks out. PiiBlur sends an X-PiiBlur-Signature header. Your handler should compute the expected HMAC over the raw request body and compare it with a constant-time comparison function.

Reject requests that fail signature verification. Do not update job state from an unsigned payload.

Make webhook handlers boring

A common mistake is doing too much inside the HTTP webhook request. Keep the handler short:

  1. Verify the signature.
  2. Parse the event.
  3. Insert the event ID into a deduplication table.
  4. Update the job status.
  5. Enqueue a download or review task.
  6. Return 200 OK.

The actual download can run in a queue worker. That gives you retries, timeouts, and observability without making PiiBlur wait for your storage provider.

Deduplicate events

Webhook delivery systems may retry. Your endpoint should be idempotent.

Use the event ID as a deduplication key. If your database already has that event ID, return 200 OK and do nothing. This prevents double downloads and duplicate status transitions.

Also make status updates monotonic. A completed job should not move back to queued because an old event arrived late.

Download only after completion

The media status response includes a download_url only when processing is complete. Your download worker should:

  1. Read the media ID from your job table.
  2. Fetch the download URL or use the URL from the processed webhook payload.
  3. Download with the same Bearer API key.
  4. Store the redacted file in your own bucket.
  5. Mark the job as downloaded.

Keep originals and redacted outputs in separate paths. For example:

s3://media-originals/claims/78391/photo.jpg
s3://media-redacted/claims/78391/photo.jpg

That separation makes accidental publication of originals less likely.

Retry policy

Not every failure should retry the same way.

  • Network timeout - retry with exponential backoff.
  • HTTP 429 - respect Retry-After, then retry.
  • HTTP 500 - retry a small number of times.
  • HTTP 409 - check whether the idempotency key is still processing or was reused with a different payload.
  • HTTP 422 - do not retry until the file or request is fixed.

After the final retry, move the job to a dead-letter queue. A human should be able to see the file, error code, selected categories, and request ID without digging through logs.

Review rules

Automated redaction is not a reason to skip review for high-risk media. Add review gates for:

  • Public release footage
  • Legal evidence
  • Health or education records
  • Media involving children
  • Any output where the source file was low resolution, blurry, dark, or heavily compressed

Low-risk internal workflows can often use sampling instead. Review 1-5% of completed outputs, track misses, and adjust category selection if the sample reveals a pattern.

Related API pages

For exact request parameters, see the API documentation. For focused examples, see the Face Blur API, License Plate Blur API, Image Redaction API, and Video Redaction API pages.