Production Checklist

✅ HTTPS Only

• Webhook URL uses https://
• SSL certificate is valid
• HTTP requests are redirected to HTTPS

✅ Signature Verification

• Implement HMAC signature verification
• Use constant-time comparison
• Reject requests with invalid signatures
• Rotate secrets if compromised

if (!verifySignature(req)) {
  return res.status(401).send('Invalid signature');
}

✅ Timestamp Validation

• Reject old requests (> 5 minutes)
• Prevent replay attacks

const timestamp = req.headers['x-webhook-timestamp'];
const now = Math.floor(Date.now() / 1000);

if (Math.abs(now - parseInt(timestamp)) > 300) {
  return res.status(401).send('Timestamp too old');
}

✅ Quick Response

• Return 2xx within 5 seconds
• Process asynchronously using queues
• Handle errors without failing the request

app.post('/webhooks/sent', (req, res) => {
  // Acknowledge immediately
  res.sendStatus(200);

  // Process in background — include X-Webhook-ID for idempotency
  queue.add('webhook', {
    eventId: req.headers['x-webhook-id'],
    ...req.body,
  });
});

✅ Idempotency

• Deduplicate events by event ID
• Store processed event IDs
• Handle out-of-order events
• Use database transactions

See Handling Retries for implementation.

✅ Retry Handling

• Expect duplicate deliveries
• Handle events at least once
• Implement exponential backoff for your retries

✅ Logging

• Log all webhook events
• Include event ID, timestamp, status
• Use structured logging (JSON)
• Set appropriate log levels

logger.info('Webhook received', {
  eventId: req.headers['x-webhook-id'],
  eventType: req.body.field,
  timestamp: new Date().toISOString()
});

✅ Alerting

• Alert on high error rates
• Monitor webhook delivery failures
• Set up PagerDuty/Opsgenie for critical issues

if (errorRate > 0.05) { // 5% error rate
  await alertTeam('High webhook error rate', { errorRate });
}

✅ Metrics

Track:

• Webhook events received (counter)
• Processing duration (histogram)
• Failed events (counter)
• Queue depth (gauge)

✅ Graceful Degradation

• Handle partial failures
• Continue processing other events
• Don't fail the entire batch

for (const event of events) {
  try {
    await processEvent(event);
  } catch (error) {
    logger.error('Event processing failed', { event, error });
    // Continue with next event
  }
}

✅ Dead Letter Queue

• Failed events go to DLQ
• Manual review process
• Retry from DLQ capability

✅ Queue-Based Processing

• Use Redis/RabbitMQ/SQS
• Multiple worker processes
• Horizontal scaling ready

const worker = new Worker('webhooks', processor, {
  connection: redisConnection,
  concurrency: 10
});

✅ Rate Limiting

• Protect downstream services
• Implement backoff
• Queue events if overloaded

✅ Local Testing

• Test with ngrok/localtunnel
• Verify signature verification
• Test retry handling

See Local Development.

✅ Load Testing

• Test with high event volume
• Verify queue doesn't overflow
• Check response times

✅ Failure Testing

• Test with invalid signatures
• Test with malformed payloads
• Test database failure scenarios

✅ High Availability

• Multiple server instances
• Load balancer configured
• Database replicas
• Redis cluster (if using)

✅ Backups

• Database backups
• Configuration backups
• Tested restore procedures