Missing Webhooks

Incident Report for Currencycloud

Postmortem

Incident Summary

On 17 June during a routine system upgrade, we encountered unexpected issues that disrupted how messages were processed and delivered across some of our services.

Root Cause

A Debian OS upgrade introduced compatibility issues affecting message handling
Our notifications system slowed down due to an inefficient database query
Some services had trouble maintaining stable connections to our messaging system (RabbitMQ)
Automatic retries overwhelmed the system, eventually redirecting messages into a backup queue

Remediation

Identified and fixed the database query causing the bottleneck
Adjusted messaging configurations to stabilize service connections
Increased system resources and improved retry logic to reduce congestion
Cleared affected queues and ensured all valid messages were processed

Service restored same day, with impacted messages reissued between 18th and 20th June.

Thank you for your patience as we worked to address the issue. If you're still experiencing any issues with missing messages, please don't hesitate to contact our support team directly.

Posted Jun 30, 2025 - 08:22 UTC

Resolved

The incident has been resolved. All missing webhooks have been resent successfully.

Posted Jun 20, 2025 - 17:17 UTC

Update

We are continuing to work on missing webhooks for clients

All Funding webhooks have been resent and will not be sent again.

Posted Jun 19, 2025 - 09:18 UTC

Monitoring

Webhooks have been resent, incident will remain open until all checks have been completed.

Posted Jun 18, 2025 - 17:59 UTC

Identified

We are in the process of resending the missing notifications; however, this may result in some clients receiving duplicate webhooks and emails.

We apologies for any inconvenience this may have caused.

Posted Jun 18, 2025 - 13:48 UTC

Update

We are working to identify missing webhooks so they can be resent, this work is ongoing

Posted Jun 18, 2025 - 10:13 UTC

Investigating

Due to the incident on the 17th June, we are investigating some clients missing important webhook notifications, after incident was mitigated.

The team is working to identify the missing webhooks so they can be resent

Posted Jun 18, 2025 - 07:50 UTC

This incident affected: Notifications.