On September 8th, 2024 from 8:08 a.m. PDT to 9:08 a.m. PDT, the web view that displays some elements of our user interface such as our giving form, live stream player, and on-demand media for the web app all experienced a partial disruption. While the underlying infrastructure was not affected, giving forms and live stream players were not accessible for some users. 

What caused the disruption?

• The issue was caused by a memory leak in the underlying framework of the Web App

What are we doing to ensure improved platform reliability during ministry critical times?

• Since the incident on Sunday, we have already made a number of improvements to enhance the reliability of the Web App such as:

  • Ran load tests specific to identifying memory management issues within the Web App to ensure memory issues will not affect users
  • Set a lower scale up threshold to trigger auto scaling and a much higher horizontal auto scaling limit so that more pods will be spun up more quickly based on demand
  • Configured memory limits for the pods, such that the pods will be restarted if memory utilization spikes again, before there is an issue

Did any elements of Subsplash Giving remain operational during the disruption?

• 93% of gifts were processed as expected on Sunday.
• While some clients were unable to process new gifts during the 60 minute disruption, many foundational elements of Giving remained fully operational such as recurring gifts, statements, Donor Management, offline giving, etc. 
• Our data also shows many users gave later in the day on Sunday, suggesting that many who were unable to access Giving during the service interruption, did indeed complete their donations at a later time. All recurring gifts processed as expected.

We’re committed to improving and investing more into platform resiliency, redundancy, and uptime. We exist to equip churches and ministries to share the gospel and make disciples. Experiencing a service disruption that impacts your ministry is simply not good enough.