Incident Report: Loyalty Endpoint Querying All Transactions and Join Tables without Filters

Date: 05 April 2023

Summary:
On 05 April 2023, a Loyalty endpoint was found to be querying all transactions and join tables without any filters due to a missing library in the Service Runner. This was caused by a failure to synchronize the library from the Service Bridge to the Service Runner during a release, as outlined in the SOP. The library in the Service Runner was updated without updating the library in the Service Bridge, which resulted in the missing library. As a result, when the runner hit the endpoint for store-transaction-item, it instead hit the default transaction endpoint, causing the query to retrieve all transactions and join tables without any filters.

Impact:
The incident resulted in a high volume of queries being sent to the Loyalty database, causing performance issues and timeouts for users trying to access the system during the incident period.

Root Cause:
The root cause of the incident was identified as a failure to synchronize the library from the Service Bridge to the Service Runner during the release, which resulted in a missing library being present in the Service Runner. This missing library caused the Loyalty endpoint to hit the default transaction endpoint instead of the intended store-transaction-item endpoint, leading to the query retrieving all transactions and join tables without any filters.

Resolution:
The issue was resolved by updating the library in the Service Bridge and synchronizing it with the Service Runner as per the SOP. Further, a review of the release process and SOPs was conducted to ensure that such incidents are prevented in the future.

Recommendations:
To prevent similar incidents in the future, we recommend the following:

Conduct regular reviews of release processes and SOPs to ensure that all necessary libraries are synced and up-to-date across all services and endpoints.
Implement automated tests to verify that all endpoints are hitting the correct APIs and payloads after each release.
Improve communication between teams to ensure that all necessary libraries and dependencies are updated and synchronized before each release.
Implement proper monitoring and alerting systems to identify and address similar incidents quickly in the future.
Conclusion:
The Loyalty Endpoint Querying All Transactions and Join Tables without Filters incident was caused by a missing library in the Service Runner due to a failure to synchronize it from the Service Bridge during a release. The issue was resolved by updating the library in the Service Bridge and synchronizing it with the Service Runner. Recommendations were made to prevent similar incidents from happening in the future.

We are pleased to inform you that the issue has been resolved. We will provide an update report shortly to explain the root cause of the issue and the steps taken to fix it.

We have discovered the cause of the timeouts in the loyalty database, and our team is currently working on a solution to resolve the issue

Session in the database has decreased. We should see some improvement in the last 15 mins

We are seeing loyalty db session going over the limit. It has cause some intermittent loyalty processor timeout. The team is working on the issue