Previous incidents
usetada.com is down
Resolved Jun 24, 2023 at 12:21pm UTC
usetada.com recovered.
1 previous update
Msite Service and Wallet API are down
Resolved Jun 19, 2023 at 01:55am UTC
Wallet API recovered.
5 previous updates
Unable to login
Resolved Jun 19, 2023 at 12:48am UTC
the storage issue has been resolved.
The database autoscale fail due to limit imposed. The limit has now been increased.
2 previous updates
usetada.com is down
Resolved Jun 16, 2023 at 04:32pm UTC
usetada.com recovered.
5 previous updates
Buyfrom.io is down
Resolved Jun 14, 2023 at 03:47pm UTC
Buyfrom.io recovered.
1 previous update
Issue with redemption
Resolved Jun 20, 2023 at 01:48am UTC
Postmortem Analysis:
Our Redis server encountered numerous persistent, unowned connections. As these connections lacked identifiable owners, they were unable to be terminated, which consequently hindered incoming connections to the server. This unexpected influx resulted in a high CPU and memory load on the server.
The resolution to this issue necessitated terminating these persistent connections. Given the unresponsiveness of the Redis server, we chose to accomplish this by rebooting t...
3 previous updates
Elasticsearch server is down
Resolved Jun 13, 2023 at 03:17am UTC
Postmortem:
Our elasticsearch server instance have the JVM setting for memory used set to 10GB. The instance is using EC2 spot instance and managed with https://spot.io for better cost optimization.
We configured the ES instance within spot.io in such a way that we can use a variety of EC2 spot instance types, depending on which type have the best value at that time.
But the problem is some of the configured instance types doesn't have the memory capacity required for the JVM setting abo...
2 previous updates
usetada.com is down
Resolved Jun 10, 2023 at 04:34am UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved May 24, 2023 at 08:01pm UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved May 18, 2023 at 08:52am UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved May 16, 2023 at 03:27pm UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved May 15, 2023 at 05:59am UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved May 12, 2023 at 03:55pm UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved May 08, 2023 at 05:32am UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved May 06, 2023 at 12:57am UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved Apr 26, 2023 at 07:57pm UTC
usetada.com recovered.
1 previous update
usetada.com is down
Resolved Apr 25, 2023 at 05:19pm UTC
usetada.com recovered.
3 previous updates
usetada.com is down
Resolved Apr 23, 2023 at 07:58pm UTC
usetada.com recovered.
1 previous update
Inventory API is down
Resolved Apr 14, 2023 at 09:15am UTC
Inventory API recovered.
1 previous update
Loyalty database session going over the limit
Resolved Apr 05, 2023 at 03:37am UTC
Incident Report: Loyalty Endpoint Querying All Transactions and Join Tables without Filters
Date: 05 April 2023
Summary:
On 05 April 2023, a Loyalty endpoint was found to be querying all transactions and join tables without any filters due to a missing library in the Service Runner. This was caused by a failure to synchronize the library from the Service Bridge to the Service Runner during a release, as outlined in the SOP. The library in the Service Runner was updated without updating...
4 previous updates
Our replica database is 11 mins behind the primary db
Resolved Apr 03, 2023 at 10:39am UTC
the replication issue has been resolved
1 previous update
Loyalty database CPU consistently going over 99%
Resolved Apr 03, 2023 at 09:26am UTC
The CPU issues has been resolved
1 previous update