Status: Resolved

Summary: All our Integration API, Wallet API down due to bugs

Impact: All request through Integration API, Delivery order and Merchant link experience timeout

Trigger: A condition in the source code open a connection but fail to close it after the condition ended.

Root cause:
We had schedule release this morning at 12am - 3am. After the release all test was done and no major issue detected. But when the real traffic coming around 7am the team get alerted of login and timeout issue. The initial suspect was the networking issue but team alerted that many connection to database fail. Normally that happen when we have large unexpected traffic. But no big traffic identified by the team.

In Parallel, another team spotted many connection to database has been opened but not close. The team then focus on identifying the code those created the issue and found the connection leak happen on certain condition only. The code was rectified and deployed to production.

Lesson learned:
Engineering team updated the Sinful Script to the team so it does not repeat again.

We've resolved the issue, all service are up.

Issue has been resolved. The team is monitoring the fixes. This ticket will still be open for update

Update: the root cause is from client application opening database transaction but failed to closed it, resulting in new database connection timed out and affecting the bridge and order API. Currently we're deploying the fix to resolve this issue.

There is a possible memory leak on the core system deployed this morning causing connection issue to our database. The team is working on the issue.

Issue has been identified. There are connection issue from our core system to database on some of the instance.

Dashboard is up and running

There maybe a case where user cannot login to Wallet apps, web, AVBO and Dashboard as our Auth system is having a problem at the moment