Experiencing issues which so far are traced down to communication between Googles Cloud Http Load Balancer and Auvious Backend sevices. A workaround seems to work so far, but requires manual action periodically.
Issue was traced down yesterday, and fix was applied immediately since it came down to configuring correctly the maximum connection limits of the http stack of our core services.
So this issue was not due to Google's Load Balancer, what initially lead us that way were the available logs and traces we had. By improving logs,instrumentation and taking more traces we were able to correctly identify and correct the issue.
System is back online since 2022-06-13 13:32:20.220 GMT. The partial outage of the service lasted for about 15 minutes, and it seems stable now. We're closely monitoring the service and investigating the cause with the assistance of Google Support.
Issue has appeared again at 2022-06-13 13:16:17.339 GMT. Users should experience disconnects or failures using the service. We're working on getting the system back in service. The issue cause is latency introduced Google Http Load balancer, so we're working closely with Google's support in order to resolve this and avoid similar issues in the future.
Issue has occured again today between 2022-06-02 16:08:10.678 GMT and 2022-06-02 16:17:58.131 GMT and it affected a small percentage of users.
The related ticket at Google has been upgraded to P1 Critical and escalated in order to resolve this faster.
Issue has not been detected again since yesterday, we're still investigating with the help of Google support the root cause of yesterday's issues.
