Uploading from GitHub fails 100% of the time

[jrfnl]

... with the following error message:

---
Error: Service Unavailable (503)
Message: <h2>This website is under heavy load (queue full)</h2><p>We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.</p>
---

Considering it is the weekend, I find the "heavy load" slightly hard to believe. This has also been going on for at least 2 hours based on what I've observed, but may well have been going on longer.

I also get the impression something bigger is going on based on the graph on the status page:

(and yes, I'm aware those only represent one of the servers, not the total picture)

/cc @afinetooth - I doubt this is related to the migration to the App, but copying you in anyway, just in case. Note: I'm seeing this across various organisations, including some not involved in the migration.

[afinetooth]

@jrfnl Thanks for the report.

First, the WEB/API RESPONSE graph is a known issue. It's in our backlog but it requires an annoyingly involved fix, which we've decided to perform after another infrastructure change scheduled for our current sprint. So it will remain that way for the next 1-2 weeks. Rest assured it does not reflect the actual response times, we are only having an issue with the graph data reporting to the status page.

The DEQUEUE TIME was more concerning, but having investigated our infra APM over the weekend, I see that we had one (1) web server (out of 6) go down over the weekend at ~04:58am PDT Sun Jul 26 for ~11hrs (we manually rebooted it and reverted the test deployment at around 04:15pm PDT Sun Jul 26). This server contained a test deployment aimed to address the occasional 503/504 responses we were seeing across the fleet, which obviously did not resolve the issue given that it went down and stayed does (the rest of the fleet has a recovery sweep that will reboot downed servers, but none went down according to our monitoring, just the one). So I'm curious why you received this more than ~20% requests given that the load balancer should have routed you to the other servers during that period.

I've asked an engineer to analyze the logs to see if we can identify your requests (we generally know where they're coming from). I'll update this ticket when I hear back.

---

P.S. this should be wholly unrelated to the GItHub App upgrades, or specific orgs.

P.S.S. If you every want to share the Cloudflare trace you get with a 504, that will let us track a specific request(s).