GitHub blames 8-hour outage on autoscaling fail

2 pointsposted 8 hours ago
by indigodaddy

4 Comments

indigodaddy

8 hours ago

"Engineers mitigated the problem by temporarily reducing gateway retries through a code change and configuring the load balancers to reject inbound Copilot Token Service requests with HTTP 403 responses."

^^ How can the LB reject a request when it hasn't hit the backend yet to known what the response would be? (eg the 403)

awsanswers

8 hours ago

Inspect headers for the request you need to filter, serve the 403 from the load balancing tier

indigodaddy

4 hours ago

Ah ok, so the point is to protect the backend then? But by inspecting the headers are you still not hitting the backend? I guess that winds up putting way less load on the backend?

tanvirk09

4 hours ago

is this something that code review tool may have caught?