linsomniac
3 hours ago
That week+-long outage over the summer inspired me to build: apt-cacher-ultra https://github.com/linsomniac/apt-cacher-ultra
It has two primary goals: be able to operate when the upstream is down, and be more robust and reliable than the other apt-cachers. I realized that if we had a major production hardware failure during a ubuntu.com outage, our processes for recovering would be severely hampered. Also, our day-to-day operations tear down and rebuild half our dev+stg stack using our automation to test it; we had to turn that off during the ubuntu.com DDoS.
apt-cacher-ng was doing nothing to help us through the DDoS, and on top of that we had to lay hands on it monthly because it would get into a weird state.