Proxying inference requests in 6ms with Pingora, Envoy, and Spanner

4 pointsposted 10 hours ago
by birdculture
(modal.com)

1 Comments

prios8

9 hours ago

Really interesting approach. How do retries work if a stream drops halfway through?