Show HN: Python Concurrency Testing with Frontrun

2 pointsposted 9 hours ago
by lucaswiman

1 Comments

lucaswiman

9 hours ago

A bit more about how it works internally. It uses sys.monitoring (or sys.set_trace for older versions of python), as well as LD_PRELOAD / DYNLIB libc patching to track arbitrary io events. It also monkeypatches basic concurrency primitives like the contents of `threading`. (Given how heavily monkeypatched that all is, I've been running frontrun tests separately from an ordinary test suite. The bundled binary handles all that when used with Pytest.)

It tracks "conflicts" at mutable boundaries, building: 1. A "shadow stack" recording when values are assigned to objects which are referenced from more than one thread. 2. A finer-grained model of SQL and Redis items that take into account some of the semantics of their concurrency models.

The code executes normally once, to get conflict points, then it uses a version of an algorithm called Dynamic Partial Order Reduction (DPOR) to efficiently search through all non-isomorphic traces looking for failures (either a crash, a statically detected deadlock, or a violation of the property/invariant provided in the test). This was partially inspired by the rust concurrency testing library Loom, though uses monkey-patching instead of customer concurrency primitives.