An adventure in writing compatible systems

46 pointsposted 9 days ago
by ezekg

12 Comments

soanvig

5 days ago

I wonder why it took so much time among entire team to actually look into the source code. That's probably third thing I would do after encountering that kind of issue. I would even look for a word "gigabyte" considering it is well documented, because bizarre things should be documented.

Joker_vD

5 days ago

The worst thing is, it's also documented in the SQLite's documentation. The file format is documented, and it describes the existence of this lock-page located at 1 GiB mark. How they've managed to implement support for SQLite DB file format without reading the file format description is, frankly, beyond me.

ozgrakkurt

5 days ago

Curious if having the automated tests do db operations on a persistent db would catch this.

It would do inserts/deletes/updates/create_table/alter_column etc. randomly for a long time, continue even if an operation fails unless there is a crash. This way the db could grow without bound and queries can get very slow

swiftcoder

5 days ago

It only catches it if you re-load the DB into sqlite after every write by turso. At which point you'd flag pretty quickly that it was the next write after 1 GB that messed things up

ozgrakkurt

5 days ago

Could also make switching an operation. So it randomly switches between implementations and keeps using it until the next switch. And can keep running two instances so it is always kind of cross checking

Joker_vD

5 days ago

This. This exact bloody attitude of people who claim "we want to write a compatible system" but can not be bothered to read — not even the source code — the freaking docs, freely available, is why we can't have any decently compatible with each other systems nowadays. There, look: [0], it's explicitly documented right on the SQLite's "Database File Format" page, and has been for a decade at least. How the fuck did you manage to write a clone of SQLite without reading the description of its file format? Like, seriously, how?

[0] https://sqlite.org/fileformat2.html#the_lock_byte_page

rawxtl

5 days ago

I don't even understand the point in rewriting something that is rock solid. And on top of that you guys are asking for full compatibility. This is just nonsense.

vendiddy

5 days ago

I think it's a valid question, but it's better to assume they had their reasons and try to understand why before drawing conclusions.

yencabulator

5 days ago

> To the point that we have a challenge [...]: if anyone can find a bug that leads to data corruption *and improve our simulator to catch it next time*, we will pay you a cash prize.

Uhh, you should be paying people if you get them to work for you.