The "original" dead man switch as I heard about it was a pedal on a train that would apply the brakes if the operator released it.

I've often wondered about how to reliably take software actions after my death or dishonor. After all, you can't really rely on me being able to pay my bills. I'm not looking to do something expensive, more like delete my accounts and send some messages.

I thought this was going to be some way to exact retribution on your employer after being laid off

Wire in pagerduty to whatever your action is. Ideally distributed on multiple infra pieces. This is just the switch side not the action side.

Normally deadmans switches can be compromised by disrupting the deadman switch hardware. This removes that attack vector and pushes it further up the chain (which may or may not help you).

It's certainly very clever.

> This is a watchdog timer / monitor / heartbeat, setting off an alert if a timer elapses.

Notice from the config file, this doesn't actually do checks, only has passive listeners with a remote switch to toggle. That makes this a dead man's switch, not a watchdog / angel / monitor / heartbeat check.

This doesn't do anything except sit there watching a clock not knowing if you're dead.

The service you want monitored, the service you want to be sure the heart is beating, has to ping this to demonstrate aliveness.

If this doesn't get a ping in time, it then acts by telling pagerduty to alert.

If your service (man) is dead (or your service otherwise can't reach this to check in in time), this switches PD to alert mode.

So, dead man's switch does seem a fair characterization.

Seems very similar to a 'watchdog', just... reporting instead of doing anything about it.

There’s no crontab for banking days. Deadcheck also requires checkins within a delta of the expected checking time.

Deadcheck also doesn’t rely on your infra to alert.

Yeah I thought "dead man's switch" was such a weird name for this.

uptime monitor?

I agree, I think people underestimate the importance of actually checking why the heart beat was missed, double checking is important. You might think you can easily click some button once a week and it is no problem to take some drastic and maybe irreversible actions after a week without any event. And then you get into a serious accident and end up in hospital for weeks. Or there is some bug unintentionally triggering the actions.

It sets up expected check in times for an app/job. The job is responsible for checking in, but the alert will fire regardless of your infra.

Doesn’t require a start command and your infra can completely fail and you’ll still get alerted.

There is a pagerduty provider for terraform, too. https://registry.terraform.io/providers/PagerDuty/pagerduty/...

It doesn't look like PagerDuty supports sending out alerts on missing heart beats

Actually I got fried and very bad remarks on my project. Not true that hn crowd is friendly to show hn projects at all.

Yes, it’s a positive signal when people take it seriously enough to criticize.

Was thinking the same!

Better Stack got on my disapproval list, for spamming me with a series of automated sales emails 15-20 minutes apart, which seemed designed that way. "One more thing.", etc.

They should've sent one welcome&upsell email, not several. And not abused contact info that was intended to reach me reliably and promptly.

For my immediate very simple need, Uptrends didn't spam me, and so far they've reliably and promptly notified me whenever my site is down even briefly (e.g., rebooting for kernel update), so they'll be getting money from me as I grow. https://www.uptrends.com/

or Checkly's heartbeat check

https://www.checklyhq.com/docs/heartbeat-checks/

Yea, and you would delete the pending message. Good idea…

This is an attorney. No subscription, no tech, no bugs, no encryption, less hackable.

I've thought about this too but I think the type of business best suited to execute this is a law firm, or you know, the normal executor of people's wills. They just need the IT bit.

Yea I’ve setup two alertmanagers that check each other before. It’s useful for multi site deployments.

If the infra goes down, you fix it while alive, otherwise the snooze timeout on the PagerDuty incident will expire and that will trigger the configured tasks. If PagerDuty goes out of business before that, then that's a different discussion.

Yea I want to support other vendors. What do you use?

Thanks, yea I’d like to support other vendors. Haven’t heard of Squadcast

I have https://github.com/adamdecaf/deadcheck/issues/12 tracking additional vendors

Edit: I'm open to a PR if you're willing. Curious, but would you use their delayed notification config to implement?

https://apidocs.squadcast.com/?version=latest#7742a9af-29fe-...

I'd argue entirely opposite, that a dead man's switch should 100% rely only on someone else's services and should avoid any dependencies on your stuff, as something happening to you is correlated to something happening to things you run or own, and the whole point of such a switch is that it should function properly when e.g. all your stuff burns down or gets confiscated or whatever.

Okay, so now your dead man’s switch depends on your home or colo’s power, internet connection, compute hardware, storage hardware, network hardware, software stack maintenance, plus the ability to pay bills for those on time?

Pre-Paying a cloud service pennies per month to do all that for you will likely be more reliable and much simpler.

Yep. I looked at Cronitor and thought about using it, but direct to PD removed a step for us at Moov.

How are they triggering the alert when the dead man’s switch isn’t tripped in time?

Your infra is relying on an ISP and colo and the power company and maybe even the Post Office. You're always relying on something that's not you. Personally I'd rely on pre-paid AWS. If all of us-east-1 goes down for days then we probably have bigger problems.

That's what I thought; this tool is "kicking the can" to whatever third party service you're using. So, it's essentially just a tool that will ping stuff for you in a slightly more convienent way?

It's a cloudification of dead man's switch.

Now two isolated systems have to fail. Redundancy is fault tolerant. Nothing is fault proof.

I ended up going this route: https://github.com/jrockway/alertmanager-status

Yea I’ve setup multiple Alertmanagers that all check each other before. That setup is useful to detecting route failures between sites.

No, you can get text, email, slack, etc alerts from PagerDuty without installing their app.