Trigger.dev customer environment variables exposed

167 pointsposted 3 days ago
by anurag
(trigger.dev)

96 Comments

corvad

3 days ago

I am sorry, this is "Resolved"? How can they even consider marking this resolved in any situation. Yeah we exposed your env vars but we told you about it over email so it's all good... We dont know how or to what extend though.

janpio

3 days ago

What would you consider appropriate action for their incident team to be able to mark the incident on their status page as resolved?

ygouzerh

3 days ago

Never put it resolved

Joker_vD

3 days ago

Why? What purpose would having an eternally open incident serve?

phs318u

2 days ago

As an eternal reminder of the need to do better? Those who fail to learn from history are doomed to repeat it?

Joker_vD

2 days ago

> As an eternal reminder of the need to do better?

Eternity is a very, very long time. Longer even than 80 years (arguably the total age of the whole "programmable electronic computing and data storage machines" industry), you know?

So I don't think that having an open issue in a bug tracker forever is even possible; not that it would even serve as a remainder — I assure you, most long-existing firms have "ignore open issues older than X years" filter enabled by default in their Jiras or whatever people use nowadays, those are only ever looked at once year by the upper management when they're out of ideas.

nilamo

2 days ago

A status page is not a historical log.

vdfs

2 days ago

At least a postmortem or what went wrong would be a start

htrp

2 days ago

Resolved = it's your problem now?

dangoodmanUT

2 days ago

As I mentioned in the Northflank breach post:

Unfortunately for infra startups, I think we're going to see a convergence back to hyperscalers as a result of incidents like this and railway.

There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises

Also, a trick i've started doing is encrypting my env vars, then hard-code the decryption key in the code

willio58

2 days ago

Is your implication that hyperscalers (big companies) will always do better at this than smaller companies?

Just becuase a company has more money doesn’t mean they direct that money well, and even money directed well at the wrong people will result in failures.

mbb70

2 days ago

No one ever got fired for putting creds in AWS Secrets Manager. I'd take the hyperscaler every time because even if something happens, it is a very defensible decision.

bitlad

2 days ago

It is probably safer if you enable HSM.

bitlad

2 days ago

Sometimes compliance do help improve security. Hyperscalers have resources for that. They have resources and they do security well. Hyperscalers are treated as critical infrastructure.

One person's machine being compromised does not bring the whole org down.

jen20

2 days ago

Not necessarily _always_ (Azure anyone?) but for the others - absolutely.

user

2 days ago

[deleted]

sebmellen

2 days ago

Yes, I’d say any infra provider that can't afford to have their own hardware security module should probably not be storing sensitive customer data.

viraptor

2 days ago

Unfortunately the cost of HSM is the cost of two of them, overprovisioned for your projected needs over years, and staff who know how to use them safely both in terms of security design and not losing access to everything in case of an issue.

bitlad

2 days ago

Just having HSM is not good enough. You also need a process around it to keep it secure. you can also use Cloud HSM solutions.

theturtletalks

3 days ago

The biggest case for building bespoke software is that you drop your attack surface area and it’s not as lucrative to hack one person. Don’t even expose it to the internet if you don’t need to and just run Tailscale.

seanw265

2 days ago

I don't see bespoke software as a viable alternative because probing for vulnerabilities is both getting cheaper by the day, and also massively parallelizable.

Someone will inevitably spin up a large swath of agents running an abliterated pareto curve model and point them at the internet.

Not to mention your bespoke tool is likely way less hardened than the existing player in the space.

cbluth

3 days ago

I think you've just explained why people rent infrastructure.

sergsoares

2 days ago

Agree in don't ever expose if not needed.

And keep private and public personal hosted services in different contexts.

pelagicAustral

3 days ago

Tailscale got hacked recently as well, didnt they (?). I think they did, the whole system is poisoned, because I believe one of the vectors had to do with eMudhra certs, which again, is not an issue specific to them, or something they had any say on it... there is literally no escape these days it would seem.

thewisenerd

3 days ago

tailscale got hacked recently? sources?

they had a priviledge escalation vulnerability that requires the attacker to already be a part of the tailnet, and more specifically allows a user-to-root escalation devices the attacker can tailscale ssh to.

> Not trying to play down the vuln (its important), but its not a free-for-all RCE that some ppl are acting like it is. [1]

eMudhra/TLS certificates have nothing to do with this (?).

[1] https://news.ycombinator.com/item?id=50266179

theturtletalks

2 days ago

That was Tailscale SSH and it’s not on by default

jen20

2 days ago

It also was not related to the mentioned CA, and was not "hacked".

bitlad

2 days ago

Or do onprem or selfhosted. Surface area reduces drastically.

seanw265

2 days ago

The tradeoff is that you become your own ops team and need to monitor and respond to cves in real time.

user

2 days ago

[deleted]

theturtletalks

2 days ago

I was also referencing that using commercial software usually brings features you probably don’t use. Those features probably require new NPM packages so by building a bespoke software, you only build features you need.

shepherdjerred

3 days ago

This is probably correct if you aren’t a target at all, e.g. you aren’t running a business

castis

2 days ago

As far as my personal life is concerned, I honestly don't think I'm going to 'build and deploy' anything to any SaaS that isn't directly in my control for a while.

fnoef

2 days ago

damn, i've been thinking about the same recently. saas is dead, at least developer oriented saas for sure. the only things i rely on are my own self hosted git (not public to the web), and that's it. all other "developer oriented saas" has been replaced with a home made version with claude. probably not safe from hacking, but id rather take the risk of someone targeting ME specifically, than let all the "saas platforms" store all my data and get hacked every monday and friday

sergsoares

2 days ago

Yep, that triggered in my mind to move the last public service (not critical) I was hosting inside a third party saas because of little conveniences.

iampims

3 days ago

It’s bold to set status page incident to resolved.

weird-eye-issue

3 days ago

If they fixed the exploit then what do you want them to do? Do they need to wait an arbitrary amount of time before setting it to resolved?

eddythompson80

3 days ago

I don't disagree with you, but I think generally when the mistake involves compromised user data, the answer is a bit more emotionally driven. People get rightfully mad when things like that happens, and services usually keep the incident in a "Mitigated" status while they "work with their customers to ensure blah blah blah" or whatever. Everywhere I've personally worked the Mitigated->Resolved involves publishing a postmortem, repair items, and general assurances or safe guards based on learnings from the incident that the issue won't happen again. However, the industry as a whole doesn't have a standard for what the line between Mitigation and Resolution is.

weird-eye-issue

3 days ago

So the active incident is now fixed and they've already reached out to impacted customers but they can't mark it as resolved yet because they are still yet to publish a more detailed blog post?

jtchang

3 days ago

It's not that they can't mark it as resolved. It's just that it is bad business to do so.

How they handle the communication on the incident is crucially as important as remediating the original cause.

Just the fact that they are doing this means I will probably never use this service or recommend this service to anyone else.

weird-eye-issue

3 days ago

Okay so let's just say you are one of those affected users

The founder of the company has already emailed you with details and their support team is around for you to ask any follow-up questions. Also the bug/exploit has been fully fixed

At this point are you saying you would actually be upset and leave because the incident on their status page has been marked as resolved?

tuwtuwtuwtuw

3 days ago

Yes.

weird-eye-issue

3 days ago

Drama queens

sebastialonso

2 days ago

> Ask questions > When showed different opinion, complains like a bitch

HN is very cesspool-ly indeed

weird-eye-issue

2 days ago

> complains like a bitch

That's what you are doing, congratulations

whilenot-dev

3 days ago

What makes you think it's fixed? They just state that they "blocked the exploit", and that could mean anything:

- did they just take the affected API offline?

- or did they just disable the account of the attacker?

- or is the exploit actually being fixed?

Environment variables are sensible data, and marking a security incident as resolved with that little detail is quite a take. Even changelog entries for harmless bug fixes are more detailed.

weird-eye-issue

3 days ago

I assume it's fixed based on it being marked resolved and from what was communicated. Yeah I agree it's a major issue but if it's fixed what they have done so far seems totally fine

tomrod

2 days ago

To use a fearful, apt analogy: resolving the operational issue isn't just plugging the hole in the dam, its also fixing the damage by pumping the water out and then addressing the damages.

weird-eye-issue

2 days ago

In your analogy they can't pump the water out. Now it's up to their users to rotate their keys. How do you expect them to do that sort of thing on behalf of their users?

tomrod

2 days ago

Depends on the TOS, but, blocking service until confirmed or decommed would get the trick done.

weird-eye-issue

2 days ago

That doesn't even make sense. People use environment variables to store things like API keys for third-party services. How do you expect them to magically rotate or decommission those keys?

tomrod

2 days ago

Trigger.dev wouldn't rotate customer BYOK. What I was saying is they disable customer use of their service until rotation occurs or is confirmed by the customer.

weird-eye-issue

2 days ago

That doesn't really help anything and just makes the exploit impact even worse since it would be effectively a denial of service

Traubenfuchs

3 days ago

They are no longer exposing anything, so they no longer have an incident.

Their customers now have an incident that will be resolved once their secrets are rotated. Maybe they also need to do previous intrusion forensics… In any case, not triggers problem anymore.

cheeze

3 days ago

I like the fancy gradient INCIDENT REPORT image as well. At least it looks fancy and AI-ey

rugma

3 days ago

they could have explained a bit better with proper postmortem and steps taken to improve, before marking it as resolved

reassess_blind

3 days ago

Have any of these companies disclosed the actual vulnerabilities used to gain access to their systems?

dboreham

3 days ago

No. Which may mean they don't know.

tough

3 days ago

the recent CVE making any non-patched TCP exposed linux server RCE-root-able seems like a good stretch to me.

But could also be any of the recent supply chain attacks hitting a dev too i guess

bbartlomiej

2 days ago

Northflank mentioned they patched CVE-2026-98432. This appears to be similar, though not confirmed by trigger.dev yet

BADCAFE

3 days ago

This also happened to a trigger.dev competitor a few months ago, inngest.

I was able to patch before it was exploited but that definitely made me question the service.

cpursley

3 days ago

The very fact that Nextjs vibe projects need an external (and paid!) background jobs / async service show how much of an anti pattern that this framework and similar are. Just use a sensible stack from day one that can handle concurrency out of the box like Elixir or something with established open source background jobs library. Then you can worry less about this kind of paid 3rd party nonsense. Ditto on auth. I just can't fathom outsourcing these types or core functionalities, the operational, security and cost risks make very little sense when there's libraries that do this out of the box.

multiplegeorges

3 days ago

Yes, outsourcing these kinds of core functionality (your queue, your users table!) is insane to me.

asdf88990

3 days ago

Auth0 and friends are useful for social logins but that should be about it, not replacing your user table wholesale like people do.

cpursley

3 days ago

There are open source libraries even for that sort of thing.

user

3 days ago

[deleted]

dgellow

3 days ago

You can run trigger in your own infra. That has very little to do with the libraries used for your web app

cpursley

3 days ago

Trigger and friends exist precisely because Node is not a serious backend runtime so you have to reach for bolt-ons for every little thing. No matter if trigger is open source or managed, it's an anti-pattern. Erlang/Elixir, C#, Go etc all handle this sort of async natively, literally out of the box. Mature frameworks like Rails, Django have pretty easy to use officially libraries as well. This has been a solved problem for literally decades.

dgellow

2 days ago

That has nothing to do with node. You have a lot of situations where a dedicated job queue makes sense

cpursley

2 days ago

The reason people are using these paid queue services is because Node sucks. Isn't it better to use an open source library or adult backend stack which solved this problem 30 years ago?

dgellow

2 days ago

Trigger is literally open source. That has nothing to do with how mature the node ecosystem is. If you want in-process queue systems you have a lots of other mature options. Trigger is a dedicated queue system, it runs in its own process and can be used from any language through its API. You seem to talk confidently about a topic I don’t think you understand.

Go has lots of mature in-process queue systems, yet River exists, because in lots of situation it makes perfect sense to have your job queue in its own process

ishandotpage

2 days ago

Have you actually tried using the open-source self hostable version of trigger? It seems to only exist in order to check a "we're open source yay" box.

It's missing lots of features like sleeping on waits, warm starts etc.

rvz

3 days ago

Given the recent Claude Code remote access tool, it would be wise to not install any new software, or use any package manager for a few weeks as they could have already been silently backdoored during the aftermath of that incident.

tough

3 days ago

I was paranoid enough to just remove all my Chrome/VSCode etc extensions.

Im now thinking about just running tart VM's on my mac to do my regular stuff from.

sergsoares

2 days ago

I really ditched now VScode/Cursor after the extension vulnerability (beside I never used that, only the Claude code CLI).

Despite no using full VM's I using Orbstack Isolated machines to some level of restrictions https://docs.orbstack.dev/machines/isolated for coding.

tough

a day ago

tart is great to orchestrate native Virtualization.Framework hypervisor VM's on macOS. I've been meaning to use more but yeah i mostly do codex/claude/opencode cli directly on my drive and havent had any scare in 3 years

cannonpalms

2 days ago

*VS Code extension for Claude Code

eddythompson80

2 days ago

*VS Code extension for Claude Code for Cursor VS Code Fork

user

3 days ago

[deleted]

asdfsa32

3 days ago

So at this rate, how long before people start hiding use of AI in development of in their SaaS and services? or people really don't care, do they?

Kiro

3 days ago

Hand-made code is riddled with dirty hacks and exploits that LLMs have no problem finding. I would not trust any software that is not using frontier models to fix the messy human code.

asdfsa32

3 days ago

Oh that is right, the frontier models yield amazing architecture and secure code in the form of aislop.

There is a difference between using AI to find bugs in written software than using AI to write bug free software, completely different set of skills and outcomes. Most of the high profile stuff you see is AI being driven by people who would have solved the same problem just in slower time going through the code. AI is as good as the operator.

CharlieDigital

2 days ago

This should not be marked as "resolved". "Mitigated" until some percentage of customers ack the email notification at least (track clicks, get teams on Zoom call, etc.).

liorgrossman

2 days ago

Another day, another infra provider hacked.

user

2 days ago

[deleted]

fathermarz

3 days ago

Holy smokes, is security just not a priority at all anymore? Like in what world do we live in, that this is acceptable. And at the volume this has been happening, bad guys are having a FIELD DAY.

OccamsMirror

3 days ago

What makes you think security was ever a priority?

_pdp_

2 days ago

In my experience, small teams put security off because someone on the dev team has some security experience, or thinks they do. But it is never the priority, there is always something more urgent to ship, and nobody has real skin in the game because it is not their job. Then shit happens. They rush to hire a security team, and that team spends the next few years fighting the culture.

fathermarz

2 days ago

It makes sense, I have a skewed perspective because I have only been in security software.

Thanks for that anecdote

dgellow

3 days ago

Security has never been a priority in the software world

fathermarz

2 days ago

I think maybe my lens is slightly skewed because I have only worked in security software? This is shocking to me.