Ask HN: AWS CloudWatch Fraud?

7 pointsposted 8 days ago
by uLogMicheal

Item id: 41501283

8 Comments

cddotdotslash

8 days ago

Based on "GetMetricData", you're not paying for services, but rather something with access to your account is making API requests to CloudWatch. Do you have any third-party monitoring tools (Splunk, Datadog, etc.) in use? Can you check your IAM portal to see if you have any users/roles with recent access?

Reubend

7 days ago

AWS definitely doesn't do any "fraudulent" billing for CloudWatch, but there are sometimes very complicated pricing schemes for AWS products, and people often setup complicated systems that they don't fully understand the cost implications of.

In your case, I'd guess that some part of your system, or perhaps some integration that you added to your account, is making API calls without you being aware of it.

ashitvora

7 days ago

Not sure about the fraud but recently we had a very heavy bill from AWS (6x of our usual AWS bill).

After much investigation I realised that one of my dev has setup complicated stuff using some Terraform config he found on Github.

I feel that AWS has very bad UX.

re-thc

8 days ago

Just in case: the thing people often forget is to change regions when checking if something exists. Did you check us-west-1 when checking if this exists in CloudWatch?

Maybe try the aws cli to list and delete?

uLogMicheal

8 days ago

Checked all regions, no services on the AWS side. It seems maybe our old Datadog was still spamming the GetMetricData API by default after cancellation. No logs are collected but they still seem to query the API for every service enabled by default.

brodouevencode

8 days ago

Yes, that's how the DD->CW integration works.

Side note: even if you shut down every service in the account, if something outside of AWS is connected to it, like Datadog, you will still incur charges. I'd recommend deleting the account if you can.

uLogMicheal

7 days ago

Thanks for the insight! I'm sure it costs Datadog a lot to continue to query connected accounts even after the user has cancelled the Datadog service. I reported to them, hopefully they take measures to fix. It would also be nice if integrations could be blocked or de-authorized from the AWS side, but I guess we can't ask for too much in 2024.

lemme_tell_ya

6 days ago

> It would also be nice if integrations could be blocked or de-authorized from the AWS side

I have no idea how DataDog integration works but if I had to guess I'd expect to setup an IAM account for them to use? If so, you could delete the IAM account they're using.