Show HN: CT Logs Search Engine

5 pointsposted 6 days ago
by Eikon

3 Comments

conradklnspl

6 days ago

Creative way to get domain name monitoring. Do you use this to keep a history of DNS records to, by any chance? If so, do you find it to be reliable?

I would suggest making the domain name clickable: https://www.merklemap.com/search?query=datafragment.com&page... Currently, I need to copy and paste in my browser.

What do I get by signing-up? Even with a very large domain like google.com, it looks like all results are available for free without limit: https://www.merklemap.com/search?query=google.com&page=1

I'm working on https://datafragment.com with a couple colleagues, and domain monitoring is definitely one of the things we'd like to try and sell to prospects.

Eikon

6 days ago

> Do you use this to keep a history of DNS records to, by any chance? If so, do you find it to be reliable?

While that wasn't the initial goal, it's an interesting application I hadn't considered. The project actually started as an attempt to build an IPv6 database, which led me down this particular rabbit hole. DNS record history could be a valuable feature to add in the future.

> What do I get by signing-up? Even with a very large domain like google.com, it looks like all results are available for free without limit: https://www.merklemap.com/search?query=google.com&page=1

Currently, signing up doesn't provide additional features. It's mainly a way for me to gauge interest and identify potential power users. I have plans to add premium features in the future, which will be available to registered users.

> I'm working on https://datafragment.com with a couple colleagues, and domain monitoring is definitely one of the things we'd like to try and sell to prospects.

That's great to hear! I'm open to collaboration. For now, you can use my API at https://api.merklemap.com/search?query=example&page=1. The rate limits are quite generous, but please be mindful of the load, searching, with wildcards across that many entries, as you might guess is _quite_ expensive :)

You can also get the live ingestion data feed using

    curl -N 'https://api.merklemap.com/live-domains?no_throttle=true'
> I would suggest making the domain name clickable: https://www.merklemap.com/search?query=datafragment.com&page... Currently, I need to copy and paste in my browser.

Excellent suggestion, thank you! I'm actually working on implementing this feature. The plan is to make each domain name clickable, leading to a detailed page that will display subdomains, certificates, and other relevant data from the CT logs.

conradklnspl

6 days ago

DNS history is really interesting for a variety of reasons.

I've seen it used to try and circumvent Cloudflare firewall rules. Some people don't replicate the firewall rules on the servers behind Cloudflare. If they've ever pointed their DNS to their servers directly before turning on Cloudflare proxy and you have that old IP address, then that IP address has value. White hat security firms for instance pay for that information when running audits.

I will sign up, curious to see where this project leads you.

Thanks for the API details, will discuss this with my teammates.