Struggling with Container HTTP Logs? Meet Logmink

13 pointsposted 2 days ago
by mutairibassam
(github.com)

16 Comments

mutairibassam

2 days ago

The Issue: When working with Docker containers, capturing and visualizing HTTP logs can be a pain. You usually need to modify your code or set up complicated logging systems just to see what’s happening with your app's HTTP requests.

The Solution: Logmink! Logmink is an easy, plug-and-play tool that lets you capture HTTP traffic from your Docker containers without touching your app code! Just add the Logmink agent, and all your logs are sent to a central hub where you can store and visualize them.

Why Use Logmink?

Quick Setup: Get started in minutes, no code changes needed. Centralized Logs: View all HTTP requests from all containers in one place. Easy to Extend: Built on Node.js, so you can easily add your own features. Try it out and make monitoring your containers a breeze: Logmink GitHub Repo

#Docker #DevOps #OpenSource #Logging #Containers #NodeJS

euph0ria

2 days ago

What is the performance like when doing this? Doing pcap capturing and processing in node.js feels slow, but that is just a feeling and I don't have any data to back it up.. are there any performance impact studies?

FooBarWidget

2 days ago

I also wonder whether the code actually works reliably. They have an "onpacket" handler on the pcap object, but this handler is, I think, called on every TCP packet. Which may not correspond to an HTTP session.

Consider an app that starts an HTTP session by sending two TCP packets:

1. GET /foo

2. HTTP/1.1\r\n...etc...

Their code would not catch this.

mutairibassam

2 days ago

I agree with you, currently I am only catching the request that have valid http method. I mentioned this in code comments there is todo task to capture different cases as well.

Moreover, I am happy to learn more from you and if you have any recommendation please mention it. This simple implementation has met my needs.

Regarding the performance, to be honest I've not tested it as I am not expert in this domain, so any hit is appreciated.

FooBarWidget

18 hours ago

It would be better to stream the captured packets to a proper HTTP parser. That way you can also handle HTTP keepalive.

Also, you can set a filter on the pcap session so that pcap only gives you http packets. Then you don't have to check yourself whether you captured non-http.

mutairibassam

a day ago

I did some fixes, now it should capture all TCP packets. Appreciate your support to validate.

firesteelrain

2 days ago

I am probably missing something. We just attached a Splunk UF container and forwarded all of our logs (events) to our centralized Splunk instance.

mutairibassam

2 days ago

Please clarify the point or the challenge if this related to the package?

firesteelrain

a day ago

Both tools (Splunk UF running as a sidecar and Logmink) serve the purpose of forwarding logs for centralized analysis. While Logmink has a narrow focus of just HTTP logs, Splunk is log general and can serve the same purpose.

Splunk is considered more enterprise grade. Logmink focuses only on HTTP logs. But Splunk seems more appropriate here for enterprise environments.

mutairibassam

a day ago

I completely agree with your points. I developed Logmink because tools like Splunk are very costly, are only managed by security teams (not application teams), and can take days to retrieve a single log packet, especially in production environments (or sometimes not at all due to security restrictions).

As an application developer, I needed a straightforward way to check my app logs. We tried other tools like Dynatrace, but they were also expensive. We turned to open-source solutions like Logstash, Prometheus, Filebeat, and Grafana. While these tools are powerful, they require significant effort and expertise to set up. What we needed was a simpler HTTP-based solution for tracing requests across servers, which is why I developed Logmink.

I hope this clarifies the difference.

szszrk

17 hours ago

It's interesting that you chose mongodb as a backend. It may be a niche that is overlooked.

Most log aggregation tools are overcomplicated, require A LOT of resources, plus they still rely on storing log term data on S3 assuming that "everyone loves that!". That's not always true.

There are a few tools that try to utilize ClickHouse for a more centralized log storage that works better for small deployments. I found them too limited, or too focused on their enterprise offering (like signoz, which is much more than logs, but lacks some basics that are only in paid versions).

This project my actually be some nice middleground.

Kudos, looking forward for the future of this.

firesteelrain

a day ago

That sounds like a valid use case. I am used to running my own Splunk instances both in Azure Stack Hub and Azure.

Great job!

szszrk

a day ago

I am curious where this conversation will go, but still fail to see what is your actual question/doubt.

These tools are vastly different and on other ends of both complexity and cost spectrums.

PeterZaitsev

a day ago

Hm. Storing logs in MongoDB. Does it really scale ?

szszrk

16 hours ago

Can't tell if this is a joke or not [1].

I think it actually makes sense. I did a few implementations of Elastic and while it was great, but even with all aspects fully managed via code, at some point it consumed insane amount of brainpower to keep expanding, adding features, securing and patching it. It also consumed very large amounts of hardware - which I was fine with mostly, as this was onprem and costs were accepted.

I tried something similar in a more cloud-based environment (both private and public) and I was shocked how much infrastructure I have to pay for just to create a reasonable PoC. Tried same with Openshift/Vector and it just gave me a headache.

There are plenty of situations where you can accept a small risk of even losing some log data (due to low redundancy, scale vertically) and deploy something that is much simpler. Small apps with small volumes, or apps with large volumes, but you are OK with high latency of the solution and you are ok with slow (and cheap) VMs hosting it. It could work well with environments where you prefer to have more of the platform tools as dedicated nodes assigned to separate teams/apps (due to easier charging, or easier handling of noise neighbors, etc).

[1] https://www.youtube.com/watch?v=HdnDXsqiPYo