Scaling Golang CI by Replacing actions/setup-go

74 pointsposted 3 days ago
by peterldowns
(cloudx.ai)

27 Comments

wannabe44

2 days ago

I always advocate having custom-built docker images for CI, periodically refreshed for security fixes. CI should not run more than few seconds over the standard time to run the same thing from a dev machine.

However, other people around me are fine with apt installs and pip installs from global mirrors in every CI run. So I may be just autistic.

bumblehean

2 days ago

I do this occasionally. But it's because my company's security org has a ridiculous policy enforcing 0 CVE images for anything running on our infra (including internal CI images).

In some cases baking a dependency into an image is signing myself up for perpetual toil to keep the image CVE free.

hackerbrother

2 days ago

Sure. Although you can expect build/test time to take a while on bigger projects, so I think it washes out a lot of the time.

pstuart

2 days ago

Yeah, that drives me bonkers. Just set up a separate CI/CD for the images themselves that update daily as needed.

ehe78qhe

2 days ago

We set up a cache that detects when the project Dockerfiles or lockfiles change. If dependencies haven't changed, we check cache first, otherwise the image rebuilds as the first step of that pipeline and following pipeline steps use it. Best of both worlds.

peterldowns

3 days ago

Hey everyone, one of the authors here. This is a "small" improvement that has saved us a LOT of developer time over the last few months. It's actually quite crazy to me that the default actions/setup-go simply does not work well if you want to have more than one golang action running at the same time.

The blogpost has a lot of technical details, but you can also just read the code and try it yourself:

https://github.com/cloudx-io/setup-go

bewuethr

2 hours ago

We recently did similar work for our pipelines, heavily based on Better GitHub Actions caching for Go [0]. Were you aware of that article, or did you (re-)discover things like the cache trimming?

What's the best way to contribute? I'd like to expose the "go-version-file" input for the setup-go action instead of setting a Go version directly. Open an issue in the repo?

Also, have you considered this potential optimization: the module cache doesn't care about architecture, it's just downloaded code. Would it make sense to cache module and build cache separately, and use an architecture-free cache key for the module cache?

[0]: https://danp.net/posts/github-actions-go-cache/

JyB

2 days ago

Have you considered submitting an upstream patch to the widely used actions/setup-go as well?

peterldowns

2 days ago

Good question — we'd be happy to submit a PR, but it's not clear to me that they'd be interested. Some background:

- Our approach writes new cache entries all the time. This can get expensive, and is a pretty big change in behavior from how actions/setup-go works today.

- actions/setup-go can basically be considered incredibly critical infrastructure for the public golang ecosystem. Any change in behavior is probably very risky and slow to happen. At this point I'd bet that we see no change, ever, in behavior.

Additionally there are a few relevant issues/prs that have been ignored for years so I'm not optimistic about contributing upstream. Frankly what we've done is write a very small bit of glue code that is likely most effective as a reference for teams writing their own custom caching actions that fit their exact needs:

- https://github.com/actions/setup-go/pull/426

- https://github.com/actions/setup-go/issues/630

- https://github.com/actions/setup-go/issues/395

- https://github.com/actions/setup-go/issues/596

That said we'd be happy if someone used our code and found it valuable! Lukas put a ton of effort into cleaning up my initial version, added the cache trimming, etc. We depend on this for all of our jobs and use it every day and think it's quite good.

nfirvine

2 days ago

Hello! I'm nfi-hashicorp from https://github.com/actions/setup-go/issues/395 :)

My 2c given that that issue has sat in TODO for several years with no movement is that the maintainers probably aren't going to be pursuing too many big swings like that.

End of the day, implementing an efficient GOCACHE with github's cache primitives is untenable for the general case IMO. It works by shipping around big tarballs, and when your cache actually needs fine grained access to thousands of usually tiny files, only a few of which change, you're likely gonna spend more time on transfer and unpacking than you gain with cache hits.

The other thing is, as mentioned elsewhere, it's really hard to measure cache perf at a granular enough level, and aggregate that across jobs, so I'm willing to bet most people don't (outside of this very good post!) and are going off of vibes or napkin math and don't realize they're wasting time caching garbage.

qmuntal

2 days ago

actions/setup-go maintainer here! Great post, I like the ideas in there. We are always open to improvements, but it's true that low-risk ones are preferred. Feel free to submit your ideas to the GitHub issue tracker. I'll do some due diligence myself.

lukasschwab

2 days ago

I'd echo Peter's #1 recommendation:

> - Allowing actions/setup-go users to specify a cache key prefix so that they can have more than one golang CI job, each with its own cache [...]

I'd actually go further: this may be a sensible default behavior.

"Always update the cache" can get expensive, but it's a neat one; "trim the cache" is definitely necessary if you enable this in a moderately active repository in our experience.

If you want really out-there ideas: rather than storing and loading the full cache monolithically, you could use a GitHub-specific GOCACHEPROG and Go-specific cache service to load only the active items. The pruning problem goes away because accretion is cheap. In theory, parallel jobs could actually share this joint cache. (This may not be a realistic initiative at GitHub.)

If you can raise feedback with your colleagues —

- The docs and settings for Actions Cache limits are really hard to navigate; at some pointed we desperately wanted to pay GitHub more money for more cache, but couldn't figure out why we were capped.

- Bulk-data endpoints for Actions performance would be a boon for optimization projects like this. I wind up either scraping `gh run` (slow) or setting up a GitHub App to collect perf data through webhooks (initially tedious, has to be continuously available).

All this aside — actions/setup-go is a pretty well-considered default and an essential part of writing Go on GitHub; ty for your work maintaining it!

peterldowns

2 days ago

Thanks for reading :) Broken down by key idea:

- Allowing actions/setup-go users to specify a cache key prefix so that they can have more than one golang CI job, each with its own cache: this is 100% worth upstreaming. I believe there are existing requests and PRs about this. Up to you guys to implement however you see fit.

- Allowing "always update the cache": also a good idea to enable as an option, very important for non-open-source teams that are trying to maximize cache hit rate.

- Allowing "trim the cache": if you're going to allow always updating the cache, probably a good idea.

But the "always update" and "trim" cache changes combine to have a lot of risks regarding cache poisoning that might be bad for open source projects. Lukas may have a different opinion or more to say on this front.

idkasam

a day ago

Try other runners that have smart Caching instead.

tonymet

2 days ago

Not OP but upstream patches are rarely worth it. Better to share the fork and if the upstream is interested they can integrate. Integration, testing, meeting upstream expectations usually takes 10x the time, and can be handled faster by those with experience.

vbernat

2 days ago

I am setting cache to `false` and directly use actions/cache:

    - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
      if: ${{ inputs.setup-go == 'true' }}
      with:
        path: |
          ${{ env.gocache }}
          ${{ env.gomodcache }}
        key: ${{ runner.os }}-${{ runner.arch }}-go${{ steps.go-setup.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ env.today }}
        restore-keys: |
          ${{ runner.os }}-${{ runner.arch }}-go${{ steps.go-setup.outputs.go-version }}-${{ hashFiles('go.sum') }}-
          ${{ runner.os }}-${{ runner.arch }}-go${{ steps.go-setup.outputs.go-version }}-
You get one new cache every day, and you can still load the most recent one if you are the first run today.

lukasschwab

2 days ago

There are some good off-cuts that didn't make the official post, but which might be of interest to HN!

It only gets a brief mention, but the cache-pruning change was an interesting one. Cache accretion happens in the default actions/setup-go too, but dramatically increasing the number of cache-writes for cloudx-io/setup-go made it an actual issue.

As the cache grows, so does the time it takes to load it from GitHub's actions cache... and that grows until it's a significant time-suck in CI. We prune with basic mark-and-sweep.

Digging deeper, the pluggable `GOCACHEPROG` (introduced in Go 1.24) is a really useful tool. Shimming the normal cache logic for measurement, for example. In theory this should also be attractive for remote caching.

PhilipRoman

2 days ago

The way caches are managed in CI is bonkers (at least in GitLab, but GitHub probably does the same). I guess it's built to conveniently work with the simplest of projects. I had to reimplement the mechanism by hand just to get basic things working, like not downloading the entire cache when only a few entries will be used for each build

lukasschwab

2 days ago

I also had fun substantiating the claim "86% of actions/setup-go test runs are unnecessary." We had a general sense things were faster, and we measured impacts immediately after we made changes, but hard to understand long-run performance vs. a counterfactual.

The trick was to run back over our git history and calculate, for each commit,

1. The test package Go cache keys at that point

2. The GitHub actions/cache keys constructed by actions/setup-go and cloudx-io/setup-go respectively

Once you have these mappings, you can

1. Pick some arbitrary HEAD commit

2. Model which prior GitHub cache blob would be loaded under each action

3. Compare the test package Go cache keys in that loaded blob against those for HEAD to determine which test packages would run vs. skip

Might write this up in greater depth sometime soon.

tonymet

2 days ago

CI is expensive, and often a blocker for critical releases (e.g. patching a production issue). Every second saved is a relief.

I’ve long wondered why setup-go was so slow and expensive it’s great to see improvements made.

y0ssar1an

2 days ago

how does this compare to WillAbides/setup-go-faster?

https://github.com/WillAbides/setup-go-faster

lukasschwab

2 days ago

They target different parts of actions/setup-go for optimization:

- WillAbides/setup-go-faster speeds up the Go toolchain setup (literally installing Go)

- cloudx-io/setup-go uses the slower actions/setup-go toolchain setup, but changes cache strategy so your `go test` and `go build` steps do less work

Those strategies are compatible. I hadn't heard of setup-go-faster — thanks for putting me on to it.

If you're deciding between one or the other, it'll probably come down to which inefficiency predominates in your codebase (i.e. how many tests you have, how quickly they run, and how much real churn there is in your test package build graph).

user

2 days ago

[deleted]

camdenclark

2 days ago

Separate but if you run CI at any scale you should have an agent working to improve CI constantly and alert you to any regressions / flakiness. It's something that agents can do in the background and open PRs for your team.

colek42

2 days ago

I'm working on something that removes CI completely and lets your agents certify their own tests. It is pretty rough right now but I'd love feedback -- pushgate.dev

lukasschwab

2 days ago

One thing to consider — one of the reasons I want our Actions CI to be fast is that I do a good deal of agentic coding work in parallel (worktrees) on very, very small VMs, but we have some computationally demanding tests and linters. Delegating the expensive validation work to a separate environment — the beefier Actions runner, with run-isolation — and then watching the check status actually tightens the feedback loop for these agents.

Like with all things CI, your mileage will vary according to where you write your code and what the code does.