jmspring
7 days ago
Pulumi is really a royal piece of shit. Why the f*ck am I writing code to do "deployment". In C# --> new Dictionary<string, object> when dealing with a values.yaml for instance. The whole need to figure out when and when not to use Apply.
Give me Terraform (as much as I hate it) any day.
stackskipton
7 days ago
As SRE dealing with former Pulumi, "Hey Devs can use code to deploy infrastructure" is not great idea you think it is. I've seen some real ugly conditional behavior where I'm like "Is this or is this not going to run? I honestly can't tell."
hinkley
7 days ago
We had so much conflict with the ops team over their choice of Terraform. The three colors of variable thing is just fucking bonkers. Getting tests wrapped around it that actually did what we thought they meant was a giant pain in the ass.
I won't go as far as to say we burned bridges arguing back and forth about it but they were definitely significantly singed.
Config files simply don't work until they do. And if it's your job to stare at them for hours and hours a day then maybe that's okay with you, but if you expect other people to 'just learn' it you're an idiot or an asshole. Or both. Ain't nobody got time for magic incantations.
I also think it should tell you you're on the wrong path when your app is named after a verb and the data it deals with is all declarative.
darkstar_16
7 days ago
Ever thought that "Ops" needs a different mindset than the devs are used to ?
user
6 days ago
hinkley
6 days ago
And that’s why we don’t delegate that work to devs.
salmo
7 days ago
Honestly, the culture/org structure is a way bigger problem in this story than any proper noun tool.
If you’re ignoring guidance and patterns and getting mad reinventing the wheel, that’s on dev. If “ops” mandates tooling and doesn’t have any skin in the game, that’s on them. And both problems are on your leadership.
If y’all just hate each other and don’t listen or participate, then you can’t be successful. It is ironic that this is the pattern that the devops movement landed us in.
slillibri
7 days ago
Honestly curious, I've been writing terraform for a while but I have never heard of "The three colors of variable thing". Could you expand on that?
skywhopper
7 days ago
They mean var vs local vs from-a-resource. There are some places you can’t use some types of variables. It can be annoying but it’s not really a huge problem if you design your approach with that in mind.
The worst part is that the Terraform team at Hashicorp often excuse not fixing these design issues as “safety measures” which isn’t entirely untrue but when over half of your users want something, sometimes you should get over yourself.
For what it’s worth, OpenTofu is fixing many of these sorts of things that cause people pain.
But my advice is to learn to use the tool. Terraform has such great benefits (in the right use cases). If you’re struggling, either you are missing something or you chose the wrong tool for your particular job. Either way, don’t gripe that this specialized tool for infra management doesn’t work exactly like every other general purpose programming language.
slillibri
7 days ago
That makes sense I guess, I just never considered locals or data resources as variables.
878654Tom
7 days ago
Same, locals are in my head like consts. You define it and it stays that way. A shortcut for a repeated value.
Data resources are you requesting a dynamic value of your environment.
Variables are dynamic values that a user can change.
hinkley
6 days ago
That’s only the case if you spend all day rerunning deployments. If your task is more frequently to transition the cluster config from A -> B then the distinction blurs and you go from a 10:1 delta ratio of the different classes of state to maybe 3:2, at which point it feels like splitting hairs.
Especially if the locals vary between prod and pre-prod, and worse if dev sandboxes end up with per-user instances, which for us was mercifully only needed for people working on the TF scripts, so we could run our tests locally.
878654Tom
6 days ago
We have multiple separate environments per application. For environment specific inputs we use variables.
The distinction is very clear in our team. Locals are used as const (like an application name), variables are for more dynamic user/environment inputs and data is to fetch dynamic information from other resources.
Zero problems. If a local becomes more environment specific a quick refactor fixes that. You can also have locals that use variable or data values if necessary.
One big win we also have is that we stopped using modules except for one big main module. We noticed from previous projects that as soon as we implemented modules everything became a big problem. Modules that are version pinned still required a lot of maintenance to upgrade. Modules that weren't version pinned caused more destruction than we planned. Modules outputs and inputs caused a lot of cycle problems,... Modules always seem too deep or too shallow.
hinkley
6 days ago
So would you go opentofu or pulumi or Sir Not Appearing in This Film?
pjmlp
7 days ago
Seconded, as someone that really does developer / operations, depending on the project assignment, I have learned the hard way that infrastructure configuration code should be as declarative as possible.
Sure "use code to deploy infrastructure" sounds great, and that is why we get stuff like Ant, Gradle, Pulumi, Jenkins Groovy scripts, .NET Aspire,.... until someone has to debug spaghetti code on a broken deployment.
AtlasBarfed
7 days ago
On the flip side dsl declarative stuff is obfuscated magic that you can't step through or drive into.
a dsl like SQL involves one basic substrate (data organized in tables) that you can compile in your head. But declarative infra as code involves a thousand different things across a dozen different clouds.
Declarative will hold off spaghetti for... A bit. But it devolves to spaghetti as well (think fine grained acls, or places where order of operations, which the dsl does not specify and is magically resolved, becomes ambiguous).
And if you need to go off the reservation (dsl support doesn't exist or is immature for rapidly evolving platforms, need some custom postprocess steps) then you are... What?
Probably writing code and scripts to autoinvoke on the new node, phone home to a central.... Yup that's code.
Finally, declarative code has an implicit execution loop. But for something like iac that is a very complicated, the execution loop that isn't well documented. And some committed changes to declarative code May trigger a destructive pass followed by a possibly broken constructive phase.
It's a tough problem.
Longwelwind
7 days ago
I would agree with you, if HCL wasn't a bad language in itself:
* You can't make have variables in an import block (for example, to specify a different "id" value for each workspace)
* There is no explicit way to make a resource conditional based on variables. Only a hacky way to do that using "count = foo ? 1 : 0"
* You can't have variables in the backend configuration, making it impossible to store states in different places depending on the environment.
* You can't have variables in the "ignore_changes" field of a resource, making it impossible to dynamically ignore changes for a field (for example, based on module variables).
* The VSCode extension for HCL is slow and buggy. Using TS with pulumi or TFCDK makes it possible to use all the existing tooling of the language.
breendreams
7 days ago
For Terraform, most of the issues with conditionals can be resolved by creating dictionaries dynamically and looping through it to generate resources.
You get the bonus of controlling the resource id and being able to selectively delete resources without worrying about ordering.
cyberpunk
6 days ago
This massively depends on your provider code. Using loops to manage tf stuff can you you into really “fun” scenarios when you want to e.g delete an openstack firewall rule from the middle of the array.
I’ve been burned so many times here that I hate all of this stuff with an extreme passion.
Crossplane seems to be a genuinely better way out but there are big gotchas there also like resources that can simply never be deleted
Hawxy
7 days ago
As much as I like it, I find C# to be too inflexible of a language for infrastructure code. I tried with Pulumi for a while but moved to TypeScript as it works so much better. Structural typing makes your life a lot easier.
MrLeap
7 days ago
I bounce back and forth between javascript and C# depending on the nature of the job at hand. I'm curious what things you'd like to do with C# that you can't?
I find that with some handwringing, C# can be forced to do almost anything. between extension methods, dispatch proxies and reflection you can pummel it into basically any shape.
Having to write a little boilerplate to make it happen can be a drag though. I do sometimes wish C# had something from a blank project that let me operate with as much reckless abandon as Object.assign does in js land.
Hawxy
7 days ago
It's not the fault of the language, it's just the nature of infrastructure code that's been ported from terraform. With Pulumi C# you end up with multiple nested objects/dictionaries with a load of `new` object calls that just add noise to your codebase. There's also some pain points with some types being Input<T> which IDEs try to autocomplete when in reality you need to call `new T()`. Typescript permits structural typing that _feels_ a lot better to write and read within this context.
I use C# extensively for most other things I do, but this the one area where I prefer not to use it.
cruffle_duffle
7 days ago
> Give me Terraform (as much as I hate it) any day
Terraform sure is a quirky little DSL ain’t it? It’s so weirdly verbose.
But at the same time I can create some azure function app, setup my GitHub build pipeline, get auth0 happy and in theory hook up parts of stripe all in one system. All those random diverse API’s plumbed together and somehow it manages to work.
But boy howdy is that language weird.
rwiggins
7 days ago
I haven't used Terraform in years (because I changed jobs, not because of the tech itself), but back in the day v0.12 solved most of my gripes. I have always wished they'd implement a better "if" syntax for blocks, because the language itself pseudo-supports it: https://github.com/hashicorp/terraform/issues/21512
But yeah, at $previous_job, Terraform enabled some really fantastic cross-SaaS integrations. Stuff like standing up a whole stack on AWS and creating a statuspage.io page and configuring Pingdom all at once. Perfect for customers who wanted their own instance of an application in an isolated fashion.
We also built an auto-approver for Terraform plans based on fingerprinting "known-good" (safe to execute) plans, but that's a story for a different day.
raffraffraff
7 days ago
I get around most of the if stuff using "for each" to iterate over a map. That map might be config (usually from the hiera data provider) or the output of another deployment. It's not generally a very flexible "if" that you need most of the time, it's more like "if this thing exists then create an X for it", or "while crafting X turn this doohickey on of that data set has this flap", which can be accomplished my munging together days with a locals var for loop (which support if statements).
Honestly, I only use terraform with hiera now, so I pretty much only write generic and reusable "wrapper" modules that accept a single block of data from Hiera via var.config. I can use this to wrap any 3rd party module, and even wrote a simple script to wrap any module by pointing at its git project.
That probably scares the shit out of folks who do the right thing, and use a bunch of vars with types and defaults. But it's so extremely flexible and it neutered all of the usual complexity and hassle I had writing terraform. I have single handedly deployed an entire infrastructure via terraform like this, from DNS domains up through networking, k8s clusters, helm charts and monitoring stack (and a heap of other AWS services like API Gateway, SQS, SES etc). The beauty of removing all of the data out to Hiera is that I can deploy new infra to a new region in about an 2 hours, or deploy a new environment to an existing region in about 10 minutes. All of that time is just waiting for AWS to spin things up. All I have to do in code is literally "cp -a eu-west-1/production eu-west-2/production" and then let all of the "stacks" under that directory tree deploy. Zero code changes, zero name clashes, one man band.
The hardest part is sticking rigidly to naming conventions and choosing good ones. That might seem hard because cloud resources can have different naming rules or uniqueness requirements. But when you build all of your names from a small collection of hiera vars like "%{product}-%{env}-%{region}-uploads", you end up with something truly reusable across any region, environment and product.
I'm pretty sure there's no chance I'd be able to do this with Pulumi.
stackskipton
7 days ago
Tip for naming, create a naming module where you pass in stuff like product, environment, region, service, have a bunch of locals for each thing like S3 bucket, RDS, EC2, EKS whatever you use then make them all outputs.
So at top of your IaC, you have module naming {variables as inputs} then all other resources are aws_s3 { name = module.naming.s3bucket }
grncdr
7 days ago
In pulumi
regions = [
“eu-west-1”,
+ “eu-west-2”,
]
for region in regions:
…raffraffraff
4 days ago
Of course Pulumi can do for loops, you're using a proper programming language.
I meant that I doubt that I could 'cp -a' on a whole deployment tree, and deploy the copy successfully without having to make any code changes.
Although thinking about it, I take it back. It may be possible with Pulumi with the right code structure and naming conventions, and if configuration were separated entirely from the codebase, and if variables were inferred from the directory structure. That is really the thing that allows me do to it.
grncdr
3 days ago
Yes, sorry for the rather pithy response, but separating out the "what changes" vs. "what doesn't" (config vs. code in your terms) is what makes these things possible.
As you also noted, doing this in plain terraform is kind of a pain, so using a tool like Hiera allows you to skip a lot of the work involved in doing it the "right" way. IMO if you're starting greenfield Pulumi (or CDK, anything that lets you use a "real" programming language) allows you to write (or consume!) that config in basically any form, instead of needing to funnel everything through a Terraform data provider.
paulgb
7 days ago
Yeah. I guess maybe terraform makes sense if the people writing it spend enough of their time writing HCL to master it, but I ported our terraform config to Pulumi a few years ago and never looked back. It meant I could spend way less time googling for the HCL way to do something (say, templated resource) and just use the JS primitives I already know.
hinkley
7 days ago
>spend enough of their time writing HCL to master it
Making Terraform changes every six weeks was enough time that we forgot everything and had to refresh our memories. Every time it felt like going into the water in a northern beach and forgetting how goddamned cold the water was, then reproaching yourself for forgetting.
postalrat
7 days ago
Why are people templating yaml for terraform like they templated html in php in 1996?
benatkin
7 days ago
Because it works fine, and is also used in for other things like Helm Charts?
https://helm.sh/docs/chart_template_guide/control_structures...
jnsaff2
7 days ago
Helm charts are a horrible example of text based templating.
You have YAML/JSON that k8s API wants, that is fed through helm which is fed through helmsman or whatever newer thing. There might be a layer or two of other templating around. Sometimes companies have built systems so developers/devops don't even have the ability to see what the final compiled version of the template is which is like the mother of all: "works on my laptop" problems.
It's super easy to break text based templating because of some space, tab, string escaping or whatever.
YAML makes it worse as there are lots of gotchas and different ways of doing. JSON, being quite verbose and inflexible at least has strong structure right in your face so it's a bit easier to figure out what went wrong.
With a proper programming language data structure you can be much better with verifying that the things you add or remove or iterate over will produce a valid result, much better refactoring and working as a team independently.
arkh
7 days ago
> Helm charts are a horrible example of text based templating.
Every time I see " | nindent whatever" I'm asking why the fuck the tool cannot manage indentation.
kevincox
7 days ago
And it breaks every time a variable gets a `:` inside of it and now you are producing invalid yaml everywhere you forgot to call `| toYaml`.
adhamsalama
5 days ago
I once got a nil pointer exception when I updated a helm chart. I wondered why the hell am I getting a nil pointer exception for updating a YAML file. After some investigation I found an issue on GitHub where the maintainers said the Go team says this is an intended behavior for some case in Go templates.
Wasn't fun.
benatkin
5 days ago
That isn't a typical nill/null exception, like in JavaScript, ruby, and python. That's in a language where a lot of values are non-nullable, and some of the ones that are have zero-values that can be used without getting a nil pointer exception. https://go.dev/tour/moretypes/12
So, there's a good chance was an error that was really unexpected and it's better to show the error than to risk producing bad output.
smw
6 days ago
Never read anything more true in my life!
jq-r
6 days ago
I’m not sure why nobody invented a way to dynamically update values.yaml based on what are writing in the template file. And maybe vice-versa. It would be such a time saver. Maybe someone did, but I didn’t find it yet.
dainiusse
7 days ago
This
arkh
7 days ago
Tried Pulumi thinking "it's gonna abstract all the k8s specifics". Welp no, still need to know and understand K8s so I still don't see the value from those kind of tools. In which case why not use something like Pkl to generate my yaml from some sensible code-like structures?
katdork
7 days ago
kubernetes is very complex and therefore any abstraction which completely glosses over the way the underlying systems work would make it very hard to avoid leaking or a bad abstraction to begin with.
the complexity in one way or another must be preserved within the abstraction (in all likelihood) or you will have cases you cannot create in that layer or breakages which now have the total complexity of both the abstraction itself AND kubernetes itself required to fix.
i would not say IaC is going to provide you a magic solution to learning k8s, although the value in using IaC (e.g. Argo CD / Flux CD + Kustomize + ...) in K8s land is that you are no longer imperatively managing your cluster resources and therefore can keep them within a repository, managed like code. the point of the solution is not to make it easier for newcomers, but to make it easier to have teams manage and work together on an established cluster for deployments, ...
in the case of Pulumi, you leverage the single language with typechecking instead of relying upon K8s flavoured YAML, which is itself beneficial in many ways (since you can use your regular developer tooling)
wrt pkl, pretending K8s manifest structure underneath does not help because you will need to know how the keys within a manifest interact with the underlying system regardless, especially to understand functionality, e.g. node selectors, taints and tolerations, node affinity, ...
i prior managed a terraform-based deployment of several k8s clusters and it still required knowledge of those keys and values, alongside knowledge of the underlying resource types.
without those you can't implement things like GPU-based node selection for jobs which require a GPU, ...
rusty-jules
6 days ago
What about pulumi's declarative yaml interface which can be exported from type-safe languages like cue? https://www.pulumi.com/blog/extending-pulumi-languages-with-...
nuker
6 days ago
> Give me Terraform (as much as I hate it) any day.
Just use CloudFormation. Easy to write, declarative, vars (Parameters and Output exports). Trick is not to pile everything in one Stack. Use several.
notyourwork
6 days ago
CDK is much better to express this. Why cfn?
nuker
6 days ago
Less lines, easier to read, declarative (cdk is interactive, less predictable).
And it generates shitty CFN, we can do better ourselves :)
notyourwork
5 days ago
How is cdk interactive? I use cdk and have it auto build and deploy.
nuker
5 days ago
It is "imperative", not interactive, sorry. From Wiki:
"There are generally two approaches to IaC: declarative (functional) vs. imperative (procedural). The difference between the declarative and the imperative approach is essentially 'what' versus 'how'."
https://en.wikipedia.org/wiki/Infrastructure_as_code#Types_o...
klysm
7 days ago
Apply is really straightforward. The dictionary stuff is very annoying overhead but it’s nice keeping everything in one language.