stackskipton
5 hours ago
SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets.
Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance.
Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load.
Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML.
Getting secrets from Kubernetes Secrets API I've seen done but lock in again. I'd strongly not recommend this approach unless you are designing a Kubernetes operator or other such type system.
I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
jppittma
5 hours ago
How is the kubernetes secret API lock in? Genuinely wondering - were you trying to use that deployment yaml for something other than a kubernetes deployment? For most applications, you should be mounting the secret on your application, then you can inject it as either an environment variable or a json file that your application reads in an environment agnostic way.
Then, on the backend, you can configure etcd to use whatever KMS provider you like for encryption.
stackskipton
5 hours ago
Because you can't run the container, even for development outside Kubernetes.
Yes, you can mount Secrets as Volumes or Env Var in Kubernetes which is fine but I'm not talking about "How you get env var/secret" but "Methods of dealing with config."
jppittma
4 hours ago
Yes you can? The container should be completely agnostic to the fact that it's running in kubernetes. You can do config the same way. Configmaps are mounted as regular files and environment variables. The application doesn't care if the configmap came from the cluster resource or a file your created on your dev machine with dev credentials. You can mount local files into the container yourself. It's docker run -v "source:destination" I think.
joshribakoff
2 hours ago
One of you is talking about mapping a secret to an environment variable and the other one of you is talking about having the work load make an API call to retrieve the secret. You’re not even talking about the same thing.
stackskipton
4 hours ago
sigh I’m extremely competent Ops type and I know. If you mount secrets as Volume or Env Var, that’s Config file or Env var from Application PoV. We are looking at this from Application PoV.
I’ve seen Applications that do direct calls to Kubernetes API and retrieve the secret from it. So they have custom role with bindings and service account and Kubernetes client libraries.
jppittma
3 hours ago
If you're not developing k8s operators, you're calling the api server directly, then complaining about lock in, then that's a skill issue. If you're developing k8s operators, then you should use a tool like kind for integration tests and dependency injection for other stuff and the concept of lock in doesn't make sense. You can also deploy your helm chart directly to kind.
cassianoleal
4 hours ago
Don’t use live system secrets and credentials when running your application locally. Then you don’t need to access the same secrets.
Keep it simple and design your applications so they’re agnostic to that fact.
It’s really not that hard, I’ve been doing this for at least 6 or 7 years. A little bit of good engineering goes a long way!
Nilocshot
5 hours ago
This is where I like things like Tilt. If you're deploying to a k8s cluster, it's probably a good idea to do local dev in as close to a similar environment as possible.
Bit more of an initial hurdle than "just run the docker image"; however.
stackskipton
4 hours ago
I've look at Tilt and it's another abstraction for Kubernetes which rarely ends well at scale.
However, most of time, Devs don't need to develop on Kubernetes since it's just Container Runtime and Networking Layer they don't care about. They run container, they connect to HTTP endpoint to talk to other containers, they are happy. Details are left to us Ops people.
nunez
an hour ago
+1 to all of this.
This is why I continue to use env vars and dotenv for configuration. They are extremely simple, work well, and are compatible with secrets managers and other secrets tooling.
Though lately I've been veering into sOps the last few years. YAML is just so nice for expressing how an app should be configured, and sops makes encrypting chunks of it so easy. Dealing with GPG keys can be challenging though, which Vault/OpenBao solve, but then lock-in becomes an issue (though less so with OpenBao).
1718627440
5 hours ago
You can have a command setting that is invoked to get the string. This way you don't have vendor login, but also don't need a separate template step.
stackskipton
5 hours ago
You still need to present that to Application.
So Command Line leaks worse than Env Var.
Config file, see original post for problems.
Env Var, see blog for problems.
sgarland
4 hours ago
I think parent is referring to something like SOPS [0], which can pass secrets via FIFO. That way, there’s nothing on disk, the pipe is cleared after first read, and /proc/cmdline doesn’t reveal anything.
1718627440
4 hours ago
No I meant a property in the application config.
For example mbsync/isync does this.
stackskipton
4 hours ago
You have a config file, it needs to have secrets so likely you are going to run some templating system where you replace dbpassword: ${dbPassword} with password from some secret system. Hopefuly you understand possible issues with any templating system that could result in replacement failures.
String manipulating is one of those "This is easy" until it's not.
1718627440
4 hours ago
Apparently I'm still unclear.
I don't mean to hardcode secrets into the config file either. I was suggesting to put a command into the configuration file that the application then calls to get the secret. This way the secret is only ever passed over a file descriptor between two processes.
sureglymop
4 hours ago
I do something similar. I usually have a flag, something like --password-file. It can only be used to specify a file containing the secret and at startup the application reads it.
1718627440
3 hours ago
Yes, this is also possible, but which the approach I stated, the secret can be generated by another program or received from the network, it isn't just limited to a file.
stackskipton
3 hours ago
My initial Ops gut feeling says "This is as lock in and error prone as Application Vault Libraries" but if Dev wanted to propose this, I'd be willing to see it in real operation.
1718627440
3 hours ago
It isn't lock in, because all the application depends on is that it gets a string it can pass to exec/the shell and then reads all data from stdout until EOF as the secret.
stackskipton
2 hours ago
Sure, but let's say you have 5 secrets to get or maybe the new vault CLI does not support just stdout printing but prints JSON only.
I still think this is worse than config file/Env Vars.
1718627440
2 hours ago
You can always pass echo "SECRET" as the command, so it is a strict superset of the config file. Also programs that tend to provide a command option also tend to provide a simple string option.