This was a really interesting article. One thing I’d like to hear more about would be alternatives to something like Bitnami’s SealedSecrets.
The articles mentions that they don’t offer a significant security benefit, but I’d humbly disagree. They’re not without their downsides, specifically around developer experience and onboarding—not that it’s “bad” when you use them often enough, but for devs who don’t use it often i have faced onboarding challenges.
From a security perspective though, if policies are created that give humans less ability to access the plaintext secrets, eg. giving them the ability to create or update sealed secrets but now view or decrypt, isn’t that eliminating, or at least minimizing the least secure aspect of software (humans)?
The other benefit, though not necessarily a security benefit, of SealedSecrets is being able to commit your secrets to your VCS, allowing you to deploy your application programmatically rather than have someone manually deploying the secrets.
I’m far from a security expert, but to me, being able to minimize the amount of eyes on sensitive materials feels like a net positive.
That said, though I’ve overall been pretty happy with it, it’s not without its downsides, and I’d love to get more perspective from people handling this situation in different ways.
Yeah documentation is hard and I'm guilty (as a former maintainer of SealedSecrets)
SealedSecrets was designed with "write only" secrets in mind.
Turns out a lot of people need to access the current secrets because they need to update a part of a "composite" secret.
There are two kinds of "composite" secrets, one easy and one harder, but if you don't know how to do it, even the easier is hard:
1. Secret with multiple data "items" (also called keys in K8s Secret jargon but that's confusing when there is encryption involved). I.e. good old "data":{"foo": "....", "bar": "..."}
2. Secrets where data within one item is actually a config file with cleartext and secrets mixed up in one single string (usually some JSON or YAML or TOML)
Case 1 is "easy" to deal with once you realize that sealed secrets files are just text files and you can just manually merge and update encryoted data items. We even created a "merge" and some "raw" encryption APIs to make that process a little less "copy pasta" but it's still hard to have a good UX that works for everyone.
Case 2 is harder. We did implement a data templating feature that allows you to generate a config file via a go-template that keeps the cleartext parts in clear and uses templating directives to inject the secret parts where you want (referencing the encrypted the items)
The main problem with case 2 is that it's undocumented.
I noticed that people at my current $dayjob used sealed secrets for years and it took me a while to understand that the reason they hated it was that they didn't know about that fundamental feature.
And how to blame them!? It's still undocumented!
In my defense I spent so much effort before and after I left VMware to lobby so that the project got the necessary staffing so it wouldn't die of bitrot that I didn't have much time left to work on documentation. Which is a bit said and probably just an excuse :-)
That said, I'm happy that the project is alive and the current maintainers are taking care of it against the forces of entropy. Perhaps some doc work would be useful too. Unfortunately I don't have time for now.
First and foremost, I really appreciate the project, so thank you for your help with it!
Documentation is honestly probably one of, if not the most difficult part of development. Even if the programming problem you’re solving is immensely difficult, how do you then simplify that in such a way that it is accessible to a wide range of backgrounds? I’ve heard that writing code is hard, but reading code is harder—and documentation is essentially writing about reading code, so I wouldn’t beat yourself up about it.
The merge feature is immensely helpful, and I’ve found that writing internally focused documentation really does help with adoption. As in, “here at XYZ we are using this tool in these cases—here is how and when you would use each of them. Here’s a link to more contextual documentation”. Even if people don’t actively seek out the documentation, it’s still a hell of a time saver to point the tenth person asking how or why to do something to a static copy of what would essentially be the response.
Documenting some code you wrote is hard because you have to forget you everything about it (all the thought processes that led you to the point of writing that piece of code, with that design, etc) and put yourself in the shoes of an arbitrary end user. To make things more complicated there are many different users coming from different backgrounds and having different sets of thins that are obvious or non-obvious to them
Unfortunately, as the author alludes to later on in the article, 'encryption at rest' is usually required by compliance standards, e.g., PCI-DSS, SOC2, etc. Just about every environment I've seen either turns on etcd encryption as a box-ticking exercise or simply forbids Kubernetes secrets.
Wait, so the argument here is that securing secrets is somehow not worth it? That’s just silly. And that’s the most kind word I can think of for it. It might actually be too kind. Stupid might be more correct. Or just wrong.
Step through it as an attacker. There are half a dozen attack vectors where securing your secrets makes sense. Even storing them in plaintext in a file on a server is more secure than k8s “secrets”.
The articles mentions that they don’t offer a significant security benefit, but I’d humbly disagree. They’re not without their downsides, specifically around developer experience and onboarding—not that it’s “bad” when you use them often enough, but for devs who don’t use it often i have faced onboarding challenges.
From a security perspective though, if policies are created that give humans less ability to access the plaintext secrets, eg. giving them the ability to create or update sealed secrets but now view or decrypt, isn’t that eliminating, or at least minimizing the least secure aspect of software (humans)?
The other benefit, though not necessarily a security benefit, of SealedSecrets is being able to commit your secrets to your VCS, allowing you to deploy your application programmatically rather than have someone manually deploying the secrets.
I’m far from a security expert, but to me, being able to minimize the amount of eyes on sensitive materials feels like a net positive.
That said, though I’ve overall been pretty happy with it, it’s not without its downsides, and I’d love to get more perspective from people handling this situation in different ways.