Grudge? You mean when Jessie Frazelle walked around dockercon with a badge saying, “I will not merge patches for systemd” just out of principal? In hindsight it is fairly obvious picking a fight at redhat summit was a very poor long term strategy from docker. It forced them to ultimately sell the orchestration parts of their business and massively limit the free tier of docker hub.
What was described could be the basis of a grudge, or it could be that Red Hat were convinced that the current project would not accept changes Red Hat would benefit by even if it helped many other people. At that point, and at the size of Red Hat and how important this technology is becoming for them, it only makes sense for them to secure some of the technological foundations they rely on. It costs them very little in resources in relation to the business they are basing on that technology.
I don't think it's a stretch to look at the events as described and see them as Red Hat doing what they need to work around someone else's grudge. It could be any number of things, but I don't think the information presented so far lends itself towards Red Hat having a grudge.
runc, containerd and the docker client are critical infrastructure to a whole lot of people, and they seem to be doing just fine running it and, when needed, contributing back. Only Red Hat seems to have a problem with using those projects without forking or re-writing them. Could it be because Red Hat considers it of strategic importance for their business, not just for those projects to be reliable (they are), but to be controlled by them to a sufficient degree that they can act as gatekeepers to upstream, thus preserving their business model?
That could be, but if Frazelle did actually walk around with a button saying “I will not merge patches for systemd” I think it's simpler and safer to assume Red Hat decided that relying on Docker was risky given how important it was to their future products and the apparent animus on display from the Docker maintainer.
If I carpool with someone and they like to go around and loudly proclaim "I refuse to listen to anything kbenson has to say", maybe I find someone else to carpool with or drive myself? That only seems prudent. That doesn't mean I have a grudge against that person, it just means I don't want to trust them with getting me to and from work anymore.
Sorry, it was poorly worded on my part. I didn't actually doubt that, but prefer not to state other people's assertions as fact unless I know it is so for myself. I prefer to have my assertions grounded by what I'm basing them on, which is less about doubt of the source and more about keeping myself honest when arguing and not taking too hard a stance on stuff I haven't verified myself, since it's easy to come across as authoritative when I don't intend to otherwise.
Thanks for the evidence though, it's always best to have, even if it can be a pain to drum up when it might not really be called into question anyways. :)
It’s no secret that there were tensions betwen Docker and Red Hat that culminated around 2016, when Red Hat’s desire to capitalize on the success of containers ran into Docker’s own ambitions to compete with Red Hat. Those tensions were eventually resolved (or at least diminished) by Red Hat shifting gears to Kubernetes as their “next Linux”.
The drama you’re talking about is from that period - 2015-16. But “crun” was launched in 2020, a full four years later. Frazelle left Docker years ago, as well as most of the people involved in those feuds. Runc, containerd and docker are chugging along drama-free, and are maintained by multi-vendor teams. There is no interest from anyone outside of Red Hat in forking or re-writing those tools. It’s just a tremendous waste of energy and in the end will be problematic for Red Hat, because they will be building their platform on less used and therefore less reliable code.
The world’s container infrastructure runs on containerd, runc and docker. Not crio, crun and podman. Someone at Red Hat clearly is having trouble accepting that reality, in my opinion because they’re having trouble moving on from old grudges. I really wish they did, because all of that wasted engineering effort could be deployed to solve more pressing problems.
> The world’s container infrastructure runs on containerd, runc and docker
I don't consider alternate implementations of widely used software to be wasted evergy almost ever. Was Clang a waste of energy because everyone was using GCC? There are many reasons why multiple implementations may be useful, competition and being able to cater to slightly different common use cases obvious ones.
I'm not sure why any end user would wish for there not to be multiple open source projects looking to satisfy a similar technical need.
> in the end will be problematic for Red Hat
That may be, but I don't really care about whether it's good for Red Hat. I care that it increased user choice and maybe at some point features and capabilities offered to users (whether through podman or pressure on docker).
That’s fair, there’s always a benefit to alternate implementations. I think those were started for the wrong reasons (interpersonal conflict rather than technical requirements) but perhaps in the end it doesn’t matter.
runc and containerd wouldn't exist if Redhat hadn't created the OCI specification and work around docker to make things interoperable. Docker, Inc was not playing ball and then redhat had to work around them. It wasn't a grudge it was just them keeping their business safe. The Open Containers Initiative was essentially designed to force Docker Inc to try to play better with others. They donated containerd to the OCI due to the reality that docker was about to be left in the dust.
For reference, this was the shirt that Docker Inc was giving away at Redhat summit years ago. I was there. I saw the shirt, and thought it was in very poor form. https://twitter.com/SEJeff/status/1125871424126767104
As Rambo said, "But Colonel, they drew first blood."
Your history is wrong. Docker created the OCI, not Red Hat. Docker donated both the spec and the runc implementation.
Containerd was donated (also by Docker) to CNCF, a different organization, and I believe a few years later.
You are correct that Docker did those things because of pressure to be more interoperable.
Reading your linked tweet (“don’t mess with an engineering behemoth”) you seem to agree that Red Hat is rewriting away Docker code because of a past grudge?
Docker and Redhat were both founding members of OCI. CoreOS probably was responsible for a lot of the initial conversations. They wanted a standardized container runtime after finding some limitations in docker which caused them to create rkt. Docker inc wasn’t super into changing anything and as history as shown, wasn’t terribly into working well with others. The industry was going to move forward without them but they were offered a spot on the OCI if they played ball.
OCI would have made docker inc entirely irrelevant if they didn’t join it. They donated runc (I was wrong originally, dyslexia sucks sometimes) as the reference implementation so docker continued to stay at the forefront.
They donated a very minimal implementation of containerd to the CNCF when kubernetes wanted to do things with the container runtime interface (CRI) to make it more pluggable. As a more purpose built CRI, crio is better suited for k8s.
Docker did not create OCI any more than Redhat did. Redhat and a collection of other orgs did. Docker and Redhat were besties after Alexander Larson added device mapper support to docker. This is what allowed docker to run on any Linux distro that didn’t include the out of tree aufs. Look it up. Redhat then made a lot of money doing container multihost orchestration with openshift v3, which deprecated their own tool, geard. When Docker inc accepted a ton of VC money, they realized they actually had to monetize things. This is the part where the relationship started to seriously sour. The company that helped make them so popular was now suddenly a huge competitor. This is what lead to the problem.
It wasn’t a grudge from Redhat I mentioned in the linked tweet. It was docker inc thinking docker was anything other than commodity. It just is some very nice UX and cli tooling with a container registry around Linux namespace and Linux control groups. They just put it all together very nicely for developers to ship code faster. Podman is also a commodity. At its core, it just is some json parsing and pretty wrapping around Linux namespace and cgroups. Redhat knows this, and they don’t hide it. They monetize openshift and quay, not podman.
Sorry if this is a bit scrambled. Long posts on mobile are difficult.
I understand what you’re saying but it is incorrect. Docker was in fact the originator of OCI in every way that matters. They made the decision to create it; drafted the founding documents and went back and forth with Linux Foundation on the content; chose the names (initially OCF, then later OCI); negotiated with LF the governance structure and initial board composition; negotiated the list of external maintainers with commit access, including a Red Hat and CoreOS employee; chose the announcement date and venue; drafted the announcement content along with LF; coordinated with PR teams of other founding members. All this in addition to being the original authors of both the spec and implementation.
Neither Red Hat nor CoreOS were involved in any of those steps, nor were they even aware of them until the last minute. When they were invited it was on a “take it or leave it” basis. They took it, then tried to pre-empt the Docker/LF announcement by a few hours to make it look like they were launching it. Those were tense days and there was very little trust between those companies.
Source: I was involved in the process of creating OCI.
Well thanks for your input, and I stand corrected. I liked Solomon and he always seemed like a super nice guy when I met him, but it did seem that there was a bit of a crisis when Docker Inc realized, "Oh shit, we have to make money for these VCs!" and then it got really awkward. I was sad to see it happen as Dan Walsh and Solomon are both really good peeps to me.
Yeah the tensions were really unfortunate. In the long run it resulted in more competition which benefits all of us; but in the short term it made people’s lives more stressful than they needed to be.
I actually have a slightly different explanation of what set it off. I don’t buy the “docker is suddenly under pressure to make money” explanation. Docker was a VC funded startup since 2010. They already had a business when they pivoted to Docker. As far as I know they kept the same investors and board. So it seems unlikely that they suddenly remembered that they needed to make money. More likely they planned to make Docker as ubiquitous a platform as they could, at which point there are many known avenues to monetize. Also, the CEO they brought on board, Ben Golub, had just spent a year at Red Hat after selling his previous startup to them. So he was certainly familiar with Red Hat’s business and strategy, and probably on friendly terms with their leadership (this is speculation on my part).
And in the early days Red Hat and Docker were in fact partners. Docker engineers even implemented features specifically to please Red Hat; for example devicemapper support and storage drivers, which were developed by Dockerfor Red Hat.
The problem is that the partnership was built on a misunderstanding. Docker was hoping for a distribution deal between the two businesses, with revenue share etc. and intended to keep control of their open-source project. Whereas Red Hat was hoping to make Docker their “new Linux” which involved no partnership with Docker and instead taking or at least sharing control of the project as they did with Linux. Each side was slow to realize the true intentions of the other, whether by naivete or deception I don’t know. This still surprises me because both side’s intentions were rational and entirely predictable. I think a lot of wishful thinking, and possibly some individual incompetence in leadership were involved. For example Red Hat has never actually partnered with a smaller startup and shared revenue with them in the way Docker was hoping, so I’m not sure what made them so confident it would happen. In any case, when the fog lifted, bitterness and conflict quickly followed.
In the end Red Hat shifted gears to Kubernetes as their “new Linux” which was a much better match for them. Then proceeded to rewrite history to minimize Docker’s role; which is unfortunate but understandable, I guess. I know a lot of Docker employees are personally hurt by it, to this day. It doesn’t feel good to see your work swept under the rug for reasons of competition and ego.
Alexander Larsson did the lions share of the work on device mapper, and here are 29 PRs to show it (including the very first with the devicemapper label):
In fact, Solomon's own words from https://github.com/moby/moby/pull/2609 "Integrate lvm/devicemapper implementation by @alexlarsson into a driver", so unless you're more knowledgeable about the history than Solomon, I'm going to have to discount your take on this entire thing.
Regardless, I think things are at a much better place now. It is unfortunate that there were tensions and more so that Solomon had such a tough time through it all. He's a super nice guy. But this is tech, and there will always be lots of personality conflicts. It is one of the precious few constants.
It is absolutely not correct that Larsson did the lion’s share. What he did was implement a Go wrapper for libdevmapper, which exposes a very low-level API. It is the Docker team that implemented devmapper-based container storage, as well as the whole storage plugin system which was now required to support more than one storage method. The original devmapper lib is utterly undocumented and Larsson’s wrapper did not fix that. So getting that feature to work was an all-consuming task and it is the Docker team that did the bulk of it.
