The delusions of debian

Published on 2022-03-30. Modified on 2022-04-01.

Debian was my favorite Linux distribution for servers where Linux where required to run for many years, and I think I have been running it in production since about 1998 up until the time about when the systemd conflict arose. Since that time I have noticed a rapid decline in many areas of Debian. This has continued up until today and Debian is no longer what Debian used to be.

Update 2022-03-31: To all those who read this as something related to systemd, it's not! I am simply using the conflict about systemd as a reference point in time.

Update 2022-03-31: I made the error of putting the name of "Michael Larabel" into the article as the previous leader of Debian when I linked to the post from Phoronix. As I was typing everything out, I got the two names mixed up.

When the dust of the conflict regarding systemd eventually settled, the Debian community was split in two, where a minority of former members and contributors decided to fork Debian into Devuan. Debian was, and still is, a very big project and the founders of Devuan struggled for years because forking Debian was no easy task. However, they where not the only ones struggling. The internal strive made several leading members simply quit Debian (without joining Devuan), and many people felt "betrayed" by how the systemd conflict was handled.

Despite the split, the Debian project steadily kept adding new third party software to its repositories. As of writing the Debian stable release has 96.729 packages in stable and 149.976 packages in unstable. That is a massive amount of software packages.

One of the primary reasons to deploy Debian on production servers in the past was because of the way Debian maintained both the kernel, the userland, and third party packages in the stable release. On the project website Debian still "boasts" of the same procedure.

On the Debian Security Information it is further stated that:

Debian takes security very seriously. We handle all security problems brought to our attention and ensure that they are corrected within a reasonable timeframe. Many advisories are coordinated with other free software vendors and are published the same day a vulnerability is made public and we also have a Security Audit team that reviews the archive looking for new or unfixed security bugs.

Furthermore, on the Debian Security Audit Project website it is stated that:

The Debian Security Audit Project is a project which is focused upon auditing Debian packages for security issues.

In the short time it has been running it has been responsible for several Debian Security Advisories proving that this auditing process really works to improve Debian security. It is hoped more advisories will result from future work.

By taking a proactive stance in auditing code we can help to ensure that Debian continues its long history of taking security seriously.

The aim of the project is to audit as many of the packages within the Debian stable release as possible for potential flaws. Important packages which are contained in the unstable distribution may also be examined for flaws, decreasing the likelihood of insecure packages entering the stable release in the first place.

And this is where things begin to spin out of control!

In the past, long before the systemd conflicts, Debian was famous in the Linux world for all of the above, and it was one of the most widely deployed Linux distributions on servers. But, as the website also rightly states further down:

Due to the sheer size of the current Debian release it is infeasible for a small team to be able to audit all the packages, so there is a system of prioritizing packages which are more security sensitive.

But that's not all. The fact of the matter is that Debian has long been experiencing a decline in the amount of people willing to participate in the project. This is further made worse by the fact that the current leadership is more worried about Debian having a black lives matter sticker somewhere on the project website than he is with the fact that tons of Debian packages are abandoned and many important packages get security updates long after the bugs have been fixed upstream.

Before that, Jonathan Carter, the former leader had these ridicules points in focus for the future of Debian.

I have been going through a number of packages during the weekend and found that the sad state of web browser support currently within Debian also extends to many other packages.

More than that, the Debian project is absolutely delusional about its long term support!

At the time when Debian 11 was about to be released, PHP 8.0 was 10 months old yet Debian's 11 was released with PHP 7.4. That makes PHP 7.4 the standard in Debian stable for at least 3 years after its release. But PHP 7.4 only got upstream support until 28 Nov 2021 and security support is permanently ended at 28 Nov 2022, which is seven month from now. This means that unless Debian has some really good C developers, no one can provide any security fixes for PHP 7.4. Not only that, no one outside of Debian will be monitoring problems with PHP 7.4 because everyone else will long since have upgraded to PHP 8.

The delusion then continues down the stream where ordinary Debian users are left with the impression that because Debian promises long term support, there is no problem running with these hopelessly outdated packages on their servers.

Maybe the problem is that a lot of myths surround many of the popular Linux distributions. When you go to e.g. Reddit, it is filled with misinformation parroted by ignorant users. But perhaps I shouldn't call it "myths", but rather misinformation because what is stated on some of these project websites, and what goes on in the real world, are often two very different things.

As such, Debian is not the only project that suffers from this. On the Void Linux website it is stated that Void Linux is a "stable rolling release" that:

focuses on stability, rather than on being bleeding-edge. Install once, update routinely and safely. Thanks to our continuous build system, new software is built into binary packages as soon as the changes are pushed to the void-packages repository.

Yet Void is also suffering from a lack of maintainers, just like Debian, and as a result, many third party packages in Void Linux are very outdated. The "rolling release" model then suddenly loses some of its appeal.

Despite the fact that Red Hat is an enterprise Linux distribution, the problems goes even further there where you e.g. still can find a so-called LTS version of PHP 5 that long since should have been permanently terminated.

Update 2022-04-01: Someone asked me about Alpine Linux, whether Alpine is suffering from the same problem. The answer is yes, only not to the same extent as Debian. Alpine Linux splits their packages up in 3 branches, "main", "testing" and "community". On the release page it is stated that, "The main repository is typically supported for 2 years and the community repository is supported until next stable release." This means that Alpine is only promising the 2 year long term support to the packages in "main", whereas the "community" packages only get support until the next release of Alpine. Alpine only has 1.549 package in main, hence that makes it a bit easier for Alpine to keep their promise. However, if upstream support is ended for a specific package in main, and a security bug is found a bit later, then the Alpine maintainers are facing the same problems as in Debian because there is no one to update the package unless the maintainer himself or herself is a skilled programmer.

What I don't understand is why all these Linux distribution projects aren't open and clear about the problems they are facing rather than writing this misinformation on their websites to the users! Most users actually believe that when the website says you're getting long term support, then that means you're getting long term support even if upstream doesn't supply this.

Stop saying that you focus on security. Stop saying that you provide long term support. Tell everyone plain and clear that your so-called long term support is conditional upon what upstream does, and that if a package version is abandoned by upstream, or orphaned by the package maintainer, then all bets are off.

These Linux projects could learn a serious lesson from both OpenBSD and FreeBSD in which all the maintenance problems of both the operating systems themselves and third party packages are out in the open. In OpenBSD, prior to version 6.5, no third party package would receive any kind of bug fix or security update unless you where running with OpenBSD current. Since 6.5, the normal release also gets important bug fixes and security updates, but OpenBSD has always been very open about how that is handled:

The ports collection does not go through the same thorough security audit that is performed on the OpenBSD base system. Although we strive to keep the quality of the packages high, we just do not have enough resources to ensure the same level of robustness and security.

Done. All OpenBSD users know and understand this and as a result, they even help keep an eye on the important packages they install.

On FreeBSD, the package manager informs you if you're installing a package that has been abandoned. It also informs you about important security issues. On the FreeBSD website the procedure is described in detail.

Now, I am not saying that all Linux distributions are suffering from the above problems, far from it! Arch Linux, Artix Linux (they have their own repositories now) and Gentoo Linux are extremely fast in providing security fixes from upstream developers. But they are also neither promising nor providing so-called long term support for packages possibly abandoned by upstream.

If you're a regular Linux user reading this, and security matters to you, don't blindly trust the promises made on the website of your favorite Linux distribution. You need to do some digging! Also, don't simply trust what other users on Reddit (sigh!) and elsewhere is yammering about, that "this is oh so secure", and "project X is more secure that project Y", and bla bla bla. Most of these discussions are worthless, people simply have no idea what they are talking about.

This is the very basic of what you need to do:

So, you thought being a system administrator was just a matter of doing some apt update; apt full-upgrade and then maybe also keep Docker running? No.

Everything is automated today, cloud this and cloud that, but very few people know just how vulnerable everything is behind the scenes, and why being a system administrator is so demanding and difficult. This is also why I care very little for all the hype and buzz, words like "DevOps" and "secure by virtualization" is even a grandeur delusion than the problems described above.

Oh, I almost forgot. This is a list of the people assigned to the Debian LTS support team. 32 people who somehow must provide long term support for 90.000+ packages. Of course many of these packages are not important, but still many others are. If even only 2.000 packages are important to Debian, that means that each of these people need to keep 62 packages updated long term, developing source code patches, for something that has long since lost any kind of support upstream.