Subject: Re: NNCP in Debian/Ubuntu Update

From: John Goerzen <[email-masked]>
Date: Mon, 20 Sep 2021 16:13:33 -0500

On Mon, Sep 20 2021, Sergey Matveev wrote:

> *** John Goerzen [2021-09-20 15:10]:
>>So Debian has a very long-standing policy, going back decades, 
>>of not
>>building dependencies from in-tree.
>
> Hmm... I probably can understand that for dynamically linked
> executables, but that seems strange for statically linked ones,
> that Go does by default. Moreover, Go's go.mod/go.sum are tied
> not to some abstract version of the library, but literally to 
> its
> exact commit/tag. And even in one Go program you can easily use
> different versions of the same library, that will force you to
> install golang-whatever-* of various versions. All of that seems
> very strange to me, but it is Debian's decision, ok :-)

Fundamentally I can't just change that on my own, so our 
conversation here is discussing something that "is what it is", 
regardless of whether we agree with it.  But, hey, I like 
conversation :-)

I'm not as familiar with the Go build, but if I could express this 
in terms of rust:

In Rust, there is a Cargo.toml, which defines the general 
requirements for dependencies (ie, package X, version 0.2.*) and 
then a cargo.lock which records the precise version and hash of 
the dependency used for a canonical build.

There are pros and cons to this...

The pro is, of course, that you can easily get a theoretically 
identical build from anywhere.

And the con is that you may have dozens of different versions of 
the same package in use on a system, making it difficult to patch 
bugs and security issues.

So any person can come down on one side or the other of that, and 
I totally understand that.  Debian as a project has chosen where 
it comes down, and carries the pros and cons with it.  FreeBSD and 
Arch probably come down differently, and live with the pros and 
cons of their choices too.

One thing that flows out of these decisions is the possibility of 
unattended-upgrades existing in Debian, but really not so much in 
FreeBSD ports or Arch.  unattended-upgrades basically can patch 
your system for security bugs automatically from cron.  That's not 
for everyone, but I love it.  It's possible primarily because 
Debian backports security fixes to the version in stable.  That 
is, you get a narrowly-targeted patch that has a low risk of 
introducing issues because it isn't pulling in major new versions 
of software and such.  But the choice here also helps, because if 
the distro follows cargo.lock of the go equivalent, that can 
preclude the possibility of narrowly-targeted patches and also 
expand the work of the security team to unsustainable levels.

Tradeoffs abound.  What's right for Debian isn't right for 
everybody.  Debian prizes stability -- that is, an installed 
system keeps working in the same way for a long time.  The 
tradeoff is slower releases and older software at certain points 
in a year compared to, say, Arch.  On the other hand, compared to 
Arch, a Debian system has fewer times where a breaking change is 
likely to be introduced.  So if you see what Debian's goals are, 
maybe this sort of thing makes some sense respective to them. 
Arch's goals are different and so different choices will flow out 
of that.  The world probably needs both, so I'm not trying to slam 
Arch/BSD here.

Actually - I ran into the versioning issue while packaging nncp. 
The blake3 package requires cpuid, and we had a newer cpuid in 
Debian than what the current blake3 was ready for.  cpuid changed 
its interface a bit, so blake3 wouldn't compile.  I patched blake3 
in the Debian build to get it going (it was a 2-line fix, not 
really worth introducing multiple cpuid versions for), and also 
submitted the issue upstream.  blake3 fixed the issue upstream as 
well, so I dropped our local Debian patch and upgraded to that 
version.  So, you could pull in newer cpuid and blake3 into your 
tarballs, if you like.

> And that frightens me, because if someone want to report a bug, 
> then it
> have to report about dozen of dependencies and their exact 
> versions,
> instead of reporting only of Go version and NNCP's one. Oh 
> well...

Ahh, well we have a tool called reportbug that captures all of 
that with a bug report (or at least enough for a Debian maintainer 
to look it up in the project databases), and sends it to the 
Debian bug-tracking system.  Then it's supposed to be the job of 
the Debian maintainer (me) to sort through that sort of thing 
before submitting it upstream (you).  In this case, if there was 
any question, I would probably ask the submitter to try verifying 
the problem with a tarball-built NNCP before passing a report on 
to you.

We've got the full build log of every package in the distro, for 
every architecture it may build on.  For instance, here's the 
build log for blake3:

https://buildd.debian.org/status/fetch.php?pkg=golang-lukechampine-blake3&arch=all&ver=1.1.6-1&stamp=1632155690&raw=0

And you can see the precise versions of go, cpuid, etc. that it 
built against.  Though again, normally this is the job of the 
Debian maintainer to look at and sort through, not the upstream 
author.

> personally I slowly moving more and more to GNU Stow type of 
> packages
> management, manually building software :-)

Haven't heard of Stow, but I have heard of Gentoo.  I tried it for 
a spell.

>>The uilive thing is a bit of an odd one; it was still in-tree in 
>>the git
>>repo and it survived review so I guess it works for now :-)
>
> As I remember I copied it and modified. It is not original 
> version. Do
> not remember why, but I believe to strip off various complicated
> unneeded stuff.

Makes sense.

- John