Dependencies, their tracking and what we need to improve.

[Auke Kok]

On 07/17/2011 03:14 AM, Duncan Gibson wrote:
> Auke wrote:
>> lunar's codebase is already a huge patch blanket, and always was. At
>> one point, you just have to decide "no more" and either just ignore
>> the architectural deficiencies, or redesign and rewrite the entire
>> thing to fix those shortcomings.
>> ...
>> It sounds rather harsh, I can understand that. But given the amount
>> of core code I wrote for lunar over the years, I think I can safely
>> say that I've learned enough from it to realize that it would be more
>> efficient to design something from scratch.
>
> Based on your experience then, and with the benefit of hindsight, what
> are the pros and cons of the current Lunar package management system?
>
> - Is the basic information/structure of the module control files,
>    DETAILS, DEPENDS, etc. sufficient?

no - the biggest gap is that "BUILD" needs to be split up in "BUILD" and 
"INSTALL" scripts. That would enable things like fakeroot, installwatch 
much more easy. It would also enable building-as-non-root.

DEPENDS/CONFLICTS are inherently the same thing, so they should be in 
one file, actually, why are we doing separate files again? None of this 
should be in separate files to begin with IMO

> - Are there things missing?

definately. subpackages for one.

> - Is it a strength or a weakness that these are written in bash and
>    can interact with the rest of the package management system?

stuff that "does something" can be bash/sh, and that should remain, 
there's no reason to stop doing that.

you'll need bash/sh to do post_install stuff anyway. But, under the 
hood, there should not be any shell code at all.

> - Is bash the Achilles' Heel of the whole system?

no.

but it will never be performant, or secure.

> If you were to build a new package management system from scratch for
> the new Sofaris :-) distribution, how would you do it differently?

no lame names, like "sofaris", for one ;)

>> From my own perspective, there are a few major problems that keep on
> biting me. The first is the same one that Dennis has already raised:
> there are too many packages with "clever" configuration systems that
> you can't turn off via command line options,

(*)

> and as a result you can
> have issues with implicit options and circular dependencies. The next
> is the old chestnut that deleted dependencies in zlocal versions are
> not handled properly. And the third is selecting the correct cpu and
> optimisation flags for gcc-4.5 as this has consequences further down
> the line.
>
> And after my recent foray into trying to build my own ISO, the fact
> that installwatch does not catch some of the more modern system calls
> for file creation makes me wonder whether we can trust it at the heart
> of any package management system.

I would completely not use any install method like installwatch, 
checkinstall, or paco. It's just not needed anymore... chroot building 
fixes that problem, and it fixes the other problem too. (*)

> Whether the package management system is written in bash, C/C++, Perl
> Python, uses a database for dependencies, or whatever, is irrelevant.

Not really. At one point the amount of packages needing to be handled 
becomes too big, exactly the problem with lunar.

That's why redoing lunar in python/perl doesn't make sense: You'd be an 
idiot not going for C/C++ in the first place. So, we stick with bash.

> Sure, some give better support for certain operations than others, and
> may be faster at run-time, but none of the languages themselves address
> the issues above. They could also introduce new problems: would you
> want to automatically re-lin pyLunar when lunar update rebuilds Python?

yes, you most definately will want to do that.

> For whatever they're worth, my suggestions would be:
> (a) improve the cpu detection/handling for the gcc plugins;

what's wrong with -O2 -march=native?

really, this seems like a non-issue. Just make that the entire 
distro-default. Anyone can then change it for themselves.

> (b) streamline the ISO build system so that we can release new ISOs
>      as each major version of gcc/udev/whatever hits the moonbase.
>      Maybe not as full-release ISOs as such, but for use "as-is".
>
> I don't see how we can code around the "smart" configuration systems
> without building each module in an almost empty chroot environment.

what's wrong with building in a "clean" environment? seems like an ideal 
situation to me...

> We could maybe switch to using the slightly more recent installwatch
> from the checkinstall sources, but that doesn't solve everything.
> I wonder how the other source based distros work round these issues.

I checked it out, and, it doesn't function properly: You'll get all 
sorts of broken compiles if you use that codebase.

> Enough waffle from me.

Auke