Dependencies, their tracking and what we need to improve.

[Jean-Michel Bruenn]

Hey

> if we had a way to build packages in a completely pristine environment 
> without non-preferred dependencies, then this would not be a problem.
> 
> the cost of building in a chroot with only dependencies are that it is 
> slow. and you might as well run an rpm-based system almost, since in 
> order to obtain the flexibility that lunar grants, we'd have to do a LOT 
> more dependency and option mapping than we do now.
> 
> Solving that problem... wouldn't be worth doing in the current codebase.

The interesting question behind all this is: Do we want an improvement
or not? Is it worth, trying to improve this or not? If it's not worth
or possible with the current codebase, what steps could we do to
make it worth it/possible. Do we have any alternatives? Apart from
switching to rpm's of course :)

Also, "if we had a way .. in a completly pristine environment" Let's
assume, we do it like this. I wrote a howto for a Chrooted Lunar,
which is working quite nice. I've been using that thing first as zone in
opensolaris, later as 32bit chroot for lunar and now just for testing.
I might be able to make it a lot smaller (~500 MB packed, 1.8 G
unpacked) for the above purpose, and i might make it install
automatically using "lchroot" or something. So let's assume further, we
take this chroot, for the purpose of doing dependency tracking, even if
that sounds a bit.. masochistic. What possible improvements could we
do, to make it "faster" (as you said it would be slow) and what
alternatives do we have?

> it's actually not the sorting that is the problem. in reality, the 
> routine that does the sorting is the most optimized thing in lunar's 
> codebase there is.
> 
> the real problem lies in the fact that for every module in the queue, we 
> need to satisfy the dependency list for the module itself as well, so 
> there are two solver steps. with a large install, we're out of control. 
> using a db might help, but with the codebase still being bash for the 
> rest of the walking about, it'll still be slow quickly with large 
> amounts of modules in the queue.

I guess, dveatch meant exactly that (correct my if I'm wrong). At least
if I'm talking about sorting, I'm talking about the whole process of
checking every app and it's dependency for dependencies (isn't that
what we do, this recursive stuff, which takes lots of time and resulted
in some never-ending-loops back ago a few years?). Also i'm quite sure
the bash stuff is very optimized - But still: Can't we make it faster?
Do we want to make it faster? Would it hurt to make it faster?

See: Databases are exactly made for this stuff. They should hold large
amounts of data and make it fast to search within them. Currently
we're using files to do this. Which is slow and disallows us to do
certain types of sorting (which won't be needed anyway) however.
Instead of opening several different files, we'd just issue several
different queries to an sqlite database. I'm sure the ones of you who
started with PHP like i, remember the times when ISPs didn't give a
MySQL Database away together with PHP. So if you wrote a Guestbook, you
wrote a text-file-based guestbook -> and the more entries, the slower
the guestbook. Right after 100 entries you could feel how it slows
down, depending on your code of course :) A few years later you got
your mysql database and you were able to see that it gives you 1000
entries (instead of 100) back without any delay.

However, back to topic and the questions: I know that some people are
against using a database, that's why i'd like to implement it as some
sort of "cache", which gets used if enabled/available and if it's
up-to-date instead of as replacement. That way, nothing would change,
except that it's in some cases faster.

Jean