Dependencies, their tracking and what we need to improve.

[Auke Kok]

On 07/14/2011 07:20 AM, Jean-Michel Bruenn wrote:
> Hey
>
>> if we had a way to build packages in a completely pristine environment
>> without non-preferred dependencies, then this would not be a problem.
>>
>> the cost of building in a chroot with only dependencies are that it is
>> slow. and you might as well run an rpm-based system almost, since in
>> order to obtain the flexibility that lunar grants, we'd have to do a LOT
>> more dependency and option mapping than we do now.
>>
>> Solving that problem... wouldn't be worth doing in the current codebase.
>
> The interesting question behind all this is: Do we want an improvement
> or not? Is it worth, trying to improve this or not? If it's not worth
> or possible with the current codebase, what steps could we do to
> make it worth it/possible. Do we have any alternatives? Apart from
> switching to rpm's of course :)

we could even switch to rpm... I'm not kidding.

the problem isn't the packaging format. the problem is the level of 
freedom that you want. no current build system will give it to you, so, 
switching package formats doesn't fill the gap.

> Also, "if we had a way .. in a completly pristine environment" Let's
> assume, we do it like this. I wrote a howto for a Chrooted Lunar,
> which is working quite nice. I've been using that thing first as zone in
> opensolaris, later as 32bit chroot for lunar and now just for testing.
> I might be able to make it a lot smaller (~500 MB packed, 1.8 G
> unpacked) for the above purpose, and i might make it install
> automatically using "lchroot" or something. So let's assume further, we
> take this chroot, for the purpose of doing dependency tracking, even if
> that sounds a bit.. masochistic. What possible improvements could we
> do, to make it "faster" (as you said it would be slow) and what
> alternatives do we have?

stop coding it in bash, start over in C.

>> it's actually not the sorting that is the problem. in reality, the
>> routine that does the sorting is the most optimized thing in lunar's
>> codebase there is.
>>
>> the real problem lies in the fact that for every module in the queue, we
>> need to satisfy the dependency list for the module itself as well, so
>> there are two solver steps. with a large install, we're out of control.
>> using a db might help, but with the codebase still being bash for the
>> rest of the walking about, it'll still be slow quickly with large
>> amounts of modules in the queue.
>
> I guess, dveatch meant exactly that (correct my if I'm wrong). At least
> if I'm talking about sorting, I'm talking about the whole process of
> checking every app and it's dependency for dependencies (isn't that
> what we do, this recursive stuff, which takes lots of time and resulted
> in some never-ending-loops back ago a few years?). Also i'm quite sure
> the bash stuff is very optimized - But still: Can't we make it faster?
> Do we want to make it faster? Would it hurt to make it faster?
>
> See: Databases are exactly made for this stuff. They should hold large
> amounts of data and make it fast to search within them. Currently
> we're using files to do this. Which is slow and disallows us to do
> certain types of sorting (which won't be needed anyway) however.
> Instead of opening several different files, we'd just issue several
> different queries to an sqlite database. I'm sure the ones of you who
> started with PHP like i, remember the times when ISPs didn't give a
> MySQL Database away together with PHP. So if you wrote a Guestbook, you
> wrote a text-file-based guestbook ->  and the more entries, the slower
> the guestbook. Right after 100 entries you could feel how it slows
> down, depending on your code of course :) A few years later you got
> your mysql database and you were able to see that it gives you 1000
> entries (instead of 100) back without any delay.
>
> However, back to topic and the questions: I know that some people are
> against using a database, that's why i'd like to implement it as some
> sort of "cache", which gets used if enabled/available and if it's
> up-to-date instead of as replacement. That way, nothing would change,
> except that it's in some cases faster.

should be trivial to write this as a plugin. just 'trap' in find_depends 
and satisfy_depends, and I think you're good.

Auke