Dependencies, their tracking and what we need to improve.

[Auke Kok]

On 07/13/2011 05:27 PM, Dennis Veatch wrote:
> On Thursday, July 14, 2011 12:11:20 AM Zbigniew Luszpinski wrote:
>>> There are still some open questions that need additional discussion or
>>> clarification about Lunar's dependency system. In the past there has
>>> been some talk about switching the dependency tracking to a database
>>> and Jean has done some work using sqlite.
>>>
>>> With the help of Ratler it is now in a usable shape as for its speed.
>>> There are a few things left such as tying it into lin, lvu, etc and
>>> what should be done as a fall back should the database become
>>> completely hosed.
>>>
>>> Before that happens (if it does) there has been mixed opinions about
>>> dependencies. Specifically;
>>>
>>> 1. those of the optional variety
>>> 2. how explicit the DEPENDS should be
>>> 3. and how far down the tree should a `lrm -p $MODULE` should go
>>> 4. do we really need to switch to a database or provide a user with a
>>> choice of database or current method
>>>
>>> Point 1 - Auke has stated he does not like seeing an optional_depends
>>> without an accompanying on/off switch. I can understand that because
>>> the optional_depends without it could cause the module to fail
>>> requiring the user to lrm the offender and then later on reinstall it.
>>>
>>> But the problem is this; there are many modules with their configure and
>>> cmake scripts that have no corresponding on/off switch. For example;
>>> kdelibs4 has no way to enable/disable grantlee even though it can be
>>> used during its build. So what is an acceptable method of handling
>>> these types of optional_depends?

realistically? if there really is no way, then you just leave $2 and $3 
empty, in which case, if the dependency is not installed, one is asked 
the appropriate question to install it or not, and so, it still is 
optional. But if the dependency is installed, you are not asked (see below).

>>> Point 2 - This has some bearing on Point 1 in that to make a modules
>>> DEPENDS more explicit we run into those that have no corresponding
>>> on/off switch. The second part to this point is; how far down the tree
>>> should we go to have a comprehensive dependency tracking system?

this is a more fundamental question and at this time we cannot solve 
this issue due to the way that we build packages.

if we had a way to build packages in a completely pristine environment 
without non-preferred dependencies, then this would not be a problem.

the cost of building in a chroot with only dependencies are that it is 
slow. and you might as well run an rpm-based system almost, since in 
order to obtain the flexibility that lunar grants, we'd have to do a LOT 
more dependency and option mapping than we do now.

Solving that problem... wouldn't be worth doing in the current codebase.

>>> Point 3 - As it stands now and I have not tried this in a very long time
>>> because it broke my box; at what point do we halt a `lrm -p $MODULE`?
>>> Even with a ultra-mega super-duper tracking system this would still be
>>> of concern.

we should probably gather all the modules up first, before deleting 
them. right now it's done inversely, and that gives the problems.

>>> Point 4 - I am amiable to the idea if only from the point that it
>>> could/would speed up the sorting which right now is not so great. Just
>>> so that we have it documented, please itemize what a database
>>> dependency tracking system would provide and potentially change how a
>>> DEPENDS is currently constructed.

it's actually not the sorting that is the problem. in reality, the 
routine that does the sorting is the most optimized thing in lunar's 
codebase there is.

the real problem lies in the fact that for every module in the queue, we 
need to satisfy the dependency list for the module itself as well, so 
there are two solver steps. with a large install, we're out of control. 
using a db might help, but with the codebase still being bash for the 
rest of the walking about, it'll still be slow quickly with large 
amounts of modules in the queue.

>>> If we are going to move forward to improve the dependency tracking I
>>> feel there needs to be additional inputs. If the work Jean has done is
>>> the direction to head it will need testing on multiple boxes by
>>> multiple users (devs). I have no problem being part of that but given
>>> the scope the more the merrier.
>>>
>>> Side note; Would it be preferable to setup an IRC meeting to discuss
>>> this or is everyone fine with the mail list?
>>
>> One more thing.  Two options in lunar menu would be useful for depends:
>> [X] If module installed use it as optional depends without asking
>
> I disagree with that; mainly it would be an extra layer/function that really
> does not address any of the points in this thread. At this point I do not
> think we need another menu selection.
>
> A user should always have the choice regarding optional_depends and is in part
> the bone of contention put forth by Auke to which I agree. See Point 1.
>
> The only idea I have to address Point 1 is one I don't particularly like and
> it goes something like this;
>
> Create our own on/off switch, call it --lunar-opt-on and --lunar-opt-off (yes my
> naming convention skills suck). If an optional_depends is already installed,
> configure/cmake will setup themselves up to use it. So you use --lunar-opt-off,
> the optional_depends will be removed and after the module is installed, re-
> install the optional_depends.

hehe, funny. good joke. can I watch when you try that code? :)

>> [X] If module not installed do not ask to install it as optional depends
>
> I don't understand what your driving at here and if I do it makes no sense.

This already happens in fact, see the code:

        if module_installed $DEP && [ -z "$2" -a -z "$3" ]; then
                 verbose_msg "Auto-adding optional dependency \"$DEP\"";
                 add_depends "$MODULE" "$DEP" "on" "optional" "$2" "$3";
                 lin --deps $DEP;

Auke