Dependencies, their tracking and what we need to improve.

[wookietreiber]

On Fri, Jul 15, 2011 at 08:42:07PM -0700, Auke Kok wrote:
> On 07/15/2011 04:36 AM, Dennis Veatch wrote:
> >On Friday, July 15, 2011 1:19:44 PM wookietreiber wrote:
> >>On Fri, Jul 15, 2011 at 06:45:21AM -0400, Dennis Veatch wrote:
> >>>On Thursday, July 14, 2011 11:50:19 PM Auke Kok wrote:
> >>>>On 07/14/2011 07:20 AM, Jean-Michel Bruenn wrote:
> >>>>>Hey
> >>>>>
> >>>>>>if we had a way to build packages in a completely pristine
> >>>>>>environment without non-preferred dependencies, then this
> >>>>>>would not be a problem.
> >>>>>>
> >>>>>>the cost of building in a chroot with only dependencies are
> >>>>>>that it is slow. and you might as well run an rpm-based
> >>>>>>system almost, since in order to obtain the flexibility that
> >>>>>>lunar grants, we'd have to do a LOT
> >>>>>>more dependency and option mapping than we do now.
> >>>>>>
> >>>>>>Solving that problem... wouldn't be worth doing in the current
> >>>>>>codebase.
> >>>>>
> >>>>>The interesting question behind all this is: Do we want an
> >>>>>improvement or not? Is it worth, trying to improve this or not?
> >>>>>If it's not worth or possible with the current codebase, what
> >>>>>steps could we do to make it worth it/possible. Do we have any
> >>>>>alternatives? Apart from switching to rpm's of course :)
> >>>>
> >>>>we could even switch to rpm... I'm not kidding.
> >>>>
> >>>>the problem isn't the packaging format. the problem is the level of
> >>>>freedom that you want. no current build system will give it to you,
> >>>>so,
> >>>>switching package formats doesn't fill the gap.
> >>>>
> >>>>>Also, "if we had a way .. in a completly pristine environment"
> >>>>>Let's
> >>>>>assume, we do it like this. I wrote a howto for a Chrooted
> >>>>>Lunar,
> >>>>>which is working quite nice. I've been using that thing first as
> >>>>>zone in opensolaris, later as 32bit chroot for lunar and now
> >>>>>just for testing. I might be able to make it a lot smaller
> >>>>>(~500 MB packed, 1.8 G unpacked) for the above purpose, and i
> >>>>>might make it install automatically using "lchroot" or
> >>>>>something. So let's assume further, we take this chroot, for
> >>>>>the purpose of doing dependency tracking, even if that sounds a
> >>>>>bit.. masochistic. What possible improvements could we do, to
> >>>>>make it "faster" (as you said it would be slow) and what
> >>>>>alternatives do we have?
> >>>>
> >>>>stop coding it in bash, start over in C.
> >>>
> >>>I wouldn't be opposed to C or perl, python or whatever.
> >>>
> >>>What I do not see any of those being able to solve is the root of the
> >>>problem or nearly so; the lack of an on/off switch. As I mentioned,
> >>>there are many instances of configure and cmake that will look for an
> >>>application but give you no way to enable/disable. How would you
> >>>address that with C?
> >>
> >>doesn't depend on the language
> >>
> >>what I think could solve the problem is to combine CONFIGURE and DEPENDS 
> >>or
> >>at least use logic in any way in depends -- both solve hell'a more
> >>problems, e.g.
> >>
> >>with-x? then
> >>	with-x-screensaver?
> >>else
> >>	with-direct-fb?
> >>...
> >>
> >>currently this kind of configuration is impossible to do right ...
> >>
> >>regards
> >>wookietreiber
> >
> >No that really doesn't solve the no on/off switch problem. Suppose you have
> >grantlee installed and for whatever reason you do not want kdelibs to use 
> >it.
> >The kdelibs cmake does not have a dis-abler and while grantlee is 
> >installed,
> >cmake will search for and include its libs/includes during its configure 
> >and
> >kdelibs will be built with grantlee. The only way I see to allow for a 
> >user to
> >truly say no to grantlee is to lrm grantlee before kdelibs runs cmake.
> >
> >So for those configure and cmake scripts that do not provide an on/off 
> >switch,
> >what do we insert into the optional_depends for grantlee to allow the user 
> >the
> >ability to truly say no?
> >
> >I do not see a level of logic we can insert or rewrite how optional_depends
> >are used when we are dependent on configure, cmake, waf -configure, etc to
> >provide us with the proper/appropriate methods to enable/disable an 
> >optional
> >dependency.
> >
> >Maybe there is a way and I am just blind.
> >
> 
> the only real solution is to build in an environment where you control 
> 100% what the configure script "sees".
> 
> there are several ways of doing that
> 
> - chroot or VM - guaranteed 100% works exactly as expected
> 
> - tricks (like my pkg-config plugin) that fool the system into thinking 
> disabled dependencies are not there - this will break in the end.
> 
> I'm not too convinced to make lunar more complex with really nasty hacks 
> anymore... I've spent enough time on it to realize that apart from 
> keeping lunar going as is, it's just not worth my time to "refine" the 
> current codebase.
> 
> changing lunar to build stuff in chroots isn't something small either, 
> and because chroot building is complex (you need to redo the chroot for 
> every compile), the current bash codebase is just too slow to do this 
> efficiently.
> 
> do I know how to solve it? sure. but it will be the end of lunar, and 
> like starting from scratch....
> 
> Auke

can't we work dynamically with either the chroot or the module solution,
i.e. flag modules to use it only if necessary? of course we would have to
implement either solution and add them to the codebase, but we still can
use the old stuff for things that work, e.g. add just add

default_module_based_config

and this is where we set

--prefix=/usr/$MODULE/$VERSION

and setup the module file containing LD_LIBRARY_PATH n stuff, so it can be
dynamically loaded.

I don't know much about how the chroot solution would work, I must admit,
but the module-based solution certainly has the advantage that we can
dynamically allow users to have multiple versions installed which fixes
the problem that some libraries like poppler usually break compatibility
even with minor version bumps. this way you can have both the old working
version as well as the cutting edge one.