Dependencies, their tracking and what we need to improve.

[wookietreiber]

On Fri, Jul 15, 2011 at 10:36:22AM -0400, Dennis Veatch wrote:
> On Friday, July 15, 2011 4:01:08 PM wookietreiber wrote:
> > On Fri, Jul 15, 2011 at 07:36:25AM -0400, Dennis Veatch wrote:
> > > On Friday, July 15, 2011 1:19:44 PM wookietreiber wrote:
> > > > On Fri, Jul 15, 2011 at 06:45:21AM -0400, Dennis Veatch wrote:
> > > > > On Thursday, July 14, 2011 11:50:19 PM Auke Kok wrote:
> > > > > > On 07/14/2011 07:20 AM, Jean-Michel Bruenn wrote:
> > > > > > > Hey
> > > > > > > 
> > > > > > >> if we had a way to build packages in a completely
> > > > > > >> pristine
> > > > > > >> environment without non-preferred dependencies, then
> > > > > > >> this
> > > > > > >> would not be a problem.
> > > > > > >> 
> > > > > > >> the cost of building in a chroot with only
> > > > > > >> dependencies are
> > > > > > >> that it is slow. and you might as well run an
> > > > > > >> rpm-based
> > > > > > >> system almost, since in order to obtain the
> > > > > > >> flexibility that
> > > > > > >> lunar grants, we'd have to do a LOT
> > > > > > >> more dependency and option mapping than we do now.
> > > > > > >> 
> > > > > > >> Solving that problem... wouldn't be worth doing in the
> > > > > > >> current
> > > > > > >> codebase.
> > > > > > > 
> > > > > > > The interesting question behind all this is: Do we want
> > > > > > > an
> > > > > > > improvement or not? Is it worth, trying to improve this
> > > > > > > or not?
> > > > > > > If it's not worth or possible with the current codebase,
> > > > > > > what
> > > > > > > steps could we do to make it worth it/possible. Do we
> > > > > > > have any
> > > > > > > alternatives? Apart from switching to rpm's of course :)
> > > > > > 
> > > > > > we could even switch to rpm... I'm not kidding.
> > > > > > 
> > > > > > the problem isn't the packaging format. the problem is the
> > > > > > level of
> > > > > > freedom that you want. no current build system will give it
> > > > > > to you,
> > > > > > so,
> > > > > > switching package formats doesn't fill the gap.
> > > > > > 
> > > > > > > Also, "if we had a way .. in a completly pristine
> > > > > > > environment"
> > > > > > > Let's
> > > > > > > assume, we do it like this. I wrote a howto for a
> > > > > > > Chrooted
> > > > > > > Lunar,
> > > > > > > which is working quite nice. I've been using that thing
> > > > > > > first as
> > > > > > > zone in opensolaris, later as 32bit chroot for lunar and
> > > > > > > now
> > > > > > > just for testing. I might be able to make it a lot
> > > > > > > smaller
> > > > > > > (~500 MB packed, 1.8 G unpacked) for the above purpose,
> > > > > > > and i
> > > > > > > might make it install automatically using "lchroot" or
> > > > > > > something. So let's assume further, we take this chroot,
> > > > > > > for
> > > > > > > the purpose of doing dependency tracking, even if that
> > > > > > > sounds a
> > > > > > > bit.. masochistic. What possible improvements could we
> > > > > > > do, to
> > > > > > > make it "faster" (as you said it would be slow) and what
> > > > > > > alternatives do we have?
> > > > > > 
> > > > > > stop coding it in bash, start over in C.
> > > > > 
> > > > > I wouldn't be opposed to C or perl, python or whatever.
> > > > > 
> > > > > What I do not see any of those being able to solve is the root
> > > > > of the
> > > > > problem or nearly so; the lack of an on/off switch. As I
> > > > > mentioned,
> > > > > there are many instances of configure and cmake that will look
> > > > > for an
> > > > > application but give you no way to enable/disable. How would you
> > > > > address that with C?
> > > > 
> > > > doesn't depend on the language
> > > > 
> > > > what I think could solve the problem is to combine CONFIGURE and
> > > > DEPENDS or at least use logic in any way in depends -- both solve
> > > > hell'a more problems, e.g.
> > > > 
> > > > with-x? then
> > > > 
> > > > 	with-x-screensaver?
> > > > 
> > > > else
> > > > 
> > > > 	with-direct-fb?
> > > > 
> > > > ...
> > > > 
> > > > currently this kind of configuration is impossible to do right ...
> > > > 
> > > > regards
> > > > wookietreiber
> > > 
> > > No that really doesn't solve the no on/off switch problem. Suppose you
> > > have grantlee installed and for whatever reason you do not want kdelibs
> > > to use it. The kdelibs cmake does not have a dis-abler and while
> > > grantlee is installed, cmake will search for and include its
> > > libs/includes during its configure and kdelibs will be built with
> > > grantlee. The only way I see to allow for a user to truly say no to
> > > grantlee is to lrm grantlee before kdelibs runs cmake.
> > > 
> > > So for those configure and cmake scripts that do not provide an on/off
> > > switch, what do we insert into the optional_depends for grantlee to
> > > allow the user the ability to truly say no?
> > > 
> > > I do not see a level of logic we can insert or rewrite how
> > > optional_depends are used when we are dependent on configure, cmake,
> > > waf -configure, etc to provide us with the proper/appropriate methods
> > > to enable/disable an optional dependency.
> > > 
> > > Maybe there is a way and I am just blind.
> > 
> > has anyone heard of modules? (http://modules.sourceforge.net/)
> > 
> > we use this at work (scientific computing) to provide different versions of
> > libraries and some stuff even compiled with different compilers (gcc, icc,
> > ...)
> > 
> 
> That looks interesting if we had an unmanageable problem with module versions. 
> That's not to say it would not be useful to us and it might be something to 
> look into or adapt to our own needs.
> 
> > works quite well and I have been thinking about using this with a source
> > based distro for a while now, but I don't know if it would work or if the
> > complexity is even remotely maintainable ... though it would certainly fix
> > our problem (I imagine the chroot environment does about the same?)
> > 
> 
> I fail to see how it obviates our reliance on the various configure/cmake/etal 
> scripts to provide us with the proper on/off switches. How would module 
> version's solve that? Since you have experience with this, perhaps you could 
> explain.

at work we build the non-system stuff with prefix

/usr/local/$MODULE/$VERSION/

so headers and libs are not visible normally (because usually LD_LIBRARY_PATH
is not set to these dirs). thus the on switch would have to include to load
the module, which contains these paths or configure scripts will fail to find
the headers/libs because they don't know where to search. following is an
example of such a module:

# module show ncl/5.2.1
-------------------------------------------------------------------
/usr/local/Modules/modulefiles/tools/ncl/5.2.1:

setenv           NCARG_ROOT /usr/local/ncarg/5.2.1
prepend-path     LD_LIBRARY_PATH /usr/local/ncarg/5.2.1/lib
prepend-path     MANPATH /usr/local/ncarg/5.2.1/man
prepend-path     PATH /usr/local/ncarg/5.2.1/bin
module           load netcdf/4.1.1_gcc41
module           load hdf/4.2.5-netcdfless
module           load gdal/1.8.0_gcc41
module           load proj/4.7.0_gcc41
-------------------------------------------------------------------

as you can see you can define all env vars in such a module that are important
for building (whats missing in this example is CPPFLAGS for headers and maybe
pkgconfig).

conclusion: if you don't want a module, don't set the paths (which is not
possible, if everything gets built to the same prefix)