concurrent builds

[wookietreiber]

Hello,

I wanted to present a use-case for concurrent builds with lin.

If all modules could be built with 'make -j CORES' this wouldn't be necessary, but unfortunately this is not the case. Some modules aren't PSAFE (they can't build concurrently), others don't use make. So with non-concurrent builds it is not always possible to achieve optimal resource usage and the minimum amount of time to install modules (with modern CPU architectures in mind, that have many cores).

on PSAFE (I subtracted what I have in zlocal):

modules in moonbase:
$ find /var/lib/lunar/moonbase/ -name *DETAILS | wc -l
3755

non-PSAFE modules:
$ grep -r PSAFE /var/lib/lunar/moonbase/ | grep DETAILS | wc -l
289


My example is about ncurses:
ncurses: sed gawk [gpm]
|--->sed:
|--->gawk:
|--->[gpm]: chkconfig bison

Now imagine ncurses, sed, gawk and gpm are not installed, but gpm's dependencies are. Imagine further that we have a six-core CPU.

Now the user wants to 'lin ncurses'

lin's procedure should be the following:
1. which modules need to be installed in which order?
1.1 modules to lin: sed gpm gawk ncurses
1.2 sed gawk and gpm can be built concurrently since all their dependencies are met, then ncurses has to be built alone
2. figure out which build type they are: they all use make and can get '-j x'
3. figure out which modules aren't PSAFE: sed is the only module that is PSAFE
4. resource distribution and build order:
4.1 lin gpm and lin gawk each with 1 core (not PSAFE) and lin sed with the remaining 4 cores (all concurrently for optimal resource usage)
4.2 some kind of barrier waits until they compiled successfully
4.3 lin ncurses with one core (not PSAFE) -- no further optimization can be achieved

One of the disadvantages of concurrent builds is that lunars voyeur feature wouldn't work -- at least not for all modules. But the speedup is definetely worth it :)

best regards
Christian Krause aka wookietreiber