[patch] Vastly improved sources() function

[Auke Kok]

On Mon, 2003-05-12 at 16:39, Simon Brenner wrote:
> I found that AUTOPRUNE took a *very* long time to complete on my 
> computer (P2 450), something like hours... (I rarely let it complete the 
> AUTOPRUNE)
> 
> So I looked into the workings of the prune command, and found that the 
> subroutine taking up the most time was the sources() function in 
> sources.lunar.
> 
> I took the sources() function, and asked myself what could be taking it 
> so much time. Not very hard to find out, the problem was the two uses of 
> sed to replace references to $SOURCE0 into $SOURCE (The first source in 
> the DETAILS file is not indexed). Two calls to `sed s/SOURCE0/SOURCE/` 
> per source variable processed.
> 
> So, to eliminate the sed calls, I removed the sed calls and added one 
> line before the source variables are looped through:
> SOURCE0=$SOURCE
> , thereby eliminating the need to translate $SOURCE0 into $SOURCE
> 
> I tested it (no fancy timing or anything, just thumbs measure), and 
> found it roughly 10 times faster (serious!). A lunar prune completed in 
> a couple of minutes, instead of an hour
> 
> Well... Just apply the patch =) Not that I am a master at the inner 
> workings of lunar, but I can see no bugs caused by this modification...
> 
> I hope I made the world a slightly better place now =)

you did! 

however, I'let you in on a few secrets that slightly destruct the 10x
speed improvement on your system...

The main problem with speed profiling is getting a fair speed
measurement. Since cache's can screw stuff up rapidly the effective
speed gain might not be 10x, but muuuuch less. Here's what I did...

the sed indeed is horrible! fork() is awfully slow, even on newer
kernels and always a bottleneck. Bash however can sed VARIABLES
perfectly, so removing the sed and replacing it by a
${VAR/SOURCE0/SOURCE} already wins quite a lot, but not 10x

try running lsh and fill up your cache with relevant data:

$ lsh
> cd $MOONBASE
> time cat */*/DETAILS 2> /dev/null

if you do that the first time nothing is cached, my P3 takes about 2:40.
Second time I do so it falls back to 0.1 second (yes, that's 1600x
faster!!!).

repeat the test above and look at the speed results, you'll get an idea
whether your cache gets saturated or not!

if you have all the moonbase's DETAILS file in your cache, you can
honestly trust speed measurements, so before benchmarking the prune
function, try filling the cache with relevant data the way I described
above.

the generate_keep_list is the big problem, since it uses the sources()
function you patched. profiling it from lsh helps:

$ lsh
> time KEEP=/tmp/x generate_keep_list

removing the sed made it run in 1:40 on my P3, so not even 2x faster!
However, if you check out the code:

> which generate_keep_list
....

you could have seen that the code actually REPEATS the whole exercise to
clear the varables before it exits. Quite bad since the ()'s around the
entire function assure no variable "escapes" when the function exits.

After I removed that code, it averaged 1:20, so about 2x faster

the above is a good guide in how to profile lunar code, anyone
interested should drop by in #lunar on irc.freenode.net, perhaps there's
more code that can be speedy'd up a bit!

thanks!

PS1: anyone thinking he can make sources() faster is invited to submit
snipplets!
PS2: I'll commit the changes suggested and a few more in due time to the
core code

sofar

-- 
Auke Kok <auke.kok at planet.nl>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dbguin.lunar-linux.org/pipermail/lunar/attachments/20030512/f80bfe11/attachment.bin