moonbase / module and code testing / qa

[Couannette]

elaine wrote:
> Hello all module developers.
> 
> Feedback on these ideas or other ways to make test/release easier are 
> solicited! Sorry this is a long email.
> 
> I wish to propose adopting guidelines for deciding the degree of testing we 
> want developers to use in determining when a module is ready for release into
> moonbase.
> 
> We're all volunteers on this project and I think we all do pretty damn
> good work. 
> 
> However from back in SGL days there have been problems when modules with
> system-wide impact are released into moonbase and trigger unexpected 
> problems. Db4, glibc, gcc, gnome have all been examples.
> 
> There's a principle in engineering that's also well known/proven in
> software. The cost of fixing bugs/flaws grows exponentially as
> they move from R&D to alpha to beta to production. I beleive we'll
> save ourselves time in the long run if we can stop dropping untested
> updates into critical sections of the moonbase.
> 
And why not save effort by validating subsets of modules that will feed a "very stable" moonbase (I
see modules that don't update very often, and are very well tested. Such modules require only
"security updates" or small fixes). And a "common" moonbase (I see modules that are well appreciated
by users such as KDE or GNOME but that are subjet to changes and evolutions that break builts and
dependencies). And a "beta test" moonbase (I see modules such as beta software, not completly
implemented, not tested....).

examples:
"very stable": kernel
"common": kde/gnome
"beta test": kino

Please relegate all new KDE/GNOME releases to "alpha test": as a common user I can't stand seeing my
lunar update complains about a lot of update that breaks/fails. Please :-).


> Personally I'm will promise time for testing updates and would rather 
> dedicate some time to that upfront than spend (more) time fixing
> things after they've gone out to the wider userbase.
> 
> Suggestions:
> 
> 1. For complex modules, testing on the developement machine isn't 
> adequate. Dependencies which most users won't have installed are 
> certain to have been installed (perhaps a few times) on the target
> machine. Because Lunar doesn't have a rollback capabity returing
> to tabula-rasa
> 
My first though was creating a sandbox. But it's too much pain.
Pearhaps increase the dependency code wits & tests.

> 2. For modules with wide impact (Qt, libgnome, gtk+* glibc<shudder> ...)
> the effects of update may vary widely depending on what's installed
> 

Please don't !!!
A security update for sendmail or ssh : good :-).
An update of all gnome tree: PAINNNNNNNNNNNN !

> 3. Lunar users are not even close to in-sync on software versions. It's
> up to the system owner to decide what to hold or update. Therefor 
> it's *certain* that the userbase is running a wide variety of configs.
> 
An install feedback for statistics on lunar-linux.org could be implemented: thus succeded / failed
builds can be tracked and subsets of modules choosen by Lunar users can be stored for further
analysis... It can be of great impact to know what are the most common modules installed.

> 4. The 'profile' of the average lunar install is very different than
> the average developer's system. The best guess of our userbase is 4-600 
> systems. *most* of these people in fact don't update more often than 
> every 30-60 days. I imagine there are a few that go a year or more without
> general updates.
> 

Old UNIX administrator: Why update a system that's working ? Except for security ?
Don't as shamefull as M$...

> 
> I think we could adopt a rating system where required degree of testing is 
> determined by a formula, something like: (criticality*delta) / impact.
> 
> criticality is high for security flaws, low for feature enhancements,
> somewhere in between for everything else.
> 
> delta = deltaV*10+ delta-majorV + delta-minorV/10 (this probably needs 
> adjusting per-module because different projects have different policies
> for release / compatibility. e.g. openssl allows API changes between 
> e.g. 0.9.6-7 while gcc (to my knowlege) is object code compatible 
> across e.g. 3.2 - 3.3 versions.
> 
> impact is high for anything that creates libraries in the ld.so.conf paths,
> is likely to affect critical operations for users
> 
> We may want to build some variation on this formula into moonbase. 
> 
> To facilitate getting testing done with minimal dev overhead, perhaps
> (rather than informally passing around dev-modules via email) we can add a 
> directory dbguin's download area for modules in development, and 
> scripts to grab that dev area into /zlocal.
> 
> I can dedicate a couple of vmware boxes which are easy to restore to
> known-state as well as my primary dev system to this work.
> 
> Feedback on this or other ways to make test/release easier are solicited!
> 
> Here's a preliminary list of modules which I beleive ought to be tested
> on at least 3-8 systems prior to release to moonbase. 
> 
> glibc
> gcc
> readline
> bash
> coreutils
> ncurses
> dialog
> db
> db4
> perl
> openssl
> openssh
> krb5
krb5 -> never succeded in building this monster :'(
> zlib
> gawk
> gnome2
Do you think all people need this ?
> gettext
> qt3
> kde3
Same
> nfs-utils
> samba
> util-linux
> nasm
> modutils
> pkgconfig
> Linux-PAM

This list clearly holds all critical modules. I think we could tag modules with such "degree" of
necessity.

We can also tag modules with a "subject" tag: server/networking, kde-gnome/normal workstation,
gimp-audacity/multimedia workstation, emacs-lyx/scientific workstation, etc....

But I described two different semantics: one for criticality and one for usability.

Pearhaps two different tags can suffice. Perhaps a more cleaver system is to be built....

IMHO I prefer rely on a stable release of my prefered distro ;-), such as the preceding list, and
build my system with thematic subsets such as: graphics, web design, development, etc....

The most simple system could be:
	zlocal -> /usr/local
		with --prefix and $PATH with a test user....

This way a sub-moonbase contained in zlocal, or whatsoever name you want, enable module testers to
install zmodules on their system what live test updates without rely on gaz-factory such as VMWare....


Bon courage !
Quand on veut on peut !

Couannette