2.5.70 kernel initial tests

[HornGábor]

Hi!

It's a bit off topic, but i hope u find it interesting. I've read some
articles about the 2.5.x series, and decided to give it a try. i
downloaded the current (2.5.70), and could compile it w/o any problem
(it's compile mechanism is highly different from 2.4.x series...i won't
go into details, just mention it because it's the first 2.5.x i've tried
so was surprising to me :).

BTW my primary goal was to find the bottleneck of our current server,
which serves around 30 million hits/month (athlon 1700+ w/ 1G reg ecc
ddram, 2.4.20 w/ some patches, apache 1.3.27). We host a php based chat
application, which is a real challange for a webserver w/ this amount of
users/hits :) 

So first step was the kernel. i don't like play w/ production servers,
so i used my own computer. The circumstances of the test:

1, i wrote a little php script, whic runs from cron every minute, and
logs the date, the 1 minute load, the number of apache, mysql and
postgres sql threads into a database. 

2, i created a test envrionment w/ the chat system. this chat uses
direct files instead of mysql to store user texts (it's faster this
way). The chat's main engine is a php file, which can be called w/ wget
to simulate user requests. So i think it's similar now to the real one.

3, i set the initial runlevel to 3 to avoid X and other stuf to confuse
the things.

4, reboot, made the tests w/ both a 2.4.20-ck6 kernel, and a 2.5.70
vanilla, more times w/ both of them. 

5, the test itself: w/ siege (http://www.joedog.org/siege/index.shtml) i
ran an 5 minute test, simulating 25 concurrent user. after the 5 min 1
minute break, then again ran the test. Then again 1 minute break, then
ran the test 3rd time. (siege w/ 25 concurrent users stresses the server
MUCH more than 25 normal users, but it's nicely comparable and
reproducable). i repeated the above steps two times for both of the
kernels (w/ reboot between them).

and the results:

I'm quite impressed by 2.5.70! I only hoped a marginal differnce, but it
is HUGE! here'r the facts:

1, w/ 2.4.20 the load went up to 73 (!!!) during the 1st 5 minute
session.  During this  almost 25 percent of transactions FAILED, and the
REAL concurrency of apache was only 5.5 (instead of 25...)!!! Even cron
dameon couldn't made the database entries...here's  the table:

+------+-------+------+------+------+--------+--------+-------+------+
| year | month | day  | hour | min  | load_1 | apache | mysql | psql |
+------+-------+------+------+------+--------+--------+-------+------+
| 2003 |     5 |   30 |    2 |   17 |   0.11 |      4 |     9 |    3 |
| 2003 |     5 |   30 |    2 |   18 |   0.04 |      5 |    10 |    3 |
| 2003 |     5 |   30 |    2 |   19 |   0.01 |      5 |     9 |    2 |
| 2003 |     5 |   30 |    2 |   24 |  57.38 |    143 |     9 |    4 |
| 2003 |     5 |   30 |    2 |   24 |  22.21 |    126 |     8 |    4 |
| 2003 |     5 |   30 |    2 |   24 |  62.76 |    155 |    13 |    2 |
| 2003 |     5 |   30 |    2 |   24 |  73.42 |    157 |    11 |    2 |
| 2003 |     5 |   30 |    2 |   24 |   76.1 |    147 |    13 |    2 |

yes u guess right, i started the test at 2:19 :) So all the cron jobs
got executed after the siege (look at the min column)...:)

ok, 1 minute break then again the same test:

| 2003 |     5 |   30 |    2 |   25 |  35.92 |    115 |    13 |    2 |
| 2003 |     5 |   30 |    2 |   31 |  13.19 |     26 |    15 |    8 |
| 2003 |     5 |   30 |    2 |   31 |  13.19 |     22 |    11 |    4 |
| 2003 |     5 |   30 |    2 |   31 |  13.19 |     28 |    17 |   10 |
| 2003 |     5 |   30 |    2 |   31 |  18.99 |     24 |    15 |    4 |
| 2003 |     5 |   30 |    2 |   31 |  18.99 |     20 |    11 |    4 |
| 2003 |     5 |   30 |    2 |   31 |  18.99 |     20 |    12 |    6 |

Now it's much lower load, but cron damen again 'froze' during the test.
This time no failed transactions, but the real concurrency only 6.

ok, 1 minute break then the final ran of the same test:

| 2003 |     5 |   30 |    2 |   32 |  12.51 |     13 |    12 |    2 |
| 2003 |     5 |   30 |    2 |   37 |  45.18 |    102 |    12 |    2 |
| 2003 |     5 |   30 |    2 |   37 |  48.65 |    105 |    17 |    8 |
| 2003 |     5 |   30 |    2 |   37 |  42.71 |    104 |    19 |    6 |
| 2003 |     5 |   30 |    2 |   37 |  42.71 |     99 |    17 |    4 |
| 2003 |     5 |   30 |    2 |   37 |  54.96 |    107 |    14 |    4 |


This time 15 percent of transactions FAILED, and cron damen froze 'of
course' during the high load. Concurrency is now 14.

Look at the wide range of value of loads and number of apache treads.

I tried to run the test for 4th time, but this time it halted completely
because of the high load.

2, reboot to 2.5.70, evrything else is the same. Now i paste it it one,
because it's absolutelly smooth:


| 2003 |     5 |   30 |    2 |   41 |  14.78 |     38 |     9 |    2 |
| 2003 |     5 |   30 |    2 |   42 |  21.63 |     37 |     9 |    3 |
| 2003 |     5 |   30 |    2 |   43 |  24.74 |     36 |    11 |    3 |
| 2003 |     5 |   30 |    2 |   44 |   25.4 |     38 |    12 |    1 |
| 2003 |     5 |   30 |    2 |   45 |  25.46 |     28 |    12 |    1 |
| 2003 |     5 |   30 |    2 |   46 |  11.45 |     12 |     9 |    1 |
| 2003 |     5 |   30 |    2 |   47 |  17.88 |     50 |    10 |    2 |
| 2003 |     5 |   30 |    2 |   48 |  23.44 |     37 |    10 |    2 |
| 2003 |     5 |   30 |    2 |   49 |  24.94 |     37 |    11 |    2 |
| 2003 |     5 |   30 |    2 |   50 |  25.86 |     36 |    12 |    3 |
| 2003 |     5 |   30 |    2 |   51 |  26.12 |     21 |    12 |    1 |
| 2003 |     5 |   30 |    2 |   52 |  12.14 |     12 |    10 |    1 |
| 2003 |     5 |   30 |    2 |   53 |     21 |     53 |     9 |    2 |
| 2003 |     5 |   30 |    2 |   54 |  24.27 |     36 |    10 |    2 |
| 2003 |     5 |   30 |    2 |   55 |  25.25 |     37 |    12 |    1 |
| 2003 |     5 |   30 |    2 |   56 |  25.72 |     36 |    13 |    3 |
| 2003 |     5 |   30 |    2 |   57 |  25.64 |     37 |    13 |    1 |


yes, u guess right. I started the test at 2:41, the breaks were 2:46 and
2:52..as it clearly can be seen from loads. No failed transactions at
all during the whole test, continous crontab entries, the load is much
balanced and lower (max 26, not 74...), the realised conccurency was
23!!! 

So all i can say to the new kernel is WOW! I'll do the same test w/
apache1 vs apache2 (w/ both of the kernels, hehe :), i'm interested if
there's a noticable differnece between them. 

Hope u don't mind i pasted those long tables, but they speak more clear
than any words imho :)

bye, hirisov