[tech] martello speed

[Matt Johnston]

On Wed, Mar 23, 2005 at 06:41:43PM +0800, James Andrewartha wrote:
> The new SATA controller has been installed in martello, and I've been
> benchmarking various RAID configurations with bonnie++. The results are

> RAID 5
> ------
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
martello         4G 38175  92 65903  27 20999   8 26989  60 58108  12 170.9   0
> 
> (fiddling with blockdev --setra didn't do much)

*snip*
> The bare disk result bears out the hdparm value of 55MB/s. The question is,
> where's the performance improvement that should happend for RAID 1+0 and
> RAID 5? The slow resync and hdparm result during array construction says to
> me that the ports on the controller are not independent.
> 
> A little searching finds
> http://forums.storagereview.net/index.php?showtopic=17794&hl=promise which
> claims "The linux driver sends only one command to the board at a time."

We've been playing with various drivers and things over the
past week with some results. The sx8.c driver in the kernel
does indeed only queue one command at a time. There's a
parameter which can be tweaked (CARM_MAX_Q), but we found
that would cause corruption for particular values - knowing
safe values seemed too hard. The author of the driver had
only tested on prototype hardware, so didn't know what would
be safe. Updating to the latest firmware didn't help either.

There's a seperate open source driver available at
promise.com, which has better performance than the sx8.c
driver in the kernel, so we're using that one. The main
downside is that it will probably bitrot more than the
in-kernel sx8.c driver.

In the process of testing we discovered a bug in Linux's
ext2 driver, and hit another bug (already known and fixed)
in the ext3 driver... These have now been patched on
martello.

I've played with --setra on the logical volume itself, and
that does seem to give performance improvements. See below
for some runs of bonnie++ with various settings.

Anyway, the drives/controller/FS now seem fairly stable in
their current configuration.

Matt


Single runs:
============

Plain RAID5, /dev/md0

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
martello         4G 44153  96 71542  16 20943   7 34897  77 144465  30 447.8   0

LVM (default readahead of 256 for /dev/sd{b..e} and /dev/mapper/reliable-home

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
martello         4G 38025  96 71307  31 28362  12 31777  67 91284  17 378.2   1

LVM, readahead 512 for /dev/sd{b..e}, 2048 for /dev/mapper/reliable-home

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
martello         4G 37334  95 73596  37 30293  12 42510  92 173115  37 407.2   1


Concurrent runs (4x bonnie++ with -y flag):
===========================================

RAID5

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
martello         4G  7706  17  7740   1  5117   1 13818  30 15072   3  91.7   0

LVM with default readahead

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
martello         4G  7566  19  9762   5  5677   2 11928  26 13976   2  79.1   0

LVM with increased readahead

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
martello         4G  9474  24  9354   4  6870   2 19414  42 20497   4  79.7   0