Simple performance tuning tips
> I don't know if it is IO bound and don't really know how to test > such things. I just know access isn't as snappy as it used to be.
When the system is running fast/normally you need to take a baseline. Look at top, see what your percentage system, user, idle and iowait is. These can be seen here:
Cpu(s): 8.6% us, 0.3% sy, 0.0% ni, 90.5% id, 0.6% wa, 0.0% hi, 0.0% si
- us : percent cpu being used by userland code
- sy : percent cpu being used by kernelspace code
- ni : like “us” but related to “niced” processes
- id : idle
- wa : cpu is idle because it waits for IO to complete
- hi : interrupts generated by hardware
- si : interrupts generated by software
Get a feel for how this looks when your system is running well. In an ideal world you want to see a load average below 1 with “wa” below 5% although what is considered “normal” varies from server to server.
Then look at more detailed vm subsystem stats, again while system is performing well. For this you can use the vmstat command.
vmstat 5 will print an initial header and stats-to-date line and then an additional line every 5 seconds with differential stats. Leave this going for a while so you have a good idea what this normally looks like. e.g.:
$ vmstat 5 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 94348 13440 13428 60304 0 0 2 2 2 1 9 0 90 1 0 0 94348 13456 13452 60304 0 0 0 86 46 70 0 0 100 0 0 0 94348 13464 13460 60304 0 0 0 5 30 65 0 0 100 0 0 0 94348 13472 13472 60304 0 0 0 17 32 66 0 0 100 0 0 0 94348 13472 13480 60304 0 0 0 3 30 64 0 0 100 0 0 0 94348 13472 13496 60304 0 0 0 17 37 68 0 0 100 0 0 0 94348 13472 13512 60304 0 0 0 18 33 67 0 0 100 0
- r : processes ready to run but are waiting for the cpu
- b : processes in uninterruptible sleep (often io-related)
- swpd : swap used (KiB)
- free : memory doing nothing (KiB)
- buff : buffers (KiB)
- cache : disk cache (KiB)
- si : memory swapped in from disk (KiB/s)
- so : memory swapped out from disk (KiB/s)
- bi : Blocks read in from block devices (blocks/sec)
- bo : Blocks written out to block devices (blocks/sec)
- in : interrupts per second (hardware, software, e.g. “the clock ticked” “the ethernet card got a packet”)
- cs : context switches per second (one process stops running on CPU, another starts)
- us : Userland CPU usage %
- sy : Kernelspace CPU usage %
- id : Idle CPU %
- wa : IO wait CPU %
Ignore the first line. You are looking for “r” and “b” to be low, certainly single digits, preferably zero. If they’re not then processes generally are lacking resources, usually CPU or disk.
Amount of swap (“swpd”) is not as important as swap churn (“si”, “so”) which should be close to zero.
The remaining io/system/cpu figures can give you an idea of how busy your machine normally is in terms of CPU and disk.
Now repeat that when your system is running slow. By looking at what is different you can narrow down performance bottlenecks.
e.g. top says load is high, “wa” is low but “us” is high and this goes on for long periods and coincides with poor performance. Therefore your processes are CPU starved, give the machine more CPU (or run better code!) and things will improve (or you’ll find a different bottleneck!)
e.g. top says load is high, “us” is low, “wa” is above 75% for a long time: your processes are waiting for IO, could be network or disk or various other things, but is most likely disk.
Well, OK so you find out your system is starved for IO, but that’s a bit vague. You need to know what is wanting the IO. This can be a bit tricky but something that can really help is iostat. iostat -x 5 will show you the IO to every partition on your system updated every 5 seconds:
(output from a RHEL box as none of my play machines have significant disk IO; 2.6 kernel output may be slightly different)
$ iostat -x 5
Linux 2.4.21-37.ELhugemem (xxxpdb02) 02/02/06
avg-cpu: %user %nice %sys %iowait %idle
7.78 0.00 8.98 30.02 53.22
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 2.76 9.67 8.56 18.87 90.36 228.49 45.18 114.25 11.62 0.00 0.07 0.19 0.53
sda1 0.03 0.00 0.00 0.00 0.06 0.00 0.03 0.00 15.54 0.00 23.64 2.36 0.00
sda2 0.22 0.89 0.34 0.66 4.45 12.37 2.23 6.18 16.83 0.00 3.42 1.48 0.15
sda3 0.00 7.78 6.50 17.42 51.97 201.71 25.98 100.86 10.61 0.00 0.34 0.01 0.02
sda5 2.21 0.99 1.46 0.78 29.38 14.11 14.69 7.06 19.40 0.00 2.65 1.81 0.41
sda6 0.16 0.02 0.17 0.01 2.61 0.24 1.30 0.12 15.97 0.00 19.21 4.77 0.09
sda7 0.00 0.00 0.00 0.00 0.00 0.04 0.00 0.02 19.35 0.00 3.50 3.09 0.00
sda8 0.15 0.00 0.09 0.00 1.89 0.00 0.94 0.00 20.99 0.00 23.10 2.38 0.02
sdb 4.26 1.50 2.90 1.84 57.29 26.77 28.64 13.38 17.75 0.01 0.99 0.76 0.36
sdb1 4.26 1.50 2.90 1.84 57.29 26.77 28.64 13.38 17.75 0.01 0.99 0.76 0.36
sdc 2.67 14.97 1.96 13.93 36.97 231.19 18.49 115.59 16.88 0.00 0.18 0.18 0.29
sdc1 2.67 14.97 1.96 13.93 36.97 231.19 18.49 115.59 16.88 0.00 0.18 0.18 0.29
sdd 38.75 2.08 29.24 1.63 12.44 29.70 6.22 14.85 1.37 0.00 0.14 0.03 0.10
sdd1 38.75 2.08 29.24 1.63 12.44 29.70 6.22 14.85 1.37 0.00 0.14 0.03 0.10
sde 3.04 4.42 1.86 2.93 39.23 58.79 19.62 29.40 20.44 0.00 1.73 0.49 0.24
sde1 3.04 4.42 1.86 2.93 39.23 58.79 19.62 29.40 20.44 0.00 1.73 0.49 0.24
sdf 12.61 1.67 9.51 2.72 176.94 35.18 88.47 17.59 17.34 0.00 0.50 0.33 0.41
sdf1 12.61 1.67 9.51 2.72 176.94 35.18 88.47 17.59 17.34 0.00 0.50 0.33 0.41
sdg 22.09 6.71 19.70 6.03 334.32 101.95 167.16 50.98 16.96 0.00 0.24 0.18 0.48
sdg1 22.09 6.71 19.70 6.03 334.32 101.95 167.16 50.98 16.96 0.00 0.24 0.18 0.48
sdh 17.39 4.64 10.43 3.94 222.60 68.70 111.30 34.35 20.26 0.00 0.66 0.01 0.01
sdh1 17.39 4.64 10.43 3.94 222.60 68.70 111.30 34.35 20.26 0.00 0.66 0.01 0.01
sdi 25.18 1.21 23.47 1.35 389.22 20.46 194.61 10.23 16.51 0.00 0.20 0.05 0.14
sdi1 25.18 1.21 23.47 1.35 389.22 20.46 194.61 10.23 16.51 0.00 0.20 0.05 0.14
sdj 22.61 7.75 19.22 9.94 334.59 141.48 167.29 70.74 16.33 0.00 0.10 0.03 0.10
sdj1 22.61 7.75 19.22 9.94 334.59 141.48 167.29 70.74 16.33 0.00 0.10 0.03 0.10
sdk 9.67 2.77 5.95 1.51 124.83 34.25 62.41 17.12 21.34 0.00 0.79 0.60 0.45
sdk1 9.65 2.77 5.95 1.51 124.78 34.25 62.39 17.12 21.33 0.00 0.79 0.60 0.45
sdl 24.37 3.64 16.00 0.90 322.99 33.88 161.50 16.94 21.11 0.00 0.25 0.21 0.36
sdl1 24.37 3.64 16.00 0.90 322.99 33.88 161.50 16.94 21.11 0.00 0.25 0.21 0.36
Don’t be scared by this mass of info! You can read the man page for iostat to see what all of the columns are, but for simple diagnostics you only need to look at a few things. Again ignore the first set of results since that’s the total since system start.
- Device : which block device we’re talking about
- rkB/s : KiB/s read from device
- wkB/s : KiB/s written to device
- await : Average time, in ms, that requests completed in the last interval (i.e. 5 secs) took to be completed, including how long they waited and how long they were actually being serviced
- %util : Percentage of the processes’ CPU time that was spent dealing with IO related to this device. When this gets close to 100 then the device is doing all it can, basically.
From the above you can see that /dev/sdi1 is saturated, /dev/sdg1 isn’t too happy either, and that’s the bottleneck on this server. In both cases it is reading rather than writing.
To try and track down which processes are doing the IO you can do a ps awux and look for processes with a “D” in the “STAT” column. This indicates uninterruptible sleep, usually IO-related. If you are lucky you will also be able to use lsof -p to see exactly what they are reading/writing from/to, but usually iostat was specific enough.
Once you know which block devices are suffering, the answer is to buy more/faster disks and put those filesystems on them. Use SoftwareRAID to increase performance and maintain redundancy which is essential with multiple disks since adding more disks is multiplying chances of device failure. See http://strugglers.net/wiki/Linux_RAID_best_practices for an existing diatribe on this subject.
Basic performance diagnostics are really useful even in a home situation where you just want to know where to best spend 200 quid.
Leave a Reply
You must be logged in to post a comment.