The UNIX iostat command (along with vmstat) is a great tool for finding bottlenecks in your disk subsystem.
The ‘tps’ field is especially useful since you can see the unique reads/writes to your disk which usually map to disk seeks (which are a performance killer).
The problem is that there are no simple tools for monitoring the iostat tps across all of your servers.
So late last week I threw one together by groking the source of iostat.
Apparently, it just reads the value of /proc/diskstats and compares the new values with the previous sample.
It took me about 10 minutes to throw together a munin plugin to measure the performance across all of our servers.
Since our disks can handle about 300 tps without being overwhelmed we’re well within operating limits.
However, this is interesting:
This is the tps on a server running MyISAM and long standing queries. As the table fills up the performance starts to fail and significantly fall over.
It’s going to be interesting to see these numbers increase over the next week.