We know there are eight fallacies of distributed computing – but I think there’s one more.
A physical machine with multiple cores and multiple disks should not be treated as one single node.
Two main reasons.
First RAID is dying. You’re not going to get linear IO scalability by striping your disks. If you have one software component accessing the disk as a logical unit you’re going to get much better overall system performance.
Second. Locking. Multicore mutex locks are evil. While the locking primitives themselves are getting faster, if the CPU is overloaded and long critical section is acquired your other processes are going to have to wait to move forward. Running one system per core fixes this problem.
We’re seeing machines with eight cores and 32G of memory. If we were to buy eight disks for these boxes it’s really like buying 8 machines with 4G each and one disk.
This partially goes into the horizontal vs vertical scale discussion. Is it better to buy one $10k box or 10 $1k boxes? I think it’s neither. Buy 4 $2.5k boxes. The new multicore stuff is super cheap.
Update: Steve Jenson sent me an email stating that this could be rephrased to consider avoid considering physical units of computation as logical units.