I’ve been looking at replacing Munin with our own higher level proprietary monitoring system for keeping track of cluster-wide statistics. This is needed for a new feature we’re trying to ship with Spinn3r so that we can expose some of our internal statistics to our customers. We weren’t able to do this before because our [...]
Categories: clustering
Spinn3r is hiring for an experienced Senior Systems Administrator with solid Linux and MySQL skills and a passion for building scalable and high performance infrastructure. About Spinn3r: Spinn3r is a licensed weblog crawler used by search engines, weblog analytic companies, and generally anyone who needs access to high quality weblog data. We crawl the entire [...]
Categories: clustering,linux,mysql,search,spinn3r
It dawned on me that if I were working for Twitter that I would just assume the service is down unless told otherwise. This lead to the conclusion that one should invert monitoring to send off a notification when Twitter is online … Seriously. I like those guys but this is getting kind of embarrassing. [...]
Categories: clustering,search
I’m going to be migrating to using ZooKeeper within Spinn3r for a myriad of reasons but this one is especially powerful. One could use ZooKeeper to configure external monitoring systems like Munin and Ganglia. ZooKeeper enables this with its support for ephemeral files. If you have an external process like a webserver, database, robot, etc [...]
Categories: clustering,java,mysql
We’ve had our SSDs in production for more than 72 hours now. We’ve had them in a slave role for nearly a week but they’ve now replaced existing hardware including the master. The drives are FAST. In our production roles they’re reading at about 45MB/s and writing to disk at about 15MB/s and using only [...]
Categories: clustering,linux
I just finished watching the Disk is the New RAM video which a number of bloggers have been talking about. If you’re lazy like me you can just read this blog post to get a nutshell on this theory of computing. FYI, I transcoded it in my scalecast podcast if you want to watch it [...]
Categories: clustering
Fully RAM based databases are being used in more and more places. For a lot of use cases throwing ALL of your data into memory will have a major performance benefit. But when should you use RAM vs SSD? RAM is about $100/GB. SSD is about $30/GB. SSDs have a finite performance of about 100MB/s [...]
Categories: clustering,linux,mysql,SSD
Process snipers are used to kill errant processes I’ve used process snipers and watchdog’s to handle realistic process management in large clusters in the past but never felt any of them were very elegant in terms of code simplicity. It dawned on me the other day that this would work out perfectly: for proc in [...]
Categories: clustering,linux
I need to give it a bit more thought but it looks like we’re going forward with deploying Spinn3r on SSD. Specifically, machines with 3 SSDs on Linux software RAID. The performance of SSDs is nothing short of astounding. When tuned correctly these drives were nearly 10x the performance of the same box running RAID. [...]
Categories: clustering,linux,mysql,SSD
Apparently, the entire computing industry is stumped by the multi-core problem. Specifically, scaling single threaded code to run across multiple cores: Both AMD and Intel have said they will ship processors using a mix of X86 and graphics cores as early as next year, with core counts quickly rising to eight or more per chip. [...]
Categories: clustering