Kevin Burton’s NEW FeedBlog

Spinn3r is hiring for an experienced Senior Operations and Infrastructure Engineer with solid Linux and MySQL skills and a passion for building scalable and high performance infrastructure.

This role is about 80% engineering in future infrastructure work (so that we can scale into the future) and 20% routine operational tasks.

It’s a great opportunity for the right candidate. The fact that most of this will be engineering and building new systems should make it a lot more challenging than a traditional operations role.

My goal is to spend the right amount of time doing long term infrastructure work so that fires and machine failures are fully routine and require no human interaction.

Right now they require little human interaction but it’s still required in some of the more serious circumstances.

This IOPS distribution is very interesting.

I’m playing with a RAID array of 5x Intel X-25E drives.

It turns out they need a lot of tuning. I’ll blog about this later.

What is more interesting is this distribution of IOPS across threads.

This is using sysbench and the seqrd file IO test.

My hypothesis is that it’s wear leveling the ext3 block group.

I also tried using ext3 striding and that did yield a performance boost. I’m going to try to use XFS but I’m on a CentOS box or testing.

Anyone have another theory as to what could be causing this?

I can’t wait until RAID is dead.

Update: This problem actually happens on a single 1x Intel X-25E so it seems like a hardware issue.

Update 2: It turns out that this is NOT a bug with the Intel SSD. I’m pretty sure it’s a bug (or misconfiguration on my part) of sysbench. Performing the same tests with ‘dd’ shows that IO scales linearly up until at least 10 parallel sequential reads. So something else is broken here.

It looks like Google and Bing are now paying for a Twitter firehose feed.

What’s interesting here isn’t the deal – what’s interesting is that Google is PAYING for content.

In the past they have refused to pay for content acquisition. Your pay was the traffic that Google sent your way.

Now the NYTimes can come to Google and they have a bit more leverage this time. You’re paying for Twitter content! Hand us over some change for the NYTimes or we’ll cut you off…

My take on the subject is that content providers like Facebook, MySpace, etc should create open networks for their users.

There’s a huge value in creating an open eco-system around content. Creating artificially walled gardens is so CompuServe/AOL. This is Web 2.0. You’re either open or you’re (eventually) dead.

What do you get when you take 7 Intel SSDs and throw them in a desktop? 1M IOPS:

“So as we look at optimizing some of those things,” he said, “like interrupts, driver speculation, improving the physical interface between SSDs, and the system, we expect great gains in power, performance, and cost.”

Gains in performance, for example, that add up to 1M IOPS (input/output operations per second) in Coulson’s lab, where he hooked up a dual Xeon 5500 desktop tower to seven Intel SSD prototypes – four in the tower and three in a PCIe expansion box – and ran a 4K read/write benchmark on it.

“This many I/Os per second is about four gigabytes per second of storage bandwidth,” he claimed. “As a storage guy, that’s a huge number, a very huge number.”

He also pointed out that during the test, the CPU utilization of the tower was about 50 per cent. “That’s really nice,” he smiled.

Mark Callaghan made some progress with 4k pages in and InnoDB. I ran some number on SSD with 8k pages but my 4k build would dump core.

The numbers for 8k show some potential.

The way I see this working is that one would buffer the first and second level btree pages in memory and the rest of the InnoDB database would be served from SSD.

We’re going to be trying to run InnoDB on 4x 64GB Intel SLC SSDs.

The key is that have high sustained reads with a much larger storage capacity.

With 4k pages we’re going to be doing far less SSD reads. However, the SSDs are so fast that reading the extra data might not impact performance significantly.

As an aside. InnoDB should be able to have a runtime reconfiguration of page size. ALTER TABLE FOO page_size=4096 would be nice.

I’m reading more about some of the InnoDB internals as I’m about to start working on a new database on the order of 1-5TB of memory.

I want to make sure everything is tight and well represented on disk.

Here are some random thoughts.

- If the table has a suffix of potentially NULLable columns it should be possible to just omit these in the field definition. It’s a tough trade off though as one would need to use a few bits extra to terminate the field list (or represent the length of the field list).

- The ‘extra bytes’ section could optionally omit the ‘next 16 bits’ pointer to the next page on disk and shave off 2 bytes of storage per record. This might not seem like a lot but I’m trying to store millions of short records per box. These are from 16-24 bits in length. Another 16 bits is a significant data overhead. These records are mostly fixed so being able to omit the next page pointer can really help use memory more efficiently. It seems like there are a few reserved bits that can be used for this purpose.

OCZ has their new PCI flash devices on the market at $3.36 per GB … these guys are FAST.

Read: Up to 500 MB/s-Write: Up to 470MB/s -Sustained Write: Up to 200MB/s

…

They still aren’t commodity just yet but flash on PCI is very interesting.

Another major issue here is the device driver.

If OCZ were to OSS their driver and get it into the Linux kernel I’m sure Debian folks would go this route rather than FusionIO (their driver is still closed source).

If you’ve studied graph theory you may have bumped into The Königsberg Bridge Problem which is essentially the first modern use of Graph Theory.

The city of Königsberg in Prussia (now Kaliningrad, Russia) was set on both sides of the Pregel River, and included two large islands which were connected to each other and the mainland by seven bridges.
The problem was to find a walk through the city that would cross each bridge once and only once. The islands could not be reached by any route other than the bridges, and every bridge must have been crossed completely every time (one could not walk halfway onto the bridge and then turn around to come at it from another side).

Euler proved that the problem has no solution.
To start with, Euler pointed out that the choice of route inside each landmass is irrelevant. The only important feature of a route is the sequence of bridges crossed. This allowed him to reformulate the problem in abstract terms (laying the foundations of graph theory), eliminating all features except the list of landmasses and the bridges connecting them. In modern terms, one replaces each landmass with an abstract “vertex” or node, and each bridge with an abstract connection, an “edge”, which only serves to record which pair of vertices (landmasses) is connected by that bridge. The resulting mathematical structure is called a graph.

… you should probably take a look at the full Wikipedia node. It’s a clever proof.

Anyway. I thought it would be fun to one days visit Königsberg which is now the modern day city of Kaliningrad in Russia.

Then it dawned on me that I could use Google Maps to find it and sure enough it’s still there.

One of the bridges appears to have been removed.

Spinn3r will be at the Real Time Stream Crunchup tomorrow (which should be fun).

There should be more announcements about realtime RSS which is interesting:

While there is an argument to be made that RSS is dying, being replaced by more instantaneous forms of content delivery such as Twitter and other real time streams, many people aren’t quite yet ready to give up on it. Instead, they want to save it by speeding it up. Tomorrow, at our Real Time Stream CrunchUp, we will see three demos of projects that do just that in slightly different ways.

Google engineers Brad Fitzpatrick and Brett Slatkin will show a demo of a new push protocol called pubsubhubub, Netvibes CEO Freddy Mini will demo his similar RSS Instant Update Hub, and WordPress engineer Andy Skelton will show off a Jabber client which uses the XMPP protocol to push blog headlines into an IM-like environment faster than RSS.

If these receive significant adoption Spinn3r will implement them pretty quickly (we push revisions every week).

Gnip also launched their early stage partner program:

Gnip is also launching a early-stage startup partner program that will let startups access to all of Gnip’s service features and data services. The program is aimed towards software development startups that have been in business for less than 3 years and generating less than $200,000 in revenue. Of course, Gnip requires that partners pay a fee of $1000 but says the services that they will receive are valued at $10,000 per month.

Spinn3r has been offering this for more than three years now. If you’re an early stage startup and you need access to Spinn3r data we can do so at a fraction of the price.

I should also note that if you’re a research organization we can provide you with data for free. We provide access to Spinn3r data to more than 100 PhDs from to universities world wide.

We’re hiring a Crawl Engineer at Spinn3r. This is a key hire (and will take a lot of work off my shoulders) so we plan on taking our time to find the right candidate.

That said, it’s an awesome opportunity to get in and work on a rapidly growing startup.

About Spinn3r:

Spinn3r is a licensed weblog crawler used by search engines, weblog analytic companies, and generally anyone who needs access to high quality weblog and social media data.

We crawl the entire blogosphere in real-time, rank, and classifying blogs, as well as remove spam. We then provide this information to our customers in a clean format for use within IR applications.

Spinn3r is rare in the startup world in that we’re actually profitable. We’ve proven our business model which gives us a significant advantage in future product design and expanding our current customer base and feature set.

We’ve also been smart and haven’t raised a dime of external VC funding which gives us a lot more flexibility in terms how how we want to grow the company moving forward.

For more information please visit our website.

Responsibilities:

• Maintain our current crawler.
• Monitor and implement statistics behind the current crawler to detect anomalies.
• Implement new features for customers
• Work on backend architecture to improve performance and stability.
• Implement custom protocol extension for enhanced metadata and site specific social media support.
• Work on new products and features using large datasets.

Requirements and Experience:

• Java (though Python, C, C++, etc would work fine).
• HTML, XML, RSS or Atom.
• HTTP
• Distributed systems.
• Databases (MySQL, etc).
• Algorithm design (especially in distributed systems).
• Ability (and appreciation) for working in a Startup environment.
• Must like cats :)

Ideal:

• Past experience running and working with large crawlers.
• Understanding of IR algorithms (K-means, naive bayes, inverted index compression, etc).
• Experience within the Open Source community.

Questions:

For bonus points. Feel free to answer the following questions in your email:

• You have a live corpus of text and HTML from 25M weblogs. You want to cluster these weblogs into logical communities (tech, politics, entertainment, etc). How (and what algorithm) would you cluster and rank the content within a reasonable time? What computational resources would this require (memory, CPU, network bandwidth, etc).
• You are building a ranking algorithm. This algorithm will execute across a large link graph. How would you store the graph to use the smallest (yet reasonable) amount of computational resources (memory and CPU).