Kevin Burton’s NEW FeedBlog

I was just looking over Techhnorati’s post which includes the total number of posts per day.

Technorati points out there are 900k blog posts per day:

As you can see by the black trend line, posting volume has followed a strong upward trend. After a brief dip last winter, the average rate of postings has grown steadily such that at the end of July 2005, there were about 900,000 posts created each day. That’s about 37,500 posts every hour, or 10.4 posts per second. It peaked at just over 1.1 million posts per day after the Live 8 concerts and Justice Sandra Day O’Connor announced her resignation from the US Supreme Court.

How much data is this? If we assume that the average HTML post is 150K this will work out to about 135G. Now assuming we’re going to average this out over a 24 hour period (which probably isn’t realistic) this works out to about 12.5 Mbps sustained bandwidth.

Of course we should assume that about 1/3 of this is going to be coming from servers running gzip content compression. I have no stats WRT the number of deployed feeds which can support gzip (anyone have a clue?). My thinking is that this reduce us down to about 9Mbps which is a bit better.

This of course assumes that you’re not fetching the RSS and just fetching the HTML. The RSS protocol is much more bloated in this regard. If you have to fetch 1 article from an RSS feed your forced to fetch the remaining 14 addition posts that were in the past (assuming you’re not using the A-IM encoding method which is even rarer). This floating window can really hurt your traffic. The upside is that you have to fetch less HTML.

Now lets assume you’re only fetching pinged blogs and you don’t have to poll (polling itself has a network overhead). The average blog post would probably be around 20k I assume. If we assume the average feed has 15 items, only publishes one story, and has a 10% overhead we’re talking about 330k per fetch of an individual post.

If we go back to the 900k posts per day figure we’re talking a lot of data – 297G most of which is wasted. Assuming gzip compression this works out to 27.5Mbps.

Thats a lot of data and a lot of bloat which is unnecessary. This is a difficult choice for smaller aggregator developers as this much data costs a lot of money. The choice comes down to cheap HTML index ing with the inaccuracy that comes from HTML or accurate RSS which costs 2.2x more.

Update: Bob Wyman commented that he’s seeing 2k average post size with 1.8M posts per day. If we are to use the same metrics as above this is 54G per day or around 5Mbps sustained bandwidth for RSS items (assuming A-IM differentials aren’t used).

Update: Slashdot picked this up and there are a few issues I want to correct. The number of posts is 900K not 9M. Not sure how this was introduced into the post. The other issue is that the 150K post is for the full HTML post not the RSS item length which a lot of people seem to be missing. I should have been more clear as this aspect can be a bit confusing.