Ouch. Check this out.
http://news.bbc.co.uk/1/hi/world/asia-pacific/4605202.stm
http://news.bbc.co.uk/2/hi/world/asia-pacific/4605202.stm
Same document. Same content. Just different layout in the navbar. This is a bad way to do A/B testing. They need to do this based on cookies!
It’s always amazing to find these little edge conditions. I’ve been writing robots/spiders for years and it never ceases to amaze me how much crazy stuff people will put into production.
-
Spinn3r News- Spinn3r at Real Time Stream Crunchup, RSS Speedups, and Startup Discounts
- Spinn3r Hiring Crawl Engineer
- Spinn3r 3.1 - Now with Twitter Support and Social Media Ranking
- Blogfa.com (Iranian Blog Network) Down Due to Iranian Protests and Civil Unrest?
- Papers using Spinn3r Data from ICWSM 2009
- Spinn3r talk from ICWSM 2009
- Spinn3r 3.0: New Features, New Architecture, New APIs - More Goodness
- Using Spinn3r to Track Swine Flu
- Researchers Using Spinn3r
- NYTimes on Crawling the Deep Web
-
Subscribe
-
Flickr
More Photos
January 13, 2006 at 1:01 pm
Kevin,
AFAIK, the Beeb has done this since time immemorial – it is not new.
January 13, 2006 at 1:22 pm
Thanks Matthew. You learn something new everyday. I guess I’ll have to write a custom URL canonicalizer for them. There are a few other mainstream sites I’ve had to do this with as well.
Kevin
January 16, 2006 at 12:56 am
What is so bad with two URLs identifying one resource?
January 16, 2006 at 2:13 am
Doing it with cookies would also mean that the non-cookie’d person could ever see the other format, that’s not exactly a good thing.