This is what I said just a few hours ago:

If you want to use them at SCALE or have a serious application (like Apple) these companies want you to execute custom terms of service to prevent you from getting kicked off their platform at some point in the future.

…

And Kara Swisher notes:

According to sources familiar with Facebook’s platform, the social networking giant essentially denied Apple’s Ping access to application programming interfaces that would allow it to search for an iTunes user’s friends on Facebook who also had signed up for Ping.

Of course if Apple were to DoS the resources of Facebook this could mean that Facebook needed to block them which Kara notes:

That is, unless some entity wants to access it a lot. In that case, Facebook requires an agreement for reasons primarily centered on protection of Facebook user data and, of course, infrastructure impact.

I think it would be fair for Facebook to charge for access, but these should be open and clear cut form day zero. Facebook has the right to do what they want with their platform of course but as developers we have the right not to code to if it the API terms are too aggressive (and this looks like the case for Apple).

We’re hearing reports of people who had access to a Facebook Connect feature in Ping earlier, which didn’t work, and has since been removed. So it looks like Apple really did pull Facebook support for Ping very late in the game. It’s still even mentioned on the Ping promo page. (And, via Peter Kafka, here’s a link to the Ping app on Facebook, which doesn’t currently seem to work.)

This is exactly why the social web needs to be fully open. Right now you don’t need to execute a terms of service to index the web, to publish HTML, to link to websites. But you DO need to agree to a terms of service to use Twitter, Facebook, etc.

If you want to use them at SCALE or have a serious application (like Apple) these companies want you to execute custom terms of service to prevent you from getting kicked off their platform at some point in the future.

You’re basically a share cropper. You are making money while working on someone else’s property. If they want to charge you EXCESSIVE rent you’re either forced to pay it or you have to go out of business.

Not fun.

This is why protocols like Salmon, PuSH, OAuth. and Status.net are so important.

Hopefully in a few years we look back at this situation and laugh at how silly it was – WAY back in 2010 when the social web was proprietary.

Which is super cool of course.

It’s cool that Facebook is crawling the web.

When does Facebook scrape my Page? “Even if you specify a longer time, Facebook will scrape your page every 24 hours.”

The user agent of the scraper is: “facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php

However, you can’t crawl Facebook. They’re blocking everyone by default (via robots.txt) still.

Unless you’re one of the big boys, you can’t play with Facebook unless you’ve been explicitly whitelisted.

It’s not really the end of the world until you realize that the ENTIRE web would melt if everyone required explicit permission from millions of websites just to crawl the web.

It’s a beg for permission and not a beg for forgiveness model which doesn’t scale.

Two winters ago I left a position as a system administrator that was paying pretty well and moved cross-country to a region with less jobs than where I moved from. Three months later, I was still unemployed, broke, and bored. I was talking to my good friend Japhy on IRC one day and he was explaining to me how the tf-idf algorithm works. For reasons involving boredom more than any other reason, I dreamed up an idea: I would write software that would take a given document and generate book suggestions based on its content.

I think that most programmers would agree with me that we put in longer hours on code when we’re not working for anybody. We don’t stop learning, either. To us, unemployment is a brief sprint of academia spent in our home office, the local coffee shop, or our parent’s house.

My imagination dreamed up this fairly straightforward process:

Take a given document and calculate tf-idf scores on all terms
Select X number of the highest scoring terms
Pass these high-scoring terms to an Amazon ItemSearch query
Receive a list of recommended books (with URLs) from Amazon

Spam on Twitter is becoming more of a problem due to the nature of @replies being very low barriers to entry. There are numerous measures to help block this but it’s still becoming a problem.

The algorithm described in the research is unique in that it incorporates what the authors call “passivity.”
 
The study found that a large majority of Twitter users act as passive information consumers and rarely forward (“retweet”) content to the network.  To become influential, users must not only catch the attention of their followers; they must also overcome their followers’ predisposition to remain passive.

There are other ways to compute this data. Retweets and @ replies can also yield results for finding influential users.

We (the Hypertable Development team) will be presenting the recently developed Hypertable Hive extension. In this presentation, we will give an overview on how to use it and will provide details on the design.

We will also be discussing the results of the recent Hypertable vs. HBase Performance Evaluation. Hypertable’s relative performance numbers were very impressive. We will describe the test in detail and hold a Q&A session after the presentation.

Much of LinkedIn’s important data is offline – it moves fairly slowly. So they use daily batch processing withHadoop as an important part of their calculations. For example, they pre-compute data for their “People You May Know” product this way, scoring 120 billion relationships per day in a mapreduce pipeline of 82 Hadoop jobs that requires 16 TB of intermediate data. This job uses a statistical model to predict the probability of two people knowing each other. Interestingly they use bloom filters to speed up large joins, yielding a 10x performance improvement.

They have two engineers who work on this pipeline, and are able to test five new algorithms per week. To achieve this rate of change, they rely on A/B testing to compare new approach to old approaches, using a “fly by instruments” approach to optimize results. To achieve performance improvements, they also need to operate on large scale data – they rely on large scale cluster processing. To achieve that they moved from custom graph processing code to Hadoop mapreduce code – this required some thoughtful design since many graph algorithms don’t translate into mapreduce in a straightforward manner.

The workflow management systems are starting to become interesting as jobs with this many steps and intermediate data can become confusing quickly.

We offer public and private source code hosting to companies and open source projects using either git or Subversion. What we want to do is lower the barrier of contributing to projects, both public and private. Submitting a patch to an open source project should be about the code, not the process of submitting the patch. Working with your coworkers, either in the same office or across the world, should be about moving your project forward and not about managing clumsy tools.

We also offer git training, provide git learning materials, and sponsor open source projects.

It’s so cool when people can control their own destinies.

I am personally and professionally concerned about their emergence at scale and the negative impacts they could have on our ecosystem and it’s probably time I spoke up.

More specifically, I’d like to touch on and expand on some things that DeWitt mentions.

Potential censorship

Having a central point for URL redirection of such a vast number of URLs means that potential censorship issues can arise.

What happens if Facebook wants to build a search engine and bulk URL resolve against a 3rd party URL redirection service?

What happens if the remote system can’t keep up with the performance requirements for bulk URL resolution?

We’ve already seen this with bit.ly. For a feature we were evaluating at Spinn3r we weren’t able to bulk resolve bit.ly URLs as fast as they were being created.

Scale and Single Point of Failure

This puts a large number of the URLs in the hands of one player and makes their scale requirements higher. It’s even more frustrating that this didn’t need to happen in the first place since the web worked pretty well without URL redirection services to begin with.

140 characters not required for uplevel clients

What I think is most frustrating is that the use of tiny URLs isn’t required for most clients.

Only downlevel SMS clients that can only support HTML and have a firm 140 character limit need to use the tiny URL.

For uplevel/advanced clients they can just render HTML with either the the tiny URL as the href and the long URL as the text of the link OR they can use the long URL in both the text and the href and then add an onClick handler and record the click in Javascript.

I would strongly recommend that Twitter adopt this second model. This would still give them most of the benefit from their URL tracking and tiny URL service but at the same time preserve the machine readability of the web and avoid all the pitfalls of these services.