Comments on: Detecting Corrupt Data in MySQL Protocol http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/ You may say I'm a dreamer, but I'm not the only one. Thu, 04 Mar 2010 00:34:59 +0000 http://wordpress.com/ hourly 1 By: Mark Callaghan http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11958 Mark Callaghan Tue, 24 Jun 2008 05:49:25 +0000 http://feedblog.org/?p=1692#comment-11958 We (and many others) suffered from the recently fixed bug that could generate corruption in relay log events. In a few cases, the corrupt relay log events were executable. We will soon add checksums to binlog events to prevent that problem. We have had at least one instance where the SQL sent from client to server was corrupted by the network hardware and TCP checksums did not catch the problem. Checksums in the client-server protocol are a great idea, but I am not sure how extensible the client-server protocol is. It is much easier for me to change server-server protocols than client-server protocol as there are at least two implementations of client-server (C library, JDBC). We (and many others) suffered from the recently fixed bug that could generate corruption in relay log events. In a few cases, the corrupt relay log events were executable. We will soon add checksums to binlog events to prevent that problem.

We have had at least one instance where the SQL sent from client to server was corrupted by the network hardware and TCP checksums did not catch the problem. Checksums in the client-server protocol are a great idea, but I am not sure how extensible the client-server protocol is. It is much easier for me to change server-server protocols than client-server protocol as there are at least two implementations of client-server (C library, JDBC).

]]> By: Kevin Burton http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11957 Kevin Burton Tue, 24 Jun 2008 01:51:45 +0000 http://feedblog.org/?p=1692#comment-11957 Kevin, Holy crap.... that's a crazy story. I've heard similar horror stories before about binlog corruption from other companies. From the number of comments this post is getting so quickly it sounds like a lot of people are thinking about this.... Anyone have any other horror stories? Kevin,

Holy crap…. that’s a crazy story.

I’ve heard similar horror stories before about binlog corruption from other companies.

From the number of comments this post is getting so quickly it sounds like a lot of people are thinking about this….

Anyone have any other horror stories?

]]> By: Kevin Marks http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11956 Kevin Marks Tue, 24 Jun 2008 01:48:15 +0000 http://feedblog.org/?p=1692#comment-11956 We had at least one scary instance of this at Technorati - an UPDATE writing to the blogs table got terminated before its WHERE clause, thereby writing the same blog title to an entire database shard (at the time, a quarter of the blogs in the index). That the blog title was something like "Huggable, kissable, loveable" made it amusing, as every results page showed that for a quarter of the results. We managed to propagate real blog titles back from memcached, slaves and other caches, but it was a very stressful day. We had at least one scary instance of this at Technorati – an UPDATE writing to the blogs table got terminated before its WHERE clause, thereby writing the same blog title to an entire database shard (at the time, a quarter of the blogs in the index). That the blog title was something like “Huggable, kissable, loveable” made it amusing, as every results page showed that for a quarter of the results. We managed to propagate real blog titles back from memcached, slaves and other caches, but it was a very stressful day.

]]> By: Xaprb http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11955 Xaprb Tue, 24 Jun 2008 01:41:31 +0000 http://feedblog.org/?p=1692#comment-11955 I've been bitten so badly by this problem (I suspect, but can't prove -- see the bug report mentioned above) that I'd almost settle for a magic number every so many bytes. 0xDEADBEEF, anyone? The fact that the protocol is unchecked has probably cost a lot of money over the years. I’ve been bitten so badly by this problem (I suspect, but can’t prove — see the bug report mentioned above) that I’d almost settle for a magic number every so many bytes. 0xDEADBEEF, anyone? The fact that the protocol is unchecked has probably cost a lot of money over the years.

]]> By: Arjen Lentz http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11954 Arjen Lentz Tue, 24 Jun 2008 00:23:11 +0000 http://feedblog.org/?p=1692#comment-11954 There should be a CRC at over protocol blocks. CRC-32 would work fine and doesn't require much computation. CRC is specifically designed to pick up bitflips, missing bytes and the like. It simply checks whether a single block was transmitted intact. MD5/SHA1 is complete overkill for this purpose. It was designed to allow signatures to be compared against eachother, equivalent to the original datasets. That's a whole different issue. There should be a CRC at over protocol blocks. CRC-32 would work fine and doesn’t require much computation. CRC is specifically designed to pick up bitflips, missing bytes and the like. It simply checks whether a single block was transmitted intact.

MD5/SHA1 is complete overkill for this purpose. It was designed to allow signatures to be compared against eachother, equivalent to the original datasets. That’s a whole different issue.

]]> By: burtonator http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11953 burtonator Mon, 23 Jun 2008 22:19:03 +0000 http://feedblog.org/?p=1692#comment-11953 Harrison, I agree it's less common but becoming more common as switches go commodity. Also, some of this stuff isn't zero copy so memory corruption can come into play.... A hashcode would nail this stuff but only if it wasn't in the last memcpy in the target MySQL box...... Kevin Harrison,

I agree it’s less common but becoming more common as switches go commodity.

Also, some of this stuff isn’t zero copy so memory corruption can come into play….

A hashcode would nail this stuff but only if it wasn’t in the last memcpy in the target MySQL box……

Kevin

]]> By: Morgan Tocker http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11951 Morgan Tocker Mon, 23 Jun 2008 20:47:45 +0000 http://feedblog.org/?p=1692#comment-11951 And it does come into play... http://bugs.mysql.com/bug.php?id=25737 It will be a whole lot worse with row based replication, since simple bit flips may still leave the replication events still syntactically correct. And it does come into play…
http://bugs.mysql.com/bug.php?id=25737

It will be a whole lot worse with row based replication, since simple bit flips may still leave the replication events still syntactically correct.

]]> By: Harrison http://feedblog.org/2008/06/23/detecting-corrupt-data-in-mysql-protocol/#comment-11950 Harrison Mon, 23 Jun 2008 20:32:30 +0000 http://feedblog.org/?p=1692#comment-11950 I suspect that since the MySQL protocol is normally used over the local LAN it would be much less common than something like S3 (since it is always over a WAN). I wonder how much enabling compression or SSL would help to check for this sort of thing. I suspect that since the MySQL protocol is normally used over the local LAN it would be much less common than something like S3 (since it is always over a WAN).

I wonder how much enabling compression or SSL would help to check for this sort of thing.

]]>