I’m pretty happy so far with Percona and the latest InnoDB changes.

Compression wasn’t living up to my expectations though.

I think the biggest problem is that the compression can only use one core in replication and ALTER TABLE statements.

We have an 80GB database that was running on 96GB boxes filled with RAM.

I wanted to try to run this on much smaller instances (32GB-48GB boxes) by compressing the database.

Unfortunately, after 24 hours of running an ALTER TABLE which would only use one core per table, the SQL replication thread went to 100% and started falling behind fast.

I think what might be happening is that the InnoDB page buffer is full because it can’t write to the disk fast enough which causes the insert thread to force compression of the pages in the foreground.

Having InnoDB only use one core / thread to compress pages seems like a very bad idea (especially on 8-16 core boxes, I’m testing on an 8 core box now but we have 16 core boxes in production).

The InnoDB page compression documentation doesn’t seem to yield any hints about when InnoDB pages are compressed and in which thread. Nor does there seem to be any configuration variables that we can change in this regard.

Perhaps a ‘compressed buffer pool only’ option could be interesting.

This way InnoDB does not have to maintain an LRU for compressed/decompressed pages. Further, it can read pages off disk, decompress them, and then leave the pages decompressed in a small buffer. Then a worker thread (executing on another core) can compress the pages and move them back into the buffer pool where they can be stored and placed back on disk.

This process could still become disk bottlenecked but at least it would use multiple cores.

For example, say you want to take a 1GB file and page through it 10MB at a time.

With a flat file I could just read 10MB off disk, read the next 10MB, etc.

This would be amazingly efficient as you could write the data sequentially, without updating indexes, and then read the data back out by byte offset.

You could page through MySQL tables by adding an index to a table:

SELECT * FROM FOO WHERE PTR > 10000 LIMIT 10000;

for example … but I REALLY want to avoid an index step because it is not cheap and only required since MySQL doesn’t support cursors.

This index slows down the import stage and I have to buffer the index data in memory which is just a waste.

I could use LIMIT with OFFSET but this isn’t going to be amazingly efficient because it will either require us to use a temporary on disk table or force us to read each row off disk.

Technically, one could implement offset efficiently with a fixed with table since you can compute the byte offset as row_width*N but I don’t think MySQL implements this optimization.

Further, my tables are are variable width.

If MySQL supported cursors then I could just tell MySQL to perform a table scan and read the data from a cursor.

I’m implementing something similar to a Map Reduce job in MySQL and Java which would work VERY well on Hadoop but is nearly impossible to implement with MySQL efficiently since I can’t page through data without over-indexing my tables.

Thoughts? I would love it if there’s some stupid solution that I’m just missing.

I suspect this will become more and more of a problem as more developers want to just flat out Map Reduce their data.

Update:

I was wrong. The solution is HANDLER. Ryan Thiessen nailed it in the comments.

This would be on Wed 10/14 2009 at 7pm … at 580 Howard Suite 301 (Spinn3r HQ)

Oracle owns MySQL, InnoDB, etc so I suspect a lot of Oracle people and MySQL hackers will be interested in attending more of an Open Source and community centered meetup.

We’ll just be hanging out at our offices … we’ll have beer and food.

Feel free to bring your laptops as we have Wifi :)

This is contingent on at least 10 RSVPs as I want to make sure there is interest from the community.

Please RSVP here

It’s a bit late notice so if you could help spread the world by blogging about this that would be GREAT!

We’re building out a new DB that uses very small records (around 32-64 bytes) so reading a whole 16k for this record should have a performance difference.

I haven’t seen any benchmarks on 16k random read IOPS on the Intel SSD but my hunch is that there will be a 20-30% penalty here.

Though even if it was a 4x penalty that would still be about 9k transactions per second which is pretty good.

On a personal note I just bought a new Mac Book Pro which will be upgraded to the Intel X-25M MLC SSD.

Needless to say I’m very excited!

You should obviously have MySQL experience. Love SQL, hate data corruption and slow queries, and preferably live in San Francisco.

Linux experience would be nice as well but not required.

Extra points if you are excited about SSD, *huge* amounts of data, have hacked on Drizzle or XtraDB

Spinn3r is a GREAT place to work. We’re growing fast and have cool new offices in SOMA.

We’re looking at deploying the Intel X-25M MLC SSD in production.

The problem being that this drive has a lower number of erase cycles but is much cheaper. Than the Intel X-25E SLC drive.

However, in our situation we’re write once, read many. I’m 99% certain that we will not burn out these drives. We write data to disk once and it is never written again.

The problem is that I can’t be 100% sure that this is the case. There is btree flushing, and binary log issues that I’m worried about…

What would be really nice is an API (SMART?) that I can enumerate the erase blocks on the drive, determine the max erase cycles, and read the current number of erase cycles.

This way, I can put an SSD into production, then determine the ETA to failure.

I can also add this to Nagios and Ganglia and trend the failure date and alert if the derivative is too high and the drive will soon fail.

Further, I can figure out if a database design is flawed. If I deploy a new database into production and the failure ETA is too high after 24 hours I know that something is wrong. Either a misconfiguration or a problem with the design.

I think this would solve a LOT of the problems with deploying SSD in enterprise environments. (MySQL, Oracle, etc)

My older post on this subject still applies for requirements.

If you’re a Linux or MySQL geek we’d love to have your help.

Did I mention we just moved to an awesome office on 2nd and Howard in downtown SF?

There is are two threads which are for the most part, unaware of each other.

One thread reads the remote binary logs, and the other writes them to disk.

While the Linux page buffer CAN work to buffer these logs, the first write will cause additional disk load.

One strategy, which could seriously boost performance in some situations, would be to pre-read say 10-50MB of data and just keep it in memory.

If a slave is catching up, it could have GIGABYTES of binary log data from the master. It would then write this to disk. These reads would then NOT come from cache.

Simply using a small buffer could solve this problem.

One HACK would be to use a ram drive or tmpfs for logs. I assume that the log thread will block if the disk fills up… if it does so intelligently, one could just create a 50MB tmpfs to store binary logs. MySQL would then read these off tmpfs, and execute them.

50MB-250MB should be fine for a pre-read buffer. Once one of the files is removed, the thread would continue reading data.

When InnoDB needs more space it auto-extends the current data file by 8MB. If this is writing out zeros to the new data (to initialize it) then using fallocate() would certainly be faster.

Apparently, XFS supports this too but needs an ioctl. XFS could support fallocate in the future as well…

Basically, the idea is that extremism is an evil and often ideological perspectives are non-optimial solutions.

The Dalai Lama has pursued a middle path solution to the issue of Tibetan independence.

The two opposing philosophies in this situation are total and complete control of Tibet by the Chinese or complete political freedom by the Tibetan people and self governance. The Dalai Lama proposes an alternative whereby China and Tibet pursue stability and co-existence between the Tibetan and Chinese peoples based on equality and mutual co-operation.

How does this apply to Linux?

The current question of swap revolves around using NO swap at all or using swap to page out some portion of your application onto disk.

Even with the SplitLRU patch we’re still seeing problems with Linux swap.

The entire question is Mu…

The solution isn’t to disable swap. The solution is to use a SMALL amount of swap (100-200MB) and monitor your application to detect when it is swapping and then tune back the memory usage of your application.

The difficulty is that you often want to use the maximum about of memory in the system. Imagine a theoretical system that efficiently uses 100% of system memory. An overcommitted application might allocate a BIT more and cause the OOM killer to kick in.

This would be horrible. Instead, why not just page 10-20 bytes.. It’s not going to have devastating effects on the system and if you’re monitoring it you can decide to tune back your memory usage by 100MB or so.

We have this happen from time to time with MySQL. We allocate slightly too much memory only to have the OOM killer kick in and kill MySQL – not fun.

Using a large swap file isn’t the solution either. If you overtune you can swap out a large portion of your application. This can essentially yield a denial of service as the load on your server becomes so high that you can’t even SSH in after hours of usage. The only solution is to reboot the box.

Using a small swap file avoids this. If you’re swapping more than say 50MB you can have your monitoring system kick in and alert you before the limit is hit and your OOM killer kicks in.

We’re experimenting with this idea this week and I’m optimistic about the results.