DimitriK's (dim) Weblog

MySQL Performance: Percona Live & Few Perf Stories..

Thu, 30 May 2013 21:14:23 +0200

It's now ~~two~~.. ~~three~~.. four weeks since Percona MySQL Conference was finished.. - and there are still many things and new ideas running in my mind inspired by this great MySQL tech + community event (regardless many public holidays and spring vacations in May (which is also the reason why I'm late with this article ;-)). There was a lot of things to learn from others, tons of stuff to share, and as well enjoy long and deep discussions with many MySQL users/ hackers/ gurus/ mentors/ devs/ etc.. etc.. We were very proud to speak about MySQL 5.6 GA during this conference, and also present the first labs release of the future MySQL 5.7 version (and you may already download and play with it, or compile yourself from sources which are always here).. - don't miss presentation made by Tomas about all MySQL Innovation happening today @Oracle! And if MySQL 5.6 for today is the best MySQL release ever, you can be sure we will yet more surprise you with MySQL 5.7, just trust me ;-))

My talk during this conference was about MySQL 5.6 Performance, and I'd like to thank all who attended it - thank you for your interest in MySQL and in what we're doing! There is yet more to come, so stay tuned! ;-) My special thanks to Domas who was seating in the first line and making me laugh all the time (while I'm usually never taking things too seriously - with Domas it was yet more easier ;-)). Well, you may find slides of my presentation on Percona site or directly from here..

As promised, I've also polled every person I've discussed with about most important MySQL Performance problems they are meeting currently on their production or dev environments.. - the results were very surprising:

For majority of people their own "dev team" creating the most of their performance troubles ;-))
Then there are many people having actually performance issues, but they are just not aware about ;-) -- my only advice here is very simple: use a right monitoring tool to see what is going on your system and your MySQL server..
And there are many as well who are taking care about everything, so speaking to them I saw all our future problems, and what is great that we are already working on it ;-)

So, again, thanks to all! It was really amazing and very productive time! Hope to see you all again during MySQL Connect as well (and next Percona Conferences of course ;-))

Then, during any MySQL Conference it's really fun to collect many true stories related to MySQL history (specially from old MySQLers).. - it helps to better understand the whole story, who is right and who is wrong, and who you should blame and who you shouldn't.. - piece by piece you're obtaining a full picture of this huge puzzle.. Probably one day someone will take a time to write a book based on such memories and for sure it'll be a great and teachable reading ;-))

But well, let's stay focused on MySQL Performance, which is my primary interest ;-)) and it'll be pity to not to continue started recently a series of fun stories around MySQL, so I'd like to add also few more ones from me.. - and you'll see it's not an off-topic ;-))

MySQL Black Horse

The story is starting on the time of MySQL 5.4 (the improved performance release of joined efforts from MySQL and Sun teams).. On that time there was not yet any useful instrumentation implemented within MySQL/InnoDB code, and finding and fixing sources of contentions and bottlenecks was yet more difficult task than today (thanks to Performance Schema, since version 5.5 MySQL internals became much more transparent ;-)) -- but on time of MySQL 5.4 such a finding was mostly based on the code analyze made by perf engineers, their expectations of "potential impact" of one or another code part, and then a patch with proposed "fix".. "Fixed" binary was then tested on various benchmarks to see if the effect was positive or not.. - and believe me, there was many right stuff rejected just because the test result was negative (while in fact fixing of one contention simply making sometimes another contention(s) more hot, making an overall result worse.. - while if we continue to dig in the same direction, we may fix further contention(s) as well, and finally fix a whole part of a problem!) - but on that time only "one shot" changes were accepted, and then again rejected, because a combination then of several supposed to be "right fixes" produced a negative result on benchmarks again (for a big surprise of all of us... ;-))

So, during that time we progressed "eyes widely closed" ;-)) and then, after some time of reflection, Mikael started to produce so called "perf builds" - tarballs of MySQL source code with various patches applied on it. There was in total over a hundred of perf builds, one more different than others.. - and I was happy to test them all on SPARC servers to compare and analyze them as in depth as on this time it was possible (we expected a lot from DTrace, but there was yet a need to add probes within the code to analyze "home made" lock and sync solutions implemented within Innodb)... So, comparing, analyzing, re-thinkingn and analyzing again was our "favorite" activity those days ;-))

The traces with test results on most of such perf builds you may still find within my blog articles during 2009 year. There were many builds the results I've published and they mostly looked promising... - But there was a one "special" perf build which results I've never published.. - on my tests it was way better in performance comparing to 5.4, but was rejected due some "unexpected behaviors".. I've called it "black horse" in my internal reports, and always tested it against any other proposed improvements in MySQL code. Even over a time we started to forget what exactly changes were applied within this build, I was still expecting than one day it'll drive us in a right direction ;-))

And this day happened! - it was during 5.5.4 development ;-) On that time MySQL and InnoDB code contained all kind of "goodies" supposed to improve performance a lot, but in fact it was still worse than our "famous black horse".. ;-)) - and once again we were out of ideas, I came back with "black horse" results and profiling pattern.. And this made Mikael thinking hard to try to remember what exactly we finally changed long time ago... - which is then resulted in impressive performance improvements we demonstrated on MySQL 5.5 ;-))

But what is amazing in the story that MySQL 5.5 is simply reached the "black horse" level.. - while only since MySQL 5.6 it was finally out-passed truly far away! ;-))

The very first NDB Benchmark

Reading all recent MySQL Cluster related articles, specially by Stewart and Mikael, remaining me the story which was not mentioned by no one until now (while Mikael was, probably, the most directly concerned ;-)) The story was starting in February 2002 when the "MySQL Cluster" name was not yet even pronounced, and there was just "NDB Cluster" software developed by Alzato.. They came to Sun Benchmark Center to test it and demonstrate that NDB is able to reach 1M TPS (yes, 1M transactions per second!) on Read-Only workload and over 200K TPS on Read-Write.. It was yet 2002, and our biggest SPARC server on that time had only 72CPUs running on 900Mhz.. - and they planned to use specifically this one. However, regarding these high ambitions, all this testing looked like a full disaster.. But we wanted to help them as much as we can, so finally this testing arrived in my hands.. But I was yet far to imagine all dramatic events waited on me ;-))

So far, they started to install their NDB "software".. - and I'm calling it "software", because at this stage it was yet far from something that you'd call a "product".. - while in Benchmark Center I've got a habit to meet customers having something already properly working and mainly coming to understand their bottlenecks, to tune, and then improve yet more better a working code (or simply get an idea about HW sizing, etc..) - but the case of NDB was probably the "most fun" I've ever seen until that time.. - the code was so poor that the problem #1 initially was just to get it compiled properly! and then to avoid to crash with a core dump on a simple probe test.. ;-))

My disappointment was so big that it was impossible to hide it.. I've finally expressed it to Alzato by explaining that we're wasting our time debugging a crappy code instead to leave this powerful HW to other customers which are, opposite to NDB, having an already working stuff and just waiting for a platform availability to test it..

I'm skipping here all dramatic moments of the story, and will just say you that the most of NDB future was played here.. And a probability that there will be no future at all was really very high.. ;-))

But well, since the first days of our common work I've started to add into NDB code various event counters and collect live stats from NDB internals via my dim_STAT monitoring tool (which I've always used and continue to use it right now).. With such a monitoring we were able then to fix all crashes and core dumping first, and then understand all the bottlenecks present in the code!.. - combined with long discussions about NDB design and why the proposed code may work or will never work ;-)

Here is a historic photo where yet 11 years younger Mikael trying to break my arguments why the NDB design is far from optimal ;-)

So, step by step, day by day we were able to improve performance more and more..

Till the Day-D when we finally made a huge jump in performance and reached 1.5M TPS(!) on Read-Only workload! - still not yet using all available CPUs on the server.. - obtained TPS level was so high (specially for those days) that even for me it was hard to believe it if I did not add my own counters within NDB code by myself ;-))

Here is the famous snapshot with dim_STAT graphs reflecting the reached TPS level:

To be sure the result is not random and not hitting any regressions over a time, we left the test workload running more than 12 hours over a night.. - and as you can see from the graph, when in the morning we were back in the office, the test was still running and showing the same near 1.5M TPS result.. That was really amazing.. And I've expected that now by using all 72CPU of the server we'll beat all imagination levels with 2M or even 3M TPS (to be honest, from the beginning nobody really expected to see 1M TPS.. - this was like "top of all expectations", so showing 1.5M TPS was still a big surprise for everybody ;-))

So, what was a TPS limit when all 72CPUs are used?.. - for my big surprise this question remained without answer.. - the goal of 1M TPS was reached, and for Mikael the testing was finished.. - and the same day the whole team packed their baggages and left the office.. ;-))

Few years later Alzato was acquired by MySQL, and MySQL Cluster was born.. And today's over(!) 20M TPS obtained on MySQL Cluster even on Read+Write workload is no more a subject of surprise for anyone.. But the main performance victory in NDB milestones for me will be still these 1.5M TPS obtained 11 years ago in Paris ;-))

~/~

Over a time you're always finishing by realize that the most surprising stuff in the life is the life itself...

Rgds,
-Dimitri

MySQL Performance: What are your top-5 performance issues?..

Sun, 21 Apr 2013 21:41:30 +0200

I have a talk about MySQL 5.6 Performance related stuff during Percona Live (this Wednesday, 24 April 2:00pm - 2:50pm @ Ballroom A). But my main interest during this Conference is to exchange with you as much as possible about what kind of performance problems you meet generally and what are your top-5 performance issues in MySQL workloads you have today right now? (ordered by priority)..

Come to discuss it live -- we have also:

MySQL 5.6 Deep Dive BoF session on Tuesday, 23 April 6:00pm - 7:00pm @ Ballroom H -- open discussion
don't miss InnoDB talk by Sunny (Tue, 23 April 4:50pm - 5:40pm @ Ballroom E)
and talk about MySQL Replication by Luis (Thu, 25 April 11:00am - 11:50am @ Ballroom D)

We all will be there ;-)

As well there are several evening events first day, second, and third - if you prefer to discuss it around a beer ;-))

While you may also left a comment about your top-5 issues here, and then let's talk about..

Rgds,
-Dimitri

MySQL Performance: Analyzing benchmarks, part 4: TRX list

Sun, 21 Apr 2013 04:17:34 +0200

This article inspired by benchmark results published by Alexey @Percona related to the "trx_list" modifications came with latest Percona Server 5.5. I was particularly curious about this feature, because the exactly the same solution was rejected by Sunny two years ago while analyzed this kind of problems (described within presented bug report and others)..

But well, one real test result is better than many discussions, so let's see what kind of results I will get on my own server. I'll test the same OLTP_RO Point-Select workload using 8 tables (by running 8 Sysbench processes in parallel). I'd say that this kind of load was initially pretty problematic for MySQL 5.6, as queries here are very short and running particularly fast. So, the idea of Read-Only transaction feature coming in 5.6 initially because of this workload, before being useful in other cases as well.

So, lets try the first case when transactions are not used and user sessions are using auto-commit (which is automatically will use Read-Only transactions in 5.6). The following graph is representing OLTP_RO Point-Select test on 16cores (MySQL server is running within a taskset with 16cores HT-enabled):

MySQL 5.6 -vs- Percona 5.5 @16cores-HT :

Observations :

Percona Server 5.5 is still hitting kernel_mutex contention here
While MySQL 5.6 is hitting now trx_sys_mutex contention here, which is much lower also and near invisible comparing to kernel_mutex waits on the InnoDB mutex waits graph...
Percona Server is reaching near 200K QPS here (seems like my server is not as fast as tested by Alexey to show higher QPS)..
while MySQL 5.6 is not far from 300K QPS within the same workload..
so, looks to me like Sunny was right ;-)
or probably we will have more details once Percona will release their 5.6 code tree? - let's see..
however, the advantage of 5.6 on this test is pretty clear and doesn't need any comment ;-)

Now, let's bring to the same picture also MySQL 5.5 and MariaDB 5.5.

MySQL 5.6/ MySQL 5.5/ MariaDB 5.5/ Percona 5.5 @16cores-HT :

Observations :

Percona 5.5 has a clear advantage here over MySQL 5.5 and MariaDB 5.5
kernel_mutex contention is dominating on all 5.5 based engines
while MySQL 5.6 is just out-passing them all..

However, more work is needed in 5.6 to get a rid of this trx_sys_mutex contention, because on 32cores it kills performance and we are hitting QPS drop rather increase. And again, have to lower this contention by using a bigger "spin delay" value (96 here) to avoid getting things even worse.. However MySQL 5.6 is still better here than Percona 5.5:

MySQL 5.6 / Percona 5.5 @32cores :

And if we enable HT here the result will be only worse:

MySQL 5.6 @32cores, HT=off / on :

Probably this is a good example of problems related to hot contentions (enabling HT on 16cores improving performance, and using 16cores-HT giving a better result than pure 32cores -- as this contention is so hot, it's as expected then..)

While, of course, as soon as you're having slightly more heavy queries within your workload, this contention will have much less impact, and you'll get higher QPS on 32cores rather on 16cores and matching your HW capacity.. But we have a good challenge here, so work continues, stay tuned ;-)

Regarding the test case #2 (when transactions are used) - well, nothing surprising in results, because it was exactly the reason to have Read-Only transactions feature, right? ;-) However, I may say you now that in the next MySQL 5.6 release the code is discovering "auto-magically" Read-Only transactions (in fact any transaction is considered Read-Only until it did not involve any writes, and switching to a "normal" transaction model as soon as the first write was involved).. - just wait little bit before the code is published..

To summarize, here are few "simple" charts presenting obtained results:

OLTP_RO 8-tables Point-Selects @16cores-HT :

OLTP_RO 8-tables Point-Selects @32cores :

All this said, there was a very good progress we made already in MySQL 5.6, but there are still tons of challenges in MySQL to improve performance and yet more to do ;-) (and many similar things coming in mind while on road to Percona Live MySQL Conference and flying from Paris to San Francisco.. - looking forward to discuss about any MySQL Performance problems there, so see you soon, it's already within 2 days! ;-))

Rgds,
-Dimitri

MySQL Performance: Analyzing Benchmarks, part-3 : Low Level Workloads with MySQL 5.6

Wed, 10 Apr 2013 03:51:26 +0200

This article was inspired by problems on low level workloads initially reported by Peter Zaitsev (Is MySQL 5.6 is slower than 5.5?, and then Why mysql performance at low concurrency is important) and Mark Callaghan (MySQL 5.6 single-threaded read-only, MySQL 5.6 single-thread update-only, MySQL 5.6 - incomplete perf-guide..)..

Sorry, it'll be long.. :-)

Of course the "low level" performance is important and remains significant. And "small servers" performance too. In many cases people supposing they have a heavy load because there are observing hundreds or thousands "active" connections, while in reality only few of them are truly active on the same time, and concurrency level is much lower than initially supposed.. Such workloads may remain near free from any internal or external contention, and have performance level depending only on the pure code path overhead..

I'll start first with Read-Only tests (Sysbench OLTP_RO, which is used so often), as this test case is the most simple and really only few factors coming into the game.

Single User OLTP_RO Performance

What is important here... - single user (or single thread) workload is completely free of any possible or potential contention, so if there is any difference in execution time, then it represents a pure overhead (coming from "somewhere").. Now, if we're comparing MySQL 5.6 with 5.5 (and yet with 5.1) we have to keep in mind that there was no fundamental changes made between them to improve the code path itself. MySQL was lacking "must have" features that people needing in production, so every new release came with a bigger code (means more and more longer code path).. - so just from a "common sense" single user performance cannot be better on MySQL 5.6 than on 5.5 (and not on 5.1 which is even not having MDL code)..

And then we may always speculate on Performance Schema (PFS) impact, Optimizer Traces, not optimal MDL, etc. etc. - forgetting that exactly all these "goodies" will completely change and speed-up your developments and workload analyzes ;-) So, yes, you may still compile MySQL 5.6 without all this stuff and win some 5% or maybe even 10% in performance.. - but it'll be likely you decided to remove from your own car all counters, electronics and air conditioning just to save in autonomy by lowering electricity consumption.. - but will you then still consider it as a "safe driving"? ;-)) (but well, we're not stopping to work on it, and things are constantly improved, stay tuned ;-))

BTW, also while analyzing various MySQL 5.6 "overheads" I've also discovered that on some queries/workloads I'm getting up to 12%(!) performance improvement with MySQL binaries are compiled without "-fno-omit-frame-pointer" GCC option.. - but will you ever consider to use such a MySQL server binaries in production? - because from such a binary you'll never able to get a complete stack trace, means if you'll have any core dumps they all will be just useless, and could be sent directly to /dev/null.. ;-)

But well, this is still yet a half of the problem.. - the main problem for me is that single user OLTP_RO is giving me completely different results from run to run (same test, same config, same HW - but different results from run to run).. Thing are going so fast on OLTP_RO workload once all the data are cached by InnoDB Buffer Pool, then performance result becomes depending on many small things (CPU cache, RAM latency, etc.).. So, when on the same run without any changes you may already get 3% difference, it becomes even harder to understand if some applied new changes will bring you real improvement or additional overhead..

However, we may expect a better stability when we're observing more concurrent sessions running, right? - so, on the following example I've started with 8 concurrent users.

Low Level OLTP_RO Workloads

Let's get a look on the following test results:

this is a "classic" Sysbench OLTP_RO workload on 12cores server
note: used here MySQL server binaries are compiled without(!) PFS
transaction clauses are not used..
MySQL server is started cold, then few tests executed to "pre-warm" the cache
then the same "referenced" OLTP_RO test executed 3 times (one after one)
then MySQL server restarted again, pre-warn and the same 3 tests executed again..
and all these 3 times too, so in total we have 9 test results which are expected to be "near the same", right?..

Le't see now what we're getting in reality:

+------------------+---------+-------+-------+-------+
| Test             | Engine  | PFS   | Users | QPS   |
+------------------+---------+-------+-------+-------+
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 69577 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 70721 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 70779 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 71245 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 71418 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 71507 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 71564 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 71721 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 72015 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 65489 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 66156 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 66833 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 67293 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 67328 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 67395 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 67494 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 68655 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 69030 |
+------------------+---------+-------+-------+-------+

Observations :

you can see that 5.6 results are in range of [65489 - 69030] QPS..
and 5.5 are in [69577 - 72015] QPS..
so, there is already 5% variation on the 5.6 results
and 3% on 5.5
however, if you're really unlucky and in your test series will match the worse time of 5.6 and the best time of 5.5 then you'll observe 10%(!) regression from 5.5 to 5.6..
and if you're unlucky in opposite side: then the difference will be only 0.7%(!) and you'll even not consider it, right? ;-))

So, seems like to understand something here the only solution is to repeat the same tests as much an possible, and then get an AVG value from obtained variations, which will look then like following:

+------------------+---------+-------+-------+------------+
| Test             | Engine  | PFS   | Users | Avg_QPS    |
+------------------+---------+-------+-------+------------+
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |     8 | 71171.8889 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |     8 | 67297.0000 |
+------------------+---------+-------+-------+------------+

which is giving us 5.8% performance regression comparing 5.5 to 5.6 on 8 user sessions.. - pretty strange how it was possible that such a difference escaped from our QA tests, because our "red line" there is around 3% (and every test is executed 3 times) - probably we were just little bit more lucky here, etc. - don't know..

But result is a result, and we cannot ignore it, right? ;-)) However, the most interesting for me here is to know exactly from where the difference is coming (and fix it then if it's possible) -- but there is another fun stuff coming ;-)

So, I'm profiling now the same 5.6 and 5.5 workloads with "perf" and getting:

5.6 : 64400 QPS
5.5 : 70800 QPS

both results are slightly worse than expected (yes, "perf" tool also has its overhead, not only MySQL stuff ;-)), however the result is the result, and there is really 10% difference happened during these tests under profiling, so we should see something pretty visible and big in profiler reports, right? (well, at least expect to see finally something and then at least to know what to fix)..

From a long time I've developed a tool helping me to compare several "perf" reports and highlight the difference. As well we have to keep in mind this "regression level" and do a kind of "normalization" between 5.5 and 5.6 reports -- CPU% time should be slightly increased for the slowest engine (5.6) to be comparable with another one (5.5), so I've used here a multiplier of x1.10 and the result is the following with top-50 most "hot" functions:

Perf List :

#1 - sb_OLTP_RO_1M-ps-8usr-mysql55-QPS_70800 x Multiplier: [1.0]
#2 - sb_OLTP_RO_1M-ps-8usr-mysql56-QPS_64400 x Multiplier: [1.10]

Top-50 Function Name(s)	CPU% #1	CPU% #2	+	+w
my_hash_sort_simple	10.68	11.58	+8.44%	+0.90
memcpy	3.71	3.67
row_search_for_mysql	3.46	2.77
ptr_compare_0	1.62	0.24
my_strnxfrm_simple	1.34	1.41	+5.06%	+0.07
libjemalloc	1.23	1.01
make_join_statistics	0.93	1.29	+38.37%	+0.36
btr_search_guess_on_hash	0.98	1.01	+3.25%	+0.03
JOIN::optimize	0.97	0.85
row_sel_store_mysql_rec	0.88	0.33
buf_block_align	0.87	0.97	+11.25%	+0.10
__merge_sort_loop	-	0.90	+0.90%	+0.90
__pthread_mutex_lock_internal	0.81	0.89	+9.99%	+0.08
rec_init_offsets	0.81	0.79
rec_get_offsets_func	0.74	0.82	+9.99%	+0.07
__merge_sort_with_buffer	-	0.76	+0.76%	+0.76
_spin_lock_bh	0.69	0.69	+0.42%	+0.00
my_strnncollsp_simple	0.66	0.73	+9.99%	+0.07
Field_string::pack	0.65	0.65
tcp_recvmsg	0.60	0.71	+17.32%	+0.10
__memset_sse2	0.55	0.36
my_lengthsp_8bit	0.54	0.54
open_table	0.51	0.31
lock_clust_rec_cons_read_sees	0.49	0.47
dispatch_command	0.43	0.53	+22.78%	+0.10
cmp_dtuple_rec_with_match_low	-	0.52	+0.52%	+0.52
buf_page_get_gen	0.39	0.52	+32.55%	+0.13
cmp_dtuple_rec_with_match	0.46	-
evaluate_join_record	0.45	0.51	+12.43%	+0.06
my_qsort2	0.45	0.22
open_tables	0.45	0.46	+2.66%	+0.01
JOIN::prepare	0.39	0.48	+24.09%	+0.09
tcp_sendmsg	0.37	0.48	+30.80%	+0.11
mysql_execute_command	0.37	0.47	+27.83%	+0.10
tcp_ack	0.42	-
filesort	0.42	0.32
__pthread_mutex_unlock	0.41	0.29
row_sel_field_store_in_mysql_format_func	-	0.44	+0.44%	+0.44
fget_light	0.30	0.44	+46.65%	+0.14
buf_page_get_known_nowait	0.30	0.42	+39.32%	+0.12
ha_innobase::general_fetch	0.23	0.42	+81.72%	+0.19
mysql_select	0.33	0.41	+23.32%	+0.08
trx_start_low	0.21	0.40	+88.55%	+0.19
Item_func_between::val_int	0.35	0.37	+6.85%	+0.02
THD::enter_stage	-	0.39	+0.39%	+0.39
copy_user_generic_string	0.28	0.37	+33.56%	+0.09
hp_write_key	0.34	0.37	+9.99%	+0.03
__pthread_getspecific_internal	0.33	0.37	+13.32%	+0.04
build_template	0.33	-
make_join_select	-	0.36	+0.36%	+0.36
			+613.88%	+6.67

Observations :

you can see that the most hot function here is my_hash_sort_simple(), however there is only 8.4% difference between 5.5 and 5.6 here, and only 0.90% of "pure weight"..
while there are function going up to 88% in difference, like trx_start_low(), but having only 0.19% in weight..
or __merge_sort_loop() and __merge_sort_with_buffer() which are completely new for 5.6 (and not present in 5.5), and their sum weight will at least out-pass 1.5% ;-)
so, from where the main difference is coming?..
while it's sure if in 5.6 we were able to divide by 2 time spent in the my_hash_sort_simple() function -- then we may expect to see a pretty positive difference in this workload, but it's not yet a case ;-))
and not the the summary percentage weight at the end of the report - it's 6.67% in difference comparing diff 5.5 vs 5.6

Now, you can also see that on some function were spending less time in 5.6 than in 5.5, so what if now I inverse the comparison and will get a report on diff 5.6 vs 5.5 (instead of 5.5 vs 5.6) ?..

Here is the result:

Perf List :

#1 - sb_OLTP_RO_1M-ps-8usr-mysql56-QPS_64400 x Multiplier: [1.10]
#2 - sb_OLTP_RO_1M-ps-8usr-mysql55-QPS_70800 x Multiplier: [1.0]

Top-50 Function Name(s)	CPU% #1	CPU% #2	+	+w
my_hash_sort_simple	11.58	10.68
memcpy	3.67	3.71	+1.02%	+0.04
row_search_for_mysql	2.77	3.46	+24.97%	+0.69
ptr_compare_0	0.24	1.62	+569.49%	+1.38
my_strnxfrm_simple	1.41	1.34
libjemalloc	1.01	1.23	+21.55%	+0.22
make_join_statistics	1.29	0.93
btr_search_guess_on_hash	1.01	0.98
JOIN::optimize	0.85	0.97	+14.53%	+0.12
row_sel_store_mysql_rec	0.33	0.88	+166.69%	+0.55
buf_block_align	0.97	0.87
__merge_sort_loop	0.90	-
__pthread_mutex_lock_internal	0.89	0.81
rec_init_offsets	0.79	0.81	+2.28%	+0.02
rec_get_offsets_func	0.82	0.74
__merge_sort_with_buffer	0.76	-
_spin_lock_bh	0.69	0.69
my_strnncollsp_simple	0.73	0.66
Field_string::pack	0.65	0.65	+0.16%	+0.00
tcp_recvmsg	0.71	0.60
__memset_sse2	0.36	0.55	+51.53%	+0.19
my_lengthsp_8bit	0.54	0.54	+0.20%	+0.00
open_table	0.31	0.51	+65.60%	+0.20
lock_clust_rec_cons_read_sees	0.47	0.49	+3.60%	+0.02
dispatch_command	0.53	0.43
cmp_dtuple_rec_with_match_low	0.52	-
buf_page_get_gen	0.52	0.39
cmp_dtuple_rec_with_match	-	0.46	+0.46%	+0.46
evaluate_join_record	0.51	0.45
my_qsort2	0.22	0.45	+104.57%	+0.23
open_tables	0.46	0.45
JOIN::prepare	0.48	0.39
tcp_sendmsg	0.48	0.37
mysql_execute_command	0.47	0.37
tcp_ack	-	0.42	+0.42%	+0.42
filesort	0.32	0.42	+31.67%	+0.10
__pthread_mutex_unlock	0.29	0.41	+43.37%	+0.12
row_sel_field_store_in_mysql_format_func	0.44	-
fget_light	0.44	0.30
buf_page_get_known_nowait	0.42	0.30
ha_innobase::general_fetch	0.42	0.23
mysql_select	0.41	0.33
trx_start_low	0.40	0.21
Item_func_between::val_int	0.37	0.35
THD::enter_stage	0.39	-
copy_user_generic_string	0.37	0.28
hp_write_key	0.37	0.34
__pthread_getspecific_internal	0.37	0.33
build_template	-	0.33	+0.33%	+0.33
make_join_select	0.36	-
			+1102.46%	+5.10

Observations :

I'm pretty sure that if I'd show this report to anyone without saying who is running faster, 5.5 or 5.6, and just asking to point to the source of regression, there are many functions which can be easily chosen as candidate for the most problematic ;-))
ptr_compare_0() is a very good candidate with it's 569% difference ;-)
and my_qsort2() too (with some others as well ;-)
while it's till 5.5 which is running faster!..
and the sum weight in 5.6 vs 5.5 is finally 5.10%, which is comparing to 6.67% in 5.5 vs 5.6 is giving only 1.57%..
as you can see, it'll be very hard to find something there..

Looks like the best "potential" candidate is still my_hash_sort_simple() function, because improved time on this one will at least be very visible as it represents 10% of the whole execution time..

A similar story is coming with 32 users on this server too, except that it has a smaller variation:

+------------------+---------+-------+-------+--------+
| Test             | Engine  | PFS   | Users | QPS    |
+------------------+---------+-------+-------+--------+
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 149592 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 149593 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 149681 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 149699 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 149745 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 149762 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 150522 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 150586 |
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 150603 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149007 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149017 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149147 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149195 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149308 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149497 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149514 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149541 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149671 |
+------------------+---------+-------+-------+--------+

So,

5.6 : [149007 - 149671] QPS, 0.4% variation
5.5 : [149592 - 150603] QPS, 0.6% variation

And in average QPS we're getting finally 0.4% regression only :

+------------------+---------+-------+-------+-------------+
| Test             | Engine  | PFS   | Users | Avg_QPS     |
+------------------+---------+-------+-------+-------------+
| sb_OLTP_RO_1M-ps | mysql55 | noPFS |    32 | 149975.8889 |
| sb_OLTP_RO_1M-ps | mysql56 | noPFS |    32 | 149321.8889 |
+------------------+---------+-------+-------+-------------+

However, if we'll add now on it 2% in binary is compiled with PFS instrumentation, then another 1-2% if you enable only PFS (keeping instrumentation inactive, but able to be activated dynamically) -- then you're getting 4%-5% difference comparing to MySQL 5.5 which is having PFS disabled by default and bigger PFS overhead when enabled (that's why it's disabled by default in 5.5 ;-))

But then, would you refuse this gold mine of information which is proposing you PFS today in exchange of 5% in QPS?.. - Seems to me I have to present you some more detailed stuff here about every potential overhead in PFS to break last doubts if any ;-)) -- and don't forget that more you test PFS and report your problems - better it will be!

Let's get a look now on the OLTP_RW (Read-Write).

Low Level OLTP_RW Workloads

Things are going slightly different on OLTP_RW:

the gap is not so big as on OLTP_RO (I/O latency is coming in game too as we have writes)
many parts of the new code are representing fixes just to do things right (so, just for that 5.6 is already better ;-))
while there are still some areas in 5.6 which should be improved (see: Heavy I/O RW workloads limits in 5.6 - and work is in progress here, stay tuned)..

While again, even on this "small server" I'm still observing a slightly lower TPS (Commit/sec) here on 5.6 vs 5.5, then looking not just on the TPS numbers, but on details about OLTP_RW on both engines we can see the following:

OLTP_RW @32usr, 5.6 -vs- 5.5 :
Observations :

as you can see, there is no drops on 5.6 in Commit/sec activity
and Checkpoint Age is remaining much lower, still allowing a room against some unexpected activity spikes
and to keep things stable, we're writing slightly more (as you can see from I/O activity graph)..
you just see 5.6 improved Adaptive Flushing in action
and just for this reason I'd migrate to 5.6 asap ;-))

While, of course, on a bigger server this workload will run way better on MySQL 5.6 (as you can see from the previous posts).

More in depth

Let's now go more in depth, but with a bigger server - my lovely 32cores host. What is good with a bigger server that we're able to simulate small one with it as well, but have a room for different test configs ;-) By binding MySQL server and Sysbench binary to 8 or 16cores we may observe less or more what we may expect from an 8 or 16cores server..

NOTE: for the following tests MySQL binaries were compiled with PFS instrumentation, but PFS was turned off during the tests.

Observations :

On 8cores OLTP_RO, 4 or 8users :
- the most hot functions on 5.5 are :
  - ptr_compare_0( 4% )
  - row_search_for_mysql( 3% )
- and on 5.6 are :
  - row_search_for_mysql( 3% )
  - merge_sort_loop( 2% )
  - merge_sort_with_buffer( 2% )
- interesting that my_hash_sort_simple() on this server is much lower (1.3% only)
- but 5.6 is still worse than 5.5..
On 16cores OLTP_RO :
- 5.6 becomes better than 5.5 only since 16 or 32 concurrent users
- and this is as expected at least, because it's since this load level the first internal contention are appearing ;-)
On OLTP_RW 8 or 16cores :
- pretty similar barrier is here too..
- on 8cores 5.5 is up to 5% better than 5.6
- on 16cores 5.6 becomes better since 16 concurrent users

Few graphs so far:

OLTP_RO @8cores-HT :

OLTP_RO @16cores-HT :

OLTP_RO Point-Selects @8cores-HT :

OLTP_RO Point-Selects @16cores-HT :

OLTP_RW @8cores-HT :

OLTP_RW @16cores-HT :

OLTP_RW Updates-only @8cores-HT :

OLTP_RW Updates-only @16cores-HT :

So far :

MySQL 5.6 showing here a direct speed-up if you have 16 (or more) CPU cores on your HW server and at least 16 concurrent user sessions..
with a smaller amount of CPU cores (or concurrent user sessions) you may see the same or slightly lower QPS level on MySQL 5.6 comparing to 5.5, and if this difference with PFS turned off is still out-passing 5-10% -- file a bug, you're really hitting something abnormal ;-)
in any case, even on a small server, you'll still get a benefit of the all other performance improvements coming with MySQL 5.6 (ex: stable writes, binlog group commit, etc. etc. etc. - the most complete lists by Rob and Peter)..

And, of course, work is continuing, so stay tuned ;-)

As well, I have yet more benchmark results that I'm planning to present and discuss during Percona MySQL Conference - so come to join us during 5.6 BoF, our talks, or just around a beer ;-) BTW, if you're still hesitating if you should attend this Conference.. - just trust me, you should! ;-)

Rgds,
-Dimitri

Speaking at MySQL Tech Tours @Paris

Wed, 13 Mar 2013 19:29:54 +0100

26 March 2013, MySQL Tech Tours comes to Paris, FRANCE, and I'm happy to cover MySQL Performance related topics during this event!

The full agenda is here: MySQL Tech Tour @Paris

Of course, the timing will be pretty short to cover all topics ;-) - but there will be also lunch time and whole afternoon if you're willing to continue, because I'll stay there for a full day and will have all my time to answer any of your questions about MySQL Performance :-))

Note: the event is free, so places are limited.. (means: hurry up to subscribe ;-))

See you there! and my strong willing is to have such kind of MySQL events @Paris more and more often (and on more regular basis) - if you have some ideas about to share, just drop me an e-mail or leave a comment..

Rgds,
-Dimitri