« MySQL Performance: 8.0 re-designed REDO log & ReadWrite Workloads Scalability | Main | MySQL Performance : my slides from MySQL Day & FOSDEM Feb.2018 »

Saturday, 30 December, 2017

dim_STAT v.9.0 CoreUpdate-17-12 is here !

Year is finishing, and as a kind of "end-of-year" gift, I'm happy to announce that a freshy new CoreUpdate-17-12 is available from now ! ;-))

IMPORTANT : this is a very "sensible" update, and you cannot just to apply it on the top of already deployed dim_STAT instance as before.. -- there was a need to fix several issues within "under hood" binaries to make the new stuff working properly, so the new code has simply no chances to work with old binaries.. So far, I decided to make it all as a "single shot move" -- align a new update shipment with a moving to "64bit" versions support only :

  • e.g. this is not a new version of dim_STAT
  • this is just a "remastered" 64bit release + several visible and internal fixes
  • any 32bit releases are remained "as it", and it makes no more sense to continue to support them..
  • 64bit versions of dim_STAT v.9.0 are available for Linux and MacOSX (macOS)
  • any further CoreUpdate will work only with 64bit version having the latest supported binaries..

So, what about this "new stuff" requiring such deep changes to go till the binaries remastering ?.. -- a very long story short, this is all about support of SVG images ! ;-)) -- I've started to develop dim_STAT exactly 20 years (!!) ago.. (hard to believe time is flying so fast..) -- initially dim_STAT used Java Applets (it was very lightweight (yes! it was so 20 years ago ;-)) and it was really cool, "live", etc.) -- but then Java support in a browser became only heavier and heavier, so I've added PNG images support (which could bring back the initial lightweight to dim_STAT and make it "usable" again ;-)) -- and today, things are changing again ;-)) -- "retina" and other "high resolution" screens become more an more popular, and on these screens my previously "good enough" PNG graphs are looking just "ugly" (to be polite ;-)). After testing and analyzing tons of various JS-based (or other) live graphing tools/libs, I've finally stopped my choice on SVG ! -- it's already supported by most of web browsers, still lightweight, and has a huge advantage -- it's extremely well "readable" !! and you can scale (!!) it very easily as much as you want ;-)) (so, no more color confusions and simply ugly graphics ;-))

An SVG graph is looking like this :


well, this is just a snapshot, but hope you already can get an idea about how much "more clean" your graphs could be ;-))

NOTE : some web browsers (for ex. FireFox) may require to not be limited in the "smallest" font size to draw SVG images correctly (as the text you see on your graphs is reproduced by your browser itself).

Now, few words about how SVG is integrated in dim_STAT :
  • you have a new option for Graph image now : SVG (default)
  • and in fact at any time you're selecting SVG or PNG -- your graph is generated in both formats on the same time
  • however, the same image "size" may look differently in PNG vs SVG..
  • for this reason there is an additional option : SVG Scale (which is x1.0 by default, but you may change it as you like)
  • and most of dim_STAT "modules" are considering SVG option since now ;-))

So far, what is new in CoreUpdate-17-12 :
  • SVG support !
    • mandatory : remastered dim_STAT v.9.0 64bit version
    • always generating PNG & SVG together
    • SVG Scale option (web interface)
    • default SVG Scale=1.0
  • Snapshots :
    • always containing PNG & SVG
    • editing :
      • allow image reordering by mouse dragging
      • use SVG or PNG images according to what was recently used..
    • redesigned Snapshots list :
      • select criteria : Title pattern, order by time / title
      • show all || first / last [N]
      • option : show first N SVG/PNG images..
      • option : show notes..
      • click on title link involves snapshot editing
    • redesigned Published Snapshots :
      • have now also PNG and SVG document versions (HTML)
      • HTML documents are now composed from first 2 images of each Snapshot + its notes, then <<More>> link is pointing on a full Snapshot page..
      • PDF documents format remains the same (single document starting with a list of links to all included Snapshots)..
  • Multi-Line stats :
    • "CacheList" feature to speed-up names look-ups ! (generated once, then only re-used, can be re-generated again at any time on demand)
    • allow bigger varchar limits for Add-On "name" collumn : 8/ 16/ 24/ 32/ 48/ 64/ 80/ 96/ 128
  • Report Tool :
    • you can now add Published Snapshots to your reports !
    • code remastering / cleanup
    • better / more accurate formatting
    • remove external / java editors
    • minor bug fixes..
  • Bookmarks :
    • Duplicate existing Preset
    • Edit Preset (via checkbox list + reorder)
    • code remastering..
  • STAT-service :
    • STAT-service script now saves PID on start
    • new commands: status, restart
    • status : check if process with saved PID is running
    • stop : call x.killSTAT with saved PID
    • x.killSTAT : remaster to use CMD or PID
    • restart : calls itself with stop, then start
    • respecting chkconfig standard
    • respecting LSB standard
    • to setup under systemd :
      • # ln -s /apps/STATsrv/STAT-service /etc/init.d
      • # systemctl enable STAT-service
      • # service STAT-service start
      • # service STAT-service status
      • # /apps/STATsrv/STAT-service status
    • involving ALARM signal : ejecting on x10 Timeout non-activity period in all MySQL/ PG/ ORA scripts !!!
      • => this could be very important in case when your monitored database engine becomes "frozen", but still continue to accept new connections..
      • so dim_STAT will not see any stats coming from your database and suppose the connection was lost..
      • and dim_STAT will then start a new stat command..
      • which may still connect to your database, but not go any further, as database is frozen..
      • and this may continue endless in loop every time once "connection timeout" is expired..
      • and your server will get tons of stat commands started on its side and trying to collect some stats from your frozen database engine..
      • so, to avoid this, each stat command is now having its own "non-activity alarm" -- if within x10 times Timeout period there was nothing collected from a given database, the stat command will self-eject by alarm event ;-))
    • innodbMUTEX : advanced aggregation with tot@ / max@ / avg@ wait stats !
    • Linux netLOAD_v2 : reports also MB/sec + fixed counters overflow
    • Linux mysqlSTACK / ProcSTACK (based on quickstack from Yoshi (@FB))
    • Linux mysqlCPU / ProcCPU (based on tiptop, not compatible with "perf", use with caution)
    • Linux PerfCPUSTAT (based on "perf", reports CPU HW counters)
    • Linux PerfSTAT_v2 (improved PerfSTAT)
      • extended with Cycles/s numbers !!
      • new option : -Event ... (ex: -Event cycles:u)
    • EasySTAT :
      • extended on Linux with : mysqlSTACK, mysqlCPU, netLOAD_v2, PerfCPUSTAT, PerfSTAT_v2
      • now also auto-generating start/end log messages (to simplify post-analyze)
      • LoadDATA.sh is allowing now to pass args from command line (path to BatchLOAD and DB name)
  • General :
    • Analyze Top-N values : choice => MIN/ MAX/ AVG/ 95% !
    • export data in CSV format via Analyze page & dim_STAT-CLI
    • tag LOG Messages with different colors according text pattern !
    • on stat names checkbox => popup with note description (when available)
    • improved auto-scaling Y-axe values according image size !
    • improved default HTML style, re-look, etc.
    • New Add-Ons & their Bookmarks :
      • mysqlSTACK
      • mysqlCPU
      • ProcSTACK
      • ProcCPU
      • netLOAD_v2
      • PerfCPUSTAT
      • PerfSTAT_v2
      • ...
    • minor bug fixes..

Hope you'll enjoy all this new stuff as I'm already doing ;-))

For those who already using dim_STAT and want to preserve their collected data -- here are few simple steps to make the migration to the latest 64bit version smooth :
  • first of all - no matter if you used 32bit version before, or 64bit (or SPARC, etc.), instructions are exactly the same
  • however, even the version remains the same, we're moving to new remastered binaries..
  • so the whole process is looking as "migration" (rather a simple CoreUpdate apply)
  • (and if your data are critical, better to try you migration steps on another host first ;-))
  • so far, let's suppose you already deployed dim_STAT into "/apps" on your host1
  • 1) stop dim_STAT : # /apps/ADMIN/dim_STAT-Server stop
  • 2) backup your whole (!!) database directory : /apps/mysql/data
  • 3) if you created any Reports or Snapshots, then backup also the whole web server docs directory : /apps/httpd/home/docs
  • 4) now on host2 install the latest 64bit dim_STAT version (supposing it'll be re-installed to the same default "/apps" path)
  • 5) restore your "docs" backup into "/apps/httpd/home/docs" on host2
  • 6) from your databases backup restore all to "/apps/mysql/data" on host2 except the following directories :
    • /apps/mysql/data/mysql
    • /apps/mysql/data/performance_schema
    • /apps/mysql/data/dim_00
  • 7) start dim_STAT on host2 : # /apps/ADMIN/dim_STAT-Server start
  • 8) connect to your newly installed dim_STAT on host2 and check you can find all your previously collected data and created documents / snapshots..
  • 9) if something is going odd.. -- then ping me back, let's see what is going wrong ;-))
  • 10) if all is fine, then test if for a while, and once everything is really ok, then you're comfortable to migrate your host1 too ;-)) (and don't hesitate to ping me with good news too ;-))

Crossing fingers.. -- hope all will go just fine for you ;-))

And for the end.. -- there is yet "one more thing"..

This "one more thing" was inspired by observing many dim_STAT users doing some completely "unexpected things" (unexpected in my mind, but looking very "natural" for those who are doing) -- honestly, I'm really surprised by all use cases I've seen over past years (even users generating graphs via CLI commands involved from app servers and then grouping them into various documents, charts, etc.) -- but the most painful for me was to see users trying to involve "web oriented" actions in dim_STAT via curl, wget, etc.. -- this could bring to something wrong as there could be tons of options used within each POST/GET order, and all expected calls could be just broken after further CoreUpdates..

And to make your life more simple for such use cases, let me present you the REST-like Interface available since CoreUpdate-17-12 ! ;-))

dim_STAT-REST interface is supporting the following commands right now :
  • DB_LIST -- list all available databases
  • DB_CREATE -- create a new database
  • HOST_LIST -- print current host list in database
  • HOST_ADD -- add new hostname into host list in database
  • HOST_STATS -- request the list of available STATS from host STAT-service
  • COLLECT_LIST -- list all available STAT collects in database
  • COLLECT_NEW -- create and start a New Collect in database
  • COLLECT_STOP -- stop Active Collect(s) in database
  • COLLECT_RESTART -- restart Stopped Collect(s) in database
  • LOG_MESSAGE -- add a LOG Message to database

the output of each command is going in simple "pre-formatted" ASCII text, easy to parse and check.

Here is an example of "COLLECT_LIST" output :
==========================================================================================
 dim_STAT-REST (dim) v.1.0
==========================================================================================
 > CMD: COLLECT_LIST
  ----------------------------------------------------------------------------------------
      1 | goldgate     | -OFF- | 1998-12-18 16:28:27 |  15 sec. | Demo 1
      6 | test         | -OFF- | 2002-10-20 23:37:01 |  15 sec. | test
      7 | bezout       | -OFF- | 2003-06-26 13:46:51 |  30 sec. | test err
      9 | fidji        | -OFF- | 2003-09-17 13:23:12 |  10 sec. | test MPXIO + nocanput
     12 | test         | -OFF- | 2003-10-06 17:15:43 |  20 sec. | test MTB bug
     15 | localhost    | -OFF- | 2004-09-26 21:48:49 |  20 sec. | test
     16 | localhost    | -OFF- | 2004-09-26 21:51:59 |  20 sec. | test
     17 | gauss        | -OFF- | 2004-10-01 12:29:37 |  20 sec. | test RESTART
     18 | neel         | -OFF- | 2004-10-01 12:30:00 |  20 sec. | test RESTART
     19 | monod        | -OFF- | 2004-10-01 12:33:52 |  20 sec. | test RESTART
     20 | monod        | -OFF- | 2004-10-01 20:36:37 |  20 sec. | test RESTART
     21 | localhost    | -OFF- | 2004-10-12 20:30:51 |  15 sec. | test statOEE
     22 | dimitri      | -OFF- | 2007-01-21 21:06:43 |   5 sec. | test IObench
     23 | dimitri      | -OFF- | 2009-06-15 16:36:49 |  10 sec. | System Load...
  ----------------------------------------------------------------------------------------

 > OK
==========================================================================================


but I think more simple is to discover this all by yourself -- right now, try to execute the following command from your shell to test dim_STAT installed on your host2 (port 80) :
 $ curl -L "http://host2:80/cgi-bin/WebX.mySQL/dim_STAT/x.REST"

this will print you the following help message :
==========================================================================================
 dim_STAT-REST (dim) v.1.0
==========================================================================================
 > Usage: curl -L "http://host2:80/cgi-bin/WebX.mySQL/dim_STAT/x.REST?CMD=Command[&..options..]" 

    CMD=DB_LIST                    -- list all available databases 

    CMD=DB_CREATE                  -- create a new database "Name"
      &DB=Name                        -- db name
      &Engine=InnoDB|MyISAM           -- db engine (InnoDB or MyISAM)
      &Passwd=password                -- optional password to protect admin actions 

    CMD=HOST_LIST                  -- print current host list in database "Name"
      &DB=Name                        -- db name 

    CMD=HOST_ADD                   -- add new hostname into host list in database "Name"
      &Host=hostname                  -- new hostname (format: [alias/]hostname[:Port])
      &DB=Name                        -- db name
      &RESET=1                        -- optionally: reset hostlist to empty 

    CMD=HOST_STATS                 -- request the list of available STATS from host STAT-service
      &Host=hostname                  -- alias OR hostname (format: [alias/]hostname[:Port])
      &DB=Name                        -- db name 

    CMD=COLLECT_LIST               -- list all available STAT collects in database "Name"
      &DB=Name                        -- db name 

    CMD=LOG_MESSAGE                -- add a LOG Message to database "Name"
      &DB=Name                        -- db name
      &Message=text                   -- text message
      [&Host=hostname]                -- hostname (multiple Host args can be used)
      [&ID=id]                        -- Collect ID (multiple ID args can be used)
                                      * Host and ID are optional :
                                        > if ID is given : use provided ID(s) only
                                        > if no ID nor Host : add the message to all active collects
                                        > if only Host : add the message to active collects matching hostname(s) 

    CMD=COLLECT_NEW                -- create and start a New Collect in database "Name"
      &DB=Name                        -- db name
      &Host=hostname                  -- hostname (only one Host can be user at time)
      &Timeout=Nsec                   -- STATs timeout in seconds
      &Title=title                    -- Collect title
      &STATS=list                     -- list of STATs to collect: stat1[,stat2[,stat3]...] or "all"
                                         all: means all STATs available from Host STAT-service
      [&LOG=filename]                 -- full filename of LOG file to watch
      [&Message=text]                 -- text message to log Collect start 

    CMD=COLLECT_STOP               -- stop Active Collect(s) in database "Name"
      &DB=Name                        -- db name
      [&Message=text]                 -- text message to log on Collect(s) stop
      [&Host=hostname]                -- hostname (multiple Host args can be used)
      [&ID=id]                        -- Collect ID (multiple ID args can be used)
                                      * Host and ID are optional :
                                        > if ID is given : use provided ID(s) only
                                        > if no ID nor Host : stop all active collects
                                        > if only Host : stop active collects matching hostname(s) 

    CMD=COLLECT_RESTART            -- restart Stopped Collect(s) in database "Name"
      &DB=Name                        -- db name
      [&Message=text]                 -- text message to log on Collect(s) restart
      [&Host=hostname]                -- hostname (multiple Host args can be used)
      [&ID=id]                        -- Collect ID (multiple ID args can be used)
                                      * Host and ID are optional :
                                        > if ID is given : use provided ID(s) only
                                        > if no ID nor Host : restart all recently stopped collects
                                        > if only Host : restart recently stopped collects matching hostname(s) 

    ...
==========================================================================================

## ERROR:
=> CMD is not filled !! 

==========================================================================================


while you can see there many options listed, there are many actions are simplified "by default" -- and to explain this, let me show you a simple use case of the whole workflow by example :
  • you're starting a testing with say "Customer 1234"
  • so, first you creating a dedicated database CU_1234_YourName
  • (adding your name to dbname to be sure the name is unique ;-))
  • then you adding to this database the hosts you're wanting to use (say: host_N1, host_N2, host_N3)
  • (note: this also can be the same HW server, but running several STAT-services (each one for different db instance, etc.)
  • once you're ready, you're :
    • starting New Collect for host_N1
    • starting New Collect for host_N2
    • starting New Collect for host_N3
    • NOTE: instead of building the list of stats you want to collect from your hosts, you can use "STATS=all" option, which will collect everything -- this could be dangerous (sometimes too much is too much ;-)) -- but you can easily limited "everything" to "just what you need" by editing the STAT-service "access" file (and leave uncommented there only the stats you'll really need) -- so, again, you can keep your own "pre-configured" STAT-service tarball, deploy & start it on your host(s) before any testings, and then in your scripts always use just "STATS=all" (regardless which system you're using and delegate it to your pre-defined STAT-service config ;-))
  • after what, you can run your first test workload..
  • your test script may contain in any place a command to send a Log Message
  • NOTE: without mentioning any ID in the command, the Message will be automatically added to all currently active Collects !
  • (so, in your scripts you even don't need to worry which exactly hosts are used, etc. -- all you need to know is the URL of your dim_STAT server and the dbname you're using ;-))
  • supposing you got your first results, and now need a feedback from Customer/Dev about, so no more tests for the moment.. -- then you just involve COLLECT_STOP for your database
  • NOTE: without any ID provided the command will stop all active collects within your database (so, no need to worry you forgot anyone ;-))
  • then, few days later, you have more info, and need to run other tests within the same conditions and on the same hosts..
  • so, all you need to do is just to involve COLLECT_RESTART command, and again, without any ID and by only giving DBNAME the tool will restart the latest most recent existing collects ;-))
  • and in case you need to run some tests only on say "host_N2" => you then just giving "DB=CU_1234_YourName&Host=host_N2" and the tool will automatically find the most recently created Collect corresponding to "host_N2" and restart it !
  • same, your test script continues to send Log Messages, and if the only host_N2 Collect is active during this time => then only host_N2 Collect will log them, and not other Collects ;-))
  • and then again, but involving COLLECT_STOP with no ID, it'll stop all your running collects, no need to worry to miss any one of them ;-))

Well, don't hesitate to ping me if you need any more details !

That's all for the moment. As usual, all the stuff above is available for free download from my site :
- http://dimitrik.free.fr
Any comments are welcome ! Happy New Year & Have Fun ! ;-))

Rgds,
-Dimitri
Posted by Dimitri at 0:29
Categories: dim_STAT, Linux
blog comments powered by Disqus
Note: if you don't see any "comment" dialog above, try to access this page with another web browser, or google for known issues on your browser and DISQUS..