<< | [up] | >> |
dim_STAT User's Guide. by Dimitri |
Add-On Statistics |
One of the most powerful features of dim_STAT is ability to integrate your OWN statistic programs into the tool. Once added, they will be considered by tool as all other from standard set of STAT(s) and give you the same kind of service: Online Monitoring, Up-Loading, Analyzing, Reporting, etc.
However, choice of external stat programs is too wide and it's quite impossible to design a wrapper adapting to any format. So, I've decided to limit input recognizer to just 2 formats (but covering maybe 95% needs) and leave you to write your own wrapped if necessary to bring the output to one of supported formats.
Formats supported by dim_STAT:
- SINGLE-Line: with one output line per measurement (ex: vmstat)
- MULTI-Line: with several output lines per measurement (ex: iostat)
To be correctly interpreted, your stat program should produce a stable output (same format for data lines, at least one line in MILTI case, keep time-out interval correctly, etc.). Lines different of data have to be declared as ignored by dim_STAT.
NOTE: lines shorter than 4 characters considered as spam! and ignored!
Let's take some examples to get it more clear...
Example of SINGLE-Line command integration |
Let's say we want to monitor a read/write cache hit on the system. This information we may get via "sar" (for example):$ sar -b 1 1000000000000000 SunOS sting 5.9 Generic_112233-05 sun4u 07/09/2004 18:10:13 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 18:10:14 0 1 100 0 0 100 0 0 18:10:15 0 14 100 0 0 100 0 0 18:10:16 0 7 100 0 0 100 0 0 18:10:17 0 0 100 0 0 100 0 0 18:10:18 0 0 100 0 0 100 0 0 18:10:19 0 135 100 0 0 100 0 0 18:10:20 0 0 100 0 0 100 0 0 18:10:21 0 69 100 0 2 100 0 0 18:10:22 0 86 100 0 2 100 0 0 18:10:23 0 0 100 0 0 100 0 0 18:10:24 0 0 100 0 0 100 0 0 18:10:25 0 0 100 0 0 100 0 0 ...And what we are interesting in are "4"-th and "7"-th columns from sar output, and ignoring any lines containing "*SunOS*" or "*read*"..Folowing "Integrate New Add-On-STAT" link:
Step 2: INTEGRATION
I give a name CacheHIT for the new Add-On.
We need only 2 columns from each line (4 and 7).
And this is a "Single-Line" output...Click on "New"...
Created!
In this step we need to explain what we want to run and which information we'll need:
Description: CacheHIT via SAR
Shell Command: sar -b %i 1000000000000000
- where %i on execution will be replaced by time interval sec.
- Note: command name is doesn't matter here because it uses only as alias for STAT-service (look at "access" file section, e.g. it's possible to name shell command "toto" and put in access file /usr/bin/sar as alias resolving)...Ignore Lines: we should ignore any lines containing "*SunOS*" or "*read*"
DATA Descriptions:
- ColumnName - leave it as it if you don't think access database directly (Note: there are 2 reserved columns for Collect-ID and measurement No.)
- Data Type - set "Float" if you're not sure it'll be always "Int"
- Column# on input - in our case we need columns 4 and 7
- Short Name - one word value descriptions, so %rcache and %wcache
- Full Name - description used in all cases detailed information needed
- Use in Multi-Host - if you choose "Yes" corresponding value will be automatically enabled in Multi-Host mode for analyzing on several hosts on the same time
Create! :))
What's Next? Will it work now?...Anti-Spam FilterYes! IF YOU DID NOT FORGET to give an access to this new command on your STAT-service! (Most common error...)
So, if you want to collect "CacheHIT" data from server "S" be sure STAT-service on "S" giving an execution permissions for command name "sar" (as we put in shell command description). Add following lines into your /etc/STATsrv/access file:
# CacheHIT Add-On command sar /usr/sbin/sar #Now it'll work! :))
NOTE: for security reason and more clean stat-to-command correspondence it'll be more preferable to create a s pecific script 'CacheHIT.sh' for this new add-on and replace with it direct access to 'sar' command...
Example:
$ cat /etc/STATsrv/bin/CacheHIT.sh #!/bin/ksh exec /usr/sbin/sar -b $1 1000000000000000$ $ CacheHIT.sh 5 ...
$ tail -3 /etc/STATsrv/access # CacheHIT Add-On command CacheHIT.sh /etc/STATsrv/bin/CacheHIT.sh #
$
And Add-On shell command will be changed to: "CacheHIT.sh %i"
Note, there is an anti-spam filter feature always active during data collecting! It rejecting all input lines having less than 4 characters in length! So, if your newly made stat command printing only one small column of numbers - please, add leading spaces before to make data correctly accepted by dim_STAT...
MULTI-Line Add-On command integration |
Multi-Line integration is quite similar to Single-Line, except few additional things:
- Line Separator pattern: new-line by default, but in some cases may be a header (like iostat)
- Attribute Column: very important! as you have several lines per measure you need to distinct them by something (like "diskname" column in iostat)
- Use In Multi-Host: has more than simple Yes/No options, you should use SUM or/and AVG for collected values...
Pre-Integrated Add-Ons |
To make your life easier, there are several additional stat programs already pre-integrated (Oracle, Java, Linux). They are not installed by default because not everybody need all features on the same time and have too much buttons on the screen is quite stressful :))So, to install any standard Add-On into your database all you need is simply click on "Install" button. All these programs normally already present in STAT-service package for corresponding platform (of course, don't look for Linux iostat inside of Solaris package :)). However, some of them may need some additional conditions/parameters for correct execution...
The good rule will be checking first the add-on works correctly starting directly from the STAT-service bin-directory on the client side (/etc/STATsrv/bin). And only after collecting from add-on output may be involved.
There are 2 new psSTAT wrappers added:Special Solaris 10: ZoneLOAD / PoolLOAD/ TaskLOAD/ ProjLOAD
- ProcLOAD: all output information on-fly summarized by process name
- UserLOAD: all output information on-fly summarized by user name
These stats become very useful when you have hundreds/thousands running processes and it's no more matter the activity of the one single process rather the groups of processes or users.
There are 4 specific to Solaris10+ new psSTAT_10 wrappers added:netLOAD
- ZoneLOAD: all output information on-fly grouped by zone id
- ProjLOAD: same, but grouped by project id
- TaskLOAD: same, but grouped by task id
- PoolLOAD: same, but grouped by pool id
These stats give you more extended info comparing to standard 'prstat', etc...
netLOAD - Solaris network activity monitoring. This tool is included into dim_STAT STAT-service for a long date now. And since v.8.0 netLOAD monitoring all present network interfaces in the system (including virtual and loopback). If some indicators are not populated by device driver - '-1' value is present instead. Also, new option '-I' is added: you may give a fixed list of network interfaces you want to monitor (run '/etc/STATsrv/bin/netLOAD' for more details). In STAT-service netLOAD is integrated via 'netLOAD.sh' script to give an easy way to change any option if needed.Oracle Add-Ons
jvmSTATNeeds correct Oracle environement for "oracle" user (default: oracle, but may be changed inside of scripts),
it means:
# su - oracle -c "sqlplus /nolog"should work correctly and enter you into "SQL>" prompt for the right database instance.
These Add-Ons given just as examples, better solution should be made via compiled binary and keeping database connection. But sometime, it's better than nothing :))
- oraIO: Oracle I/O stats for data/temp files
- oraEXEC: Oracle SQL Executions/sec, Commits/sec, Sessions
- oraLATCH: Oracle latch stats
- oraSLEEP: Oracle latch sleeps stats
As well you may add any other new one (some customers even collecting statspack reports directly into dim_STAT! :))
Is a wrapper to bring information from "jvmstat" package developed by Sun engineers. jvmstat is officially integrated now within JVM 1.5 distribution. Wrapper is able to monitor all running JVMs on your server.jvmGCTo run jvmstat you need first of all to have jdk 1.5 installed on your host.
To get 'jvmSTAT.sh' wrapper working:
- edit "/etc/STATsrv/bin/jvmSTAT.sh" file (STAT-service) on each client machine to set right PATH environment for JAVA_HOME.
- enable jvmSTAT in STAT-service (edit /etc/STATsrv/access file)
- Before to start any new collect including jvmSTAT, be sure jvmSTAT Add-On is already installed (Add-On interface from Main Page)
NOTE: this wrapper is still here, but I don't see any reason why one should use old JVM version today, no? jvmSTAT is more preferable solution for any kind of GC collection now.LINUX specific STATsWrapper to collect on-fly information about GC activity of any JVM running with "-verbose:gc" option (before JVM 1.4.2 the only possible way to get GC activity infos of standard JVM was a log output dumping, so this wrapper is simply based on log file scanning).
Usage: Let's say you want to see GC activity of one of JVM running on server "J".
0.) Install "jvmGC" via Add-Ons page.
1.) jvmGC using $LOG file for data input (you may change name & permissions according your needs (default filename: /var/tmp/jvm.log), modify if needs on the server "J" STAT-service side (/etc/STATsrv/bin).
2.) Start collect including "jvmGC" via Web interface
3.) on server "J" add "-verbose:gc" option to java in your starting application script and redirect output into application log file (for ex. app.log)
4.) once you want to monitor your JVM:
$ tail -f app.log | /etc/STATsrv/bin/grepX GC >> /var/tmp/jvm.log
5.) observe jvmGC output data & have fun! :))
- LvmSTAT (Linux vmstat)
- LcpuSTAT (Linux mpstat)
- LioSTAT (Linux iostat)
- LnetLOAD (Linux netLOAD)
- LpsSTAT (Linux psSTAT)
- LprcLOAD (Linux ProcLOAD)
- LusrLOAD (Linux UserLOAD)
For further details, see following special Linux note...
Administation tasks |
At any moment you can:Save Add-On Description - this will give you an ASCII text file which may be reused for another database by you or other users. In this way you may share with other any new findings and any new tools you found useful!
Restore Add-On Description - from information on given Desc.File re-create all Add-On needed database structures and fill all needed information for its correct functionality. WARNING: all previous data will be destroyed if you're already using the same Add-On in the current database!
Delete Add-On - Removes Add-On and all corresponding data from the current database...
<< | [up] | >> |