The idea for BatchLOAD came (as many things) from day to day needs. Sometimes you are facing customers/users who want to know what happens on their machines, but then they don't allow the installation of any additional software (a very constructive approach :-)).

All you can do now is to ask them to run some stat commands on their systems and send you the output files. While loading their files every day via the Web interface, you start to think harder and harder if there isn't a way to do this automatically. Are you ready for BatchLOAD??

I decided to add a new component to dim_STAT, but I kept in mind that other tools already exist that are collecting output from stat commands. All these tools are keeping data in their own format, so I've tried to design the input format for BatchLOAD to be easily adaptable. Of course, I didn't think to create something universal :)), but I hope it shouldn't be too hard to write a script that can convert from an existing format into BatchLOAD.

Some words about the internals of BatchLOAD. There is no dependency on the name of loaded files. All needed information is given by command options and in the contents of the loaded file. The loaded file must have special TAGs. At least two: to give the STAT name and to confirm the END.

USAGE:

Usage: /apps/ADMIN/BatchLOAD -cmd NEW/ADD options 


    Options [NEW]:        -- force new collect creation
       -base DBname       -- database name
       -ID id             -- Collect ID, if 0 use max+1 id automatically
       -title Title       -- Collect Title
       -host Hostname     -- Collect Host Name
       -isec sec          -- Collect STATs Interval (sec)
       -start datetime    -- Collect Start DateTime in format YYYYMMDDHHMISS
       -skip1 yes/no      -- Yes/No skip first STAT measurement (often wrong values)
       -file Filename     -- Full path to file with STATs outputs
       -verbose on/off    -- verbose output on/off


    Options [ADD]:        -- add to existing collect whenever possible
       -base DBname       -- database name
       -host Hostname     -- Collect Host Name (optional)
       -ID id             -- Collect ID, if 0 : 
                             -- if host is given - use max id used by host
                             -- otherwise, use max (last) id automatically
       -skip1 yes/no      -- Yes/No skip first STAT measurement (often wrong values)
       -file Filename     -- File with STATs outputs
       -verbose on/off    -- verbose output on/off

Example :

$ /apps/ADMIN/BatchLOAD -cmd NEW -ID 0 -base ANT -file `pwd`/vmstat.out -skip1 no -title "Test BatchLOAD" -host V880 -isec 20 -start 20031024100000
$ /apps/ADMIN/BatchLOAD -cmd ADD -ID 0 -base ANT -file `pwd`/iostat.out -skip1 no
$ /apps/ADMIN/BatchLOAD -cmd ADD -ID 0 -base ANT -file `pwd`/mpstat.out -skip1 no -verbose on

In this example the first line will create a new STAT Collect using an automatic new ID (max+1), with the title "Test BatchLOAD" and it will load the first file: "vmstat.out" The second and third lines load into the new Collect the next data, "iostat.out" and "mpstat.out". Once it is finished, we can connect to the dim_STAT web server and start to analyze.

Note : multiple "-file" options can be used at the same time. For example:

$ /apps/ADMIN/BatchLOAD -cmd NEW -ID 0 -base ANT -skip1 no -title "Test BatchLOAD" 
        -host V880 -isec 20 -start 20031024100000 -file `pwd`/vmstat.out 
        -file `pwd`/mpstat.out -file `pwd`/iostat.out 

File Format of STAT output

The file format is designed in such a way as to give maximum flexibility on data grouping and processing.

The main TAGs are STAT and END:

==> STAT StatName                  -- after this point all following data corresponds
                                      to given STAT command (StatName)
    Supported STAT names: 
        VMSTAT
        MPSTAT
        IOSTAT (iostat -x)
        IOSTAT-xn (iostat -xn)
        VXSTAT (vxstat -v)
        psSTAT


    And all other Add-On STAT you are able to create,
    like some already shipped:


        netLOAD
        T3stat
        oraEXEC
        oraIO
        ...

==> END                            -- end of STAT data

At any time the following TAGs may also be inserted:

==> DTSET yyyy-mm-dd hh:mi:ss      -- set date+time point for next STAT data

==> LOGMSG message                 -- add log message into database corresponding
                                      to the currently loading data

Outside of the "STAT" - "END" blocks, any other lines are ignored.

Note : TAGs are exactly as it shown: "==> STAT", "==> END", "==> DTSET", "==> LOGMSG". Don't miss any characters!

BatchLOAD Example

A small example, let's say you have three vmstat and three iostat files corresponding to let's say "morning", "day" and "night" activity for some special tasks. Therefore you can make six load files, each one containing its own "STAT", "DTSET", "END" TAGs, or put all in one.

...
==> DTSET 2004-01-19 10:30:00                  -- set "morning" point
==> LOGMSG Morning workload
==> STAT VMSTAT                                -- load vmstat
  ... output of vmstat.out1
==> LOGMSG Strange CPU activity                -- marking time period to analyze (example)
  ... continue ...
==> END                                        -- end of first vmstat

==> STAT IOSTAT-xn
  ... output of iostat.out1
==> END

==> DTSET 2004-01-19 14:30:00                  -- set "day" point
==> LOGMSG Day workload
==> STAT VMSTAT            
  ... output of vmstat.out2
==> END                    

==> STAT IOSTAT-xn
  ... output of iostat.out2
==> END

==> DTSET 2004-01-19 23:30:00                  -- set "night" point
==> LOGMSG Night workload
==> STAT VMSTAT            
  ... output of vmstat.out3
==> END                    

==> STAT IOSTAT-xn
  ... output of iostat.out3
==> END

All information is placed in one single file ready to load:

$ /apps/ADMIN/BatchLOAD -cmd NEW -ID 0 -base ANT -skip1 no -title "Customer Workload"
        -host V880 -isec 20 -start 20040119100000  -file `pwd`/all_stat.out

In the same way, you can group all data of the same STAT command in a single file. Or all outputs corresponding to the same collecting time period.

NOTE : don't forget to create your database before starting any load!! In this example the database name is 'ANT'.


Special NOTE

Please, take care - there is no option to give a name of loaded stat command! That's why "STAT" and "END" tags are mandatory!. Even you want to load just one vmstat file, tool have no idea about your file contents till it'll meet a "STAT" tag inside!




GUDs integration

If you already worked with Sun support or you're Sun employe - you may know or already used GUDs (shell script collecting various system information + stats and saving them into special case archive). GUDs was created by Sun France engineer, and another French engineer made an integration script to load GUDs data into dim_STAT via BatchLOAD - 'guds2dim.sh'. This script is shipped now with dim_STAT and may be found in /apps/ADMIN directory. To obtain GUDs script - please, contact directly Sun support.