The dim_STAT installation package is either delivered as a TAR archive (dim_STAT.tar) or, when on CDs, already "untarred".

Before install: Verify your available diskspace - you will need ~60MB for the initial install, mostly to store Web Server and Database Server data. The database volume will grow according to the number of (future) STAT collections and the web directory may grow with your reports. So reserve enough space for your data ...

During installation: a new user "dim" and a group "dim" will be created. User "dim" is the owner of the dim_STAT database and the web server. In case your system has special rules or restrictions, you may create these manually beforehand, or you may choose other user and group names that are following your system policies. Please, after installation, don't forget to set a password to this user! (otherwise cron is not allowing execution of regular clean-up tasks via 'crontab')...

INSTALL.sh

As the root user, unload the tar archive into some directory and start the installation script:

 # cd /tmp
 # tar xvf /path_to_tar/dim_STAT.tar
 # cd dim_STAT-INSTALL
 #
 # INSTALL.sh

During installation you will be asked to confirm your host IP address (found automatically), host and domain name, the script verifies if the user "dim" already exists on the system, if not it will be created, and you will be asked about WebX and home directories (Web Server, Database Server, Administration and Client scripts, etc.) and about port numbers to be used.

Mainly you have to choose 3 application directories:

• WebX home (default: /opt/WebX)
• Data home (default: /apps)
• Temporary space (default: /tmp)

And a user/group name which will be the owner of the dim_STAT data in your system (default: 'dim')

If you are not sure about the meaning of some values, leave them by default.

NOTE: WebX is the main interpreter (or execution engine), it interprets all application script files and absolutely needs a fixed and trusted root (home) directory. Otherwise, anyone may execute whatever they want on your machine (like /etc/passwd to crack logins, etc.). So, as a first step protection for its root directory: you may choose one of 4 available paths (hey, 4 choices anyway, better then one :) ). Also, the WebX engine itself is very small (only a few MB) and not growing.

After install, the dim_STAT software will be distributed on your system in the following way:

    + /WebX, /apps/WebX, /opt/WebX or /etc/WebX - WebX main directory (only 4 possibilities)
    |
    + /apps - default dim_STAT home directory
      |
      +-- /ADMIN      - administration scripts (start/stop dim_STAT Server, BatchLOAD, etc.)
      |
      +-- /mysql      - MySQL database server main directory
      |
      +-- /httpd      - Apache Web server directory
      |
      +-- /client     - client collect script(s)
      |
      +-- /Java2GIF   - Java applet graph to GIF convertor
      |
      +-- /htmldoc    - HTML to PDF converting tool
      |
      +-- ...         - there may be other directories depending on dim_STAT release :))

NOTE: To simplify things, the next examples assume that your home directory is '/apps' and owner's user name is 'dim'.




Silent INSTALL

Since version 8.1 there is a silent "auto install" feature integrated in the install script. It may be very useful in case you need to automate the installation of dim_STAT on your servers. To activate it, use the '-Auto yes' option.

Then add more options if you need to have any settings different from the default:

• -HOST `hostname`
• -IP ip_address
• -USER dim
• -GROUP dim
• -WebX_DIR /opt/WebX
• -TEMP_DIR /tmp
• -HOME_DIR /apps
• -HTTP_PORT 80
• -DB_PORT 3306
• -STAT_PORT 5000
• -USERADD yes (add user/group )
• -AutoLink yes (make auto-start links in /etc/rc*.d)

Examples :

  Default install:
   # ./INSTALL.sh -Auto yes


  With customized Home:
   # ./INSTALL.sh -Auto yes -HOME_DIR /export/home/apps/dim_STAT


  With existing User:
   # ./INSTALL.sh -Auto yes -USER stat -GROUP staff -ADDUSER no -HOME_DIR /staff/stat


  etc...







Starting Web and Database servers

As you saw before, administration scripts are placed in /apps/ADMIN :

  # cd /apps/ADMIN 
  # dim_STAT-Server start

To stop servers:

  # cd /apps/ADMIN 
  # dim_STAT-Server stop

NOTE: a global dim_STAT-Server script is working as the main admin interface and replaces various separate httpd / mysql scripts. This global script also checks before a stop/start action if there are any active collects running and restarts them automatically during the next startup. Also, if the shutdown was not properly done, startup script will print a warning messages about a possible need of index rebuild on some databases...

At any moment you may look in the database for any active connections.

  $ su - root
  # /apps/mysql/bin/mysql -S /apps/mysql/data/mysql.sock
  
  mysql>
  mysql> show processlist;


    +------+------+-----------+----------+---------+-------+-------+------------------+
    | Id   | User | Host      | db       | Command | Time  | State | Info             |
    +------+------+-----------+----------+---------+-------+-------+------------------+
    |    3 | dim  | localhost | Mind     | Sleep   | 18    | NULL  | NULL             |
    |    4 | dim  | localhost | Mind     | Sleep   | 17    | NULL  | NULL             |
    |    5 | dim  | localhost | Mind     | Sleep   | 2     | NULL  | NULL             |
    |    6 | dim  | localhost | Mind     | Sleep   | 1     | NULL  | NULL             |
    |    7 | dim  | localhost | Mind     | Sleep   | 2     | NULL  | NULL             |
    |    8 | dim  | localhost | Mind     | Sleep   | 16    | NULL  | NULL             |
    |    9 | dim  | localhost | Mind     | Sleep   | 104   | NULL  | NULL             |
    |   10 | dim  | localhost | Mind     | Sleep   | 1     | NULL  | NULL             |
    |   11 | dim  | localhost | Mind     | Sleep   | 0     | NULL  | NULL             |
    |   53 | dim  | localhost | UPC      | Sleep   | 108   | NULL  | NULL             |
    |   54 | dim  | localhost | UPC      | Sleep   | 103   | NULL  | NULL             |
    |   56 | dim  | localhost | UPC      | Sleep   | 115   | NULL  | NULL             |
    |   57 | dim  | localhost | UPC      | Sleep   | 118   | NULL  | NULL             |
    |   58 | dim  | localhost | UPC      | Sleep   | 112   | NULL  | NULL             |
    |   59 | dim  | localhost | UPC      | Sleep   | 105   | NULL  | NULL             |


    ...

and even kill any of them (however, be very careful !!)

  mysql> kill 57;
  mysql> quit
  Bye
  
  #
  #




MySQL Admin Tips

MySQL administration is very easy. However, depending on a user's past experience, here are some tips which may help...

First of all, be aware, dim_STAT is using MySQL MyISAM engine to save data. This engine has no transactions support nor transaction log, etc., but it's very easy to manage, it does all needed stuff quite well, providing a reasonable SQL interface, and keep all saved data fully platform-independent! (you may simply copy your data files from Linux/x86 to Solaris/SPARC station and continue to work with them without any problem!). Of course, without transaction log there is still a risk to loose some data due system crash or power outage... But if you'll put to the list of priorities all important points you'll see that loosing few minutes of collected data are much less important rather database software cost as well having skills to administrate it.. - you don't need any DBA skills to administrate MySQL for dim_STAT! UNIX admin habits will be enough :-)



As much as you can, use separated databases: it's much more easier for administration, it avoids possible future activity conflicts, etc. Since v.8.3 there is a possibility to add an Admin password while creating a new database - all administration action then will require giving this password (start/ stop/ restart of collects, data drop, etc.)



Limitation in the number of connections: each MySQL connection uses 5 file descriptors (avg). This means that with a maximum of 1024 file descriptors per process (default in some old systems), we can't create more than ~200 connections on a multi-threaded MySQL server (Note: each STAT command in collect uses its own single connection). In case you run dim_STAT server on Solaris and need more connections (several hosts, many stats, etc), first check the values of your /etc/system parameters : rlim_fd_cur and rlim_fd_max. Next, in the file /apps/mysql/mysql.server replace the default value of 2000 by a new one (current dim_STAT server is just configured with a limit of 2000 connections, however it depends on the system how much it'll be able to acquire, as well you may always increase again this value)...



Accidental "power off" on your machine: MySQL server within dim_STAT is configured in way to force data flush every 5 minutes. So, if your database was not used for a long time - your data should be safe.. However for active databases it's very possible some of their index files will be corrupted. The dim_STAT-Server script will print a warning message in this case, but you'll need to run manually the data checks..
NOTE: you do NOT need to stop dim_STAT server! :-)

Supposing you discovering some data errors on the database "Demo" (for example):

• First of all you stop all collects on this database (and check via 'Preferences' there is no connections anymore to this database)..
• Wait 15 minutes (MySQL will flush data and close files)
• Start "repair" MySQL command:

  # cd /apps/mysql/data/Demo
  # /apps/mysql/bin/myisamchk -r *.MYI
  

• Restart all your collects you previously stopped

Since v.8.2 auto-repair was removed from dim_STAT-Server script, because:

• Recovery process blocked all users from using database during whole recovery time..
  
  
• It's extremely difficult to say which table/database will need or not need a data recovery (even if it was closed properly it doesn't mean yet indexes were not corrupted - during system crash filesystem buffers may still stay dirty and not flushed to disk(s))..
  
  
• Finally the only running "myisamchk -r" gives you a true repair in this case and it may take a lot of time.

Since v.8.2:

• Every 5 minutes mysql daemon is forced to flush key buffers and close all table files - it's protecting at least non-active databases, their data normally will still stay stable in case of system crash!
  
  
• If system crash happens, MySQL server will still start correctly but with a warning message - probably some of the databases will need a data repair!..
  
  
• If you discover your database is broken:
   - stop all active collects on it
   - wait 5 minutes (within 5 minutes all your tables will be closed)
   - start recovery on your database (see above)
  
  
• This solution is give a way to recover databases in preferred by user order, as well leave other working (if they don't need to repair) or just create a new database and still continue your work!

Probably with a time for some critical system environment there will be a possibility to upgrade databases to InnoDB engine and not take care anymore about system crashes, but it's just a part of future plan for the moment :-)



No more disk space: just add disks if possible :). The collect part of dim_STAT is done in such a way to "keep the flow", in case of errors nothing will be stopped. Once you have added space, the collects will continue, but you probably will get some holes during this period.



To get a backup/copy of your collects in the fastest way: one of the great features of MySQL is its support of cross-platform data compatibility. As an example, the same database files may be moved from a Solaris machine and successfully reused on a Linux laptop. And most cases, copying the whole database to another machine will be much more faster than exporting and again importing collects via flat files. The exception is if you want to move only a very small amount of data from a large database.
Fine, but can we do this on-line? - Yes!! Like in "repair" steps:

• Stop all collects in your database
• Wait 15 minutes
• Backup the database (ex. "Demo"):

  # cd /apps/mysql/data /your_backup
  # cp -rp Demo /your_backup_path
  OR:
  # tar cf - Demo | gzip > /your_backup_path/Demo.tgz
  

• Restart all previously stopped collects...
• NOTE: since v.8.3 there is a web interface added to safely backup whole database.
  
  

Delete the database: there is no way to delete a database via the web interface (generally, I don't like deleting :) ). Delete by error is such a common thing ... so, if you really need to delete your database, the only way is:

• Check there is no more connections to your database
• Delete database files (ex. "Demo"):

  # rm -rf /apps/mysql/data/Demo
  

  
  
  

Running several MySQL instances on the same host: long time ago it was one of the bigger problems to avoid dim_STAT to conflict with already installed and running databases on an existing system. The solution I found is isolating the dim_STAT database completely from existing instances, but the price for it is a few more complexity for simple things. The tool now uses its own parameters for TCP/IP ports and UNIX sockets. For example, to connect locally to your database server, instead of the usual:

# /apps/mysql/bin/mysql DatabaseName

you should now use:

# /apps/mysql/bin/mysql -S/apps/mysql/data/mysql.sock DatabaseName




MySQL: datafile corruption

This section is covering a particular case when table is not repaired by "myisamchk", and usually you get a following message:

"table TABLE doesn't have a correct index definition" etc.

The solution is:

• Stop dim_STAT server
• Start only MySQL instance
• Connect to your database
• Execute CHECK, then REPAIR of your TABLE
• Stop MySQL instance
• Start dim_STAT server

The following example is demonstrating a real case with "dim_MPSTAT" table:

bash# /apps/ADMIN/dim_STAT-Server stop
  
bash# /apps/mysql/bin/myisamchk -r -f dim_MPSTAT.MYI
  
IF IT DID NOT HELP:
  
bash# /apps/mysql/mysql.server start
  
bash# mysql -S /apps/mysql/data/mysql.sock Benchmark_TTT
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
  
Didn't find any fields in table 'dim_MPSTAT'
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 1 to server version: 3.23.53
  
Type 'help;' or 'h' for help. Type 'c' to clear the buffer.
  
mysql> check table dim_MPSTAT;
+--------------------------+-------+----------+-------------------------------------------------------------+
| Table                    | Op    | Msg_type | Msg_text                                                    |
+--------------------------+-------+----------+-------------------------------------------------------------+
| Benchmark_TTT.dim_MPSTAT | check | warning  | Table is marked as crashed                                  |
| Benchmark_TTT.dim_MPSTAT | check | warning  | Size of datafile is: 1251942400       Should be: 1251942360 |
| Benchmark_TTT.dim_MPSTAT | check | error    | Found 16918142 keys of 16918140                             |
| Benchmark_TTT.dim_MPSTAT | check | error    | Corrupt                                                     |
+--------------------------+-------+----------+-------------------------------------------------------------+
4 rows in set (19.39 sec)
   
mysql> repair table dim_MPSTAT;
+--------------------------+--------+----------+----------+
| Table                    | Op     | Msg_type | Msg_text |
+--------------------------+--------+----------+----------+
| Benchmark_TTT.dim_MPSTAT | repair | status   | OK       |
+--------------------------+--------+----------+----------+
1 row in set (7 min 34.16 sec)
   
mysql>
   
bash# /apps/mysql/mysql.server stop
   
bash# /apps/ADMIN/dim_STAT-Server start

The doc reference is here (see comments) - http://dev.mysql.com/doc/refman/5.0/en/myisamchk-repair-options.html (Thanks Google! :-))


Using InnoDB Engine instead of MyISAM

Since dim_STAT v.9.0 it is possible to use InnoDB Storage Engine within MySQL instead of MyISAM. This Engine is a true transactional one and pretty safe against server power-off or system crashes.. You may choose to use this InnoDB instead of MyISAM on Database creation, or at any moment convert your Database from one Engine to another. The only thing you'll not be able to do with InnoDB is a full "physical" backup of your Database files (in this case you'll need to convert your Database to MyISAM first). However there is no problem with Import or Export.

NOTE: bigger is your database, more it'll take time to convert it from one Engine to another one..

Since v.9.0 to simplify DBA-like tasks there is an admin tool included: dim_STAT-Admin .


dim_STAT-Admin Tool

dim_STAT-Admin is shipped since v.9.0 to avoid to use a web interface for sometimes heavy DBA tasks.

With dim_STAT-Admin you're able from the command line:

• Create a new Database
• Convert existing Database to another Storage Engine
• Backup a whole Database
• Export STAT Collect(s)
• Import STAT Collect(s)
• Recycle STAT Collect(s)

Command line:

$ ./dim_STAT-Admin

dim_STAT-Admin CLI (dim) v.1.0
> Usage: dim_STAT-Admin  [options]
    Options:
       -CMD Command                Commands: CREATE, BACKUP, CONVERT, EXPORT, IMPORT, RECYCLE
       -Base DBname                Database Name  (if empty: prints database name list)
       ... 


    Additional options: (depending on Command)
      CREATE :
       -Engine Name                MyISAM (default) or InnoDB
       -Passwd PASSWORD            optional password setting for Admin actions 


      BACKUP :
       -Passwd PASSWORD            if password was assigned for Admin actions
       -File Filename              full path output file name for tar.Z backup file 


      CONVERT :
       -Engine Name                MyISAM or InnoDB
       -Passwd PASSWORD            optional password setting for Admin actions 


      EXPORT :
       -ID id1[,id2,..]            Collect ID(s) to export (if empty: prints available Collect list)
       -Begin YYYYMMDDhhmiss       optional begin date+time
       -End YYYYMMDDhhmiss         optional end date+time
       -File filename              full path output file name for tar.Z export file 


      IMPORT :
       -ID id1[,id2,..]            optional Collect ID(s) to import  (if known)
       -File filename              full path file name for input tar.Z import file 


      RECYCLE :
       -Days N                     keep data collected during last N days
       -ID CollectID               optional collect ids (ex: id1,id2,id3 or "ALL" for any ID)
                                   (if empty: All active collects only) 







Migration from any old dim_STAT version to the new one

The migration procedure is quite easy:



• Stop all activity on your current dim_STAT installation
• dim_STAT-Server stop
• Backup all your databases from '/apps/mysql/data/' (see below) except: dim_00, mysql and dim
  - mysql: system database, don't play with it !!
  - dim_00: is a reference database and changing with every release
  - dim: is a "Default" database, and if you really need it, rename it before backup
  
  
• Install the new dim_STAT distribution
• Restore your backup-ed data into '/apps/mysql/data'
• Start dim_STAT-Server

Enjoy :))

NOTE: The old database should be seen as before and work correctly, but if you want to get an advantage of the all new features coming within new version, then create a new database and start new collects.




First-Level Security

The main point: ANY SECURE SYSTEM IS NEVER SECURE ENOUGH... The question is only, what will you consider secure ENOUGH for you :))

Anyway, during discussions with our engineers and customers, the security issue was so often raised that I cannot leave it without attention.

For paranoia-users: there is a Solaris X86 or Linux version of dim_STAT and if you really need maximum protection, spend some money on a small dedicated PC, run dim_STAT on it and protect any access with firewalls, etc.

In my experience, I suggest to protect access to the web server, to prevent somebody from just by error stopping or suspending active collects. For this kind of first-level access protection, a good candidate is Apache's ".htaccess". For a more detailed information, please refer to the Apache documentation. But in short, just to make it work with dim_STAT:

• 1) via /apps/httpd/bin/htaccess create /apps/httpd/etc/.htpasswd file and add any pairs of user/password you need
• 2) create ".htaccess" file with context:

    AuthName "Welcome to dim_STAT Host"
    AuthType Basic
    AuthUserFile /apps/httpd/etc/.htpasswd
  
  
    require valid-user
  

• 3) copy ".htaccess" file into /apps/httpd/home/docs and /apps/httpd/home/cgi-bin
• 4) try to connect to your web server now and check the access user/password - that's all! ;-)

Example:

  $ /apps/httpd/bin/htpasswd
  
  Usage: htpasswd [-c] passwordfile username
  The -c flag creates a new file.
   
  $ /apps/httpd/bin/htpasswd -c /apps/httpd/etc/.htpasswd  login1
    Password:
    ...
   
  $ vi /tmp/.htaccess
  $ cat /tmp/.htaccess
  AuthName "Welcome to dim_STAT Host"
  AuthType Basic
  AuthUserFile /apps/httpd/etc/.htpasswd
   
  require valid-user
  $
  $ cp /tmp/.htaccess /apps/httpd/home/cgi-bin
  $ cp /tmp/.htaccess /apps/httpd/home/docs