nuknad.com

Nagios is one of those pretty essential tools for managing rather large systems. Having been around for almost 10 years now, most of the work has been focused around getting a stable capable system. The user interface has taken a bit of a back seat, staying a little clunky in true web 1.0 style, while the rest of the web has geared up into more interactive and usable interfaces. Ninja, a frontend developed by Op5, is looking to change this and bring Nagios up to speed. Enter the Ninja.

Op5 sells custom monitoring SaaS based off Nagios and have made the majority of their code public and accessible. They have made a huge contribution to Nagios in their release of Ninja and while not their only contribution, Ninja offers a new interface with customizable components and widgets. These features have set it on a path to become the new face of Nagios. It has a shiny interface with some new features and reporting.

One of my favourite features, was unfortunately also one of the hardest features to get working. Geomap is a flash application that sits on the dashboard (Tactical Overview) and lets you visualize your network status across the globe. You can setup nodes and assign them to points on the map, and create connections between points. This way, a quick trip to the Dash can tell you if an international connection has gone down or whether one of your routers is causing an outage for several machines.

The complete setup can be a little tricky and involves following multiple guides which you just copy and paste. The geomap is still in beta, and so there is no guide for it at all. So, if you want to give it a go, I suggest firing up an Ubuntu EC2 machine and following along to get going. For ease of access I have the entire script at pastebin

The first step to get going is to get the basic nagios install going. The following little snippet will install the dependencies (apache+some libraries), add a nagios user, install nagios and setup some basic auth for apache. Note that you could also install Nagios from your package manager if you want, but if it installs to a different location, just take note. You’ll also need to check the users/groups are setup

# install our dependencies
apt-get install --assume-yes apache2
apt-get install --assume-yes libapache2-mod-php5
apt-get install --assume-yes build-essential
apt-get install --assume-yes libgd2-xpm
apt-get install --assume-yes libgd2-xpm-dev

# stop apache running since we're going to reconfigure it 
/etc/init.d/apache2 stop

# add our nagios user
/usr/sbin/useradd -m -s /bin/bash nagios

# set the password
passwd nagios

# add the nagios user to the apache group (www-data)
/usr/sbin/groupadd nagcmd
/usr/sbin/usermod -a -G nagcmd nagios
/usr/sbin/usermod -a -G nagcmd www-data

# let's fetch nagios + plugins and install

mkdir nagiosInstalls && cd nagiosInstalls
wget --quiet http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.1.tar.gz
wget --quiet http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.15.tar.gz
 
tar xzf nagios-3.2.1.tar.gz && cd nagios-3.2.1
./configure --with-command-group=nagcmd
make all
make install
make install-init
make install-config
make install-commandmode
make install-webconf
cd ..

# optional - change email address in nagios config to your email
sed -i 's/nagios@localhost/nagiosadmin@yourcompany.com/' /usr/local/nagios/etc/objects/contacts.cfg

# setup htaccess for the front-end. If you want to skip this you can, but just remember to
# edit /etc/apache/conf/nagios.conf and remove the basic auth section

htpasswd -b -c /usr/local/nagios/etc/htpasswd.users nagiosadmin nagiosadmin

# install the plugins

tar xzf nagios-plugins-1.4.15.tar.gz && cd nagios-plugins-1.4.15
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make && make install
cd ..

Next up we want to install Merlin (another contribution from Op5). Merlin converts nagios information into a database where each entity is a table. This offers good flexibility and scalability, the kind of things we’re looking for when monitoring large networks!

Unfortunately Merlin hasn’t been packaged on any distro that I know of yet, so for now we’ll need to install from source.

# first up, install mysql, if already installed, skip this...

# set some debconf stuff for mysql so we don't get asked too many questions, this is optional...
# NB: change your_password to whatever you want your password for mysql to be.
cat <<EOF | debconf-set-selections
debconf debconf/frontend select Readline
mysql-server-5.0 mysql-server/root_password_again your_password
mysql-server-5.0 mysql-server/root_password your_password
mysql-server-5.0 mysql-server-5.0/really_downgrade boolean false
mysql-server-5.0 mysql-server-5.0/need_sarge_compat boolean false
mysql-server-5.0 mysql-server-5.0/start_on_boot boolean true
mysql-server-5.0 mysql-server-5.0/nis_warning note
mysql-server-5.0 mysql-server-5.0/postrm_remove_databases boolean false
mysql-server-5.0 mysql-server-5.0/need_sarge_compat_done boolean true
EOF

apt-get install --assume-yes mysql-server

# merlin will need these libraries if you don't have them installed
apt-get install --assume-yes libapache2-mod-php5 libdbi0 libdbi0-dev libdbd-mysql php5-cli php5-mysql

# READ - DON'T JUST PASTE
# this section we login to our database and create a new table, and a new user, and grant the privileges accordingly

mysql -u root -e 'create database merlin'
mysql -u root -e "grant all privileges on merlin.* to merlin@localhost identified by 'merlin'"
mysql -u root -e 'flush privileges'

# finally, install merlin

wget --quiet http://www.op5.org/op5media/op5.org/downloads/merlin-0.6.8.tar.gz
tar -zxvf merlin-0.6.8.tar.gz && cd merlin-0.6.8
make

# Note: You'll probably see an error message at the end. I'm not sure if everyone gets it, but i did and it didnt have an impact on anything.
./install-merlin.sh --nagios-cfg=/usr/local/nagios/etc/nagios.cfg --dest-dir=/usr/local/nagios/addons/merlin --batch || echo #merlin install always fails, but not really.

cd ..

Next up is ninja. This is where things start getting a little harder and we have to juggle some config files.
Install paths matter hugely, if you installed nagios different, be sure to check the following snippet carefully.

# download ninja and the reports module
wget --quiet http://www.op5.org/op5media/op5.org/downloads/ninja-1.0.1.tar.gz
wget --quiet http://www.op5.org/op5media/op5.org/downloads/reports-module-2.0.10.tar.gz

# installing ninja is just copying the folder across (it's a plugin for nagios)

tar -zxvf ninja-1.0.1.tar.gz
cp -a ninja-1.0.1 /usr/local/nagios/addons/ninja

# now we need to modify some configs

pushd /usr/local/nagios/addons/ninja
sed -i 's~$merlin_path = .*$~$merlin_path = "/usr/local/nagios/addons/merlin";~g' install_scripts/auth_import_mysql.php
sed -i 's~$nagios_cfg_path = .*$~$nagios_cfg_path = "/usr/local/nagios/etc";~g' install_scripts/auth_import_mysql.php
sed -i 's/\/bin\/awk/\/usr\/bin\/awk/g' install_scripts/auth_import_mysql.php
install_scripts/ninja_db_init.sh /usr/local/nagios/addons/ninja
popd

# install reports module
tar zxf reports-module-2.0.10.tar.gz
cp -a reports-module-2.0.10 /usr/local/nagios/addons/reports-module
pushd /usr/local/nagios/addons/reports-module

# ninja also needs graphviz
apt-get install --assume-yes php5-gd graphviz

# now we need the libmysql
failed=0
apt-get install --assume-yes libmysqld-dev || failed=1
if [ $failed == 1 ]
then
        echo "WARNING: Failed to isntall the right mysql development files. Trying to find right package"
        p=`apt-cache search libmysql | grep 'database development files' | awk '{print $1}'
        echo "Installing $p"
        [ "x$p" != "x" ] && apt-get install $p
fi

# we're getting ready to run the ninja setup script which integrates itself with nagios and merlin
# before we do that, we need to change some paths

sed -i 's~mod_path=/opt/monitor/op5/reports/module~mod_path=/usr/local/nagios/addons/reports-module~g' scripts/setup.sh
sed -i 's~prefix=/opt/monitor~prefix=/usr/local/nagios~g' scripts/setup.sh
sed -i 's~php $mod_path/find_configured.php \\~~g' scripts/setup.sh
sed -i 's~> /tmp/$name.interesting~~g' scripts/setup.sh
sed -i 's~archived="/opt/monitor/var/archives/nagios-*.log"~archived="$prefix/var/archives/nagios-*.log"~g' scripts/setup.sh
sed -i 's~nagioslog=/opt/monitor/var/nagios.log~nagioslog=$prefix/var/nagios.log~g' scripts/setup.sh
sed -i 's~/etc/rc.d/init.d/monitor start~~g' scripts/setup.sh

# run it!

bash scripts/setup.sh
make

# install reports into the db
mysql monitor_reports < /usr/local/nagios/addons/ninja/install_scripts/reports.sql

# change an installed config

cd ../ninja/application/config/
sed -i '/nagios_base_path/ s~/opt/monitor~/usr/local/nagios~g' config.php
popd

Ok, so we’ve got Nagios, Ninja and Merlin installed. Note that your apache isn’t configured yet, so don’t worry that you can’t load up ninja or nagios just yet!

This next step is to get Geomap working. As I said in the opening paragraph, Geomap is a funky little contribution which maps hosts/connections on a world map.

If you want geomap working, this next snippet is for that!

Currently, nagvis does not have the bits and pieces we need. Op5 has a custom build which does. So this next step involves taking nagvis from op5-monitor (Nagvis falls under the GPL and Op5 have kindly said that everything is fine)

# We need to rip nagvis from the rpm
apt-get install --assume-yes cpio rpm
wget http://download.op5.com/shop/op5-monitor-software-install-latest.tar.gz
tar -xzvf op5-monitor-software-install-latest.tar.gz && pushd monitor-software*
cd rpm
nv=`ls | grep nagvis`
rpm2cpio $nv | cpio -dimv
cd opt/monitor/op5/nagvis/
 
# install nagvis 
chmod u+x install.sh
./install.sh -n /usr/local/nagios -B /usr/local/nagios/bin/nagios -b /usr/local/bin -p /usr/local/nagios/addons/nagvis -u nagios -g nagcmd -w /tmp/ -i merlinmy -q || echo

# fix up some permissions and configs
chmod g+wx /usr/local/nagios/addons/nagvis/var/
cd /usr/local/nagios/addons/nagvis/etc
sed -i 's~base="/opt/monitor/op5/nagvis/"~base="/usr/local/nagios/addons/nagvis/"~g' nagvis.ini.php
chmod g+w /usr/local/nagios/addons/nagvis/etc/geomap/*
popd

Ok, whew, this is quite a process. Everything is installed and waiting, now we have to do now is get our apache setup running. A bit of an annoyance is that geomap needs some nagvis stuff behind ssl. So we need to generate a certificate

We need /etc/ssl/server.crt and /etc/ssl/server.key.insecure. You can read up on generating certificates here or here

#enable apache ssl and rewrite
a2enmod ssl
a2enmod rewrite

#backup old apache conf.d directory
mv /etc/apache2/conf.d /etc/apache2/conf.d.pre_nagios
mkdir /etc/apache2/conf.d

# if you know you're ip, just set ip= instead of the following attempt to auto-detect your ip
ip_list=`ifconfig | grep 'inet addr' | tr ':' ' ' | awk '{print $3}'`
ip=${ip_list%%[^0-9\.]*}
[ "x$ip_list" != "x$ip" ] && echo "WARNING : Multiple IP Addresses Found, using first one for apache configs."

# let's create our apache config, you might want yours to be different. This is just something to get you going

cat > /etc/apache2/sites-available/nagios.conf << EOF
<IfModule !mod_alias.c>
        LoadModule alias_module modules/mod_alias.so
</IfModule>
 
NameVirtualHost *:80
 
<VirtualHost $ip:80>
 
 
ScriptAlias /nagios/cgi-bin "/usr/local/nagios/sbin"
 
<Directory "/usr/local/nagios/sbin">
   Options ExecCGI
   AllowOverride None
   Order allow,deny
   Allow from all
   AuthName "Nagios Access"
   AuthType Basic
   AuthUserFile /usr/local/nagios/etc/htpasswd.users
   Require valid-user
</Directory>
 
Alias /nagios "/usr/local/nagios/share"
 
<Directory "/usr/local/nagios/share">
   Options None
   AllowOverride None
   Order allow,deny
   Allow from all
   AuthName "Nagios Access"
   AuthType Basic
   AuthUserFile /usr/local/nagios/etc/htpasswd.users
   Require valid-user
</Directory>
 
 
Alias /nagvis "/usr/local/nagios/addons/nagvis/share"
 
<Directory "/usr/local/nagios/addons/nagvis/share">
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  Allow from all
 
  AuthName "NagVis Access"
  AuthType Basic
  AuthUserFile /usr/local/nagios/etc/htpasswd.users
  Require valid-user
 
  <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /nagvis
   
    RewriteCond %{REQUEST_URI} ^/nagvis(/index\.php|/|)(\?.*|)$
    RewriteRule ^(index\.php|)(\?.*|)$ /nagvis/frontend/nagvis-js/$1$2 [R=301,L]
    RewriteCond %{REQUEST_URI} ^/nagvis/config\.php.*$
    RewriteRule ^config\.php(.*) /nagvis/frontend/wui/$1 [R=301,L]
   
    RewriteCond %{REQUEST_URI} ^/nagvis/frontend/nagvis-js
    RewriteCond %{QUERY_STRING} map=(.*)
    RewriteRule ^(.*)$ /nagvis/frontend/nagvis-js/index.php?mod=Map&act=view&show=%1 [R=301,L]
 
    RewriteCond %{REQUEST_URI} ^/nagvis/frontend/wui
    RewriteCond %{QUERY_STRING} map=(.*)
    RewriteRule ^(.*)$ /nagvis/frontend/wui/index.php?mod=Map&act=edit&show=%1 [R=301,L]
 
    RewriteCond %{REQUEST_URI} ^/nagvis/frontend/nagvis-js
    RewriteCond %{QUERY_STRING} !mod
    RewriteCond %{QUERY_STRING} rotation=(.*)
    RewriteRule ^(.*)$ /nagvis/frontend/nagvis-js/index.php?mod=Rotation&act=view&show=%1 [R=301,L]
  </IfModule>
</Directory>
 
        <IfModule !mod_alias.c>
                LoadModule alias_module modules/mod_alias.so
        </IfModule>
 
        Alias /ninja /usr/local/nagios/addons/ninja
        <Directory "/usr/local/nagios/addons/ninja">
                Order allow,deny
                Allow from all
                DirectoryIndex index.php
        </Directory>
 
</VirtualHost>
 
<VirtualHost $ip:443>
        SSLEngine On
        SSLCertificateFile /etc/ssl/server.crt
        SSLCertificateKeyFile /etc/ssl/server.key.insecure
       
        DocumentRoot /usr/local/nagios/addons/nagvis/share/netmap/
        Alias /nagvis "/usr/local/nagios/addons/nagvis/share/"
       
        RewriteEngine On
        RewriteRule ^crossdomain.xml$ nagvis/netmap/crossdomain.xml
 
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
 
 
        <Directory "/usr/local/nagios/addons/nagvis/share/">
                Order allow,deny
                Allow from all
                DirectoryIndex index.php
                AllowOverride None
        </Directory>
 
</VirtualHost>
EOF

LASTLY! and this is important. We need to fix up some permissions and start our services!

# Permissions

mkdir /usr/local/nagios/addons/ninja/application/logs || echo
chown nagios:nagcmd -R /usr/local/nagios/addons/
chmod g+wx /usr/local/nagios/addons/ninja/application/logs/

# Daemons
/etc/init.d/merlind stop
/etc/init.d/nagios stop
/etc/init.d/apache2 stop
sleep 10
/etc/init.d/nagios start
/etc/init.d/merlind start
/etc/init.d/apache2 start

That’s it. It’s quite a complicated process for now, but hopefully in the coming months, ninja will be integrated more with nagios and we’ll see the need for this process dissappear!

These scripts come with no guarentee and it’s recommended you don’t run anything on a production machine unless you know what you’re doing.