Solution to bypass Distributed Monitoring in Zabbix

Jeez NO BitCoins ... So, PLEASE .. If this helped you in any way and you have some spare BitCoins, you may donate them to me :-). If you don't have any, please buy some and donate them to me....  - 16tb2Rgn4uDptrEuR94BkhQAZNgfoMj3ug


I started with distributed monitoring with Zabbix, but after I did my

• Zabbix Certified Specialist course is intended for those who have just begun to understand Zabbix and want to get on track fast and in the right way. During the course you will learn about the main functionality of Zabbix monitoring software, its installation, setup and further maintenance.
   
• Zabbix Certified Professional course is designed for administrators of large enterprises and companies that use Zabbix to monitor large amounts of devices located in multiple datacenters.

courses, to my dismay I found out that Zabbix might not support DMs anymore. I suppose the solution is to go to proxies, but it just seemed to be a bit of a slepph to me. So, I decided to write some scripts.
Place a 'master' script in crontab, query the Zabbix database for all the active alarms and get the Master Zabbix server to pull these values and alarm on them.
Quick and dirty solution, but for now it works for me.

So lets get started, create this script below, I use the directory /etc/zabbix/scripts, where you place yours is up to you. I use Postgres for my Zabbix databases in the various countries, but I'm sure to adopt the sql for some other database will not be that difficult...

#!/bin/bash

# This script will fire various sql queries to get the different priority 

# alarms in the Zabbix database which are active at the moment.
# The resulting logfiles's (alarmlogs) owner will then be changed to Zabbix, 

# so that the Zabbix agent can read the files
# The Items and Triggers are then fired by the Master Zabbix server 

# indicating the amount of alarms on the Zabbix Server in country
# This script will fix the problem with Distributed Monitoring, where the 

# Master Zabbix server does not get updated from the in-country
# Zabbix servers

# King Rat 25 July 2014

zabdir="/etc/zabbix/scripts/*.alarmlog"
for d in $zabdir
do
 echo " priority |        host        |                       description                        |      date_time      
----------+--------------------+----------------------------------------------------------+---------------------" > $d
chown postgres:postgres $d
done

if [ -f /etc/zabbix/scripts/all_alarms.alarmlog ]; then
 chown postgres:postgres /etc/zabbix/scripts/all_alarms.alarmlog
else
 touch /etc/zabbix/scripts/all_alarms.alarmlog
 chown postgres:postgres /etc/zabbix/scripts/all_alarms.alarmlog
fi

su - postgres -c "psql -d BOSS -U zabbix_server -f /etc/zabbix/scripts/priority5.sql -o /etc/zabbix/scripts/priority5alarms.alarmlog"
su - postgres -c "psql -d BOSS -U zabbix_server -f /etc/zabbix/scripts/priority4.sql -o /etc/zabbix/scripts/priority4alarms.alarmlog"
su - postgres -c "psql -d BOSS -U zabbix_server -f /etc/zabbix/scripts/priority3.sql -o /etc/zabbix/scripts/priority3alarms.alarmlog"
su - postgres -c "psql -d BOSS -U zabbix_server -f /etc/zabbix/scripts/priority2.sql -o /etc/zabbix/scripts/priority2alarms.alarmlog"
su - postgres -c "psql -d BOSS -U zabbix_server -f /etc/zabbix/scripts/priority1.sql -o /etc/zabbix/scripts/priority1alarms.alarmlog"
su - postgres -c "psql -d BOSS -U zabbix_server -f /etc/zabbix/scripts/all_alarms.sql -o /etc/zabbix/scripts/all_alarms.alarmlog"

for d in $zabdir
do
 chown zabbix:zabbix $d
done

This script is fired from crontab every 5 minutes, this line below must be placed in root's crontab

*/5 * * * * /etc/zabbix/scripts/get_zabbix_alarms.sh

The main script get_zabbix_alarms.sh fires the following sql scripts
priority5.sql, priority4.sql, priority3.sql, priority2.sql, priority1.sql, all_alarms.sql
The scripts check_*.sh is fired by the Master Zabbix server towards the Zabbix server in country.
The *.sh scripts must be made executable with chmod +x *.sh and the following owners must be set on the scripts, see below

srv1:/etc/zabbix/scripts # ls -ltr
total 72
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority5.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority4.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority3.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority2.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority1.sql
-rwxr-xr-x 1 root     root     2038 Jul 25 07:45 get_zabbix_alarms.sh
-rwxr-xr-x 1 zabbix   zabbix    374 Jul 25 07:45 check_war.sh
-rwxr-xr-x 1 zabbix   zabbix    394 Jul 25 07:45 check_info.sh
-rwxr-xr-x 1 zabbix   zabbix    376 Jul 25 07:45 check_high.sh
-rwxr-xr-x 1 zabbix   zabbix    386 Jul 25 07:45 check_dis.sh
-rwxr-xr-x 1 zabbix   zabbix    380 Jul 25 07:45 check_ave.sh
-rw-r--r-- 1 postgres postgres  893 Jul 25 07:45 all_alarms.sql
srv1:/etc/zabbix/scripts #

Below is what the priority*.sql scripts look like, make 6 files, named
priority5.sql, priority4.sql, priority3.sql, priority2.sql, priority1.sql and all_alarms.sql

Change the
t.priority
line to 5 for the priority5.sql
line to 4 for the priority4.sql
line to 3 for the priority3.sql
line to 2 for the priority2.sql
line to 1 for the priority1.sql
line to t.priority>=0 for the all_alarms.sql

srv1:/etc/zabbix/scripts # cat priority5.sql
--SQL script to get all priority 5 alarms
SELECT DISTINCT
case
 when t.priority = 0 then 'Not classified'
 when t.priority = 1 then 'Information'
 when t.priority = 2 then 'Low '
 when t.priority = 3 then 'Medium '
 when t.priority = 4 then 'High '
 when t.priority = 5 then 'Emergency'
 else 'Error no priority found'
end priority,
h.host,
t.description,
to_char(to_timestamp(t.lastchange),'YYYY-MM-DD HH24:MI:SS') date_time
FROM
zabbix_server.triggers AS t ,
zabbix_server.hosts AS h ,
zabbix_server.items AS i ,
zabbix_server.functions AS f ,
zabbix_server.acknowledges
Inner Join zabbix_server.events ON events.eventid = acknowledges.eventid
WHERE
t.priority=5
AND
f.itemid=i.itemid
AND h.hostid=i.hostid 
AND t.triggerid=f.triggerid
AND t.status=0
AND i.status=0
AND h.hostid not in (-1) 
AND h.status=0 
AND ((t.value=1)) 
AND t.value<>2
order by 1 desc, t.description
srv1:/etc/zabbix/scripts #

And the sql produces a log (*.alarmlog) with the contents below

srv1:/etc/zabbix/scripts # cat priority4alarms.alarmlog
 priority | host |                    description                     |      date_time     
----------+------+----------------------------------------------------+---------------------
 High     | srv1 | Free disk space is less than 10% on volume /backup | 2014-07-24 02:03:32
(1 row)
 
srv1:/etc/zabbix/scripts #

The scripts that are fired by the Master Zabbix server, the check*.sh scripts looks like this below

srv1:/etc/zabbix/scripts # cat check_high.sh
#!/bin/bash
# This is the script that gets fire by the Zabbix Master server to 

# list the High Priority alarms
# King Rat 25 July 2014
 
if [ `cat /etc/zabbix/scripts/priority4alarms.alarmlog | grep "row" | wc -l` -gt 0 ]; then
    cat /etc/zabbix/scripts/priority4alarms.alarmlog | grep "row" | awk '{print substr($1,2,4)}'
else
    echo "No High priority alarms"
fi
 
srv1:/etc/zabbix/scripts #

You would need 5 scripts, check_war.sh, check_info.sh, check_high.sh, check_dis.sh and check_ave.sh, just change the alarmlog name in the sh scripts to suit the priority of the alarm

Run the get_zabbix_alarms.sh manually to see if it works, if you get a Postgres password failure, add the user and password to the /opt/directory_where_postgres_resides/.pgpass

srv1:/etc/zabbix/scripts # ./get_zabbix_alarms.sh
Password for user zabbix:
Session terminated, killing shell... ...killed.
Password for user zabbix:
Session terminated, killing shell... ...killed.
Password for user zabbix:
Session terminated, killing shell... ...killed.
Password for user zabbix:
Session terminated, killing shell... ...killed.
Password for user zabbix:
Session terminated, killing shell... ...killed.
Password for user zabbix:
Session terminated, killing shell... ...killed.
srv1:/etc/zabbix/scripts #
srv1:/etc/zabbix/scripts #

localhost:5432:*:zabbix:the_password
And try again

srv1:/etc/zabbix/scripts # ./get_zabbix_alarms.sh
srv1:/etc/zabbix/scripts # ls -ltr
total 72
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority5.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority4.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority3.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority2.sql
-rw-r--r-- 1 postgres postgres  884 Jul 25 07:45 priority1.sql
-rwxr-xr-x 1 root     root     2038 Jul 25 07:45 get_zabbix_alarms.sh
-rwxr-xr-x 1 zabbix   zabbix    374 Jul 25 07:45 check_war.sh
-rwxr-xr-x 1 zabbix   zabbix    394 Jul 25 07:45 check_info.sh
-rwxr-xr-x 1 zabbix   zabbix    376 Jul 25 07:45 check_high.sh
-rwxr-xr-x 1 zabbix   zabbix    386 Jul 25 07:45 check_dis.sh
-rwxr-xr-x 1 zabbix   zabbix    380 Jul 25 07:45 check_ave.sh
-rw-r--r-- 1 postgres postgres  893 Jul 25 07:45 all_alarms.sql
-rw-r--r-- 1 zabbix   zabbix     98 Jul 25 13:34 priority5alarms.alarmlog
-rw-r--r-- 1 zabbix   zabbix    507 Jul 25 13:34 priority4alarms.alarmlog
-rw-r--r-- 1 zabbix   zabbix    245 Jul 25 13:34 priority3alarms.alarmlog
-rw-r--r-- 1 zabbix   zabbix   2589 Jul 25 13:34 priority2alarms.alarmlog
-rw-r--r-- 1 zabbix   zabbix    432 Jul 25 13:34 priority1alarms.alarmlog
-rw-r--r-- 1 zabbix   zabbix   3406 Jul 25 13:34 all_alarms.alarmlog
srv1:/etc/zabbix/scripts # date
Fri Jul 25 13:35:02 SAST 2014
srv1:/etc/zabbix/scripts #

On the Master Zabbix server I created a template that I then assign to the hosts (Remote Zabbix servers) that I want to monitor

And these are the triggers

Remember to set your zabbix_agentd.conf to accept the incoming requests from the Master Zabbix Server and of course restart the agent once done

REMEMBER those BitCoins....