Welcome to Monitoring Blog: Icinga - Nagios Monday, March 18 2019 @ 03:42 PM UTC

Icinga 1.10.2 IDO2DB dump latency

  • Contributed by:
  • Views: 43,421


This will be a short article about the migration of an Icinga 1.9.3 infrastructure to a 1.10.2 version, with ido2db/mysql update too.

The update is classic, and as there are a lot of (official) docs about the upgrades, I won't explain it here.

But there is one thing to know, which is missing in the Mysql update schema.

After the upgrade, I had these symptoms:
- Icinga core is starting up as quickly as before
- Icinga-web and NagiosBP takes hours to show something normal  (icinga-web shows nothing and NagiosBP is showing everything as  critical)
- Ido2db give me lines like this in syslog:
Feb 11 16:33:48 HOST ido2db: IDO2DB proxy stats (p=0x1571de0): left=60535614 bytes, right=0 bytes; iostats=80588552 bytes
-  Icinga is on a cluster (bi-xeon E5-2630), the Mysql db is on a separate  T4 server with lot of power and nearly all tables in ram. There are 600  hosts, 17k services and about 120 users.
- On the mysql, as for debug, I can view lot of process like:
|  1267016 | nagios | icinga:46902      | nagios | Query   | 0     |  Updating | UPDATE icinga_commenthistory SET comment_type=2,  entry_type=2, object_id=19961, author_name='user1',  |
- This didn't happen with 1.9.3 (well, these symptomes lasted for about 20 seconds)

So why?
Well, it happens on some specific configurations, and it is because there is an index missing for the icinga_commenthistory table.

Create the index
CREATE INDEX commenthistory_delete_idx ON icinga_commenthistory (instance_id, comment_time, internal_comment_id);

This solved the problem!