We currently have all polling located on dedicated polling agent servers. While polling servers can still collect data when the STM server is off-line due to database maintenance this is still an issue:
1) We cannot view reports or perform analysis.
2) Alerting for performance related events cannot be escalated (agents can be pointed at other secondary SNMP receivers, however this requires another tool, is not centralised and adds management overhead).
3) STM has to work hard to catchup with the agents (3+ hours of backlog, more often than not STM and the agents must be restarted).
We ensure that the STM DB has dedicated high IO available. However, the full maintenance process still takes a an extended period of time.
Moving away from the MyISAM engine (even to InnoDB/XtraDB on MySQL/MariaDB) would be beneficial. Not just for DB maintenance, but also to resolve a lot of the DB locking issues that occur due to day-to-day STM activities and reports.
There is currently always a need to optimise the DB for disk space management reasons. This is because MyISAM does not automatically compact files on disk to account for deleted records which causes disk utilisation to needlessly grow..