We had an issue last weekend whereby one of our servers triggered a monitor set which had been applied to it by a policy. Now this usually wouldn't be a problem, however this monitor set was a service check. It resulted in us returning to work on Monday morning with around 4,000 tickets in our AutoTask queue. This was due to the monitor set being triggered every 5 minutes and sending an email as configured.
I'm sure there's a way to create an agent procedure to restart these services upon failure, but I can't find a way to link a monitor set and an agent procedure together with my limited knowledge. Would some kind soul be able to help me in this? What we're aiming for is a monitor set which watches over a set of given services, when these services stop it then triggers a procedure to restart them. If this fails, it then raises an email as per normal.
Easiest way to do this is in policy management.
When you assign a monitor set, you can choose several options:
This is actually in the monitor set itself. When adding the service there is a box called "Re-start attempts"
So it is, i've even got it configured! I've rejigged the re-arm time to make it a bit less spammy in the future. Not sure why I didn't think of actually looking at the monitor set in the first place...