Kaseya Community

How do you handle offline alerts after hours?

  • Was just wondering how everyone handles offline alerts after business hours.

    We have a script set to dial a modem on our Kserver that calls a tech on duty. We have 7 engineers and everyone takes a week turn at getting calls after hours. But these guys are already overworked, and they hate getting calls at 4am when someone's Internet goes down, or EVEN WORSE, a Kaseya agent goes offline for no reason and they log in, and all is well.

    Plus if a client has multiple servers, each one will call the tech.

    Looking for a better way to do this as we move to K2.

    Legacy Forum Name: How do you handle offline alerts after hours?,
    Legacy Posted By Username: jasonrdavis
  • jasonrdavis
    Was just wondering how everyone handles offline alerts after business hours.

    We have a script set to dial a modem on our Kserver that calls a tech on duty. We have 7 engineers and everyone takes a week turn at getting calls after hours. But these guys are already overworked, and they hate getting calls at 4am when someone's Internet goes down, or EVEN WORSE, a Kaseya agent goes offline for no reason and they log in, and all is well.

    Plus if a client has multiple servers, each one will call the tech.

    Looking for a better way to do this as we move to K2.


    It depends on the alert, all alerts go to our "monitoring staff" but only critical alerts (site/server down) will wel respond to. I handle all after hours alerts/trouble calls myself. How many end points do you have?

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: thirteentwenty
  • We have After Hours information for all of our clients and the techs use that information to determine what they need to do. Most of them have it memorized.

    We e-mail text messages to the on-call phones and if it's someone who has an After Hours procedure where we need to do something, it gets taken care of. Otherwise the tech goes back to sleep. When techs are on-call they get up at 5am to do a system check of anything they didn't already attend to.

    This works really well, but our techs are on a 2 month on-call rotation (one week on-call every two months) so there's not a lot of burnout.

    I had started on a system to automate the notifications based on After-Hours contact procedure information, but into some issues and haven't gotten back to it. Since we're in the process of moving to K2 right now, it didn't really make sense for me to do it on K5...though, I do feel bad for the techs who have to wake up because of an internet outage and not a real server offline.

    If you're already using a script, I would recommend just implementing after-hours contact information into the script. (X customer wants to know if their server goes down 24/7...Y customer only wants to know between 5am and 7pm,etc).

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: Intech-Jason
  • We're using Kaseya's level 1 NOC service. They leverage Kaseya's monitoring and a text to speech system by Telalert to call the on call technician.

    It works ok for the most part but there are some kinks. It's 10/server/month. We will get called for any type of definable critical outage including ISP outage. I think anything better than this, it would have to be a 24 hour NOC with real people.

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: raymondm
  • Our after-hours work is handled on a rotating schedule amongst 4 techs, currently. Call schedule is 2 weeks, starting on a Friday. The on-call tech has a voice mail box at the office routed to his company phone, where customers with after-hours requests can call in and leave a message, 30 minute call back window.This message is also sent to everyone's email box. If the message is not cleared from the mailbox in 30 minutes, it notifies 3 people, the NOC manager, the CTO, and the CEO. On-call guy also gets a 2nd phone which our monitoring systems will send periodic alerts to based on both Kaseya and Servers Alive checks. The tech is responsible for calling the client and finding out how they would like us to handle the problem, along with getting a ticket open in Connectwise, documenting client contact, and any pertinent support ticket numbers. Some clients have defined after-hours support procedures, some do not.

    Call is 24-7 for those 2 weeks. With the current level of techs, we wind up doing 2 weeks every 2 months. Typically when I sleep, it's in the guest room for those 2 weeks.

    Charles

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: cmccloud
  • I sometimes envy the larger companies that can afford to have a rotating after hours staff... But then I think about it, most of our "critical" after hours events are Bandwidth related, so a quick phone call is all that is really needed for us... I'd imagine that for you larger companies, you have clients that run 24/7, or have to deal with more environmental issues that we do.

    [QUOTE=cMcCloud]
    The tech is responsible for calling the client and finding out how they would like us to handle the problem, along with getting a ticket open in Connectwise, documenting client contact, and any pertinent support ticket numbers. Some clients have defined after-hours support procedures, some do not. [/QUOTE]

    This is pretty much my routine for anything that comes in after hours...

    How many after hour calls do you take, say in a weeks period? For me, I'll see maybe 3 or 4 calls after hours or on the weekends.

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: thirteentwenty
  • Typically, I see at least 1 call per night, sometimes 2. Weekends tend to be a bit more hectic. I think this past Saturday, I saw 6 voice mails hit the VM Afterhours box, but I wasn't on call. I'm not sure how many alerts went out over the pager, but typically I'll see at least 10-12 alerts per night, most of which do not require intervention. That pager/phone gets alarms that are triggered by anything though, as it monitors modems, firewalls, switches, servers, iLO cards, and some critical desktops.

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: cmccloud
  • Do you bill your clients at a premium rate for after-hours/weekends? Say, 1.5 times a normal rate? If you have a client that is 24x7, how much would you mark up your regular contract to include the 24x7 coverage?

    Thanks!

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: billmccl
  • We have 1500 endpoints.
    I figure this thread is about to be locked and moved to the new forums.
    Ugh, nobody seems to have a better answer than what we're doing - on call hell for the engineer on duty.

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: jasonrdavis
  • Setting up remote support for NOC purpose would solve this issue. We are doing this for handful of clients and if there are issues that need escalation we make calls to the onsite guys.

    We leave email status on the morning things as well keeping everything up ready by 7 AM.

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: mnagra
  • I'm just curious about those of you who rotate your technicians through an after hours "on-call" schedule, do you compensate them extra for the week they are on call?

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: Don_Cockayne
  • We pay them for taking a mobile phone for a week. For BEING on call.
    And we pay them a fixed hourly rate for any calls they take. For WORK after hours.

    Our SLAs are based on business hours support and after hours support is charged as "out of scope" at an after hours rate.
    We used to get a lot of customers trying to get us to give them free after hours work.
    So we changed the message on the phone system to clearly state that they have called after hours and will be charged at their after hours rate. If they understand the rates and have appropriate authority to approve after hours payments hold on the phone to be forwarded to an engineer.

    Our After hours calls have almost stopped now. The problem is the phone runs hotter the next morning.

    Legacy Forum Name: General Discussion,
    Legacy Posted By Username: garry