Kaseya Community

Patch Reboot script not shutting down services properly

  • Hi,



    I'd posted something like this in the past, but had sort of relegated it to the back of my mind since it was just annoying to have the "what was the reason for the unexpected shutdown?" prompt on several servers but never a real issue ( My old post http://community.kaseya.com/xsp/f/30/t/10466.aspx ).



    However this week we ran in Kaseya's crummy method of rebooting causing a serious issue. And it wasn't on an SBS server.



    The issue was: Patches applied early Monday morning, Kaseya forced a reboot (the server has BES/BPS and SQL/MSDE on it) but because Kaseya does not reboot properly (*see below) when the server came up SQL started but didn't start the BESMgmt database as it was replaying logs (I assume), so the BES services hung on start-up such that no one could connect to it (logon - it's a DC, RDP, even Kaseya said it was off line), but it was powered on.



    Luckily I'd seen this before and with some non-Kaseya remote software (an old copy of Radmin still installed) I remotely killed the BES services until the logon screen came up, I rebooted properly and all was well.



    *How do I know Kaseya was to blame? In Pending Scripts I saw:

    Patch Reboot 1:23:10 am 14-Dec-09 Success THEN *System*



    In the System Log I saw at 1:23:15AM:

    "The process AgentMon.exe has initiated the restart of computer..."



    And at 1:25:28AM I saw:

    "The previous system shutdown at 1:23:15 AM on 12/14/2009 was unexpected."



    It looks like Kaseya waited a whole 5 seconds before somehow forcing the system to shutdown. Possibly by killing LSASS, CSRSS, SMSS, WinLogon, or one of the other critical processes, I don't know. But I have seen this on SBS boxes, non-SBS all-in-one-boxes (like this one), and single purpose member servers that run only 1 thing (like SQL, or file shares, or Exchange).



    But the bottom line is that this is totally unacceptable! Kaseya should not be "rebooting" any system - let alone a server - in this manner. It's like hitting the reset button! When I rebooted this server properly it took 4 minutes from eventlog Stop to eventlog Start, meaning it took 2 minutes to shutdown (when it was unexpected it took only 2 mins to start up). I would be willing to wait 2 minutes to avoid this issue!



    So my thought are:



    Since there is no Patch Management "run script after" option for regular patches (like there is for initial updates) I would edit the System Script named "Patch Reboot" and delete the Reboot action and replace it with something like:

    Copy over a Generic Server reboot batch file

    Run the batch file



    And the Batch file would do something like:

    run shutdown.exe command with a long, 5 minute delay (could use psshutdown.exe for Win2K servers)

    net stop Various Services /yes (for dependencies)

    (I'm thinking of listing all BES, SQL, Exchange and some other services -- if they don't exist net stop will just say it doesn't exist)

    if it reaches the end then issue a shutdown command with a short delay



    This way if one of the net stop commands hangs stopping a service the previously issued shutdown command should still kick in.



    Or just have it run shutdown.exe /R and not stop services. Apart from it taking a while to shutdown I've never had a problem with shutdown.exe not waiting long enough and causing an improper shutdown.



    Any thoughts? Has anyone successfully edited a System Script and had it continue to work? This http://community.kaseya.com/xsp/f/28/t/6404.aspx thread seems to imply that editing System Scripts works.



    But obviously I don't want to make things worse. Is K6 any better with reboots? Anyone have a batch with NET STOPs for all sorts of services (SQL 2000/5/8, Exchange 2000/3/7, BES, other things)



    At this point I'm hesitant to use Kaseya's reboot action for ANY server (and I won't for my own machine either).



    --Saul

    Legacy Forum Name: Patch Reboot script not shutting down services properly,
    Legacy Posted By Username: PSaul
  • We had this same exact issue. Patch management does some sort of forced reboot without waiting for things to shut down properly. Pair this with data stored on a SAN and iSCSI and you can see where improper shutdowns can lead to a headache. It doesn't seem to want to "wait" gracefully for services to shut down.

    Here's a response I received ultimately when working with their support on this issue:

    "It would seem that the best plan of action to avoid a forced reboot would be to stop the reboot script as it is currently set up. Create a scheduled reboot time for ongoing maintenance for once a week. This way the reboot is planned and does not force a reboot."

    Legacy Forum Name: Patch Management,
    Legacy Posted By Username: donjjones
  • I'm not sure I understood support's suggestion Confused

    Did support ever explain exactly how they implement reboots?

    Legacy Forum Name: Patch Management,
    Legacy Posted By Username: ReedMikel
  • In the scripts forum I just found out that "You cannot actually edit a System Script. You can Save and/or Export/Import a System script to make your own copy, but you cannot actually make any changes to the System Script."

    So that stops that idea -- unless I directly change the ksubscribers database, which I think is doable (2 small changes), but probably unwise all the same...

    ReedMikel
    I'm not sure I understood support's suggestion Confused

    Did support ever explain exactly how they implement reboots?


    I think what they're saying is change the Patch Management action to "Do not reboot after update" and then manually schedule a reboot for shortly after. (ie. Do server updates at 1am, don't reboot but just always schedule a proper reboot at about 3am to give patches 2 hours to complete) You could probably push out a reboot script and schedule it with SCHTASKS or AT with Kaseya. Good for something at least Wink

    That or go back to manual patching. But I can't recommend Kaseay's Reboot action for servers anymore.

    --Saul

    Legacy Forum Name: Patch Management,
    Legacy Posted By Username: PSaul
  • Are you getting your errors everytime? I'm running Initial Update on one of my new test servers (2008 R2) and it's automatically rebooting without any errors being written to the event log.

    Legacy Forum Name: Patch Management,
    Legacy Posted By Username: CCDave
  • CCDave
    Are you getting your errors everytime? I'm running Initial Update on one of my new test servers (2008 R2) and it's automatically rebooting without any errors being written to the event log.


    Is your server in use though? With applications running and potentially open connections from clients?


    We don't get the errors with every Kaseya scripted Reboot action, and we don't get it on all servers. Or even certain servers all the time.

    It does seem to happen more often with reboots after patch installation -- or that's just the most common usage of Kaseya scripted Reboots for servers perhaps.

    But when the scripted reboots happen (for us it's early Monday mornings: no backups Sunday night and we're in the office Monday) it is not unusual to log into a server on Monday or Tuesday and be greeted with the "what was the reason for the unexpected shutdown?" message. So much so no one even bothers saying "hey, X server didn't shutdown OK, that's weird!" because it happens so often.

    And it's all kinds of servers.

    But I don't recall having the problem if I run: shutdown -r -f

    --Saul

    Legacy Forum Name: Patch Management,
    Legacy Posted By Username: PSaul