I have created this community ticket mostly just as a warning to others who may be troubleshooting issues that result in Windows not being bootable today, 6/18/15. I have submitted a (very) long post to Kaseya about our 8 hours of troubleshooting and dealing with this issue.
We discovered after several hours of troubleshooting and working with Microsoft that this issue is NOT DUE TO JUNE 2015 SECURITY UPDATES, but is in fact a direct issue with the latest KES/AVG UPDATE that was pushed out automatically last night/this morning. Many systems affected by this update are unable to boot afterwards. Some can be corrected by a system recovery/restore point, but most cannot.
Our only current solution is to boot into safe mode and remove AVG or otherwise cripple its services through the registry. We do not have a streamlined fix for those that are unable to boot, as it requires calling the user and booting into safe mode, running a complex command, etc. This also requires a local user admin account which, in most cases, cannot just be given out on a whim. Obviously we cannot make changes to user accounts as these systems are offline, else we could just create a temporary account for them to use.
I just wanted everyone to be aware that this is an issue and Kaseya is allegedly working with AVG in trying to work out a fix. They have a master ticket, and for most immediate updates I recommend that you put in a ticket. At this point we have reached out to all of our clients and asked that they don't reboot their computers out of fear that they may not come back after, due to the KES update.
I'm also curious if anyone has already found a better fix that doesn't require so much user-end input. Kaseya has not yet, but I'm sure they're spending a lot of their time dealing with customers and with AVG. If anyone has any input or suggestions (or cries for help) please post them here so we can work together.
We've been bit by this as well. We hadn't found a cause, so thanks!
Any chance it affected servers as well? We had a high number go offline overnight. Fortunately, all recovered on their own.
Which version of KES is causing this issue? is it KES 126.96.36.199 or KES 271.1.1/9543, or another version?
The machines I know that had problems are on 271.1.1/9543.
2012.0.2250.271.1/9534 on all of our affected clients.
jerryd we are also having this issue across a number of customers. Rebooted the servers brought them back online. I followed up with kaseya and they said AVG is still investigating the issue.
Any thoughts on why this affected some servers (and few workstations) and not others?
@jerryd - Do you know the master ticket number?
@Elliot - I've scanned the handful of workstations I knew had the problem. I've not found anything common among them yet.
Thanks. I dont have the master ticket but this is my ticket that they linked to the master. 89441
@Mike_Judd I was never provided a link to their master ticket, I assume this is an internal ticket which they can connect external requests to for tracking purposes. They only informed me of this as a comforting way to let me know that my ticket would be updated as they find workarounds/resolutions.
The hope is that they find a solution by tomorrow that we can just send out via agent procedure or will go through automatically with KES update scripts.
Today, we had to call (many) users and walk them through doing a safe boot with command prompt and uninstalling AVG and rebooting.
We've had what seems like countless workstations and servers which have become unresponsive to all but ICMP. In every case AVG initiated an msiexec based update prior to the issue. We can see that the system initiated a manual update under the Security > Protection > Manual Update page.
We have found that one or sometimes two reboots are required get return the machine to normal operation. After this they seem OK but we are watching them closely. This is another giant nail in the coffin for this product for us.
@Mike_Judd we tried that, and we also needed to perform the same on AVGgrkx64 which allows the workstation to boot but without AVG.. I then changed all values of any AVG service to 1 (Delayed start) that allowed workstations to boot but AVG modules wouldn't load correctly.
It took Kaseya 34 minutes to respond to our initial ticket and they indicated that they were already aware of the issue and were working with AVG. My question is couldn't Kaseya have helped their partners out by communicating this issue proactively?
Right - Or at least sent out a brief communication to clients which actively use the KES module for many of their clients. I'd imagine this would have taken just a few minutes and would have saved labor hours for troubleshooting for many companies.
Everyone on this ticket whom has already communicated their issues to Kaseya should question why we don't get updates for such major conflicts which our service provider is already aware of. Of course, Kaseya isn't directly responsible for the issues we saw today... But is the point of an RMM not to keep tasks like this simpler?
We had this happen to over 50 servers and over 100 workstations.
We found that it was update 9542 and 9543 that has been breaking the machines.
If you do a "shutdown" then you are more likely to face the issues and have the PC crash on you, If you pull the power cable from the PC or do a "dirty" shutdown then it seems to work afterwards and allow you to update AVG and bypass the issue.
We have been given a few "fixes" from AVG that do not work!
We have tested a few things to get our customers back up and running but we have failed at most hurdles... we have tested the AVG uninstaller and then a reinstall from kaseyas servers and we get exactly the same issue, so at present we have had to create a script that will download and install 32/64bit MSE and a script that will uninstall it when AVG is "fixed"
The scripts are straight forward but let me know if you want a copy and I will attach them.