Kaseya Community

AVG9 SP3 and Server Lockups - A fix is coming

  • Its been a LONG 3 weeks.

    The issue: If you catch the other forum posts about server lockups, we've been experiencing a bunch of AVG locking up servers just after updates. This started in early March.

    The progress: Lloyd Wolf has been working deeply with us and AVG on capturing a dump after forcing a crash on lockup. AVG has captured the information and believe it is due to Resident Shield interaction with the new cache feature of AVG. An update is planned for release on Monday to fix that problem.

    The update is currently in test so there is a chance it doesn't get released. AVG isn't 100% sure that this problem will be resolved with the release, but they believe it should fix it.

    The update on Monday will require a reboot.

    If anyone experiences a server lockup after this, please let us know as soon as possible.

    An aside, this issue has changed the dynamics between AVG and Kaseya. We've always had a good relationship with them, but this was the first issue that I needed to "shake the tree" if you will. I've been trading emails with the CEO and two key changes have come of it.

    1. Phone support. For server down / mission critical type issues, we are now going to have AVG support on the phone / skype with you as needed.
    2. Developer response. For these critical issues, we now have access to the development team for helping isolate and debug.




    Jeff

    Legacy Forum Name: AVG9 SP3 and Server Lockups - A fix is coming,
    Legacy Posted By Username: Jeff.Keyes
  • I neglected to mention a workaround

    The issue appears to be related to Resident Shield and the new cache feature in AVG9/SP3. If you disable resident shield, then the servers shouldn't lock up.

    Alternatively - you can just disable the cache working with resident shield and leave resident shield operational. The Kaseya interface doesn't support that today and would need to happen in the AVG GUI.

    Jeff

    Legacy Forum Name: KES,
    Legacy Posted By Username: Jeff.Keyes
  • Jeff,

    Thanks for the long awaited update, any more specific information on the issue? Who is and who isn't at risk, I just got done rebooting all of my servers from the SP3 update, I don't have the resources to handle another mass reboot a week later. We found this issue to happen on 15-20% of servers that have AVG9 installed, so I'm wondering... do we assume it will happen on the rest or is there a specific variable on the server that causes this issue?

    Because the Fix is only "believed" to resolve the issue, I can't risk another round of reboots just to have servers lock-up again.

    Legacy Forum Name: KES,
    Legacy Posted By Username: cnwicsurrett
  • I don't have all the details other than it appears to be focused on servers with SBS or Exchange with AVG9. It doesn't happen all of the time either...which has made this one bug a real nasty one to hunt down / isolate.

    The workaround i prescribe here should get you over the hump without having to reboot servers...but just know that they will start reporting a reboot is needed.

    Legacy Forum Name: KES,
    Legacy Posted By Username: Jeff.Keyes
  • Jeff, thanks so much for the update and communication. We haven't experienced this issue, but the fact that you're communicating it to us, along with the info you have and the progress, is extremely helpful!

    Legacy Forum Name: KES,
    Legacy Posted By Username: rbjsea
  • your welcome. I thought I'd share information directly from AVG on this topic (just received it).


    The reason why it appeared after SP3 was released is the fact that we added two more caches there. The bigger the number of NTFS disks/partitions on the system, the bigger the chance of the issue appearing there. Adding two more caches per disk/partition makes it even more probable.

    A better workaround for the issue than disabling RS would be disabling the cache server – there is a checkbox for that in the Advanced settings section of the user interface. I am not sure whether you have a way to do that using KES. If the cache server is disabled RS can be turned on. There should be no significant performance degradation without the cache server – it should be at the 8.5 level. This is safe to apply on all machines – servers and workstations..

    We are testing the fix at the moment rather intensively and if all goes well we will release it on Monday as a program update. This program update will require computer restart so it may be wise to reschedule the server restarts from Monday, 1 am to Tuesday, 1 am.

    By the way it coincides with the update as one of the update steps is to invalidate some of the records in the caches and this provides the bug with a window of opportunity. However it may appear independently from the update as well. We managed to induce it on reboot actually.



    Legacy Forum Name: KES,
    Legacy Posted By Username: Jeff.Keyes
  • cnwicsurrett
    Jeff,

    Thanks for the long awaited update, any more specific information on the issue? Who is and who isn't at risk, I just got done rebooting all of my servers from the SP3 update, I don't have the resources to handle another mass reboot a week later. We found this issue to happen on 15-20% of servers that have AVG9 installed, so I'm wondering... do we assume it will happen on the rest or is there a specific variable on the server that causes this issue?

    Because the Fix is only "believed" to resolve the issue, I can't risk another round of reboots just to have servers lock-up again.


    Yea, we just got hit. This issue locked up about 15-20 servers we support. We had a lot of unhappy clients..Mad

    Legacy Forum Name: KES,
    Legacy Posted By Username: netwiz001
  • To all,

    Per a suggestion from AVG Support, I have written a script to disable the AVGv9 Cache Server feature as a way to help (temporarily) prevent the server lockup issue. Per AVG support, this *should* prevent the lockup problem (while waiting on the program changes from their Developers).

    Below are two scripts that I wrote. The first script disables the Cache Server feature (Tools->Advanced Settings->CacheServer). The second script re-enables it.

    I have two separate steps to restart the Process because the avgchsvx.exe filename is used for 32-bit systems, and the avgchsva.exe is used for 64-bit systems, and I didn't feel like taking the time to make subscriptiosn for determining the Operatign System.

    After running, the checkmarks in the Tools->Advanced Settings->CacheServer will change, and the PID in Task Manager for avgchsva.exe or avgchsvx.exe will change.

    You should make a View to show only machines that contain avgwdsvc.exe > 9.0, and for OS Type= Servers.

    I have tested on my own in-house systems, and they seemed to work fine. But as with any scripts that anyone posts, please review and test. YMMV. Use at your own risk. No warranties provided. Etc.



    Script Name: AVG - disable Cache Server
    Script Description: Modify registry (HKEY_LOCAL_MACHINE\SOFTWARE\AVG\Avg9\CacheServer\chjwParams) to disable Cache Server, and restart the Process. (used to temporarily prevent problem with AVG9 and server lockups.)

    IF True
    THEN
    Set Registry Value
    Parameter 1 : HKEY_LOCAL_MACHINE\SOFTWARE\AVG\Avg9\CacheServer\chjwParams
    Parameter 2 : 0
    Parameter 3 : REG_DWORD
    OS Type : 0
    Execute Shell Command
    Parameter 1 : taskkill /IM avgchsva.exe /F
    Parameter 2 : 1
    OS Type : 0
    Execute Shell Command
    Parameter 1 : taskkill /IM avgchsvx.exe /F
    Parameter 2 : 1
    OS Type : 0
    Write Script Log Entry
    Parameter 1 : Modify registry (HKEY_LOCAL_MACHINE\SOFTWARE\AVG\Avg9\CacheServer\chjwParams) to disable Cache Server, and restart the Process. (used to temporarily prevent problem with AVG9 and server lockups.)
    OS Type : 0
    ELSE




    Script Name: AVG - enable Cache Server
    Script Description: Modify registry (HKEY_LOCAL_MACHINE\SOFTWARE\AVG\Avg9\CacheServer\chjwParams) to enable Cache Server, and restart the Process. (used to reset to original settings after temporarily preventing problem with AVG9 and server lockups.)

    IF True
    THEN
    Set Registry Value
    Parameter 1 : HKEY_LOCAL_MACHINE\SOFTWARE\AVG\Avg9\CacheServer\chjwParams
    Parameter 2 : 3
    Parameter 3 : REG_DWORD
    OS Type : 0
    Execute Shell Command
    Parameter 1 : taskkill /IM avgchsva.exe /F
    Parameter 2 : 1
    OS Type : 0
    Execute Shell Command
    Parameter 1 : taskkill /IM avgchsvx.exe /F
    Parameter 2 : 1
    OS Type : 0
    Write Script Log Entry
    Parameter 1 : Modify registry (HKEY_LOCAL_MACHINE\SOFTWARE\AVG\Avg9\CacheServer\chjwParams) to enable Cache Server, and restart the Process. (used to temporarily prevent problem with AVG9 and server lockups.)
    OS Type : 0
    ELSE


    I hope this can help some others - while AVG is finalizing the formal fix.


    I will also post to the other thread, as some people may be looking in one or the other.

    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • Thanks Jeff for the update

    A huge thank you to Lloyd for sticking with it and taking the risk of keeping AVG running just to help the vendors get to the bottom of it.

    As someone commented on another thread this hasn't done us any good at all as our clients pay us to keep their systems running and what is essentially seen as our software has caused these kinds of serious issues.

    I'll be giving serious consideration to whether I re-deploy KES to my servers at all.

    Lee

    Legacy Forum Name: KES,
    Legacy Posted By Username: leeevans
  • @jeff
    Really REALLY unhappy about this issue, but I understand is not directly a Kaseya problem.

    My real issue is why I have to actively chase down this information.
    Why doesn't Kaseya have a knowledgebase and critical issues email list where they COMMUNICATE to clients problems like this.

    Have just spent 8 hours in the last day diagnosing and troubleshooting a server freezing issue with a new client site which turns out to be an AVG issue.

    @lwolf
    Appreciate the script - you're a champion! Its going to save me a lot of headaches.

    Not at all happy at the moment...i pushed for KES internally and this issue has caused a lot of pain. Not going to be easy to dig out of but if I known this was a known issue in advance we could have applied the fix as we deployed KES.

    Legacy Forum Name: KES,
    Legacy Posted By Username: tbone2345
  • To all,

    FYI... AVG released the program update for v9.0.798 this afternoon. This version is *suppose* to fix the Cache Server problem. And according to AVG, and reboot is NOT required.

    It has been about 6 hours or so, and I am still only seeing it deployed to abotu 1 out of 10 of my AVG9 machines. The balance are still running v9.0.791.

    FYI, I manually clicked the Udpate button in the AVG GUI and a few machines, and saw the Update get downloaded and installed, and there was no reboot.

    Be on the lookout for this program update.

    I will also post to the other thread.

    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • We are still seeing lockups on Version 8.5.437


    will a similar fix be released for 8?

    Legacy Forum Name: KES,
    Legacy Posted By Username: camorton
  • Thanks Lloyd

    This has appeared on a couple of our systems now so i will re-enable the cache on the servers and see what happens.

    Cheers

    Legacy Forum Name: KES,
    Legacy Posted By Username: PeterS
  • I came in this morning to 3 internal servers locked up while updating to AVG 9.0.798

    Legacy Forum Name: KES,
    Legacy Posted By Username: cnwicsurrett
  • I'm super interested in any other server lockups...let me know if we have any problems here

    Thanks for the patience...

    Jeff

    Legacy Forum Name: KES,
    Legacy Posted By Username: Jeff.Keyes