Kaseya Community

Server lockups/freezes after AVG virus def udpates

  • Add us and a bunch of servers to the list. I think we've had ~10 servers affected thus far with and the number is growing. They're all Dell PowerEdge servers however that's primarily what we support.

    Lloyd - to answer your questions:
    1) How many different servers were there?
    ~10

    2) Are they all running AVG v9? or are some running AVG v8.5?
    Both 8.5 and 9 seem to be affected. Two of the servers that have hung within the last couple weeks have been running 8.5. FWIW, these two servers have never hung before.

    3) Just before the lockup, are there events logged in the History in the AVG GUI for "Updated Finished"?
    Yes, on every one.

    4a) Are they running SBS2003 or SBS2008 (which means SharePoint is installed, and the AVG SharePoint component in installed?
    SBS2003, SBS2008, Windows 2003 and Windows 2008 and at least 2 without SharePoint

    4b) If there are NOT running SBS, is Microsoft SharePoint installed?
    Not necessarily

    5) And to reverse the questions, have you had any lockups with servers running AVG8.5? Or any lockups with servers that do NOT have SharePoint installed?
    Yes and Yes. And to reiterate my answer for #2, these are all fairly new, solid servers - we rarely, if ever, experience 'frozen' servers so I can say with pretty high certainty that the recent occurrences stem from the same problem


    Also, FYI, we've had one server nearly 'frozen' but still semi-operable. It may just be a symptom of the problem, but almost everything that relied on RPC had failed or was failing.

    Legacy Forum Name: KES,
    Legacy Posted By Username: dsecker
  • Just to throw a wrench in here. We have since disabled automatic update from the KES Side and decided to let AVG update itself to it's default schedule (Every 4 hours). We are still having lockups, our server locked up last night.

    It's a sbs 2008.

    Link scanner is not installed

    Exchange module is NOT installed (we removed it last week as we went to off site smtp email scanning.)

    Had 3 or 4 servers lock up on us in the last week. 3 of those were on KES one client is not using KES but using AVG for SBS Those were SBS 2003.

    I am leaning towards this being an AVG issue because the client that locked up with the same symptoms as other servers, does not have KES installed.

    -------------------------

    What we have done and this has seemed to help on some servers.

    Scheduled all tasks to not happen at the same time, things like (patch scans, auto updates, Disabled KES updating, AV scans.)

    Also we have changed AVG on the console itself by overwriting the sched.cfg file to only update AVG once a day and to only update definitions on the server once a day.

    Legacy Forum Name: KES,
    Legacy Posted By Username: jestes
  • Have we or any of you receive a decent response from support recently? What about a comment by Jeff Keyes?

    This seems like a pretty important problem we are having...

    Legacy Forum Name: KES,
    Legacy Posted By Username: jestes
  • tjibbe@nexusict.nl

    I suspect you are working with support/avg on this?

    Yes, K Support has gotten AVG Support involved, and AVG Support is working with me on trying to identify the root cause of the problem. They are reviewing some memory dump files that I have provided.


    tjibbe@nexusict.nl
    Have we or any of you receive a decent response from support recently? What about a comment by Jeff Keyes? This seems like a pretty important problem we are having...

    Jeff Keyes is aware of the problem, and its large scope. He has been helpful in getting AVG Support involved in the troubleshooting and diagnosing with me.

    tjibbe@nexusict.nl
    2) Are they all running AVG v9? or are some running AVG v8.5?
    Both 8.5 and 9 seem to be affected. Two of the servers that have hung within the last couple weeks have been running 8.5. FWIW, these two servers have never hung before.

    Yikes, so far I have only seen the issue on servers running AVG v9. I was thinking that my servers still running AVG 8.5 were safe. But, thanks very much for sharing the information. I will be sure to pass it along to AVG Support.

    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • I opneed a ticket yesterday with Kaseya Support and below is the resposne I got. Wonder if anyone can figure out where they are going with this:

    Dear Customer,

    Thank you for contacting Kaseya support.

    In order resolve the issue, i would suggest that you please refer the installation procedure to install the AVG on the server from the link belowr:
    http://www.avg.com/us-en/faq?num=2175

    After installing the AVG if the issue persists, please provide the following additional information:
    1. Please let me know the profile that you used to install the AVG.
    2. Also let us know the Installation option that you have selected to install AVG.

    Regards,
    Kaseya support

    Legacy Forum Name: KES,
    Legacy Posted By Username: chimoe
  • Guys seems like many are having this issue so I would advise the following:

    1. If you haven't, please open a support ticket with Kaseya Support and obtain a ticket number for your issue.

    The more users that open ticket's I feel the better resources will be assigned to this.

    I have spoken to Jeff Keyes and they are fully aware of the issue. His first question to me was "have you opened a ticket". My response, no! I agree with Jeff that tickets need to be opened to show proof of this issue to AVG to get this resolved.

    At this time I have one server that I just have to uninstall AVG from at this time due to my client's busy tax season currently!

    Let's all get tickets created and explain the issues we have seen and point support to this post where others are having the same issue.

    If anyone has any update to this, please advise as I am very interested in finding out what the root cause of this is so we can resolve for other SBS Server's that haven't (Knock on Wood) locked up yet.....

    Joe Axne
    IT-Guru, LLC

    Legacy Forum Name: KES,
    Legacy Posted By Username: itgurullc
  • We have a client that manages their own AVG infrastructure separate from KES, and the same issue occurred there. Still have not seen any recurrence of the problem anywhere, though.

    It seems that the problem is occurring after AVG updates to 8.5.437 or 9.0.971, which requires a reboot. Sometime after the reboot to complete the engine update, a definition update occurs that creates the hang.

    Legacy Forum Name: KES,
    Legacy Posted By Username: drodden
  • Add another Pittsburgher with the same frustrating issue I've spent my morning researching, before finding this thread. None of my clients without AVG 9 / Kaseya are affected, yet ALL of them with it are. It always occurs after an AVG update.


    To answer Lloyd's questions:

    1) How many different servers were there?
    6

    2) Are they all running AVG v9? or are some running AVG v8.5?
    9 on all accounts. Only one of these servers has been known to hang without rhym or reason, and it's a 6 year old "entry level" box.

    3) Just before the lockup, are there events logged in the History in the AVG GUI for "Updated Finished"?
    Yes.

    4a) Are they running SBS2003 or SBS2008 (which means SharePoint is installed, and the AVG SharePoint component in installed?
    2 are 2003 Standard, 3 are SBS 2003, one is SBS 2008. All are equally affected. The wo 2003 standards do NOT have Sharepoint services installed.

    4b) If there are NOT running SBS, is Microsoft SharePoint installed?
    No

    The servers would all allow me to log in from the console, but never load properly. Some still showed a Kaseya connection, but none would respond to a reboot. In all cases I had to drive out to resolve the issue, which essentially meant power cycling - that's not a resolution.

    This is not the kind of service I was promised from Kaseya / KES, and in turn I'm frankly providing poor service to my clients. I might resort to removing KES from the effected servers until this issue is resolved.

    I love the integration, but boy has KES been a pain Cool

    -Matt Weaver
    Echo Systems

    Legacy Forum Name: KES,
    Legacy Posted By Username: FCCSonline
  • Has anyone experienced similar lockups with desktops or laptops?

    We just put 2 and 2 together, and realized that a recurring freezing/lockup issue on an important client's desktop PC (newer computer, running Windows XP Pro) is a match for this problem - lockup after AVG udpate occurs.

    Looking back through this thread, I see a post yesterday from tjibbe@nexusict.nl where a workstation PC lockup was mentioned. But otherwise, the focus has all been on servers.

    Has anyone experienced similar lockups after an AVG Update with desktops or laptops?

    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • I have only seen this on Servers, not seen it on XP,Vista or Win7 - so far. Probably means that my Vista Laptop will die in the next few minutes though....Smile

    Legacy Forum Name: KES,
    Legacy Posted By Username: PeterS
  • Thinking about this, if anyone has Blue Screen dumps, I would highly advise also getting Microsoft involved in this to see if they can find the conflict.

    I was speaking with another VAR that stated they saw similar issues with SBS 2008 and what they thought was Trend causing the server to lockup the server after updates. However after time they found it was related with Microsoft. I will try and find details out on this and see if this apply's to our situation at all.

    If it isn't related to Microsoft they will be the first to point that out and hopefully point us in the right direction of what is going on here. (i.e. specific .dll, .sys file etc)

    I will be placing Blue Screen traps on a few of my SBS Server's that have AVG 9.x installed currently. I will blue screen once, but have made the decision to remove AVG currently if the problem persists... I don't have DRAC cards on a lot of these server's and have to drive onsite if one of my server's keeps locking.....

    Just a sugestion, but interesting to see that standalone AVG may be causing this issue. I am also going to place a post on ASCII tech group and see if anyone else uses standalone product and has issues....

    Regards,

    Joe Axne
    IT-Guru, LLC

    Legacy Forum Name: KES,
    Legacy Posted By Username: itgurullc
  • It doesnt blue screen, in my case at least. The server just becomes unresponsive with no activity

    Legacy Forum Name: KES,
    Legacy Posted By Username: PeterS
  • Just had this (we believe) on our internal Exchange server.


    1) How many different servers were there?
    1 (so far)

    2) Are they all running AVG v9? or are some running AVG v8.5?
    AVG9

    3) Just before the lockup, are there events logged in the History in the AVG GUI for "Updated Finished"?
    Yes.

    4a) Are they running SBS2003 or SBS2008 (which means SharePoint is installed, and the AVG SharePoint component in installed?
    No Sharepoint.

    4b) If there are NOT running SBS, is Microsoft SharePoint installed?

    No linkscanner. Just AV, antispyware, and Exchange plugin.
    Server was offline, but still on though nothing could be done at the console. Had to power off.
    Server 2003 SP2 with Exchange 2003 SP2.

    Legacy Forum Name: KES,
    Legacy Posted By Username: Coldfirex
  • I got a reply from Kaseya Support suggesting a short term fix is to disable Resident Shield on AVG. Has anyone tried this?

    Legacy Forum Name: KES,
    Legacy Posted By Username: chimoe
  • PeterS
    It doesnt blue screen, in my case at least. The server just becomes unresponsive with no activity


    FYI, they were talking about using the special registry keys in order to allow a hotkey sequence to induce a blue screen, whenever the server is in a locked up state, so that the resulting memory dump file could be analyzed.





    PeterS
    I got a reply from Kaseya Support suggesting a short term fix is to disable Resident Shield on AVG. Has anyone tried this?




    In working with AVG Support earlier this week, they suspected some type of conflict with Resident Shield, during a virus def udpate. So it was suspected that either stopping virus def updates, or disabling resident shield would eliminate the combination fo circumstances in which a lockup would occur. On a short term temporary basis, about 2 days ago, I disabled Resident Shield on the majority of our servers running AVG9, and have not had any lockups on those servers since then. Although this is not 100% certain that future lockups may not occur.







    I can report to some good news..... late today I received a report from AVG Support on my support ticket. I was told they they think their developers have pinpointed the culprit of the issue. According to them it something related to the interaction between the Cache Server and the Resident Shield that may cause the deadlock.



    I was told that the reason why it appeared after SP3 was released for AVG v9 (around first week of March) is the fact that they added two more caches there. The bigger the number of NTFS disks/partitions on the system, apparently the bigger the chance of the issue appearing there. Adding two more caches per disk/partition makes it even more probable. Since servers often tend to have c-partitions, d-partitions, etc. - the probelm is more likely to occur than on desktops or laptops that have only one partition.



    I was also told by AVG Support that a better workaround (for preventing the lockup) than disabling Resident Shield would be disabling the cache server – there is a checkbox for that in the Advanced settings section of the user interface.



    Due to time zone differences, I was not able to get any more info today. I believe that changing the Cache Server options in the AVG GUI updates the cachesrv.cfg file, which we could probably distribute via script. But I need to learn more.



    I was also told they are intensively testing the fix at the moment and if all goes well we will release it early next week as a program update. This program update will require computer restart. Refer to a new thread created by Jeff

    AVG9 SP3 and Server Lockups - A fix is coming

    http://community.kaseya.com/xsp/f/94/t/9680.aspx





    Hopefully this nightmare will be over by this time next week !



    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf