Kaseya Community

Server lockups/freezes after AVG virus def udpates

  • I am starting a new thread for Server lockups/freezes after AVG virus def updates.



    Some of you may be following the other thread I started regarding PERC RAID card issues with DELL PowerEdge 2900 servers with AVG installed (http://community.kaseya.com/xsp/f/94/t/9618.aspx). I believe that PERC problem and related crashes seems to have been caused by having Link Scanner installed with AVGv9, on servers running Windows Server 2008. Do NOT install AVG Link Scanner on any servers. See this other post for details (http://community.kaseya.com/xsp/f/94/t/9608.aspx)



    Now, back to the problem of Server lockups/freezes after AVG virus def updates....

    - We have about 45 servers running AVG v9.

    - We have had 5 of them randomly locked-up/froze over the past several weeks. (separately we had 4 others with crashes due to PERC errors, but that seems to have been resolved by uninstalling Link Scanner).

    - of the 5 that have locked-up/froze, some of them were running Windows Server 2003, some were running Windows Server 2008.

    - All were running AVGv9, none were running AVG v8.5.

    - In my case, all were running SBS2003 (with Windows Server 2003 32bit) or SBS2008 (with Windows Server 2008 64bit)

    - For these 5 servers, just before the lockup, an AVG virus definition update was performed. If I look in the AVG GUI->History menu->Event History Log, in every case of the lockups/freezes, the log shows an entry for "Update was Finished" anywhere from 1-5 minutes prior to the lockup (and us receiving an OFFLINE alert from Kaseya).

    - Also, within a minute or so of the AVG Log file showing "Update was Finished", there are events in the Windows Application Log for SharePoint items.

    --- for an SBS2008 server, the Windows Application Log shows event id=6398, Source= Windows SharePoint Services 3, description = The Execute method of job definition Microsoft.SharePoint.Administration.SPAntivirusJobDefinition (ID 58610d66-6427-4a35-b71b-ddfdbd4c1fb0) threw an exception.

    --- for an SBS2003 server, in the Windows Application log, there are events logged at 9:28PM, Source= Windows SharePoint Services 2.0, Event ID= 1000, descriptions = “#96000d: Unloading antivirus scanner...” and another for “#96000e: Antivirus scanner has been unloaded.” And another for “#96000f: Loading antivirus scanner.”



    Some info that I learned during my work with Kaseya Support and AVG Support...

    - Kaseya periodically runs the "KES Update" script. This passes commands to the local AVG software to perform a virus def update/check.

    - Independent of Kaseya, the AVG software itself does its own virus def updates (Tools ->Advanced Settings->Schedules->virus database update schedule).

    - AVG GUI->History menu->Event History Log will show "Update was Started" and "Updated was Finished" - no matter how the virus def update was initiated - either via Kaseya or via AVG itself.

    - The Kaseya VSA->Security tab->View Logs screen will show an entry for "Update Succeeded" - no matter how the virus def update was initiated - either via Kaseya or via AVG itself.

    - Starting with AVGv9, AVG started to offer a SharePoint plug-in option. Apparently they did not inform Kaseya. And apparently, since there are no Kaseya switches to tell it otherwise, the AVG installer will automatically install the SharePoint Plug-in if it detects SharePoint on the server. And we are all aware that SBS2003 includes Windows SharePoint Services v2, and that SBS2008 includes Windows SharePoint Services v3.

    - If you log onto the server, and open the AVG GUI, on the left-hand side you will see a 4th link (that you do not see on other machines) with the caption Server Components. Click that link, and you will see the SharePoint component installed.



    I know for a fact that the issue is related to something happening at the time of AVG virus def updates. We are still troubleshooting and diagnosing. But at this point, there is some thought that it might be related to the AVG SharePoint component, or possibly related to something that happens inside of SharePoint itself, whenever the installed virus software does something.



    So that is what I wanted to share with everyone.



    Here are my questions for everyone, which can help me in working with Kaseya Support and AVG Support to troubleshoot and solve the problem for all of us ..... if you have experienced some server lockups/freezes lately, could you please reply with your answers….



    1) How many different servers were there?

    2) Are they all running AVG v9? or are some running AVG v8.5?

    3) Just before the lockup, are there events logged in the History in the AVG GUI for "Updated Finished"?

    4a) Are they running SBS2003 or SBS2008 (which means SharePoint is installed, and the AVG SharePoint component in installed?

    4b) If there are NOT running SBS, is Microsoft SharePoint installed?

    5) And to reverse the questions, have you had any lockups with servers running AVG8.5? Or any lockups with servers that do NOT have SharePoint installed?



    Thanks!



    Lloyd

    Legacy Forum Name: Server lockups/freezes after AVG virus def udpates,
    Legacy Posted By Username: lwolf
  • I just had a SBS 2003 server lockup after AVG update. It is running AVG
    9.0.791 271.1.1/2754 [KES 2.1.0.118]

    I need to know how to disable this plug-in ASAP!! This is a TPA Firm in the middle of Tax Season and this is killing them. I can't afford to loose these clients due to this "AVG push what we want, when we want prior to any testing occur" Any more!! This isn't how to run an effective, efficient MSP Practice.

    Last, Lloyd I know you have been struggling with this and apprciate bringing this to our attention. I think you are on to something here as SBS is a big product with a lot of technology running.

    At the end of the day I want a simple AntiVirus program running as little as neccessary to assure good protection for a File Server/Exchange Server/Share Point Server/SBS Server.

    I want keep this solution simple, with no extra "Fluff" and keep the changes to a minimum. All these program changes are what is causing the issues. 80% of all issues come from change and I as the MSP should have some control over this for my client's environment.

    Thanks again Lloyd and keep us posted on any findings

    Joe Axne
    IT-Guru, LLC

    Legacy Forum Name: KES,
    Legacy Posted By Username: itgurullc
  • Sorry, I don't have much time this morning, but here are some quick answers.

    1) How many different servers were there? 15/400

    2) Are they all running AVG v9? or are some running AVG v8.5? All are running AVG9 (SP3?) This has started happening since all clients went to reboot required status a few weeks back.

    3) Just before the lockup, are there events logged in the History in the AVG GUI for "Updated Finished"? Yes

    4a) Are they running SBS2003 or SBS2008 (which means SharePoint is installed, and the AVG SharePoint component in installed? I did a quick check on the 8 servers I have tickets on, none of them are SBS.

    4b) If there are NOT running SBS, is Microsoft SharePoint installed? I checked one of the 8 servers, it is not running SharePoint and is still locking up.

    5) And to reverse the questions, have you had any lockups with servers running AVG8.5? Or any lockups with servers that do NOT have SharePoint installed? We made a push a few months back to get everyone on AVG 9.0 while also uninstalling web shield and link scanner on all servers.

    Because of the way the servers have locked up, almost all of mine failed on reboot and were stuck at "Applying computer settings" with 100% CPU usage. I would like to know what "Use DNS update" is under additional update options. My update options are also set to "Complete at next computer restart" which might lead to why they are freezing on reboot.

    Legacy Forum Name: KES,
    Legacy Posted By Username: cnwicsurrett
  • this is getting extremely scary when more and more servers are locking up. Its a great way for the customer to lose confidence in our ability to manage their sites when out installed components cause more problems than resolve.
    Are most of these Dell server?

    We have not noticed too many problems but we only have HP servers under the avg.

    Legacy Forum Name: KES,
    Legacy Posted By Username: gdoubinin
  • Uuggghhhh. We had two more servers lockup last night - shortly after running AVG updates. One was a 1-year old SBS2008 server, one was a 3-year old 2003 terminal server.

    It is looking like the probelm is not related to their Sharepoint plugin, but is definatelyr elaetd to something that happens when the AVG virus defs are udpated.

    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • lwolf
    Uuggghhhh. We had two more servers lockup last night - shortly after running AVG updates. One was a 1-year old SBS2008 server, one was a 3-year old 2003 terminal server.

    It is looking like the probelm is not related to their Sharepoint plugin, but is definatelyr elaetd to soemthign that happens when the AVG virus dfs are udpated.

    Lloyd


    Do you think it would be possible to have a standard AVG install (without KES in the picture), to see if that helps in determine the cause?

    Legacy Forum Name: KES,
    Legacy Posted By Username: Coldfirex
  • From my research, I don;t really think that it is Kaseya/KES related, but caused by the underlying AVG software.

    Over the past day or so, we just got done with an effort to get a hotfix and registry keys created on each of our servers running KES/AVG v9. They will will allow us (via dell DREC hardeare remote access cards, or via talking our customer's thought the procedre) to generate a BSOD and corresponding memory dump, by pressing right Control-ScrollLock-ScrollLock key combination, while the machine is in a locked-up state. AVG Support has said, via our K Support, that they will then be able to reveiw the memory dump and determine what is going on.

    We had two lockup last night, but after installing the hotfixes and makign the registry keys, a reboot requries before it takes effect, and we had not yet rebooted the servers (waiting for our weekly reboots via Kaseya,s cheduled for early Monday AM).

    If anyone is interested in reveiwing the KB articles, I beleiev they are as follows:
    - Windows 2003 (KB244139)
    - Windows 2008 32-bit (KB971284)
    - Windows 2008 64-bit (KB971284)

    It is sad, but nice I suppose, that we can use Kaseya (our IT automation and scripting management software, to deploy hotfixes and registry keys and reboots) so we can troubleshoot why our KES/AVG addon software, is causing servers to lockup :-)

    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • OMG, I think I this is what is causing one of my clients server to keep freezing up. They are running Win2003 Server on Dell PowerEdge and all of a sudden lately their server would just freeze up. The console shows the deskop but you can't seem to load any apps what-so-ever and have to do a cold reboot. Happens once in a while but has happened 2 times in teh past 2-days. So i'm going to go ahead and uninstall AVG. This is horrible. To answer question, Server is 2003 with Sharepoint installed Running AVG9 I notice in the AGent the following runs just before the server locked up: Run Now - KES Update 4:50:49 pm 21-Mar-10 Success THEN *System* KES Update 4:50:47 pm 21-Mar-10 Success THEN *System* KES Update AVG via LAN 4:50:46 pm 21-Mar-10 Success THEN *System* KES Update AVG via LAN/If_1.A 4:50:45 pm 21-Mar-10 Success THEN *System* KES Update LAN Share 4:34:19 pm 21-Mar-10 Success THEN *System*

    Legacy Forum Name: KES,
    Legacy Posted By Username: chimoe



    [edited by: Anonymous at 12:53 PM (GMT -8) on 3-3-2011] Ok. I have same problems. AVG 9 and 8.5, freesing servers. I have 230 servers, found some things. Virtual do the same things. As I can know, same symptomps. Server is only OK on ICMP, no other service. No logging, no SNMP traps, but last thing done by the server is update. Even on virruals, nothing. No CPU, no memory problems. seems to me, some incorrect writing to protected memory, or so.
  • chimoe,

    That sounds like a match for the server lockup that is *sometimes* occuring rigth after an AVG virus def udpate.

    Please note, as stated in my original post, that the AVG udpate may be initiated by Kaseya, or by the AVG software itself, so you'd need to check the local AVG GUE for the History Log too, in case you don;t see anythign related in Kaseya Script History.

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • 4 more servers over the weekend with this same issue.

    Legacy Forum Name: KES,
    Legacy Posted By Username: cnwicsurrett
  • One today

    Server locked up almost immediatley - loggin a support call with K

    11:05:18 22-Mar-10 UpdateTimeout Script succeeded but client never reported update success
    11:05:18 22-Mar-10 UpdateScriptFailed Script succeeded but client never reported update success--Retrying with Internet update...
    10:47:23 22-Mar-10 AVGComponentStatus AVGAPI_COMP_ON_ACCESS=100|AVGAPI_COMP_ON_DEMAND=10|AVGAPI_COMP_EMAIL_CLIENT=0|AVGAPI_COMP_UPDATE_NORMAL=100|AVGAPI_COMP_UPDATE_SPECIAL=10|AVGAPI_COMP_ADMIN=0|AVGAPI_COMP_WORKSTATION=10|AVGAPI_COMP_SERVER=10|AVGAPI_COMP_LINUX=10|AVGAPI_COMP_FIREWALL=0|AVGAPI_COMP_ANTIVIRUS=100|AVGAPI_COMP_ANTISPY=100|AVGAPI_COMP_ANTISPAM=0|AVGAPI_COMP_ANTISPAM_SERVER=0|AVGAPI_COMP_ANTIROOTKIT=100|AVGAPI_COMP_HTTPSCANNING=0|AVGAPI_COMP_SAFESURF=100|AVGAPI_COMP_SAFESEARCH=100|AVGAPI_COMP_SYSTEMTOOLS=10|AVGAPI_COMP_EMAILSERVER=10|AVGAPI_COMP_FILESERVER=10|AVGAPI_COMP_ADMIN_LITE=10|AVGAPI_COMP_TOOLBAR=0|AVGAPI_COMP_ALERT_MANAGER=10|AVGAPI_COMP_LICENSE=100|AVGAPI_COMP_NET_SCANNER=0|AVGAPI_COMP_XPL=0|AVGAPI_COMP_ANALYSIS=80| AVGAPI_COMP_ON_ACCESS=0|AVGAPI_COMP_ON_DEMAND=0|AVGAPI_COMP_EMAIL_CLIENT=0|AVGAPI_COMP_UPDATE_NORMAL=0|AVGAPI_COMP_UPDATE_SPECIAL=0|AVGAPI_COMP_ADMIN=0|AVGAPI_COMP_WORKSTATION=0|AVGAPI_COMP_SERVER=0|AVGAPI_COMP_LINUX=0|AVGAPI_COMP_FIREWALL=0|AVGAPI_COMP_ANTIVIRUS=0|AVGAPI_COMP_ANTISPY=0|AVGAPI_COMP_ANTISPAM=0|AVGAPI_COMP_ANTISPAM_SERVER=0|AVGAPI_COMP_ANTIROOTKIT=0|AVGAPI_COMP_HTTPSCANNING=0|AVGAPI_COMP_SAFESURF=0|AVGAPI_COMP_SAFESEARCH=0|AVGAPI_COMP_SYSTEMTOOLS=0|AVGAPI_COMP_EMAILSERVER=0|AVGAPI_COMP_FILESERVER=0|AVGAPI_COMP_ADMIN_LITE=0|AVGAPI_COMP_TOOLBAR=0|AVGAPI_COMP_ALERT_MANAGER=0|AVGAPI_COMP_LICENSE=0|AVGAPI_COMP_NET_SCANNER=0|AVGAPI_COMP_XPL=1|AVGAPI_COMP_ANALYSIS=0|

    Legacy Forum Name: KES,
    Legacy Posted By Username: PeterS
  • Where the H is support on this?

    Legacy Forum Name: KES,
    Legacy Posted By Username: jestes
  • I have this isseu at this moment with about 10 servers.

    All are running windows 2003 and most on HP Proliant hardware, some Dell Servers.

    My conclusion it is not hardware dependent and it is an issue with AVG. This because one server I had this on was freezing several times a week and when I uninstalled the AVG the hookup didn't happen again.

    Also, referring to the reboot earlier this month, since then this problem happens. Anyone reviewed the changelog/releasenotes and related that to a freeze yet?

    Legacy Forum Name: KES,
    Legacy Posted By Username: tjibbe@nexusict.nl
  • tjibbe@nexusict.nl
    I have this isseu at this moment with about 10 servers.

    All are running windows 2003 and most on HP Proliant hardware, some Dell Servers.


    tjibbe,

    Sorry to hear of your server lockup problems. I can tell that you you are not alone.

    To followup on your post... are these servers all running Small Business Server 2003? or not? If they are not, is Microsoft Sharepoint Services installed?

    This information will help me when workign with K Support and AVG Support.

    Lloyd

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • lwolf
    tjibbe,

    Sorry to hear of your server lockup problems. I can tell that you you are not alone.

    To followup on your post... are these servers all running Small Business Server 2003? or not? If they are not, is Microsoft Sharepoint Services installed?

    This information will help me when workign with K Support and AVG Support.

    Lloyd


    Lloyd,

    This are not all sbs 2003 servers. I had just a server which was frozen and was a regular server 2003. This server did not have sharepoint services installed. Also a normal windows xp workstation had this issue just before I typed this. However this happens only on server 2003 machines so far.

    The most recent 2003 server which had this issue froze on the moment the "Kes_update" script was scheduled. there were no script log entries yet and the server froze. It had a Gray screen just like the GSOD screen's which can happen when running BUDR with VSS.

    I suspect you are working with support/avg on this?

    Legacy Forum Name: KES,
    Legacy Posted By Username: tjibbe@nexusict.nl