Windows Defender/Anti-malware Causing Performance Issues after CU update
rdodev opened this issue · comments
So, after uninstalling my old Ubuntu Bash lxrun /uninstall /full and after the update lxrun /install I noticed, as promised, that processes running in WSL could be seen in the Windows task manager (yay!); however, there is an issue that consistently causes Windows Defender/Anti-Malware service to consume over 50% of CPU utilization whenever code compilation or builds are happening inside bash. Back last version (before the processes were visible to Windows -- or more sandboxed, not sure) this was not an issue at all. So I'm wondering if this is a worthy trade off? I, for one, as a dev, would prefer to maximize build/compile performance even if that means reverting to a more sandboxed environ (or at least the option to do so).
For what it's worth, I always add an exclusion for my big source-code build directories to Windows Defender, for exactly this reason. I trust my own code, and the performance hit is too big otherwise.
Adding to @aseering's comment, this is a problem even with yarn: yarnpkg/yarn#990
Just tried it out @aseering and noticed a pretty big speedup of a rails app load time. Nice!
I added an exclusion for both my networked drive and my lxss folder.
Adding an exclusion does NOT fix the performance problem. Adding an exclusion for VS does NOT fix the performance problem. Adding an exclusion for IIS does NOT fix the performance problem.
The only solution that works for us is to disable REALTIME protection in Windows Defender.
I also want to note that the performance impact is DRAMATIC.
For instance, a rebuild of a VS2017 solution that takes ~20seconds to complete with REALTIME protection turned off, takes more than 3 minutes to complete with Windows Defender REALTIME protection enabled.
Another example: a GIT local check-in with one file changed, completes almost instantly with Windows Defender REALTIME protection DISABLED. This same checkin takes more than 2 minutes with Windows Defender REALTIME protextion enabled.
The exclusions DO NOTHING to improve the situation.
@tanseydavid yes exclusions do not seem to have helped at all for the linux side. :(
Yeah, it feels like adding exclusions is just a placebo - the performance is still incredibly slow for a lot of stuff related to development. But how do we even raise an issue with the windows defender team about this?
@tanseydavid - The issue that you have described, is that under WSL? Or, just in Windows general outside WSL?
@rdodev & @kumarharsh - In WSL, are you guys seeing any differences when defender realtime protection is turned off for the build directory? If you have any data to quantify, that will be useful.
I would like to get some clarification here before reaching out to the defender team internally.
Also, can you share your Windows build numbers, using the ver command from CMD?
10.0.15063
I stopped using WSL due to performance issues, and don't usually run it now... but even with normal powershell, the performance of, for example yarn install is not anywhere near linux when defender is running. You can see some benchmarks in the yarn issue listed above. I realise I'm not adding much to the WSL part of the discussion, but just wanted to add some details for the defender team's benefit.
@sunilmut this should be very easy for you folks to replicate and grab as many benchmarks as you want.
- Install WSL
- Install git/gnumake, etc.
- Clone kubernetes/kubernetes (for example, or any large repo)
- See your CPUs pegged at 100%
- Profit
Aww, just ran into this, while I was trying the Bash for Windows 10 for the first time. It actually freezes the whole Webpack process and it will never finnish, no matter how long you wait.
Guess the Bash is still unusable for web developers.
Blaaaah this is killing me as projects get bigger. Has nobody at Microsoft looked at all? I've seen a ton of closed threads but no action. I have to disable Defender all the time to get Git to finish in reasonable amounts of time (2s for a checkout with it disabled vs over a minute). I've tried the workaround - it does nothing.
Same here. I also encounter a lot of disk errors while doing large npm installs. I can see windows defender killing my CPU even though I have a total exclusion on my E drive. It seems like npm may be caching things on an un-excluded directory before unpacking which is triggering windows defender and causing disk issues.
The team is looking into this and will continue investigating. FWIW this rolls up into a broader filesystem perf improvement endeavor.
In the meantime if you can provide us with more specifics so we can generate a repro that would be very helpful. @ow which repo are you using? @justinmchase which errors do you hit?
@tara-raj thanks for the update. Just to toss my own experience out there, I am running 16299 Enterprise and my Windows Defender is locked on due to company policy. As soon as I start WSL I see constant CPU usage from the Defender service hovering around 25%. This goes away the moment I shut down WSL. Inside I am just running tmux and bash, running ssh does not seem to affect it much. Any filesystem actions can take a while especially on /mnt/c. The high CPU usage reduces my computers overall performance and especially the battery life. Adding an exception on the Linux rootfs folder helped, it was 50%+ without it, but it does not solve the issue entirely.
Thanks for looking into this.
Here is a pretty simple repro for the bug I keep seeing:
- In WSL install node v8.7.0
- Unzip attached file to
/mnt/c/bug, cd into dir npm install
This could be a node/npm bug... I'm not sure yet but I use the same directory linking feature on osx and it doesn't have this problem. The problem appears to be a combination of being in a linked directory and having the same dependency in both modules. Also i am doing everything under /mnt/e, which I have an exclusion for and ends up being reasonably fast but if I were to copy these files to mv bug ~/bug it will trigger windows defender scans which massively slow it down. Also global npm installs or installing a module for the first time which I think is globally caching off of /mnt/e. If I knew where the unmounted files for wsl were located I'd add that to windows defender exclusion. Might be nice if that was done during wsl install by you guys too.
Error code:
22:04:52:justin:/mnt/e/code/bug$ npm install
npm WARN bug@1.0.0 No description
npm WARN bug@1.0.0 No repository field.
npm ERR! path /mnt/e/code/bug/test/node_modules/mongoose/node_modules/async
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/async' -> '/mnt/e/code/bug/test/node_modules/mongoose/node_modules/.async.DELETE'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm ERR! A complete log of this run can be found in:
npm ERR! /home/justin/.npm/_logs/2017-12-20T04_04_57_053Z-debug.log
I'm having heavy performance issues related to Windows Defender and WSL as well. For me, it's when I'm switching tmux panes and opening files in vim. Disabling realtime protection in the Windows Defender Security Center causes a massive (>100x) speedup in performance for these tasks.
I'd really appreciate it if anyone had a workaround besides just disabling realtime protection.
Microsoft Windows [Version 10.0.16299.125]
Linux goose 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux
FWIW this rolls up into a broader filesystem perf improvement endeavor.
@tara-raj Would this endeavor be limited to WSL, or would also transcend to Windows' NTFS filesystem too?
@tara-raj I can get the poor performance to occur on almost any repo.
Do a git clone on something like this - the performance will be slow in general, but can push it further by committing new changes or just installing with npm - in all seriousness I'm turning Defender off 6-8 times a day to get the performance bottleneck out of the way. I've not timed it, but I'd hazard a guess it's more than 4x as bad with Defender enabled.
There's a ton of documentation about this across repositories — Yarn has a bunch of ongoing threads about this and has been writing workarounds to handle it. Generally speaking, WSL works great, but I will say these performance issues have been crippling and frustrating.
I'm working on migrating from the old version of bash to the windows store version of bash and cannot find the new location of the lxss folder. Anyone know where volfs is being stored these days?
For those here having the same issue, I've just completely given up on Defender pinning my CPU all the time. Would prefer to have it on, but currently nigh unusable on my setup.
If you want to force Realtime protection off, you can use this registry key, since Microsoft insists that it will turn itself back on within a few hours on Fall Creator's and up.
Paste below into a file, add .reg extension, double click. Reboot and boom, hey, it's actually usable.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender]
"DisableAntiSpyware"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection]
"DisableBehaviorMonitoring"=dword:00000001
"DisableOnAccessProtection"=dword:00000001
"DisableScanOnRealtimeEnable"=dword:00000001
The sad part is that this issue was reported over 8 months ago and we've yet to see any progress or patch to remedy the issue. Instead of having to disable Windows Defender, they should revert to the way WSL worked before and let it be its own isolated subsystem. Integration with Windows is what caused all these perf problems.
Please bear with us. We are investigating. If it was a simple issue to fix, it'd have been fixed by now 😉
@ow Weird, I tried cloning crafty-vagrant, both within the WSL file system and in /mnt/c and neither seemed to take super long, nor did my CPU go above 15%. I only heard about WSL a week ago and installed it very recently. Have you been using it for long? Maybe you've got an older version installed? Or some cruft left over from an old version?
@marcfor I have been using WSL since the day it launched, and have written much about my experience with it so far. Initial repositories appear fine, but once you begin to get complex directories in them -- run npm install, composer install, vagrant up on Crafty-vagrant -- then start trying to commit on a regular basis and it's agony. I can reproduce the behavior across three machines, one is just a few weeks old.
I once ran git status in a big repository under /mnt/c from WSL, it's unresponsive for a minute.
While the same operation using the Windows native git is completed within one second.
I suppose this is also related to Windows Defender.
As an FYI. We use the WSL Switcher to use centos under WSL. Adding exclusions helped improve git performance by 50%.
$lxss = $env:LOCALAPPDATA
Add-MpPreference -ExclusionPath "${lxss}\lxss"Adding: @bitcrazed , please feel free to reach out to me. Our team uses WSL extensively, and we're happy to get you use cases / situations to reproduce.
Cannot reproduce effects of disabling real-time protection or adding exclusion to project folder. Most noticeable is when running yarn start. On MacOs this takes roughly 2-10s, on Ubuntu 16.04 around 2s and on WSL I get between 40-60s which is unbearable. rebuilding takes 200-800ms on Ubuntu, 1-2s on MacOs and 9-20s on WSL. No effect on this by changing Defender/Firewall settings
Cannot reproduce effects of disabling real-time protection or adding exclusion to project folder.
Note there are a few unrelated patterns that show poor performance on WSL. Turning off Defender will probably (possibly?) help for the many itty-bitty-files case. It won't save you when doing git status on a ginormous repo on /mnt/c though, for example. It won't help at all for other cases, notably when thrashing large blocks of virtual memory. I am not saying that's what is happening with yarn. Maybe there are other patterns, who knows. It is difficult to get good data. Worse, #1577 gperf doesn't work. The most constructive thing to do, for now, is generate a strace -tt and start looking for red flags. Often where the program is spending the time is obvious. Understandably this is not always a very satisfying exercise, because you find out "it's that" and then the follow-up is "yeah it's that". But it does help.
I can reproduce this very easily, doing a large cp with real-time protection turned on and off. Open and close calls in particular seem to take a long time, for example with protection then without protection:
10:15:49.752801 lstat("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000075>
10:15:49.752940 stat("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000063>
10:15:49.753057 open("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_RDONLY|O_NOFOLLOW) = 3 <0.003043>
10:15:49.756173 fstat(3, {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000042>
10:15:49.756358 open("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_WRONLY|O_TRUNC) = 4 <0.001099>
10:15:49.757559 fstat(4, {st_mode=S_IFREG|0777, st_size=0, ...}) = 0 <0.000071>
10:15:49.757727 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000059>
10:15:49.757878 read(3, "$ Grid indexing is 1-based\n\n$ 15"..., 131072) = 889 <0.000099>
10:15:49.758068 write(4, "$ Grid indexing is 1-based\n\n$ 15"..., 889) = 889 <0.000110>
10:15:49.758231 read(3, "", 131072) = 0 <0.000071>
10:15:49.758356 close(4) = 0 <0.000809>
10:15:49.759221 close(3) = 0 <0.000045>
10:15:17.106272 lstat("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000061>
10:15:17.106379 stat("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000056>
10:15:17.106483 open("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_RDONLY|O_NOFOLLOW) = 3 <0.000093>
10:15:17.106655 fstat(3, {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000066>
10:15:17.106800 open("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_WRONLY|O_TRUNC) = 4 <0.000260>
10:15:17.107106 fstat(4, {st_mode=S_IFREG|0777, st_size=0, ...}) = 0 <0.000024>
10:15:17.107179 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000013>
10:15:17.107229 read(3, "$ Grid indexing is 1-based\n\n$ 15"..., 131072) = 889 <0.000034>
10:15:17.107301 write(4, "$ Grid indexing is 1-based\n\n$ 15"..., 889) = 889 <0.000041>
10:15:17.107381 read(3, "", 131072) = 0 <0.000025>
10:15:17.107442 close(4) = 0 <0.000077>
10:15:17.107554 close(3) = 0 <0.000027>
Note there are a few unrelated patterns that show poor performance on WSL. Turning off Defender will probably (possibly?) help for the many itty-bitty-files case
Erm, I don't know if I buy this. Git status, commit and anything NPM related improve dramatically with Windows Defender disabled. I had to resort to force disabling via Group Policy to make any of these tools usable.
Well, that the performance of the filesystem of WSL is slower than an actual system is a known issue since day 1 and well documented in benchmarks (e.g. Phoronix). The team even acknowledged that improving the performance is a high priority goal at the moment.
The virus scanner performance impact also has always been there for me and if anything filesystem-related is happening it gets involved. McAfee is even worse than Defender. Disabling the Defender real-time protection helps but it's still held back by basic performance limitations.
I thought somebody once mentioned that it is a known issue caused by the way WSL is stored and it it would require reworking of it to get this fixed.
So far it could live with it and had to move some more complex things I did on WSL to an real VM eventually because of other reasons so I hit me less. But since CLion is now having support for WSL and having no control over the virus scanner at all in an corporate environment makes this issue more severe for me now.
Just as a benchmark, I find turning off Windows Defender speeds up cp of a large directory by about a factor of 8.
Just to reinforce this (old) issue that isn't solved yet, I just did a sudo apt upgrade. Look at this picture and see who is always fighting the Dpkg process for resources:
If I'm compiling something, it is even worse... I think the exclusion of the source directory is being totally ignored.
I understand that disable Windows Defender on registry is not the proper fix as we would have other security issues later on, but have this issue that long open is frustrating. We see more and more fixes/enhancements getting added to WSL but this, which is critical to any use case, is not sorted yet. :(
just noticed this too, windows 10 pro version 1709, it looks like even reading simple file repeatedly triggers this. This one just flies with zero cpu taken by antimalware service
while :; do echo -n Hello ; done
And this one takes >10% cpu by antimalware service and the output is slow as typewriter
echo -n "Hello" >file
while :; do cat file ; done
And btw I have excluded %LOCALAPPDATA%\lxss folder but don't see any difference
Doing an apt-get upgrade can also cause some totally gnarly disk access
Doing an apt-get upgrade can also cause some totally gnarly disk access
To the point people think it is stuck.
A few people in here have said that adding an exclusion in Windows Defender does nothing but I wanted to go ahead and give it a try anyway. A quick google search for a directory that I should exclude was C:\Users\{user}\AppData\Local\lxss, but pretty sure that was legacy.
Most instructions I could find, granted was a quick search all pointed to excluding a directory, but I noticed that Windows Defender Security Center also allows excluding process. So I excluded
Apt
Bash
Dash
Dpkg
Init
Mandb
Sudo
Ubuntu
And noticed an immediate improvement. Although it is not blazing fast, it has sped up a little bit during this huge install it was already working on. Also for the first time during this install, I am not seeing my disk constantly at 100%. I'm at 58% right now, yes - I have seen it go back up into the high 90s but it goes back down. This is much preferred to constant use.
As for the speed increase. I cant say for sure. I was mid install but it definitely seems like its going faster. Even if not, I will take the disk saving. One thing I also noticed is there is a subprocess of Ubuntu, ubuntu - but Windows Defender will not allow adding Ubuntu and ubuntu. Not sure if that's an issue.
@gmcclins Is this on Skip-Ahead Insiders? They're supposed to allow per process control now. If you are on Skip Ahead (and the reason you were able to able to add those processes as I suspect it is), I would create an issue for not allowing whatever sub-processes can't be added.
@DarthSpock I am on Windows 10 Pro Version 1709 (OS Build 16299.309). I am in Release Preview. Also I edited the post now, but it seems my grammar check made things unclear. What I was trying to say was there is a process Ubuntu and then also ubuntu. I'm not sure how Defender handles a difference in capitalization and it won't allow adding both.
I reset Ubuntu back to default today and am currently updating. I'm about 15 minutes in and at 37% so it is still pretty slow and no surprise Antimalware is chugging away
There is still the improvement in disk use, it gets high at times, but it's not a constant 100% anymore.
There are probably other processes that should be excluded but what I have so far I got from performing an update and install within Ubuntu and watching Windows Task Manager. As far as the blocking itself, Windows Defender Security Center, Virus & threat protection settings, Add or remove exclusions.
Ah ok, that makes a difference. Yeah, you have to exclude the entire rootfs for all pico processes to run without errors or latency due to Defender (or any other A/V/FW vendor for that matter). It isn't a satisfactory solution but a workable one until the next release of Windows or upon your A/V/FW vendor supporting pico processes.
As far as case sensitivity goes, this is fixed in the Insiders and will be available in 1803.
Can confirm that I just installed Ubuntu yesterday and after installing all the goodies whenever I launched Ubuntu bash it would slow down to like 20 seconds before I get the terminal prompt.
After adding the exclusion it only takes 3 seconds now. This is only on startup, not sure that i'll experience further lag doing other work but i'll watch out for it.
After the series of updates (17120 -> 17128) I'm also facing both #3034 and this (Windows Defender high CPU load) issues. Even after full stack of exclusions I can't run dev tools (tests for example, check attached pics please).
Same here, I don't understand why microsoft defender won't wsl by default?
upgrade to 17133 for the other issue. As far as Defender goes, you'll have to be on Skip-Ahead to see the change in Defender's behavior. SCU won't have it unfortunately.
Be wary of skip ahead, last I chefked has a breaking change where Hyper V is enabled by default on all Windows 10 versions. You won't be able to boot VMs with anything else.
I just disable real time protection all the time now and performance is amazing - but this really isn't a solution. It's a real shame.
After going 13+ hours for a large install I refused to make any major changes for the last couple of months, but then got a dreaded Error while moving old database out of the way. AppStream cache update failed and decided to clean which also failed, so a fresh restart was in order and I figured I'd add a little timing info for those deciding if they want real-time on/off...
Fresh ubuntu install; first upgrade on a 5400rpm platter drive without Realtime Protection and with.
Personally, I'll be disabling when doing large installs (like Tex Live) and living with the pain until some form of relief comes along...
PS - Windows 10.0.16299
At this point I have the windows defender config panel open in the background, and I just repeatedly go to it and turn real time protection off while I'm working. It's a real pain, and it needs to be fixed. Not to mention that it leaves my PC vulnerable.
what is the progress here?
@Deshke I have received many reassurances this is being looked into but little feedback or insight. I'm on the bleeding edge builds from Skip Ahead and there is no improvement as far as I can tell.
The only solutions I've found to date are:
- Forcefully disabling Windows Defender via Group Policy (huge impact, worth it if you are able to manage in this scenario but be aware of the implications.)
- Disabling Bitlocker entirely (minimal impact, don't do it)
thanks @ow but both workarounds should not be used
The status is as it was when I last replied to this thread:
I appreciate this is frustrating, but the fact that it's not yet resolved is just a glimpse into how complex this issue is.
For those who're interested, #873 is the main issue tracking disk perf issues in general, and you'll find more commentary, and detailed feedback there, including from the engineer responsible for most of WSL's disk IO engine.
You can be sure that as soon as we have a decent solution ready for testing that this issue will be updated, and there will be much rejoicing on Twitter and elsewhere.
Until then, please bear with us while @tara-raj works across several teams to figure out and implement a decent solution.
Thanks.
Thanks for the update @bitcrazed.
Looking forward for the final fix.
Hi I am facing similar issue when i start any node.exe process even as simple as npm -v on my windows 10 laptop with pretty good configurations as i7 and 8 gb RAM it takes around 40 seconds to come up with version number. My previous laptop with much less configuration took 3 to 4 seconds, is it windows defender causing this issue , i have tried disabling it and adding exceptions in my symantac antivirus but still no progress. Any help would be appreciated
@rohityadavrv If you're running Symantec, then Defender should be disabled by default. If you're running node.exe, then that is just typical Windows behavior and not pertinent to WSL (though Windows itself has suffered as a result of changes to Defender). If you run the Linux node binary however, then I would suggest creating an exception for the entire rootfs which you will find under AppData in your user profile under the distro in use and see if that helps. If not, then you need to file a bug with Symantec to implement WSL support. There's a MS blog post for them to do that and MS is willing to work with A/V vendors to help support WSL. That said, there are other underlying perf issues. Defender is just one of them (if you're only running it and not something else).
In my experience, adding the WSL Linux filesystem root to the directory exceptions for Defender is ineffective, or at least, doesn't make a significant difference to the AV chewing the CPU inspecting running processes. The only method I've found effective is adding each WSL Linux process I use commonly to the process exclusions.
Works better in Avast (which disables Defender), Symantec may or may not get the same results. Both options are valid. This situation is better in Insiders for Defender with Linux processes being treated the same as Windows processes (which makes your method the "right" way when the next build is released)
Here's another workaround which works for me: for each directory in your linux path add a process exclusion rule with the equivalent windows path with a wildcard. So, instead of excluding just bash with a process exclusion rule of C:\Users\ipc\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs\bin\bash use C:\Users\ipc\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs\bin\*. I did that for /bin, /usr/bin, /sbin, /usr/sbin... basically every linux dir in my $PATH that had an executable in it. I also added a folder exclusion rule for the base dir of the installation because that prevents the executable itself from being scanned by real-time (see this page for details).
With that change I can leave real-time scanning on; my compile times in WSL are just as fast as they are when Defender's real-time scanning is disabled and Defender no longer spikes the CPU as much. Defender still takes up some CPU time but it didn't seem to affect my build times so I'm not too concerned.
Here's a powershell script that I was using to add my rules. Change the username and package name to your own usernames and package.
Does this work for anyone else's test case?
thanks @DarthSpock and everyone. @DarthSpock , unfortunately i cant use WSL , but these slowness issues goes away in safe mode. Moreover for some of our guys the speed is ok for windows 10 and for some it slow even on Windows 7 and for some not. We need to find out if it is antivirus which is interfering with it or is it Forcepoint DLP.
Forcepoint DLP
Haven't used that particular product but having experience with McAfee DLP, I'd say if it wasn't trusted or exempted, that could cause problems to performance if not outright blockage of some executions. (Thankfully I don't use WSL with DLP installed so don't have this particular factor to consider) Another factor may just be the combo of the Symantec being used with Forcepoint in terms of performance. Try doing a comparison of removing one, executing a large binary, then reinstalling the one you remove and remove the other product and run some benchmarks for both of those scenarios. Also, since this issue is for Defender, you may be better off on the Symantec and/or Forcepoint forum.
@ian-p-cooke your solution to exclude processes in bin directories works for me on Ubuntu 16.04 too. Thanks!
Last update (Windows 10, version 1803) was on 6/5/2018
I use WSL nearly daily. On 7/27/2018 I went from it being perfectly usable to 10s of seconds to perform cd ~/git/<TAB>, git status, etc. I found that MsMpEng, via resource Monitor, was at 25% CPU utilization and was performing a steady 1+MB/s read.
This was observed through this morning, 7/30/2018
EDIT: It looks like it might now be resolved (more testing needed). I checked Windows Defender and I did get a definition update at 6:09AM on 7/30/2018 which was after I experienced it this morning. I wonder if I just got a bad definition.
Thanks @ian-p-cooke your solution appears to have helped 👍
Although I have only added the paths explicitly mentioned, if it's still playing up I'll attempt to add everything from my path.
Is there a way to identify where you have executable files located in bash?
Thanks @ian-p-cooke I think I can finally use git efficiently in WSL. Thanks!
@spences10 The executables can vary a bit, but 99% of the time they're in standard locations. Starting from the WSL root filesystem directory in Windows, which is something like C:\Users\YOURUSERNAME\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs the paths to the Linux executables will be be at:
bin\*sbin\*usr\bin\*usr\sbin\*usr\local\bin\*usr\local\sbin\*
This is using Ubuntu - the list might vary in CentOS or other distro.
If you have special cases, you could add those too. For example Ian Cooke's powershell script also excludes home\$linuxuser\.cargo\* - I don't know what .cargo is for, presumably some build tool that he uses. You probably don't have that and can omit it.
I've tried the script and changed the .cargo entry to point to a toolchain I installed in my home directory but no such luck, AntiMalware still pins the CPU.
I can see things like cc1 and cc1plus up there in the task manager every now and then, but even at that I manually added the libexec directories containing binaries to the exclusions but with no luck.
I don't see CPU spinning a lot by the Defender process. I see I/O reads/writes overdose though
@bitcrazed hey Rich, is there a good place to follow along for the status on this, are you having any luck getting this solved? :)
@ian-p-cooke works for me, thanks!
For what it's worth, I have this issue outside WSL with Windows git as well. A simple git branch on the windows command line takes minutes. So I'm not sure this is a WSL problem, but it's a critical issue impacting my entire team right now.
@mlhpdx Did you add an exclusion to Defender for the folder that your source code lives in?
You guys literally saved my life! My WSL is performing like hell after running the script! It's a world of a difference! Welcome back productivity! :-)
None of these suggestions worked for me unfortunately. I'm using "git status" to test and the same exact project which takes 0.4 seconds on my Ubuntu running within VirtualBox takes over 4.4 seconds on WSL. I've not disabled real time protection because that seems like a major security threat. So WSL is just about useless for real work, unfortunately.
Same. None of the suggestions work for me, too - on my primary PC, which I don't have admin rights to (therefore cannot use these workarounds), these workarounds are hopeless. On my secondary PC, I've tried it, and got similar results to @colinmollenhour . If virtualbox can do it, then it's certainly possible (yes, virtualbox requires a driver - but the performance should still be in the same ballpark. Please don't do anything to WSL that makes it require admin rights on startup/for simple tasks, the guy who as admin rights to my computer is very rarely around).
hoW aBoUt opTimIze WiNdoWs defeNDEr, oR mAkE It noT SCAn EveRy sIngLe file BY DEFAULT (IT hAS tO NOT DO ThiS BY DEFAULT becauSe THeRE ARE sTANdArD uSERS OuT tHEre, AND somE, LiKE mE, WiLl gO fOr 6-7 MoNtHS At a TiMe wIThoUt COntAct WiTH tHe PerSon wHO hAs tHE aDmIn accOuNt)?
Windows defender's probably pretty optimized already tho.
@colinmollenhour Did you try the ruby script from here:
https://gist.github.com/dayne/313981bc3ee6dbf8ee57eb3d58aa1dc0#file-1-wsl-defender-fix-md
I had to delete all the Windows Path variables from the output. But after running the script in an Adminstrator Powershell, "git status" was about 10x times faster than before.
I didn't use the Ruby script but rather hand-edited the Powershell script. Afterwards a large number of new entries were seen in the excludes list. It didn't help me.
as @bitcrazed stated
"Dear WSL fans, despite the perf impact of Defender (which we are working to address), THIS is why we DO NOT recommend excluding folders from or disabling Defender"
(https://twitter.com/richturn_ms/status/1095207032292929536)
So, in order to check about the interaction between WSL and Windows Defender, I ran extensive tests with the configure script on a large package (emacs, git master branch).
Actually I ran:
time (make distclean ; ./configure)
My conclusions are as follow :
- without adding any exceptions, the real time was about 2'30, user time was 0'21 and sys time 1'19
- adding a directory exception to Defender with the rootfs of the WSL distribution dropped the runtime to 2'00 and the user time and sys time did not change at 0'21 and 1'19
- adding any other processus exception did not change this result at all (I even added a '*' processus exception which had no effect)
- disabling the real time protection dropped the real time further to 1'45, user time and sys time stayed constant (and btw real is now almost equal to user + sys)
I would not advise to disable the real time protection. However, I have added the directory exception to Defender.
Regards,
Do not exclude any directories - take the hit or switch to WSL2
see #1932 (comment)
Just want to throw in there something positive for a change—after all of this thread and the frustrations with slowness, Microsoft really came up with a great solution in WSL2, and I'm pretty excited about what that means going forward. I'll have to break my own no-beta rule and install Insiders for this, because it's going to change everything!
From the article linked:
Initial builds of WSL 2 will be available through the Windows insider program by the end of June 2019.
It's great that they've come up with a fix but that doesn't change the fact that this is an issue right now.
#somethingwehavetolivewith #itisnotthatbad #iuseitdailyanditsfine #justsayin
@lewisdonofrio Yeah, that's pretty unfair—I can't work on a number of my repos in the current scenario without disabling Windows Defender entirely, which I'm doing daily, because they take 5-10 minutes to build vs less than 30 seconds without. This is a real problem, and discounting other people's well-documented problems with it isn't helpful.
it's been solved in WSL2 go join insiders and enjoy the power.......
@lewisdonofrio a) it's not even in insiders for another month or so as far as I can tell and b) of course it's great that it's been solved and that some are willing to help test. But it's super counterproductive to claim that there is an immediate solution right now when taking it involves trading the stability of your entire operating system by joining the fastest insiders track...
@ow @anstosa I would say it is safe to assume at this point that there is no easy (or even remotely feasible) fix for this issue in WSL1 and this is one of the top motivations for WSL2 so this simply won't get fixed in WSL1. No amount of complaining and begging is going to change this.
The terrible performance in WSL also makes it unusable for my work, which is why I've chosen instead to... not use it and do my work inside a VM until a better solution becomes available. If WSL is not the best solution then quit using it.
@colinmollenhour I don't think anyone here is claiming that WSL2 isn't the fix for this issue. We're just pointing out that @lewisdonofrio is being insensitive by claiming that it's fixed now because it just isn't.
Just wondering how this is a 'me issue' when the advisory came from @bitcrazed
@lewisdonofrio He made one post which was factually accurate and recommended people not take action to mitigate the problems (reasonable, constructive). You're making many posts which are factually inaccurate and insisting that people do what you say (unreasonable, just plain annoying). I won't reply again, sorry everyone else for the noise...
You were saying to speed up wsl1 exclude directives - exactly what he stated not to do, and that this was already fixed in wsl2....that is all.
Seems with Version 1903 McAfee seriously screwed the pooch. Execution of certain processes in WSL now takes ages and McAfee Process Validation Service is going totally bonkers now. Will try to open a ticket with them as well.
@firewave Strongly recommend dumping McAfee if there is any possible way. Windows already has a built-in anti virus and, to put it lightly, its more trustworthy.
Suggesting different antivirus software isn't a solution to the problem. Many companies enforce policies that require specific antivirus or firewall software, which may not be user-configurable.
There definitely seem to be changes in 1903 that allow Defender to get out of the way of other installed real-time antivirus scanners. While that is helpful from a system management viewpoint, it also means that these antivirus scanners may not know what they should or shouldn't be scanning, so they just scan everything all the time.
I know this is not perfect, but SSD helps a lot if you can't turn off third party scanners.
@ithiru I'm using SSDs and the performance is still horrible. I see this regularly with Debian 10 using apt, like:
sudo apt-get update
sudo apt-get dist-upgradeAlways very slow, Task Manager shows Antimalware Service Executable busy. The same thing under Hyper-V is faster (and that one is backed by an old spinning disk).
I do not know what you all are talking about with performance issues, I live daily in xfce4 on my w10 workstations - tinyurl.com/donofrioworkremmina2020 and I daily update and it is fine, I mean sure firefox is not working but arora does everything I need within wsl1 and ubuntu 18.04 granted I do mostly ssh/rdp (yes I rdp from ubuntu because I use remmina - it is cross platform) I but I have used many application/projects from aweather to ansible. my update as documented in my raw setup text file in that link:
"sudo apt-get update && sudo apt-get upgrade -y && sudo apt-get dist-upgrade -y && sudo apt-get autoremove -y && sudo apt-get autoclean -y && sudo sync && sudo sync"
@drichardson I am guessing your Debian distro is still running on WSL1.
When you run wsl.exe -l -v at your Windows command-line, what do you see?
C:\> wsl -l -v
NAME STATE VERSION
* Ubuntu Running 2
openSUSE-Leap-15-1 Stopped 1As you can see from above, I am currently running Ubuntu on WSL2 and openSUSE on WSL1.
If your Debian is running as Version 1, then you're running Debian atop WSL1 whereupon you'll likely bump into the well known WSL1 file IO perf challenges that come from running Linux binaries atop Windows IO stack. If, on the other hand, you run a distro atop WSL2, your distro/binaries will be running in a container directly on a Linux kernel instance in lightweight VM, which doesn't use Windows' IO stack and isn't affected by anti-malware tools like Defender etc.