mx checkstyle gives "OSError: [Errno 7] Argument list too long" on MacOS
zakkak opened this issue · comments
Hi @zakkak thanks for the report. What suite are you running this on? Also, how did you come up with 21135? The safer thing to do would be to simply pass in the command line prefix to _chunk_files_for_command_line so that there's no need for heuristics at all. Any chance you want to try and put up a PR with that change?
Hello @dougxc.
What suite are you running this on?
This is with the maxine suite https://github.com/beehive-lab/Maxine-VM/tree/develop
Also, how did you come up with 21135?
I came up with 21135 through trial and error, no calculation in there.
The safer thing to do would be to simply pass in the command line prefix to _chunk_files_for_command_line so that there's no need for heuristics at all.
The thing is that the prefix is 680 characters long :/ (see bellow), so I don't see any connection with
21135.
env MX__SUITEMODEL=sibling LD_LIBRARY_PATH=/Users/zakkak/Code/maxine-graal/maxine/com.oracle.max.vm.native/build/darwin/substrate MX_HOME=/Users/zakkak/Code/maxine-graal/mx MX_PRIMARY_SUITE_PATH=/Users/zakkak/Code/maxine-graal/maxine MX_SUBPROCESS_COMMAND_FILE=/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/mx_subprocess_command.vwJalS \
/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/bin/java -d64 -Xmx1g -jar /Users/zakkak/.mx/cache/CHECKSTYLE_6.15_db9ade7f4ef4ecb48e3f522873946f9b48f949ee/checkstyle-6.15.jar -f xml -c /Users/zakkak/Code/maxine-graal/maxine/com.sun.max/.checkstyle_checks.xml -o /Users/zakkak/Code/maxine-graal/maxine/checkstyleOutput.txt
Any chance you want to try and put up a PR with that change?
The only reason I did not is because last time I opened a PR there was some issue with my OCA form, so for this tiny change I thought it was not worth trying :)
Can you please try this patch to see if it fixes the problem.
I tried the patch and unfortunately it does not work.
The reason is that it does not account for the length of the prefix added by the run_java method in checkstyle.
In maxine's case this is
env MX__SUITEMODEL=sibling LD_LIBRARY_PATH=/Users/zakkak/Code/maxine-graal/maxine/com.oracle.max.vm.native/build/darwin/substrate MX_HOME=/Users/zakkak/Code/maxine-graal/mx MX_PRIMARY_SUITE_PATH=/Users/zakkak/Code/maxine-graal/maxine MX_SUBPROCESS_COMMAND_FILE=/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/mx_subprocess_command.vwJalS \
/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/bin/java
I guess (didn't try it) it will also not always work for the formatter as well, since env is not taken in account.
Can you please provide me steps to reproduce including which GitHub repo to clone.
The env line is not part of the command line so that should not matter. It's just printed to look like a command line prefix to make it easier for copy-pasting when reproducing a failed mx command.
I've pushed a branch that tries to account for the run_java prefix: https://github.com/graalvm/mx/commits/ds/GR-14378. Please let me know if that works.
Can you please provide me steps to reproduce including which GitHub repo to clone.
git clone --branch develop --depth 1 https://github.com/beehive-lab/Maxine-VM.git maxine
cd maxine
export MAXINE_HOME=$(pwd)
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/
mx --suite=maxine checkstyle -f
I've pushed a branch that tries to account for the
run_javaprefix: https://github.com/graalvm/mx/commits/ds/GR-14378. Please let me know if that works.
I don't have access to that link, I also checked on the repository for other branches but did not find any.
Thanks. I'm going to apply this patch:
diff --git a/mx.py b/mx.py
index 19ac544..99d4a44 100755
--- a/mx.py
+++ b/mx.py
@@ -12842,20 +12842,20 @@ def _chunk_files_for_command_line(files, limit=None, separator=' ', pathFunction
chunkSize = 0
chunkStart = 0
if limit is None:
- commandLinePrefixAllowance = 3000
if get_os() == 'windows':
# The CreateProcess function on Windows limits the length of a command line to
# 32,768 characters (http://msdn.microsoft.com/en-us/library/ms682425%28VS.85%29.aspx)
- limit = 32768 - commandLinePrefixAllowance
+ limit = 32768
else:
- # Using just SC_ARG_MAX without extra downwards adjustment
- # results in "[Errno 7] Argument list too long" on MacOS.
- commandLinePrefixAllowance = 20000
syslimit = os.sysconf('SC_ARG_MAX')
if syslimit == -1:
syslimit = 262144 # we could use sys.maxint but we prefer a more robust smaller value
- limit = syslimit - commandLinePrefixAllowance
- assert limit > 0
+ limit = syslimit
+ # Reduce the limit by 20% to account for the space required by environment
+ # variables and other things that use up the command line limit.
+ # This is not an exact calculation as calculating the exact requirements
+ # is complex (https://www.in-ulm.de/~mascheck/various/argmax/)
+ limit = limit * 0.8
for i in range(len(files)):
path = pathFunction(files[i])
size = len(path) + len(separator)
I've tested it with your reproducer.