Parallel fabricate on windows fork-bombs.
GoogleCodeExporter opened this issue · comments
What steps will reproduce the problem?
1. Set 'parallel_ok=True' with any fabricate script.
What is the expected output? What do you see instead?
1. Parallel goodness.
What version of the product are you using? On what operating system?
1. Tip of tree.
Please provide any additional information below.
main(parallel_ok=True, jobs=1) begins creating the same command over-and-over,
never stopping.
Original issue reported on code.google.com by jaros...@gmail.com
on 7 Jan 2013 at 11:20
I can't reproduce this with the following script:
import fabricate
def build():
fabricate.run('dir', '/w', shell=True)
fabricate.main(parallel_ok=True, jobs=1)
Can you attach (a minimal version) of the script you're using to reproduce this?
Original comment by benh...@gmail.com
on 8 Jan 2013 at 1:37
- Added labels: ****
- Removed labels: ****
As Lex Trotman noted on fabricate-users, parallel jobs only works with strace
runner, which is not available on Windows. This is automatically detected in
the code, however -- line 921 of fabricate.py checks that the system supports
strace:
self.parallel_ok = parallel_ok and is_strace and _pool is not None
Original comment by benh...@gmail.com
on 8 Jan 2013 at 1:38
- Added labels: ****
- Removed labels: ****
from fabricate import *
import os, shutil
def build():
for i in xrange(10):
run('dir', '/w', shell=True)
def clean():
autoclean()
main(parallel_ok=True, jobs=2)
Causes the fork bomb. Windows 8; Python 2.7.3 x64.
Also, have you considered the use of the SDK took, 'Tracker.exe'. You can find
it in
c:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\bin\NETFX 4.0 Tools\Tracker.exe
Although it exists in other SDKs (7.0), and is redistributable. The usage of
tracker is like this:
tracker.exe /if <intermediate-files> /c <command-line>
This results in a set of files printed to <intermediate-files> of the form:
xxx.read.tlog
yyy.write.tlog
Where each line in the *.tlog files after the first line are the files accessed
(either read or written).
Original comment by jaros...@gmail.com
on 8 Jan 2013 at 10:30
- Added labels: ****
- Removed labels: ****
I see, explicitly specifying jobs > 1 tries to use multiprocessing even on
Windows. Wonder why that is -- we'll look into it. Thanks.
Oh, and thanks for the pointer to Microsoft's tracker.exe -- that looks great,
basically like strace for Windows! What do you mean by "redistributable" -- the
copyright notice on my version says "(C) Microsoft All rights reserved".
Original comment by benh...@gmail.com
on 9 Jan 2013 at 7:16
- Added labels: ****
- Removed labels: ****
Usually, MS executables that aren't marked as redistributable may not be
redistributed outside of MS official distribution. Executables marked as
redistributable can be packaged outside of official distribution.
I suppose the proper strategy would be to try to detect the existence of
tracker.exe in SmartRunner, and not deal with distribution headaches.
I've attached a hacked-up version of a 'TrackerRunner' as a patch file. This
still craps out on Windows with parallel jobs, though; not sure why. Works well
enough for my simple projects, though.
Original comment by jaros...@gmail.com
on 9 Jan 2013 at 7:28
- Added labels: ****
- Removed labels: ****
Original comment by jaros...@gmail.com
on 9 Jan 2013 at 7:30
- Added labels: ****
- Removed labels: ****
Attachments:
Very cool, thanks. To keep things separate, could you add those TrackerRunner
patch to issue 3?
You indicated in comment 3 that tracker.exe is marked as redistributable. Where
does it say that, or how did you know?
Original comment by benh...@gmail.com
on 9 Jan 2013 at 7:38
- Added labels: ****
- Removed labels: ****
[deleted comment]
I don't know that if it is redistributable ... I'll try to find out tomorrow,
and move the patch, then, as well. I'm no longer on a real computer.
Original comment by jaros...@gmail.com
on 9 Jan 2013 at 7:42
- Added labels: ****
- Removed labels: ****
I have been working on parallel building for windows. I now have it working
with a modified version of the Tracker patch.
I think fork issue that initially generated it issue may be down to a special
case in windows when using the multiprocessing module. You must protect the
entry point of the build script with this piece of code.
if __name__ == "__main__":
freeze_support()
main(parallel_ok=True)
I do not fully understand what this does, but it boils down to Windows no
having an os.fork() function and the way fork is emulated on a windows system.
Anyway, look out for a git push soon with the modifications that make fabricate
parallel on windows.
Original comment by simon.al...@gmail.com
on 3 May 2013 at 11:09
- Changed state: Started
- Added labels: ****
- Removed labels: ****
@SimonAlfie was this ever completed?