Degrade from 3.4 to 3.5: Windows Protobuf Compiler does no longer understand * operator for multiple input files

Question

Degrade from 3.4 to 3.5: Windows Protobuf Compiler does no longer understand * operator for multiple input files

apacha opened this issue 7 years ago · comments

I tried to follow the instructions from tensorflow-models repo up until the point where to run

protoc object_detection/protos/*.proto --python_out=.

So I downloaded the appropriate Windows version and added it to my Path. But when running the corresponding windows command:

protoc .\object_detection\protos\*.proto --python_out=.
or even with the forward slash
protoc object_detection/protos/*.proto --python_out=.

I only get the error object_detection/protos/*.proto: No such file or directory, indicating that it does not understand the *-operator. When compiling the files individually that reside in that directory, it works as expected.

Jisi Liu · Answer 1 · Tue Nov 28 2017 05:33:34 GMT+0800 (China Standard Time)

@laszlocsomor I suspect this has something to do with the long path support.

László Csomor · Answer 2 · Fri Dec 01 2017 18:56:29 GMT+0800 (China Standard Time)

I suspect this never worked on Windows.

On Linux, Bash expands the wildcards so the main method's argv already contains the list of files, not the wildcard.

Alexander Pacha · Answer 3 · Fri Dec 01 2017 21:22:51 GMT+0800 (China Standard Time)

@laszlocsomor Very likely, this behavior never worked, but it would be nice, if it did work.

László Csomor · Answer 4 · Fri Dec 01 2017 21:55:13 GMT+0800 (China Standard Time)

I have a fix for it. Stay tuned.

Aisuko · Answer 5 · Thu Dec 14 2017 16:52:40 GMT+0800 (China Standard Time)

I have the same issue on my win10 and anaconda3 , with the instructions from tensorflow-models repo up until the point where to run

 protoc object_detection/protos/*.proto --python_out=.

Alexander Pacha · Answer 6 · Thu Dec 14 2017 18:56:59 GMT+0800 (China Standard Time)

There is a nice script here that you can use instead: tensorflow/models#1934

Aisuko · Answer 7 · Fri Dec 15 2017 13:04:29 GMT+0800 (China Standard Time)

Thanks 😘 @apacha
This my solution:

Head to the protoc releases page and download the protoc-3.4.0-win32.zip, extract it, and you will find protoc.exe in the bin directory.

Move this to models/research folder clone to local tensorflow model project

Next,you can use the protoc command like

protoc object_detection/protos/*.proto --python_out=.

Reference linking

Aisuko · Answer 8 · Tue Dec 19 2017 09:01:18 GMT+0800 (China Standard Time)

@apacha I think that i have been succeed.

Alexander Pacha · Answer 9 · Wed Dec 20 2017 05:14:10 GMT+0800 (China Standard Time)

@Aisuko You're right. This actually works in Version 3.4.0 of protoc but not with version 3.5.0 and is therefore a degrade of the Windows protoc.exe.

Adam Cozzette · Answer 10 · Tue Apr 03 2018 01:38:45 GMT+0800 (China Standard Time)

@pherl What's the current status of this issue? Is this now fixed on master?

Jisi Liu · Answer 11 · Tue Apr 03 2018 02:00:32 GMT+0800 (China Standard Time)

My understanding we probably would have to build the protoc directly using mingw in windows without using cross-compile. I will see if we can fix this in 3.6

jeffreytang · Answer 12 · Thu Jun 21 2018 00:28:41 GMT+0800 (China Standard Time)

Just tried latest 3.6.0 on Windows 10 command window, still could not recognize "*.proto"

vthoppae · Answer 13 · Wed Jul 25 2018 22:52:32 GMT+0800 (China Standard Time)

Using batch file try this option using for loop command
for /f %%G IN ('dir /b "C:\Protobuf\protoc-3.6.0\in*.proto"') do (
protoc --proto_path=C:\Protobuf\protoc-3.6.0\in --java_out=C:\Protobuf\protoc-3.6.0\out %%G
)

moshebitan · Answer 14 · Thu Aug 02 2018 22:53:18 GMT+0800 (China Standard Time)

This is one liner to replace google command for compliing proto files.
ORIGINAl (Doesn't work on windows):
protoc object_detection\protos\*.proto --python_out=.

Batch version works on windows:
for /f %G in ('dir /b object_detection\protos\*.proto') do protoc object_detection\protos\%G --python_out=.

László Csomor · Answer 15 · Tue Aug 14 2018 22:18:33 GMT+0800 (China Standard Time)

I fixed this, and it works with long paths too:

In MSYS:

$ find /c/tmp/proto/aaaaaaaa.aaa/ -name '*.proto' -type f
/c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/a.proto
/c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/b.proto

$ find /c/tmp/proto/aaaaaaaa.aaa/ -name '*.h' -type f | wc -l
0

In cmd.exe:

c:\>cd c:\tmp\proto

c:\tmp\proto>protoc-dev.exe aaaaaaaa.aaa\bbbbbbbb.bbb\cccccccc.ccc\dddddddd.ddd\eeeeeeee.eee\ffffffff.fff\gggggggg.ggg\hhhhhhhh.hhh\iiiiiiii.iii\jjjjjjjj.jjj\kkkkkkkk.kkk\llllllll.lll\mmmmmmmm.mmm\nnnnnnnn.nnn\oooooooo.ooo\pppppppp.ppp\qqqqqqqq.qqq\rrrrrrrr.rrr\ssssssss.sss\tttttttt.ttt\uuuuuuuu.uuu\vvvvvvvv.vvv\wwwwwwww.www\*.proto --cpp_out=.

In MSYS:

$ find /c/tmp/proto/aaaaaaaa.aaa/ -name '*.h' -type f
/c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/a.pb.h
/c/tmp/proto/aaaaaaaa.aaa/bbbbbbbb.bbb/cccccccc.ccc/dddddddd.ddd/eeeeeeee.eee/ffffffff.fff/gggggggg.ggg/hhhhhhhh.hhh/iiiiiiii.iii/jjjjjjjj.jjj/kkkkkkkk.kkk/llllllll.lll/mmmmmmmm.mmm/nnnnnnnn.nnn/oooooooo.ooo/pppppppp.ppp/qqqqqqqq.qqq/rrrrrrrr.rrr/ssssssss.sss/tttttttt.ttt/uuuuuuuu.uuu/vvvvvvvv.vvv/wwwwwwww.www/b.pb.h

David Zalewski · Answer 16 · Tue Feb 12 2019 14:52:29 GMT+0800 (China Standard Time)

any updates? Will the fix be merged in at some point?

Li Yangyang · Answer 17 · Thu Mar 14 2019 09:44:21 GMT+0800 (China Standard Time)

I encountered this problem recently, but unfortunately, this bug is still not fixed in the recent release of Protobuf.
The easiest way to solve this instead of manually fixing is to download the old version protoc-3.4.0-win32https://github.com/protocolbuffers/protobuf/releases?after=v3.4.1 and then following the usual steps. It worked for me.

OsamaNaeem1995 · Answer 18 · Thu Mar 21 2019 18:35:04 GMT+0800 (China Standard Time)

Thanks 😘 @apacha
This my solution:

Head to the protoc releases page and download the protoc-3.4.0-win32.zip, extract it, and you will find protoc.exe in the bin directory.

Move this to models/research folder clone to local tensorflow model project

Next,you can use the protoc command like
protoc object_detection/protos/*.proto --python_out=.
Reference linking

i'm still getting the same error ,
**no such file or directory **

Thanks 😘 @apacha
This my solution:

Head to the protoc releases page and download the protoc-3.4.0-win32.zip, extract it, and you will find protoc.exe in the bin directory.

Move this to models/research folder clone to local tensorflow model project

Next,you can use the protoc command like
protoc object_detection/protos/*.proto --python_out=.
Reference linking

i am still getting the error despite following the steps you mentioned

Xianyi Cui · Answer 19 · Mon Jul 15 2019 18:05:48 GMT+0800 (China Standard Time)

Seems that it is keeping break since 3.5.0. And this regression bug has already kept for almost 2 years.

László Csomor · Answer 20 · Mon Jul 15 2019 20:05:02 GMT+0800 (China Standard Time)

@xycui : You're right. Sorry for the long time to fix. I'm not on the protobuf team so this was low priority for me.

Xianyi Cui · Answer 21 · Tue Jul 16 2019 12:01:06 GMT+0800 (China Standard Time)

@laszlocsomor Thank you so much for the help. But it seems that the fixing PR is still pending because of the 'release tag' missing.

László Csomor · Answer 22 · Tue Jul 16 2019 15:08:35 GMT+0800 (China Standard Time)

You're welcome. I asked the reviewer to add it.

Alexander Pacha · Answer 23 · Tue Jul 30 2019 05:39:20 GMT+0800 (China Standard Time)

Halleluja!

JamesOldfield · Answer 24 · Fri Mar 13 2020 18:58:57 GMT+0800 (China Standard Time)

Just for the record: this fix was first released in protobuf 3.10. I haven't properly checked but the commit 5b6238e can be found in the 3.10.x branch.