Support for M1 Macs (osx-aarch_64)

Question

Support for M1 Macs (osx-aarch_64)

cmardini opened this issue 4 years ago · comments

% java -jar .m2/repository/com/github/os72/protoc-jar/3.11.4/protoc-jar-3.11.4.jar -pp
------------------------------------------------------------------------
Detecting the operating system and CPU architecture
------------------------------------------------------------------------
os.detected.name: osx
os.detected.arch: aarch_64
os.detected.version: 11.1
os.detected.version.major: 11
os.detected.version.minor: 1
os.detected.classifier: osx-aarch_64
Detected platform: osx-aarch_64 (mac os x/aarch64)

Would it be possible to add to protoc-jar/bin// a protoc--osx-aarch_64.exe? Temporarily, it can probably even be a copy of protoc-*-osx-x86_64.exe as it should run on M1 macs.

I also have a brew installed protoc that seems to be compiled for arm, if you need it.

% file /opt/homebrew/Cellar/protobuf/3.14.0/bin/protoc
/opt/homebrew/Cellar/protobuf/3.14.0/bin/protoc: Mach-O 64-bit executable arm64

% ls -l /opt/homebrew/Cellar/protobuf/3.14.0/bin/protoc
-r-xr-xr-x  1 cmardini  staff  71872  4 Jan 16:22 /opt/homebrew/Cellar/protobuf/3.14.0/bin/protoc

% md5 /opt/homebrew/Cellar/protobuf/3.14.0/bin/protoc 
MD5 (/opt/homebrew/Cellar/protobuf/3.14.0/bin/protoc) = 0da061ae62c179f668ff4a544377cec3

The protoc-jar-maven-plugin seems to request this for the embedded protoc.

[INFO] Protoc version: 3.11.0
protoc-jar: protoc version: 3.11.0, detected platform: osx-aarch_64 (mac os x/aarch64)
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: saved: /var/folders/gq/883qwjg91_x4pyq995m7vct40000gn/T/protocjar.webcache/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/github/os72/protoc/maven-metadata.xml
protoc-jar: saved: /var/folders/gq/883qwjg91_x4pyq995m7vct40000gn/T/protocjar.webcache/com/github/os72/protoc/maven-metadata.xml
protoc-jar: cached: /var/folders/gq/883qwjg91_x4pyq995m7vct40000gn/T/protocjar.webcache/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/3.11.0/protoc-3.11.0-osx-aarch_64.exe
protoc-jar: cached: /var/folders/gq/883qwjg91_x4pyq995m7vct40000gn/T/protocjar.webcache/com/github/os72/protoc/maven-metadata.xml
protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/github/os72/protoc/3.11.0/protoc-3.11.0-osx-aarch_64.exe
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.323 s
[INFO] Finished at: 2021-01-05T09:25:11-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project webapi-grpc-clients: Error extracting protoc for version 3.11.0: Unsupported platform: protoc-3.11.0-osx-aarch_64.exe -> [Help 1]

To workaround, I've done the following:

mvn install:install-file -DgroupId=com -DartifactId=github -Dversion=os72 -Dclassifier=3.14.0 -Dpackaging=protoc -Dfile=/opt/homebrew/Cellar/protobuf/3.14.0/bin/protoc

and then in the plugin configuration in pom.xml:

                        <protocVersion>3.14.0</protocVersion>
                        <protocArtifact>com:github:os72:protoc:3.14.0</protocArtifact>

os72 · Answer 1 · Wed Jan 06 2021 10:55:53 GMT+0800 (China Standard Time)

Ack. I see. We're saying special handling for osx-aarch_64:

look for native osx-aarch_64 binary
if not found, fallback to osx-x86_64 (cause M1 platform handles compatibility)

Correct?

If you could provide clean native binaries (static build, stripped, etc), good for wide consumption, I could host them on maven central:
https://repo.maven.apache.org/maven2/com/github/os72/protoc/

You could make them available via PR:
https://github.com/os72/protoc-jar/tree/master/bin

That would be a nice contribution. And if you're using the project professionally, I'd appreciate support (sponsorship) for it..

Christian Mardini · Answer 2 · Wed Jan 06 2021 21:39:21 GMT+0800 (China Standard Time)

I think if we populate the osx-aarch_64 binary, that would be sufficient.

I've built protoc with static linkage (./configure --disable-shared) and stripped the symbols. I'll make a PR shortly.

Christian Mardini · Answer 3 · Wed Jan 06 2021 22:22:04 GMT+0800 (China Standard Time)

Added a PR: #94

Jerry.Wang · Answer 4 · Sat Mar 13 2021 08:18:07 GMT+0800 (China Standard Time)

when can this be released?

Harry · Answer 5 · Fri May 14 2021 17:15:44 GMT+0800 (China Standard Time)

Suggested workaround for the time being:

Use brew to install the protobuf cli locally brew install protobuf https://formulae.brew.sh/formula/protobuf
Set the protocCommand property to protoc

I was able to add the property to a m1 maven build profile

<properties>
    <protocCommand>protoc</protocCommand>
</properties>

Nikita Dauhashei · Answer 6 · Mon Aug 02 2021 22:23:39 GMT+0800 (China Standard Time)

protoc-3.17.3-osx-aarch_64.exe
https://repo1.maven.org/maven2/com/google/protobuf/protoc/3.17.3/

Viktor BORBÉLY · Answer 7 · Wed Aug 04 2021 23:46:49 GMT+0800 (China Standard Time)

protoc-3.17.3-osx-aarch_64.exe
https://repo1.maven.org/maven2/com/google/protobuf/protoc/3.17.3/

I created this comment for the solution.

protocolbuffers/protobuf#8062 (comment)

Christian Mardini · Answer 8 · Thu Aug 05 2021 22:09:15 GMT+0800 (China Standard Time)

As per @hazmeister's comment, I added the following profile to my pom.xml that uses protoc. Maybe it can be shortened.

  <profile>
    <build>
      <plugins>
        <plugin>
          <groupId>com.github.os72</groupId>
          <artifactId>protoc-jar-maven-plugin</artifactId>
          <executions> 
            <execution>
              <configuration>
                <protocCommand>protoc</protocCommand> <!-- brew install protobuf -->
              </configuration>
            </execution> 
          </executions>
        </plugin>
      </plugins>
    </build>
    <activation>
      <os>
        <name>mac os x</name>
        <arch>aarch64</arch>
        <family>mac</family>
      </os>
    </activation>
  </profile>
</profiles>

os72 · Answer 9 · Wed Sep 01 2021 05:14:29 GMT+0800 (China Standard Time)

I'm afraid the pace in this project will remain slow until more support/sponsorship can be found (most users are enterprises)..

Rigoberto Vazquez · Answer 10 · Fri Jan 21 2022 01:07:29 GMT+0800 (China Standard Time)

in the meantime, I was able to run the process in a Apple M1 chip using the parameter "-Dos.arch=x86_64" with the maven goals

pratikraj1 · Answer 11 · Thu May 19 2022 00:35:32 GMT+0800 (China Standard Time)

Can someone provide final step to how to resolve this on M1 mac ?

Sylwester Lachiewicz · Answer 12 · Wed Jun 01 2022 05:01:12 GMT+0800 (China Standard Time)

simply do something like

<plugin>
       <groupId>com.github.os72</groupId>
       <artifactId>protoc-jar-maven-plugin</artifactId>
       <version>3.11.4</version>
      <configuration>
        <protocArtifact>com.google.protobuf:protoc:3.21.1</protocArtifact>

KAMRUJJAMAN JOY · Answer 13 · Mon Jun 27 2022 19:35:31 GMT+0800 (China Standard Time)

Where I can add those codes or suggestion codes?
<properties> <protocCommand>protoc</protocCommand> </properties>

os72 · Answer 14 · Tue Jun 28 2022 02:14:24 GMT+0800 (China Standard Time)

Where I can add those codes or suggestion codes? <properties> <protocCommand>protoc</protocCommand> </properties>

"properties" doesn't seem right, it goes under <configuration>. example: #93 (comment)

Lee · Answer 15 · Mon Jul 25 2022 22:00:55 GMT+0800 (China Standard Time)

in the meantime, I was able to run the process in a Apple M1 chip using the parameter "-Dos.arch=x86_64" with the maven goals

@Rigoberto-Vazquez thanks for this -- works for me too.

Another way I found that works is to switch to using a x86_64 JDK instead. Seems to run just fine on my M1 Mac.

Reason94 · Answer 16 · Wed Nov 02 2022 22:48:38 GMT+0800 (China Standard Time)

I found that works. Add the following configuration to maven settings.xml.

<profile>
      <id>apple-m1</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
</profile>

huanwei · Answer 17 · Tue Jan 03 2023 23:36:55 GMT+0800 (China Standard Time)

firstly, clone protobuf sourc code: git@github.com:protocolbuffers/protobuf.git
check out your specify version: (e.g. v3.11.4: git checkout v3.11.4 -b v3.11.4)
compile profobuf of specify version
a.(optional) ( in version v3.11.4, that must to generate configure.sh): autoconf
b.(optional) (then generate Makefile): ./configure
c. make -j $(nproc)
then you will find protoc under src directory
change the configuration <-- protoc-jar-maven-plugin -->
a. add protocCommand tag
b. example:

    <configuration>
          <protocCommand>/Users/huanwei/git_files/github/protobuf/src/protoc</protocCommand>
          <protocVersion>${protoc.version}</protocVersion>
          <inputDirectories>
                <include>src/main/resources</include>
          </inputDirectories>
     </configuration>

then re-run maven command, you will success

Florian Enner · Answer 18 · Fri Jan 06 2023 06:42:13 GMT+0800 (China Standard Time)

Don't you only need to specify a <protocVersion> of 3.17.3 or higher?

I'm not using protoc-jar by itself, but at least for the protoc-jar-maven-plugin that is sufficient.

Kirill Lassounski · Answer 19 · Tue Jul 25 2023 16:21:17 GMT+0800 (China Standard Time)

Make sure that you're running on a JDK that is x86_64.

I was previously running on Zulu JDK with arch64 architecture and having that issue.

After switching to OpenJDK the problem was gone.

Yuxiang Zhuang · Answer 20 · Wed Aug 30 2023 16:14:30 GMT+0800 (China Standard Time)

Make sure that you're running on a JDK that is x86_64.

I was previously running on Zulu JDK with arch64 architecture and having that issue.

After switching to OpenJDK the problem was gone.

it's good for me