Giters

gvansickle / ucg

UniversalCodeGrep (ucg) is an extremely fast grep-like tool specialized for searching large bodies of source code.

Home Page:https://gvansickle.github.io/ucg/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Homebrew formula

equal-l2 opened this issue · comments

commented

Currently I'm working on new ucg formula for submitting to homebrew-core.

Build fails with following error.

Undefined symbols for architecture x86_64:
  "FileScanner::CountLinesSinceLastMatch_sse2(char const*, char const*)", referenced from:
      _resolve_CountLinesSinceLastMatch in libsrc.a(libsrc_la-FileScanner.o)
ld: symbol(s) not found for architecture x86_64

Configure summary:

  Configuration Summary for universalcodegrep 0.3.1
  =================================================

  Compilers
  ---------
  C compiler:                 clang -std=gnu11
  AM_CFLAGS:                  -ggdb3 -O3 -fno-omit-frame-pointer -ftree-vectorize -pthread
  CFLAGS:

  C preprocessor:             clang -E
  AM_CPPFLAGS:                -Wall -Wextra
  CPPFLAGS:

  C++ compiler:               clang++ -std=gnu++1z
  AM_CXXFLAGS:                -ggdb3 -O3 -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -Wformat -Wformat-security -Werror=format-security
  CXXFLAGS:

  C++ preprocessor:           clang++ -E -std=gnu++1z

  LTCC:                       clang -std=gnu11
  LTCFLAGS:
  AM_LDFLAGS:
  LD:                         /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
  LDFLAGS:
  LIBS:                       -largp

  clang++ -std=gnu++1z Search Paths
  -----------------
  programs
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
  libraries
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0

  clang++ -E -std=gnu++1z Include Paths
  ---------------------
  #include "..." search starts here:
  #include <...> search starts here:
  /usr/local/include
  /usr/include/libxml2
  /System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/include
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
  /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include
  /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks (framework directory)
  End of search list.

  PCRE Info
  ---------
  HAVE_LIBPCRE                no
  HAVE_LIBPCRE2               yes

  libtool info
  ------------
  sys_lib_search_path_spec:   /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0  /usr/local/lib
  sys_lib_dlsearch_path_spec: /usr/local/lib /lib /usr/lib
  runpath_var:

Full formula:

class Ucg < Formula
  desc "grep-like tool for searching large bodies of source code"
  homepage "https://github.com/gvansickle/ucg"
  url "https://github.com/gvansickle/ucg/releases/download/0.3.1/universalcodegrep-0.3.1.tar.gz"
  sha256 "62f9ef88ea5c0777696c4322bed1fb9a3fb62eb85bd053af50f75d42ec259086"
  head "https://github.com/gvansickle/ucg.git"

  depends_on "pkg-config" => :build
  depends_on "autoconf" => :build
  depends_on "libtool" => :build
  depends_on "automake" => :build
  depends_on "argp-standalone" => :build
  depends_on "pcre2"

  def install
    system "autoreconf", "-i" if build.head?

    system "./configure", "--disable-debug",
                          "--disable-dependency-tracking",
                          "--disable-silent-rules",
                          "--prefix=#{prefix}"
    system "make", "install"
  end

  test do
    # TODO: Write test
  end
end

Feel free to tell me if any further information is needed.

Additionally, I'd appreciate if you could provide some test for formula's test block.

commented

Hey @equal-l2 ,

Currently I'm working on new ucg formula for submitting to homebrew-core.

This is the best Christmas ever! ;-) Thanks a ton for doing this.

Undefined symbols for architecture x86_64:
  "FileScanner::CountLinesSinceLastMatch_sse2(char const*, char const*)", referenced from:
      _resolve_CountLinesSinceLastMatch in libsrc.a(libsrc_la-FileScanner.o)
ld: symbol(s) not found for architecture x86_64

This sounds like an issue with the multiversioning mechanism. What does the "make V=1" output look like? Towards the end of the build, you should see three sections which look like this:

MULTIVERSIONING BUILD START: Building libsrc_sse2.fmv.la
make "CPPFLAGS=$(echo  | /usr/bin/sed -r 's/-m((arch=)|(sse)|(avx))[^[:space:]]*//g')" "CFLAGS=$(echo  | /usr/bin/sed -r 's/-m((arch=)|(sse)|(avx))[^[:space:]]*//g')" "CXXFLAGS=$(echo  | /usr/bin/sed -r 's/-m((arch=)|(sse)|(avx))[^[:space:]]*//g')" libsrc_sse2.la
make[4]: Entering directory '/cygdrive/c/cygwin/home/Gary/src/ucg/UltimateCodeSearch/build/src'
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -std=gnu++1z -DHAVE_CONFIG_H -I. -I../../src -I..  -Wall -Wextra  -ggdb3 -O3 -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -Wformat -Wformat-security -Werror=format-security -msse2  -c -o libsrc_sse2_la-FileScanner_sse4_2.lo `test -f 'FileScanner_sse4_2.cpp' || echo '../../src/'`FileScanner_sse4_2.cpp
libtool: compile:  g++ -std=gnu++1z -DHAVE_CONFIG_H -I. -I../../src -I.. -Wall -Wextra -ggdb3 -O3 -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -Wformat -Wformat-security -Werror=format-security -msse2 -c ../../src/FileScanner_sse4_2.cpp -o libsrc_sse2_la-FileScanner_sse4_2.o
../../src/FileScanner_sse4_2.cpp:32:68: note: #pragma message: Have SSE2 at line 32
/bin/sh ../libtool  --tag=CXX   --mode=link g++ -std=gnu++1z -ggdb3 -O3 -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -Wformat -Wformat-security -Werror=format-security -msse2    -o libsrc_sse2.la  libsrc_sse2_la-FileScanner_sse4_2.lo  -largp
libtool: link: ar cru .libs/libsrc_sse2.a  libsrc_sse2_la-FileScanner_sse4_2.o
libtool: link: ranlib .libs/libsrc_sse2.a
libtool: link: ( cd ".libs" && rm -f "libsrc_sse2.la" && ln -s "../libsrc_sse2.la" "libsrc_sse2.la" )
make[4]: Leaving directory '/cygdrive/c/cygwin/home/Gary/src/ucg/UltimateCodeSearch/build/src'
/bin/sh ../libtool  --tag=CXX   --mode=link g++ -std=gnu++1z -ggdb3 -O3 -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -Wformat -Wformat-security -Werror=format-security    -o libsrc_sse2.fmv.la libsrc_sse2.la
libtool: link: (cd .libs/libsrc_sse2.fmv.lax/libsrc_sse2.a && ar x "/home/Gary/src/ucg/UltimateCodeSearch/build/src/./.libs/libsrc_sse2.a")
libtool: link: ar cru .libs/libsrc_sse2.fmv.a   .libs/libsrc_sse2.fmv.lax/libsrc_sse2.a/libsrc_sse2_la-FileScanner_sse4_2.o
libtool: link: ranlib .libs/libsrc_sse2.fmv.a
libtool: link: rm -fr .libs/libsrc_sse2.fmv.lax
libtool: link: ( cd ".libs" && rm -f "libsrc_sse2.fmv.la" && ln -s "../libsrc_sse2.fmv.la" "libsrc_sse2.fmv.la" )
MULTIVERSIONING BUILD END: Built libsrc_sse2.fmv.la

In particular, in the first MULTIVERSIONING BUILD section, you should see a line which looks very much like this one from the above:

libtool: compile:  g++ -std=gnu++1z -DHAVE_CONFIG_H -I. -I../../src -I.. -Wall -Wextra -ggdb3 -O3 -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -Wformat -Wformat-security -Werror=format-security -msse2 -c ../../src/FileScanner_sse4_2.cpp -o libsrc_sse2_la-FileScanner_sse4_2.o

The -msse2 flag is critical here; if it's not there, or if there are additional -m's such as -march=native or -mavx, things could break such that a function named CountLinesSinceLastMatch_sse2 never gets built.

Additionally, I'd appreciate if you could provide some test for formula's test block.

make check runs a number of regression tests, and also a few benchmarks. Is that sufficient? (I apologize for the probably dumb question, I'm quite new to the OS X ecosystem).

I'll be in and out today, what with it being xmas and all, but I'll try to give this issue my fullest attention. Thanks again @equal-l2 , I would love to see ucg in homebrew-core!

GRVS

commented

@gvansickle

The -msse2 flag is critical here; if it's not there, or if there are additional -m's such as -march=native or -mavx, things could break such that a function named CountLinesSinceLastMatch_sse2 never gets built.

I've found that Homebrew has a mechanism called superenv to override compiler flags with their favorite one, which includes -march.
Installation has succeeded once I have disabled the mechanism

make check runs a number of regression tests, and also a few benchmarks. Is that sufficient?

Unfortunately, the test should be done only by installed files.
(Homebrew has binary-installation mechanism, which not includes original build scripts)

However, I have noticed Homebrew only needs very easy test.

Look, this is the_platinum_searcher's test.

  test do
    path = testpath/"hello_world.txt"
    path.write "Hello World!"

    lines = `#{bin}/pt 'Hello World!' #{path}`.strip.split(":")
    assert_equal "Hello World!", lines[2]
  end

This one is for ripgrep.

  test do
    (testpath/"Hello.txt").write("Hello World!")
    system "#{bin}/rg", "Hello World!", testpath
  end

Even I can write this sort of test.
There is no need to bother you.

Now I think the formula is ready to submit.
Thank you for your support.

commented

Hey @equal-l2 ,

I've found that Homebrew has a mechanism called superenv to override compiler flags with their favorite one, which includes -march.
Installation has succeeded once I have disabled the mechanism

Fabulous! I see superenv mentioned in the Formula Cookbook, but it's pretty light on details. Could you copy-paste the compile command line it generates before the disabling? I added some filtering of the options to the multiversioning infrastructure a while ago (so it should in fact build correctly with a "make CFLAGS=-march=native" for example), but I must have missed a case. I'd like to fix this.

Thanks for all the work on this. How would you like to proceed going forward? With luck I should have the next release out soon (this week?), would you like me to ping you on new releases?

commented

@gvansickle

Could you copy-paste the compile command line it generates before the disabling?

This is a part of the log from superenv.

clang++ called with: -std=gnu++1z -DHAVE_CONFIG_H -I. -I.. -Wall -Wextra -ggdb3 -O3 -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -Wformat -Wformat-security -Werror=format-security -msse2 -c FileScanner_sse4_2.cpp -o libsrc_sse2_la-FileScanner_sse4_2.o
superenv removed:  -Wall -Wextra -ggdb3 -O3 -Wformat -Wformat-security -Werror=format-security
superenv added:    -pipe -w -Os -march=native -isystem/usr/local/include -isystem/usr/include/libxml2 -isystem/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
superenv executed: clang++ -pipe -w -Os -march=native -std=gnu++1z -DHAVE_CONFIG_H -I. -I.. -fno-omit-frame-pointer -fdiagnostics-color=auto -ftree-vectorize -pthread -msse2 -c FileScanner_sse4_2.cpp -o libsrc_sse2_la-FileScanner_sse4_2.o -isystem/usr/local/include -isystem/usr/include/libxml2 -isystem/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers

Superenv behaves as a wrapper for the compiler, so you can't see the change by build log (output from make) itself.

would you like me to ping you on new releases?

I'd be grateful if you could so.

commented

Hey @equal-l2 ,

superenv executed: clang++ -pipe -w -Os -march=native -std=gnu++1z [...]

Ah, I see what they did there. They did the equivalent of:

make CXX='clang++ -pipe -w -Os -march=native'

i.e., the makefiles consider the C++ compiler to be that whole string; I'm only filtering CFLAGS/CXXFLAGS. That's probably going to require a non-trivial amount of additional non-fun battling with Automake . I think your solution in the Homebrew formula is the right way to go for now. I've pulled a branch to address this (and any other issues which your work here might find), but don't wait on that; automake work tends to take forever, and like I say I think you have the right solution at this time.

would you like me to ping you on new releases?

I'd be grateful if you do so.

On the contrary, I'm delighted you're helping the project out!

commented

Yep, I just reproduced the original issue here on Fedora with this:

make CXX='g++ -march=native'

Good find!

commented

That's probably going to require a non-trivial amount of additional non-fun battling with Automake .

Sometimes I wonder if it takes more effort using the autocrap tools than just writing a configuration shell script manually...

commented

Sometimes I wonder if it takes more effort using the autocrap tools than just writing a configuration shell script manually...

Well, this time it didn't turn out so bad, just two more lines in the Makefile.am.

But we feel each others' pain. About half of the languages I know are either primarily or solely connected with the GNU Build System (aka "autotools", among other names ;-)), and that's only a marginal exaggeration. I mean seriously, m4?

Putting on my Karnak turban, what I think we just might see soon is a well-documented (so Cmake's pretty much out) tool combining the functionality of make, the GNU Build System, and every one of the hundreds of package managers out there, written in Python and exposing some subset of Python to the input "ConfigMakeBuildFiles". It's absolute loony-tunes that in the 21st century, I have to know every minute vagary of dozens of different tools (makes, seds, autoconfs, automakes, libtools, awks, scripts, shells, etc etc, (and that's before I even get to the compilers, linkers, etc etc etc), and no two versions of any of them work the same.

I hate to say it, but this really isn't the unsolvable problem people want to make it into. Here's the problem statement in full:

  1. I write some source code for a program in my selection of favorite languages.
  2. I write a very small amount of "meta" info about that source code, like which files go in it, informational text about what it does, what name and version I want to call it... and that's pretty much it.
  3. (The missing part) I hit a proverbial button, the computer grinds for a bit, and out pops a binary ready for deployment to the world.
    3a. (Also missing) I tar up the source, send it to someone else, and they can build it themselves - on whatever platform they happen to be running, even one I've never seen before.
  4. Profit.

The GNU Build System has all along strived to be # 3a. They've tried to do it by going least-common-denominator, which turned out to be a loser's game, because in the Unix diaspora, there is no bottom to the least common denominator. The time utility/builtin is not usably portable for God's sake! And GBS has come the closest to meeting all the challenges of the problem statement above.

Ok, rant off. I say Python above for a couple reasons:
a. It's ubiquitous. Any GBS replacement needs ubiquity or it's going nowhere.
b. It has an impressive standard library. There's nothing the GBS is doing that Python couldn't do, plus...
c. It's one language. Not three or four for applying regexes to strings, not another one for specifying the guzintas and comesoutas, not yet another one that nobody really understands for templating.
d. Scripting language.

All that said, I got nothing on my radar. Don't make me write it myself @kirbyfan64 !!!! ;-)

commented

I think part of the trouble is:

"meta" info

Almost no one can agree on how to describe the build process, let alone how to actually build (IDE generator? hashes vs timestamps?), and then the ones that cover all use cases are too complicated too figure out...

commented

Hi again @equal-l2 ,

Sorry it took so long, but I finally cut a new tarball for ucg version 0.3.2: https://github.com/gvansickle/ucg/releases/tag/0.3.2. This should not require the env :std to build under Homebrew.

I'll post this in the Homebrew-core issue as well. Let me know how that tarball works; travis-ci is showing no problems, so fingers crossed.

Thanks again,
GRVS

commented

Hi @equal-l2 ,

I see your pull request has been rejected again. In addition to the ~stylistic comments about the *.rb, there appears to be a real ucg issue as well which I can reproduce. If I try your test:

$ echo "Hello World!" > newdir/test.txt
$ ./ucg 'Hello World' newdir

I get no matches. So something's broken, which is pretty odd since it passes make check and make distcheck with no problems.
I'm looking into it now, I'm certain this used to work. Sorry for the inconvenience, I'll get back to you as soon as I know something.

GRVS

commented

Hi again @equal-l2 ,
I've found it and it looks like I fixed it, and I added a regression test for it. I'm going to do some more testing, and I have one legitimate (but benign at the moment) Coverity issue to fix, then I'll spin 0.3.3.
Again my apologies for the inconvenience, and thanks again for your efforts.
GRVS

commented

Ok @equal-l2 , third time's the charm:
https://github.com/gvansickle/ucg/releases/tag/0.3.3
Thanks again,
GRVS

commented

Hey @equal-l2 ,
So it's been approved and merged! I've tried homebrew install ucg and it installs and works fine. This is remotely via Travis-CI since I don't have an OS X machine myself, but that's probably better anyway, since it installs against several OS X versions and configurations.
Again, thanks a million, this is just great. I'll keep this Issue open until I get a chance to update the Web site and README.md, hopefully in time to ring in the new year ;-).
GRVS

commented

Closing since the formula has been accepted.

commented

Thanks @equal-l2 , I should have done this myself. I did get the README.md updated, the website not so much.

Thanks again,
GRVS