aychtang / scip-clang

scip-clang: SCIP indexer for C and C++

scip-clang is a precise code indexer based on Clang 16, which supports cross-repository code navigation for C and C++ in Sourcegraph.

Here are some code navigation examples:

• Cross-repository navigation in Boost
• In Chromium.
• In llvm/llvm-project:
  • Find references for #include
  • Find references for macros
  • Find references for types

Table of Contents

• Supported Platforms
• Quick Start
• System Requirements
• Usage
  • Generating a compilation database
  • Building code
  • Initial scip-clang testing
  • Running scip-clang on a single repo
  • Setting up cross-repo code navigation
• Troubleshooting
• Reporting issues
• Documentation
• Contributing

Supported Platforms

Binary releases are available for x86_64 Linux (glibc 2.16 or newer) and x86_64 macOS (supported on arm64 macOS via Rosetta).

We're exploring Windows support.

Codebases using GCC and/or Clang for routine compilation are both supported. For codebases exclusively built using GCC, compatibility should be as good as Clang's compatibility (i.e. most features should work, with graceful degradation for features that don't).

scip-clang currently supports indexing using a JSON compilation database. CMake, Bazel and Meson support emitting this format for compatibility with clang-based tooling. We're interested in exploring more native Bazel support in the future.

Quick Start

The easiest way to use scip-clang, once you have a JSON compilation database, is to invoke scip-clang from the project root like so:

scip-clang --compdb-path=path/to/compile_commands.json

If you see any errors, see the Troubleshooting section.

If all goes well, indexing will generate a file index.scip which can be uploaded to a Sourcegraph instance using src-cli v4.5 or newer.

# See https://docs.sourcegraph.com/cli/references/code-intel/upload
# Make sure to authenticate earlier or provide an access token
src code-intel upload -file=index.scip

See the Usage section for step-by-step instructions.

System Requirements

1. About 2MB of temporary space for every TU in the compilation database.

   echo "$(perl -e "print $(jq 'length' build/compile_commands.json) / 512.0") GB"

2. On Linux, about 2MB of space in /dev/shm per core (df -h /dev/shm). This may particularly be an issue when using Docker on a high core count machine, as default size of /dev/shm in Docker is 64MB. See also: how to troubleshoot low disk space for IPC.
3. 2GB RAM per core is generally sufficient.

Usage

Generating a compilation database

• CMake: Add -DCMAKE_EXPORT_COMPILE_COMMANDS=ON to the cmake invocation. For typical projects, the overall invocation will look like:

  cmake -B build -DCMAKE_EXPORT_COMPILE_COMMANDS=ON

• Bazel: Use either hedronvision/bazel-compile-commands-extractor or grailbio/bazel-compilation-database. Caveat: The grailbio generator sometimes accidentally adds unexpanded Make variables in compilation commands, so you may need to remove them as a preprocessing step, before invoking scip-clang.

• Meson: Use the Ninja backend, which generates a compilation database.

Building code

Large projects typically use various forms of code generation. scip-clang re-runs type-checking, so it needs access to generated code. This means that scip-clang should preferably run after building compilation artifacts.

Initial scip-clang testing

For large codebases, we recommend first testing scip-clang on a subset of a compilation database with diagnostics turned on. For example:

# Using jq (https://github.com/stedolan/jq)
jq '.[0:5]' build/compile_commands.json > build/small_compdb.json
scip-clang --compdb-path=build/small_compdb.json --show-compiler-diagnostics

If there are errors about missing system or SDK headers, install the relevant system dependencies.

If there are errors about missing generated headers, make sure to build your code first.

If there are any other errors, such as standard library or platform headers not being found, please report an issue.

Running scip-clang on a single repo

scip-clang --compdb-path=build/compile_commands.json

The --show-compiler-diagnostics flag is deliberately omitted here, since scip-clang is still able to index code in the presence of compiler errors, and any errors in headers will get repeated for each translation unit in which the header is included.

Setting up cross-repo code navigation

See the cross-repository setup docs.

Troubleshooting

See the Troubleshooting docs.

Reporting issues

Create a new GitHub issue with any relevant logs attached.

Sourcegraph customers may ask their Customer Engineers for help with filing an issue confidentally, as the log may contain information about file names etc.

Documentation

Run scip-clang --help to see documentation for different flags.

A CHANGELOG is also available.

Contributing

• Development.md covers build instructions etc.
• Design.md covers the high-level architecture and design considerations.
• This GitHub comment covers why we decided to write a new C++ indexer.