Java: Include all XML files with codeql-cli
RickGY opened this issue Β· comments
I've trying to use the codeql command line interface to analyze a project locally. I would like the extractor to include all the XML files, but it seems that the "codeql database create command" will only include pom.xml.
- I noticed that by including a lgtm.yml file in the repo and write the following, LGTM can include all xml files:
extraction:
java:
index:
xml_mode: all
I wonder if it's possible to do something similar while using codeql-cli?
Hi @RickGY, yes, instead of doing
codeql database create --source-root <src> --language java --command <java command> <db>
You can split up the database creation process to manually include the XML files:
codeql database init --source-root=<src> --language java <db>
codeql database trace-command --working-dir=<src> <db> <java command>
codeql database index-files --language xml --include-extension .xml --working-dir=<src> <db>
codeql database finalize <db>
If you're just using the current directory as your source, you can skip the --working-dir <src>
parts.
Hi @RasmusWL , I actually had a similar question. I am attempting to include xml files for a python CodeQL database I am building through the CLI, but am running into some issues. What should I include as the component in step 2 (but if I am working in python)? I am a bit unsure about what that command does, and if I run all of the commands except that step, my database includes the xml files I would like but nothing else (so none of the source code at all). Would you mind providing some guidance?
Hi @dilanbhalla. I agree that this part is a bit tricky, and requires internal knowledge of how to do things. I was a bit confused about how to do it as well for #3911 (we're having internal talks about making it easier).
You can use this trick to trigger the same behavior as with codeql database create
π
codeql database trace-command --working-dir=<src> <db> $(codeql resolve extractor --language=python)/tools/autobuild.sh
No problem π
Gonna close this now then
Hi Rasmus, apologies in advance for the long follow up, but I have a question unrelated to this query and am unsure how else to contact you (since our discussion thread from earlier is now closed). This may seem a little silly, but the trace-command you showed my for python won't work due to the simple error that the '$' is not recognized. My end goal is to simply use the CLI to build a python database that includes some custom xml files I wrote, so I believe your method would work for python (init, index the xml files, trace-command, finalize). Does your PR need to be merged before this trace-command will work? Or is it something simple that I may be doing wrong with regards to the expression starting with '$'? And lastly, if including the xml is not at all possible, would you happen to know any other method to include custom data (maybe through something like a csv) and reference it within a python ql file? Thank you so much!
- #3878 (comment)
Hi @dilanbhalla, you can comment on closed issues, no problem. I think the most appropriate place to ask questions about the CLI in the future is on https://github.com/github/codeql-cli-binaries/issues. You can always use https://github.com/github/securitylab/discussions as well for CodeQL questions π
I guess the part you're having trouble with is $(codeql resolve extractor --language=python)/tools/autobuild.sh
. That is just using command substitution in bash. Basically it means, run codeql resolve extractor --language=python
and insert the output of that command in place of $(codeql resolve extractor --language=python)
. So if you're running a different shell where command substitution doesn't work, you can do the following:
$ codeql resolve extractor --language=python
/foo/bar/codeql/python
$ codeql database trace-command --working-dir=<src> <db> /foo/bar/codeql/python/tools/autobuild.sh
if you're on windows, you should probably use autobuild.cmd
instead of autobuild.sh
π
Hi @RasmusWL! This worked like a charm. Sorry I did not realize I can continue to comment on a closed issue and thank you so much for the advice. Everything is running perfectly for me now, with the .xml files I was attempting to include earlier. Thanks again!