Stringize and concatenate macros (# and ##) are not highlighted properly

[Pratyush Yadav]

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

When using # and ## operators, the tree-sitter parser incorrectly highlights them like #defines, which is not consistent with the part preceeding them. This was also partly a problem without tree-sitter, but only for using ## in function name.

With tree-sitter:

Without tree-sitter:

In the images, name is being highlighted incorrectly whenever the # and ## operators are used. taskqueue_define_##name should instead be highlighted like a function name. #name should be highlighted like a variable name (or a string? I'll let you guys decide).

Steps to Reproduce

1. Use the stringify and token concatenation preprocessor operators.

edit by @rsese to add code for copy/paste and more screenshots

/*
 * Define and initialize...
 */
#define TASKQUEUE_DEFINE(name, enqueue, context, init)                     \
                                                                           \
struct taskqueue *taskqueue_##name;                                        \
                                                                           \
static void                                                                \
taskqueue_define_##name(void *arg)                                         \
{                                                                          \
  taskqueue_##name =                                                       \
    taskqueue_create(#name, M_WAITOK, (enqueue), (context));               \
  init;                                                                    \
}                                                                          \

Tree-sitter:

Without:

Reproduces how often: 100%

Versions

atom --version:
Atom : 1.34.0
Electron: 2.0.16
Chrome : 61.0.3163.100
Node : 8.9.3

apm --version:
apm 2.1.3
npm 6.2.0
node 8.9.3 x64
atom 1.34.0
python 2.7.15rc1
git 2.17.1

OS:
Ubuntu 18.04

[Robert Sese]

Thanks for the report! Reproduce with 1.34.0 on macOS 10.12.6:

Tree-sitter:

Without: