Documentation
IngwiePhoenix opened this issue · comments
Hello!
I am working towards building a way to automatically generate bindings between C and V. One way to do so is to parse C directly - which I am trying to accomplish with mpc - but this will require to solve macros and other preprocessor definitions. For that, I would like to use TPP.
Building it on Windows was completely flawless (Windows 10, amd64, VS2019). But trying to piece together the API from frontend.c is a little difficult.
Which functions are "public" and which ones are not? And what do I basically need to process a C file, add include directives and possibly other definitions and then obtain the output?
Thank you in advance!
Kind regards,
Ingwie
Hi,
I only just realized that I haven't commited to TPP in quite a while. This wasn't actually because I didn't improve it over that time, but because it kind-of became embedded inside of another project of mine (I've committed all of the changes here now, too). I'm sorry if you've went to all the troubles to go through my age-old code, only for me to come out and say "Yeah... There's a newer version you should be using" (sorry: this was entirely my fault).
To answer your question on what's meant to be "public" and what not, that depends on how much you want to integrate tpp into your project: If you only intend to use it as a stand-along preprocessor, then you'd only invoke the commandline utility built by "frontend.c" (or some variation there-of). Though I do a assume that this isn't what you want to do.
If your intend is to use the actual C-interface, then the idea is to copy+paste "tpp.c" into your /src directory, and put "tpp.h", "tpp-defs.inl" and "tpp-gcc-defs.inl" into your /include folder (That's (kind-of) what I've been doing). TPP is in no ways meant to be treated/used as a library (if you've ever used dlmalloc, TPP's kind-of like that: A single-source-file drop-in, fully featured implementation to-be used and integrated into some other project). You would then create 2 files "/include/mywrapper-for-tpp.h" and "/src/mywrapper-for-tpp.c" that include the respective tpp-files after overriding the various implementation hooks/overrides/config-options which are checked for by TPP. (Most of these are documented at the top of "tpp.h")
Once you've integrated TPP in this manner into your project, you'd initialize & use it like this:
...
#if !TPP_CONFIG_ONELEXER
TPPLexer_Current = (struct TPPLexer *)malloc(sizeof(struct TPPLexer));
if (!TPPLexer_Current)
ERROR_HANDLING;
#endif
if (!TPPLexer_Init(TPPLexer_Current))
ERROR_HANDLING;
/* Configure `TPPLexer_Current' to your liking */
TPPLexer_Current->l_flags = ...;
TPPLexer_Current->l_extokens = ...;
/* -DMY_MACRO=42 */
char const *name = "MY_MACRO";
char const *val = "42";
if (!TPPLexer_Define(name, strlen(name), val, strlen(val), TPPLEXER_DEFINE_FLAG_NONE))
ERROR_HANDLING;
/* -I/usr/include */
char *incpath = strdup("/usr/include");
if (!TPPLexer_AddIncludePath(incpath, strlen(incpath)))
ERROR_HANDLING;
free(incpath);
/* Push an initial file onto the #include-stack */
struct TPPFile *file = TPPLexer_OpenFile(
TPPLEXER_OPENFILE_MODE_NORMAL | TPPLEXER_OPENFILE_FLAG_CONSTNAME,
"input.c");
if (!file)
ERROR_HANDLING;
TPPLexer_PushFile(file);
/* Process input one token at a time.
* Hint: emission of certain tokens depends on `TPPLEXER_FLAG_WANT*' and `TPPLEXER_TOKEN_*' */
while (TPPLexer_Yield() > 0) {
int id = TPPLexer_Current->l_token.t_id;
char *tokstr = TPPLexer_Current->l_token.t_begin;
size_t toklen = (size_t)(TPPLexer_Current->l_token.t_end - tokstr);
printf("token: %d: '%.*s'\n", id, (int)toklen, tokstr);
}
/* Check if something went wrong (stuff like `#error' directives, or syntax errors) */
if ((TPPLexer_Current->l_flags & TPPLEXER_FLAG_ERROR) ||
(TPPLexer_Current->l_errorcount != 0))
ERROR_HANDLING;
/* Cleanup the lexer (must be called after successful a `TPPLexer_Init()') */
TPPLexer_Quit(TPPLexer_Current);
#if !TPP_CONFIG_ONELEXER
free(TPPLexer_Current);
#endif
...As you can see, output generated by TPP is in token-based, with the idea that one would build a compiler directly on-top of the tokens generated by tpp, using TPPLexer_Yield() in order to advance the lexer to the next token (performing tokenization, dealing with preprocessor #if-blocks, macro expansion, and the many extensions implemented by TPP). So if you just want to get back the "preprocessed output", you'll have to do what "frontend.c" does by re-constructing individual tokens into something more coherent.
As far as language-specific integration (such as language-specific keywords, configurable language features, or warning messages) goes, this sort of thing should be done in a file "/include/mywrapper-for-tpp.h" that is hooked in your "/include/my-custom-tpp-defs.h":
...
#define TPP_USERDEFS <my-custom-tpp-defs.h>
#include <tpp.h>
...in "my-custom-tpp-defs.h":
/* Custom keywords (but be careful not to re-define ones already defined by TPP)
* When parsed, you can check for these keywords like:
* >> switch (TPPLexer_Current->l_token.t_id) {
* >> case KWD_async:
* >> ...;
* >> break;
* >> case KWD_function:
* >> ...;
* >> break;
* >> }
* */
DEF_K(async)
DEF_K(function)
DEF_K(def)
DEF_K(awesome_keyword)
/* A pre-defined macro (as in `#ifdef __MY_PREDEFINED_MACRO__')
* This should be used for stuff like `__GNUC__' or `__cplusplus', etc... */
PREDEFINED_MACRO_IF(__MY_PREDEFINED_MACRO__, "42", should_be_defined ? 1 : 0)
/* Custom warnings groups. (see "tpp-defs.inl" for the default groups)
* Warnings can be enabled/disabled on a per-group basis by parsed text:
* >> #pragma warning("-Wmygroup") // Enable
* >> #pragma warning("-Wno-mygroup") // Disable
* >> #pragma GCC diagnostic error "-Wmygroup"
* >> #pragma GCC diagnostic warning "-Wmygroup"
* >> #pragma GCC diagnostic ignored "-Wmygroup"
*/
WGROUP(WG_MYGROUP, "mygroup", WSTATE_FATAL) // Warnings controlled by "-Wmygroup" / "-Wno-mygroup" / ...
/* Custom warnings.
* Your compiler would trigger this like (note: it's a varargs function):
* >> if (!TPPLexer_Warn(W_MYWARNING, "first variable argument"))
* >> HANDLE_AS_CRITICAL_ERROR;
* >> TRY_TO_CONTINUE_COMPILING;
*/
DEF_WARNING(W_MYWARNING, (WG_MYGROUP, WG_SYNTAX), WSTATE_ERROR, {
char *mesg = ARG(char *);
WARNF("My warning handler: %s",mesg);
})
/* Custom extension. Input code can enable/disable this by:
* >> #pragma extension("-fawesome") // Turn on
* >> #pragma extension("-fno-awesome") // Turn off
* Your compiler can check if it's enabled with `TPPLexer_HasExtension(EXT_AWESOME)' */
EXTENSION(EXT_AWESOME, "awesome", enabled_by_default ? 1 : 0)I hope that answers all of your questions.
Greetings,
GrieferAtWork
Hi, can you put all the content of your reply to the homepage's README.md
I think people need to know how to use your library. Without the document, it is really hard to do that.
Thanks.
I found an error in your example code:
#include "<tpp.h>"
Do you think the " " and < > is duplicated?
OK, I can build the sample project now, but I see it get crashed when I hit here:
TPPLexer_Quit(TPPLexer_Current);
Here is the full source code of the main.cpp
#include <iostream>
using namespace std;
#include "mywrapper-for-tpp.h"
#include <string.h> // for the strlen() function call
#define ERROR_HANDLING return(0)
int main()
{
#if !TPP_CONFIG_ONELEXER
TPPLexer_Current = (struct TPPLexer*)malloc(sizeof(struct TPPLexer));
if(!TPPLexer_Current)
ERROR_HANDLING;
#endif
if(!TPPLexer_Init(TPPLexer_Current))
ERROR_HANDLING;
/* Configure `TPPLexer_Current' to your liking */
// TPPLexer_Current->l_flags = ...;
// TPPLexer_Current->l_extokens = ...;
/* -DMY_MACRO=42 */
char const* name = "MY_MACRO";
char const* val = "42";
if(!TPPLexer_Define(name, strlen(name), val, strlen(val), TPPLEXER_DEFINE_FLAG_NONE))
ERROR_HANDLING;
/* -I/usr/include */
char* incpath = strdup("F:/code/test-prep/tpp/test");
if(!TPPLexer_AddIncludePath(incpath, strlen(incpath)))
ERROR_HANDLING;
free(incpath);
char* inputFilename = "input.c";
/* Push an initial file onto the #include-stack */
struct TPPFile* file = TPPLexer_OpenFile(
TPPLEXER_OPENFILE_MODE_NORMAL | TPPLEXER_OPENFILE_FLAG_CONSTNAME,
inputFilename, strlen(inputFilename), NULL);
if(!file)
ERROR_HANDLING;
TPPLexer_PushFileInherited(file);
/* Process input one token at a time.
* Hint: emission of certain tokens depends on `TPPLEXER_FLAG_WANT*' and `TPPLEXER_TOKEN_*' */
while(TPPLexer_Yield() > 0)
{
int id = TPPLexer_Current->l_token.t_id;
char* tokstr = TPPLexer_Current->l_token.t_begin;
size_t toklen = (size_t)(TPPLexer_Current->l_token.t_end - tokstr);
printf("token: %d: '%.*s'\n", id, (int)toklen, tokstr);
}
/* Check if something went wrong (stuff like `#error' directives, or syntax errors) */
if((TPPLexer_Current->l_flags & TPPLEXER_FLAG_ERROR) ||
(TPPLexer_Current->l_errorcount != 0))
ERROR_HANDLING;
/* Cleanup the lexer (must be called after successful a `TPPLexer_Init()') */
TPPLexer_Quit(TPPLexer_Current);
#if !TPP_CONFIG_ONELEXER
free(TPPLexer_Current);
#endif
cout << "Hello world!" << endl;
return 0;
}
The content of the "input.c" is very simple, i just wrote:
#define HI 5
printf(HI);
I see that the tokens were printed in the console window:
token: 801: 'printf'
token: 40: '('
token: 48: '5'
token: 41: ')'
token: 59: ';'
The call stack are below:
#0 0x00007fff8678f2d3 in ntdll!RtlIsZeroMemory () from C:\WINDOWS\SYSTEM32\ntdll.dll
#1 0x00007fff86798092 in ntdll!RtlpNtSetValueKey () from C:\WINDOWS\SYSTEM32\ntdll.dll
#2 0x00007fff8679837a in ntdll!RtlpNtSetValueKey () from C:\WINDOWS\SYSTEM32\ntdll.dll
#3 0x00007fff8679e001 in ntdll!RtlpNtSetValueKey () from C:\WINDOWS\SYSTEM32\ntdll.dll
#4 0x00007fff866b6625 in ntdll!RtlGetCurrentServiceSessionId () from C:\WINDOWS\SYSTEM32\ntdll.dll
#5 0x00007fff866b5b74 in ntdll!RtlGetCurrentServiceSessionId () from C:\WINDOWS\SYSTEM32\ntdll.dll
#6 0x00007fff866b47b1 in ntdll!RtlFreeHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
#7 0x00007fff8678943a in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\WINDOWS\SYSTEM32\ntdll.dll
#8 0x00007fff866b5cc1 in ntdll!RtlGetCurrentServiceSessionId () from C:\WINDOWS\SYSTEM32\ntdll.dll
#9 0x00007fff866b5b74 in ntdll!RtlGetCurrentServiceSessionId () from C:\WINDOWS\SYSTEM32\ntdll.dll
#10 0x00007fff866b47b1 in ntdll!RtlFreeHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
#11 0x00007fff84fa9c9c in msvcrt!free () from C:\WINDOWS\System32\msvcrt.dll
#12 0x00007ff7f7e82ef3 in TPPFile_Destroy (self=0x1b4600) at F:\code\test-prep\tpp\src\tpp.c:2087
#13 0x00007ff7f7e8cc4d in cleanup_keyword (self=0x1b4590) at F:\code\test-prep\tpp\src\tpp.c:5514
#14 0x00007ff7f7e8cf45 in destroy_keyword_map (self=0x7ff7f7ec6098 <TPPLexer_Global+88>) at F:\code\test-prep\tpp\src\tpp.c:5563
#15 0x00007ff7f7e8e765 in TPPLexer_Quit (self=0x7ff7f7ec6040 <TPPLexer_Global>) at F:\code\test-prep\tpp\src\tpp.c:6268
#16 0x00007ff7f7e8168c in main () at F:\code\test-prep\main.cpp:64
It crashes in the function:
Line 2087 in 1ca163d
Any ideas?
Thanks.
Hi, can you put all the content of your reply to the homepage's README.md
I think people need to know how to use your library. Without the document, it is really hard to do that.
Thanks.
Good suggestion. I'll make a note for adding this tiny sample-project as an actual part of the repo (and then link it in the README)
Edit: sample added in 468e2fe
I found an error in your example code:
#include "<tpp.h>"
Jup. that's supposed to be #include <tpp.h> (or #include "tpp.h" depending on how you're integrating it)
It crashes in the function:
Any ideas?
continued in #4