tokenizer-parser

There are 0 repository under tokenizer-parser topic.

risesoft-y9 / Data-Labeling
数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。
chinese data-annotation-tools data-annotations docker elasticsearch java nacos springboot2 tokenizer tokenizer-parser vue3
Language:Java 690
parser-toolkit
ikskuh / parser-toolkit
A toolkit that makes it easier to write recursive-descent parsers in Zig.
zig zig-package ziglang compiler-frontend compiler parser recursive-descent-parser tokenizer tokenizer-parser
Language:Zig 79
ehwan / C-language-Parser-In-Rust
C language lexer & parser & virtual interpreter from scratch
abstract-syntax-tree c code-generation lexer parser rust rust-lang tokenizer tokenizer-parser virtual-machine compiler interpreter llvm llvm-ir
Language:Rust 27
bytexenon / LuaXen
Lua Compiler, (De)Obfuscator, Minifier, Beautifier, And more
assembler assembly assembly-language beautifier decompiler interpreter lexer lua lua51 minifier obfuscator parser programming-language pseudo-assembler pseudo-assembly tokenizer tokenizer-parser virtual-machine virtualmachine vm
Language:Lua 23
recp / xml
🔋 In-place lightweight XML parser
xml xml-parser fast-xml tokenizer header-only tokenizer-parser optimized-xml lightweight-xml in-place-parser c-xml xml-loader parser
Language:C 15
antlr4-experiments
melchisedech333 / antlr4-experiments
:wrench: My studies on context-free grammar, using ANTLR4 (C++) to generate the parser files. Some basics are developed, such as token processing, recursion, variable definition, array processing, Abstract Syntax Tree (AST) manipulation, UNICODE support, and error handling.
antlr4 grammar-parser grammars semantics syntax tokenizer antlr4-grammar grammar-checker grammar-parser-generator grammar-rules semantic-analysis syntax-analysis syntax-tree tokenization antlr-language-development language-development language-processing syntax-analyzer tokenizer-parser grammars-utility
Language:Java 11
zbroderson / HumanTimeParser
Simple to use parser capable of parsing a usable time object from human input
parsing human readable time dotnet tokenizer-parser tokenizer
Language:C# 9
aboss123 / Ice-Programming-Language-Submission
This is a short and modern JIT compiler that transform source text, into LLVM IR bytecode that executes machine code at runtime. This project was developed at the hths.hacks() hackathon against more 250+ participants internationally and was placed as a winner. Among the winners, my project was the only one developed solo.
programming-language compiler llvm llvm-ir tokenizer-parser
Language:C 8
lex-yacc-experiments
melchisedech333 / lex-yacc-experiments
:wrench: My studies involving context-free grammar analysis. The analyzers were built using familiar tools such as YACC, Lex and Bison. Topics covered include token filtering, simple variable manipulation, and arrays.
lexical-analysis syntactic-analysis semantic-analysis language-development compiler-development parsing bison bison-grammar lex lexical-analyzer parser-generator tokenizer-parser yacc yacc-lex bison-parser bison-yacc tokenization yacc-example lex-example lex-yacc-parser
Language:Yacc 6
spydaz / AI_ZX81
A Basic Experiment in Parser and Compilers and Stack VM . A basic stack based CPU with Assembly language and basic commands. A basic programming Languge Parsed to Tokens to e parsed to expressions to be compiled to assembly code to be executed on the virtual CPU... Also to be used to Parse English grammar to make abstract syntax trees.
abstract-syntax-tree context-free-grammar stack-vm stack-virtual-cpu programming-language compiler compiler-design parser tokenizer-parser langugae-design
Language:Visual Basic .NET 4
Rajspeaks / Machine-Learning-approach-to-Bengali-Corpus-Tokenization-Stemming-POS-Tagging-using-BNLTK
Machine Learning approach to Bengali Corpus POS Tagging using BNLTK. This is an experimenting project under the mentorship of Prof. Sandipan Ganguly, HIT-K.
nlp nlp-machine-learning nlp-library natural-language-processing natural-language-understanding bengali-natural-language-processing bengali-nlp bengali bengali-language-processing bengali-dataset postagging postagger stemming stemmer tokenizer-parser english machine-learning rajdeep-das rajspeaks
Language:Jupyter Notebook 3
stcjs / flkit
A JS/HTML/CSS Toolkit(Tokenizer、Parser) Support Template Syntax
css compress tokenizer-parser
Language:HTML 3
2439905184 / bkscr-parser
bkengine脚本的解析器（开源实现）基于python3.8.4
bkengine tokenizer-parser python compiler
Language:Python 2
daireto / odata-v4-query
A lightweight, simple and fast parser for OData V4 query options supporting standard query parameters. Provides helper functions to apply OData V4 query options to ORM/ODM queries such as SQLAlchemy and Beanie.
beanie-odm odata odata-query-parser odatav4 sqlalchemy tokenizer-parser pymongo
Language:Python 2
DuffsDevice / tiny-parser
Write use-case specific parsers within minutes!
context-free-grammar parser parser-generator parser-library tokenizer tokenizer-parser
Language:Python 2
ldaniels528 / oxide
Oxide is a hybrid database and streaming messaging system (think Kafka + MySQL); supporting data access via REST and SQL.
rust golang lollypop tokenizer tokenizer-parser api dataframes sql database microservices
Language:Rust 2
Critter-World-README
MitchellGray100 / Critter-World-README
A README for my private CS 2112 Critter World Project
readme simulation-game javafx java game-development parser threading tokenizer-parser
2
vxern / text_expressions
A tiny and complete tool to supercharge static JSON strings with dynamic, user-defined expressions.
parser language translation user-defined-expressions expression expression-evaluator lexer lexer-parser tokenizer tokenizer-parser udf
Language:Dart 2
yetnt / jaiva
An esoteric functional programming language written in Java
esolang esoteric-language esoteric-programming-language interpreted-programming-language interpreter java new-programming-language programming-language tokenizer tokenizer-parser jaiva yetnt functional-programming maven new programming
Language:Java 2
ander-db / Tokenizer-C-Plus-Plus
A very fast and low memory usage C++ automaton tokenizer that breaks an input string into a list of tokens looking at tabs, spaces, new lines, and detects special tokens like numbers, prces, personal noms, emails, lexemes, etc. It allows to specify delimeters and detect special cases.
automaton cplusplus tokenizer tokenizer-parser
Language:C++ 1
atmay / Tokenization
tokeniser for math in c#
csharp open-source formal-grammar tokenization tokenizer tokenizer-parser
Language:C# 1
idris-aitmoulay / minimal-parser
this repository contient a minimal parser
parser ast tokenizer tokenizer-parser typescript javascript
Language:TypeScript 1
Simple-Pascal-Like-Language-Interpreter
jackr276 / Simple-Pascal-Like-Language-Interpreter
An interpreter for a custom-made, Pascal-Like Programming Language
programming-language compiler-design interpreters tokenizer-parser
Language:C++ 1
jasont14 / tokenizer
Python Token Tokenizer for SQL using Postgresql Keywords
postgresql sql token tokenizer-parser parser
Language:Python 1
antlr4-demonstration-project
melchisedech333 / antlr4-demonstration-project
:wrench: Demonstration of using ANTLR4 (with runtime for C++) in projects for context-free grammar processing. The ANTLR4 (Java) package is included, and the project is configured to compile on Linux.
grammar-parser grammar-checker grammar-rules grammars language-processors semantic-analysis syntax syntax-analysis syntax-analyzer language-development semantics tokenization tokenizer tokenizer-parser syntax-tree antlr antlr-grammar antlr4-grammar antlr-development-environment grammars-utility
Language:C++ 1
nguraijs / nguraijs
A lightweight, fast, and extensible line-by-line content tokenizer and parser
javascript javascript-library syntax-tree tokenizer tokenizer-parser trie-structure typescript
Language:TypeScript 1
nousantx / nguraijs
A small and lightweight library for string tokenizing
javascript-library tokenizer tokenizer-parser typescript typescript-library
Language:TypeScript 1
paulo-granthon / uml-gen-tw
An automatic UML generator for Java that *actually works*
devtool devtools tokenizer tokenizer-parser uml uml-diagram
Language:Rust 1
xxEBxx / Math_eval
Command-line application designed to parse and evaluate mathematical expressions
math tokenizer-parser
Language:C++ 1
benjamin-trainor / json-parser
Recreating JSON.parse
abstract-syntax-tree json parsing tokenizer-parser typescript
Language:TypeScript 0
zavvdev / elise-lang
Mini programming language.
lexer parser programming-language rust interpreter language semantic-analysis semantic-analyzer tokenizer tokenizer-parser
Language:Rust 0
ardac67 / lisp_lisp
some works on lisp
lisp tokenizer-parser
Language:Common Lisp
gabsfredes / LISPtoPOSFIX
Software produzido na disciplina de Linguagens Formais que deverá receber como entrada uma expressão em LISP e retornar a expressão posfixa equivalente.
linguagens-formais parser ply tokenizer-parser
Language:Python
ldaniels528 / tinyDB.go
An experimental database implementation written in pure Go.
go golang rust lollypop tokenizer tokenizer-parser
Language:Go
luminati-io / Golang-html-parsing
Parse HTML in Go using Node Parser, Tokenizer, and tools like Goquery and Colly, with practical examples and efficient web scraping techniques
bright-data colly go golang goquery html node-html-parser parser tokenizer tokenizer-parser web-scraping
mikiyadd / my-c-array
Dynamic array implementation in C with a modular, folder-based structure.
antlr-language-development array c cipher conventions cpp data-structures language-processing semantic-analysis semantics syntax syntax-analyzer tokenization tokenizer-parser
Language:C

tokenizer-parser

risesoft-y9 / Data-Labeling

ikskuh / parser-toolkit

ehwan / C-language-Parser-In-Rust

bytexenon / LuaXen

recp / xml

melchisedech333 / antlr4-experiments

zbroderson / HumanTimeParser

aboss123 / Ice-Programming-Language-Submission

melchisedech333 / lex-yacc-experiments

spydaz / AI_ZX81

Rajspeaks / Machine-Learning-approach-to-Bengali-Corpus-Tokenization-Stemming-POS-Tagging-using-BNLTK

stcjs / flkit

2439905184 / bkscr-parser

daireto / odata-v4-query

DuffsDevice / tiny-parser

ldaniels528 / oxide

MitchellGray100 / Critter-World-README

vxern / text_expressions

yetnt / jaiva

ander-db / Tokenizer-C-Plus-Plus

atmay / Tokenization

idris-aitmoulay / minimal-parser

jackr276 / Simple-Pascal-Like-Language-Interpreter

jasont14 / tokenizer

melchisedech333 / antlr4-demonstration-project

nguraijs / nguraijs

nousantx / nguraijs

paulo-granthon / uml-gen-tw

xxEBxx / Math_eval

benjamin-trainor / json-parser

zavvdev / elise-lang

ardac67 / lisp_lisp

gabsfredes / LISPtoPOSFIX

ldaniels528 / tinyDB.go

luminati-io / Golang-html-parsing

mikiyadd / my-c-array