lexical-analysis

There are 7 repositories under lexical-analysis topic.

DoctorWkt / acwj
A Compiler Writing Journey
compiler lexical-analysis parsing c
Language:C 12440
baidu / lac
百度NLP：分词，词性标注，命名实体识别，词重要性
chinese-nlp chinese-word-segmentation java lexical-analysis named-entity-recognition part-of-speech-tagger python word-segmentation
Language:C++ 3973
SeanLee97 / xmnlp
xmnlp：提供中文分词, 词性标注, 命名体识别，情感分析，文本纠错，文本转拼音，文本摘要，偏旁部首，句子表征及文本相似度计算等功能
pinyin nlp spell-checker radical sentiment-analysis ner lexical-analysis segmentation postagging sentence-embeddings sentence-similarity
Language:Python 1291
katef / libfsm
DFA regular expression library & friends
fsm regex lexer-generator regexp regexes regex-validator dfa nfa finite-state-machine finite-state-automata finite-state-machines lexer lexical-analysis lexing lex automata compiler compilers compiler-principles
Language:C 968
DmitryRyumin / INTERSPEECH-2023-24-Papers
INTERSPEECH 2023-2024 Papers: A complete collection of influential and exciting research papers from the INTERSPEECH 2023-24 conference. Explore the latest advances in speech and language processing. Code included. Star the repository to support the advancement of speech technology!
acoustic adaptation asr audio-signals interspeech interspeech2023 interspeech2024 language-modeling lexical-analysis linguistic-analysis machine-translation prosody self-supervised-learning signal-processing speech-analysis speech-production speech-recognition speech-synthesis speech-technology transmission
685
qinwf / jiebaR
Chinese text segmentation with R. R语言中文分词（文档已更新 🎉 ：https://qinwenfeng.com/jiebaR/ )
cppjieba jieba chinese-text-segmentation lexical-analysis chinese nlp
Language:C++ 362
WGrape / lexer
A lexical analyzer based on DFA that is built using JS and supports multi-language extensions / 一个基于DFA的支持多语言扩展的JS版开源词法分析器
lexer lexical-analysis lexical-analyzer dfa javascript
Language:JavaScript 346
ghaiklor / llvm-kaleidoscope
LLVM Tutorial: Kaleidoscope (Implementing a Language with LLVM)
llvm llvm-tutorial lexer llvm-ir kaleidoscope language lexical-analysis
Language:C++ 265
gecko0307 / dlib
Allocators, I/O streams, math, geometry, image and audio processing for D
utility-library dlang linear-algebra computational-geometry image-processing audio-processing networking xml-parser containers threading memory-allocation lexical-analysis json-parser
Language:D 224
drifter1 / compiler
This is an implementation of a compiler for a simple C-like language that uses the C tools Flex and Bison.
lexical-analysis syntax-analysis symbol-table bison flex compiler-design compiler semantic-analysis machine-code lexer parser c assembly mips mips-assembly
Language:C 189
SwiftStudies / OysterKit
OysterKit is a framework that provides a native Swift scanning, lexical analysis, and parsing capabilities. In addition it provides a language that can be used to rapidly define the rules used by OysterKit called STLR
decoder language lexical-analysis parsing-capabilities swift
Language:Swift 178
AnjaneyaTripathi / c-compiler
A compiler that accepts any valid program written in C. It is made using Lex and Yacc. Returns a symbol table, parse tree, annotated syntax tree and intermediate code.
lex lexical-analysis yacc parse-tree intermediate-code-generation compiler compiler-construction symbol-table frontend semantic-analysis syntax-tree c parser ast
Language:Yacc 142
brucewlee / lingfeat
[EMNLP 2021] LingFeat - A Comprehensive Linguistic Features Extraction ToolKit for Readability Assessment
discourse feature-extraction flesch-kincaid lexical-analysis linguistic-analysis natural-language-processing nlp readability-metrics readability-scores semantic-analysis spacy syntactic-analysis text-classification text-simplification
Language:Python 129
LSYS / LexicalRichness
:smile_cat: :speech_balloon: A module to compute textual lexical richness (aka lexical diversity).
python linguistic-analysis data-mining natural-language-processing nlp data-science information-retrieval lexical-analysis natural-language lexical-analyzer
Language:Python 110
h2337 / clex
clex is a simple lexer generator
lex lexer lexer-framework lexer-generator lexer-library lexical-analysis lexical-analyzer finite-state-machine nfa regex regex-engine regexp
Language:C 102
kamadorueda / santiago
Santiago is a lexing and parsing toolkit for Rust
parser parser-combinators lexer bison flex bison-yacc rust bison-flex bison-grammar parsing parser-generator lexical-analysis
Language:Rust 102
elle
acquitelol / elle
A procedural programming language built in Rust which compiles to QBE
c-style-lang compiler educational language rust elle lexer lexical-analysis qbe procedural parser programming-language
Language:Rust 86
Aurore54F / JStap
Modular static malicious JavaScript detection system
javascript static-analysis data-flow-analysis control-flow-analysis syntactic-analysis lexical-analysis classification malware-detection python3
Language:Python 73
techcentaur / PyLex
Perform lexical analysis on words, one word at a time.
python3 words scraping poets cli lexical-analysis nlp
Language:Python 64
gbroques / compiler
A simple compiler written from scratch in C++ for an undergraduate course in program translation.
compiler scanner lexical-analysis parser parse-trees assembly-language compiler-optimization compiler-frontend compiler-design
Language:C++ 63
quil-lang / alexa
A Lexical Analyzer Generator
lexer lex lexical-analysis common-lisp
Language:Common Lisp 63
PratikSavla / compiler-design-lab
These are my programs for compiler design lab work in my sixth semester
python python-3 regular-expression nfa regex regular-expression-to-nfa compiler-design theory-of-computation lexical-analyzer lexical-analysis eliminate-left-recursion left-recursion-elimination
Language:Python 59
YaccConstructor / YaccConstructor
Platform for parser generators and other grammarware research and development. GLL, RNGLR, graph parsing algorithms, and many others are included.
parser-generator fsharp glr-parser gll-parser parsing lexical-analysis lexing grammarware ebnf
Language:F# 41
ghaiklor / pascal-interpreter
A simple interpreter for a large subset of Pascal language written for educational purposes
pascal interpreter lexer tokenizer scanner parser ast pascal-language syntax-analysis lexical-analysis ast-nodes semantic-analysis visitor symbol-table grammar-rules parse ast-node pascal-interpreter
Language:JavaScript 39
truongpt / meo
subset c compiler
compiler 100daysofcode lexical-analysis x86-64
Language:C++ 37
soul-soft / SqlBatis
A high performance Micro-ORM supporting SQL Server, MySQL, Sqlite etc..
mybatis linq orm mybatis-net expression lexical-analysis dapper sqlbatis
Language:C# 35
shivananda199 / Mini-C-Compiler
compiler compiler-design lexical-analysis parser syntax-analysis c-compiler semantic-analysis intermediate-code lexer lex yacc
Language:Yacc 34
alirezakay / RegexToDFA
converting regex (regular expression) to DFA directly by creating syntax tree in java
regular-expression dfa syntax-tree netbeans java regex directly regex-to-dfa compiler straightly in-java regex-to-dfa-java deterministic finite-state-machine automata synthesis lexical-analysis regular-expressions compilers syntax-analysis
Language:Java 30
SwiLex
yassram / SwiLex
A universal lexer library in Swift.
lexer lexical-analysis syntax-analysis swift-package-manager tokenizer
Language:Swift 30
devangi2000 / Compiler-Design
A repository containing all experiments for the course Compiler Design.
automata-theory compiler-design computer-science first-follow-compiler intermediate-code-generation lexical-analysis nfa-to-dfa-conversion parsing-expression-grammar predictive-parsing python3 regex-to-nfa shift-reduce-parsers
Language:Python 28
ftchirou / blink-lexer
Starting code of the first challenge in the Let's Build a Programming Language series.
compiler-design interpreter lbpl lexical-analysis programming-language-concepts
Language:JavaScript 28
jnyfah / CuriousX
A simple miniature Compiler
compiler lexical-analysis syntax-analysis semantic-analysis codegenerator lexer lexer-parser
Language:C++ 28
FelipeTomazEC / Lexical-Analyzer
A basic Lexical-Analyzer written in Java.
javafx-application java javafx lexical-analysis compilers
Language:Java 27
amirbawab / EasyCC-CPP
Easy Compiler Compiler written in C++
compiler c-plus-plus lexical-analysis syntax-analysis compiler-design compiler-construction programming-language
Language:C++ 25
phorward / libphorward
C/C++ library for dynamic data structures, regular expressions, lexical analysis & more...
linked-list hash-table dynamic-arrays lexical-analysis regular-expressions regular-expression-engine documentation-tool prototype-generator regular-expression c c-plus-plus
Language:C 25
PELIC-dataset
ELI-Data-Mining-Group / PELIC-dataset
The University of Pittsburgh English Language Institute Corpus (PELIC) dataset
learner-corpus second-language-acquisition tesol concordancer corpus intensive-english-program english-for-academic-purposes second-language-writing esl longitudinal-data lexical-analysis
Language:HTML 24