[Feature] Optimize Tokenzation incluing multi-mode problems, Parser and Formula optimization
KenelmQLH opened this issue · comments
Longhu Qin commented
Description
(A clear and concise description of what the feature is.)
- Handle multi-mode problems
- AST Graph
- Image
- Handle noise problems when identify
$...$ in Parser (need better rules) - Handle Formula ast problems when identify
$AB=BC$ and$123$ (consider preprocessing)