vx416 / fts

FTS

FTS is a practice project which implement a inveret index.

Full-Text Search Solution

SQL LIKE Solution

if strings.Contains(doc.Text, term) {
	result.Add(doc)
}

Regex Solution

re, err := regexp.Compile(`(?i)\b` + term + `\b`)
...
if re.MatchString(doc.Text) {
	result.Add(doc)
}

Invert Index Solution

...
tokens := s.Normailze(s.Tokenize(term))

ids := make([]int, 0, 1)

for _, tok := range tokens {
  subIDs := s.index[tok]
  if len(ids) == 0 {
    ids = subIDs
  } else {
    ids = s.intersection(ids, subIDs)
  }
}
return s.findDocs(ids), nil

Benchmark

References

• let's build a full-text search engine
• source data