Giters

HisStar / search-legend

docs for search system and ai infra

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

search legend

中文版|Release Notes|Contributors

what it is?

In 2023 or the second half of 2022, after several years of dormancy, the big model finally showed his powerful ability to the world in the form of chatgpt. This project is the entry project of the Gottingen organization, which aims to introduce how to implement a search system and provide a suitable way to access the components of the large model. The project takes engineering realization and actual landing projects as the main line, and gradually explains the principle, design and function of the system.

Technical discussions

reference project

flowchart TB
    A[turbo]-->B[libtext]
    A[turbo]-->BB[titan-redis]
    BB-->C[lambda]
    A[turbo]-->C[lambda]
    A[turbo]-->D[hercules]
    A[turbo]-->E[rigel]
    E[rigel]-->D[hercules]
    D[hercules]-->F[hvm]
    C[lambda]-->F[hvm]
    B[libtext]-->F[hvm]
    E[rigel]-->G[melon-rpc]
    G[melon-rpc]-->F[hvm]

  • turbo c++ foundation library.

  • libtext chinese segment and normalize library.

  • titan-redis disk storage using redis interface.

  • lambda host local search engine for mix term and vector search.

  • rigel ai and rpc runtime library, to manage memory and fiber or thread schedule.

  • hercules python Aot framework for integrated components interface rpc and ai backend like tesorflow or pytorch

  • hvm (not start now) framework gateway, let us write python and generate c++ code and run in llvm.

the goals

flowchart LR
    A[python code]-->B[hvm.compile]
    B-->C[so library and graph]
    G[ci/cd]-->C[so library and graph]
    D[c++ runtime]-.->G
    D[c++ runtime]-->E[c++ app]
    F[service mess]-->E[c++ app]

it will reduce c++ communication and development phrase during business development。 may reduce business development days a lot and provide bug free code.

how ever, in some Modularity is not good system, you need to do a lot work int test and online, it is a very frustrating thing that， if many people develop together, so fuzz..

flowchart LR
  A[python algorithm]-->B[c++ review]-->C[c++ develop]-->D[test and online]
  AA[python algorithm]-->E[generate c++]-->D[test and online]

inspire by the ai aot. design it write a few python code like below

import hvm

q:str='刘德华'
def do_search(q:tr)->List[Any]
  qp = hvm.QU.parse(q)
  qt = hvm.query.boolen_query
  sq = hvm.query.term_query(qp)
  sq.must('title')
  qt.add_sub_query(sq)
  searcher  = hvm.search
  searcher.set_relavece('bm25')
  searcher.set_ranker('dw_fm')
  searcher.set_l1_sort('default')
  searcher.set_l2_sort('ctr_pri_sorter')
  result = searcher.search(engine)
  return result
hvm.compile(do_search, "./search_demo")

at c++ side,

hvm::Session session;
bool rs = session.load("./search_demo")
if(!rs){
    return false;
}      
string query;
session.run(query);

so it can do things in c++ code,Enjoy the joy brought by c++. and hidden the difficult code via c++ and hidden complex service governance logic behind.

nlp roadmap

Table of contents

About

docs for search system and ai infra

License:Apache License 2.0

Languages

Language:Jupyter Notebook 100.0%