wuruilongll / pg_jieba

Postgresql full-text search extension for chinese

pg_jieba

pg_jieba is a PostgreSQL extension for full-text search of Chinese.

NOTE

It is tested on

This branch require C++11(gcc4.8+), because the new version of cppjieba upgrade to C++11.
If the OS compiler did not support C++11, please try old version of pg_jieba as branch v1.0.1

PREPARE

Make sure PostgreSQL is installed and command pg_config could be runnable.

Install Postgres:

• Option:
  download from http://www.postgresql.org/download/

• Option:
  CentOS:

  sudo yum install postgresql postgresql-server postgresql-devel
  

• Option:
  Download From http://www.enterprisedb.com/products-services-training/pgdownload

INSTALL

1. Downloads

git clone https://github.com/jaiminpan/pg_jieba

2. Init submodule

cd pg_jieba

# initilized sub-project
git submodule update --init --recursive

3. Compile

cd pg_jieba

mkdir build
cd build

cmake ..

make
make install 
# if got error when doing "make install"
# try "sudo make install"

Compile Failed Q&A

Q: Postgresql is installed customized
A: Try cmd as following
cmake -DCMAKE_PREFIX_PATH=/PATH/TO/PGSQL_INSTALL_DIR ..

Q: Ubuntu, To specify version of pg(missing: PostgreSQL_TYPE_INCLUDE_DIR)
A: cmake -DPostgreSQL_TYPE_INCLUDE_DIR=/usr/include/postgresql/10/server ..

Q: In some OS such as Ubuntu.
A: Try cmd as following
cmake -DCMAKE_CXX_FLAGS="-Wall -std=c++11" ..

HOW TO USE & EXAMPLE

General

jieba=# create extension pg_jieba;
CREATE EXTENSION

jieba=# select * from to_tsquery('jiebacfg', '是拖拉机学院手扶拖拉机专业的。不用多久，我就会升职加薪，当上CEO，走上人生巅峰。');
                                        to_tsquery
-----------------------------------------------------------------------------------------------
'拖拉机' & '学院' & '手扶拖拉机' & '专业' & '不用' & '多久' & '会' & '升职' & '加薪' & '当上' & 'ceo' & '走上' & '人生' & '巅峰'
(1 row)

jieba=# select * from to_tsvector('jiebacfg', '是拖拉机学院手扶拖拉机专业的。不用多久，我就会升职加薪，当上CEO，走上人生巅峰。');
                                          to_tsvector
-----------------------------------------------------------------------------------------------------
'ceo':18 '不用':8 '专业':5 '人生':21 '会':13 '加薪':15 '升职':14 '多久':9 '学院':3 '巅峰':22 '当上':17 '手扶拖拉机':4 '拖拉机':2 '走上':20
(1 row)

Token And Tag

jieba=# select * from ts_token_type('jieba');
 tokid | alias |         description
-------+-------+-----------------------------
     1 | eng   | letter
     2 | nz    | other proper noun
     3 | n     | noun
... ...
... ...
    55 | ug    | ug
    56 | rz    | rz
    57 |       |
(56 rows)

jieba=# select * from ts_debug('jiebacfg', '是拖拉机学院手扶拖拉机专业的。不用多久，我就会升职加薪，当上CEO，走上人生巅峰。');
 alias |  description  |   token    | dictionaries | dictionary |   lexemes
-------+---------------+------------+--------------+------------+--------------
 v     | verb          | 是         | {jieba_stem} | jieba_stem | {}
 n     | noun          | 拖拉机     | {jieba_stem} | jieba_stem | {拖拉机}
 n     | noun          | 学院       | {jieba_stem} | jieba_stem | {学院}
 n     | noun          | 手扶拖拉机 | {jieba_stem} | jieba_stem | {手扶拖拉机}
 n     | noun          | 专业       | {jieba_stem} | jieba_stem | {专业}
 uj    | uj            | 的         | {jieba_stem} | jieba_stem | {}
 x     | unknown       | 。         | {jieba_stem} | jieba_stem | {}
 v     | verb          | 不用       | {jieba_stem} | jieba_stem | {不用}
 m     | numeral       | 多久       | {jieba_stem} | jieba_stem | {多久}
 x     | unknown       | ，         | {jieba_stem} | jieba_stem | {}
 r     | pronoun       | 我         | {jieba_stem} | jieba_stem | {}
 d     | adverb        | 就         | {jieba_stem} | jieba_stem | {}
 v     | verb          | 会         | {jieba_stem} | jieba_stem | {会}
 v     | verb          | 升职       | {jieba_stem} | jieba_stem | {升职}
 nr    | person's name | 加薪       | {jieba_stem} | jieba_stem | {加薪}
 x     | unknown       | ，         | {jieba_stem} | jieba_stem | {}
 t     | time          | 当上       | {jieba_stem} | jieba_stem | {当上}
 eng   | letter        | CEO        | {jieba_stem} | jieba_stem | {ceo}
 x     | unknown       | ，         | {jieba_stem} | jieba_stem | {}
 v     | verb          | 走上       | {jieba_stem} | jieba_stem | {走上}
 n     | noun          | 人生       | {jieba_stem} | jieba_stem | {人生}
 n     | noun          | 巅峰       | {jieba_stem} | jieba_stem | {巅峰}
 x     | unknown       | 。         | {jieba_stem} | jieba_stem | {}

Here is alternative configs;

• jiebamp: Use mp
• jiebahmm: Use hmm
• jiebacfg: Combine MP&HMM(Mix). Used in most situation (Recommand)
• jiebaqry: First use Mix, then use full. Similar to the one used by web search engines.

Config	Statment	Result
jiebamp	我来到北京清华大学	'来到' & '北京' & '清华大学'
jiebamp	他来到了网易杭研大厦	'来到' & '网易' & '杭' & '研' & '大厦'
jiebamp	小明硕士毕业于**科学院计算所，后在日本京都大学深造	'明' & '硕士' & '毕业' & '**科学院' & '计算所' & '日本京都大学' & '深造'

Config	Statment	Result
jiebahmm	我来到北京清华大学	'我来' & '北京' & '清华大学'
jiebahmm	他来到了网易杭研大厦	'他来' & '网易' & '杭' & '研大厦'
jiebahmm	小明硕士毕业于**科学院计算所，后在日本京都大学深造	'小明' & '硕士' & '毕业于' & '**' & '科学院' & '计算' & '日' & '本京' & '大学' & '深造'

Config	Statment	Result
jiebacfg	我来到北京清华大学	'来到' & '北京' & '清华大学'
jiebacfg	他来到了网易杭研大厦	'来到' & '网易' & '杭研' & '大厦'
jiebacfg	小明硕士毕业于**科学院计算所，后在日本京都大学深造	'小明' & '硕士' & '毕业' & '**科学院' & '计算所' & '日本京都大学' & '深造'

Config	Statment	Result
jiebaqry	我来到北京清华大学	'来到' & '北京' & '清华' & '华大' & '大学' & '清华大学'
jiebaqry	他来到了网易杭研大厦	'来到' & '网易' & '杭研' & '大厦'
jiebaqry	小明硕士毕业于**科学院计算所，后在日本京都大学深造	'小明' & '硕士' & '毕业' & '**' & '科学' & '学院' & '科学院' & '**科学院' & '计算' & '计算所' & '日本' & '京都' & '大学' & '日本京都大学' & '深造'

USER DEFINED DICTIONARY

Dictionary Format

• Words weight type
• Words type
• Words

  云计算
  韩玉鉴赏
  蓝翔 nz
  区块链 10 nz
  

Reference jieba_user.dict

How to use your own dictionary

cd /PATH/TO/POSTGRESQL_INSTALL/share/postgresql/tsearch_data
OR
cd /PATH/TO/POSTGRESQL_INSTALL/share/tsearch_data

cp 'YOUR DICTIONARY' jieba_user.dict

Dictionary Sharing

• dict.367W.utf8 iLife

Parameter

When pg_jieba loaded by shared_preload_libraries, The following configuration options are available and can be added into postgresql.conf

• pg_jieba.hmm_model (Need Restart) HMM Model file.
• pg_jieba.base_dict (Need Restart) Base dictionary.
• pg_jieba.user_dict (Need Restart) csv list of specific user dictionary name(Exclude suffix .dict). All should located in dir tsearch_data.

Postgresql parameter

# shared_preload_libraries = 'pg_jieba.so'  # (change requires restart)

# default_text_search_config='pg_catalog.simple'; default value
# default_text_search_config='jiebacfg'; uncomment to make 'jiebacfg' as default

Online Test

You can test for result by test link (Suggest opened by Chrome)

HISTORY

history

Package Dependency

• cppjieba v5.0.0

Docker

There is docker file by @ssfdust.

# scripts
docker run --name testjieba -e POSTGRES_PASSWORD=passwd -e POSTGRES_USER=test -e POSTGRES_DB=testdb -d ssfdust/psql_jieba_swsc
docker exec -ti testjieba psql -U test testdb

THANKS

jieba project by SunJunyi
CppJieba project by WuYanyi