The readme demo code

[suntong]

How to fix the problem of

panic: no tokenizer with name or type 'gse' registered?

This is the program that I tried:

package main

import (
	"fmt"
	"os"

	"github.com/blevesearch/bleve"
)

func main() {
	INDEX_DIR := "bleve.gse"
	message := "工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作"

	mapping := bleve.NewIndexMapping()
	os.RemoveAll(INDEX_DIR)
	defer os.RemoveAll(INDEX_DIR)

	if err := mapping.AddCustomTokenizer("gse", map[string]interface{}{
		"type":       "gse",
		"user_dicts": "",
	}); err != nil {
		panic(err)
	}
	if err := mapping.AddCustomAnalyzer("gse", map[string]interface{}{
		"type":      "gse",
		"tokenizer": "gse",
	}); err != nil {
		panic(err)
	}
	mapping.DefaultAnalyzer = "gse"

	index, err := bleve.New(INDEX_DIR, mapping)
	if err != nil {
		panic(err)
	}
	if err := index.Index("1", message); err != nil {
		panic(err)
	}

	query := "干事亲口交待"
	req := bleve.NewSearchRequest(bleve.NewQueryStringQuery(query))
	req.Highlight = bleve.NewHighlight()
	res, err := index.Search(req)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Result of: '%s': %d matches\n", query, res.Total)
	for i, hit := range res.Hits {
		rv := fmt.Sprintf("%d. %s, (%f)\n", i+res.Request.From+1, hit.ID, hit.Score)
		for fragmentField, fragments := range hit.Fragments {
			rv += fmt.Sprintf("%s: ", fragmentField)
			for _, fragment := range fragments {
				rv += fmt.Sprintf("%s", fragment)
			}
		}
		fmt.Printf("%s\n", rv)
	}

	index.Close()
}

Thanks

[Song Liu]

add import bleve-gse-tokenizer: _ "github.com/leopku/bleve-gse-tokenizer"

import (
	"fmt"
	"os"

	"github.com/blevesearch/bleve"
	_ "github.com/leopku/bleve-gse-tokenizer"
)

[suntong]

Thanks for the swift reply.

Now I get:

Could not load dictionaries: "./data/dict/zh/dict.txt", open ./data/dict/zh/dict.txt: no such file or directory

Would that be specified by the line:

"user_dicts": "",?

Where can i find such file?

thx

[suntong]

Moreover, seems to me that fragments should be printed, but this is what I get:

Result of: '干事亲口交待': 1 matches
1. 1, (0.062850)

Is it normal, or fragments are indeed not printed? thx

[Song Liu]

Thanks for the swift reply.

Now I get:

Could not load dictionaries: "./data/dict/zh/dict.txt", open ./data/dict/zh/dict.txt: no such file or directory

Would that be specified by the line:

"user_dicts": "",?

Where can i find such file?

thx

Ooops. It is a bug. I will fix it later.

You are right. You can specify user dicts by "user_dicts: "/path/of/your/user/dict" to make things work.

[suntong]

?

[suntong]

The 761e58d, hmm, was not done right. I've reverted it since.

So this "Could not load dictionaries" remains as the problem.

Please reopen. thx

[Song Liu]

Fixed both v1 and v2.

Feel free to reopen for questions.