Custom Comparator in Go

[Muthu Krishnan]

Comparator interface needs Native method

Native() *C.rocksdb_comparator_t

Cgo translates C types into equivalent unexported Go types. Because the translations are unexported, a Go package should not expose C types in its exported API: a C type used in one Go package is different from the same C type used in another.

Because of this, it is not possible to implement this interface.

[Muthu Krishnan]

@linxGnu

[Linh Tran Tuan]

@muthukrishnan24 please forgive me that I did not fully understand your question?

Could you please describe it in simpler way?

• What you want to archive? New API? Expose native pointer?
• What is your motivation to do so

[Muthu Krishnan]

I'm trying to create a Custom Comparator for a CF

package main

// #include "rocksdb/c.h"
import "C"
import (
	"bytes"
	"fmt"
	"os"

	"github.com/linxGnu/grocksdb"
)

func main() {
	opts := grocksdb.NewDefaultOptions()
	opts.SetCreateIfMissing(true)

	defOpts := grocksdb.NewDefaultOptions()
	defOpts.SetComparator(&CustomComparator{})

	db, cfHands, err := grocksdb.OpenDbColumnFamilies(opts, "testdb", []string{"default"}, []*grocksdb.Options{defOpts})
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	defer db.Close()

	defCF := cfHands[0]

	writeOpts := grocksdb.NewDefaultWriteOptions()
	db.PutCF(writeOpts, defCF, []byte("Hello"), []byte("World"))
	writeOpts.Destroy()

	readOpts := grocksdb.NewDefaultReadOptions()
	defer readOpts.Destroy()

	data, err := db.GetCF(readOpts, defCF, []byte("Hello"))
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	if data.Exists() {
		fmt.Println(string(data.Data()))
	}
}

type CustomComparator struct{}

func (f *CustomComparator) Compare(a, b []byte) int {
	return bytes.Compare(b, a)
}

// The name of the comparator.
func (f *CustomComparator) Name() string {
	return "custom"
}

// Return native comparator.
func (f *CustomComparator) Native() *C.rocksdb_comparator_t {
	return nil
}

// Destroy comparator.
func (f *CustomComparator) Destroy() {}

// also not valid
// func (f *CustomComparator) Native() *grocksdb._Ctype_struct_rocksdb_comparator_t { return nil }

Since Comparator needs to implement method Native() *C.rocksdb_comparator_t

Above code throws following errors

./main.go:18:24: cannot use &CustomComparator{} (type *CustomComparator) as type grocksdb.Comparator in argument to defOpts.SetComparator:
        *CustomComparator does not implement grocksdb.Comparator (wrong type for Native method)
                have Native() *_Ctype_struct_rocksdb_comparator_t
                want Native() *grocksdb._Ctype_struct_rocksdb_comparator_t

Directly referencing the type, also not possible since it is always unexported

./main.go:59:38: cannot refer to unexported name grocksdb._Ctype_struct_rocksdb_comparator_t

[Linh Tran Tuan]

@muthukrishnan24 I will reproduce at my side and tell you.

[Linh Tran Tuan]

main.go file:

package main

// #cgo LDFLAGS: -lrocksdb -pthread -lstdc++ -ldl -lm -lzstd -llz4 -lz -lsnappy
// #include "rocksdb/c.h"
// #include "a.h"
import "C"

import (
	"fmt"
	"os"
	"unsafe"

	"github.com/linxGnu/grocksdb"
)

func main() {
	opts := grocksdb.NewDefaultOptions()
	opts.SetCreateIfMissing(true)

	// work:
	cmp := C.comparator_create(0)
	opts.SetNativeComparator(unsafe.Pointer(cmp))

	// Also work:
	// opts.SetComparator(grocksdb.NewComparator("abc", func(a, b []byte) int {
	// 	return bytes.Compare(b, a)
	// }))

	db, err := grocksdb.OpenDb(opts, "testdb")
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	defer db.Close()

	// insert keys
	givenKeys := [][]byte{[]byte("key1"), []byte("key2"), []byte("key3")}
	wo := grocksdb.NewDefaultWriteOptions()
	for _, k := range givenKeys {
		if err := db.Put(wo, k, []byte("val")); err != nil {
			panic(err)
		}
	}

	// create a iterator to collect the keys
	ro := grocksdb.NewDefaultReadOptions()
	iter := db.NewIterator(ro)
	defer iter.Close()

	// we seek to the last key and iterate in reverse order
	// to match given keys
	for iter.SeekToFirst(); iter.Valid(); iter.Next() {
		key := make([]byte, 4)
		copy(key, iter.Key().Data())
		fmt.Println(string(key))
	}
}

a.h file:

#include <stdlib.h>
#include "rocksdb/c.h"

// This API provides convenient C wrapper functions for rocksdb client.

/* Base */
static void gorocksdb_destruct_handler(void* state) {
}

static const char* cmp_name(void*) { 
	return "abc";
}

static int compare_custom(void* state, const char* a, size_t alen, const char* b, size_t blen) {
	for (size_t i = 0; i < alen; ++i)
	    if (a[i] > b[i]) return -1;
	return 1;
}

/* Comparator */
rocksdb_comparator_t* comparator_create(uintptr_t idx) {
	return rocksdb_comparator_create(
        (void*)idx,
        gorocksdb_destruct_handler,
        compare_custom,
        cmp_name);
}

[Linh Tran Tuan]

@muthukrishnan24 I have updated the API to make your code built-able, as well as easier to create custom comparator:

• Using go code logic as comparator
• Using c code logic as comparator (I highly recommend this way)

The patch is included in the release: https://github.com/linxGnu/grocksdb/releases/tag/v1.6.47
PR for this patch: #67

I also gave an example above to test. PTAL

[Muthu Krishnan]

Thanks @linxGnu