[Question] Are multiple file backends possible? Can i add an additional file-backend for specific resource?

[Karsten Kraus]

Hi,

i'd like to have a default (file-)backend for all my resources. For a specific resource i'd like to add another, additional (file-)backend with keys specific for this resource. Is this possible? For now, the backend-definition in the resource seems to overwrite the default-backend(s)?

So in my remco.toml i've got this:

[default_backends]
    [default_backends.file]
        watch = true
        interval = 60
        keys = [ "/" ]
        filepath = "default-config.yml"

And in one of my resource-definitions (separate file) i got this:

[backend]
    [backend.file]
        onetime = true
        keys = [ "/" ]
        prefix = "/my-service/"
        filepath = "specific-resource-config.yml"

In the templates for this resource i'd like to be able to access the keys of the default-config and the keys for this specific resource, ideally without having to re-declare the default-file-backend in the resource.

Kind regards!

[Karsten Kraus]

Hi there,

is this a really simple (or stupid question)? Or where my many edits just to annoying? How can i make amends? ;-)

Kind regards!

[Rene Kaufmann]

Sorry for the late response! :)

Currently, each backend can only be used once per resource (except the backend plugins).

Technically it would be easy to support multiple backends.
But I am not sure if it is possible to do this without changing the format of the current configuration file.

[backend]
    [backend.file]
        onetime = true
        keys = [ "/" ]
        prefix = "/my-service/"
        filepath = "specific-resource-config.yml"

this would turn to

[backend]
   [[backend.file]]
        onetime = true
        keys = [ "/" ]
        prefix = "/my-service/"
        filepath = "specific-resource-config.yml"

what would break everyones config files.

You could of course rewrite the file backend as a plugin and use this instead.

[Karsten Kraus]

Hi,

thanks for the replay and, yeah, i get that. Changing the datastructure would break things...

I'm mostly a java developer so my solution would be to allow arbitrary suffixes (e.g. [backend.mycustomuniqueid]) and then have an attribute "class" or "implementation" which has the name of the class implementing the backend as value which i'd instantiate by reflection.

So either the suffix matches a "well known" backend or the "implementation" attribute must be present or, to stay more in line with the current syntax for plugins, when the suffix of backend.suffix does not match a known backend and the "path"-attribute does not start with a dot or a slash then try to match it with a known/built-in backend)?

Could that be a backwards compatible way to implement my feature request? If yes, then i could try to create a pr, but that would take some time as i've got no expirience with go whatsoever...

[Rene Kaufmann]

That could work, but let me think about it a little bit before you put too much work into it.

Here is the file backend wrapped as a plugin in the meantime:

package main

import (
	"context"
	"fmt"
	"log"
	"net/rpc/jsonrpc"

	"github.com/HeavyHorst/easykv"
	"github.com/HeavyHorst/easykv/file"
	"github.com/HeavyHorst/remco/pkg/backends/plugin"
	"github.com/natefinch/pie"
)

func NewFileClient(fpath string) (*file.Client, error) {
	c, err := file.New(fpath)
	if err != nil {
		return nil, err
	}
	return c, nil
}

type FileRPCServer struct {
	client *file.Client
}

func main() {
	p := pie.NewProvider()
	if err := p.RegisterName("Plugin", &FileRPCServer{}); err != nil {
		log.Fatalf("failed to register Plugin: %s", err)
	}
	p.ServeCodec(jsonrpc.NewServerCodec)
}

func (c *FileRPCServer) Init(args map[string]string, resp *bool) error {
	var err error
	if fpath, ok := args["fpath"]; ok {
		c.client, err = NewFileClient(fpath)
		if err != nil {
			return err
		}
		*resp = true
		return nil
	}
	return fmt.Errorf("I need an File Path !")
}

func (c *FileRPCServer) GetValues(args []string, resp *map[string]string) error {
	r, err := c.client.GetValues(args)
	if err != nil {
		return err
	}

	*resp = r
	return nil
}

func (c *FileRPCServer) Close(args interface{}, resp *interface{}) error {
	c.client.Close()
	return nil
}

func (c *FileRPCServer) WatchPrefix(args plugin.WatchConfig, resp *uint64) error {
	r, err := c.client.WatchPrefix(context.Background(), args.Prefix, easykv.WithKeys(args.Opts.Keys), easykv.WithWaitIndex(args.Opts.WaitIndex))
	*resp = r
	return err
}

The config would look like this:

[backend]
    [[backend.plugin]]
      path = "/etc/remco/plugins/fileplug"
      onetime = true
      keys = [ "/" ]
      prefix = "/my-service/"
      [backend.plugin.config]
      	fpath = "specific-resource-config.yml"