opendilab / DI-store

OpenDILab RL Object Store

Introduction

Decision AI Store

Installation

Prerequisites

• Linux or macOS

• Python >= 3.6

pip install .

Quick Start

Start Etcd Server

di_store etcd_server ./conf/di_store.yaml

Start Node Tracker

di_store node_tracker ./conf/di_store.yaml

Start Storage Server

di_store storage_server ./conf/di_store.yaml

Start Storage Client

from di_store import Client
client = Client('./conf/di_store.yaml')
ref = client.put(b'Hello world.')
data = client.get(ref)
print('data:', data)
client.delete(ref)

Object prefetch

start storage server with group

di_store storage_server ./conf/di_store.yaml --group actor

or register group in client

client.register_group('actor')

specify prefetch_group when put object

ref = client.put(b'Hello world.', prefetch_group='actor')

When an object is put with prefetch_group specified, a background schedule is triggered to spread the object to all nodes corresponding to the group. The following diagram illustrates the data dissemination.

Running DI-store on Kubernetes

k8s/README.md

Directory Structure

├── di_store: python related codes
│     ├── bin: executable files needed by "etcd_server", "node_tracker" and "storage_server" commands
│     ├── cmd: driver for "di_store" command
│     ├── common: utilities for common
│     ├── driver
│     │     └── etcd_server_driver.py: etcd_server driver
│     ├── node_tracker
│     │     └── node_tracker_client.py: node_tracker client
│     ├── storage
│     │     └── storage_client.py: storage client
│     └── tracing
│           └── tracing.py: utilities for tracing
└── go: golang related codes
      ├── fb: generated by the FlatBuffers compiler
      ├── metadata: etcd client operations
      ├── node_tracker
      │     ├── fetch.go: fetch implementation of node_tracker
      │     ├── main
      │     │     └── node_tracker.go: the main entry point for node_tracker
      │     ├── node_tracker.go
      │     └── node_tracker_client.go
      ├── pb: generated by ProtoBuffers
      ├── plasma_client: golang codes that wraps libplasma
      ├── plasma_server: golang codes that wraps executable plasma-store-server
      ├── protos: .proto and .fbs files
      ├── release: outputs of command "go build", executable files of node_tracker and storage_server
      ├── storage_server
      │     ├── fetch.go: fetch implementation of storage_server
      │     ├── main
      │     │     └── storage_server.go: the main entry point for storage_server
      │     └── storage_server.go
      ├── tracing: utilities for tracing
      └── util

License

DI-store released under the Apache 2.0 license.