Database Storage of URLs
heynemann opened this issue · comments
Is your feature request related to a problem?
I really want to use this at my company for everyone to use together, but we need to store the links in a database such that we can run this in Kubernetes and not depend on the filesystem. Any chance this has come up already? If it is something you would consider adding as a feature I'd be happy to contribute.
Describe the solution you'd like
Provide the capability to abstract url storage and retrieval to allow filesystem and other storage options such as relational or document databases.
Additional context
No response
Hi @heynemann, we also have plans to support other databases instead of SQLite. And I'm a bit curious as to which you're more comfortable with, MySQL or PostgreSQL?
Either would be fine. What do you think of supporting Gorm and then we can easily allow for a myriad of dbs. We can definitely start with either pg or MySQL.
How about just raw SQL? Just like the way it is now for SQLite. Personally, I think that Go's ORM is kind of not good and writing raw SQL is more clearer and easier to understand.
A related discussion: usememos/memos#2517 (comment)
Can surely do that :) In such a small problem space TBH whatever works best for slash. Will try to send something your way soon.
FYI, we just implemented postgres driver. If you want, you can start with the yourselfhosted/slash:test docker image. All you need to do is add the related flags in startup command. e.g.,
docker run -d --name slash -p 5231:5231 -v ~/.slash/:/var/opt/slash yourselfhosted/slash:test --driver postgres --dsn 'postgresql://postgres:PASSWORD@localhost:5432/slash'With this docker-compose I'm getting an error migrating the DB. There seems to be an opportunity here to improve the logging of errors in migrating as it seems to be swallowing errors.
version: '3'
services:
postgres:
image: postgres
restart: always
user: root
ports:
- "5432:5432"
environment:
PGUSER: slash
POSTGRES_USER: slash
POSTGRES_PASSWORD: slash
POSTGRES_DATABASE: slash
healthcheck:
test: ["CMD-SHELL", "pg_isready"]
interval: 5s
timeout: 5s
retries: 5
networks:
- local
slash:
image: yourselfhosted/slash:test
container_name: slash
ports:
- "5231:5231"
volumes:
- ~/.slash/:/var/opt/slash
command: --driver postgres --dsn 'postgresql://slash:slash@postgres:5432/slash'
depends_on:
postgres:
condition: service_healthy
networks:
- local
networks:
local:
The error I got:
slash | ---
slash | Server profile
slash | dsn: /var/opt/slash/slash_prod.db
slash | port: 5231
slash | mode: prod
slash | version: 0.5.0
slash | ---
slash | 2023-12-18T16:47:10.322Z ERROR slash/main.go:49 failed to migrate db {"error": "failed to apply latest schema: migrate error: DROP TABLE IF EXISTS migration_history CASCADE;\nDROP TABLE IF EXISTS workspace_setting CASCADE;\nDROP TABLE IF EXISTS \"user\" CASCADE;\nDROP TABLE IF EXISTS user_setting CASCADE;\nDROP TABLE IF EXISTS shortcut CASCADE;\nDROP TABLE IF EXISTS activity CASCADE;\nDROP TABLE IF EXISTS collection CASCADE;\nDROP TABLE IF EXISTS memo CASCADE;\n\n-- migration_history\nCREATE TABLE migration_history (\n version TEXT NOT NULL PRIMARY KEY,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())\n);\n\n-- workspace_setting\nCREATE TABLE workspace_setting (\n key TEXT NOT NULL UNIQUE,\n value TEXT NOT NULL\n);\n\n-- user\nCREATE TABLE \"user\" (\n id SERIAL PRIMARY KEY,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',\n email TEXT NOT NULL UNIQUE,\n nickname TEXT NOT NULL,\n password_hash TEXT NOT NULL,\n role TEXT NOT NULL CHECK (role IN ('ADMIN', 'USER')) DEFAULT 'USER'\n);\n\nCREATE INDEX idx_user_email ON \"user\"(email);\n\n-- user_setting\nCREATE TABLE user_setting (\n user_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n key TEXT NOT NULL,\n value TEXT NOT NULL,\n PRIMARY KEY (user_id, key)\n);\n\n-- shortcut\nCREATE TABLE shortcut (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',\n name TEXT NOT NULL UNIQUE,\n link TEXT NOT NULL,\n title TEXT NOT NULL DEFAULT '',\n description TEXT NOT NULL DEFAULT '',\n visibility TEXT NOT NULL CHECK (visibility IN ('PRIVATE', 'WORKSPACE', 'PUBLIC')) DEFAULT 'PRIVATE',\n tag TEXT NOT NULL DEFAULT '',\n og_metadata TEXT NOT NULL DEFAULT '{}'\n);\n\nCREATE INDEX idx_shortcut_name ON shortcut(name);\n\n-- activity\nCREATE TABLE activity (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n type TEXT NOT NULL DEFAULT '',\n level TEXT NOT NULL CHECK (level IN ('INFO', 'WARN', 'ERROR')) DEFAULT 'INFO',\n payload TEXT NOT NULL DEFAULT '{}'\n);\n\n-- collection\nCREATE TABLE collection (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n name TEXT NOT NULL UNIQUE,\n title TEXT NOT NULL DEFAULT '',\n description TEXT NOT NULL DEFAULT '',\n shortcut_ids INTEGER ARRAY NOT NULL,\n visibility TEXT NOT NULL CHECK (visibility IN ('PRIVATE', 'WORKSPACE', 'PUBLIC')) DEFAULT 'PRIVATE'\n);\n\nCREATE INDEX idx_collection_name ON collection(name);\n\n-- memo\nCREATE TABLE memo (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',\n name TEXT NOT NULL UNIQUE,\n title TEXT NOT NULL DEFAULT '',\n content TEXT NOT NULL DEFAULT '',\n visibility TEXT NOT NULL CHECK (visibility IN ('PRIVATE', 'WORKSPACE', 'PUBLIC')) DEFAULT 'PRIVATE',\n tag TEXT NOT NULL DEFAULT ''\n);\n\nCREATE INDEX idx_memo_name ON memo(name);\n: missing \"=\" after \"/var/opt/slash/slash_prod.db\" in connection info string\"", "errorVerbose": "missing \"=\" after \"/var/opt/slash/slash_prod.db\" in connection info string\"\nmigrate error: DROP TABLE IF EXISTS migration_history CASCADE;\nDROP TABLE IF EXISTS workspace_setting CASCADE;\nDROP TABLE IF EXISTS \"user\" CASCADE;\nDROP TABLE IF EXISTS user_setting CASCADE;\nDROP TABLE IF EXISTS shortcut CASCADE;\nDROP TABLE IF EXISTS activity CASCADE;\nDROP TABLE IF EXISTS collection CASCADE;\nDROP TABLE IF EXISTS memo CASCADE;\n\n-- migration_history\nCREATE TABLE migration_history (\n version TEXT NOT NULL PRIMARY KEY,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())\n);\n\n-- workspace_setting\nCREATE TABLE workspace_setting (\n key TEXT NOT NULL UNIQUE,\n value TEXT NOT NULL\n);\n\n-- user\nCREATE TABLE \"user\" (\n id SERIAL PRIMARY KEY,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',\n email TEXT NOT NULL UNIQUE,\n nickname TEXT NOT NULL,\n password_hash TEXT NOT NULL,\n role TEXT NOT NULL CHECK (role IN ('ADMIN', 'USER')) DEFAULT 'USER'\n);\n\nCREATE INDEX idx_user_email ON \"user\"(email);\n\n-- user_setting\nCREATE TABLE user_setting (\n user_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n key TEXT NOT NULL,\n value TEXT NOT NULL,\n PRIMARY KEY (user_id, key)\n);\n\n-- shortcut\nCREATE TABLE shortcut (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',\n name TEXT NOT NULL UNIQUE,\n link TEXT NOT NULL,\n title TEXT NOT NULL DEFAULT '',\n description TEXT NOT NULL DEFAULT '',\n visibility TEXT NOT NULL CHECK (visibility IN ('PRIVATE', 'WORKSPACE', 'PUBLIC')) DEFAULT 'PRIVATE',\n tag TEXT NOT NULL DEFAULT '',\n og_metadata TEXT NOT NULL DEFAULT '{}'\n);\n\nCREATE INDEX idx_shortcut_name ON shortcut(name);\n\n-- activity\nCREATE TABLE activity (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n type TEXT NOT NULL DEFAULT '',\n level TEXT NOT NULL CHECK (level IN ('INFO', 'WARN', 'ERROR')) DEFAULT 'INFO',\n payload TEXT NOT NULL DEFAULT '{}'\n);\n\n-- collection\nCREATE TABLE collection (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n name TEXT NOT NULL UNIQUE,\n title TEXT NOT NULL DEFAULT '',\n description TEXT NOT NULL DEFAULT '',\n shortcut_ids INTEGER ARRAY NOT NULL,\n visibility TEXT NOT NULL CHECK (visibility IN ('PRIVATE', 'WORKSPACE', 'PUBLIC')) DEFAULT 'PRIVATE'\n);\n\nCREATE INDEX idx_collection_name ON collection(name);\n\n-- memo\nCREATE TABLE memo (\n id SERIAL PRIMARY KEY,\n creator_id INTEGER REFERENCES \"user\"(id) NOT NULL,\n created_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n updated_ts BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW()),\n row_status TEXT NOT NULL CHECK (row_status IN ('NORMAL', 'ARCHIVED')) DEFAULT 'NORMAL',\n name TEXT NOT NULL UNIQUE,\n title TEXT NOT NULL DEFAULT '',\n content TEXT NOT NULL DEFAULT '',\n visibility TEXT NOT NULL CHECK (visibility IN ('PRIVATE', 'WORKSPACE', 'PUBLIC')) DEFAULT 'PRIVATE',\n tag TEXT NOT NULL DEFAULT ''\n);\n\nCREATE INDEX idx_memo_name ON memo(name);\n\ngithub.com/yourselfhosted/slash/store/db/postgres.(*DB).applyLatestSchema\n\t/backend-build/store/db/postgres/migrator.go:137\ngithub.com/yourselfhosted/slash/store/db/postgres.(*DB).Migrate\n\t/backend-build/store/db/postgres/migrator.go:33\nmain.glob..func1\n\t/backend-build/bin/slash/main.go:47\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\nmain.Execute\n\t/backend-build/bin/slash/main.go:93\nmain.main\n\t/backend-build/bin/slash/main.go:160\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\nfailed to apply latest schema\ngithub.com/yourselfhosted/slash/store/db/postgres.(*DB).Migrate\n\t/backend-build/store/db/postgres/migrator.go:34\nmain.glob..func1\n\t/backend-build/bin/slash/main.go:47\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\nmain.Execute\n\t/backend-build/bin/slash/main.go:93\nmain.main\n\t/backend-build/bin/slash/main.go:160\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
slash | main.glob..func1
slash | /backend-build/bin/slash/main.go:49
slash | github.com/spf13/cobra.(*Command).execute
slash | /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987
slash | github.com/spf13/cobra.(*Command).ExecuteC
slash | /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
slash | github.com/spf13/cobra.(*Command).Execute
slash | /go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
slash | main.Execute
slash | /backend-build/bin/slash/main.go:93
slash | main.main
slash | /backend-build/bin/slash/main.go:160
slash | runtime.main
slash | /usr/local/go/src/runtime/proc.go:267
slash exited with code 0
Found the issue. This diff fixes it:
diff --git a/server/profile/profile.go b/server/profile/profile.go
index 7091304..c941b0a 100644
--- a/server/profile/profile.go
+++ b/server/profile/profile.go
@@ -9,7 +9,6 @@ import (
"github.com/pkg/errors"
"github.com/spf13/viper"
-
"github.com/yourselfhosted/slash/server/version"
)
@@ -82,16 +81,18 @@ func GetProfile() (*Profile, error) {
}
}
- dataDir, err := checkDSN(profile.Data)
- if err != nil {
- fmt.Printf("Failed to check dsn: %s, err: %+v\n", dataDir, err)
- return nil, err
- }
+ if profile.Driver == "sqlite" {
+ dataDir, err := checkDSN(profile.Data)
+ if err != nil {
+ fmt.Printf("Failed to check dsn: %s, err: %+v\n", dataDir, err)
+ return nil, err
+ }
- profile.Data = dataDir
- dbFile := fmt.Sprintf("slash_%s.db", profile.Mode)
- profile.DSN = filepath.Join(dataDir, dbFile)
- profile.Version = version.GetCurrentVersion(profile.Mode)
+ profile.Data = dataDir
+ dbFile := fmt.Sprintf("slash_%s.db", profile.Mode)
+ profile.DSN = filepath.Join(dataDir, dbFile)
+ profile.Version = version.GetCurrentVersion(profile.Mode)
+ }
return &profile, nil
}Did you update the docker image with label test?
I got it running but now I get an error whenever I try to create a new shortcut like this:
And the log in the DB is:
go-postgres-1 | 2023-12-19 19:57:25.019 UTC [101] ERROR: function json_extract(text, unknown) does not exist at character 141
go-postgres-1 | 2023-12-19 19:57:25.019 UTC [101] HINT: No function matches the given name and argument types. You might need to add explicit type casts.
go-postgres-1 | 2023-12-19 19:57:25.019 UTC [101] STATEMENT:
go-postgres-1 | SELECT
go-postgres-1 | id,
go-postgres-1 | creator_id,
go-postgres-1 | created_ts,
go-postgres-1 | type,
go-postgres-1 | level,
go-postgres-1 | payload
go-postgres-1 | FROM activity
go-postgres-1 | WHERE 1 = 1 AND type = $1 AND level = $2 AND json_extract(payload, '$.shortcutId') = 1
go-postgres-1 | 2023-12-19 19:57:40.920 UTC [101] ERROR: duplicate key value violates unique constraint "shortcut_name_key"
go-postgres-1 | 2023-12-19 19:57:40.920 UTC [101] DETAIL: Key (name)=(test) already exists.
go-postgres-1 | 2023-12-19 19:57:40.920 UTC [101] STATEMENT:
go-postgres-1 | INSERT INTO shortcut (creator_id,name,link,title,description,visibility,tag,og_metadata)
go-postgres-1 | VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
go-postgres-1 | RETURNING id, created_ts, updated_ts, row_status
go-postgres-1 |
go-postgres-1 | 2023-12-19 19:58:47.769 UTC [101] ERROR: duplicate key value violates unique constraint "shortcut_name_key"
go-postgres-1 | 2023-12-19 19:58:47.769 UTC [101] DETAIL: Key (name)=(test) already exists.
go-postgres-1 | 2023-12-19 19:58:47.769 UTC [101] STATEMENT:
go-postgres-1 | INSERT INTO shortcut (creator_id,name,link,title,description,visibility,tag,og_metadata)
go-postgres-1 | VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
go-postgres-1 | RETURNING id, created_ts, updated_ts, row_status
go-postgres-1 |
And even though the error says the uniqueness has been violated, the list of shortcuts is empty:
Thanks for the report. Fixed with bec2c15. It is recommended to empty the database before restarting slash.
Editing the shortcut gives me this:
I will test postgres uniformly in the next few days. Thanks for your feedbacks again!
Already released in the v0.5.1.
Is mysql still being considered or is postgres the only db option outside of sqlite for the foreseeable future?
@Huskydog9988 It's already in the roadmap, but it won't be a priority.