macu / crowdspec

Crowdsourced Specifications - example custom architecture for Google App Engine project in Go

CrowdSpec

© Matt Cudmore 2020

Mac setup

A postgres password must be set before the app can connect.

$ brew install postgresql go node jq
$ psql postgres
postgres=# ALTER USER matt WITH PASSWORD 'somepassword';
postgres=# \q
$ cd crowdspec
// Update crowdspec/env.json
$ npm install
$ sh init.sh

Create Postgres database

$ psql postgres
postgres=# CREATE DATABASE crowdspec;

Access database from command line

$ psql crowdspec
postgres=# \dt
postgres=# \d+ spec

Set up env.json for local development

{
	"dbUser": "testuser",
	"dbPass": "testpass",
	"dbName": "crowdspec",
	"httpPort": "2020",
	"adminUserId": 1,
	"recaptchaSiteKey": "...",
	"recaptchaSecretKey": "...",
	"mailjetApiKey": "...",
	"mailjetSecretKey": "...",
	"versionStamp": ""
}

Version stamp is updated automatically when running restart.sh.

Build and run

On first use, or to re-initialize database:

$ sh init.sh

To rebuild client and server, and run:

$ sh restart.sh

Publish

Make two separate deploys for database upgrades, setting MAINTENANCE_MODE to "true" in app.yaml for the first deploy to disable the site temporarily.

$ gcloud app deploy

Update cron jobs:

$ gcloud app deploy cron.yaml

Reindex postgres collations after version upgrade

To refresh a collation, run in postgres:

reindex table user_account;
reindex table user_signup_request;
reindex table spec;
reindex table spec_subspec;
reindex table spec_url;
ALTER COLLATION public.case_insensitive REFRESH VERSION;

Vacuum

postgres=# SELECT pg_size_pretty(pg_database_size('postgres'));
postgres=# SELECT pg_size_pretty(pg_database_size('crowdspec'));
postgres=#

Changelog

2022-02

• Upgrade to Vue 3
• Make public specs publicly accessible
• Add PL/pgSQL functions to verify access permissions

2022-01

• Refactoring
• Jump to subspec modal update
• Edit block modal ref form updates

2021-02

• Add maintenance mode
• Signup requests
• Add unreadOnly community setting
• Use YouTube API for fetching video previews
• Add Markdown support
• Add community review page
• Initialize add block modal style type according to siblings
• Add code highlighting

2021-01

• Add community space, comments, sub-comments, read records, and unread counts
• Scroll viewport as needed after move or insert block
• Multiselect bulk block move (within or across contexts)

2020-12

• Add user settings
• Add forgot password form and email service
• Freely move blocks between spec/subspec contexts
• Change routes structure
• Add loading messages when navigating specs
• Case insensitive username login
• Add username highlight
• Allow moving blocks into empty contexts

2020-08-31

• spec/subspec last modified time displayed to visitors now reflects block updates
• change password form
• reCAPTCHA on login

2020-08-30

• toggle spec public
• public specs on dashboard
• remove editing features for visitors
• show content unavailable when block refs have been deleted
• record and display last modified times
• change all timestamp columns to TIMESTAMPTZ