decred / dcrpool

This issue will serve as an umbrella issue for all of the changes needed to support Postgres as a database option.

The intention is to add new config items which will allow an administrator to select between using the current bbolt database, or an external postgres database.

This will require separating the current database code from the the codebase, making it generic enough to create an abstraction layer, re-implementing the current db code in Postgres, and adding the new config items which will allow switching between the two modes.

The above PRs extract the majority of the database code from the rest of the code. I will soon push a PR which abstracts the database behind an interface which looks something like the following:

type database interface {
    createShare(share) err
    retrieveShare(share) (share, err)
    deleteShare(share) (share, err)
    ... etc
}

We need to start considering how to handle database backups. dcrpool currently:

writes a database backup on shutdown
allows downloading a database backup through the web UI

Another consideration is the behaviour when dcrpool is changed from solo mode to pool mode (or vice versa). Currently we make a backup of the database file and start a new empty database in the new mode. Is this desirable (or even feasible) with postgres?

I think we should maintain writing a database backup as well on shutdown as downloading a backup via the web UI for the bolt db driver, the postgres db driver will not support these functionality. Backing up for postgres will require a separate db instance synchronizing to the live instance. We can outline how to set this up in the README, I think that's as far as we go for postgres regarding backup.

I've hit a little bit of a roadblock because of how heavily the tests depend on the database implementation. To provide an example, when the payment manager is tested, we pass it a real database implementation and then we verify the paymentmgr behaviour by checking the contents of the database. A good example is here

dcrpool/pool/paymentmgr_test.go

Lines 192 to 212 in 11b44c2

    
           err = persistShare(db, xID, weight, eightyBefore) // Share A 
        
           if err != nil { 
        
           	t.Fatal(err) 
        
           } 
        
           err = persistShare(db, yID, weight, thirtyBefore) // Share B 
        
           if err != nil { 
        
           	t.Fatal(err) 
        
           } 
        
           err = mgr.pruneShares(sixtyBefore) 
        
           if err != nil { 
        
           	t.Fatal(err) 
        
           } 
        
           // Ensure share A got pruned with share B remaining. 
        
           shareAID := shareID(xID, eightyBefore) 
        
           _, err = fetchShare(db, shareAID) 
        
           if err == nil { 
        
           	t.Fatal("expected value not found error") 
        
           }

This single test is verifying both the paymentmgr behaviour, and the database access code.

In a perfect world we would pass the payment manager a mock database and validate its behaviour against the mock, rather than using a real db.
We would then have a separate test for database access code.

And in future, when we add postgres support, we would add a new test for database access via postgres, and leave the payment mgr test untouched

Encountered an issue with how shares are encoded and stored in boltDB.

Share created on time is not stored as a discrete value in boltDB. Instead, it is converted from an integer value into bytes, hex encoded, and stored as the first 16 characters of the share ID.

This means that we cannot select for it in SQL queries, eg SELECT * FROM shares WHERE created_on <= max;. And because of the encoding, we can't even do something like SELECT * FROM shares WHERE SUBSTRING(uuid, 0, 15) <= max;.

One way to tackle this would be just select all shares from the DB and filter them in the Go code, but this could be very inefficient if we have a huge number of shares in the DB and we only want to select a small number of them. Another approach is to start storing created_on as a discrete value in the shares bucket/table, but thats going to need a boltDB migration.

Appreciate some thoughts @dnldd

Chatted through the issue above with @dnldd and he kindly created #275 to resolve.
He also created #276 which, coupled with #277, resolves a similar issue where payment uuid was not being stored as a discrete value.

	err = persistShare(db, xID, weight, eightyBefore) // Share A
	if err != nil {
	t.Fatal(err)
	}

	err = persistShare(db, yID, weight, thirtyBefore) // Share B
	if err != nil {
	t.Fatal(err)
	}

	err = mgr.pruneShares(sixtyBefore)
	if err != nil {
	t.Fatal(err)
	}

	// Ensure share A got pruned with share B remaining.
	shareAID := shareID(xID, eightyBefore)
	_, err = fetchShare(db, shareAID)
	if err == nil {
	t.Fatal("expected value not found error")
	}