This commit contains a complete redesign and rewrite of the database
package that approaches things in a vastly different manner than the
previous version. This is the first part of several stages that will be
needed to ultimately make use of this new package.
A quick summary of the changes is as follows:
- The previous database could only contain blocks on the main chain and
reorgs required deleting the blocks from the database. This made it
impossible to store orphans and could make external RPC calls for
information about blocks during the middle of a reorg fail.
- The previous database interface forced a high level of coin-specific
intelligence such as spend tracking into each backend driver.
- The aforementioned point led to making it difficult to implement new
backend drivers due to the need to repeat a lot of non-trivial logic
which is better handled at a higher layer, such as the blockchain
package.
- The old database stored all blocks in leveldb. This made it extremely
inefficient to do things such as lookup headers and individual
transactions since the entire block had to be loaded from leveldb (which
entails it doing data copies) to get access.
In order to address all of these concerns, and others not mentioned, the
database interface has been redesigned as follows:
- Two main categories of functionality are provided: block storage and
metadata storage
- All block storage and metadata storage are done via read-only and
read-write MVCC transactions with both manual and managed modes
- Support for multiple concurrent readers and a single writer
- Readers use a snapshot and therefore are not blocked by the writer
- Some key properties of the block storage and retrieval API:
- It is generic and does NOT contain additional coin logic such spend
tracking and block linking
- Provides access to the raw serialized bytes so deserialization is not
forced for callers that don't need it
- Support for fetching headers via independent functions which allows
implementations to provide significant optimizations
- Ability to efficiently retrieve arbitrary regions of blocks
(transactions, scripts, etc)
- A rich metadata storage API is provided:
- Key/value with arbitrary data
- Support for buckets and nested buckets
- Bucket iteration through a couple of different mechanisms
- Cursors for efficient and direct key seeking
- Supports registration of backend database implementations
- Comprehensive test coverage
- Provides strong documentation with example usage
This commit also contains an implementation of the previously discussed
interface named ffldb (flat file plus leveldb metadata backend). Here
is a quick overview:
- Highly optimized for read performance with consistent write performance
regardless of database size
- All blocks are stored in flat files on the file system
- Bulk block region fetching is optimized to perform linear reads which
improves performance on spindle disks
- Anti-corruption mechanisms:
- Flat files contain full block checksums to quickly an easily detect
database corruption without needing to do expensive merkle root
calculations
- Metadata checksums
- Open reconciliation
- Extensive test coverage:
- Comprehensive blackbox interface testing
- Whitebox testing which uses intimate knowledge to exercise uncommon
failure paths such as deleting files out from under the database
- Corruption tests (replacing random data in the files)
In addition, this commit also contains a new tool under the new database
directory named dbtool which provides a few basic commands for testing the
database. It is designed around commands, so it could be useful to expand
on in the future.
178 lines
5.4 KiB
Go
178 lines
5.4 KiB
Go
// Copyright (c) 2015-2016 The btcsuite developers
|
|
// Use of this source code is governed by an ISC
|
|
// license that can be found in the LICENSE file.
|
|
|
|
package database2_test
|
|
|
|
import (
|
|
"bytes"
|
|
"fmt"
|
|
"os"
|
|
"path/filepath"
|
|
|
|
"github.com/btcsuite/btcd/chaincfg"
|
|
database "github.com/btcsuite/btcd/database2"
|
|
_ "github.com/btcsuite/btcd/database2/ffldb"
|
|
"github.com/btcsuite/btcd/wire"
|
|
"github.com/btcsuite/btcutil"
|
|
)
|
|
|
|
// This example demonstrates creating a new database.
|
|
func ExampleCreate() {
|
|
// This example assumes the ffldb driver is imported.
|
|
//
|
|
// import (
|
|
// "github.com/btcsuite/btcd/database"
|
|
// _ "github.com/btcsuite/btcd/database/ffldb"
|
|
// )
|
|
|
|
// Create a database and schedule it to be closed and removed on exit.
|
|
// Typically you wouldn't want to remove the database right away like
|
|
// this, nor put it in the temp directory, but it's done here to ensure
|
|
// the example cleans up after itself.
|
|
dbPath := filepath.Join(os.TempDir(), "examplecreate")
|
|
db, err := database.Create("ffldb", dbPath, wire.MainNet)
|
|
if err != nil {
|
|
fmt.Println(err)
|
|
return
|
|
}
|
|
defer os.RemoveAll(dbPath)
|
|
defer db.Close()
|
|
|
|
// Output:
|
|
}
|
|
|
|
// This example demonstrates creating a new database and using a managed
|
|
// read-write transaction to store and retrieve metadata.
|
|
func Example_basicUsage() {
|
|
// This example assumes the ffldb driver is imported.
|
|
//
|
|
// import (
|
|
// "github.com/btcsuite/btcd/database"
|
|
// _ "github.com/btcsuite/btcd/database/ffldb"
|
|
// )
|
|
|
|
// Create a database and schedule it to be closed and removed on exit.
|
|
// Typically you wouldn't want to remove the database right away like
|
|
// this, nor put it in the temp directory, but it's done here to ensure
|
|
// the example cleans up after itself.
|
|
dbPath := filepath.Join(os.TempDir(), "exampleusage")
|
|
db, err := database.Create("ffldb", dbPath, wire.MainNet)
|
|
if err != nil {
|
|
fmt.Println(err)
|
|
return
|
|
}
|
|
defer os.RemoveAll(dbPath)
|
|
defer db.Close()
|
|
|
|
// Use the Update function of the database to perform a managed
|
|
// read-write transaction. The transaction will automatically be rolled
|
|
// back if the supplied inner function returns a non-nil error.
|
|
err = db.Update(func(tx database.Tx) error {
|
|
// Store a key/value pair directly in the metadata bucket.
|
|
// Typically a nested bucket would be used for a given feature,
|
|
// but this example is using the metadata bucket directly for
|
|
// simplicity.
|
|
key := []byte("mykey")
|
|
value := []byte("myvalue")
|
|
if err := tx.Metadata().Put(key, value); err != nil {
|
|
return err
|
|
}
|
|
|
|
// Read the key back and ensure it matches.
|
|
if !bytes.Equal(tx.Metadata().Get(key), value) {
|
|
return fmt.Errorf("unexpected value for key '%s'", key)
|
|
}
|
|
|
|
// Create a new nested bucket under the metadata bucket.
|
|
nestedBucketKey := []byte("mybucket")
|
|
nestedBucket, err := tx.Metadata().CreateBucket(nestedBucketKey)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// The key from above that was set in the metadata bucket does
|
|
// not exist in this new nested bucket.
|
|
if nestedBucket.Get(key) != nil {
|
|
return fmt.Errorf("key '%s' is not expected nil", key)
|
|
}
|
|
|
|
return nil
|
|
})
|
|
if err != nil {
|
|
fmt.Println(err)
|
|
return
|
|
}
|
|
|
|
// Output:
|
|
}
|
|
|
|
// This example demonstrates creating a new database, using a managed read-write
|
|
// transaction to store a block, and using a managed read-only transaction to
|
|
// fetch the block.
|
|
func Example_blockStorageAndRetrieval() {
|
|
// This example assumes the ffldb driver is imported.
|
|
//
|
|
// import (
|
|
// "github.com/btcsuite/btcd/database"
|
|
// _ "github.com/btcsuite/btcd/database/ffldb"
|
|
// )
|
|
|
|
// Create a database and schedule it to be closed and removed on exit.
|
|
// Typically you wouldn't want to remove the database right away like
|
|
// this, nor put it in the temp directory, but it's done here to ensure
|
|
// the example cleans up after itself.
|
|
dbPath := filepath.Join(os.TempDir(), "exampleblkstorage")
|
|
db, err := database.Create("ffldb", dbPath, wire.MainNet)
|
|
if err != nil {
|
|
fmt.Println(err)
|
|
return
|
|
}
|
|
defer os.RemoveAll(dbPath)
|
|
defer db.Close()
|
|
|
|
// Use the Update function of the database to perform a managed
|
|
// read-write transaction and store a genesis block in the database as
|
|
// and example.
|
|
err = db.Update(func(tx database.Tx) error {
|
|
genesisBlock := chaincfg.MainNetParams.GenesisBlock
|
|
return tx.StoreBlock(btcutil.NewBlock(genesisBlock))
|
|
})
|
|
if err != nil {
|
|
fmt.Println(err)
|
|
return
|
|
}
|
|
|
|
// Use the View function of the database to perform a managed read-only
|
|
// transaction and fetch the block stored above.
|
|
var loadedBlockBytes []byte
|
|
err = db.Update(func(tx database.Tx) error {
|
|
genesisHash := chaincfg.MainNetParams.GenesisHash
|
|
blockBytes, err := tx.FetchBlock(genesisHash)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// As documented, all data fetched from the database is only
|
|
// valid during a database transaction in order to support
|
|
// zero-copy backends. Thus, make a copy of the data so it
|
|
// can be used outside of the transaction.
|
|
loadedBlockBytes = make([]byte, len(blockBytes))
|
|
copy(loadedBlockBytes, blockBytes)
|
|
return nil
|
|
})
|
|
if err != nil {
|
|
fmt.Println(err)
|
|
return
|
|
}
|
|
|
|
// Typically at this point, the block could be deserialized via the
|
|
// wire.MsgBlock.Deserialize function or used in its serialized form
|
|
// depending on need. However, for this example, just display the
|
|
// number of serialized bytes to show it was loaded as expected.
|
|
fmt.Printf("Serialized block size: %d bytes\n", len(loadedBlockBytes))
|
|
|
|
// Output:
|
|
// Serialized block size: 285 bytes
|
|
}
|