This adds code to migrate the existing block index in ffldb to the new
format managed by the blockchain package and updates the code to use the
new infrastructure.
This adds a new field to the block node struct named status that
consists of bit flags for keeping tracking of the validation state of
each block. The status is only stored in memory as of this commit, but
it will be stored as a part of the block index in the upcoming block
index migration code.
Since the field will be updated after node creation, this also
introduces some new functions for interacting with the status field in a
concurrent-safe fashion.
This is largely based on upstream commits 2492af0 and fb0d13c; however,
it does not include some of the logic in the reorganization paths those
commits include as that approach doesn't match the intended direction
this package is moving towards.
This simplifies and consolidates the logic for adding and removing nodes
to and from the block index by moving the initial add into
maybeAcceptBlock just after the node is created, adding the child
connection logic to the AddNode function, and introducing a RemoveNode
function that is called in dry run mode which undoes the aforementioned
changes.
In addition to being simpler logic to follow, it helps make it clear
that all blocks which are written to the database end up with a block
index entry, even if they ultimately fail to connect.
This modifies the newBlockNode function to accept the parent as an
argument to automatically connect the newly created node. When it is
not nil, the work sum will automatically be summed and the parent of the
new node will be set accordingly.
This simplifies the block node construction a bit and allows some
redundant code to be removed. It also paves the way for easier simpler
full block index construction in the future.
This separates the logic for populating the stake information in a block
node from the construction of the node in order to better delineate the
difference between the two and to pave the way for that information to
be stored separately in the database versus needing to load full blocks
to retrieve it.
This modifies the code in maybeAcceptBlock to return an error if there
is no previous entry in the block index for a given block indicating it
is either an orphan or the genesis block.
The function should never be called in either circumstance since the
genesis block is valid by definition and orphans are handled prior to
calling the function.
This modifies the blockchain code to store all blocks that have passed
proof-of-work and contextual validity tests in the database even if they
may ultimately fail to connect.
This eliminates the need to store those blocks in memory, allows them to
be available as orphans later even if they were never part of the main
chain, and helps pave the way toward being able to separate the download
logic from the connection logic.
Note that it also updates the blockExists function since the code base
is currently in the process of changing over to decouple download and
connection logic, but not all of the necessary parts are updated yet,
to ensure blocks that are in the database, but do not have an associated
main chain block index entry, are treated as if they do not exist for
the purposes of chain connection and selection logic.
This moves the checkBlockContext function to back to validate.go next to
checkBlockHeaderContext where it belongs to help minimize the
differences between the upstream code in order to facilitate easier
syncs due to less merge conflicts as a result of superfluous changes.
This moves the test for validating the block height specified by the
header matches the height it actually connects to the chain into the
checkBlockHeaderContext function where it more naturally belongs since
it is only dependent on the header and its contextual position within
the chain.
This makes the semantics of block fetching much more explicit by
introducing a new function named fetchMainChainBlockByHash which only
attempts to the load a block from the main chain block cache and then
falls back to the database. While currently only blocks in the main
chain are in the database, the intention is for future commits to
allow the database to house side chain blocks as well, so having clearly
defined semantics will help make that transition easier to verify.
While here, it also renames {f,F}etchBlockFromHash to {f,F}etchBlockByHash
to be consistent with the naming used by the other existing functions
and updates the comments to better describe the function semantics.
This modifies the block connection code paths to accept the parent block
in addition to the block as a parameter versus loading it on demand in
the called functions. This makes the dependency more explicit and also
ensures it doesn't potentially get loaded multiple times in the reorg
case.
This refactors the block index logic into a separate struct and
introduces an individual lock for it so it can be queried independent of
the chain lock.
It also modifies the `newBlockNode` function to accept nil for the
ticket spend information parameter and updates all of the test code that
doesn't require it to use nil.
This modifies the IsSStx, IsSSGen, and IsSSRtx functions to only return
a bool and introduces CheckSStx, CheckSSGen, and CheckSSRtx to return
the actual error as needed by consensus.
This is being done because "is" functions are much nicer to use when
they don't return an error and the callers that use them almost never
care why they aren't of the type, they just want to determine if they
are. In the few cases where the caller does care, they can use of the
new check functions.
While here, also update the comments to call out what the more common
names for the transaction types are and to add comments to the test
functions for consistency.
Finally, it updates all callers in the repo accordingly.
This adds a new function FindSpentTicketsInBlock that extracts all spent and
revoked tickets, plus the vote bits of a given block and uses that function
instead of the individual functions when loading a new node into the blockchain.
This improves startup time a bit, as the isSSGen, isSSRtx and determineTxType
are somewhat slow, as they need to decode the output script of the transactions.
Looping over the transactions once and doing a single test is faster.
Numbers for my computer:
```
| variation | mainnet | testnet |
| with patch | 23.7s | 42.6s |
| without patch | 35.9s | 118.0s |
```
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This implements the agenda for voting on the LN features as defined in
DCP0002 and DCP0003 along with consensus tests to ensure its correctness.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 5
- New block version on testnet is version 6
- Introduce a convenience function for determining if the vote passed
and is now active
- Introduce a new function for getting the consensus script verification
flags and setting the appropriate flags to enforce the new consensus
semantics in accordance with the state of the vote
- Enforce transaction finality checks based on the past median time in
accordance with the state of the vote
- Enforce relative time locks via sequence numbers in accordance with
the state of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0002 in accordance with the state of the vote
- It should be noted that the flag for DCP0003 is always set for
the more strict standardness checks because it is a soft fork while
the DCP0002 enforcement must depend on the result of the vote
because it is a hard fork
- Add tests for determining consensus script verification flags for both
mainnet and testnet
- Add tests for determining if the agenda is active for both mainnet and
testnet
This repurposes the sequence number of version 2 transaction inputs to
provide consensus-enforced relative lock-time semantics so that a
transaction can require inputs to have a specified relative age before
they are allowed to be included in a block. Each relative time lock can
either specify a relative number of seconds (with a granularity of 512
seconds and a maximum value of 33,553,920) or a specific number of
blocks (max 65535).
The number of seconds is calculated relative to the past median time of
the block before the one that contains the referenced output. This is
done because, due to other changes that will also be included in the
same agenda vote, said time will become the earliest possible time the
block that contains the referenced output could have been (technically
it will be one second after that, but that complexity is ignored since
there is already a granularity involved anyways).
It is also possible to disable the behavior by setting bit 31 of the
sequence number (which all transactions currently do by default since
they are set to the max).
In order for the transaction to be permitted to the mempool, relayed,
considered for inclusion into block templates, and allowed into a block,
the specified relative time locks for all of its inputs must be
satisfied.
This only implements the required logic and tests to enforce the new
behavior. Code to enforce the new behavior when considering candidate
transactions for acceptance to the mempool, relaying, and inclusion into
block templates will be added in a separate commit.
A consensus vote is required in order to reject blocks which contain
transactions that violate the new rules at a consensus level. Code to
selectively enable consensus enforcement based on the result of an
agenda vote will be added in a separate commit.
In order to accomplish this new behavior, the concept of a sequence lock
is introduced which allows the minimum possible time and height at which
a transaction can be included into a block to be calculated from all
inputs with non-disabled relative time locks, and functions to calculate
and evaluate the sequence lock are added.
The following is an overview of the changes:
- Introduce a new struct named SequenceLock to represent the previously
described sequence lock
- Define new constants related to sequence numbers named
SequenceLockTimeDisabled, SequenceLockTimeIsSeconds,
SequenceLockTimeMask, and SequenceLockTimeGranularity
- Add a new function named calcSequenceLock to calculate the sequence
lock for a given transaction
- Add a new function named SequenceLockActive to determine if a given
sequence lock is satisfied for a given block height and past median
time
- Add a convenience function named LockTimeToSequence which can be used
to convert a relative lock time to a sequence number
- Add a comprehensive set of tests for all of the new funcs
This removes the memory block node (header) pruning. It should be noted
that this does not apply to stake node pruning.
This is being done for two primary reasons:
- The goal is to ultimately have all block nodes in memory in the same
way the upstream code has done since it provides significant
optimization and code simplification opportunities
- Upcoming code that deals with calculating sequence locks on inputs
far in the past requires the ability to quickly calculate the median
time for arbitrarily old nodes and consequently pruning the memory
block nodes could lead to significant performance implications under
those conditions
Contains the following upstream commits:
- 42a4366ba8
- This is a NOOP since it has already previously been applied
- 77913ad2e8
Most of the changes related to this merge have already previously
applied, so this primarily just minimizes the differences versus
upstream.
Propagate version and bits on the getstakeversions call. Now we can
interrogate all raw vote version/bits data in addition to cooked values.
Diff is mostly mechanical exporting the Version:Bits tuple.
With this change we can keep the version and the bits in blockNode.
This is used to fix an oversight where we weren't strictly enforcing
the vote version. This also adds tests that validate quorum-1, quorum,
75%-1 and 75% rules during tallying.
Also adjust getvoteinfo totals to only return the specified vote version
totals.
This implements a new voting agenda for the testnet and simnet networks
for increasing the maximum block size from 1MiB to 1.25MB.
The following is an overview of the changes:
- Bump the maximum protocol block size limit to 1.25MB
- Bump the protocol version to 4
- Add wire tests for v3 protocol sizes and update tests for latest
- Update all wire values that are defined in terms of the max block size
to respect the protocol version
- Update the MaxSigOpsPerBlock constant to maintain existing value
regardless of the raised max protocol block size
- Decouple the maximum tx size from the block size by creating a chain
parameter for it with the current sizes so it is not a part of the
hard fork vote
- Add definition for new version 4 stake vote along with agenda to vote
on block size to the testnet and simnet networks
- Convert the MaximumBlockSize chain parameter to a slice in order to
hold the different possible sizes
- Adds a new function that returns the maximum block size based upon the
result of the vote
- Change the existing context-free block size check to enforce the max
protocol size and introduce a context-specific check which enforces a
restricted size based upon the network consensus parameters and the
result of the vote
- Set the max allowed size in generated block templates based upon the
network params and result of the vote
- Generate version 4 blocks and reject version 3 blocks after a super
majority has been reached
Port BIP0009 over from btcd. The following is an overview of the
changes:
- Add new configuration options to the chaincfg package which allows the
rule deployments to be defined per chain
- Implement code to calculate the threshold state as required by voting
- Use threshold state caches that are stored to the database in order
to accelerate startup time
- Remove caches that are invalid due to definition changes in the
params including additions, deletions, and changes to existing
entries
- Verify that PoW and PoS majorities have been reached.
- Add tests for new error type
- Add tests for threshold state
- Deployments are per stake version.
- Add human readable vote/choice infrastructure.
- Add RPC command to interrogate vote status (getvoteinfo).
Contains the following commits:
- 711f33450c
- b6b1e55d1e
- Reverted because Travis is already at a more recent version
- bd4e64d1d4
Also, the merge commit contains the necessary decred-specific
alterations, converts all other references to sha to hash to keep with
the spirit of the merged commits, and various other cleanup intended to
bring the code bases more in line with one another.
Bump block version and add stake version to wire. Currently
StakeVersions are unused and once the enforcement logic goes in the
versions will bump again.
This fixes#435
This modifies the blockchain.ProcessBlock function to return an
additional boolean as the first parameter which indicates whether or not
the block ended up on the main chain.
This is primarily useful for upcoming test code that needs to be able to
tell the difference between a block accepted to a side chain and a block
that either extends the main chain or causes a reorganize that causes it
to become the main chain. However, it is also useful for the addblock
utility since it allows a better error in the case a file with out of
order blocks is provided.
The blockchain pruning functions would run at every newly inserted
block node, causing performance degradation. Now the blockchain will
only prune nodes to return the memory to the garbage collector at
prespecified (5 minute) intervals, resulting in much less overhead
when syncing the blockchain.
The legacy ticket database, which was GOB serialized and stored on
shut down, has been removed. Ticket state information is now held in
a stake node, which acts as a modularized "black box" to contain all
information about the state of the stake system. Stake nodes are now
a component of the blockchain blockNode struct, and are updated with
them.
Stake nodes, like their internal treap primitives, are immutable
objects that are created with their connect and disconnect node
functions. The blockchain database now stores all information about
the stake state of the best node in the block database. The blockchain
makes the assumption that the stake state of the best node is known at
any given time. If the states of former blocks or sidechains must be
evaluated, this can be achieved by iterating backwards along the
blockchain from the best node, and then connecting stake nodes
iteratively if necessary.
Performance improvements with this new module are dramatic. The long
delays on start up and shut down are removed. Blockchain
synchronization time is improved approximately 5-10x on the mainnet
chain. The state of the database is atomic, so unexpected shut downs
should no longer have the ability to disrupt the chain state.
An upgrade path has been added for version 1 blockchain databases.
Users with this blockchain database will automatically update when
they start their clients.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
Contains the following upstream commits:
- ef9c50be57
- eb882f39f8
In addition to merging the fixes in the commits, this also fixes a few
more misspellings that were introduced in the new Decred code.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
This commit converts all block height references to int32 instead of
int64. The current target block production rate is 10 mins per block
which means it will take roughly 40,800 years to reach the maximum
height an int32 affords. Even if the target rate were lowered to one
block per minute, it would still take roughly another 4,080 years to
reach the maximum.
In the mean time, there is no reason to use a larger type which results
in higher memory and disk space usage. However, for now, in order to
avoid having to reserialize a bunch of database information, the heights
are still serialized to the database as 8-byte uint64s.
This is being mainly being done in preparation for further upcoming
infrastructure changes which will use the smaller and more efficient
4-byte serialization in the database as well.
This commit refactors the consensus rule checks for block headers and
blocks in the blockchain package into separate functions. These changes
contain no modifications to consensus rules and the code still passes all
block consensus tests. It is only a refactoring.
This is being done to help pave the way toward supporting concurrent
downloads. While the package already supports headers-first mode up
through the latest checkpoint through the use of the BFFastAdd flag and
hard-coded checkpoints, it currently only works when downloading from a
single peer. In order to support concurrent downloads from multiple
peers, the ability for the caller to do things such as independently
checking a block header (both context-free and full-context checks) will
be needed.
There are several more changes that will be necessary to support
concurrent downloads as well, such as making the package concurrent safe,
modifying it to make use of the new database API, etc. Those changes are
planned for future commits.
