This finishes separating the mining code from the mempool that was
partially done in commit 9031d85 by extending the TxSource interface to
include the new functionality required by Decred.
The following is an overview of the changes to accomplish this:
- Move the VoteTx struct out of the mempool package in the mining
package and rename it to VoteDesc instead to signify it is a vote
descriptor and be consistent with the naming of TxDesc
- Update mempool to use the new mining.VoteDesc struct
- Rename CheckIfTxsExist to HaveAllTransactions to be more consistent
with HaveTransaction in the TxSource interface as it now lives
alongside it
- Add VoteHashesForBlock, VotesForBlocks, and IsTxTreeKnownInvalid
to the TxSource interface
- Switch the mining code to call the functions on the TxSource interface
instead and remove the reference to the concrete mempool
This replaces the mempool IsTxTreeValid function with
IsTxTreeKnownInvalid instead which more accurately describes its
semantics and purpose.
The actual purpose of this function is related to mempool acceptance and
building block templates for mining where the assumption is that the
previous block's regular transaction tree will be valid and is only
invalid specifically if there are enough known votes that vote against
it.
This change is desirable because the function provides semantics that
are actually opposite to that of the consensus rules where a tree is
only valid if it has a majority of yes votes and thus claiming it's
valid when that is not actually known for certain yet can easily cause
confusion.
The current code creates block templates with coinbase transactions that
have 4 uint64 extra nonces, but only ever uses one of them. Rather than
wasting the extra space, this modifies the code so it only uses a single
uint64 extra nonce.
It should be noted that realistically there isn't even a real need for
an extra nonce in the coinbase at all because there is extra space in
the header specifically for that purpose and miners can't modify the
coinbase via getwork anyways.
However, it is still necessary to insert for the current code in order
to ensure every block template has a unique merkle root since that is
what is used to uniquely identify the block template.
This modifies all files in the main package so that function definitions
are on the a single line as is the style used in the upstream code as
well as throughout its code base. This helps minimize the differences
between them and the upstream code in order to facilitate easier syncs
due to less merge conflicts as a result of superfluous changes.
This moves the priority-related code from the mempool package to the
mining package and also exports a new constant named UnminedHeight which
takes the place of the old unexported mempoolHeight.
Even though the mempool makes use of the priority code to make decisions
about what it will accept, priority really has to do with mining since
it influences which transactions will end up into a block. This change
also has the side effect of being a step towards enabling separation of
the mining code into its own package which, as previously mentioned,
needs access to the priority calculation code as well.
Finally, the mempoolHeight variable was poorly named since what it
really represents is a transaction that has not been mined into a block
yet. Renaming the variable to more accurately reflect its purpose makes
it clear that it belongs in the mining package which also needs the
definition now as well since the priority calculation code relies on it.
This will also benefit an outstanding PR which needs access to the same
value.
This fixes a bug where a transaction would lose reference to other
transactions dependant on it when being considered for inclusion in a
new block template. The issue only occurs when the transaction being
considered triggers a change of priority queue ordering to ordering by
fee. It results in none of the dependant transactions being considered
for inclusion in the new block template.
This prevents the ability for duplicate transactions to be added to the
mining priority heap when a transaction spends multiple outputs from the
same input transaction by converting the dependency tracking to use a
map keyed by the transaction hash instead of a linked list.
This modifies the mining code to request the desired blocks from chain
by hash as opposed to using a separate function to get the top block and
removes the now unused plubming through block manager.
Not only is this faster because it avoids the need to go through block
manager, it is also more robust since it does not rely on the tip of the
chain not having changing for correctness.
This replaces the GetGeneration function which allowed getting the
entire generation (all children stemming from the same parent) of an
arbitrary bock with TipGeneration which only returns the same
information for the tip block.
This is being done because the function is only used for mining purposes
to get the generation of the current tip. The code is simplified by
reducing its scope to its actual purpose as an initial benefit. It also
provides much better optimization opportunities later.
This modifies the exported CheckConnectBlock function to call the
checkBlockSanity and checkBlockContext functions to ensure all
validation rules are enforced from the mining code.
Since both the mining code and tests typically work with unsolved
blocks, this also introduces a new parameter on CheckConnectBlock to
pass behavior flags which allows the caller to skip the proof of work
check.
It also modifies forceHeadReorganization to call checkBlockContext for
the same reason.
Finally, all tests and call sites are updated with the appropriate
flags and various tests are updated for the changes accordingly.
This modifies the SSGenBlockVotedOn function to directly copy the hash
the vote commits to into a statically sized array versus calling the
hash function and thus can remove the potential runtime error.
This is preferred because the code will fail to compile if the size
of chainhash.HashSize is every changed which is desirable because it
would mean the assumptions the code makes would be invalidated.
It also updates all callers in the repository accordingly.
This modifies the BestState struct in the blockchain package to store
hashes directly instead of pointers to them and updates callers to deal
with the API change in the exported BestState struct.
In general, the preferred approach for hashes moving forward is to store
hash values in complex data structures, particularly those that will be
used for cache entries, and accept pointers to hashes in arguments to
functions.
Some of the reasoning behind making this change is:
- It is generally preferred to avoid storing pointers to data in cache
objects since doing so can easily lead to storing interior pointers into
other structs that then can't be GC'd
- Keeping the hash values directly in the structs provides better
cache locality
This modifies the IsSStx, IsSSGen, and IsSSRtx functions to only return
a bool and introduces CheckSStx, CheckSSGen, and CheckSSRtx to return
the actual error as needed by consensus.
This is being done because "is" functions are much nicer to use when
they don't return an error and the callers that use them almost never
care why they aren't of the type, they just want to determine if they
are. In the few cases where the caller does care, they can use of the
new check functions.
While here, also update the comments to call out what the more common
names for the transaction types are and to add comments to the test
functions for consistency.
Finally, it updates all callers in the repo accordingly.
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This implements the agenda for voting on the LN features as defined in
DCP0002 and DCP0003 along with consensus tests to ensure its correctness.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 5
- New block version on testnet is version 6
- Introduce a convenience function for determining if the vote passed
and is now active
- Introduce a new function for getting the consensus script verification
flags and setting the appropriate flags to enforce the new consensus
semantics in accordance with the state of the vote
- Enforce transaction finality checks based on the past median time in
accordance with the state of the vote
- Enforce relative time locks via sequence numbers in accordance with
the state of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0002 in accordance with the state of the vote
- It should be noted that the flag for DCP0003 is always set for
the more strict standardness checks because it is a soft fork while
the DCP0002 enforcement must depend on the result of the vote
because it is a hard fork
- Add tests for determining consensus script verification flags for both
mainnet and testnet
- Add tests for determining if the agenda is active for both mainnet and
testnet
This modifies the way standard verification flags are handled so that it
is possible to selectively enable them based on the result of agenda
votes.
First, it moves the StandardVerifyFlags constant from the txscript
package to the mempool/policy code and rename it to
BaseStandardVerifyFlags. As the TODO in the comment of the moved
constant indicated, these flags are policy related and thus really
belong in policy. Ideally there would be a completely separate policy
package, but since the policy code currently lives in mempool/policy.go,
the constant has been moved there.
Next, it introduces a new function named standardScriptVerifyFlags,
which accepts the chain as an argument and, for now, just returns the
BaseStandardVerifyFlags along with a nil error. This will allow
additional flags to be selectively enabled depending on the result of an
agenda vote.
Finally, it updates the mempool policy struct to require a closure for
obtaining the flags so it can remain decoupled from the chain which in
turn allows easier and more robust unit testing of mempool functionality
since it allows a mocks to be used.
This implements a new stake difficulty algorithm along with a voting
agenda for all networks to change and a comprehensive set of tests. It
also implements estimation using the new algorithm for the
estimatestakediff RPC.
The following is an overview of the changes:
- Add new agenda to vote on changing the stake difficulty algorithm to
all networks
- The version on mainnet is version 4
- The version on testnet and simnet is version 5
- Modifies the stake difficulty calculation function to calculate the
difficulty based on the result of the vote
- Modifies the stake difficulty estimation function to calculate the
difficulty based on the result of the vote
- Makes the stake difficulty estimation function concurrent safe
- Calls it directly from the RPC server instead of going through block
manager
- Removes no longer needed code from the block manager
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 4
- New block version on testnet and simnet is version 5
- Add tests for the supply estimation used in the new algorithm
- Add tests for the new algorithm calculations
- Add tests for the estimation based on the new algorithm
This implements a new voting agenda for the testnet and simnet networks
for increasing the maximum block size from 1MiB to 1.25MB.
The following is an overview of the changes:
- Bump the maximum protocol block size limit to 1.25MB
- Bump the protocol version to 4
- Add wire tests for v3 protocol sizes and update tests for latest
- Update all wire values that are defined in terms of the max block size
to respect the protocol version
- Update the MaxSigOpsPerBlock constant to maintain existing value
regardless of the raised max protocol block size
- Decouple the maximum tx size from the block size by creating a chain
parameter for it with the current sizes so it is not a part of the
hard fork vote
- Add definition for new version 4 stake vote along with agenda to vote
on block size to the testnet and simnet networks
- Convert the MaximumBlockSize chain parameter to a slice in order to
hold the different possible sizes
- Adds a new function that returns the maximum block size based upon the
result of the vote
- Change the existing context-free block size check to enforce the max
protocol size and introduce a context-specific check which enforces a
restricted size based upon the network consensus parameters and the
result of the vote
- Set the max allowed size in generated block templates based upon the
network params and result of the vote
- Generate version 4 blocks and reject version 3 blocks after a super
majority has been reached
* Bump block header version to 3.
* Enforce new rules once block version gets to 3.
* Use Voter Versions to control bumping of Stake Version.
* Calculate appropriate stake version based on history and prior
intervals in the mining code.
* Add bunch of UT to validate code.
This is the minimally correct working implementation. This code
desperately needs to memoize the questions that are asked repeatedly.
Fixes#523
Upstream commit 7fac099bee.
Also, the merge commit contains the necessary decred-specific
alterations, adds a new policy parameter for AllowOldVotes to keep with
the spirit of the merged commit, modifies the RawMempoolVerbose to
accept an optional filter type to retain that functionality, and various
other cleanup intended to bring the code bases more in line with one
another.
This reverts to the correct upstream majority version code which
properly detects when certain thresholds of the network have been
updated in order to determine when to start rejecting old version
blocks and enforcing the rules in the new version block.
This splits the mining-specific code that deals with sorting blocks
according to the number of votes available for it in the mempool into
mining.go.
This is being done for a couple of reasons:
1) Mining-specific code does not belong directly in the mempool
2) It is required to be able to split the mempool into a separate
package which an upcoming upstream sync does
In order to accomplish this, a new function named VotesForBlocks has been
exposed on the mempool and the SortParentsByVotes code has been moved to
mining.go and modified to no longer return errors. The reasoning for
this is the same as the recent change to VoteHashesForBlock. That is to
say it is not the responsibility of a sorting function to dictate
caller-specific block eligibility logic.
This renames the GetVoteHashesForBlock on the mempool type to
VoteHashesForBlock so it is named according to Effective Go guidelines
as required by the project's code contribution guidelines and removes
the error return and paths from the function.
The error has been removed since the function should not be assuming how
the caller wishes to use the data. Consequently, now when there are no
votes for a given block hash, the function simply returns a nil slice to
reflect that. The caller can then decide if the lack of vote hashes for
a block is an error or not.
Also, the error path regarding an unset vote hash in the metadata is an
impossible condition because the mempool has full control over the vote
insertion and thus will never insert vote metadata without it.
Upstream commit a7b35d9f9e.
Also, the merge commit contains necessary decred-specific alterations.
It also has the side effect of correcting a couple of min reduction bugs
even though they can't currently ever be hit due to the min reduction
interval flag being disabled for all networks.
Contains the following commits:
- 711f33450c
- b6b1e55d1e
- Reverted because Travis is already at a more recent version
- bd4e64d1d4
Also, the merge commit contains the necessary decred-specific
alterations, converts all other references to sha to hash to keep with
the spirit of the merged commits, and various other cleanup intended to
bring the code bases more in line with one another.
This exports the transaction tree constants from the wire package
instead of from dcrutil. The tree is a fundamental part of a
transaction and therefore its definitions belong in wire alongside the
other constants that are also related to fundamental transaction fields.
Further, the wire package most definitely should not depend on dcrutil
since that would create a cyclic import cycle.
Bump block version and add stake version to wire. Currently
StakeVersions are unused and once the enforcement logic goes in the
versions will bump again.
This fixes#435
Profiling indicated that significant time was being spent validating
coinbase outputs, ensuring that they paid to the development
organization's P2SH tax address. This check was more inefficient than
necessary for a couple of reasons:
* The tax address was always decoded from a string to a dcrutil.Address.
* The actual script being validated was always parsed for addresses to
check if they matched the tax address.
Neither of these are needed. To optimize the algorithm, only the
equality of the output script and script version are checked.
A bug when updating the chain state during forced reorganizations
was fixed. An assertion was added into the mining code so that it
now errors out if the blockchain and chainState of the block
manager are inconsistent.
The blockchain function to fetch the latest block header was
renamed for consistency and now returns a copy of the block header
instead of the pointer itself.
The legacy ticket database, which was GOB serialized and stored on
shut down, has been removed. Ticket state information is now held in
a stake node, which acts as a modularized "black box" to contain all
information about the state of the stake system. Stake nodes are now
a component of the blockchain blockNode struct, and are updated with
them.
Stake nodes, like their internal treap primitives, are immutable
objects that are created with their connect and disconnect node
functions. The blockchain database now stores all information about
the stake state of the best node in the block database. The blockchain
makes the assumption that the stake state of the best node is known at
any given time. If the states of former blocks or sidechains must be
evaluated, this can be achieved by iterating backwards along the
blockchain from the best node, and then connecting stake nodes
iteratively if necessary.
Performance improvements with this new module are dramatic. The long
delays on start up and shut down are removed. Blockchain
synchronization time is improved approximately 5-10x on the mainnet
chain. The state of the database is atomic, so unexpected shut downs
should no longer have the ability to disrupt the chain state.
An upgrade path has been added for version 1 blockchain databases.
Users with this blockchain database will automatically update when
they start their clients.
This does the minimum work necessary to refactor the mempool code into
its own package. The idea is that separating this code into its own
package will greatly improve its testability, allow independent
benchmarking and profiling, and open up some interesting opportunities
for future development related to the memory pool.
There are likely some areas related to policy that could be further
refactored, however it is better to do that in future commits in order
to keep the changeset as small as possible during this refactor.
Overview of the major changes:
- Create the new package
- Move several files into the new package:
- mempool.go -> mempool/mempool.go
- mempoolerror.go -> mempool/error.go
- policy.go -> mempool/policy.go
- policy_test.go -> mempool/policy_test.go
- Update mempool logging to use the new mempool package logger
- Rename mempoolPolicy to Policy (so it's now mempool.Policy)
- Rename mempoolConfig to Config (so it's now mempool.Config)
- Rename mempoolTxDesc to TxDesc (so it's now mempool.TxDesc)
- Rename txMemPool to TxPool (so it's now mempool.TxPool)
- Move defaultBlockPrioritySize to the new package and export it
- Export DefaultMinRelayTxFee from the mempool package
- Export the CalcPriority function from the mempool package
- Introduce a new RawMempoolVerbose function on the TxPool and update
the RPC server to use it
- Update all references to the mempool to use the package.
- Add a skeleton README.md
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
This moves several of the chain constants to the Params struct in the
chaincfg package which is intended for that purpose. This is mostly a
backport of the same modifications made in Decred along with a few
additional things cleaned up.
The following is an overview of the changes:
- Comment all fields in the Params struct definition
- Add locals to BlockChain instance for the calculated values based on
the provided chain params
- Rename the following param fields:
- SubsidyHalvingInterval -> SubsidyReductionInterval
- ResetMinDifficulty -> ReduceMinDifficulty
- Add new Param fields:
- CoinbaseMaturity
- TargetTimePerBlock
- TargetTimespan
- BlocksPerRetarget
- RetargetAdjustmentFactor
- MinDiffReductionTime
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
Contains the following upstream commits:
- ef9c50be57
- eb882f39f8
In addition to merging the fixes in the commits, this also fixes a few
more misspellings that were introduced in the new Decred code.
Contains the following upstream commits:
- 83bcfea271
- 2f6aeacfab
- Cherry-picked because it fixes an issue originally introduced by the
commit being merged
In addition to the normal required changes for syncing, the following
changes have been made in order to facilitate integration into Decred:
- Configure the NewestSha function to the new mempool config as a
closure over the block manager state instead of using the database
func directly
- Add a new NextStakeDifficulty callback to the new mempool config for
obtaining the next stake difficulty from the block manager and
configure it as a closure over the block manager state.
Upstream commit 2b6a9a56e5.
NOTE: This is only merging in the upstream changes, so while they are
useful for Decred as well, this merge commit does fully separate the
mempool as upstream does due to the new functionality provided by
Decred.
