This implements the agenda for voting on the sequence lock fixes as
defined in DCP0004 along with consensus tests and mempool acceptance
tests to ensure its correctness.
It also modifies the mempool to conditionally treat all transactions
with enabled sequence locks as non standard until the vote passes at
which point the will become standard with the modified semantics
enforced.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 6
- New block version on testnet is version 7
- Introduce a convenience function for determining if the vote passed
and is now active
- Enforce modified sequence lock semantics in accordance with the state
of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0004 in accordance with the state of the vote
- Make all transactions with enabled sequence locks non standard until
the agenda vote passes
- Add tests to ensure acceptance and relay behave according to the
aforementioned description
- Add tests for determining if the agenda is active for both mainnet and
testnet
- Add tests to ensure the corrected sequence locks are handled properly
depending on the result of the vote
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
This modifies simnet to include the latest consensus changes that have
been successfully voted into mainnet to date starting from its genesis
block. This means it is no longer necessary to vote those changes in to
simnet first before making use of them.
The changes entail removing the deployment entries for those votes,
updating the mining templates to start from version 5, which matches the
current mainnet version, and modifying the various sections in
blockchain which make decisions accordingly.
The chaincfg and blockchain modules are affected, but since their
version has already been bumped since their last release tags, they are
not bumped again.
this updates the function signatures of minimumMedianTime and
medianAdjustedTime and removes all chainState usage in
rpcserver.go in favour of using BestState.
Port of upstream commit 2274d36 (subset). In preparation of the
removal of chainState.
This modifies the BlockByHash function to return blocks regardless of
whether or not the block is on a side chain and removes the
FetchBlockByHash function since this change makes it redundant.
This is being changed to simplify the API and because there are no
current cases where calls into chain to retrieve a block need to be
limited to the main chain and if the need should ever arise, the caller
can simply make use of the MainChainHasBlock function to verify.
This removes the CheckConnectBlock function in favor of a new function
named CheckConnectBlockTemplate which more accurately reflects its
purpose of testing block template proposals and uses this fact to be
more restrictive about the allowed inputs and to avoid performing the
proof of work check.
In particular, the provided block template must only build from the
current tip or its parent or an error is returned.
All mining code has been updated to call the new function accordingly.
Finally, it also adds a full suite of tests for the new function using
the chaingen framework in order to ensure proper functionality.
This corrects the code to create a new template when first starting up
with aggressive mining enable before votes on the most recent block are
known.
First, the code is expecting the current best block, not its parent, and
second, the coinbase script should match what is created in the normal
path as well.
This finishes separating the mining code from the mempool that was
partially done in commit 9031d85 by extending the TxSource interface to
include the new functionality required by Decred.
The following is an overview of the changes to accomplish this:
- Move the VoteTx struct out of the mempool package in the mining
package and rename it to VoteDesc instead to signify it is a vote
descriptor and be consistent with the naming of TxDesc
- Update mempool to use the new mining.VoteDesc struct
- Rename CheckIfTxsExist to HaveAllTransactions to be more consistent
with HaveTransaction in the TxSource interface as it now lives
alongside it
- Add VoteHashesForBlock, VotesForBlocks, and IsTxTreeKnownInvalid
to the TxSource interface
- Switch the mining code to call the functions on the TxSource interface
instead and remove the reference to the concrete mempool
This replaces the mempool IsTxTreeValid function with
IsTxTreeKnownInvalid instead which more accurately describes its
semantics and purpose.
The actual purpose of this function is related to mempool acceptance and
building block templates for mining where the assumption is that the
previous block's regular transaction tree will be valid and is only
invalid specifically if there are enough known votes that vote against
it.
This change is desirable because the function provides semantics that
are actually opposite to that of the consensus rules where a tree is
only valid if it has a majority of yes votes and thus claiming it's
valid when that is not actually known for certain yet can easily cause
confusion.
The current code creates block templates with coinbase transactions that
have 4 uint64 extra nonces, but only ever uses one of them. Rather than
wasting the extra space, this modifies the code so it only uses a single
uint64 extra nonce.
It should be noted that realistically there isn't even a real need for
an extra nonce in the coinbase at all because there is extra space in
the header specifically for that purpose and miners can't modify the
coinbase via getwork anyways.
However, it is still necessary to insert for the current code in order
to ensure every block template has a unique merkle root since that is
what is used to uniquely identify the block template.
This modifies all files in the main package so that function definitions
are on the a single line as is the style used in the upstream code as
well as throughout its code base. This helps minimize the differences
between them and the upstream code in order to facilitate easier syncs
due to less merge conflicts as a result of superfluous changes.
This moves the priority-related code from the mempool package to the
mining package and also exports a new constant named UnminedHeight which
takes the place of the old unexported mempoolHeight.
Even though the mempool makes use of the priority code to make decisions
about what it will accept, priority really has to do with mining since
it influences which transactions will end up into a block. This change
also has the side effect of being a step towards enabling separation of
the mining code into its own package which, as previously mentioned,
needs access to the priority calculation code as well.
Finally, the mempoolHeight variable was poorly named since what it
really represents is a transaction that has not been mined into a block
yet. Renaming the variable to more accurately reflect its purpose makes
it clear that it belongs in the mining package which also needs the
definition now as well since the priority calculation code relies on it.
This will also benefit an outstanding PR which needs access to the same
value.
This fixes a bug where a transaction would lose reference to other
transactions dependant on it when being considered for inclusion in a
new block template. The issue only occurs when the transaction being
considered triggers a change of priority queue ordering to ordering by
fee. It results in none of the dependant transactions being considered
for inclusion in the new block template.
This prevents the ability for duplicate transactions to be added to the
mining priority heap when a transaction spends multiple outputs from the
same input transaction by converting the dependency tracking to use a
map keyed by the transaction hash instead of a linked list.
This modifies the mining code to request the desired blocks from chain
by hash as opposed to using a separate function to get the top block and
removes the now unused plubming through block manager.
Not only is this faster because it avoids the need to go through block
manager, it is also more robust since it does not rely on the tip of the
chain not having changing for correctness.
This replaces the GetGeneration function which allowed getting the
entire generation (all children stemming from the same parent) of an
arbitrary bock with TipGeneration which only returns the same
information for the tip block.
This is being done because the function is only used for mining purposes
to get the generation of the current tip. The code is simplified by
reducing its scope to its actual purpose as an initial benefit. It also
provides much better optimization opportunities later.
This modifies the exported CheckConnectBlock function to call the
checkBlockSanity and checkBlockContext functions to ensure all
validation rules are enforced from the mining code.
Since both the mining code and tests typically work with unsolved
blocks, this also introduces a new parameter on CheckConnectBlock to
pass behavior flags which allows the caller to skip the proof of work
check.
It also modifies forceHeadReorganization to call checkBlockContext for
the same reason.
Finally, all tests and call sites are updated with the appropriate
flags and various tests are updated for the changes accordingly.
This modifies the SSGenBlockVotedOn function to directly copy the hash
the vote commits to into a statically sized array versus calling the
hash function and thus can remove the potential runtime error.
This is preferred because the code will fail to compile if the size
of chainhash.HashSize is every changed which is desirable because it
would mean the assumptions the code makes would be invalidated.
It also updates all callers in the repository accordingly.
This modifies the BestState struct in the blockchain package to store
hashes directly instead of pointers to them and updates callers to deal
with the API change in the exported BestState struct.
In general, the preferred approach for hashes moving forward is to store
hash values in complex data structures, particularly those that will be
used for cache entries, and accept pointers to hashes in arguments to
functions.
Some of the reasoning behind making this change is:
- It is generally preferred to avoid storing pointers to data in cache
objects since doing so can easily lead to storing interior pointers into
other structs that then can't be GC'd
- Keeping the hash values directly in the structs provides better
cache locality
This modifies the IsSStx, IsSSGen, and IsSSRtx functions to only return
a bool and introduces CheckSStx, CheckSSGen, and CheckSSRtx to return
the actual error as needed by consensus.
This is being done because "is" functions are much nicer to use when
they don't return an error and the callers that use them almost never
care why they aren't of the type, they just want to determine if they
are. In the few cases where the caller does care, they can use of the
new check functions.
While here, also update the comments to call out what the more common
names for the transaction types are and to add comments to the test
functions for consistency.
Finally, it updates all callers in the repo accordingly.
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This implements the agenda for voting on the LN features as defined in
DCP0002 and DCP0003 along with consensus tests to ensure its correctness.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 5
- New block version on testnet is version 6
- Introduce a convenience function for determining if the vote passed
and is now active
- Introduce a new function for getting the consensus script verification
flags and setting the appropriate flags to enforce the new consensus
semantics in accordance with the state of the vote
- Enforce transaction finality checks based on the past median time in
accordance with the state of the vote
- Enforce relative time locks via sequence numbers in accordance with
the state of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0002 in accordance with the state of the vote
- It should be noted that the flag for DCP0003 is always set for
the more strict standardness checks because it is a soft fork while
the DCP0002 enforcement must depend on the result of the vote
because it is a hard fork
- Add tests for determining consensus script verification flags for both
mainnet and testnet
- Add tests for determining if the agenda is active for both mainnet and
testnet
This modifies the way standard verification flags are handled so that it
is possible to selectively enable them based on the result of agenda
votes.
First, it moves the StandardVerifyFlags constant from the txscript
package to the mempool/policy code and rename it to
BaseStandardVerifyFlags. As the TODO in the comment of the moved
constant indicated, these flags are policy related and thus really
belong in policy. Ideally there would be a completely separate policy
package, but since the policy code currently lives in mempool/policy.go,
the constant has been moved there.
Next, it introduces a new function named standardScriptVerifyFlags,
which accepts the chain as an argument and, for now, just returns the
BaseStandardVerifyFlags along with a nil error. This will allow
additional flags to be selectively enabled depending on the result of an
agenda vote.
Finally, it updates the mempool policy struct to require a closure for
obtaining the flags so it can remain decoupled from the chain which in
turn allows easier and more robust unit testing of mempool functionality
since it allows a mocks to be used.
This implements a new stake difficulty algorithm along with a voting
agenda for all networks to change and a comprehensive set of tests. It
also implements estimation using the new algorithm for the
estimatestakediff RPC.
The following is an overview of the changes:
- Add new agenda to vote on changing the stake difficulty algorithm to
all networks
- The version on mainnet is version 4
- The version on testnet and simnet is version 5
- Modifies the stake difficulty calculation function to calculate the
difficulty based on the result of the vote
- Modifies the stake difficulty estimation function to calculate the
difficulty based on the result of the vote
- Makes the stake difficulty estimation function concurrent safe
- Calls it directly from the RPC server instead of going through block
manager
- Removes no longer needed code from the block manager
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 4
- New block version on testnet and simnet is version 5
- Add tests for the supply estimation used in the new algorithm
- Add tests for the new algorithm calculations
- Add tests for the estimation based on the new algorithm
This implements a new voting agenda for the testnet and simnet networks
for increasing the maximum block size from 1MiB to 1.25MB.
The following is an overview of the changes:
- Bump the maximum protocol block size limit to 1.25MB
- Bump the protocol version to 4
- Add wire tests for v3 protocol sizes and update tests for latest
- Update all wire values that are defined in terms of the max block size
to respect the protocol version
- Update the MaxSigOpsPerBlock constant to maintain existing value
regardless of the raised max protocol block size
- Decouple the maximum tx size from the block size by creating a chain
parameter for it with the current sizes so it is not a part of the
hard fork vote
- Add definition for new version 4 stake vote along with agenda to vote
on block size to the testnet and simnet networks
- Convert the MaximumBlockSize chain parameter to a slice in order to
hold the different possible sizes
- Adds a new function that returns the maximum block size based upon the
result of the vote
- Change the existing context-free block size check to enforce the max
protocol size and introduce a context-specific check which enforces a
restricted size based upon the network consensus parameters and the
result of the vote
- Set the max allowed size in generated block templates based upon the
network params and result of the vote
- Generate version 4 blocks and reject version 3 blocks after a super
majority has been reached
* Bump block header version to 3.
* Enforce new rules once block version gets to 3.
* Use Voter Versions to control bumping of Stake Version.
* Calculate appropriate stake version based on history and prior
intervals in the mining code.
* Add bunch of UT to validate code.
This is the minimally correct working implementation. This code
desperately needs to memoize the questions that are asked repeatedly.
Fixes#523
Upstream commit 7fac099bee.
Also, the merge commit contains the necessary decred-specific
alterations, adds a new policy parameter for AllowOldVotes to keep with
the spirit of the merged commit, modifies the RawMempoolVerbose to
accept an optional filter type to retain that functionality, and various
other cleanup intended to bring the code bases more in line with one
another.
This reverts to the correct upstream majority version code which
properly detects when certain thresholds of the network have been
updated in order to determine when to start rejecting old version
blocks and enforcing the rules in the new version block.
This splits the mining-specific code that deals with sorting blocks
according to the number of votes available for it in the mempool into
mining.go.
This is being done for a couple of reasons:
1) Mining-specific code does not belong directly in the mempool
2) It is required to be able to split the mempool into a separate
package which an upcoming upstream sync does
In order to accomplish this, a new function named VotesForBlocks has been
exposed on the mempool and the SortParentsByVotes code has been moved to
mining.go and modified to no longer return errors. The reasoning for
this is the same as the recent change to VoteHashesForBlock. That is to
say it is not the responsibility of a sorting function to dictate
caller-specific block eligibility logic.
This renames the GetVoteHashesForBlock on the mempool type to
VoteHashesForBlock so it is named according to Effective Go guidelines
as required by the project's code contribution guidelines and removes
the error return and paths from the function.
The error has been removed since the function should not be assuming how
the caller wishes to use the data. Consequently, now when there are no
votes for a given block hash, the function simply returns a nil slice to
reflect that. The caller can then decide if the lack of vote hashes for
a block is an error or not.
Also, the error path regarding an unset vote hash in the metadata is an
impossible condition because the mempool has full control over the vote
insertion and thus will never insert vote metadata without it.
Upstream commit a7b35d9f9e.
Also, the merge commit contains necessary decred-specific alterations.
It also has the side effect of correcting a couple of min reduction bugs
even though they can't currently ever be hit due to the min reduction
interval flag being disabled for all networks.
Contains the following commits:
- 711f33450c
- b6b1e55d1e
- Reverted because Travis is already at a more recent version
- bd4e64d1d4
Also, the merge commit contains the necessary decred-specific
alterations, converts all other references to sha to hash to keep with
the spirit of the merged commits, and various other cleanup intended to
bring the code bases more in line with one another.
This exports the transaction tree constants from the wire package
instead of from dcrutil. The tree is a fundamental part of a
transaction and therefore its definitions belong in wire alongside the
other constants that are also related to fundamental transaction fields.
Further, the wire package most definitely should not depend on dcrutil
since that would create a cyclic import cycle.
Bump block version and add stake version to wire. Currently
StakeVersions are unused and once the enforcement logic goes in the
versions will bump again.
This fixes#435
