BgBlkTmplGenerator represents the background process that
generates block templates and notifies all subscribed clients
on template regeneration. It generates new templates based
on mempool activity for vote and non-vote transactions and
the time elapsed since last template regeneration.
This also adds a template pool to the background block
generator for recreating submitted blocks.
This adds the go 1.11 directive to all of the modules in order to
clearly mark they build and work with that version. Go 1.12 modified
the tools such that tidy will automatically add the new version to
modules that do not already have a directive and that would prevent
builds on Go 1.11 through Go 1.11.3 which is not desirable.
Currently, there is a tight coupling between the rpcserver and the
mempool by way of the RawMempoolVerbose function returning a
dcrjson.GetRawMempoolVerboseResult.
In order to remove the tight coupling, this introduces a new type and
function to the mempool module named VerboseTxDesc and VerboseTxDescs,
respectively, which provide all of the details necessary for the
rpcserver to be able to properly generate the dcrjson response
directly.
Also, it removes the old RawMempoolVerbose function and related
dependency imports on dcrjson since it is no longer required as a part
of moving to mempool/v2.
Note that since this constitutes a major break to the API, the mepool
module version will need to be bumped to version 2.
This removes the deprecated MinHighPriority constant as part of moving
to mempool/v2.
Note that since this constitutes a major break to the API, the mepool
module version will need to be bumped to version 2.
This modifies all of the modules, with the exception of the root module,
to remove all replacement directives from their go.mod files and update
the requirements and module sums accordingly.
While it is nice to be able to build and test directly from each module
directory and have it pull in the latest untagged changes when
developing, having all of the overrides in each module makes it
infeasible to use the module tools to help maintain the modules and thus
makes it quite difficult to ensure they are all independently accurate
for external consumers.
By maintaining all of the overrides in the root module and invoking all
builds and tests from it, the overrides will apply to ensure the latest
code is being built and tested.
This also modifies the tests script used with in CI to run all of the
tests from the root module accordingly.
Currently, the memory pool policy is to allow votes which double spend
the same ticket (up to a maximum amount) since it is possible that the
same ticket is selected in competing branches and thus the double spend
must be allowed and will resolve itself depending on which chain is
ultimately extended.
A side effect of that is that it also currently accepts votes with
different hashes but otherwise both spend the same ticket and vote on
the same block. This can happen when there are multiple wallets casting
the vote with different settings such as when one has been upgraded to a
new stake version while the others have not or an improperly configured
wallet where the vote choices have been set on one, but not the others.
Consequently, this modifies the policy to explicitly reject these
duplicate votes while still allowing votes on competing branches.
It also adds tests to ensure the new functionality works as expected.
This modifies the check which limits the maximum allowed votes double
spending the same ticket to exactly the specified constant instead of
one more.
It also adds tests for the max vote double spend handling to both ensure
they are accepted up until the max, rejected thereafter, and that
removing votes allows them to be accepted again.
This modifies CreateVote in the mempool test harness to accept munge
functions similar to how CreateSignedTx works. This allows the test
code to more easily mutate the transactions as desired prior to them
being signed which provides greater flexibility.
While here, it also sets the block that is being voted on to the current
best chain tip of the harness. While there is not currently anything in
the tests that relies on that behavior, it will allow future tests to
more easily create votes which involve the block being voted on.
This implements the agenda for voting on the sequence lock fixes as
defined in DCP0004 along with consensus tests and mempool acceptance
tests to ensure its correctness.
It also modifies the mempool to conditionally treat all transactions
with enabled sequence locks as non standard until the vote passes at
which point the will become standard with the modified semantics
enforced.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 6
- New block version on testnet is version 7
- Introduce a convenience function for determining if the vote passed
and is now active
- Enforce modified sequence lock semantics in accordance with the state
of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0004 in accordance with the state of the vote
- Make all transactions with enabled sequence locks non standard until
the agenda vote passes
- Add tests to ensure acceptance and relay behave according to the
aforementioned description
- Add tests for determining if the agenda is active for both mainnet and
testnet
- Add tests to ensure the corrected sequence locks are handled properly
depending on the result of the vote
This adds infrastructure to the mempool test harness to allow the faked
chain to handle sequence lock calculation for use in upcoming sequence
lock tests.
In particular, it modifies the existing CalcSequenceLock function on the
faked chain to perform the full sequence lock calculation versus simply
returning a lock that is always valid.
Since by-time sequence lock calculation in CalcSequenceLock depends on
the ability to calculate the median time for the block before the block
that contains the input being spent and the test harness only provides
an incomplete faked chain, this introduces a map which tracks the median
times per utxo and provides two new functions named FakeUxtoMedianTime
and AddFakeUtxoMedianTime which the tests can make use of to properly
fake the necessary data to perform the calculations.
Finally, since sequence locks require v2 transactions, the pool harness
policy now allows a maximum transaction version of 2.
This modifies CreateSignedTx in the mempool test harness to accept munge
functions similar to how NextBlock in chaingen works and updates the
callers accordingly. This allows the test code to more easily mutate
the transactions as desired prior to them being signed which provides
greater flexibility.
While here, it also sets the input amount to the proper value in the
generated transaction. While there is not currently anything in the
tests that relies on that value being accurate, it is more desirable to
create transactions in the test code which are fully valid in case they
end up being used where it does matter in future tests.
The changes to reverse the utxoset semantics also had the side effect of
correcting the behavior for sequence locks when inside of a block to the
expected behavior instead of the actual current consensus behavior.
In order to avoid that, this modifies the blockchain consensus rules to
properly preserve the old incorrect behavior of sequence locks within
blocks and also updates the mempool to to reject transactions
accordingly thus providing parity between the two.
The behavior needs to be corrected, but since it constitutes a change to
the consensus rules, a vote will be necessary.
This bumps the various module versions as follows:
- github.com/decred/dcrd/dcrec/secp256k1@v1.0.1
- github.com/decred/dcrd/dcrjson@v1.1.0
- github.com/decred/dcrd/database@v1.0.3
- github.com/decred/dcrd/blockchain/stake@v1.1.0
- github.com/decred/dcrd/mining@v1.1.0
- github.com/decred/dcrd/certgen@v1.0.2
- github.com/decred/dcrd/connmgr@v1.0.2
- github.com/decred/dcrd/mempool@v1.1.0
In addition, it serves as a base for tagging releases of the following
module versions that have previous been bumped since the last release,
but not yet tagged:
- github.com/decred/dcrd/wire@v1.2.0
- github.com/decred/dcrd/chaincfg@v1.2.0
- github.com/decred/dcrd/dcrutil@v1.2.0
- github.com/decred/dcrd/txscript@v1.0.2
- github.com/decred/dcrd/blockchain@v1.1.0
- github.com/decred/dcrd/hdkeychain@v1.1.1
- github.com/decred/dcrd/peer@v1.1.0
- github.com/decred/dcrd/rpcclient@v1.1.0
Finally, it updates all of the dependencies for every module accordingly,
adds a few missing overrides for transitive dependencies, and tidies up
some of the go module sum files.
This commit performs the necessary modifications to hook the fee
estimation facility added by the previous commit into the node software.
The block manager is modified to notify the estimator of all new blocks,
such that their transactions can be accounted for, and the mempool is
modified to relay transactions entering and leaving it, so that those
transactions can be tracked.
The results of the estimator can be queried by issuing estimatesmartfee
rpc commands to the node.
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
This modifies the majority of the tests that make use of chain
parameters and the RPC tests to use the resurrected regression test
network.
It also bumps the affected module versions as follows:
- github.com/decred/dcrd/txscript@v1.0.2
- github.com/decred/dcrd/blockchain/stake@v1.0.3
- github.com/decred/dcrd/mempool@v1.0.2
The blockchain and dcrutil modules are also affected, but since their
version has already been bumped since their last release tags, they are
not bumped again.
This removes a redundant check for validating that votes and revocations
are not in a block before they are allowed from CheckTransactionInputs since
it has nothing to do with validation of transaction inputs and it's an
impossible condition to hit in the block validation path because the
same check is done much earlier in the validation process.
It should also be noted that the checks this removes are also more
permissive than they need to be because the first vote is not allowed
until stake validation height, which is after stake enabled height.
There is no effect on consensus because, as previously mentioned, the
more restrictive checks earlier in the process prevent these checks from
ever being exercised.
However, the mempool previously relied on this duplicate check to reject
early votes and revocations, so this adds the checks, with the corrected
heights, directly to the mempool early in the process where they more
naturally belong. It is more efficient to reject the transactions in
that scenario earlier before doing more work as well.
This does some minor cleanup of the code in the maybeAcceptTransaction
function to use some local bools with more easily readable names for
dealing with stake transactions.
It also improves the readability of the double spending exceptions
section and avoids some additional if statements.
This removes the standardness check to reject transactions with a lock
time greater than a maxint32 because the requirement for the check was
due to legacy Bitcoin reasons that have never applied to Decred.
This improves trace logs related to transaction removal from the
mempool. During some debugging, they were found to be slightly
misleading in their current form.
This bumps the various module versions as follows:
- github.com/decred/dcrd/addrmgr@v1.0.2
- github.com/decred/dcrd/wire@v1.1.0
- github.com/decred/dcrd/chaincfg@v1.1.1
- github.com/decred/dcrd/connmgr@v1.0.1
- github.com/decred/dcrd/dcrutil@v1.1.1
- github.com/decred/dcrd/database@v1.0.1
- github.com/decred/dcrd/hdkeychain@v1.1.0
- github.com/decred/dcrd/txscript@v1.0.1
- github.com/decred/dcrd/blockchain/stake@v1.0.1
- github.com/decred/dcrd/gcs@v1.0.1
- github.com/decred/dcrd/blockchain@v1.0.1
- github.com/decred/dcrd/mining@v1.0.1
- github.com/decred/dcrd/mempool@v1.0.1
- github.com/decred/dcrd/peer@v1.0.1
- github.com/decred/dcrd/rpcclient@v1.0.1
It also updates all of the dependencies for every module accordingly and
adds a few missing overrides for transitive dependencies.
This modifies the test code for several unit tests to prefer using
another network, such as mainnet or simnet, when not specifically
testing something that requires testnet.
In the cases where where testnet is required, it also now prefers a
local reference over directly accessing the testnet parameters directly
from the chaincfg package.
This is being done because the test network undergoes periodic resets
and by reducing the number of direct references, it simplifies the
process of resetting it.
This further refines the modules to add the following new modules
instead of depending on the entire dcrd module:
- github.com/decred/dcrd/dcrjson@v1.0.0
- github.com/decred/dcrd/blockchain@v1.0.0
- github.com/decred/dcrd/blockchain/stake@v1.0.0
- github.com/decred/dcrd/gcs@v1.0.0
- github.com/decred/dcrd/mining@v1.0.0
- github.com/decred/dcrd/mempool@v1.0.0
- github.com/decred/dcrd/peer@v1.0.0
- github.com/decred/dcrd/rpcclient@v1.0.0
Also, it ensures modules that rely on other modules within the repo are
provided replacements to the latest repo code to ensure builds against
master and continuous integration use the latest code.
- github.com/decred/dcrd/addrmgr
- github.com/decred/dcrd/blockchain
- github.com/decred/dcrd/blockchain/stake
- github.com/decred/dcrd/chaincfg
- github.com/decred/dcrd/connmgr
- github.com/decred/dcrd/database
- github.com/decred/dcrd/dcrec/secp256k1
- github.com/decred/dcrd/dcrjson
- github.com/decred/dcrd/dcrutil
- github.com/decred/dcrd/gcs
- github.com/decred/dcrd/hdkeychain
- github.com/decred/dcrd/mempool
- github.com/decred/dcrd/mining
- github.com/decred/dcrd/peer
- github.com/decred/dcrd/rpcclient
- github.com/decred/dcrd/txscript
- github.com/decred/dcrd/wire
This modifies the way orphan removal and processing is done to more
aggressively remove orphans that can no longer be valid due to other
transactions being added or removed from the primary transaction pool.
The net effect of these changes is that orphan pool will typically be
much smaller which greatly improves its effectiveness. Previously, it
would typically quickly reach the max allowed worst-case usage and
effectively stay there forever.
The following is a summary of the changes:
- Modify the map that tracks which orphans redeem a given transaction to
instead track by the specific outpoints that are redeemed
- Modify the various orphan removal and processing functions to accept
the full transaction rather than just its hash
- Introduce a new flag on removeOrphans which specifies whether or not
to remove the transactions that redeem the orphan being removed as
well which is necessary since only some paths require it
- Add a new function named removeOrphanDoubleSpends that is invoked
whenever a transaction is added to the main pool and thus the outputs
they spent become concrete spends
- Introduce a new flag on maybeAcceptTransaction which specifies whether
or not duplicate orphans should be rejected since only some paths
require it
- Modify processOrphans as follows:
- Make use of the modified map
- Use newly available flags and logic work more strictly work with tx
chains
- Recursively remove any orphans that also redeem any outputs redeemed
by the accepted transactions
- Several new tests to ensure proper functionality
- Removing an orphan that doesn't exist is removed both when there is
another orphan that redeems it and when there is not
- Removing orphans works properly with orphan chains per the new
remove redeemers flag
- Removal of multi-input orphans that double spend an output when a
concrete redeemer enters the transaction pool
Ticket purchase, vote and revocation orphan tests have also been added.
This modifies the mempool to handle pruning of expired transactions in a
self-contained way so that the caller is not responsible for worrying
about the specific semantics of height and expiration interplay and
updates all callers in the repository accordingly.
It also adds a test which ensures the behavior is correct by creating a
series of transactions with expirations, adding them just before the
point at which they will expire, and then advancing the chain so that
each transaction expires and thus should be pruned.
This removes a bunch of build modules that rely on a version of dcrd
that doesn't exist and updates the root module so dcrd can be built with
the upcoming go1.11 release.
This removes the ScriptVerifyMinimalData flag from the txscript package,
changes the default semantics to always enforce its behavior, and
updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_tests.json
specifically dealt with ensuring equivalency of different ways to encode
the same numbers when the ScriptVerifyMinimalData flag is not set.
Therefore, they are no longer necessary.
A few tests which dealt with equivalency that did not already have
expected failing counterparts were converted to expected failure.
Also, several of the tests which dealt with ensuring the specific
encoding of numeric opcodes is being used have been converted to use
hashes since the minimal data requirements specifically prevent
alternate ways of pushing the same encoding which is necessary for
directly checking equality of the raw bytes.
Finally, the MINIMALDATA indicator to enable the flag in the test data
has been retained for now in order to isolate the logic changes as much
as possible.
This removes the ScriptVerifyDERSignatures flag from the txscript
package, changes the default semantics to always enforce its behavior
and updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_tests.json
specifically dealt with ensuring non-DER-compliant signatures were
handled properly when the ScriptVerifyDERSignatures flag was not set.
Therefore, they are no longer necessary.
Finally, the DERSIG indicator to enable the flag in the test data has
been retained for now in order to keep the logic changes separate.
This removes the ScriptBip16 flag from the txscript package, changes the
default semantics to always enforce its behavior, and updates all
callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so there is
no need to require a flag to conditionally toggle it.
Also, since it is no longer possible to invoke the script engine without
the flag with the clean stack flag, it removes the now unused
ErrInvalidFlags error and associated tests.
It should be noted that the test removed from script_tests.json
specifically dealt with ensuring a signature script that contained
non-data-pushing opcodes was successful when neither the ScriptBip16 or
ScriptVerifySigPushOnly flags were set. Therefore, it is no longer
necessary.
Finally, the P2SH indicator to enable the flag in the test data has been
retained for now in order to keep the logic changes separate.
This removes the ScriptVerifyStrictEncoding flag from the txscript
package, changes the default semantics to always enforce its behavior
and updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_valid.json
specifically dealt with ensuring signatures not compliant with DER
encoding did not cause execution to halt early on invalid signatures
when neither of the ScriptVerifyStrictEncoding or
ScriptVerifyDERSignatures flags were set. Therefore, they are no longer
necessary.
For nearly the same reason, the tx test related to the empty pubkey
tx_valid.json was moved to tx_invalid.json. In particular, an empty
pubkey without ScriptVerifyStrictEncoding simply failed the signature
check and continued execution, while the same condition with the flag
halts execution. Thus, without the flag the final NOT in the script
would allow the script to succeed, while it does not under the strict
encoding rules.
Finally, the STRICTENC indicator to enable the flag in the test data has
been retained for now in order to keep the logic changes separate.
This removes the ScriptVerifyLowS flag from the txscript package,
changes the default semantics to always enforce its behavior and updates
all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
This lowers the default minimum relay fee to 0.0001 DCR/Kb from its
previous value of 0.001 DCR/Kb and increases the high fee multiplier to
keep the same default high fee threshold of 1 DCR/kB.
In order to keep the comments accurate, it also updates the comments in
the isDust function and the sample config file to match the new default
relay fee as it's nice to have the most recent values as a reference in
the comments.
Finally, it updates the tests for the expected new values as a result of
the reduced default relay fee and adds add a couple of dust tests just
below and above the dust point for the new reduced default relay fee.
It should be noted that this is only a default node policy change and as
such does not affect the consensus rules in any way.
This adds module support for the versioned go toolchain. In particular,
the following packages are defined as modules:
* addrmgr
* blockchain
* certgen
* chaincfg
* connmgr
* database
* dcrjson
* dcrutil
* gcs
* hdkeychain
* mempool
* mining
* peer
* rpcclient
* txscript
* wire
It does not update the travis build environment or README since it is
experimental at this point.
This modifies all packages within the repository to the use the Decred
fork of btclog which has been renamed to slog and updates the dependency
files accordingly.
This PR replaces chainec function calls within the mempool packagewith the underlining secp256k1 functions. This is in preparation forremoving the chainec package.
This modifies the pruning code to make use of the newly available
function in blockchain to test if a transaction is expired.
While here, it makes the comments on the functions more consistent with
others and moves the unexported definition before the exported function
from which it is used.
