This bumps the various module versions as follows:
- github.com/decred/dcrd/addrmgr@v1.0.2
- github.com/decred/dcrd/wire@v1.1.0
- github.com/decred/dcrd/chaincfg@v1.1.1
- github.com/decred/dcrd/connmgr@v1.0.1
- github.com/decred/dcrd/dcrutil@v1.1.1
- github.com/decred/dcrd/database@v1.0.1
- github.com/decred/dcrd/hdkeychain@v1.1.0
- github.com/decred/dcrd/txscript@v1.0.1
- github.com/decred/dcrd/blockchain/stake@v1.0.1
- github.com/decred/dcrd/gcs@v1.0.1
- github.com/decred/dcrd/blockchain@v1.0.1
- github.com/decred/dcrd/mining@v1.0.1
- github.com/decred/dcrd/mempool@v1.0.1
- github.com/decred/dcrd/peer@v1.0.1
- github.com/decred/dcrd/rpcclient@v1.0.1
It also updates all of the dependencies for every module accordingly and
adds a few missing overrides for transitive dependencies.
This modifies the test code for several unit tests to prefer using
another network, such as mainnet or simnet, when not specifically
testing something that requires testnet.
In the cases where where testnet is required, it also now prefers a
local reference over directly accessing the testnet parameters directly
from the chaincfg package.
This is being done because the test network undergoes periodic resets
and by reducing the number of direct references, it simplifies the
process of resetting it.
This further refines the modules to add the following new modules
instead of depending on the entire dcrd module:
- github.com/decred/dcrd/dcrjson@v1.0.0
- github.com/decred/dcrd/blockchain@v1.0.0
- github.com/decred/dcrd/blockchain/stake@v1.0.0
- github.com/decred/dcrd/gcs@v1.0.0
- github.com/decred/dcrd/mining@v1.0.0
- github.com/decred/dcrd/mempool@v1.0.0
- github.com/decred/dcrd/peer@v1.0.0
- github.com/decred/dcrd/rpcclient@v1.0.0
Also, it ensures modules that rely on other modules within the repo are
provided replacements to the latest repo code to ensure builds against
master and continuous integration use the latest code.
- github.com/decred/dcrd/addrmgr
- github.com/decred/dcrd/blockchain
- github.com/decred/dcrd/blockchain/stake
- github.com/decred/dcrd/chaincfg
- github.com/decred/dcrd/connmgr
- github.com/decred/dcrd/database
- github.com/decred/dcrd/dcrec/secp256k1
- github.com/decred/dcrd/dcrjson
- github.com/decred/dcrd/dcrutil
- github.com/decred/dcrd/gcs
- github.com/decred/dcrd/hdkeychain
- github.com/decred/dcrd/mempool
- github.com/decred/dcrd/mining
- github.com/decred/dcrd/peer
- github.com/decred/dcrd/rpcclient
- github.com/decred/dcrd/txscript
- github.com/decred/dcrd/wire
This modifies the way orphan removal and processing is done to more
aggressively remove orphans that can no longer be valid due to other
transactions being added or removed from the primary transaction pool.
The net effect of these changes is that orphan pool will typically be
much smaller which greatly improves its effectiveness. Previously, it
would typically quickly reach the max allowed worst-case usage and
effectively stay there forever.
The following is a summary of the changes:
- Modify the map that tracks which orphans redeem a given transaction to
instead track by the specific outpoints that are redeemed
- Modify the various orphan removal and processing functions to accept
the full transaction rather than just its hash
- Introduce a new flag on removeOrphans which specifies whether or not
to remove the transactions that redeem the orphan being removed as
well which is necessary since only some paths require it
- Add a new function named removeOrphanDoubleSpends that is invoked
whenever a transaction is added to the main pool and thus the outputs
they spent become concrete spends
- Introduce a new flag on maybeAcceptTransaction which specifies whether
or not duplicate orphans should be rejected since only some paths
require it
- Modify processOrphans as follows:
- Make use of the modified map
- Use newly available flags and logic work more strictly work with tx
chains
- Recursively remove any orphans that also redeem any outputs redeemed
by the accepted transactions
- Several new tests to ensure proper functionality
- Removing an orphan that doesn't exist is removed both when there is
another orphan that redeems it and when there is not
- Removing orphans works properly with orphan chains per the new
remove redeemers flag
- Removal of multi-input orphans that double spend an output when a
concrete redeemer enters the transaction pool
Ticket purchase, vote and revocation orphan tests have also been added.
This modifies the mempool to handle pruning of expired transactions in a
self-contained way so that the caller is not responsible for worrying
about the specific semantics of height and expiration interplay and
updates all callers in the repository accordingly.
It also adds a test which ensures the behavior is correct by creating a
series of transactions with expirations, adding them just before the
point at which they will expire, and then advancing the chain so that
each transaction expires and thus should be pruned.
This removes a bunch of build modules that rely on a version of dcrd
that doesn't exist and updates the root module so dcrd can be built with
the upcoming go1.11 release.
This removes the ScriptVerifyMinimalData flag from the txscript package,
changes the default semantics to always enforce its behavior, and
updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_tests.json
specifically dealt with ensuring equivalency of different ways to encode
the same numbers when the ScriptVerifyMinimalData flag is not set.
Therefore, they are no longer necessary.
A few tests which dealt with equivalency that did not already have
expected failing counterparts were converted to expected failure.
Also, several of the tests which dealt with ensuring the specific
encoding of numeric opcodes is being used have been converted to use
hashes since the minimal data requirements specifically prevent
alternate ways of pushing the same encoding which is necessary for
directly checking equality of the raw bytes.
Finally, the MINIMALDATA indicator to enable the flag in the test data
has been retained for now in order to isolate the logic changes as much
as possible.
This removes the ScriptVerifyDERSignatures flag from the txscript
package, changes the default semantics to always enforce its behavior
and updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_tests.json
specifically dealt with ensuring non-DER-compliant signatures were
handled properly when the ScriptVerifyDERSignatures flag was not set.
Therefore, they are no longer necessary.
Finally, the DERSIG indicator to enable the flag in the test data has
been retained for now in order to keep the logic changes separate.
This removes the ScriptBip16 flag from the txscript package, changes the
default semantics to always enforce its behavior, and updates all
callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so there is
no need to require a flag to conditionally toggle it.
Also, since it is no longer possible to invoke the script engine without
the flag with the clean stack flag, it removes the now unused
ErrInvalidFlags error and associated tests.
It should be noted that the test removed from script_tests.json
specifically dealt with ensuring a signature script that contained
non-data-pushing opcodes was successful when neither the ScriptBip16 or
ScriptVerifySigPushOnly flags were set. Therefore, it is no longer
necessary.
Finally, the P2SH indicator to enable the flag in the test data has been
retained for now in order to keep the logic changes separate.
This removes the ScriptVerifyStrictEncoding flag from the txscript
package, changes the default semantics to always enforce its behavior
and updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_valid.json
specifically dealt with ensuring signatures not compliant with DER
encoding did not cause execution to halt early on invalid signatures
when neither of the ScriptVerifyStrictEncoding or
ScriptVerifyDERSignatures flags were set. Therefore, they are no longer
necessary.
For nearly the same reason, the tx test related to the empty pubkey
tx_valid.json was moved to tx_invalid.json. In particular, an empty
pubkey without ScriptVerifyStrictEncoding simply failed the signature
check and continued execution, while the same condition with the flag
halts execution. Thus, without the flag the final NOT in the script
would allow the script to succeed, while it does not under the strict
encoding rules.
Finally, the STRICTENC indicator to enable the flag in the test data has
been retained for now in order to keep the logic changes separate.
This removes the ScriptVerifyLowS flag from the txscript package,
changes the default semantics to always enforce its behavior and updates
all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
This lowers the default minimum relay fee to 0.0001 DCR/Kb from its
previous value of 0.001 DCR/Kb and increases the high fee multiplier to
keep the same default high fee threshold of 1 DCR/kB.
In order to keep the comments accurate, it also updates the comments in
the isDust function and the sample config file to match the new default
relay fee as it's nice to have the most recent values as a reference in
the comments.
Finally, it updates the tests for the expected new values as a result of
the reduced default relay fee and adds add a couple of dust tests just
below and above the dust point for the new reduced default relay fee.
It should be noted that this is only a default node policy change and as
such does not affect the consensus rules in any way.
This adds module support for the versioned go toolchain. In particular,
the following packages are defined as modules:
* addrmgr
* blockchain
* certgen
* chaincfg
* connmgr
* database
* dcrjson
* dcrutil
* gcs
* hdkeychain
* mempool
* mining
* peer
* rpcclient
* txscript
* wire
It does not update the travis build environment or README since it is
experimental at this point.
This modifies all packages within the repository to the use the Decred
fork of btclog which has been renamed to slog and updates the dependency
files accordingly.
This PR replaces chainec function calls within the mempool packagewith the underlining secp256k1 functions. This is in preparation forremoving the chainec package.
This modifies the pruning code to make use of the newly available
function in blockchain to test if a transaction is expired.
While here, it makes the comments on the functions more consistent with
others and moves the unexported definition before the exported function
from which it is used.
The CheckTransactionInputs, as it name suggested, is only intended to
perform validation that requires the inputs to transactions.
Consequently, this moves the check for validating the transactions in
the block are not expired into the checkBlockContext function where it
more naturally belongs since it is only dependent on the block and its
contextual position within the chain.
Since this check is also needed by the mempool, it refactors the check
into separate functions named IsExpired and IsExpiredTx, which work with
both util transactions and raw wire transactions, respectively, and
updates mempool to perform the necessary check by making use of them.
This introduces a new pool membership test function to the mempool
testing infrastructure and refactors the tests to make use of it.
It is useful since it is common logic that is not only needed in the
existing tests, but will be needed by most mempool-related tests.
This finishes separating the mining code from the mempool that was
partially done in commit 9031d85 by extending the TxSource interface to
include the new functionality required by Decred.
The following is an overview of the changes to accomplish this:
- Move the VoteTx struct out of the mempool package in the mining
package and rename it to VoteDesc instead to signify it is a vote
descriptor and be consistent with the naming of TxDesc
- Update mempool to use the new mining.VoteDesc struct
- Rename CheckIfTxsExist to HaveAllTransactions to be more consistent
with HaveTransaction in the TxSource interface as it now lives
alongside it
- Add VoteHashesForBlock, VotesForBlocks, and IsTxTreeKnownInvalid
to the TxSource interface
- Switch the mining code to call the functions on the TxSource interface
instead and remove the reference to the concrete mempool
This replaces the mempool IsTxTreeValid function with
IsTxTreeKnownInvalid instead which more accurately describes its
semantics and purpose.
The actual purpose of this function is related to mempool acceptance and
building block templates for mining where the assumption is that the
previous block's regular transaction tree will be valid and is only
invalid specifically if there are enough known votes that vote against
it.
This change is desirable because the function provides semantics that
are actually opposite to that of the consensus rules where a tree is
only valid if it has a majority of yes votes and thus claiming it's
valid when that is not actually known for certain yet can easily cause
confusion.
This optimizes the way in which the mempool oprhan map is limited in the
same way the server block manager maps were previously optimized.
Previously the code would read a cryptographically random value large
enough to construct a hash, find the first entry larger than that value,
and evict it.
That approach is quite inefficient and could easily become a
bottleneck when processing transactions due to the need to read from a
source such as /dev/urandom and all of the subsequent hash comparisons.
Luckily, strong cryptographic randomness is not needed here. The primary
intent of limiting the maps is to control memory usage with a secondary
concern of making it difficult for adversaries to force eviction of
specific entries.
Consequently, this changes the code to make use of the pseudorandom
iteration order of Go's maps along with the preimage resistance of the
hashing function to provide the desired functionality. It has
previously been discussed that the specific pseudorandom iteration order
is not guaranteed by the Go spec even though in practice that is how it
is implemented. This is not a concern however because even if the
specific compiler doesn't implement that, the preimage resistance of the
hashing function alone is enough.
The following is a before and after comparison of the function for both
speed and memory allocations:
benchmark old ns/op new ns/op delta
----------------------------------------------------------------
BenchmarkLimitNumOrphans 3727 243 -93.48%
benchmark old allocs new allocs delta
-----------------------------------------------------------------
BenchmarkLimitNumOrphans 4 0 -100.00%
This moves the priority-related code from the mempool package to the
mining package and also exports a new constant named UnminedHeight which
takes the place of the old unexported mempoolHeight.
Even though the mempool makes use of the priority code to make decisions
about what it will accept, priority really has to do with mining since
it influences which transactions will end up into a block. This change
also has the side effect of being a step towards enabling separation of
the mining code into its own package which, as previously mentioned,
needs access to the priority calculation code as well.
Finally, the mempoolHeight variable was poorly named since what it
really represents is a transaction that has not been mined into a block
yet. Renaming the variable to more accurately reflect its purpose makes
it clear that it belongs in the mining package which also needs the
definition now as well since the priority calculation code relies on it.
This will also benefit an outstanding PR which needs access to the same
value.
This replaces the ErrDoubleSpend and ErrMissingTx error codes with a
single error code named ErrMissingTxOut and updates the relevant errors
and expected test results accordingly.
Once upon a time, the code relied on a transaction index, so it was able
to definitively differentiate between a transaction output that
legitimately did not exist and one that had already been spent.
However, since the code now uses a pruned utxoset, it is no longer
possible to reliably differentiate since once all outputs of a
transaction are spent, it is removed from the utxoset completely.
Consequently, a missing transaction could be either because the
transaction never existed or because it is fully spent.
Also, while here, consistently use the LookupEntry function on the
UtxoView instead of directly accessing the internal map as intended.
This renames the mempool.Config.RelayNonStd option to AcceptNonStd which
more accurately describes its behavior since the mempool was refactored
into a separate package.
The reasoning for this change is that the mempool is not responsible for
relaying transactions (nor should it be). Its job is to maintain a pool
of unmined transactions that are validated according to consensus and
policy configuration options which are then used to provide a source of
transactions that need to be mined.
Instead, it is the server that is responsible for relaying transactions.
While it is true that the current server code currently only relays txns
that were accepted to the mempool, this does not necessarily have to
be the case. It would be entirely possible (and perhaps even a good
idea as something do in the future), to separate the relay policy from
the mempool acceptance policy (and thus indirectly the mining policy).
This modifies the SSGenBlockVotedOn function to directly copy the hash
the vote commits to into a statically sized array versus calling the
hash function and thus can remove the potential runtime error.
This is preferred because the code will fail to compile if the size
of chainhash.HashSize is every changed which is desirable because it
would mean the assumptions the code makes would be invalidated.
It also updates all callers in the repository accordingly.
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This updates the default policy for standard transactions to enforce the
new CSV opcode. In other words, with this change, the new opcode will
be enforced when considering candidate transactions for acceptance to
the mempool, relaying, and inclusion into block templates.
This adds enforcement of the newly introduced relative time locks via
transaction sequence numbers when considering candidate transactions for
acceptance to the mempool, relaying, and inclusion into block templates.
It also raises the maximum standard transaction version to 2
accordingly. This is acceptable because it is a soft forking change and
the aforementioned areas only enforce policy.
The following is an overview of the changes:
- Modify mempool to consider version 2 transactions standard
- Introduce a new callback to the mempool config for obtaining the
sequence lock for block after the current best chain tip so it can
remain decoupled from the chain
- Update mempool to enforce relative locks on all candidate transactions
- Update the mock chain in the mempool test harness accordingly
This reverts the changes related to the CheckSequenceVerify opcode that
were merged from upstream since additional changes are needed and it's
much cleaner to implement all of code related to the sequence locks in
the same PR which will be referenced by the DCP as opposed to being
split up in multiple.
This modifies the mempool transaction finality checks to use the past
median time instead of the adjusted network time. This ensures the
standardness rules match the upcoming consensus rules which reject
transactions that are not after the median time of the last several
blocks. Without this change, it is possible for transactions which can
never make it into a block to be admitted to the mempool thereby wasting
memory.
The following is an overview of the changes:
- Introduce a new callback to the mempool config named PastMedianTime
for obtaining the median time for the current best chain so it can
remain decoupled from the chain
- Update the tx finality standardness check to use the callback
- Remove the now unused TimeSource mempool config parameter
- Update the mempool test harness and mock chain to use a mock median time
- Update server to provide a closure for the mempool config that uses
the cached median time for the current best chain to negate any
additional performance impact the new calculations would otherwise
cause from recalculating the median time for every new candidate tx
validation
This modifies the way standard verification flags are handled so that it
is possible to selectively enable them based on the result of agenda
votes.
First, it moves the StandardVerifyFlags constant from the txscript
package to the mempool/policy code and rename it to
BaseStandardVerifyFlags. As the TODO in the comment of the moved
constant indicated, these flags are policy related and thus really
belong in policy. Ideally there would be a completely separate policy
package, but since the policy code currently lives in mempool/policy.go,
the constant has been moved there.
Next, it introduces a new function named standardScriptVerifyFlags,
which accepts the chain as an argument and, for now, just returns the
BaseStandardVerifyFlags along with a nil error. This will allow
additional flags to be selectively enabled depending on the result of an
agenda vote.
Finally, it updates the mempool policy struct to require a closure for
obtaining the flags so it can remain decoupled from the chain which in
turn allows easier and more robust unit testing of mempool functionality
since it allows a mocks to be used.
The old code ran dust checks on regular, vote, and revocation
transactions, while ignoring any dust checking on ticket purchases.
This change also removes the dust checking for votes and revocations
as these are required for the network to run and should not be
rejected just to minimize dust. The output amounts for these votes
and revocations are determined by consensus rules so standard checks
must be relaxed on them.
If there were to be any dust checks performed on stake transactions,
it should only be done on the commitment amounts in ticket purchases.
By calculating the output amounts that a revocation would require
creating, the mempool could determine if the revocation outputs would
be considered dust with the current policy and reject the ticket
purchase.
Decred's serialized format for transactions split the 32-bit version
field into two 16-bit components such that the upper bits are used to
encode a serialization type and the lower 16 bits are the actual
transaction version.
Unfortunately, when this was done, the in-memory transaction struct was
not also updated to hide this complexity, which means that callers
currently have to understand and take special care when dealing with the
version field of the transaction.
Since the main purpose of the wire package is precisely to hide these
details, this remedies the situation by introducing a new field on the
in-memory transaction struct named SerType which houses the
serialization type and changes the Version field back to having the
desired semantics of actually being the real transaction version. Also,
since the maximum version can only be a 16-bit value, the Version field
has been changed to a uint16 to properly reflect this.
The serialization and deserialization functions now deal with properly
converting to and from these fields to the actual serialized format as
intended.
Finally, these changes also include a fairly significant amount of
related code cleanup and optimization along with some bug fixes in order
to allow the transaction version to be bumped as intended.
The following is an overview of all changes:
- Introduce new SerType field to MsgTx to specify the serialization type
- Change MsgTx.Version to a uint16 to properly reflect its maximum
allowed value
- Change the semantics of MsgTx.Version to be the actual transaction
version as intended
- Update all callers that had special code to deal with the previous
Version field semantics to use the new semantics
- Switch all of the code that deals with encoding and decoding the
serialized version field to use more efficient masks and shifts
instead of binary writes into buffers which cause allocations
- Correct several issues that would prevent producing expected
serializations for transactions with actual transaction versions that
are not 1
- Simplify the various serialize functions to use a single func which
accepts the serialization type to reduce code duplication
- Make serialization type switch usage more consistent with the rest of
the code base
- Update the utxoview and related code to use uint16s for the
transaction version as well since it should not care about the
serialization type due to using its own
- Make code more consistent in how it uses bytes.Buffer
- Clean up several of the comments regarding hashes and add some new
comments to better describe the serialization types
This removes the function IsSupportedMsgTxVersion function from wire
since that is a policy decision and does not belong in wire.
It also updates the mempool standard transaction checks to be more
inline with the upstream code such that the maximum supported
transaction version is specified via a field in the mempool policy
configuration struct.
Finally, it adds a new test to the standard transaction tests to ensure
transactions that are not serialized with the full seriaization type are
considered nonstandard.
This updates the comments in the isDust function to match the current
default relay fee of 1e5, correct tx size values, and make them
more consistent with the field order. It's nice to have the most recent
values as a reference in the comments.
Also, while here, add a couple of tests just below and above the dust
point for the current default relay fee.
This contains the following upstream commits:
- dc5486a579
- 815ded348e
- c6d50b7abf
The merge commit also includes the contains necessary Decred-specific
alterations.
Upstream commit 15bace88dc.
In addition to the necessary Decred-specific alterations to the existing
upstream code, the merge commit also introduces several new functions on
the fake chain instance to handle the additional capabilities needed by
the Decred mempool such as the NextStakeDifficulty, BlockByHash, and
BestHash functions along with a SubsidyCache instance.
