This replaces the ErrDoubleSpend and ErrMissingTx error codes with a
single error code named ErrMissingTxOut and updates the relevant errors
and expected test results accordingly.
Once upon a time, the code relied on a transaction index, so it was able
to definitively differentiate between a transaction output that
legitimately did not exist and one that had already been spent.
However, since the code now uses a pruned utxoset, it is no longer
possible to reliably differentiate since once all outputs of a
transaction are spent, it is removed from the utxoset completely.
Consequently, a missing transaction could be either because the
transaction never existed or because it is fully spent.
Also, while here, consistently use the LookupEntry function on the
UtxoView instead of directly accessing the internal map as intended.
This modifies the newBlockNode function to accept the parent as an
argument to automatically connect the newly created node. When it is
not nil, the work sum will automatically be summed and the parent of the
new node will be set accordingly.
This simplifies the block node construction a bit and allows some
redundant code to be removed. It also paves the way for easier simpler
full block index construction in the future.
This separates the logic for populating the stake information in a block
node from the construction of the node in order to better delineate the
difference between the two and to pave the way for that information to
be stored separately in the database versus needing to load full blocks
to retrieve it.
This moves the checks for validating that votes commit to the parent of
the block they are contained in and the vote bits specified by the
header match the result of the actual votes included in the block to the
checkBlockSanity function where they more naturally belong since they do
not require access to any previous chain context.
This modifies the exported CheckConnectBlock function to call the
checkBlockSanity and checkBlockContext functions to ensure all
validation rules are enforced from the mining code.
Since both the mining code and tests typically work with unsolved
blocks, this also introduces a new parameter on CheckConnectBlock to
pass behavior flags which allows the caller to skip the proof of work
check.
It also modifies forceHeadReorganization to call checkBlockContext for
the same reason.
Finally, all tests and call sites are updated with the appropriate
flags and various tests are updated for the changes accordingly.
This moves the check for validating that there are only ticket purchases
included in blocks before stake validation height to checkBlockSanity
where is more naturally belongs since it does not require access to any
previous chain context.
This moves the test for validating the number of votes specified by the
header is at least the required minimum into the checkBlockHeaderSanity
function where it more naturally belongs since it is solely dependent on
information the header.
Also, remove the redundant check for the same condition from
checkBlockSanity since it has now already been checked according to the
value the header commits to and therefore validating the header
commitment for the number of votes implies correctness.
This adds a new function FindSpentTicketsInBlock that extracts all spent and
revoked tickets, plus the vote bits of a given block and uses that function
instead of the individual functions when loading a new node into the blockchain.
This improves startup time a bit, as the isSSGen, isSSRtx and determineTxType
are somewhat slow, as they need to decode the output script of the transactions.
Looping over the transactions once and doing a single test is faster.
Numbers for my computer:
```
| variation | mainnet | testnet |
| with patch | 23.7s | 42.6s |
| without patch | 35.9s | 118.0s |
```
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This repurposes the sequence number of version 2 transaction inputs to
provide consensus-enforced relative lock-time semantics so that a
transaction can require inputs to have a specified relative age before
they are allowed to be included in a block. Each relative time lock can
either specify a relative number of seconds (with a granularity of 512
seconds and a maximum value of 33,553,920) or a specific number of
blocks (max 65535).
The number of seconds is calculated relative to the past median time of
the block before the one that contains the referenced output. This is
done because, due to other changes that will also be included in the
same agenda vote, said time will become the earliest possible time the
block that contains the referenced output could have been (technically
it will be one second after that, but that complexity is ignored since
there is already a granularity involved anyways).
It is also possible to disable the behavior by setting bit 31 of the
sequence number (which all transactions currently do by default since
they are set to the max).
In order for the transaction to be permitted to the mempool, relayed,
considered for inclusion into block templates, and allowed into a block,
the specified relative time locks for all of its inputs must be
satisfied.
This only implements the required logic and tests to enforce the new
behavior. Code to enforce the new behavior when considering candidate
transactions for acceptance to the mempool, relaying, and inclusion into
block templates will be added in a separate commit.
A consensus vote is required in order to reject blocks which contain
transactions that violate the new rules at a consensus level. Code to
selectively enable consensus enforcement based on the result of an
agenda vote will be added in a separate commit.
In order to accomplish this new behavior, the concept of a sequence lock
is introduced which allows the minimum possible time and height at which
a transaction can be included into a block to be calculated from all
inputs with non-disabled relative time locks, and functions to calculate
and evaluate the sequence lock are added.
The following is an overview of the changes:
- Introduce a new struct named SequenceLock to represent the previously
described sequence lock
- Define new constants related to sequence numbers named
SequenceLockTimeDisabled, SequenceLockTimeIsSeconds,
SequenceLockTimeMask, and SequenceLockTimeGranularity
- Add a new function named calcSequenceLock to calculate the sequence
lock for a given transaction
- Add a new function named SequenceLockActive to determine if a given
sequence lock is satisfied for a given block height and past median
time
- Add a convenience function named LockTimeToSequence which can be used
to convert a relative lock time to a sequence number
- Add a comprehensive set of tests for all of the new funcs
Rather than redefining the simnet parameters in the tests, just use the
existing chaincfg simnet parameters. Since some of the tests rely on
the modified legacy version, introduce a clone function for the params
and simply override the parameters that are different.
This is far preferable to redefining them which is cumbersome to update
when the chaincfg params add new definitions.
Decred's serialized format for transactions split the 32-bit version
field into two 16-bit components such that the upper bits are used to
encode a serialization type and the lower 16 bits are the actual
transaction version.
Unfortunately, when this was done, the in-memory transaction struct was
not also updated to hide this complexity, which means that callers
currently have to understand and take special care when dealing with the
version field of the transaction.
Since the main purpose of the wire package is precisely to hide these
details, this remedies the situation by introducing a new field on the
in-memory transaction struct named SerType which houses the
serialization type and changes the Version field back to having the
desired semantics of actually being the real transaction version. Also,
since the maximum version can only be a 16-bit value, the Version field
has been changed to a uint16 to properly reflect this.
The serialization and deserialization functions now deal with properly
converting to and from these fields to the actual serialized format as
intended.
Finally, these changes also include a fairly significant amount of
related code cleanup and optimization along with some bug fixes in order
to allow the transaction version to be bumped as intended.
The following is an overview of all changes:
- Introduce new SerType field to MsgTx to specify the serialization type
- Change MsgTx.Version to a uint16 to properly reflect its maximum
allowed value
- Change the semantics of MsgTx.Version to be the actual transaction
version as intended
- Update all callers that had special code to deal with the previous
Version field semantics to use the new semantics
- Switch all of the code that deals with encoding and decoding the
serialized version field to use more efficient masks and shifts
instead of binary writes into buffers which cause allocations
- Correct several issues that would prevent producing expected
serializations for transactions with actual transaction versions that
are not 1
- Simplify the various serialize functions to use a single func which
accepts the serialization type to reduce code duplication
- Make serialization type switch usage more consistent with the rest of
the code base
- Update the utxoview and related code to use uint16s for the
transaction version as well since it should not care about the
serialization type due to using its own
- Make code more consistent in how it uses bytes.Buffer
- Clean up several of the comments regarding hashes and add some new
comments to better describe the serialization types
This implements a new stake difficulty algorithm along with a voting
agenda for all networks to change and a comprehensive set of tests. It
also implements estimation using the new algorithm for the
estimatestakediff RPC.
The following is an overview of the changes:
- Add new agenda to vote on changing the stake difficulty algorithm to
all networks
- The version on mainnet is version 4
- The version on testnet and simnet is version 5
- Modifies the stake difficulty calculation function to calculate the
difficulty based on the result of the vote
- Modifies the stake difficulty estimation function to calculate the
difficulty based on the result of the vote
- Makes the stake difficulty estimation function concurrent safe
- Calls it directly from the RPC server instead of going through block
manager
- Removes no longer needed code from the block manager
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 4
- New block version on testnet and simnet is version 5
- Add tests for the supply estimation used in the new algorithm
- Add tests for the new algorithm calculations
- Add tests for the estimation based on the new algorithm
This renames the chaincfg parameter for the vote choice which represents
an abstaining vote to be named IsAbstain instead of IsIgnore since that
more accurately describes its intent and behavior.
It also updates the RPC server choice result field for isignore to be
named isabstain to match and bumps the major version accordingly.
Finally, it renames other internal variables which make use of the
choice to include the word abstain as well for clarity and renames a
couple of other internal variables.
This implements a new voting agenda for the testnet and simnet networks
for increasing the maximum block size from 1MiB to 1.25MB.
The following is an overview of the changes:
- Bump the maximum protocol block size limit to 1.25MB
- Bump the protocol version to 4
- Add wire tests for v3 protocol sizes and update tests for latest
- Update all wire values that are defined in terms of the max block size
to respect the protocol version
- Update the MaxSigOpsPerBlock constant to maintain existing value
regardless of the raised max protocol block size
- Decouple the maximum tx size from the block size by creating a chain
parameter for it with the current sizes so it is not a part of the
hard fork vote
- Add definition for new version 4 stake vote along with agenda to vote
on block size to the testnet and simnet networks
- Convert the MaximumBlockSize chain parameter to a slice in order to
hold the different possible sizes
- Adds a new function that returns the maximum block size based upon the
result of the vote
- Change the existing context-free block size check to enforce the max
protocol size and introduce a context-specific check which enforces a
restricted size based upon the network consensus parameters and the
result of the vote
- Set the max allowed size in generated block templates based upon the
network params and result of the vote
- Generate version 4 blocks and reject version 3 blocks after a super
majority has been reached
Port BIP0009 over from btcd. The following is an overview of the
changes:
- Add new configuration options to the chaincfg package which allows the
rule deployments to be defined per chain
- Implement code to calculate the threshold state as required by voting
- Use threshold state caches that are stored to the database in order
to accelerate startup time
- Remove caches that are invalid due to definition changes in the
params including additions, deletions, and changes to existing
entries
- Verify that PoW and PoS majorities have been reached.
- Add tests for new error type
- Add tests for threshold state
- Deployments are per stake version.
- Add human readable vote/choice infrastructure.
- Add RPC command to interrogate vote status (getvoteinfo).
This updates the various definitions of the simnet params used
exclusively in the tests to match their definitions used by the actual
simnet parameters defined in chaincfg.
While it would certainly be easier to simply use the params defined in
chaincfg, that also would not allow the tests to detect them being
inadvertently changed.
This reverts to the correct upstream majority version code which
properly detects when certain thresholds of the network have been
updated in order to determine when to start rejecting old version
blocks and enforcing the rules in the new version block.
Upstream commit a7b35d9f9e.
Also, the merge commit contains necessary decred-specific alterations.
It also has the side effect of correcting a couple of min reduction bugs
even though they can't currently ever be hit due to the min reduction
interval flag being disabled for all networks.
Contains the following commits:
- 711f33450c
- b6b1e55d1e
- Reverted because Travis is already at a more recent version
- bd4e64d1d4
Also, the merge commit contains the necessary decred-specific
alterations, converts all other references to sha to hash to keep with
the spirit of the merged commits, and various other cleanup intended to
bring the code bases more in line with one another.
This function had several issues:
- It is completely unnecessary since the ErrorCode field is
intentionally exported so it is accessible
- It goes against effective Go naming guidelines which specifically
state getters should not be named GetFoo
- The comment on it regarding implementing the error interface was
incorrect
Bump block version and add stake version to wire. Currently
StakeVersions are unused and once the enforcement logic goes in the
versions will bump again.
This fixes#435
Profiling indicated that significant time was being spent validating
coinbase outputs, ensuring that they paid to the development
organization's P2SH tax address. This check was more inefficient than
necessary for a couple of reasons:
* The tax address was always decoded from a string to a dcrutil.Address.
* The actual script being validated was always parsed for addresses to
check if they matched the tax address.
Neither of these are needed. To optimize the algorithm, only the
equality of the output script and script version are checked.
A bug prevented STXO data from being written correctly in the event
that a block became invalidated. This could cause failures on testnet
when a block was invalidated. There was also some debug code and a panic
that were in master that have now been removed. A new test has been
added to ensure that STXO data is being properly written, and more
stringent checks for STXO data validity when added or removing blocks
has been added.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
Contains the following upstream commits:
- d4852101d4
- This commit has already been independently applied so it is mostly a
NOOP
- f389742b39
In addition, gofmt -s has been run again to simplify the new additions
to Decred and and all simplifications are included in the merge commit.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
This commit converts all block height references to int32 instead of
int64. The current target block production rate is 10 mins per block
which means it will take roughly 40,800 years to reach the maximum
height an int32 affords. Even if the target rate were lowered to one
block per minute, it would still take roughly another 4,080 years to
reach the maximum.
In the mean time, there is no reason to use a larger type which results
in higher memory and disk space usage. However, for now, in order to
avoid having to reserialize a bunch of database information, the heights
are still serialized to the database as 8-byte uint64s.
This is being mainly being done in preparation for further upcoming
infrastructure changes which will use the smaller and more efficient
4-byte serialization in the database as well.
