This modifies the PrivateKey type to be opaque and updates the internal
implementation of it to use the new ModNScalar type instead of a
big.Int.
It also modifies NewPrivateKey to accept the new ModNScalar type instead
of a big.Int and improves the PrivKeyFromBytes comment to explicitly
call out the semantics that realistically already existed when using the
value in calculations.
It is also worth noting that there are still a couple of internal
conversions to big.Int when dealing with signatures that are required
since that code still relies on them for now. The goal is to eventually
update them to use the more efficient specialized types and this change
help facilitate that process.
The comment says this is only exported for test code, but the test
code is able to call the unexported function, and is not used outside
of the package.
stake/v3 is not released yet so a breaking change is permitted here to
remove this API.
This modifies all prelease modules to use the latest commit to work out
more transitive dependency issues. Several of these are needed due to
the dependency chain.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/hdkeychain/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/mining/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200215031403-6b2ce76f0986
This modifies all prelease modules to use the latest commit to work out
more transitive dependency issues. Several of these are needed due to
the dependency chain.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/hdkeychain/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/mining/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200215023918-6247af01d5e3
This modifies all prelease modules to use the latest version to work out
more transitive dependency issues.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/hdkeychain/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/mining/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200215015031-3283587e6add
This modifies all prelease modules to use the latest version so they can
be used in require statements in consumer code that is also under
development without running into transitive dependency issues.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/mining/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200214194519-928737b3e580
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200214194519-928737b3e580
This modifies the PrivateKey type to be defined as its own struct as
opposed of in terms of an ecdsa.PrivateKey and updates the code and
tests accordingly. It also removes the PrivKeyFromScalar and
GenerateKey funcs while here since they are effectively duplicates of
existing funcs that were added to satisfy the chainec interface which no
longer exists.
For now, it essentially keeps the same format with the exception of
embedding the ecdsa.PublicKey, but being its own type will alllow for it
to eventually be made opaque and thus provide greater optimizations.
This is continued work towards making the internals of the package
entirely independent of the crypto/elliptic and crypto/ecdsa stdlib
interfaces.
Upcoming changes constitute breaking public API changes to the secp256k1
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the new
version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
Upcoming changes constitute breaking public API changes to the chaincfg
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This modifies the various modules to use a valid prerelease version so
they can be used in require statements in consumer code that is also
under development.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/mining/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200104000002-54b67d3474fb
This reworks the checkpoint logic in blockchain to greatly simplify it
and pave the way to support checkpoints based on only the headers being
available.
Previously there was more complex logic that attempted to find the most
recent checkpoint that is already available in the downloaded portion of
the block chain on demand along with the next checkpoint that is
expected. Instead, the new logic this introduces updates the most
recent checkpoint as the blocks are processed as needed and finds the
most recent checkpoint once during initialization.
This approach will eventually make it possible to update the most recent
checkpoint based on headers alone since it is no longer tightly tied to
the active best chain which will be important when decoupling the
connection code from the download logic.
This modifies the CalcNextRequiredDifficulty function to accept a hash
instead of using the current chain tip as part of an overall effort to
make the blockchain module more explicit and less reliant on the current
state.
This modifies blockchain to expose a new field named Checkpoints in the
Config struct which allows the checkpoints to use to be specified by the
caller instead of assuming the checkpoint associated with the current
chain parameters. This allows the caller to customize the checkpoints
as it sees fit, such as adding custom checkpoints or removing default
checkpoints.
In addition, this removes the DisableCheckpoints function in favor of
the caller simply configuring the chain instance without checkpoints
when they're disabled and updates the code to remove the associated flag
accordingly. This produces simpler code and is faster since the mutex
that was required to protect the aforementioned flag is no longer
required.
Finally, this also paves the way to significantly simplify the
checkpoint finding logic and make further progress towards the overall
effort to decouple the connection code from the download logic.
These checks are no longer necessary since they applied to orphans and
blockchain no longer handles orphans. Further, orphans are currently
otherwise limited by the block manager and this is another step towards
the overall effort to decouple the connection code from the download
logic. Ultimately, the goal is to remove orphan handling altogether.
Finally, it also has the added benefit of further minimizing the
reliance on trusted known good checkpoints.
This decouples and removes the orphan handling from blockchain in favor
of implementing it in the block manager as part of the overall effort to
decouple the connection code from the download logic.
The change might not make a ton of sense in isolation, since there is no
major functional change, however, decoupling the orphan handling
independently helps make the review process easier and alleviates what
would otherwise result in additional intermediate code to handle cases
that ultimately will no longer exist.
The following is a high level overview of the changes:
- Introduce blockchain function to more easily determine if an error is
a rule error with a given error code
- Move core orphan handling code from blockchain to block manager
- Move data structures used to cache and track orphan blocks
- Move all functions releated to orphans
- BlockChain.IsKnownOrphan -> blockManager.isKnownOrphan
- BlockChain.GetOrphanRoot -> blockManager.orphanRoot
- BlockChain.removeOrphanBlock -> blockManager.removeOrphanBlock
- BlockChain.addOrphanBlock -> blockManager.addOrphanBlock
- Implement orphan handling in block manager
- Rework to use the moved functions and data structs
- Add check for known orphans in addition HaveBlock calls to retain
the same behavior
- Modify the semantics of the process block func exposed by the block
manager so that it no longer stores orphans since all consumers of
it ultimately reject orphans anyway
- Remove remaining orphan related code from blockchain
- Update ProcessBlock to return an error when called with an orphan
- Remove additional orphan processing from ProcessBlock
- Remove orphan cache check from HaveBlock
- Adjust example to account for the removed parameter
- Change chaingen harness tests to detect orphans via returned error
- Modify fullblock tests to detect orphans via returned error
- Adapt process order logic tests to cope with lack of orphan
processing
- Update all other tests accordingly
- Update various comments and the README.md and doc.go to properly
reflect the removal of orphan handling
There are several places when processing blocks and headers that require
the ability to traverse the chain to ancestors of a given block such as
tallying votes and versions to determine threshold states, finding
forks, and calculating PoW difficulty, ticket price, and sequence locks.
Currently most traversal of this type is linear and therefore does not
scale well as the chain height grows. In addition, the ability to
quickly locate ancestors, potentially deep in history, will be needed
when decoupling the connection code from the download logic as well as
several planned future optimizations.
Consequently, this significantly optimizes ancestor traversal by
introducing a deterministic skip list and adds tests to ensure proper
functionality.
The following is a before and after comparison of ancestor traversal for
a large number of nodes:
benchmark old ns/op new ns/op delta
------------------------------------------------------
BenchmarkAncestor 67901581 158 -100.00%
benchmark old allocs new allocs delta
------------------------------------------------------
BenchmarkAncestor 0 0 +0.00%
benchmark old bytes new bytes delta
------------------------------------------------------
BenchmarkAncestor 0 0 +0.00%
This adds a new parameter to the chain parameters to allow a minimum
total known cumulative amount of chain work to be specified as of a
given point in time and updates the check which determines if the
blockchain believes it is current to make use of the value instead of
the height of the final checkpoint. In other words, the chain is not
considered synced if the best chain does not yet have the amount of work
specified.
This approach is preferred because it easier to maintain, does not rely
on trusted known good checkpoints, is valid regardless of the order that
blocks are received and processed, and, in general, is a more objective
value that can't be invalidated by a chain reorganization.
It also adds initial values for the main and test networks as follows:
mainnet: 0x0000000000000000000000000000000000000000000ae01920a7ee4b769cc620
testnet: 0x0000000000000000000000000000000000000000000000005df2701ec6263182
Currently, the full block tests, which are intended to test the
consensus rules include some tests that are related to
implementation-dependent processing specifics as opposed to consensus
rules. For example, rejection of duplicate blocks, accepting orphan
blocks, and handling of reorgs to invalid blocks when processed out of
order.
In order to better separate the consensus rules from the processing
aspects, this introduces a new test function specifically for testing
the process order logic and moves the related tests out of the full
block consensus tests into the new function.
This renames the KnownValid and associated status flag of statusValid to
HasValidated and statusValidated, respectively in order to better
clarify the behavior.
While this distinction doesn't currently make much difference the
intention in the future to decouple the block process and download logic
at which point the distinction will become more important.
This removes the main chain block size instance variable since it is
merely set to a constant and is not configurable anyway. While here, it
also moves the prune height calculation outside of the loop and renames
the variables related to the main chain block cache to be more
consistent with the rest of the code.
When an index is enabled for the first time (or after having previously
been disabled and re-enabled), the index needs to be updated to account
for the current state of the main chain.
This poses a problem for indexes that require access to the previous
scripts, such as the address index, since the outputs are no longer in
the utxo set due to pruning. Currently, in order to handle this, the
code has to load all of the referenced transactions in order to get
access to the output scripts which is extremely inefficient. The
address index is already notoriously time consuming to generate due to
the sheer amount of information, but it is only exacerbated by this
inefficiency.
However, the necessary information exists in a much more efficient form
in the spend journal and the recently introduced interfaces which
decouple the script lookup from the concrete type provide a path to
significantly optimize the index catch up process.
Consequently, this modifies the indexer.ChainQueryer interface to
provide access to a source of previous transaction scripts and their
associated versions spent by the given block, updates the index
initialization to make use of it, and updates the blockchain
implementation of the interface to provide the required information via
the spend journal.
As an example of the speedup obtained by this, the following shows how
long it took to catch up the address index from block 0 to block 391500
on a 7200 RPM HDD:
before: 55 minutes, 25 seconds
after: 34 minutes, 4 seconds
That amounts to about a 38.5% reduction. It is also worth noting that
the speedup is not linear, so as the number of transactions continues to
the grow, the speedup will be increasingly more significant.
Some indexes require all of the previous output scripts referenced as
inputs by the blocks they are indexing. Currently, the indexing code
relies on a concrete blockchain.UtxoViewpoint to provide that
information which makes the indexing code more rigid than it needs to be
and prevents the possibility of using other sources, such as the spend
journal, to provide the necessary scripts.
In addition, the index initialization code also currently requires
access to a concrete blockchain.BlockChain to query the current chain
state which also makes the code more rigid than it should ideally be.
Consequently, this modifies the indexing code to accept a PrevScripter
interface to remove the tight coupling to a specific type in the former
case, and introduces a new ChainQueryer interface to remove the tight
coupling in the latter case.
In order to fully remove the blockchain dependency from the indexers
package, this also moves the IndexManager interface from blockchain to
indexers and modifies it to use the new ChainQueryer interface in order
to provide the necessary information.
Finally, all code is updated to satisfy and use the interfaces.
This introduces a new interface named PriorityInputser and updates the
functions that currently take a pointer to a blockchain.UtxoViewpoint
struct to accept the interface instead.
This removes the tight coupling between the two packages at the API
boundary and allows callers to easily provide custom values if desired.
It also updates the blockchain.UtxoViewpoint struct to implement the
interface.
Several upcoming changes constitute breaking public API changes to the
blockchain module, therefore, this follows the process for introducing
major API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This updates the blockchain module dependencies and serves as a base for
blockchain/v2.1.0.
The updated direct dependencies in this commit are as follows:
- github.com/decred/dcrd/blockchain/stake/v2@v2.0.2
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/blockchain/standalone@v1.1.0
- github.com/decred/dcrd/chaincfg/v2@v2.3.0
- github.com/decred/dcrd/database/v2@v2.0.1
- github.com/decred/dcrd/dcrutil/v2@v2.0.1
- github.com/decred/dcrd/gcs/v2@v2.0.0
- github.com/decred/dcrd/txscript@v2.1.0
- github.com/decred/dcrd/wire@v1.3.0
The full list of updated direct dependencies since the previous
blockchain/v2.0.2 release are as follows:
- github.com/decred/dcrd/blockchain/stake/v2@v2.0.2
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/blockchain/standalone@v1.1.0
- github.com/decred/dcrd/chaincfg/v2@v2.3.0
- github.com/decred/dcrd/database/v2@v2.0.1
- github.com/decred/dcrd/dcrec/secp256k1/v2 v2.0.0
- github.com/decred/dcrd/dcrutil/v2@v2.0.1
- github.com/decred/dcrd/gcs/v2@v2.0.0
- github.com/decred/dcrd/txscript@v2.1.0
- github.com/decred/dcrd/wire@v1.3.0
This updates the blockchain/stake module dependencies and serves as a
base for blockchain/stake/v2.0.2.
The updated direct dependencies in this commit are as follows:
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/chaincfg/v2@v2.3.0
- github.com/decred/dcrd/database/v2@v2.0.1
- github.com/decred/dcrd/dcrutil/v2@v2.0.1
- github.com/decred/dcrd/txscript@v2.1.0
- github.com/decred/dcrd/wire@v1.3.0
The full list of updated direct dependencies since the previous
blockchain/stake/v2.0.1 release are as follows:
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/chaincfg/v2@v2.3.0
- github.com/decred/dcrd/database/v2@v2.0.1
- github.com/decred/dcrd/dcrutil/v2@v2.0.1
- github.com/decred/dcrd/txscript@v2.1.0
- github.com/decred/dcrd/wire@v1.3.0
This updates the blockchain/standalone module dependencies and serves as
a base for blockchain/standalone/v1.1.0.
The updated direct dependencies in this commit are as follows:
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/wire@v1.3.0
The full list of updated direct dependencies since the previous
blockchain/standalone/v1.0.0 release are as follows:
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/wire@v1.3.0
This implements an agenda for voting to repurpose the stake root field
of the block header to house a commitment root that includes an initial
commitment to a version 2 block filter as defined in DCP0005 along with
consensus tests.
In particular, once the vote has passed and is active, the stake root
field of the block header will contain the merkle root of a merkle tree
that consists of the hash of the version 2 GCS block filter as the sole
leaf.
In order to accomplish validation efficiently and such that it provides
a well-defined path for adding future commitments, this introduces a new
unexported struct in blockchain to house header commitment data, named
headerCommitmentData, and modifies the relevant funcs to accept an
instance of it.
Next, it introduces a new function named CalcCommitmentRootV1 which
takes the filter hash to commit to and returns the resulting commitment
root. It is certainly the case that this function is not strictly
necessary yet since the version 1 header commitment only consists of a
single item, and hence the root will be the same as the hash provided.
However, this approach is used in order to provide a clear path for
future commitment versions, help make it clear exactly what each version
commits to, and to provide for more consistent code the supports
multiple versions.
Finally, since the version 2 block filters require all previous output
scripts referenced as inputs by the block, and some of those scripts may
no longer be availabled in the pruned utxo set in the case the current
tip block of the main chain does not have enough votes, this introduces
a new blockchain method named FetchUtxoViewParentTemplate to load utxo
details from the point of view of just having connected the given block
template to the parent of the tip of the main chain. It also ensures
the provided template connects to the parent as expected.
It is also worth noting that this makes use of the already introduced
header commitments agenda and associated changes to generate new version
blocks and therefore must be merged at the same time as the commits
which introduce those changes along with the other consensus changes
that the agenda entails.
The following is an overview of the changes:
- Generate block templates with the stake root value set to either the
existing stake root or the new commitment root in accordance with the
state of the vote
- Add new ErrCalcCommitmentRoot mining error code
- Introduce a new function named calcBlockCommitmentRootV1 which
handles creating the version 2 block filter and calculating the
resulting commitment root for block templates
- Remove the no longer used mining calcTxTreeMerkleRoot function
- Modify block validation to enforce the commitment root field in
accordance with the state of the vote
- Add new ErrBadCommitmentRoot rule error code to uniquely identify
the new consensus violation
- Introduce new unexported struct to house header commitment data and
modify relevant funcs to accept an instance of it
- Introduce a new function named CalcCommitmentRootV1 for calculating
the commitment root
- Add tests to ensure the commitment root is calculated properly
- Add a new blockchain method named FetchUtxoViewParentTemplate
This modifies the chain logic to create and store version 2 block
filters for all new blocks and also adds code to migrate the database to
retroactively create and store the v2 filters for all historical blocks.
Since this requires a database upgrade and the next release of the
software will include a vote to change the consensus rules, this also
takes the opportunity to unmark all blocks previously marked as having
failed validation so they are eligible for validation again under what
will likely become new consensus rules. This ensures clients that did
not update prior to new rules activating are able to automatically
recover under the new rules without having to download the entire chain
again.
The following is a high level overview of the changes:
- Introduce a new database bucket to house v2 block filters
- Make UtxoViewpoint satisfy the PrevScripter interface so it may be
used as a source previous scripts when creating filters
- Create and store the new filters in the db when connecting blocks
- Introduce exported function named FilterByBlockHash to retrieve the
new block filters so they are available to be served in the future
- Implement database migration code to retroactively create the new
filters for all historical blocks
- Bump the chain database version to 6
- Introduce code to allow spent txout entries from the spend journal
to be used as a source of previous scripts to significantly optimize
the filter creation as compared to what would be required to
reconstruct all the utxo views as of each block
- Mark all blocks that failed validation under the current consensus
rules as eligible for validation again
- Export a new constant named HeaderCmtFilterIndex which indicates the
header proof index for the upcoming filter header commitment
This adds two new functions to the blockchain/standalone module named
GenerateInclusionProof and VerifyInclusionProof which can be used to
generate and verify inclusion proofs for merkle trees, respectively.
It also updates the documentation and includes comprehensive tests.
