This modifies the PrivateKey type to be opaque and updates the internal
implementation of it to use the new ModNScalar type instead of a
big.Int.
It also modifies NewPrivateKey to accept the new ModNScalar type instead
of a big.Int and improves the PrivKeyFromBytes comment to explicitly
call out the semantics that realistically already existed when using the
value in calculations.
It is also worth noting that there are still a couple of internal
conversions to big.Int when dealing with signatures that are required
since that code still relies on them for now. The goal is to eventually
update them to use the more efficient specialized types and this change
help facilitate that process.
This modifies all prelease modules to use the latest commit to work out
more transitive dependency issues. Several of these are needed due to
the dependency chain.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/hdkeychain/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/mining/v3@v3.0.0-20200215031403-6b2ce76f0986
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200215031403-6b2ce76f0986
This modifies all prelease modules to use the latest commit to work out
more transitive dependency issues. Several of these are needed due to
the dependency chain.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/hdkeychain/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/mining/v3@v3.0.0-20200215023918-6247af01d5e3
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200215023918-6247af01d5e3
This modifies all prelease modules to use the latest version to work out
more transitive dependency issues.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/hdkeychain/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/mining/v3@v3.0.0-20200215015031-3283587e6add
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200215015031-3283587e6add
This modifies all prelease modules to use the latest version so they can
be used in require statements in consumer code that is also under
development without running into transitive dependency issues.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/mining/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200214194519-928737b3e580
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/chaincfg/v3@v3.0.0-20200214194519-928737b3e580
- github.com/decred/dcrd/dcrec/secp256k1/v3@v3.0.0-20200214194519-928737b3e580
This change moves tickets that spend from transactions in
the mempool into a separate data structure, so they are not
considered for block template inclusion or fee estimation.
This updates the allowed votes range for mainnet, simnet
and regnet to coinbase maturity from tip. Testnet has been
exempted due to longer reorgs that have already happened
in testnet's history.
This modifies the PrivateKey type to be defined as its own struct as
opposed of in terms of an ecdsa.PrivateKey and updates the code and
tests accordingly. It also removes the PrivKeyFromScalar and
GenerateKey funcs while here since they are effectively duplicates of
existing funcs that were added to satisfy the chainec interface which no
longer exists.
For now, it essentially keeps the same format with the exception of
embedding the ecdsa.PublicKey, but being its own type will alllow for it
to eventually be made opaque and thus provide greater optimizations.
This is continued work towards making the internals of the package
entirely independent of the crypto/elliptic and crypto/ecdsa stdlib
interfaces.
Upcoming changes constitute breaking public API changes to the secp256k1
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the new
version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
Upcoming changes constitute breaking public API changes to the chaincfg
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This modifies the various modules to use a valid prerelease version so
they can be used in require statements in consumer code that is also
under development.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/blockchain/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/connmgr/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/dcrutil/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/mempool/v4@v4.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/mining/v3@v3.0.0-20200104000002-54b67d3474fb
- github.com/decred/dcrd/txscript/v3@v3.0.0-20200104000002-54b67d3474fb
The current max orphan transaction size causes problems with dependent
transaction relay due to its artificially small size in relation to the
max standard transaction size.
Consequently, this modifies the orphan transaction policy by increasing
the max size of each orphan to the same value allowed for standard
non-orphan transactions and reducing the default max allowed number of
orphans to 100.
From a memory usage standpoint, the worst case max mem usage prior to
this change was 5MB plus structure and tracking overhead (1000 max
orphans * 5KB max each). With this, that is raised to 10MB (100 max
orphans * 100KB max each) in the worst case.
It is important to note that the values were originally implemented as a
naive means to control the size of the orphan pool before several of the
recent enhancements which more aggressively remove orphans from the
orphan pool were added, so they needed to be evaluated again.
For a very long time prior to recent changes, the orphan pool would
quickly reach the max allowed worst-case usage and effectively stay
there forever whereas with more recent changes, the actual run-time
orphan pool usage is usually much smaller.
Finally, as another point in favor of this change, as the network has
evolved, nodes have generally become better about orphan management and
as such missing ancestors will typically either be broadcast or mined
fairly quickly resulting in fewer overall orphans.
This allows a caller-provided tag to be associated with orphan
transactions. This is useful since the caller can use the tag for
purposes such as keeping track of which peers orphans were first seen
from.
Also, since a parameter is required now anyways, it associates the peer
ID with processed transactions from remote peers.
This implements orphan expiration in the mempool such that any orphans
that have not had their ancestors materialize within 15 minutes of their
initial arrival time will be evicted which in turn will remove any other
orphans that attempted to redeem it.
In order to perform the evictions with reasonable efficiency, an
opportunistic scan interval of 5 minutes is used. That is to say that
there is not a hard deadline on the scan interval and instead it runs
when a new orphan is added to the pool if enough time has passed.
The following is an example of running this code against the main
network for around an hour while intentionally generating orphans to
test:
2019-10-28 01:44:36.272 [DBG] TXMP: Expired 2 orphans (remaining: 46)
2019-10-28 01:49:39.915 [DBG] TXMP: Expired 17 orphans (remaining: 49)
2019-10-28 01:54:47.704 [DBG] TXMP: Expired 15 orphans (remaining: 43)
2019-10-28 01:59:53.414 [DBG] TXMP: Expired 16 orphans (remaining: 56)
2019-10-28 02:05:09.858 [DBG] TXMP: Expired 16 orphans (remaining: 68)
2019-10-28 02:10:10.567 [DBG] TXMP: Expired 16 orphans (remaining: 78)
2019-10-28 02:15:11.736 [DBG] TXMP: Expired 26 orphans (remaining: 82)
2019-10-28 02:48:34.352 [DBG] TXMP: Expired 85 orphans (remaining: 0)
As can be seen from the above, without orphan expiration on this data
set, the orphan pool would have grown an additional 193 entries.
Several upcoming changes constitute breaking public API changes to the
mempool module, therefore, this follows the process for introducing major
API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
Several upcoming changes constitute breaking public API changes to the
mining module, therefore, this follows the process for introducing major
API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
Several upcoming changes constitute breaking public API changes to the
blockchain module, therefore, this follows the process for introducing
major API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This updates the mempool module dependencies and serves as a base for
mempool/v3.1.0.
The updated direct dependencies in this commit are as follows:
- github.com/decred/dcrd/blockchain/stake/v2@v2.0.2
- github.com/decred/dcrd/blockchain/standalone@v1.1.0
- github.com/decred/dcrd/blockchain/v2@v2.1.0
- github.com/decred/dcrd/chaincfg/v2@v2.3.0
- github.com/decred/dcrd/dcrutil/v2@v2.0.1
- github.com/decred/dcrd/mining/v2@v2.0.1
- github.com/decred/dcrd/txscript/v2@v2.1.0
- github.com/decred/dcrd/wire@v1.3.0
The full list of updated direct dependencies since the previous
mempool/v3.0.0 release are as follows:
- github.com/decred/dcrd/blockchain/stake/v2@v2.0.2
- github.com/decred/dcrd/blockchain/standalone@v1.1.0
- github.com/decred/dcrd/blockchain/v2@v2.1.0
- github.com/decred/dcrd/chaincfg/v2@v2.3.0
- github.com/decred/dcrd/dcrec/secp256k1/v2@v2.0.0
- github.com/decred/dcrd/dcrutil/v2@v2.0.1
- github.com/decred/dcrd/mining/v2@v2.0.1
- github.com/decred/dcrd/txscript/v2@v2.1.0
- github.com/decred/dcrd/wire@v1.3.0
This updates the following modules to use the secp256k1/v2 module:
- blockchain
- chaincfg/v2
- dcrutil/v2
- hdkeychain/v2
- mempool/v3
- txscript/v2
- main
The hdkeychain/v3 and txscript/v2 modules both use types from secp256k1
in their public API.
Consequently, in order avoid forcing them to bump their major versions,
secp256k1/v1.0.3 was released with the types redefined in terms of the
secp256k1/v2 module so callers still using v1 of the module that are not
ready to upgrade to the v2 module yet can interoperate by updating to
the latest patch version.
This adds the ErrorCode member to TxRuleError, filling it with
appropriate values throughout the mempool package. This allows clients
of the package to correctly identify error causes with a greater
granularity and respond appropriately.
It also deprecates the RejectCode attribute and ErrToRejectError
functions, to be removed in the next major version update of the
package.
All call sites that inspect mempool errors were updated to use the new
error codes instead of using RejectionCodes. Additional mempool tests
were added to ensure the correct behavior on some relevant cases.
Finally, given the introduction and use of a new public field, the main
module was updated to use an as-of-yet unfinished mempool v3.1.0, which
will include the required functionality.
This udpates the mempool module to use the latest module major versions
as well as the new blockchain/standard module.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v2@v2.0.1
- github.com/decred/dcrd/blockchain/standalone@v1.0.0
- github.com/decred/dcrd/blockchain/v2@v2.0.2
- github.com/decred/dcrd/chaincfg/v2@v2.2.0
- github.com/decred/dcrd/dcrutil/v2@v2.0.0
- github.com/decred/dcrd/mining/v2@v2.0.0
- github.com/decred/dcrd/txscript/v2@v2.0.0
This updates the mempool` module dependencies and serves as a base for
mempool/v2.1.0.
The updated direct dependencies in this commit are as follows:
- github.com/decred/dcrd/blockchain@v1.2.0
- github.com/decred/dcrd/blockchain/stake@v1.2.1
- github.com/decred/dcrd/chaincfg@v1.5.2
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/dcrec/secp256k1@v1.0.2
- github.com/decred/dcrd/dcrutil@v1.4.0
- github.com/decred/dcrd/mining@v1.1.1
- github.com/decred/dcrd/txscript@v1.1.0
The full list of updated direct dependencies since the previous
mempool/v2.0.0 release are as follows:
- github.com/decred/dcrd/blockchain@v1.2.0
- github.com/decred/dcrd/blockchain/stake@v1.2.1
- github.com/decred/dcrd/chaincfg@v1.5.2
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/dcrec@v1.0.0
- github.com/decred/dcrd/dcrec/secp256k1@v1.0.2
- github.com/decred/dcrd/dcrutil@v1.4.0
- github.com/decred/dcrd/mining@v1.1.1
- github.com/decred/dcrd/txscript@v1.1.0
This updates the main module to use the latest available minor and patch
versions of all modules and reverts the recent change that incorrectly
removed all of the blake256 references from the various go.sum files.
The following required direct dependencies are bumped as follows:
- github.com/decred/dcrd/blockchain/stake@v1.2.1
- github.com/decred/dcrd/blockchain/stake/v2@v2.0.1
- github.com/decred/dcrd/certgen@v1.1.0
- github.com/decred/dcrd/chaincfg@v1.5.2
- github.com/decred/dcrd/chaincfg/chainhash@v1.0.2
- github.com/decred/dcrd/chaincfg/v2@v2.2.0
- github.com/decred/dcrd/dcrutil/v2@v2.0.0
- github.com/decred/dcrd/gcs@v1.1.0
- github.com/decred/dcrd/hdkeychain/v2@v2.0.1
- github.com/decred/dcrd/txscript/v2@v2.0.0
- github.com/decred/dcrwallet/rpc/jsonrpc/types@v1.2.0
This updates the code to make use of the new crypto/blake256 module instead of
github.com/dchest/blake256.
* change the references in the chaincfg/chainhash and gcs modules
* update chaincfg/chainhash to use the new Sum256 API
* remove references to dchest/blake256 from all go.sum files
Currently, block templates that provide work to PoW miners are generated
in response to them calling the getwork RPC. However, unlike in a pure
PoW system, in Decred, when a new block is connected, a new block
template extending that block can't be generated until the minimum
number of required votes for it has been received. This poses a
challenge for PoW miners because they are typically unaware of when
votes are received. Consequently, miners poll getwork waiting until
they see a new template generated when a new block is connected.
Since only a minimum of 3 votes is required to build a template and new
templates are only generated in response to the miners calling getwork,
as just described, miners often, unfortunately, end up receiving a
template that only has 3 votes and begin working on it. Worse, since
many new miners are not well versed in the intricacies of Decred voting,
they often aren't aware they really need to continue polling for new
votes and immediately switch over to the new template with more votes in
it in order to avoid receiving a reduced subsidy. This often results in
new miners producing blocks with 3 votes.
Moreover, blocks with less than all 5 votes are not only undesirable for
the PoW miners because they receive reduced subsidy, they're also
undesirable for PoS stakers since it means the votes that were not
included don't receive the reward despite having voted simply due to a
propagation delay.
Another issue with the existing approach is that since it takes a bit of
time to generate and validate templates, particularly as the number of
transactions they include rises, miners periodically requesting updated
work to include the latest transactions have to wait for the template to
be built when they call getwork once the existing cached template is
expired. This can result in undesirable delays for them.
In order to address the need to wait for templates to be built in that
case, there was recently some preliminary work that implemented a
background template generator which asynchronously generates templates
in response to events with the intention of allowing access to the
latest one without said random delays in the future.
However, that preliminary work did not address the issues around vote
propagation such as the tendency for miners to end up with the first
block template only containing the 3 fastest-propagating votes, nor did
it have robust handling for all potential corner cases and error
conditions.
Consequently, this overhauls the background block template generator to
add support for intelligent vote propagation handling, handle chain
reorganization to alternative blocks in the case the current tip is
unable to obtain enough votes, provide a subscription for a stream of
template updates, handle template generation errors, consider the
synchronization state as a part of determining if the chain is current,
track the reason for template updates, block current template retrieval
during operations that are in the process of making it stale, and
correct several corner cases.
It should be noted that this only implements the infrastructure and does
not switch getwork or the CPU miner over to use the background templates
yet. This will be done in future commits.
The following is a high-level overview of the semantics this implements:
- Generate new templates immediately when prior to stake validation
height since no votes are required
- Do not generate templates for intermediate blocks during a chain
reorganization
- Do not generate templates before the chain is considered synchronized
- Prefer to create templates with maximum votes through the use of a
timeout once the minimum votes have been received to provide the votes
an opportunity to propagate with a fallback for immediate generation
as soon as all votes are received
- In the case the timeout expires and a template is created with less
than the maximum number of votes, generate a new template
immediately upon receiving more votes for the block it extends
- In the event there are competing blocks at the current tip, prefer to
build a template on the first seen block so long as it receives the
minimum number of required votes within a few seconds
- Generate new templates immediately when a block is disconnected to
support future planned chain invalidation capabilities
- Generate new templates periodically when there are new regular transactions
to include
- Schedule retries in the rare event template generation fails
- Allow clients to subscribe for updates every time a new template is
successfully generated along with a reason why it was generated
- Provide direct access to the most-recently generated template
- Block while generating new templates that will make the current template
stale (e.g. new parent or new votes)
The following is a high-level overview of the changes:
- Initialize PRNG once instead of at every invocation
- Implement an asynchronous queue for all events to ensure normal chain
and vote processing is not blocked
- Provide ability to subscribe for all template updates over a single
channel to ensure none can indadvertently be missed as is possible with
the current design
- Always deliver existing template upon registration
- Drop templates if the receiver falls too far behind in the same way
tickers work
- Introduce tracking the reason for template updates so that it can
ultimately be presented to miners to allow them to make better
decisions about when to force their workers to switch
- Consider the current network sync state when checking if the chain is
current
- Introduce a regen handler state such as timeouts, blocks to monitor
for votes, and the block the next template should extend
- Add logic for selectively reacting to votes on the block the current
template is extending, the current tip, and alternate competing blocks
at the current tip using the aforementioned semantics
- Perform all template generation in separate goroutines with a
cancellable context
- Cancel any in progress templates that are being generated whenever
generating a new one
- Introduce blocking current template retrieval when a new template that
would cause the existing one to become stale is being generated
- Modify periodic regen handling to use a resettable timer for better
efficiency and more fine grained control
- Remove template pool as that is something that should be handled by
the code that is actually handing the templates out
- Rename and export the event notification funcs to make it clear
they're not internal functions and also make it easier to eventually
move the code into the mining package
- Expand and improve comments
- Store and return template generation errors
- Schedule retry in the case of failure
- Correct several cases that were not being handled correctly and some
undesirable behaviors such as block disconnects (as opposed to
reorgs), reorganization to side chains, and notifying with stale
templates
This updates all of the modules that still rely on v1 of the chaincfg
module to use v1.5.1 which builds correctly with the major API bump
introduced in the edwards v1.0.0 module.
While here, it also updates all references to the v0 edwards and dcrec
modules to their tagged v1 counterparts, tidies all of the modules via
go mod tidy, and removes all unnecessary indirect entries from the mod
files to keep the modules using the versions the dependencies are tested
with.
The primary motivation for this change is that the chaincfg/v2 module
requires edwards/v1 which is not API compatible with edwards/v0.
Consequently, in order for each module to be incrementally updated to
use chaincfg/v2, all of its dependencies must also be able to build with
edwards/v1 which is the case as of chaincfg v1.5.1.
BgBlkTmplGenerator represents the background process that
generates block templates and notifies all subscribed clients
on template regeneration. It generates new templates based
on mempool activity for vote and non-vote transactions and
the time elapsed since last template regeneration.
This also adds a template pool to the background block
generator for recreating submitted blocks.
This adds the go 1.11 directive to all of the modules in order to
clearly mark they build and work with that version. Go 1.12 modified
the tools such that tidy will automatically add the new version to
modules that do not already have a directive and that would prevent
builds on Go 1.11 through Go 1.11.3 which is not desirable.
Currently, there is a tight coupling between the rpcserver and the
mempool by way of the RawMempoolVerbose function returning a
dcrjson.GetRawMempoolVerboseResult.
In order to remove the tight coupling, this introduces a new type and
function to the mempool module named VerboseTxDesc and VerboseTxDescs,
respectively, which provide all of the details necessary for the
rpcserver to be able to properly generate the dcrjson response
directly.
Also, it removes the old RawMempoolVerbose function and related
dependency imports on dcrjson since it is no longer required as a part
of moving to mempool/v2.
Note that since this constitutes a major break to the API, the mepool
module version will need to be bumped to version 2.
This removes the deprecated MinHighPriority constant as part of moving
to mempool/v2.
Note that since this constitutes a major break to the API, the mepool
module version will need to be bumped to version 2.
