Currently, block templates that provide work to PoW miners are generated
in response to them calling the getwork RPC. However, unlike in a pure
PoW system, in Decred, when a new block is connected, a new block
template extending that block can't be generated until the minimum
number of required votes for it has been received. This poses a
challenge for PoW miners because they are typically unaware of when
votes are received. Consequently, miners poll getwork waiting until
they see a new template generated when a new block is connected.
Since only a minimum of 3 votes is required to build a template and new
templates are only generated in response to the miners calling getwork,
as just described, miners often, unfortunately, end up receiving a
template that only has 3 votes and begin working on it. Worse, since
many new miners are not well versed in the intricacies of Decred voting,
they often aren't aware they really need to continue polling for new
votes and immediately switch over to the new template with more votes in
it in order to avoid receiving a reduced subsidy. This often results in
new miners producing blocks with 3 votes.
Moreover, blocks with less than all 5 votes are not only undesirable for
the PoW miners because they receive reduced subsidy, they're also
undesirable for PoS stakers since it means the votes that were not
included don't receive the reward despite having voted simply due to a
propagation delay.
Another issue with the existing approach is that since it takes a bit of
time to generate and validate templates, particularly as the number of
transactions they include rises, miners periodically requesting updated
work to include the latest transactions have to wait for the template to
be built when they call getwork once the existing cached template is
expired. This can result in undesirable delays for them.
In order to address the need to wait for templates to be built in that
case, there was recently some preliminary work that implemented a
background template generator which asynchronously generates templates
in response to events with the intention of allowing access to the
latest one without said random delays in the future.
However, that preliminary work did not address the issues around vote
propagation such as the tendency for miners to end up with the first
block template only containing the 3 fastest-propagating votes, nor did
it have robust handling for all potential corner cases and error
conditions.
Consequently, this overhauls the background block template generator to
add support for intelligent vote propagation handling, handle chain
reorganization to alternative blocks in the case the current tip is
unable to obtain enough votes, provide a subscription for a stream of
template updates, handle template generation errors, consider the
synchronization state as a part of determining if the chain is current,
track the reason for template updates, block current template retrieval
during operations that are in the process of making it stale, and
correct several corner cases.
It should be noted that this only implements the infrastructure and does
not switch getwork or the CPU miner over to use the background templates
yet. This will be done in future commits.
The following is a high-level overview of the semantics this implements:
- Generate new templates immediately when prior to stake validation
height since no votes are required
- Do not generate templates for intermediate blocks during a chain
reorganization
- Do not generate templates before the chain is considered synchronized
- Prefer to create templates with maximum votes through the use of a
timeout once the minimum votes have been received to provide the votes
an opportunity to propagate with a fallback for immediate generation
as soon as all votes are received
- In the case the timeout expires and a template is created with less
than the maximum number of votes, generate a new template
immediately upon receiving more votes for the block it extends
- In the event there are competing blocks at the current tip, prefer to
build a template on the first seen block so long as it receives the
minimum number of required votes within a few seconds
- Generate new templates immediately when a block is disconnected to
support future planned chain invalidation capabilities
- Generate new templates periodically when there are new regular transactions
to include
- Schedule retries in the rare event template generation fails
- Allow clients to subscribe for updates every time a new template is
successfully generated along with a reason why it was generated
- Provide direct access to the most-recently generated template
- Block while generating new templates that will make the current template
stale (e.g. new parent or new votes)
The following is a high-level overview of the changes:
- Initialize PRNG once instead of at every invocation
- Implement an asynchronous queue for all events to ensure normal chain
and vote processing is not blocked
- Provide ability to subscribe for all template updates over a single
channel to ensure none can indadvertently be missed as is possible with
the current design
- Always deliver existing template upon registration
- Drop templates if the receiver falls too far behind in the same way
tickers work
- Introduce tracking the reason for template updates so that it can
ultimately be presented to miners to allow them to make better
decisions about when to force their workers to switch
- Consider the current network sync state when checking if the chain is
current
- Introduce a regen handler state such as timeouts, blocks to monitor
for votes, and the block the next template should extend
- Add logic for selectively reacting to votes on the block the current
template is extending, the current tip, and alternate competing blocks
at the current tip using the aforementioned semantics
- Perform all template generation in separate goroutines with a
cancellable context
- Cancel any in progress templates that are being generated whenever
generating a new one
- Introduce blocking current template retrieval when a new template that
would cause the existing one to become stale is being generated
- Modify periodic regen handling to use a resettable timer for better
efficiency and more fine grained control
- Remove template pool as that is something that should be handled by
the code that is actually handing the templates out
- Rename and export the event notification funcs to make it clear
they're not internal functions and also make it easier to eventually
move the code into the mining package
- Expand and improve comments
- Store and return template generation errors
- Schedule retry in the case of failure
- Correct several cases that were not being handled correctly and some
undesirable behaviors such as block disconnects (as opposed to
reorgs), reorganization to side chains, and notifying with stale
templates
BgBlkTmplGenerator represents the background process that
generates block templates and notifies all subscribed clients
on template regeneration. It generates new templates based
on mempool activity for vote and non-vote transactions and
the time elapsed since last template regeneration.
This also adds a template pool to the background block
generator for recreating submitted blocks.
Instead of waiting for a second inv to process what the first inv
may have requested, just send it all at once. This allows dropping
of a request queue for each server peer.
This updates the main module to use version 2 of the mempool module.
The following is a summary of changes:
- Update all imports to use mempool/v2
- Update module requirements to include new module and remove no longer
needed old versions
- Add an override for the v2 module so CI is always builds with the
latest code
- Update rpcserver getrawmempool to make use of new mempool
VerboseTxDescs instead of the removed RawMempoolVerbose
- Update rpcserver getrawmempool to properly return the parameter
provided for the transaction type instead of the type of variable in
the case an invalid type is provided
- Update rpcserver getrawmempool to return the supported transaction
types in the error when an invalid type is provided
This commit performs the necessary modifications to hook the fee
estimation facility added by the previous commit into the node software.
The block manager is modified to notify the estimator of all new blocks,
such that their transactions can be accounted for, and the mempool is
modified to relay transactions entering and leaving it, so that those
transactions can be tracked.
The results of the estimator can be queried by issuing estimatesmartfee
rpc commands to the node.
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
This resurrects the regression test network that was removed before
initial launch although it really should not have been. The simulation
test network and the regression test network do not serve the same
purpose. Specifically, the regression test network is intended for unit
tests, RPC server tests, and consensus tests. On the other hand, the
simulation test network is intended for private use within a group of
individuals doing simulation testing and full integration tests between
different applications such as wallets, voting service providers, mining
pools, block explorers, and other services that build on Decred.
Keeping the concerns separate will allow the simulation test network to
be modified in ways such as activating consensus changes that have been
successfully voted into mainnet without also needing to vote them in on
the simulation test network while still preserving the ability for the
unit tests to properly test the voting semantics and handling to help
prevent regressions.
In addition to resurrecting the regression test network, this also fully
fleshes out new values for the various addresses prefixes (Rk, Rs, Re,
etc), HD key prefixes (rprv, rpub), and treasury multisig details.
As a part of resurrecting the network, a new CLI flag `--regnet` is
added to allow the RPC test harness connect to a running instance, the
areas of the code which involve votes have been modified to allow the
votes to apply to the new network, and tests have been added to the
relevant modules.
This bumps the affected module versions as follows:
- github.com/decred/dcrd/wire@v1.2.0
- github.com/decred/dcrd/chaincfg@v1.2.0
- github.com/decred/dcrd/dcrutil@v1.2.0
- github.com/decred/dcrd/hdkeychain@v1.1.1
The blockchain module is also affected, but since its version has
already been bumped since the last release tag, it is not bumped again.
Finally, this does not include switching unit tests or the RPC test
harness over the new network since that will be done in a separate
commit.
Clients are able to choose between two modes of notification about new
blocks via the wire protocol. The first, and current default, is via
inv messages and the second is via direct header announcements.
There is code in the peer package which deals with filtering duplicate
inventory announcements via the QueueInventory message. This works as
expected for inv-based announcements, however, direct header
announcements are not inv messages and hence do not use that code path.
Consequently, the recent addition of the fast relay path which relied on
that filtering to avoid sending duplicate announcements resulted in
duplicate announcements being sent to peers who requested the header
announcements mode.
This resolves the issue by adding some logic directly to the
notifications themselves to deal with filtering duplicates, both for inv
and header-based announcements.
This makes use of the newly exposed notification from blockchain when a
block that intends to extend the main chain has passed all sanity and
contextual checks to relay the block the rest of the network at that
point rather than needing to wait for the more expensive connection code
to complete.
This modifies the infrastructure for the server inventory relay to allow
the caller to specify the inventory should be announced immediately
versus using the typical trickle mechanism.
It also updates all callers in the repository and changes the relay of
accepted blocks use the new ability.
Current code results in transactions from invalidated blocks being
removed from the mempool instead of remaining there to be mined in a
future block.
The new version ensures that transactions are *only* removed once they
are in a block voted as valid.
This only applies to regular transactions, since stake transactions are
not affected by the validity bits.
This modifies the way orphan removal and processing is done to more
aggressively remove orphans that can no longer be valid due to other
transactions being added or removed from the primary transaction pool.
The net effect of these changes is that orphan pool will typically be
much smaller which greatly improves its effectiveness. Previously, it
would typically quickly reach the max allowed worst-case usage and
effectively stay there forever.
The following is a summary of the changes:
- Modify the map that tracks which orphans redeem a given transaction to
instead track by the specific outpoints that are redeemed
- Modify the various orphan removal and processing functions to accept
the full transaction rather than just its hash
- Introduce a new flag on removeOrphans which specifies whether or not
to remove the transactions that redeem the orphan being removed as
well which is necessary since only some paths require it
- Add a new function named removeOrphanDoubleSpends that is invoked
whenever a transaction is added to the main pool and thus the outputs
they spent become concrete spends
- Introduce a new flag on maybeAcceptTransaction which specifies whether
or not duplicate orphans should be rejected since only some paths
require it
- Modify processOrphans as follows:
- Make use of the modified map
- Use newly available flags and logic work more strictly work with tx
chains
- Recursively remove any orphans that also redeem any outputs redeemed
by the accepted transactions
- Several new tests to ensure proper functionality
- Removing an orphan that doesn't exist is removed both when there is
another orphan that redeems it and when there is not
- Removing orphans works properly with orphan chains per the new
remove redeemers flag
- Removal of multi-input orphans that double spend an output when a
concrete redeemer enters the transaction pool
Ticket purchase, vote and revocation orphan tests have also been added.
This modifies the mempool to handle pruning of expired transactions in a
self-contained way so that the caller is not responsible for worrying
about the specific semantics of height and expiration interplay and
updates all callers in the repository accordingly.
It also adds a test which ensures the behavior is correct by creating a
series of transactions with expirations, adding them just before the
point at which they will expire, and then advancing the chain so that
each transaction expires and thus should be pruned.
This modifies the logic which sends winning ticket notifications to RPC
clients to consider the reorganization depth and refuse to send the
notifications for blocks on side chains that would ultimately result in
a reorg of 6 or more blocks.
Also, the code used to send the notifications was needlessly repeated in
multiple places since any blocks accepted via ProcessBlock also result
in an accepted notification and therefore the RPC notification only
needs to be sent from there.
This modifies the ProcessBlock function in the blockchain package to
return the fork length for the connected block and updates all callers
and tests accordingly. Several of the internal functions which
ProcessBlock calls are also updated in order to bubble the necessary
information back up so it can be returned. It does not make any
behavioral changes.
This is being done to better expose information about the position of
the block within the chain to callers without them having to make
additional queries.
The current code creates block templates with coinbase transactions that
have 4 uint64 extra nonces, but only ever uses one of them. Rather than
wasting the extra space, this modifies the code so it only uses a single
uint64 extra nonce.
It should be noted that realistically there isn't even a real need for
an extra nonce in the coinbase at all because there is extra space in
the header specifically for that purpose and miners can't modify the
coinbase via getwork anyways.
However, it is still necessary to insert for the current code in order
to ensure every block template has a unique merkle root since that is
what is used to uniquely identify the block template.
This modifies the mining code to request the desired blocks from chain
by hash as opposed to using a separate function to get the top block and
removes the now unused plubming through block manager.
Not only is this faster because it avoids the need to go through block
manager, it is also more robust since it does not rely on the tip of the
chain not having changing for correctness.
This replaces the GetGeneration function which allowed getting the
entire generation (all children stemming from the same parent) of an
arbitrary bock with TipGeneration which only returns the same
information for the tip block.
This is being done because the function is only used for mining purposes
to get the generation of the current tip. The code is simplified by
reducing its scope to its actual purpose as an initial benefit. It also
provides much better optimization opportunities later.
This propagates the interrupt channel through to blockchain and the
indexers so that it is possible to interrupt long-running operations
such as catching up indexes.
This modifies the BestState struct in the blockchain package to store
hashes directly instead of pointers to them and updates callers to deal
with the API change in the exported BestState struct.
In general, the preferred approach for hashes moving forward is to store
hash values in complex data structures, particularly those that will be
used for cache entries, and accept pointers to hashes in arguments to
functions.
Some of the reasoning behind making this change is:
- It is generally preferred to avoid storing pointers to data in cache
objects since doing so can easily lead to storing interior pointers into
other structs that then can't be GC'd
- Keeping the hash values directly in the structs provides better
cache locality
This modifies the IsSStx, IsSSGen, and IsSSRtx functions to only return
a bool and introduces CheckSStx, CheckSSGen, and CheckSSRtx to return
the actual error as needed by consensus.
This is being done because "is" functions are much nicer to use when
they don't return an error and the callers that use them almost never
care why they aren't of the type, they just want to determine if they
are. In the few cases where the caller does care, they can use of the
new check functions.
While here, also update the comments to call out what the more common
names for the transaction types are and to add comments to the test
functions for consistency.
Finally, it updates all callers in the repo accordingly.
This adds a new function FindSpentTicketsInBlock that extracts all spent and
revoked tickets, plus the vote bits of a given block and uses that function
instead of the individual functions when loading a new node into the blockchain.
This improves startup time a bit, as the isSSGen, isSSRtx and determineTxType
are somewhat slow, as they need to decode the output script of the transactions.
Looping over the transactions once and doing a single test is faster.
Numbers for my computer:
```
| variation | mainnet | testnet |
| with patch | 23.7s | 42.6s |
| without patch | 35.9s | 118.0s |
```
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This implements a new stake difficulty algorithm along with a voting
agenda for all networks to change and a comprehensive set of tests. It
also implements estimation using the new algorithm for the
estimatestakediff RPC.
The following is an overview of the changes:
- Add new agenda to vote on changing the stake difficulty algorithm to
all networks
- The version on mainnet is version 4
- The version on testnet and simnet is version 5
- Modifies the stake difficulty calculation function to calculate the
difficulty based on the result of the vote
- Modifies the stake difficulty estimation function to calculate the
difficulty based on the result of the vote
- Makes the stake difficulty estimation function concurrent safe
- Calls it directly from the RPC server instead of going through block
manager
- Removes no longer needed code from the block manager
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 4
- New block version on testnet and simnet is version 5
- Add tests for the supply estimation used in the new algorithm
- Add tests for the new algorithm calculations
- Add tests for the estimation based on the new algorithm
