Upcoming changes constitute breaking public API changes to the chaincfg
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This modifies the CalcNextRequiredDifficulty function to accept a hash
instead of using the current chain tip as part of an overall effort to
make the blockchain module more explicit and less reliant on the current
state.
This adds the regentemplate comand to the node, which asks it to
regenerate is block template.
The node now maintains a background template generator, with various
timeouts to enable efficient generation of block templates in a
production setting.
However, external users might want to force a new template to be
generated after new regular transactions have been received but before
the regular regen timeout (currently 30 seconds) has elapsed. This is
true specially for external simnet tests that publish transactions and
rely on them being mined shortly after the node receives them.
The most flexible solution found was to add a new regentemplate command
which rpc clients can issue to force a node to regenerate the latest
template with any new transactions.
Several upcoming changes constitute breaking public API changes to the
mining module, therefore, this follows the process for introducing major
API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
Several upcoming changes constitute breaking public API changes to the
blockchain module, therefore, this follows the process for introducing
major API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This updates the new RPC websocket work notification that is sent when a
new template is generated and a websocket client requests updated via
notifywork to include the reason the template was generated.
In order to accomplish this, it modifies the background template
generator subscription notifications, the RPC websockets notification
manager, and the JSON-RPC work notification to include the reason and
updates all of the intermediate plumbing code accordingly.
It also updates the JSON-RPC documentation to include the new field and
enumerate the supported reasons.
This exports the template update reason type and associated constants in
preparation to modify the template notifications to include the reason a
new block template was generated.
This updates the getwork RPC to make use of the background template
generator infrastructure and removes code that is no longer needed as a
result.
This has a lot of benefits, many of which were discussed more in depth
in the previous commit that overhauled the background template
generator, however the following is a non-exhaustive overview that
highlights the major benefits of the change:
- Requests for updated templates during the normal mining process in
between tip changes will now be nearly instant instead of potentially
taking several seconds to build the new template on the spot
- When the chain tip changes, requesting a template will now attempt to
wait until either all votes have been received or a timeout occurs
prior to handing out a template which is beneficial for PoW miners,
PoS miners, and the network as a whole
- PoW miners are much less likely to end up with template with less than
the max number of votes which means they are less likely to receive a
reduced subsidy
- PoW miners will be much less likely to receive stale templates during
chain tip changes due to vote propagation
- PoS stakers whose votes end up arriving to the miner slightly slower
than the minimum number required are much less likely to have their
votes excluded despite having voted simply due to propagation delay
The following is a high level overview of the changes:
- Move the template pool to the work state struct
- Remove fields from work state struct that dealt with determining when
to update the template
- Introduce a new function named getWorkTemplateKey to construct a
unique template key for storing and retrieving delivered block
templates
- Make pruneOldBlockTemplates a method of the work state and clean it up
- Completely rework handleGetWorkRequest to use the background template
generator infrastructure
- Wait for a new template to be generated with a timeout when the best
chain tip to bias towards giving out templates with max possible
votes
- Ensure the timestamp is updated with each invocation
- Shallow copy the header (versus deep copying the entire template as
before) to avoid mutations to the shared templates
- Rework handleGetWorkSubmission
- Improve logging
- Use new template key to lookup the template used to reconstruct the
block
- Shallow copy the block from the template before updating it with the
solved versus the more expensive method used before
- Remove no longer used funcs and methods:
- extractCoinbaseExtraNonce
- extractCoinbaseTxExtraNonce
- deepCopyBlockTemplate
This implements an agenda for voting to repurpose the stake root field
of the block header to house a commitment root that includes an initial
commitment to a version 2 block filter as defined in DCP0005 along with
consensus tests.
In particular, once the vote has passed and is active, the stake root
field of the block header will contain the merkle root of a merkle tree
that consists of the hash of the version 2 GCS block filter as the sole
leaf.
In order to accomplish validation efficiently and such that it provides
a well-defined path for adding future commitments, this introduces a new
unexported struct in blockchain to house header commitment data, named
headerCommitmentData, and modifies the relevant funcs to accept an
instance of it.
Next, it introduces a new function named CalcCommitmentRootV1 which
takes the filter hash to commit to and returns the resulting commitment
root. It is certainly the case that this function is not strictly
necessary yet since the version 1 header commitment only consists of a
single item, and hence the root will be the same as the hash provided.
However, this approach is used in order to provide a clear path for
future commitment versions, help make it clear exactly what each version
commits to, and to provide for more consistent code the supports
multiple versions.
Finally, since the version 2 block filters require all previous output
scripts referenced as inputs by the block, and some of those scripts may
no longer be availabled in the pruned utxo set in the case the current
tip block of the main chain does not have enough votes, this introduces
a new blockchain method named FetchUtxoViewParentTemplate to load utxo
details from the point of view of just having connected the given block
template to the parent of the tip of the main chain. It also ensures
the provided template connects to the parent as expected.
It is also worth noting that this makes use of the already introduced
header commitments agenda and associated changes to generate new version
blocks and therefore must be merged at the same time as the commits
which introduce those changes along with the other consensus changes
that the agenda entails.
The following is an overview of the changes:
- Generate block templates with the stake root value set to either the
existing stake root or the new commitment root in accordance with the
state of the vote
- Add new ErrCalcCommitmentRoot mining error code
- Introduce a new function named calcBlockCommitmentRootV1 which
handles creating the version 2 block filter and calculating the
resulting commitment root for block templates
- Remove the no longer used mining calcTxTreeMerkleRoot function
- Modify block validation to enforce the commitment root field in
accordance with the state of the vote
- Add new ErrBadCommitmentRoot rule error code to uniquely identify
the new consensus violation
- Introduce new unexported struct to house header commitment data and
modify relevant funcs to accept an instance of it
- Introduce a new function named CalcCommitmentRootV1 for calculating
the commitment root
- Add tests to ensure the commitment root is calculated properly
- Add a new blockchain method named FetchUtxoViewParentTemplate
This implements the agenda for voting on combining the regular and stake
tree merkle roots into a single merkle root that is stored in the merkle
root field of the block header as defined in DCP0005 along with
consensus tests.
In particular, once the vote has passed and is active, the final merkle
root is the result of a merkle tree that itself has the individual
merkle roots of the two transaction trees as leaves.
It is also worth noting that this does not repurpose the stake root
field once the vote has passed as that will be done in a future commit.
This implies that it is important for both this commit and the
aforementioned future commit which changes the stake root field
semantics to be merged at the same time.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 7
- New block version on testnet is version 8
- Generate block templates with the correct merkle root set in
accordance with the state of the vote
- Introduce a convenience function for determining if the vote passed
and is now active
- Introduce a new function in the standalone module for calculating the
combined merkle root
- Add tests to ensure the combined merkle root calculation produces
the correct results
- Modify block validation to enforce the merkle root field in accordance
with the state of the vote
- Add tests for determining if the agenda is active for both mainnet and
testnet
This combines the two conditions for the aggressive mining path into a
single condition and does a bit of light cleanup to remove the template
copies that are no longer necessary due to the removal of the old style
template caching.
This removes the UpdateExtraNonce function which updated an extra nonce
in the coinbase transaction and recalculated the merkle root since it is
not necessary and wasteful for Decred due to the extra nonce being
available the block header.
Further, due to the aforementioned and the fact the template doesn't
have a height set, it isn't currently actually being called anyway as
can be seen by diffing the decoded output of subsequent getwork calls
and noting the only thing that is being updated in between full
regeneration of new templates is the timestamp as expected.
$ diff -uNp work1.txt work2.txt
--- work1.txt 2019-09-07 08:18:58.410917100 -0500
+++ work2.txt 2019-09-07 08:19:01.216456300 -0500
@@ -16,7 +16,7 @@
"sbits": 0.00021026,
"height": 98,
"size": 7221,
- "time": 1567862338,
+ "time": 1567862341,
"nonce": 0,
"extradata": "0000000000000000000000000000000000000000000000000000000000000000",
"stakeversion": 0,
This removes all of the code related to setting and updating cached
templates in the block manager since they are no longer used.
It is easy to see this is the case by considering that the only places
that set cachedCurrentTemplate and cachedParentTemplate set them to nil.
This updates all code in the main module to use the latest major modules
versions to pull in the latest updates.
A more general high level overview of the changes is provided below,
however, there is one semantic change worth calling out independently.
The verifymessage RPC will now return an error when provided with
an address that is not for the current active network and the RPC server
version has been bumped accordingly.
Previously, it would return false which indicated the signature is
invalid, even when the provided signature was actually valid for the
other network. Said behavior was not really incorrect since the
address, signature, and message combination is in fact invalid for the
current active network, however, that result could be somewhat
misleading since a false result could easily be interpreted to mean the
signature is actually invalid altogether which is distinct from the case
of the address being for a different network. Therefore, it is
preferable to explicitly return an error in the case of an address on
the wrong network to cleanly separate these cases.
The following is a high level overview of the changes:
- Replace all calls to removed blockchain merkle root, pow, subsidy, and
coinbase funcs with their standalone module equivalents
- Introduce a new local func named calcTxTreeMerkleRoot that accepts
dcrutil.Tx as before and defers to the new standalone func
- Update block locator handling to match the new signature required by
the peer/v2 module
- Introduce a new local func named chainBlockLocatorToHashes which
performs the necessary conversion
- Update all references to old v1 chaincfg params global instances to
use the new v2 functions
- Modify all cases that parse addresses to provide the now required
current network params
- Include address params with the wsClientFilter
- Replace removed v1 chaincfg constants with local constants
- Create subsidy cache during server init and pass it to the relevant
subsystems
- blockManagerConfig
- BlkTmplGenerator
- rpcServer
- VotingWallet
- Update mining code that creates the block one coinbase transaction to
create the output scripts as defined in the v2 params
- Replace old v2 dcrjson constant references with new types module
- Fix various comment typos
- Update fees module to use the latest major module versions and bump it v2
Currently, block templates that provide work to PoW miners are generated
in response to them calling the getwork RPC. However, unlike in a pure
PoW system, in Decred, when a new block is connected, a new block
template extending that block can't be generated until the minimum
number of required votes for it has been received. This poses a
challenge for PoW miners because they are typically unaware of when
votes are received. Consequently, miners poll getwork waiting until
they see a new template generated when a new block is connected.
Since only a minimum of 3 votes is required to build a template and new
templates are only generated in response to the miners calling getwork,
as just described, miners often, unfortunately, end up receiving a
template that only has 3 votes and begin working on it. Worse, since
many new miners are not well versed in the intricacies of Decred voting,
they often aren't aware they really need to continue polling for new
votes and immediately switch over to the new template with more votes in
it in order to avoid receiving a reduced subsidy. This often results in
new miners producing blocks with 3 votes.
Moreover, blocks with less than all 5 votes are not only undesirable for
the PoW miners because they receive reduced subsidy, they're also
undesirable for PoS stakers since it means the votes that were not
included don't receive the reward despite having voted simply due to a
propagation delay.
Another issue with the existing approach is that since it takes a bit of
time to generate and validate templates, particularly as the number of
transactions they include rises, miners periodically requesting updated
work to include the latest transactions have to wait for the template to
be built when they call getwork once the existing cached template is
expired. This can result in undesirable delays for them.
In order to address the need to wait for templates to be built in that
case, there was recently some preliminary work that implemented a
background template generator which asynchronously generates templates
in response to events with the intention of allowing access to the
latest one without said random delays in the future.
However, that preliminary work did not address the issues around vote
propagation such as the tendency for miners to end up with the first
block template only containing the 3 fastest-propagating votes, nor did
it have robust handling for all potential corner cases and error
conditions.
Consequently, this overhauls the background block template generator to
add support for intelligent vote propagation handling, handle chain
reorganization to alternative blocks in the case the current tip is
unable to obtain enough votes, provide a subscription for a stream of
template updates, handle template generation errors, consider the
synchronization state as a part of determining if the chain is current,
track the reason for template updates, block current template retrieval
during operations that are in the process of making it stale, and
correct several corner cases.
It should be noted that this only implements the infrastructure and does
not switch getwork or the CPU miner over to use the background templates
yet. This will be done in future commits.
The following is a high-level overview of the semantics this implements:
- Generate new templates immediately when prior to stake validation
height since no votes are required
- Do not generate templates for intermediate blocks during a chain
reorganization
- Do not generate templates before the chain is considered synchronized
- Prefer to create templates with maximum votes through the use of a
timeout once the minimum votes have been received to provide the votes
an opportunity to propagate with a fallback for immediate generation
as soon as all votes are received
- In the case the timeout expires and a template is created with less
than the maximum number of votes, generate a new template
immediately upon receiving more votes for the block it extends
- In the event there are competing blocks at the current tip, prefer to
build a template on the first seen block so long as it receives the
minimum number of required votes within a few seconds
- Generate new templates immediately when a block is disconnected to
support future planned chain invalidation capabilities
- Generate new templates periodically when there are new regular transactions
to include
- Schedule retries in the rare event template generation fails
- Allow clients to subscribe for updates every time a new template is
successfully generated along with a reason why it was generated
- Provide direct access to the most-recently generated template
- Block while generating new templates that will make the current template
stale (e.g. new parent or new votes)
The following is a high-level overview of the changes:
- Initialize PRNG once instead of at every invocation
- Implement an asynchronous queue for all events to ensure normal chain
and vote processing is not blocked
- Provide ability to subscribe for all template updates over a single
channel to ensure none can indadvertently be missed as is possible with
the current design
- Always deliver existing template upon registration
- Drop templates if the receiver falls too far behind in the same way
tickers work
- Introduce tracking the reason for template updates so that it can
ultimately be presented to miners to allow them to make better
decisions about when to force their workers to switch
- Consider the current network sync state when checking if the chain is
current
- Introduce a regen handler state such as timeouts, blocks to monitor
for votes, and the block the next template should extend
- Add logic for selectively reacting to votes on the block the current
template is extending, the current tip, and alternate competing blocks
at the current tip using the aforementioned semantics
- Perform all template generation in separate goroutines with a
cancellable context
- Cancel any in progress templates that are being generated whenever
generating a new one
- Introduce blocking current template retrieval when a new template that
would cause the existing one to become stale is being generated
- Modify periodic regen handling to use a resettable timer for better
efficiency and more fine grained control
- Remove template pool as that is something that should be handled by
the code that is actually handing the templates out
- Rename and export the event notification funcs to make it clear
they're not internal functions and also make it easier to eventually
move the code into the mining package
- Expand and improve comments
- Store and return template generation errors
- Schedule retry in the case of failure
- Correct several cases that were not being handled correctly and some
undesirable behaviors such as block disconnects (as opposed to
reorgs), reorganization to side chains, and notifying with stale
templates
This ensures the vote handling for the new background miner template
happens in a goroutine in order to prevent blocking more transactions
being added to the mempool and the vote itself from being immediately
relayed to the rest of the network.
It also prevents a potential deadlock that can occur due to the cycle
created by being called under the mempool lock and in turn generating a
template which itself attempts to grab the mempool lock to get the list
of transactions.
This really should be handled better overall, but this serves as a quick
solution until the rest of the mining updates are made.
This modifies the lifecycle of the background generator to use the
expected pattern for running subsystems based on contexts in the future.
In particular, this renames the Start function to Run and arranges for
it to block until the provided context is cancelled. This is more
flexible for the caller since it can easily turn blocking code into
async code while the reverse is not true.
Next, the server is modified to ensure it waits for the background
template generator to shutdown before shutting down itself to help
ensure an orderly shutdown.
While here, it also makes the various lifecycle debug messages more
consistent with the rest of the code base.
This factors out the code which deals with generating a unique block
template key to a separate function, modifies it to only be comprised of
the merkle root + timestamp, and renames the associated variables
accordingly. The separate function is useful because work submission
will ultimately need to create the same key.
The rationale for switching to only using the merkle root + timestamp is
as follows:
- It is more efficient to use smaller keys
- The combination is guaranteed to be unique for every template so long
as there are no more than one per second generated with the exact same
txns (which is already the case with the inclusion of the stake merkle
root in the key too)
- There are future plans to combine the stake root and merkle roots into
the merkle root field and repurpose the stake root
Finally, the naming is also being changed from ID to key because the
name templateID already has a distinct meaning in the rpcserver code in
how it applies to long polling clients and it is better to avoid
potential confusion.
This modifies the code which notifies subscribed clients about a new
block template to avoid creating an unnecessary goroutine since the
channel is single use and intentionally buffered to ensure the sender
can't block.
BgBlkTmplGenerator represents the background process that
generates block templates and notifies all subscribed clients
on template regeneration. It generates new templates based
on mempool activity for vote and non-vote transactions and
the time elapsed since last template regeneration.
This also adds a template pool to the background block
generator for recreating submitted blocks.
This implements the agenda for voting on the sequence lock fixes as
defined in DCP0004 along with consensus tests and mempool acceptance
tests to ensure its correctness.
It also modifies the mempool to conditionally treat all transactions
with enabled sequence locks as non standard until the vote passes at
which point the will become standard with the modified semantics
enforced.
The following is an overview of the changes:
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 6
- New block version on testnet is version 7
- Introduce a convenience function for determining if the vote passed
and is now active
- Enforce modified sequence lock semantics in accordance with the state
of the vote
- Modify the more strict standardness checks (acceptance to the mempool
and relay) to enforce DCP0004 in accordance with the state of the vote
- Make all transactions with enabled sequence locks non standard until
the agenda vote passes
- Add tests to ensure acceptance and relay behave according to the
aforementioned description
- Add tests for determining if the agenda is active for both mainnet and
testnet
- Add tests to ensure the corrected sequence locks are handled properly
depending on the result of the vote
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
This modifies simnet to include the latest consensus changes that have
been successfully voted into mainnet to date starting from its genesis
block. This means it is no longer necessary to vote those changes in to
simnet first before making use of them.
The changes entail removing the deployment entries for those votes,
updating the mining templates to start from version 5, which matches the
current mainnet version, and modifying the various sections in
blockchain which make decisions accordingly.
The chaincfg and blockchain modules are affected, but since their
version has already been bumped since their last release tags, they are
not bumped again.
this updates the function signatures of minimumMedianTime and
medianAdjustedTime and removes all chainState usage in
rpcserver.go in favour of using BestState.
Port of upstream commit 2274d36 (subset). In preparation of the
removal of chainState.
This modifies the BlockByHash function to return blocks regardless of
whether or not the block is on a side chain and removes the
FetchBlockByHash function since this change makes it redundant.
This is being changed to simplify the API and because there are no
current cases where calls into chain to retrieve a block need to be
limited to the main chain and if the need should ever arise, the caller
can simply make use of the MainChainHasBlock function to verify.
This removes the CheckConnectBlock function in favor of a new function
named CheckConnectBlockTemplate which more accurately reflects its
purpose of testing block template proposals and uses this fact to be
more restrictive about the allowed inputs and to avoid performing the
proof of work check.
In particular, the provided block template must only build from the
current tip or its parent or an error is returned.
All mining code has been updated to call the new function accordingly.
Finally, it also adds a full suite of tests for the new function using
the chaingen framework in order to ensure proper functionality.
This corrects the code to create a new template when first starting up
with aggressive mining enable before votes on the most recent block are
known.
First, the code is expecting the current best block, not its parent, and
second, the coinbase script should match what is created in the normal
path as well.
