This adds --peeridletimeout config option and updates the peer
connection to enforce idle timeouts using the connection's read
deadline. Associated testsand documentation have also been updated.
This updates the allowed votes range for mainnet, simnet
and regnet to coinbase maturity from tip. Testnet has been
exempted due to longer reorgs that have already happened
in testnet's history.
Upcoming changes constitute breaking public API changes to the chaincfg
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This introduces a new flag named allowunsyncedmining which allows checks
that prevent unsynchonrized mining to be disabled on testnet. This is
an advanced option and should not be used unless you understand exactly
why it exists and what it implications it carries.
This modifies blockchain to expose a new field named Checkpoints in the
Config struct which allows the checkpoints to use to be specified by the
caller instead of assuming the checkpoint associated with the current
chain parameters. This allows the caller to customize the checkpoints
as it sees fit, such as adding custom checkpoints or removing default
checkpoints.
In addition, this removes the DisableCheckpoints function in favor of
the caller simply configuring the chain instance without checkpoints
when they're disabled and updates the code to remove the associated flag
accordingly. This produces simpler code and is faster since the mutex
that was required to protect the aforementioned flag is no longer
required.
Finally, this also paves the way to significantly simplify the
checkpoint finding logic and make further progress towards the overall
effort to decouple the connection code from the download logic.
This removes any remaining orphan transactions that were sent by a peer
when it disconnects since it is extremely unlikely that the missing
parents will ever materialize from elsewhere.
Electron 6 has dropped support to the P-521 curve, so in order to
upgrade Decrediton dcrd needs to support generating curve P-256.
This adds a --tlscurve parameter similar to what exists in dcrwallet to
select which curve to use when generating the RPC certificates.
The helper function parseListeners has been changed to return a
slice of net.Addrs with Network() returning tcp4 or tcp6 instead of
returning two slices of IPv4 and IPv6 addresses to simplify calling
code. Also improves how local addresses are added to the address
manager when listening on wildcard addresses.
Also splits some newServer logic into new method initListeners.
Network interface parsing has been updated as
well.
Some indexes require all of the previous output scripts referenced as
inputs by the blocks they are indexing. Currently, the indexing code
relies on a concrete blockchain.UtxoViewpoint to provide that
information which makes the indexing code more rigid than it needs to be
and prevents the possibility of using other sources, such as the spend
journal, to provide the necessary scripts.
In addition, the index initialization code also currently requires
access to a concrete blockchain.BlockChain to query the current chain
state which also makes the code more rigid than it should ideally be.
Consequently, this modifies the indexing code to accept a PrevScripter
interface to remove the tight coupling to a specific type in the former
case, and introduces a new ChainQueryer interface to remove the tight
coupling in the latter case.
In order to fully remove the blockchain dependency from the indexers
package, this also moves the IndexManager interface from blockchain to
indexers and modifies it to use the new ChainQueryer interface in order
to provide the necessary information.
Finally, all code is updated to satisfy and use the interfaces.
This adds a warning when alternative DNS names are specified and the TLS
certificate key or cert already exist to make it clear the additional
names will NOT be included unless the certificate and key are deleted
first.
This properly associates ports applicable to discovered
IPs based on the listeners specified and the network
address type of the discovered host. Inbound peers
can now corroborate valid sugmissions from outbound
peer
Submissions are now bounded, address discovery also
accounts for setups using NAT.
Several upcoming changes constitute breaking public API changes to the
mempool module, therefore, this follows the process for introducing major
API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This modifies the lifecycle of the server to use the expected pattern
for running subsystems based on contexts in the future.
In particular, this renames the Start function to Run and arranges for
it to block until the provided context is cancelled. This is more
flexible for the caller since it can easily turn blocking code into
async code while the reverse is not true.
The new Run function ensures server waits for all goroutines that it
starts to shutdown before shutting down itself to help ensure an orderly
shutdown.
Finally, this removes the WaitForShutdown function from the server since
it is no longer needed.
Several upcoming changes constitute breaking public API changes to the
mining module, therefore, this follows the process for introducing major
API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
Several upcoming changes constitute breaking public API changes to the
blockchain module, therefore, this follows the process for introducing
major API breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
The purpose is to remove the dependency of blockmanager on serverPeer,
which is defined in the main package. Instead, we split out some of
the fields from serverPeer into a separate struct called peerSyncState
in blockmanager.go. While they are in the same package now, this
change makes it easier to move blockManager into its own package along
with peerSyncState. The blockManager tracks a map of Peer pointers to
the peer state and keeps it updated as peers connect and disconnect.
This refactors the RPC server to accept and take ownership of already
configured listeners and refactors the logic to setup those listeners to
the server. This mirrors the logic used by the connection manager and
is desirable since it is another step closer to being able to split the
RPC server code out into a separate package and will make it much
easier to internally test since it allows creating mock listeners.
This decouples the RPC server from the internal dcrd server to move
closer to being able to split it out into a separate package.
In order to accomplish this, it introduces an rpcserverConfig type and
several new interfaces, named rpcserverPeer, rpcserverConnManager, and
rpcserverBlockManager, which are necessary to break the direct
dependencies on the main server and block manager instances.
It also adds concrete implementations of the new interfaces and uses
them to configure the RPC server.
Ultimately, the RPC server should ideally be decoupled even more such
that all of the types in the configuration struct use interfaces instead
of the concrete types. Doing this would make the RPC server much easier
to internally test since it would allow creating lightweight stubs for
the various pieces.
This updates the getwork RPC to make use of the background template
generator infrastructure and removes code that is no longer needed as a
result.
This has a lot of benefits, many of which were discussed more in depth
in the previous commit that overhauled the background template
generator, however the following is a non-exhaustive overview that
highlights the major benefits of the change:
- Requests for updated templates during the normal mining process in
between tip changes will now be nearly instant instead of potentially
taking several seconds to build the new template on the spot
- When the chain tip changes, requesting a template will now attempt to
wait until either all votes have been received or a timeout occurs
prior to handing out a template which is beneficial for PoW miners,
PoS miners, and the network as a whole
- PoW miners are much less likely to end up with template with less than
the max number of votes which means they are less likely to receive a
reduced subsidy
- PoW miners will be much less likely to receive stale templates during
chain tip changes due to vote propagation
- PoS stakers whose votes end up arriving to the miner slightly slower
than the minimum number required are much less likely to have their
votes excluded despite having voted simply due to propagation delay
The following is a high level overview of the changes:
- Move the template pool to the work state struct
- Remove fields from work state struct that dealt with determining when
to update the template
- Introduce a new function named getWorkTemplateKey to construct a
unique template key for storing and retrieving delivered block
templates
- Make pruneOldBlockTemplates a method of the work state and clean it up
- Completely rework handleGetWorkRequest to use the background template
generator infrastructure
- Wait for a new template to be generated with a timeout when the best
chain tip to bias towards giving out templates with max possible
votes
- Ensure the timestamp is updated with each invocation
- Shallow copy the header (versus deep copying the entire template as
before) to avoid mutations to the shared templates
- Rework handleGetWorkSubmission
- Improve logging
- Use new template key to lookup the template used to reconstruct the
block
- Shallow copy the block from the template before updating it with the
solved versus the more expensive method used before
- Remove no longer used funcs and methods:
- extractCoinbaseExtraNonce
- extractCoinbaseTxExtraNonce
- deepCopyBlockTemplate
This removes the getblocktemplate and its helpers from the codebase.
Ongoing mining updates focused on the voting/block validation process
with respect to generating block templates for getwork makes it the
better option for decred. Also getblocktemplate rpc was buggy and
has been disabled for a while.
Some lint related issues have been addressed as well.
Currently, the filters provide two different serialization formats per
version. The first is the raw filter bytes without the number of items
in its data set and is implemented by the Bytes and FromBytesV1
functions. The second includes that information and is implemented by
the NBytes and FromNBytesV1 functions.
In practice, the ability to serialize the filter independently from the
number of items in its data set is not very useful since that
information is required to be able to query the filter and, unlike the
other parameters which are fixed (e.g. false positive rate and key), the
number of items varies per filter. For this reason, all usage in
practice calls NBytes and FromNBytesV1.
Consequently, this simplifies the API for working with filters by
standardizing on a single serialization format per filter version which
includes the number of items in its data set.
In order to accomplish this, the current Bytes and FromBytesV1 functions
are removed and the NBytes and FromNBytesV1 functions are renamed to
take their place.
This also updates all tests and callers in the repo accordingly.
This refactors the primary gcs filter logic into an internal struct with
a version parameter in in order to pave the way for supporting v2
filters which will have a different serialization that makes them
incompatible with v1 filters while still retaining the ability to work
with v1 filters in the interim.
The exported type is renamed to FilterV1 and the new internal struct is
embedded so its methods are externally available.
The tests and all callers in the repo have been updated accordingly.
This removes the unused and undesired FromPBytes and FromNPBytes
functions and associated tests from the gcs module in preparation for
upcoming changes aimed to support new version filters for use
in header commitments.
Since these changes, and several planned upcoming ones, constitute
breaking pubic API changes, this bumps the major version of the gcs
module, adds a replacement for gcs/v2 to the main module and updates all
other modules to make use of it.
It also bumps the rpcclient module to v5 since it makes use of the
gcs.Filter type in its API, adds a replacement for rpcclient/v5 to the
main module and updates all other modules to make use of it.
Note that this also marks the start of a new approach towards handling
module versioning between release cycles to reduce the maintenance
burden.
The new approach is as follows.
Whenever a new breaking change to a module's API is introduced, the
following will happen:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already
done since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the
new version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
Finally, once the repo is frozen for software release, all modules will
be tagged in dependency order to stabilize them and all module
replacements will be removed in order to ensure releases are only using
fully tagged and released code.
This discovers the network address(es) of the daemon
through connected outbound peers. The address(es)
discovered are advertised to subsequent connecting peers.
This updates all code in the main module to use the latest major modules
versions to pull in the latest updates.
A more general high level overview of the changes is provided below,
however, there is one semantic change worth calling out independently.
The verifymessage RPC will now return an error when provided with
an address that is not for the current active network and the RPC server
version has been bumped accordingly.
Previously, it would return false which indicated the signature is
invalid, even when the provided signature was actually valid for the
other network. Said behavior was not really incorrect since the
address, signature, and message combination is in fact invalid for the
current active network, however, that result could be somewhat
misleading since a false result could easily be interpreted to mean the
signature is actually invalid altogether which is distinct from the case
of the address being for a different network. Therefore, it is
preferable to explicitly return an error in the case of an address on
the wrong network to cleanly separate these cases.
The following is a high level overview of the changes:
- Replace all calls to removed blockchain merkle root, pow, subsidy, and
coinbase funcs with their standalone module equivalents
- Introduce a new local func named calcTxTreeMerkleRoot that accepts
dcrutil.Tx as before and defers to the new standalone func
- Update block locator handling to match the new signature required by
the peer/v2 module
- Introduce a new local func named chainBlockLocatorToHashes which
performs the necessary conversion
- Update all references to old v1 chaincfg params global instances to
use the new v2 functions
- Modify all cases that parse addresses to provide the now required
current network params
- Include address params with the wsClientFilter
- Replace removed v1 chaincfg constants with local constants
- Create subsidy cache during server init and pass it to the relevant
subsystems
- blockManagerConfig
- BlkTmplGenerator
- rpcServer
- VotingWallet
- Update mining code that creates the block one coinbase transaction to
create the output scripts as defined in the v2 params
- Replace old v2 dcrjson constant references with new types module
- Fix various comment typos
- Update fees module to use the latest major module versions and bump it v2
Currently, block templates that provide work to PoW miners are generated
in response to them calling the getwork RPC. However, unlike in a pure
PoW system, in Decred, when a new block is connected, a new block
template extending that block can't be generated until the minimum
number of required votes for it has been received. This poses a
challenge for PoW miners because they are typically unaware of when
votes are received. Consequently, miners poll getwork waiting until
they see a new template generated when a new block is connected.
Since only a minimum of 3 votes is required to build a template and new
templates are only generated in response to the miners calling getwork,
as just described, miners often, unfortunately, end up receiving a
template that only has 3 votes and begin working on it. Worse, since
many new miners are not well versed in the intricacies of Decred voting,
they often aren't aware they really need to continue polling for new
votes and immediately switch over to the new template with more votes in
it in order to avoid receiving a reduced subsidy. This often results in
new miners producing blocks with 3 votes.
Moreover, blocks with less than all 5 votes are not only undesirable for
the PoW miners because they receive reduced subsidy, they're also
undesirable for PoS stakers since it means the votes that were not
included don't receive the reward despite having voted simply due to a
propagation delay.
Another issue with the existing approach is that since it takes a bit of
time to generate and validate templates, particularly as the number of
transactions they include rises, miners periodically requesting updated
work to include the latest transactions have to wait for the template to
be built when they call getwork once the existing cached template is
expired. This can result in undesirable delays for them.
In order to address the need to wait for templates to be built in that
case, there was recently some preliminary work that implemented a
background template generator which asynchronously generates templates
in response to events with the intention of allowing access to the
latest one without said random delays in the future.
However, that preliminary work did not address the issues around vote
propagation such as the tendency for miners to end up with the first
block template only containing the 3 fastest-propagating votes, nor did
it have robust handling for all potential corner cases and error
conditions.
Consequently, this overhauls the background block template generator to
add support for intelligent vote propagation handling, handle chain
reorganization to alternative blocks in the case the current tip is
unable to obtain enough votes, provide a subscription for a stream of
template updates, handle template generation errors, consider the
synchronization state as a part of determining if the chain is current,
track the reason for template updates, block current template retrieval
during operations that are in the process of making it stale, and
correct several corner cases.
It should be noted that this only implements the infrastructure and does
not switch getwork or the CPU miner over to use the background templates
yet. This will be done in future commits.
The following is a high-level overview of the semantics this implements:
- Generate new templates immediately when prior to stake validation
height since no votes are required
- Do not generate templates for intermediate blocks during a chain
reorganization
- Do not generate templates before the chain is considered synchronized
- Prefer to create templates with maximum votes through the use of a
timeout once the minimum votes have been received to provide the votes
an opportunity to propagate with a fallback for immediate generation
as soon as all votes are received
- In the case the timeout expires and a template is created with less
than the maximum number of votes, generate a new template
immediately upon receiving more votes for the block it extends
- In the event there are competing blocks at the current tip, prefer to
build a template on the first seen block so long as it receives the
minimum number of required votes within a few seconds
- Generate new templates immediately when a block is disconnected to
support future planned chain invalidation capabilities
- Generate new templates periodically when there are new regular transactions
to include
- Schedule retries in the rare event template generation fails
- Allow clients to subscribe for updates every time a new template is
successfully generated along with a reason why it was generated
- Provide direct access to the most-recently generated template
- Block while generating new templates that will make the current template
stale (e.g. new parent or new votes)
The following is a high-level overview of the changes:
- Initialize PRNG once instead of at every invocation
- Implement an asynchronous queue for all events to ensure normal chain
and vote processing is not blocked
- Provide ability to subscribe for all template updates over a single
channel to ensure none can indadvertently be missed as is possible with
the current design
- Always deliver existing template upon registration
- Drop templates if the receiver falls too far behind in the same way
tickers work
- Introduce tracking the reason for template updates so that it can
ultimately be presented to miners to allow them to make better
decisions about when to force their workers to switch
- Consider the current network sync state when checking if the chain is
current
- Introduce a regen handler state such as timeouts, blocks to monitor
for votes, and the block the next template should extend
- Add logic for selectively reacting to votes on the block the current
template is extending, the current tip, and alternate competing blocks
at the current tip using the aforementioned semantics
- Perform all template generation in separate goroutines with a
cancellable context
- Cancel any in progress templates that are being generated whenever
generating a new one
- Introduce blocking current template retrieval when a new template that
would cause the existing one to become stale is being generated
- Modify periodic regen handling to use a resettable timer for better
efficiency and more fine grained control
- Remove template pool as that is something that should be handled by
the code that is actually handing the templates out
- Rename and export the event notification funcs to make it clear
they're not internal functions and also make it easier to eventually
move the code into the mining package
- Expand and improve comments
- Store and return template generation errors
- Schedule retry in the case of failure
- Correct several cases that were not being handled correctly and some
undesirable behaviors such as block disconnects (as opposed to
reorgs), reorganization to side chains, and notifying with stale
templates
This modifies the lifecycle of the background generator to use the
expected pattern for running subsystems based on contexts in the future.
In particular, this renames the Start function to Run and arranges for
it to block until the provided context is cancelled. This is more
flexible for the caller since it can easily turn blocking code into
async code while the reverse is not true.
Next, the server is modified to ensure it waits for the background
template generator to shutdown before shutting down itself to help
ensure an orderly shutdown.
While here, it also makes the various lifecycle debug messages more
consistent with the rest of the code base.
BgBlkTmplGenerator represents the background process that
generates block templates and notifies all subscribed clients
on template regeneration. It generates new templates based
on mempool activity for vote and non-vote transactions and
the time elapsed since last template regeneration.
This also adds a template pool to the background block
generator for recreating submitted blocks.
