This adds support for the getblockchaininfo RPC to rpcclient,
adding asynchronous and blocking functions to rpcclient.Client.
The rpcclient module version is bumped to v1.1.0.
This commit performs the necessary modifications to hook the fee
estimation facility added by the previous commit into the node software.
The block manager is modified to notify the estimator of all new blocks,
such that their transactions can be accounted for, and the mempool is
modified to relay transactions entering and leaving it, so that those
transactions can be tracked.
The results of the estimator can be queried by issuing estimatesmartfee
rpc commands to the node.
This adds the first version of the fees package, responsible for
performing fee estimation of network transactions.
The main goal of fee estimation is to allow the usage of dynamic fees
by wallets, contingent on block contention and the desired confirmation
range for a given transaction.
This version was based on bitcoin core fee estimation.
This corrects an ultra rare and nearly impossible to hit edge case where
the case of r == N in a deterministic signature could lead to an invalid
signature being produced versus returning an error as desired.
This bumps the major version of the JSON RPC server to reflect the
breaking change of reordering the reorg and block connected
notifications. The reorg notification now indicates a successful reorg
was completed rather than initiated.
This modifies the forceHeadReorganization function to remove checks that
are now duplicates due to the recently updated conversion of the
reorganization logic.
Also, correct the exported comment while here to explain what the
function does as opposed to referring to an unexported function that
won't show up in the generated documentation.
This modifies the version 5 migration code to update the database
version atomically with setting a reindex tip and resetting the best
chain tip back to the genesis block so the chain can be reindexed.
This is necessary because, unlike the modifications that come before it,
storing the current chain tip and resetting it is not idempotent, so
lack of atomicity in those updates can lead to failure to recover in
unclean shutdown circumstances.
This adds code to migrate the database to the new schema required by the
recent utxo set semantics reversal changes.
It involves removing the existing utxo set, spend journal, ticket
database, various indexes, resetting the best chain tip back to the
genesis block, and finally performing a full reindex. The process can
be interrupted at any point and future invocations will resume from the
point it was interrupted.
In addition, some new infrastructure to remove the ticket database and
support for index versions is added to support the migration.
This modifies the chain reorganization logic to directly perform the
reorg one block at a time with rollback in the case of failure, as
opposed to the existing memory-based two-step approach, so that it is
more optimized for the typical case, better handles large reorgs, gives
the ability to implement better caching strategies, and helps provide a
path to decouple the chain processing and connection code from the
download logic. It also removes the cached stxos from the view since
the aforementioned changes make them no longer necessary.
A side effect of these changes is that it is no longer possible to know
if a reorg will succeed before actually performing it, so the
NTReorganization notification is now sent after a successful reorg. The
notification really should have been sent after the reorg before anyway.
Prior to these changes, chain reorganization used a two-step approach
such that the first step involved checking all of the blocks along the
reorg path in memory and then actually performing the reorg in a second
step if those checks succeeded. While that approach does have some
benefits in terms of avoiding any intermediate mutation to the current
best chain for failed reorgs, and thus not requiring a rollback in that
case, it also has some disadvantages such as not scaling well with large
reorgs, being more difficult to make use of different caching
strategies, and hindering the ability to decouple the connection code
from the download logic.
In a certain sense, the approach this replaces assumed that a reorg
would fail and took measures to detect that condition prior to
performing the reorg, while the new approach assumes the reorg will
succeed and rolls back the changes in the very rare case it doesn't.
This is an acceptable and safe assumption because the proof-of-work
requirements make it exceedingly expensive to create blocks that are
valid enough to trigger a reorg yet ultimately end up failing to
connect, thus miners are heavily disincentivized from creating such
invalid blocks and attackers are also unable to easily create such
blocks either. Even in the case of attack, the only result would be
nodes performing slightly more database updates than the existing
approach.
The following results show the difference between performing the large
reorg full block tests before and after these changes:
before: 4.3GB memory usage, 2m18.629s to complete
after: 2.8GB memory usage, 2m04.056s to complete
As can be seen, the new approach takes much less memory and is also a
bit faster as well.
This modifies the way the unspent transaction output set is handled to
reverse its current semantics so that it is optimized for the typical
case, provides simpler handling, and resolves various issues with the
previous approach. In addition, it updates the transaction, address,
and existsaddress indexes to no longer remove entries from blocks that
have been disapproved as, in all cases, the data still exists in the
blockchain and thus should be queryable via the indexes even though
there is special handling applied which treats them as if they did not
exist in certain regards.
Prior to this change, transactions in the regular tree were not applied
to the utxo set until the next block was processed and did not vote
against them. However, that approach has several undesirable
consequences such as temporarily "invisible" utxos that are actually
spendable, disapproved transactions missing from indexes even though
they are still in the blockchain, and poor performance characteristics.
In a certain sense, the previous approach could be viewed as the
transactions not being valid until they were approved, however, that is
not really true because it was (and still is) perfectly acceptable to
spend utxos created by transactions in the regular tree of the same
block so long as they come before the transactions that spend them.
Further, utxos from a transaction in the regular tree of a block can be
spent in the next block so long as that block does not disapprove them,
which further illustrates that the utxos are actually valid unless they
are disapproved.
Consequently, this modifies that behavior to instead make the utxo set
always track the most recent block and remove the regular transactions
in the parent when a block votes against them. This approach is
significantly more efficient for the normal case where the previous
block is not disapproved by its successor.
Also, the terminology is changed in several places to refer to
disapproved blocks and transaction trees as opposed to invalid, because
invalid implies the tree/block is malformed or does not follow the
consensus rules. On the contrary, when a block votes against its
parent, it is only voting against regular transaction tree of the
parent. Both the block and transaction tree are still valid in that
case, only the regular transaction tree is treated as if it never
existed in terms of effects on the utxo set and duplicate transaction
semantics.
High level overview of changes:
- Modify the utxo viewpoint to reverse semantics as previously described
- Remove all code related to stake viewpoints
- Change all block connection code in the viewpoint to first undo all
transactions in the regular tree of the parent block if the current
one disapproves it then connect all of the stake txns followed by
the regular transactions in the block
- NOTE: The order here is important since stake transactions are not
allowed to spend outputs from the regular transactions in the same
block as the next block might disapprove them
- Change all block disconnection code in the viewpoint to first undo
all the transactions in the regular and stake trees of the block
being disconnected, and then resurrect the regular transactions in
the parent block if the block being disconnected disapproved of it
- Introduce a new type named viewFilteredSet for handling sets
filtered by transactions that already exist in a view
- Introduce a function on the viewpoint for specifically fetching the
inputs to the regular transactions
- Update mempool block connection and disconnection code to match the
new semantics
- Update all tests to handle the new semantics
- Modify the best state number of transactions to include all
transactions in all blocks regardless of disapproval because they
still had to be processed and still exist in the blockchain
- Remove include recent block parameter from mempool.FetchTransaction
since the utxoset now always includes the latest block
- This also has the side effect of correcting some unexpected results
such as coinbases in the most recent block being incorrectly
reported as having zero confirmations
- Modify mempool utxo fetch logic to use a cached disapproved view, when
needed, rather than recreating the view for every new transaction
added to it
- Update spend journal to include all transactions in the block instead
of only stake transactions from the current block and regular
transactions from the parent block
- Modify tx and address indexes to store the block index of each tx
along with its location within the files and update the query
functions to return the information as well
- Change the tx, address, and existsaddress indexes to index all
transactions regardless of their disapproval
- This also corrects several issues such as the inability to query and
retrieve transactions that exist in a disapproved block
- Update all RPC commands that return verbose transaction information
to set that newly available block index information properly
- Rename IsRegTxTreeKnownDisapproved in the mining.TxSource interface to
IsRegTxTreeKnownDisapproved
- NOTE: This will require a major bump to the mining module before
the next release
- Rename several utxoView instances to view for consistency
- Rename several variables that dealt with disapproved trees from
names that contained Invalid to ones that contain Disapproved
NOTE: This does not yet have database migration code and thus will
require a full chain download. It will exit with error in the case you
attempt to run it against an existing v4 database. The new database it
creates will be v5, so attempting to run an older version will reject
the new database to prevent corruption.
The database migration will be added in a separate commit.
This separates the block and header context checks to make it more
explicit exactly which checks require the availability of the full data
for all ancestors and which checks do not.
Currently, all ancestor data is always available due to the way the
download and block processing logic is implemented, however, ultimately,
the plan is to decouple the chain processing and connection code from the
download logic. In order to help pave the way towards that goal, this
more clearly delineates the checks by redefining the current meaning of
the concept of contextual checks and creating a new concept of
positional checks.
Positional checks are defined as those which depend on the position of a
block or header within the block chain and have all of the ancestor
block headers available, but do NOT have all of the full block data for
all of the ancestor blocks.
Contextual checks are now defined as those which depend on having the
full block data for all of the ancestors available, which also implies
all of the ancestor block headers are available too.
To that end, this introduces two new functions named
checkBlockHeaderPositional and checkBlockPositional, moves all of the
checks which adhere to the aforementioned semantics to the new
functions, and updates all call sites which invoke the contextual
variants to call the positional variants first.
This modifies the code which enforces a maximum number of peers to
always allow whitelisted inbound peers even when they would exceed the
maximum number of peers to ensure that operators can always allow their
own clients.
When an accepted block is attached to the main chain, potentially
three notifications are emitted: NTBlockConnected,
NTSpentAndMissedTickets, and NTNewTickets. This change moves
NTBlockConnected to being the first notification, followed by
NTSpentAndMissedTickets and NTNewTickets.
This is important for applications which react to the stake
notifications since they will have already been notified of the most
recently connected block. For example, dcrwallet uses the
spentandmissedtickets JSON-RPC notification to create revocation
transactions for missed votes. However, if the missed tickets are
notified before the connected block, a race is possible where the
revocations are rejected as they double spend a vote on what wallet
believes is still the main chain tip block.
This is considered a patch bump to the JSON-RPC API semantic version
due to affecting and improving the ordering of the blockconnected and
spentandmissedtickets notifications.
This reworks the stake node handling logic a bit to clean it up,
slightly optimize it, and to make it easier to decouple the chain
processing and connection code from the download logic in the future.
To accomplish this, the fetchStakeNode dependence on the
getReorganizeNodes function is removed in favor of directly iterating
the block nodes in the function as needed. Not only is this more
efficient, it also allows the function to return stake nodes for
branches regardless of their validation status. Currently, this is
irrelevant due to the connection and download logic being tightly
coupled. However, it will be necessary in the future when those are
separated.
Also, the flushing and pruning logic is modified to no longer rely on
the download logic being tightly coupled to the the connection logic.
In particular, a new function named flushBlockIndex is added as a
wrapper to the index flush function which populates stake information in
a node before flushing it to the database as needed, and all flushing
invocations use the wrapper instead.
Finally, the stakeUndoData field is removed since it is only used to
avoid some database loads on reorgs that are large enough to exceed the
pruning depth, which never happens in practice anyway, so there is no
point in using up extra memory for it.
This updates the code which deals with marking block nodes modified when
setting and clearing status flags to only mark them modified if the
status flags actually change.
This is more efficient as it prevents the need to write nodes that
haven't actually changed to the database.
This simplifies the CheckConnectBlockTemplate by making use of the fact
that a template may only build off the current tip or the parent of the
current tip in order to remove the reorganization detach loop and all
code to attach nodes.
This can be done because there can never be any blocks that need to be
attached given the aforementioned constraints, and the only two
possibilities for the blocks that need to be detached are none in the
case the the template is building on the current tip, or the current tip
itself in the case the template is building on its parent.
In other words, there will always be 0 nodes to attach and either 0 or 1
node(s) to detach. Consequently, it is not necessary to include any
attach code nor have a detach loop.
This modifies the getblock and getblockheader RPC results to include the
total number of hashes expected to produce the chain up to the requested
block in hex and updates the JSON-RPC API documentation accordingly.
This modifies the getblock and getblockheader RPC results to be a little
more consistent in their ordering, to include the extra data field in
the getblockheader results, and to update the JSON-RPC API documentation
for both to match reality. The documentation was missing several fields
and had some typos.
This removes the CheckWorklessBlockSanity function since nothing uses
it anymore. It was used for testing templates and unit tests in the
past, but the former is now handled by CheckConnectBlockTemplate and the
latter is handled by the full block tests.
It should be noted that this is a breaking change to the API and thus
will need a major version bump of the blockchain module to be published
before the next release.
This cleans up code recently added to support the --altdnsnames flag and
DCRD_ALT_DNSNAMES environment variable to match the project standards,
simplify it, and correct some issues with standalone test binaries.
- Gets rid of the setup func which had several issues:
- It was creating a file that was not needed (and without even
checking the error)
- Since it was parsing flags and missing with os.Args in a func, it
was not possible to individual run the test funcs without failure
due to the additional flags
- Performs the flag parsing and removal in an init func that only runs
once when the tests are run (and before TestMain)
- Avoids creating empty objects when nil will suffice
- Uses full sentences and periods in the comment and properly spaces
them out so they are consistent with the rest of the code
- Adds comments to all of the added test functions as required by the
code contribution guidelines
- Closes the create temp files before trying to write to them in the cert
generation
- Defers the removal of the temp files directly after they are created so
they are removed properly if the next one fails to create
- Uses consistent camel case in the variable names
This renames the unexported and exported versions
of ThresholdState to NextThresholdState as well as
related test helpers.
NextThresholdState better describes the function
because the threshold state being returned is for the
block after the provided block hash.
- rpcserver: Adds ability to allow alternative dns names for TLS.
- Adds AltDNSNames config for passing additional hostnames to the
TLS certificate generate method. Without this change there is no
way to pass in alternative dns names for the rpc server.
- Additionally this commit allows the user to set 'DCRD_ALT_DNSNAMES'
as an environment variable when starting dcrd. This is useful for
docker and other app runtimes that require apps to be 12 factor.
These errors are only used internally and thus should not be exported.
While here, it also removes the completely unused VoteBitsNotFound
error.
It should be noted that this is a breaking change to the API and thus
will need a v2 major version bump of the chaincfg module to be published
before the changes can be externally consumed.
This modifies the ChainTips function in blockchain to return the results
using a type defined in the blockchain package itself instead of
directly returning a dcrjson type. This is preferable since nothing
else in the module depends on dcrjson and therefore allows the
dependency on the dcrjson module to be broken.
It also updates the RPC server that makes use of the function to perform
the necessary conversions.
Finally, the associated module hierarchy documentation is updated to
properly remove the no longer required dependency.
It should be noted that this is a breaking change to the API and thus
will need a v2 major version bump of the blockchain module to be
published before the changes can be externally consumed.
This modifies simnet to include the latest consensus changes that have
been successfully voted into mainnet to date starting from its genesis
block. This means it is no longer necessary to vote those changes in to
simnet first before making use of them.
The changes entail removing the deployment entries for those votes,
updating the mining templates to start from version 5, which matches the
current mainnet version, and modifying the various sections in
blockchain which make decisions accordingly.
The chaincfg and blockchain modules are affected, but since their
version has already been bumped since their last release tags, they are
not bumped again.
This modifies the majority of the tests that make use of chain
parameters and the RPC tests to use the resurrected regression test
network.
It also bumps the affected module versions as follows:
- github.com/decred/dcrd/txscript@v1.0.2
- github.com/decred/dcrd/blockchain/stake@v1.0.3
- github.com/decred/dcrd/mempool@v1.0.2
The blockchain and dcrutil modules are also affected, but since their
version has already been bumped since their last release tags, they are
not bumped again.
This resurrects the regression test network that was removed before
initial launch although it really should not have been. The simulation
test network and the regression test network do not serve the same
purpose. Specifically, the regression test network is intended for unit
tests, RPC server tests, and consensus tests. On the other hand, the
simulation test network is intended for private use within a group of
individuals doing simulation testing and full integration tests between
different applications such as wallets, voting service providers, mining
pools, block explorers, and other services that build on Decred.
Keeping the concerns separate will allow the simulation test network to
be modified in ways such as activating consensus changes that have been
successfully voted into mainnet without also needing to vote them in on
the simulation test network while still preserving the ability for the
unit tests to properly test the voting semantics and handling to help
prevent regressions.
In addition to resurrecting the regression test network, this also fully
fleshes out new values for the various addresses prefixes (Rk, Rs, Re,
etc), HD key prefixes (rprv, rpub), and treasury multisig details.
As a part of resurrecting the network, a new CLI flag `--regnet` is
added to allow the RPC test harness connect to a running instance, the
areas of the code which involve votes have been modified to allow the
votes to apply to the new network, and tests have been added to the
relevant modules.
This bumps the affected module versions as follows:
- github.com/decred/dcrd/wire@v1.2.0
- github.com/decred/dcrd/chaincfg@v1.2.0
- github.com/decred/dcrd/dcrutil@v1.2.0
- github.com/decred/dcrd/hdkeychain@v1.1.1
The blockchain module is also affected, but since its version has
already been bumped since the last release tag, it is not bumped again.
Finally, this does not include switching unit tests or the RPC test
harness over the new network since that will be done in a separate
commit.
This modifies the code in run_tests.sh which obtains the stripped list
of modules to test based on the output of the go list command to work
with MacOS as well. It seems that sed on MacOS does not like the "or"
regexp syntax, so this modifies it to perform each branch separately.
This adds some tests to ensure blocks that disapprove the regular
transaction tree of the parent as follows:
- Attempt to spend an output from a disapproved tree in the block that
disapproves it is rejected
- Reorg to a side chain that contains two consecutive blocks that
disapprove their parents is accepted without error
- Reorg back to the original chain which causes the disapproving blocks
to be removed is accepted without error
While here, it adds a new function named AssertTipDisapprovesPrevious
which, as its name suggests, allows the caller to assert the vote bits
in the header disapprove of the previous block.
This modifies the test chain generator infrastructure that automatically
tracks missed votes and generates revocations for them to only generate
a single revocation and properly handle reorgs by removing revoked
tickets from the pool of missed tickets when a block is connected and
adding back any revoked tickets to the pool of missed tickets when a
block that contains them is disconnected.
In addition, it modifies the handling of stake ticket purchases such
that the associated ticket purchases are created after the munge
functions run to ensure all changes are accurately account for. In
particular, the indexes of the ticket purchase transactions within the
block can change if the munger adds or removes transactions before them.
This allows the large reorg tests to work as intended again.
This adds a pull request template which provides a checklist and links
to the code contribution guidelines to help contributors ensure they are
adhering to project standards.
