This corrects the tests in the rpctest package to work properly with the
latest code.
The tests were not failing by default because many of the tests in the
rpctest package are not executed when only performing short tests which
is the default mode used in run_tests.sh.
This adds a new method to ScriptBuilder named AddOpsUnchecked that is
similar to AddDataUnchecked in that it allows the limits that are
ordinarily impose to be bypassed for regression testing purposes where
sizes are intentionally made larger than allowed.
It also updates the test code to make use of the method instead of
reaching into the internals to manually concatenate the opcodes.
This adds benchmarks for decoding and generating the payment script for
each supported version 0 address type.
This is part of a series of commits to fully implement the stdaddr
package.
This adds a new type to fully support version 0 pay-to-script-hash
addresses along with associated tests.
This is part of a series of commits to fully implement the stdaddr
package.
This adds a new type to fully support version 0
pay-to-pubkey-hash-schnorr-secp256k1 addresses along with associated
tests.
This is part of a series of commits to fully implement the stdaddr
package.
This adds a new type to fully support version 0
pay-to-pubkey-hash-ed25519 addresses along with associated tests.
This is part of a series of commits to fully implement the stdaddr
package.
This adds a new type to fully support version 0
pay-to-pubkey-hash-ecdsa-secp256k1 addresses along with associated
tests.
This is part of a series of commits to fully implement the stdaddr
package.
This adds a new type to fully support version 0
pay-to-pubkey-schnorr-secp256k1 addresses along with associated tests.
This is part of a series of commits to fully implement the stdaddr
package.
This adds a new type to fully support version 0 pay-to-pubkey-ed25519
addresses along with associated tests.
This is part of a series of commits to fully implement the stdaddr
package.
This adds a new type to fully support version 0
pay-to-pubkey-ecdsa-secp256k1 addresses along with associated tests.
This is part of a series of commits to fully implement the stdaddr
package.
This adds the overall infrastracture for decoding version 0 addresses
along with associated testing infrastructure.
Note that, in order to help ease the review process, this only adds the
relevant infrastructure and does not yet support any version 0 address
types. Each supported version 0 address type will be added in future
commits.
This is part of a series of commits to fully implement the stdaddr
package.
The current code for handling standard addresses implemented in dcrutil
was written many years ago prior a wide variety of changes and several
new features added by Decred. As a result, it entirely lacks support
for some features and supports others in a roundabout and non-intuitive
way.
Specifically, it does not support or provide a clean path to enable
support for different script versions and the way they are handled in
stake transactions is entirely non-intuitive.
Further, back when the original address code was implemented, it was
necessary to implement script creation in the txscript package which led
to the current design of providing methods such as
txscript.PayTo{AddrScript,SStx,SStxChange}, and others, which need to
type assert the specific concrete types of addresses in order to produce
the necessary scripts. This, unfortunately, effectively negates the use
of an interface to support generic addresses because it means callers,
such as dcrwallet, are not able to implement their own types without
somewhat invisibly breaking the script creation.
Finally, the aforementioned blending of the address code into txscript
has led to confusion regarding what is considered standard and what is
considered consensus which has tripped up several contributors over the
years.
This is part of a series of commits that aims to resolve all of the
aforementioned issues by introducing a new package named stdaddr which
entirely reworks the way addresses are handled.
For the time being, the package is introduced into the internal staging
area for initial review.
The following provides an overview of some of the key features of the
new design:
- Supports versioned addresses
- Produces scripts directly via methods implemented on underlying types
- Provides direct support for creation of the scripts necessary for the
staking system
- Uses a capabilities-based approach via interfaces so callers can
cleanly and generically determine under what circumstances addresses
can be used
- Allows callers to create their own concrete address types without
worrying about breaking the existing ones
- Clearly denotes that addresses are a standardized construction that
must not be used directly in consensus code
In order to help ease the review process, this commit only contains the
overall generic infrastructure without adding support for any specific
address types. Each supported version 0 address type will be added in
future commits.
This corrects the generic public key address constructor to pass the
actual serialized ed25519 public key to specialized constructor instead
of incorrectly including the initial byte that identifies the signature
suite and y bit oddness.
With this update, when dcrd is run with the LOGFLAGS=nodatetime
environment variable, the logs will not contain any date or time.
This is useful when logs already only end up in another logging system
which provides its own logged dates and times.
Fast block announcements were added a few years ago as a way to increase
the propagation speed of new blocks that extend the main chain through
the network to significantly reduce voting latency as well as how
quickly proof-of-work miners discover new blocks. This is desirable
behavior, however, it should only apply to full nodes since only they
are equipped to deal with the fast announcements.
As a case in point, header commitments and committed block filters can
pose a problem for lightweight clients that rely on them in the case of
fast block announcements because lightweight clients tend to request the
filters as soon as they learn about the new blocks and those filters
have not been created yet when fast block announcements are sent.
Since there is network latency involved in the request process, the
result is that sometimes the filters are available by the time the
requests from lightweight clients are received and sometimes they are
not.
In order to help prevent the aforementioned case, and other similar
ones, this modifies the code that deals with block announcements to only
send fast block announcements to peers that advertise themselves as full
nodes instead of all nodes and separates out the block announcement
logic slightly to make future enhancements easier in the process.
It also modifies the filtering of duplicate block announcements to work
on a per peer basis since not all peers receive the fast block
announcement now.
This adds tests for the UtxoCache Initialize method. These tests
include recovery scenarios that are particularly hard to simulate to
ensure that those code paths are not broken in the future.
In order to test recovery scenarios, this introduces a test utxo cache
that allows for toggling flushing on and off.
Additionally, this adds an ExpectUtxoSetState method to chaingenHarness
that allows for easily validating the last flushed block for the utxo
set.
This adds a UtxoCacher interface so that alternative utxo cache
implementations can be provided. In particular, this will be used to
provide a mock implementation for testing in order to more easily
simulate various scenarios.
In addition to introducing the UtxoCacher interface, this updates
BlockChain and UtxoViewpoint to use the interface rather than the
concrete type.
This changes the InitUtxoCache method on the BlockChain type to a
Initialize method on the UtxoCache type instead. This simplifies
providing alternative implementations for testing since the Initialize
method deals with internal fields of the UtxoCache type.
This updates the utxo cache flush tests to use a mock function for
getting the current time. Without this the flush tests will fail if
the flush happens fast enough for time.Now to not have elapsed any time.
This updates the error messages in the chainio tests to consistently use
the newer style of `t.Errorf("%q: ...", test.name)` rather than
including extra information for the function name, test index, etc.
The extra information is no longer needed to pinpoint an error since the
test framework now shows proper line numbers for any errors that occur.
This updates the minimum recommended memory (RAM) from 1GB to 2GB in the
main README.md. The minimum recommended memory is being increased due
to the introduction of the utxo cache.
This adds full test coverage to the UtxoCache type and its methods.
Additionally, since this uses the testing Cleanup function that was
introduced in Go 1.14, this bumps the required go version for the
blockchain package from 1.13 to 1.14.
UtxoCache is an unspent transaction output cache that sits on top of the
utxo set database and provides significant runtime performance benefits
at the cost of some additional memory usage. It drastically reduces the
amount of reading and writing to disk, especially during initial block
download when a very large number of blocks are being processed in quick
succession.
The UtxoCache is a read-through cache. All utxo reads go through the
cache. When there is a cache miss, the cache loads the missing data
from the database, caches it, and returns it to the caller.
The UtxoCache is a write-back cache. Writes to the cache are
acknowledged by the cache immediately but are only periodically flushed
to the database. This allows intermediate steps to effectively be
skipped. For example, a utxo that is created and then spent in between
flushes never needs to be written to the utxo set in the database.
Due to the write-back nature of the cache, at any given time the
database may not be in sync with the cache, and therefore all utxo reads
and writes MUST go through the cache, and never read or write to the
database directly.
An overview of the changes is as follows:
- Add UtxoCache and UtxoCacheConfig struct types and NewUtxoCache method
- Update server to create the utxo cache with the configured max size
and pass to the block chain instance that is created
- Update all test block chains to create a utxo cache
- Add FetchEntry to UtxoCache
- FetchEntry returns the specified transaction output from the utxo
set
- If the output exists in the cache, it is returned immediately.
Otherwise, it uses an existing database transaction to fetch the
output from the database, caches it, and returns it to the caller.
- Add AddEntry to UtxoCache
- AddEntry adds the specified output to the cache
- Add SpendEntry to UtxoCache
- SpendEntry marks the specified output as spent
- Remove entries that are marked as fresh and then subsequently spent.
This is an optimization to skip writing to the database for outputs
that are added and spent in between flushes to the database.
- Update UtxoViewpoint to hold the UtxoCache
- Update fetching entries from the database to fetch entries from the
cache instead
- Add Commit to UtxoCache
- Commit updates all entries in the cache based on the state of each
entry in the provided view
- All entries in the provided view that are marked as modified and
spent are removed from the view
- Additionally, all entries that are added to the cache are removed
from the provided view
- Add MaybeFlush to UtxoCache
- MaybeFlush conditionally flushes the cache to the database
- If the maximum size of the cache has been reached, or if the
periodic flush duration has been reached, then a flush is required
- A flush can be forced by setting the force flush parameter
- Flushing commits all modified entries to the database and
conditionally evicts entries
- Entries that are nil or spent are always evicted since they are
unlikely to be accessed again. Additionally, if the cache has
reached its maximum size, entries are evicted based on the height of
the block that they are contained in.
- Update connect block and disconnect block to commit to the cache and
conditionally flush to the database
- Rather than writing to the utxo set in the database every time that
a block is connected or disconnected, commit the updated view to the
cache and call MaybeFlush on the cache to conditionally flush it to
the database
- Add InitUtxoCache to UtxoCache
- InitUtxoCache initializes the utxo cache by ensuring that the utxo
set is caught up to the tip of the best chain
- Since the cache is only flushed to the database periodically, the
utxo set may not be caught up to the tip of the best chain
- InitUtxoCache catches the utxo set up by replaying all blocks from
the block after the block that was last flushed to the tip block
through the cache
- Add ShutdownUtxoCache to BlockChain
- ShutdownUtxoCache flushes the utxo cache to the database on
shutdown. Since the cache is flushed periodically during initial
block download and flushed after every block is connected after
initial block download is complete, this flush that occurs during
shutdown should finish relatively quickly
- Note that if an unclean shutdown occurs, the cache will still be
initialized properly when restarted as during initialization it will
replay blocks to catch up to the tip block if it was not fully
flushed before shutting down. However, it is still preferred to
flush when shutting down versus always recovering on startup since
it is faster
- Track the hit ratio of UtxoCache
- Track the number of hits and misses when accessing the cache in
order to calculate the overall hit ratio of the cache to gauge its
performance
This adds a utxoSetState type that is tracked in the database. The utxo
set state contains information regarding the current state of the utxo
set. In particular, it tracks the block height and block hash of the
last completed flush.
The utxo set state is tracked in the database since at any given time,
the utxo cache may not be consistent with the utxo set in the database.
This is due to the fact that the utxo cache only flushes changes to the
database periodically. Therefore, during initialization, the utxo set
state is used to identify the last flushed state of the utxo set and it
can be caught up to the current best state of the main chain.
This additionally adds full test coverage to the new serialization and
deserialization functions.
This updates utxo viewpoints to deep copy the script when getting it
from a msg tx. This is required since the tx out script is a subslice
of the overall contiguous buffer that the msg tx houses for all scripts
within the tx. It is deep copied here since this entry may be added to
the utxo cache, and we don't want the utxo cache holding the entry to
prevent all of the other tx scripts from getting garbage collected.
This adds a utxocachemaxsize configuration option which represents the
maximum size in MiB of the utxo cache. The default value is 150 MiB,
the minimum value is 25 MiB, and the maximum value is 32768 MiB
(32 GiB).
This adds a size method to UtxoEntry, which returns the number of bytes
that the entry uses on a 64-bit platform. This will be used as part of
tracking the total size of the utxo cache.
This adds utxoStateFresh to UtxoEntry to indicate that a txout is fresh,
which means that it exists in the utxo cache but does not exist in the
underlying database.
The utxo cache will use the fresh flag as an optimization to skip
writing to the database for outputs that are added and spent in between
flushes to the database.
This splits the utxo packed flags into two separate types, utxoState
and utxoFlags. The reasoning is that:
- This cleanly separates the purpose of the flags. utxoState defines
the in-memory state of a utxo entry, whereas utxoFlags defines
additional information for the containing transaction of a utxo
entry.
- This makes room for an additional state that is required for the utxo
cache, namely whether or not a utxo entry is fresh (does not exist as
an unspent transaction output in the database).
This modifies the vote notification logic so that wallets will no longer
vote on mainnet blocks once height 534304 has been reached and the block
version is prior to 8.
This change is being made to make use of the staking system to force
proof-of-work miners who have had well over a month now to upgrade to
the latest version so voting on the new consensus changes can commence.
This modifies the net sync manager to track the recently rejected
transactions using a much more efficient APBF instead of map which needs
store the entirety of the key for the items added to it.
It also significantly increases the number of tracked rejected
transactions thereby further lowering bandwidth usage in high rejection
scenarios while simultaneously increasing robustness against malicious
peers.
More concretely, tracking the new higher number with the current map
would take around 4.47 MiB per profiling while the new APBF only takes
around 568 KiB, a reduction of around 88%, while exhibiting roughly the
same computational performance.
The handler for getheaders currently ignores the request when the chain
is not yet believed to be current. This is generally desirable and
correct behavior, since it might otherwise lead peers to incorrect
conclusions about the state of the peer.
However, on private networks, such as simnet, it is not at all uncommon
for every node in the network to no longer be current if a block hasn't
been mined in a long time or when all nodes are stopped and restarted.
In order to better handle these types of edge conditions, this modifies
the server to respond to getheaders when the local chain tip is exactly
the same as the requested locator even when it is not marked current
yet.
This results in more robust handling for private networks while still
providing the normal desirable behavior prior to being current.
Upcoming changes constitute breaking public API changes to the addrmgr
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the new
version in the same commit
- Repeat the process for any other modules that require a new major as a
result of consuming the new major(s)
This deprecates the reject wire protocol message by bumping the protocol
version to 9 and making the reject message illegal under the new
protocol version. It also bumps the default user agent to
dcrwire:1.0.0.
Note that the message is not removed yet because the code still has to
be able to negotiate to older protocols where it is still supported
until some future software version makes the entire network require the
new protocol version is deployed at which time the entire message can be
removed.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This removes PushRejectMsg from the peer package as it is no longer
used or desired.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This removes the deprecated reject message return value from the
peer.OnVersion callback and updates all callers in the repo accordingly.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This modifies the server to stop sending reject messages.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This modifies the netsync package to stop sending reject messages.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
