This updates the error messages in the chainio tests to consistently use
the newer style of `t.Errorf("%q: ...", test.name)` rather than
including extra information for the function name, test index, etc.
The extra information is no longer needed to pinpoint an error since the
test framework now shows proper line numbers for any errors that occur.
This updates the minimum recommended memory (RAM) from 1GB to 2GB in the
main README.md. The minimum recommended memory is being increased due
to the introduction of the utxo cache.
This adds full test coverage to the UtxoCache type and its methods.
Additionally, since this uses the testing Cleanup function that was
introduced in Go 1.14, this bumps the required go version for the
blockchain package from 1.13 to 1.14.
UtxoCache is an unspent transaction output cache that sits on top of the
utxo set database and provides significant runtime performance benefits
at the cost of some additional memory usage. It drastically reduces the
amount of reading and writing to disk, especially during initial block
download when a very large number of blocks are being processed in quick
succession.
The UtxoCache is a read-through cache. All utxo reads go through the
cache. When there is a cache miss, the cache loads the missing data
from the database, caches it, and returns it to the caller.
The UtxoCache is a write-back cache. Writes to the cache are
acknowledged by the cache immediately but are only periodically flushed
to the database. This allows intermediate steps to effectively be
skipped. For example, a utxo that is created and then spent in between
flushes never needs to be written to the utxo set in the database.
Due to the write-back nature of the cache, at any given time the
database may not be in sync with the cache, and therefore all utxo reads
and writes MUST go through the cache, and never read or write to the
database directly.
An overview of the changes is as follows:
- Add UtxoCache and UtxoCacheConfig struct types and NewUtxoCache method
- Update server to create the utxo cache with the configured max size
and pass to the block chain instance that is created
- Update all test block chains to create a utxo cache
- Add FetchEntry to UtxoCache
- FetchEntry returns the specified transaction output from the utxo
set
- If the output exists in the cache, it is returned immediately.
Otherwise, it uses an existing database transaction to fetch the
output from the database, caches it, and returns it to the caller.
- Add AddEntry to UtxoCache
- AddEntry adds the specified output to the cache
- Add SpendEntry to UtxoCache
- SpendEntry marks the specified output as spent
- Remove entries that are marked as fresh and then subsequently spent.
This is an optimization to skip writing to the database for outputs
that are added and spent in between flushes to the database.
- Update UtxoViewpoint to hold the UtxoCache
- Update fetching entries from the database to fetch entries from the
cache instead
- Add Commit to UtxoCache
- Commit updates all entries in the cache based on the state of each
entry in the provided view
- All entries in the provided view that are marked as modified and
spent are removed from the view
- Additionally, all entries that are added to the cache are removed
from the provided view
- Add MaybeFlush to UtxoCache
- MaybeFlush conditionally flushes the cache to the database
- If the maximum size of the cache has been reached, or if the
periodic flush duration has been reached, then a flush is required
- A flush can be forced by setting the force flush parameter
- Flushing commits all modified entries to the database and
conditionally evicts entries
- Entries that are nil or spent are always evicted since they are
unlikely to be accessed again. Additionally, if the cache has
reached its maximum size, entries are evicted based on the height of
the block that they are contained in.
- Update connect block and disconnect block to commit to the cache and
conditionally flush to the database
- Rather than writing to the utxo set in the database every time that
a block is connected or disconnected, commit the updated view to the
cache and call MaybeFlush on the cache to conditionally flush it to
the database
- Add InitUtxoCache to UtxoCache
- InitUtxoCache initializes the utxo cache by ensuring that the utxo
set is caught up to the tip of the best chain
- Since the cache is only flushed to the database periodically, the
utxo set may not be caught up to the tip of the best chain
- InitUtxoCache catches the utxo set up by replaying all blocks from
the block after the block that was last flushed to the tip block
through the cache
- Add ShutdownUtxoCache to BlockChain
- ShutdownUtxoCache flushes the utxo cache to the database on
shutdown. Since the cache is flushed periodically during initial
block download and flushed after every block is connected after
initial block download is complete, this flush that occurs during
shutdown should finish relatively quickly
- Note that if an unclean shutdown occurs, the cache will still be
initialized properly when restarted as during initialization it will
replay blocks to catch up to the tip block if it was not fully
flushed before shutting down. However, it is still preferred to
flush when shutting down versus always recovering on startup since
it is faster
- Track the hit ratio of UtxoCache
- Track the number of hits and misses when accessing the cache in
order to calculate the overall hit ratio of the cache to gauge its
performance
This adds a utxoSetState type that is tracked in the database. The utxo
set state contains information regarding the current state of the utxo
set. In particular, it tracks the block height and block hash of the
last completed flush.
The utxo set state is tracked in the database since at any given time,
the utxo cache may not be consistent with the utxo set in the database.
This is due to the fact that the utxo cache only flushes changes to the
database periodically. Therefore, during initialization, the utxo set
state is used to identify the last flushed state of the utxo set and it
can be caught up to the current best state of the main chain.
This additionally adds full test coverage to the new serialization and
deserialization functions.
This updates utxo viewpoints to deep copy the script when getting it
from a msg tx. This is required since the tx out script is a subslice
of the overall contiguous buffer that the msg tx houses for all scripts
within the tx. It is deep copied here since this entry may be added to
the utxo cache, and we don't want the utxo cache holding the entry to
prevent all of the other tx scripts from getting garbage collected.
This adds a utxocachemaxsize configuration option which represents the
maximum size in MiB of the utxo cache. The default value is 150 MiB,
the minimum value is 25 MiB, and the maximum value is 32768 MiB
(32 GiB).
This adds a size method to UtxoEntry, which returns the number of bytes
that the entry uses on a 64-bit platform. This will be used as part of
tracking the total size of the utxo cache.
This adds utxoStateFresh to UtxoEntry to indicate that a txout is fresh,
which means that it exists in the utxo cache but does not exist in the
underlying database.
The utxo cache will use the fresh flag as an optimization to skip
writing to the database for outputs that are added and spent in between
flushes to the database.
This splits the utxo packed flags into two separate types, utxoState
and utxoFlags. The reasoning is that:
- This cleanly separates the purpose of the flags. utxoState defines
the in-memory state of a utxo entry, whereas utxoFlags defines
additional information for the containing transaction of a utxo
entry.
- This makes room for an additional state that is required for the utxo
cache, namely whether or not a utxo entry is fresh (does not exist as
an unspent transaction output in the database).
This modifies the vote notification logic so that wallets will no longer
vote on mainnet blocks once height 534304 has been reached and the block
version is prior to 8.
This change is being made to make use of the staking system to force
proof-of-work miners who have had well over a month now to upgrade to
the latest version so voting on the new consensus changes can commence.
This modifies the net sync manager to track the recently rejected
transactions using a much more efficient APBF instead of map which needs
store the entirety of the key for the items added to it.
It also significantly increases the number of tracked rejected
transactions thereby further lowering bandwidth usage in high rejection
scenarios while simultaneously increasing robustness against malicious
peers.
More concretely, tracking the new higher number with the current map
would take around 4.47 MiB per profiling while the new APBF only takes
around 568 KiB, a reduction of around 88%, while exhibiting roughly the
same computational performance.
The handler for getheaders currently ignores the request when the chain
is not yet believed to be current. This is generally desirable and
correct behavior, since it might otherwise lead peers to incorrect
conclusions about the state of the peer.
However, on private networks, such as simnet, it is not at all uncommon
for every node in the network to no longer be current if a block hasn't
been mined in a long time or when all nodes are stopped and restarted.
In order to better handle these types of edge conditions, this modifies
the server to respond to getheaders when the local chain tip is exactly
the same as the requested locator even when it is not marked current
yet.
This results in more robust handling for private networks while still
providing the normal desirable behavior prior to being current.
Upcoming changes constitute breaking public API changes to the addrmgr
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the new
version in the same commit
- Repeat the process for any other modules that require a new major as a
result of consuming the new major(s)
This deprecates the reject wire protocol message by bumping the protocol
version to 9 and making the reject message illegal under the new
protocol version. It also bumps the default user agent to
dcrwire:1.0.0.
Note that the message is not removed yet because the code still has to
be able to negotiate to older protocols where it is still supported
until some future software version makes the entire network require the
new protocol version is deployed at which time the entire message can be
removed.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This removes PushRejectMsg from the peer package as it is no longer
used or desired.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This removes the deprecated reject message return value from the
peer.OnVersion callback and updates all callers in the repo accordingly.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This modifies the server to stop sending reject messages.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This modifies the netsync package to stop sending reject messages.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This modifies the peer package to stop sending and logging reject
messages.
This is part of deprecating the reject wire protocol message towards its
eventual removal in a future version of the software.
This modifies the server to track the per peer known addresses using a
much more efficient APBF instead of an LRU cache which has significant
overhead in addition to having to store the entirety of all items added
to it.
False positives are acceptable as the goal is to avoid sending duplicate
addresses which is an ideal case for APBFs.
More concretely, the current LRU cache that stores the known addresses
can grow to around 1.69 MiB *per peer* when full while the new APBF only
takes around 40 KiB, a reduction of around 97.7%, while exhibiting
roughly the same computational performance.
Since there is a maximum of 125 peers by default, that means the current
LRU cache can potentially grow to around ~211 MiB if fully populated for
max peers. The new APBF, on the other hand, will only consume ~4.88 MiB
under the same scenario.
This modifies the server to track the recently confirmed transactions
using a much more efficient APBF instead of an LRU cache which has
significant overhead in addition to having to store the entirety of all
items added to it.
This is acceptable because false positives are acceptable as the goal is
to deduplicate transaction requests which is an ideal case for APBFs.
More concretely, the current LRU cache that stores the recently
confirmed transactions takes around 2.67 MiB per profiling while the new
APBF only takes around 180 KiB, a reduction of around 93%, while
exhibiting roughly the same computational performance.
Upcoming changes constitute breaking public API changes to the peer
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the new
version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
This modifies the modular reduction step to make use of the same fast
reduce mechanism used in the gcs filters. As can be seen in the
following benchmark results, it more than doubles the performance of the
primary filter operations.
In order to experimentally validate the mapping adheres to the
theoretical results and doesn't have any adverse effects on the false
positive rates, the same validation methodology described in README.md
was conducted again and said README is updated accordingly. Everything
is well within the margin of error as expected.
Finally, the README is also updated with the new benchmark results and
the required go version is bumped in the go.mod due to the addition of
the math/bits import.
name old time/op new time/op delta
-----------------------------------------------------------------------------
capacity=1000, fprate=0.1%
--------------------------
BenchmarkAdd 158ns ± 1% 59ns ± 1% -62.59% (p=0.008 n=5+5)
BenchmarkContainsTrue 183ns ± 1% 69ns ± 2% -62.27% (p=0.008 n=5+5)
BenchmarkContainsFalse 61.3ns ±40% 42.0ns ±26% -31.41% (p=0.032 n=5+5)
capacity=1000, fprate=0.01%
---------------------------
BenchmarkAdd 211ns ± 1% 69ns ± 1% -67.07% (p=0.008 n=5+5)
BenchmarkContainsTrue 236ns ± 2% 80ns ± 1% -66.16% (p=0.008 n=5+5)
BenchmarkContainsFalse 59.6ns ±24% 37.7ns ± 5% -36.74% (p=0.008 n=5+5)
BenchmarkReset
capacity=1000, fprate=0.001%
----------------------------
BenchmarkAdd 247ns ± 0% 78ns ± 2% -68.32% (p=0.008 n=5+5)
BenchmarkContainsTrue 272ns ± 1% 89ns ± 1% -67.50% (p=0.008 n=5+5)
BenchmarkContainsFalse 58.6ns ±26% 37.0ns ± 4% -36.98% (p=0.008 n=5+5)
capacity=100000, fprate=0.01%
-----------------------------
BenchmarkAdd 205ns ± 2% 80ns ± 1% -61.12% (p=0.008 n=5+5)
BenchmarkContainsTrue 219ns ± 1% 80ns ± 1% -63.39% (p=0.008 n=5+5)
BenchmarkContainsFalse 70.3ns ±46% 37.6ns ±10% -46.61% (p=0.008 n=5+5)
capacity=100000, fprate=0.0001%
-------------------------------
BenchmarkAdd 275ns ± 2% 110ns ± 1% -60.10% (p=0.008 n=5+5)
BenchmarkContainsTrue 287ns ± 1% 98ns ± 1% -65.98% (p=0.008 n=5+5)
BenchmarkContainsFalse 56.6ns ±45% 36.3ns ± 6% -35.93% (p=0.008 n=5+5)
capacity=100000, fprate=0.00001%
--------------------------------
BenchmarkAdd 413ns ± 3% 205ns ± 2% -50.41% (p=0.016 n=5+5)
This adds the ability to run 'go generate' to generate a table of k and
l parameter combinations along with the false positive rate they
maintain and average expected number of accesses for false queries to
help callers that wish to fine tune the filter behavior select
appropriate parameters.
This implements an Age-Partitioned Bloom Filter (APBF) that is safe for
concurrent access.
An APBF is a probabilistic data structure suitable for use in processing
unbounded data streams where more recent items are more significant than
older ones and some false positives are acceptable. It has similar
computational costs as traditional Bloom filters and provides space
efficiency that is competitive with the current best-known, and more
complex, Dictionary approaches.
Similar to classic Bloom filters, APBFs have a non-zero probability of
false positives that can be tuned via parameters and are free from false
negatives up to the capacity of the filter. However, unlike classic
Bloom filters, where the false positive rate climbs as items are added
until all queries are a false positive, APBFs provide a configurable
upper bound on the false positive rate for an unbounded number of
additions.
The unbounded property is achieved by adding and retiring disjoint
segments over time where each segment is a slice of a partitioned Bloom
filter. The slices are conceptually aged over time as new items are
added by shifting them and discarding the oldest one.
While APBFs are useful in a variety of use cases, the primary motivation
for adding them to Decred at the current time is for use in more
efficiently deduplicating various continuous event streams, such as
recently confirmed and rejected transactions and inventory (e.g.
transactions, blocks, addresses) other peers are known to have.
Currently, a LRU cache is used for this purpose in several places which
has a non-trivial amount of overhead. For a concrete example of one
particular instance, tracking the known addresses for the maximum
default number of 125 peers can currently consume up to around 150 MiB.
On the other hand, using the APBF implementation this introduces will,
once properly updated, allow that to be reduced to around 3.63 MiB while
still maintaining a false positive rate of 1 in a million and exhibiting
similar computational performance.
This modifies the server to only notify the sync manager after it has
had a chance to process and potentially disconnect it and introduces an
additional flag to explicitly track when the sync manager is notified or
not instead of relying on the result of version negotiation in order to
ensure stable ordering.
This modifies the gcs and txscript modules to use the latest siphash
v1.2.2.
While here, it also updates the main module to use the latest rpcclient.
- github.com/dchest/siphash@v1.2.2
- github.com/decred/dcrd/rpcclient/v7@v7.0.0-20210129214723-fc227a05904d
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
Several commits are needed since there is a dependency chain that
involves transitive deps.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/v4@v4.0.0-20210129200153-14fd1a785bf2
- github.com/decred/dcrd/rpc/jsonrpc/types/v3@v3.0.0-20210129200153-14fd1a785bf2
- github.com/decred/dcrd/rpcclient/v7@v7.0.0-20210129200153-14fd1a785bf2
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
Several commits are needed since there is a dependency chain that
involves transitive deps.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/v4@v4.0.0-20210129195202-a4265d63b619
- github.com/decred/dcrd/gcs/v3@v3.0.0-20210129195202-a4265d63b619
- github.com/decred/dcrd/rpcclient/v7@v7.0.0-20210129195202-a4265d63b619
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
Several commits are needed since there is a dependency chain that
involves transitive deps.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v4@v4.0.0-20210129192908-660d0518b4cf
- github.com/decred/dcrd/blockchain/v4@v4.0.0-20210129192908-660d0518b4cf
- github.com/decred/dcrd/gcs/v3@v3.0.0-20210129192908-660d0518b4cf
- github.com/decred/dcrd/peer/v2@v2.2.1-0.20210129192908-660d0518b4cf
- github.com/decred/dcrd/rpcclient/v7@v7.0.0-20210129192908-660d0518b4cf
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
Several commits are needed since there is a dependency chain that
involves transitive deps.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/v4@v4.0.0-20210129190127-4ebd135a82f1
- github.com/decred/dcrd/database/v2@v2.0.3-0.20210129190127-4ebd135a82f1
- github.com/decred/dcrd/gcs/v3@v3.0.0-20210129190127-4ebd135a82f1
- github.com/decred/dcrd/hdkeychain/v3@v3.0.1-0.20210129190127-4ebd135a82f1
- github.com/decred/dcrd/peer/v2@v2.2.1-0.20210129190127-4ebd135a82f1
- github.com/decred/dcrd/rpcclient/v7@v7.0.0-20210129190127-4ebd135a82f1
- github.com/decred/dcrd/txscript/v4@v4.0.0-20210129190127-4ebd135a82f1
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
Several commits are needed since there is a dependency chain that
involves transitive deps.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/blockchain/stake/v4@v4.0.0-20210129181600-6ae0142d3b28
- github.com/decred/dcrd/blockchain/v4@4.0.0-20210129181600-6ae0142d3b28
- github.com/decred/dcrd/dcrutil/v4@v4.0.0-20210129181600-6ae0142d3b28
- github.com/decred/dcrd/hdkeychain/v3@v3.0.0-20210129181600-6ae0142d3b28
- github.com/decred/dcrd/txscript/v4@v4.0.0-20210129181600-6ae0142d3b28
This modifies some recently-updated modules to use a valid prerelease
version so they can be used in require statements in consumer code that
is also under development.
Several commits are needed since there is a dependency chain that
involves transitive deps.
The updated direct dependencies are as follows:
- github.com/decred/dcrd/dcrec/secp256k1/v4@v4.0.0-20210127014238-b33b46cf1a24
This modifies the server code that handles connection requests from the
RPC server to prevent duplicate pending connections to the same
address (ip/port).
It makes use of the new functionality of the connection manager to
iterate all connection requests which including pending requests.
This adds a new func named ForEachConnReq to the connection manager
which accepts a closure to call with each connection req known to the
connection manager, including pending ones.
A test is also added to ensure proper functionality.
This change modifies the mempool to store staged transactions
as a *TxDesc rather than a *dcrutil.Tx. The advantage of doing
this is to avoid recalculating transaction types through
stake.DetermineTxType since the type would have been captured
in a TxDesc when the transaction initially entered the mempool.
Additionally, rather than returning an array of redeemers for a
given transaction, a function provided by the caller is invoked
for each redeemer. This eliminates unnecessary allocations by
allowing the caller to access each redeemer without needing to
rely on an intermediary buffer.
This removes several functions that are no longer used as they are
defined directly on the RPC adapator for the rpcserver.ConnManager
interface instead.
This changes the log for head reorg failures to a debug log instead of
an error log since it is actually expected to fail reorgs from time to
time due to the fact mining is a race to find blocks.
It also cleans up the code slightly while here and adds an additional
check to avoid additional work in the case reorgs to other tips fail and
the current tip then becomes the best candidate among the eligible
parents.
