Attempting to publish a recently-mined transaction using the
sendrawtransaction JSON-RPC method will now use the same error code as
for duplicate transactions currently present in the mempool, rather
than a generic error for double spending or spending unknown inputs.
There is a very small chance that this check results in a false
positive and that a transaction rejected for other reasons is reported
as a duplicate when it had never been mined at all.
This allows returning the P2P address of the given harness node, which
is needed when trying to test third party services that connect via the
P2P network via rpctest (e.g. SPV wallets).
This adds an ExpiringNextBlock method on the Node type, which returns
the tickets that will expire in the next block. This allows consumers
to determine the tickets that are about to expire when validating or
constructing a new block.
This will be used for the automatic ticket revocations agenda since one
of the new rules for that agenda is that blocks must contain revocations
for expired tickets in the block that they become expired.
This changes the amount used for the large tspend in the rpc treasury
simnet test so that it is compatible to the new expenditure policy and
fails as expected.
This renames the existing maxTreasuryExpenditure function to
maxTreasuryExpenditureDCP0006 to identify it as the one implementing the
legacy policy.
A future commit will add a new policy and selection of the active one
will be based on the results of an on-chain voting.
This switches the current blockchain go.mod file to use a replace
directive for the chaincfg code, which is needed due to the new agenda
introduced in a previous commit.
This environment variable was preventing the script from running
properly on OpenBSD. GCC is no longer provided in base on several
architectures, and GCC installed from packages is named 'egcc' to not
conflict with base GCC provided by the other architectures which still
require that toolchain.
This significantly optimizes the NAF conversion code by rewriting it to
avoid all heap allocations as well as switch to an O(1) algorithm.
The following benchmark shows a before and after comparison of the NAF
conversion as well as how that translates to scalar multiplication and
signature verification:
name old time/op new time/op delta
----------------------------------------------------------------------
NAF 1.16µs ± 1% 0.08µs ± 1% -93.02% (p=0.008 n=5+5)
ScalarMult 138µs ± 1% 135µs ± 0% -1.77% (p=0.016 n=5+4)
SigVerify 164µs ± 0% 162µs ± 0% -0.98% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
----------------------------------------------------------------------
NAF 96.0B ± 0% 0.0B -100.00% (p=0.008 n=5+5)
ScalarMult 816B ± 0% 720B ± 0% -11.76% (p=0.008 n=5+5)
SigVerify 1.54kB ± 0% 1.44kB ± 0% -6.25% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
----------------------------------------------------------------------
NAF 2.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
ScalarMult 15.0 ± 0% 11.0 ± 0% -26.67% (p=0.008 n=5+5)
SigVerify 32.0 ± 0% 28.0 ± 0% -12.50% (p=0.008 n=5+5)
This reworks the NAF tests to actually test the produced encoding
adheres to the requires of NAF encoding as well as to make them more
consistent with modern practices in the code.
Specifically, the following properties are asserted:
- There must not be a leading zero byte and the number of bytes used to
encode the negative portion must not exceed the positive portion
- There must not be any adjacent non-zero digits
- The negative and positive portions must sum back to the original value
The existing tests only ensure the positive and negative portions sum
back up to the original value.
The updated tests also revealed that the existing naf function doesn't
strip leading zeroes, so that logic has been added. That behavior is
not a problem in terms of correctness given how it is used in the code
since extra leading zeroes don't change the resulting value, but it
could potentially cause more work to be done than necessary and having
leading zeroes technically doesn't conform to NAF encoding.
This slightly reworks the scalar multiplication logic to improve its
readability by making use of zero values and a single mask instead of
shifting every individual byte of the positive and negative portions of
k1 and k2.
It also improves the comments to clarify the intention of the code which
might not otherwise be obvious given the bit twiddling nature of it.
The reduction in the number of shifts in the double and add loop is also
technically an optimization, but the calculation is by far dominated by
the ec operations, so saving a few nanoseconds on shifts is so negigible
that it's in the margin of error in benchmarks.
This modifies the ecdsa and schnorr examples to call blake256 directly
instead of indirecting through chainhash in order to avoid the chainhash
dependency which is otherwise not needed.
The go.mod and go.sum files are updated accordingly to remove the
no longer required dependency.
This modifies the getwork RPC to properly handle the case when both the
template and the error are nil which happens if the caller makes a
request during a chain reorganization.
It also adds a test for the condition to prevent regressions.
This optimizes the pre-computed byte points used to accelerate scalar
base multiplication to store the data in affine coordinates instead of
Jacobian coordinates which reduces the memory usage requirement to 66%
of what it current requires and also has the important benefit of
further speeding up the computation.
This is the case because projecting affine coordinates into Jacobian
space is essentially free and the point doubling and addition routines
have optimizations which allow them to avoid additional operations when
the Z coordinate is 1, which is the case for an initial affine
projection.
Further, since the compressed table is stored in the string table of the
binary, it also reduces the size the of final binary by ~385KiB.
The following benchmark shows a before and after comparison of scalar
base multiplication as well as how that translates to signature
verification:
name old time/op new time/op delta
-------------------------------------------------------------------------
ScalarBaseMult 34.5µs ± 1% 24.7µs ± 1% -28.43% (p=0.008 n=5+5)
ScalarBaseMultLarge 48.2µs ± 1% 38.0µs ± 1% -21.08% (p=0.008 n=5+5)
SigVerify 181µs ± 5% 163µs ± 2% -9.86% (p=0.008 n=5+5)
While 18 µs less per signature verification might not seem like much on
the surface, consider that every transaction requires at least one
signature operation, so there are a ton of them when doing no checkpoint
syncs. For a concrete number, verifying 100 million signatures would
take 30 minutes less time.
This removes the code that deals with only initializing the adaptor
instance on first use since the adaptor code no longer houses the
pre-computed byte points which motivated that behavior.
This separates the code that loads the pre-computed byte points used to
accelerate scalar base multiplication from the elliptic adaptor code
which further makes the internals of the package independent of the
crypto/elliptic and crypto/ecdsa stdlib interfaces.
It also takes this opportunity to improve the related code a bit by
making it less dependent on magic numbers and defining a proper type for
the data table.
Finally, it retains and improves the logic which only loads the data on
first use by making use of a closure to house and access the loaded data
so it is no longer possible to accidentally access the uninitialized
pointer.
When a shutdown is requested, the sleep should not continue in
(*Client).wsReconnectHandler. Instead, use a select with the shutdown
channel and a time.After.
This updates the addrmgr and peer dependencies to list the
pseudo-versions of their next major versions as dependencies instead of
the as-of-yet unreleased tagged versions.
This is needed to allow clients of the main dcrd module to correctly
import it while those dependencies are still under development.
This updates all users of the github.com/decred/dcrd/databasev/2 module
to use the recently introduced v3 version.
Replace directives are added as needed to ease development while the
final version of the database/v3 v3.0.0 isn't tagged.
The relevant documentation is also updated to use the new database/v3
module.
This prepares the dependent modules for a version bump of the database
module by pointing their required version to the most recent one and
removing replace directives where they exist.
This modifies the block index hash -> node mapping to use shortened keys
with fallback to the full 32-byte key in the event of a collision in the
shortened key. This saves a significant amount of memory because there
are hundreds of thousands of nodes in the index and Go maps duplicate
keys. As an added bonus, due to the implementation choices, it also
results in slightly faster overall lookups on average. This is because
even though lookups of colliding keys are slightly slower, lookups of
non-colliding keys are slightly faster and non-colliding keys are by far
the most common case (typically > 99.5%).
It accomplishes this by taking advantage of the fact the full hash is
part of the node itself and therefore can be used to disambiguate any
collisions in the shortened key and splitting the non-colliding entries
from the colliding entries across two maps where one is keyed by the
shorter key and the other by the full key.
The shortened key is chosen to be the big-endian uint32 formed by the
first 4 bytes of the full hash. The primary reasons for this choice
are:
- The hash function used to produce the block hashes is a
uniformly-random distribution, so the number of collisions will
coincide with the expected value of a uniform distribution
- The bits zeroed by the mining process start at the end of the array,
so there would need to be effectively impossible to achieve hash rates
exceeding ~2^215.77 hashes/sec (aka ~89.8 peta yotta yotta hashes/sec)
in order to start zeroing out the chosen bits
- Using 4 bytes per node results in a savings of 28 bytes per node minus
a small amount of amortized overhead for the collision handling
- 4-byte keys produce a relatively small expected number of collisions
- The cutoff point where the overhead of the collisions is expected to
reach just 10% of the savings gained is around 5800 years given the
average mainnet block production rate
- Conversion of the bytes to a uint32 is basically free in terms of
processing overhead
Profiling of the implemented code as well as simulations show that both
the actual number of observed collisions and observed memory savings
match the expected theoretical values. Specifically, as of block height
575166 on mainnet, there are 35 collisions in the shortened keys and the
memory savings is around 30 MiB.
This modifies a couple of cases that lookup nodes directly from the
block index to use the methods for that purpose instead. This allows
future modifications to the way the nodes are keyed.
This moves the UTXO database from a "metadata" subdirectory to the
parent directory. The UTXO database existed in a "metadata"
subdirectory during the transition phase of moving the UTXO set to a
separate database since it was using the database package, but the
subdirectory is no longer necessary now that the UTXO database uses
leveldb directly.
Additionally, it bumps the overall block database version to 11 in order
to prevent downgrades.
This updates the UtxoBackend implementation to use leveldb directly
rather than using the database package. This results in reduced memory
usage and improved processing time since it avoids the overhead of
things that the database package provides, such as caching with periodic
flushing, that are not needed for the UtxoBackend since it has its own
caching layer (UtxoCache).
An overview of the changes is as follows:
- Introduce simple UTXO key sets to be used for constructing prefixes of
database keys rather than using the "buckets" concept that the
database package uses
- All keys in the UTXO backend now start with a serialized prefix
consisting of the key set (1 byte) and version of that key set (1
byte)
- Add migration to move all keys in the UTXO backend from buckets to the
newly defined key sets
- Add Get function to the UtxoBackend interface and corresponding
implementation in LevelDbUtxoBackend to allow for directly retrieving
the value for a given key
- Add NewIterator function to the UtxoBackend interface and
corresponding implementation in LevelDbUtxoBackend to allow for
iterating over the key/value pairs in the UtxoBackend
- Add Update function to the UtxoBackend interface and corresponding
implementation in LevelDbUtxoBackend to allow for directly updating
values in the UtxoBackend in the context of a transaction
- Update LevelDbUtxoBackend to use an underlying leveldb.DB instance
rather than a database.DB instance
- Update all LevelDbUtxoBackend methods to use the Get, NewIterator, and
Update functions rather than the database package functions
- Update all database queries to use keys with the new key set prefixes
rather than bucketized keys
- Update dbFetchUtxoBackendInfo to check the legacy bucket for database
info if it doesn't exist in the new location
- Update upgrade logic to use the UtxoBackend interface rather than the
concrete LevelDbUtxoBackend type
- Update all tests to create leveldb.DB databases instead of database.DB
databases for the UtxoBackend
This introduces the levelDbUtxoBackendTx type, which wraps an underlying
leveldb transaction and implements the UtxoBackendTx interface.
This is part of updating the UtxoBackend to use leveldb directly.
This introduces a UtxoBackendTx interface, which represents a
UtxoBackend transaction. As would be expected with a transaction, no
changes will be saved to the underlying UtxoBackend until it has been
committed.
This is part of updating the UtxoBackend to use leveldb directly.
This introduces a UtxoBackendIterator interface, which represents an
iterator over the key/value pairs in a UtxoBackend in key order.
This is part of updating the UtxoBackend to use leveldb directly.
This adds a helper function and associated tests to convert leveldb
errors into context errors with the equivalent error kind and the passed
description.
This adds new blockchain error kinds for the UTXO backend.
Additionally, it adds a RawErr field to ContextError which contains the
original error in the case where an error has been converted.
This adds a missing Method type declaration to the "work" and "tspend"
notification method names.
This allows importing both v2 and v3 versions of this package without
running into RPC command registration errors.
This updates the static generation code so that it compiles as intended
by locally calculating the curve byte size based on its bit size since
that value was removed from the curve definition.
This updates the mempool maybeAcceptTransaction method to accept agenda
flags rather than just a boolean indicating if the treasury is active.
This allows for accommodating future agendas without having to add
additional boolean fields for each agenda.
This is useful in particular since maybeAcceptTransaction makes use of
the blockchain.CheckTransaction function to perform validation checks on
transactions, which includes checks which depend on whether or not an
agenda is active.
This updates the mempool ProcessOrphans method to accept agenda flags
rather than just a boolean indicating if the treasury is active.
This allows for accommodating future agendas without having to add
additional boolean fields for each agenda.
