This environment variable was preventing the script from running
properly on OpenBSD. GCC is no longer provided in base on several
architectures, and GCC installed from packages is named 'egcc' to not
conflict with base GCC provided by the other architectures which still
require that toolchain.
This significantly optimizes the NAF conversion code by rewriting it to
avoid all heap allocations as well as switch to an O(1) algorithm.
The following benchmark shows a before and after comparison of the NAF
conversion as well as how that translates to scalar multiplication and
signature verification:
name old time/op new time/op delta
----------------------------------------------------------------------
NAF 1.16µs ± 1% 0.08µs ± 1% -93.02% (p=0.008 n=5+5)
ScalarMult 138µs ± 1% 135µs ± 0% -1.77% (p=0.016 n=5+4)
SigVerify 164µs ± 0% 162µs ± 0% -0.98% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
----------------------------------------------------------------------
NAF 96.0B ± 0% 0.0B -100.00% (p=0.008 n=5+5)
ScalarMult 816B ± 0% 720B ± 0% -11.76% (p=0.008 n=5+5)
SigVerify 1.54kB ± 0% 1.44kB ± 0% -6.25% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
----------------------------------------------------------------------
NAF 2.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
ScalarMult 15.0 ± 0% 11.0 ± 0% -26.67% (p=0.008 n=5+5)
SigVerify 32.0 ± 0% 28.0 ± 0% -12.50% (p=0.008 n=5+5)
This reworks the NAF tests to actually test the produced encoding
adheres to the requires of NAF encoding as well as to make them more
consistent with modern practices in the code.
Specifically, the following properties are asserted:
- There must not be a leading zero byte and the number of bytes used to
encode the negative portion must not exceed the positive portion
- There must not be any adjacent non-zero digits
- The negative and positive portions must sum back to the original value
The existing tests only ensure the positive and negative portions sum
back up to the original value.
The updated tests also revealed that the existing naf function doesn't
strip leading zeroes, so that logic has been added. That behavior is
not a problem in terms of correctness given how it is used in the code
since extra leading zeroes don't change the resulting value, but it
could potentially cause more work to be done than necessary and having
leading zeroes technically doesn't conform to NAF encoding.
This slightly reworks the scalar multiplication logic to improve its
readability by making use of zero values and a single mask instead of
shifting every individual byte of the positive and negative portions of
k1 and k2.
It also improves the comments to clarify the intention of the code which
might not otherwise be obvious given the bit twiddling nature of it.
The reduction in the number of shifts in the double and add loop is also
technically an optimization, but the calculation is by far dominated by
the ec operations, so saving a few nanoseconds on shifts is so negigible
that it's in the margin of error in benchmarks.
This modifies the ecdsa and schnorr examples to call blake256 directly
instead of indirecting through chainhash in order to avoid the chainhash
dependency which is otherwise not needed.
The go.mod and go.sum files are updated accordingly to remove the
no longer required dependency.
This modifies the getwork RPC to properly handle the case when both the
template and the error are nil which happens if the caller makes a
request during a chain reorganization.
It also adds a test for the condition to prevent regressions.
This optimizes the pre-computed byte points used to accelerate scalar
base multiplication to store the data in affine coordinates instead of
Jacobian coordinates which reduces the memory usage requirement to 66%
of what it current requires and also has the important benefit of
further speeding up the computation.
This is the case because projecting affine coordinates into Jacobian
space is essentially free and the point doubling and addition routines
have optimizations which allow them to avoid additional operations when
the Z coordinate is 1, which is the case for an initial affine
projection.
Further, since the compressed table is stored in the string table of the
binary, it also reduces the size the of final binary by ~385KiB.
The following benchmark shows a before and after comparison of scalar
base multiplication as well as how that translates to signature
verification:
name old time/op new time/op delta
-------------------------------------------------------------------------
ScalarBaseMult 34.5µs ± 1% 24.7µs ± 1% -28.43% (p=0.008 n=5+5)
ScalarBaseMultLarge 48.2µs ± 1% 38.0µs ± 1% -21.08% (p=0.008 n=5+5)
SigVerify 181µs ± 5% 163µs ± 2% -9.86% (p=0.008 n=5+5)
While 18 µs less per signature verification might not seem like much on
the surface, consider that every transaction requires at least one
signature operation, so there are a ton of them when doing no checkpoint
syncs. For a concrete number, verifying 100 million signatures would
take 30 minutes less time.
This removes the code that deals with only initializing the adaptor
instance on first use since the adaptor code no longer houses the
pre-computed byte points which motivated that behavior.
This separates the code that loads the pre-computed byte points used to
accelerate scalar base multiplication from the elliptic adaptor code
which further makes the internals of the package independent of the
crypto/elliptic and crypto/ecdsa stdlib interfaces.
It also takes this opportunity to improve the related code a bit by
making it less dependent on magic numbers and defining a proper type for
the data table.
Finally, it retains and improves the logic which only loads the data on
first use by making use of a closure to house and access the loaded data
so it is no longer possible to accidentally access the uninitialized
pointer.
When a shutdown is requested, the sleep should not continue in
(*Client).wsReconnectHandler. Instead, use a select with the shutdown
channel and a time.After.
This updates the addrmgr and peer dependencies to list the
pseudo-versions of their next major versions as dependencies instead of
the as-of-yet unreleased tagged versions.
This is needed to allow clients of the main dcrd module to correctly
import it while those dependencies are still under development.
This updates all users of the github.com/decred/dcrd/databasev/2 module
to use the recently introduced v3 version.
Replace directives are added as needed to ease development while the
final version of the database/v3 v3.0.0 isn't tagged.
The relevant documentation is also updated to use the new database/v3
module.
This prepares the dependent modules for a version bump of the database
module by pointing their required version to the most recent one and
removing replace directives where they exist.
This modifies the block index hash -> node mapping to use shortened keys
with fallback to the full 32-byte key in the event of a collision in the
shortened key. This saves a significant amount of memory because there
are hundreds of thousands of nodes in the index and Go maps duplicate
keys. As an added bonus, due to the implementation choices, it also
results in slightly faster overall lookups on average. This is because
even though lookups of colliding keys are slightly slower, lookups of
non-colliding keys are slightly faster and non-colliding keys are by far
the most common case (typically > 99.5%).
It accomplishes this by taking advantage of the fact the full hash is
part of the node itself and therefore can be used to disambiguate any
collisions in the shortened key and splitting the non-colliding entries
from the colliding entries across two maps where one is keyed by the
shorter key and the other by the full key.
The shortened key is chosen to be the big-endian uint32 formed by the
first 4 bytes of the full hash. The primary reasons for this choice
are:
- The hash function used to produce the block hashes is a
uniformly-random distribution, so the number of collisions will
coincide with the expected value of a uniform distribution
- The bits zeroed by the mining process start at the end of the array,
so there would need to be effectively impossible to achieve hash rates
exceeding ~2^215.77 hashes/sec (aka ~89.8 peta yotta yotta hashes/sec)
in order to start zeroing out the chosen bits
- Using 4 bytes per node results in a savings of 28 bytes per node minus
a small amount of amortized overhead for the collision handling
- 4-byte keys produce a relatively small expected number of collisions
- The cutoff point where the overhead of the collisions is expected to
reach just 10% of the savings gained is around 5800 years given the
average mainnet block production rate
- Conversion of the bytes to a uint32 is basically free in terms of
processing overhead
Profiling of the implemented code as well as simulations show that both
the actual number of observed collisions and observed memory savings
match the expected theoretical values. Specifically, as of block height
575166 on mainnet, there are 35 collisions in the shortened keys and the
memory savings is around 30 MiB.
This modifies a couple of cases that lookup nodes directly from the
block index to use the methods for that purpose instead. This allows
future modifications to the way the nodes are keyed.
This moves the UTXO database from a "metadata" subdirectory to the
parent directory. The UTXO database existed in a "metadata"
subdirectory during the transition phase of moving the UTXO set to a
separate database since it was using the database package, but the
subdirectory is no longer necessary now that the UTXO database uses
leveldb directly.
Additionally, it bumps the overall block database version to 11 in order
to prevent downgrades.
This updates the UtxoBackend implementation to use leveldb directly
rather than using the database package. This results in reduced memory
usage and improved processing time since it avoids the overhead of
things that the database package provides, such as caching with periodic
flushing, that are not needed for the UtxoBackend since it has its own
caching layer (UtxoCache).
An overview of the changes is as follows:
- Introduce simple UTXO key sets to be used for constructing prefixes of
database keys rather than using the "buckets" concept that the
database package uses
- All keys in the UTXO backend now start with a serialized prefix
consisting of the key set (1 byte) and version of that key set (1
byte)
- Add migration to move all keys in the UTXO backend from buckets to the
newly defined key sets
- Add Get function to the UtxoBackend interface and corresponding
implementation in LevelDbUtxoBackend to allow for directly retrieving
the value for a given key
- Add NewIterator function to the UtxoBackend interface and
corresponding implementation in LevelDbUtxoBackend to allow for
iterating over the key/value pairs in the UtxoBackend
- Add Update function to the UtxoBackend interface and corresponding
implementation in LevelDbUtxoBackend to allow for directly updating
values in the UtxoBackend in the context of a transaction
- Update LevelDbUtxoBackend to use an underlying leveldb.DB instance
rather than a database.DB instance
- Update all LevelDbUtxoBackend methods to use the Get, NewIterator, and
Update functions rather than the database package functions
- Update all database queries to use keys with the new key set prefixes
rather than bucketized keys
- Update dbFetchUtxoBackendInfo to check the legacy bucket for database
info if it doesn't exist in the new location
- Update upgrade logic to use the UtxoBackend interface rather than the
concrete LevelDbUtxoBackend type
- Update all tests to create leveldb.DB databases instead of database.DB
databases for the UtxoBackend
This introduces the levelDbUtxoBackendTx type, which wraps an underlying
leveldb transaction and implements the UtxoBackendTx interface.
This is part of updating the UtxoBackend to use leveldb directly.
This introduces a UtxoBackendTx interface, which represents a
UtxoBackend transaction. As would be expected with a transaction, no
changes will be saved to the underlying UtxoBackend until it has been
committed.
This is part of updating the UtxoBackend to use leveldb directly.
This introduces a UtxoBackendIterator interface, which represents an
iterator over the key/value pairs in a UtxoBackend in key order.
This is part of updating the UtxoBackend to use leveldb directly.
This adds a helper function and associated tests to convert leveldb
errors into context errors with the equivalent error kind and the passed
description.
This adds new blockchain error kinds for the UTXO backend.
Additionally, it adds a RawErr field to ContextError which contains the
original error in the case where an error has been converted.
This adds a missing Method type declaration to the "work" and "tspend"
notification method names.
This allows importing both v2 and v3 versions of this package without
running into RPC command registration errors.
This updates the static generation code so that it compiles as intended
by locally calculating the curve byte size based on its bit size since
that value was removed from the curve definition.
This updates the mempool maybeAcceptTransaction method to accept agenda
flags rather than just a boolean indicating if the treasury is active.
This allows for accommodating future agendas without having to add
additional boolean fields for each agenda.
This is useful in particular since maybeAcceptTransaction makes use of
the blockchain.CheckTransaction function to perform validation checks on
transactions, which includes checks which depend on whether or not an
agenda is active.
This updates the mempool ProcessOrphans method to accept agenda flags
rather than just a boolean indicating if the treasury is active.
This allows for accommodating future agendas without having to add
additional boolean fields for each agenda.
This updates the blockchain BlockConnectedNtfnsData and
BlockDisconnectedNtfnsData types to contain agenda flags rather than
just a boolean indicating if the treasury is active.
This allows for accommodating future agendas without having to add
additional boolean fields for each agenda.
This adds an IsTreasuryEnabled method to the AgendaFlags type that
returns whether the flags indicate that the treasury agenda is enabled.
This is useful since checking the flags directly is pretty verbose,
especially when using the type outside of the blockchain package.
This also updates existing references to the AgendaFlags type to use
the IsTreasuryEnabled function.
This introduces a child context with cancellation to the manual Connect
method and cancels it when the client is manually stopped via Shutdown
to ensure callers are still able to shutdown the client via that method
when using manual connections in addition to when using automatic ones.
This corrects the ban reason string when too many transactions are
advertised and then claimed as not found to display the number of
transactions as intended.
This makes the CORS policy for websocket clients more strict by ensuring
any requests from other domains are rejected. It is worth noting that,
in practice, the current less strict mechanism is secure due to both the
use of a self-signed TLS certificate and the requirement for
authentication to issue any commands, but further hardening is still
desirable.
This imposes additional per-connection read limits on the RPC server to
help further harden it against potential abuse in non-standard
configurations on poorly-configured networks.
In practice, these changes will not have any noticeable effect for the
vast majority of nodes since the RPC server is not publicly accessible
by default and requires authentication.
Nevertheless, it can still be useful to apply additional read limits for
scenarios such as authenticated fuzz testing and poorly-configured
networks that have disabled all other security measures.
The following are the updated per-connection limits:
- 0 B / 8 MiB for pre and post auth HTTP connections
- 4 KiB / 16 MiB for pre and post auth websocket connections
This modifies the test code in stake to avoid using standardness code
since it is testing consensus level code which means it needs to be
stable and thus must not change if the policy regarding standardness can
change.
This forces the UTXO cache to flush to the database when the chain just
latched to current. This ensures that the full UTXO set is always
flushed to the database when the chain becomes current, which allows for
fetching up-to-date UTXO set stats.
The behavior before these changes is that the UTXO cache will not be
forced to flush to the database until a block is connected AFTER the
chain is already latched to current. Therefore, if you fetch UTXO stats
when the chain is current but the next block has not been connected yet,
the UTXO stats may not be up to date.
This adds explicit test cases for the consensus-specific
isAnyKindOfScriptHash method of the engine instead of using the script
class tests which are only intended for use by standardness code
This adds explicit test cases for the consensus-specific
IsPayToScriptHash func instead of using the script class tests which are
only intended for use by standardness code.
This removes PushedData along with its associated test and benchmark
from txscript since it is no longer used by anything and hasn't been for
quite some time due to the introduction of the zero allocation script
tokenizer which allows it to be done by callers more efficiently.
