This makes the calcNextRequiredDifficulty method a little more
consistent with the code in terms of the name of the parameter and using
a shorter local convenience var to avoid overly long lines.
This modifies the entire repository to use the new formatting of doc
comments in the upcoming Go 1.19 release.
The primary motivating factors for this are:
- Builds check that files are formatted per gofmt and that will no
longer be true as of Go 1.19 without these changes
- Separating all the updates into a single commit ensures these
documentation only formatting changes do not clutter up diffs that
actually change code
For the most part, the changes are just the automated changes suggested
by the Go 1.19 version of gofmt, but there are also a few cases where
the comments were reworded a bit to play nicely with the new formatting
requirements.
For example, the new version of gofmt reformats and collapses nested
lists where as the existing version does not. Thus, instances of nested
lists have been changed to either eliminate them or use mixed markers
which produce expect results.
The existing code prior to this change updated the relevant fields in
the header and called its own hash method which internally makes use of
the existing writer interface based serialization and shared buffers to
prevent a lot of allocations. That makes sense most places in the code,
but it is really bad for concurrent performance in a tight mining loop.
Thus, this significantly optimizes the CPU miner workers to instead
remove all mutex contention by serializing the header once outside of
the loop, directly updating the relevant bytes in the serialized data,
using the allocation-free hash method to compute the hash from the
serialized bytes directly, and finally updating the block template
header when/if a solution is found.
It also now makes use of the much more efficient primitives and uint256
packages instead of big integers to further reduce allocations and speed
up the calculations.
The net result is a major speedup when mining with a single core and
nearly linear scalar with multiple cores.
Concretely, prior to these changes, I was only seeing around 400-570
kh/s with 1 or 2 cores and mining on more cores actually made the
performance worse due to the all of the contention. With these changes,
I'm seeing around 1.2Mh/s on a single core and over 10MH/s with
10 cores.
This reworks the speed stat tracking to make it more accurate as well as
significantly reduce lock contention.
A high level overview of the changes is as follows:
- Each worker is given its own stats state to independently update
- Each worker now has a unique ID which is used to store and clean up
said state
- The elapsed hashing time per worker is now tracked
- The individual speed stats are updated via lock-free atomic primitives
versus the previous single mutex that all workers shared
- Each worker now periodically updates its individual speed stats every
fixed number of nonces instead of relying on a ticker
- The speed monitor goroutine periodically sums the results from all of
the individual worker stats
Some local test mining shows that prior to these changes the reported
hash speed varies pretty wildly between 43 kh/s and 427 kh/s and more
wild swings with more workers. With the changes, the reporting is more
in line with what I would expect without additional changes and comes in
a much tigher (and a bit higher due to a bit of reduced lock contention)
range of 408 kh/s and 568 kh/s.
This implements private key generation directly for the secp256k1 curve
instead of calling the generic version in the standard library that
requires the adaptor code and thus all of the extra allocations and big
integer conversions involved with it.
Not only is this implementation significantly more efficient, both in
terms of execution speed and memory allocations, it also is
theoretically more secure since it does not have the modulo bias the
implementation in the standard lib does.
The following is a before and after comparison of generating a private
key:
name old time/op new time/op delta
-----------------------------------------------------------------------------
PrivateKeyGenerate 38.6µs ± 4% 0.4µs ± 5% -99.01% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
-----------------------------------------------------------------------------
PrivateKeyGenerate 14.0 ± 0% 3.0 ± 0% -78.57% (p=0.008 n=5+5)
This exposes a new function on the ModNScalar type named IsZeroBit which
operates similarly to IsZero except it returns a 0 or 1.
This is being provided because it is not possible in Go to convert from
a bool to numeric value in constant time and many constant-time
operations require a numeric value.
Since the type is now exported for external use, consumers would
otherwise have no way to perform the conversion.
This downgrades an error when rebroadcasting the winning tickets to a
logged warning.
This is necessary because it is a legitimate state to have a chain tip
header returned by the Chain.TipGeneration() while the actual block data
is not yet available for the LotteryDataForBlock to complete.
Prior to this change, a voting wallet connecting to a dcrd instance with
multiple chain tip headers without the associated data would be unable
to re-start voting for blocks due to the associated RPC call failing.
This removes the rate limit flag from MaybeAcceptTransaction since it is
no longer used.
This is part of the overall removal of the low-fee/free tx relay policy
and associated rate limiting.
This removes the rate limit flag from maybeAcceptTransaction since it is
no longer used.
This is part of the overall removal of the low-fee/free tx relay policy
and associated rate limiting.
This removes ErrInsufficientPriority now that it is no longer used.
This is part of the overall removal of the low-fee/free tx relay policy
and associated rate limiting.
This removes the code that accepts transactions with low fees to the
mempool when they have sufficient priority along with the associated
rate limiting of such transactions.
It also updates the error message when a transaction does not pay a high
enough fee to clearly state it was rejected for that reason along with
the minimum fee it is required to pay.
Finally, it slightly reworks the low fee rejection logic to include all
types of transactions it applies to in the same check instead of
independently per type.
This is part of the overall removal of the low-fee/free tx relay policy
and associated rate limiting.
This deprecates the --limitfreerelay CLI option and removes the
associated code that changes behavior based on the option.
This is part of the overall removal of the low-fee/free tx relay policy
and associated rate limiting.
Prior to the introduction of child-pays-for-parent (CPFP), it was
possible for transactions to essentially become stuck forever if they
didn't pay a high enough fee for miners to include them in a block.
In order to prevent this, a policy was introduced to allow relaying of
low-fee/free transactions based on a priority that is calculated based
on the fee as well as the age of coins being spent. This means that the
priority slowly increases over time as the coins age to ensure such
transactions would eventually be relayed and mined. Further, in order
to prevent abuse the behavior could otherwise allow, this policy
includes additional rate-limiting of these types of transactions.
While the policy served its purpose in the past, there are some
downsides such as:
- A confusing UX where transactions that don't pay enough fees and also
aren't old enough to meet the dynamically changing priority
requirements are rejected due to having insufficient priority instead
of not paying enough fees as the user might expect
- The priority requirements dynamically change over time which leads to
non-deterministic behavior and thus ultimately results in what appear
to be intermittent/transient failures to users
Since the policy is no longer necessary given said transactions can now
simply use CPFP to increase the overall fee of the entire transaction
chain thereby ensuring they are mined, this is the first of a series of
commits to remove the aforementioned policy along with associated rate
limiting from the mempool and transaction relay.
Specifically, it deprecates the --norelaypriority CLI option and removes
the associated code that changes behavior based on the option.
This updates the mempool to explicitly reject standalone treasurybases
and adds a test to ensure proper functionality.
Note that treasurybases are already rejected because IsCoinBaseTx
currently detects a treasurybase as a coinbase as well, but it is safer
to be explicit in case the coinbase detection function is updated in the
future to exclude treasurybases.
Also since treasurybases are rejected early, remove the code that deals
with special casing them later given there is no way the transaction is
one.
This modifies the test harness to create transactions that use valid
fees instead of zero fees and cleans up a couple of minor things in the
process.
Of particular note is TestMaybeAcceptTransactions which was reworked to
use the existing transaction chain generation func and calculate the
expected ancestor fees from the transactions instead of manually
overriding.
Since the address index has been removed there is no longer a need for a spend
pruner component to keep track of spend journal entries. This removes the
spend pruner components and its integrations. DropConsumerDepsBucket has been
added to remove persisted spend pruner consumer dependencies which are now
unneeded.
This updates the docker image to golang:1.18.3-alpine3.16.
To confirm the new digest:
$ docker pull golang:1.18.3-alpine3.16
...
1.18.3-alpine3.16: Pulling from library/golang
...
Digest: sha256:7cc62574fcf9c5fb87ad42a9789d5539a6a085971d58ee75dd2ee146cb8a8695
...
This modifies the method that returns the total cumulative work for a
given block to return a more efficient uint256 instead of a big integer
and updates all consumers accordingly.
Go 1.14 added runtime support for handling the windows
CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, and CTRL_SHUTDOWN_EVENT events by
adding a definition for syscall.SIGTERM to windows and converting the
events to that signal. It also added definitions for other common
signals, including SIGHUP, and treats them as NOOPs on windows.
Consequently, this modifies the logic that deals with conditionally
handling SIGTERM and SIGHUP to handle them for windows as well as all
unix-like operating systems, including newer ones that are supported by
Go since the code was originally written.
Although the additional signals could probably just be added
unconditionally without too much issue now due to the runtime adding
stubs to all operating systems the project officially supports, it is
safer to be explicit when dealing with syscall definitions since there
is no guarantee that they will exist for newly-added operating systems.
Stated more plainly, this change means dcrd will now be shutdown cleanly
on more variants of unix as well as when being terminated by windows
itself due to various things such as the user logging off, the window
terminal being closed, and the system shutting down.
Live profiling data of performing an initial sync shows that roughly 36%
of all in-use allocations are the result of the big integers used to
store the cumulative work for each block. Further, around 12% of the
entire CPU time is spent scaning the heap for garbage collection which
is a direct result of the total number of inuse allocations. Therefore,
a reasonable expectation is that eliminating those heap objects should
produce a speedup of around 4-5%.
Consequently, this modifies the blockchain package to make use of the
much more efficient zero-alloc uint256s and associated work calculation
funcs in the new primitives package that is under development.
Profiling shows the result is about 100MiB less heap usage on average
and a reduction of about 5% to the initial sync time which is in line
with the expected result.
The moves the blockchain package from the blockchain module to an
internal package of the root module meaning that it is no longer a part
of the exported blockchain module. Nearly all of the logic it provides
is really for the internal implementation of dcrd itself and thus having
it exported module significantly increases the maintenance burden.
Note that as of this change the blockchain module still exists and
provides the chaingen and fullblocktests packages both or which are
useful and used by external consumers.
This is part of a continuing overall effort to reduce the total number
of exported packages and modules and eventually get to the point it will
be possible to follow semver for the root module.
Overview of the major changes:
- Move all go files from blockchain -> internal/blockchain
- Move testdata from blockchain -> internal/blockchain
- Remove doc.go in favor of the README.md since godoc now displays it
- Move README.md from blockchain -> internal/blockchain and update to
match the new reality
- Add a new README.md in the exported blockchain module that documents
it contains the remaining exported packages
- Update all import paths in the repository accordingly
- Run go mod tidy on all modules
This modifies the chain logic to create and store the individual
commitment hashes covered by the commitment root field of the header of
each block and also adds code to migrate the database to retroactively
create and store entries for all applicable historical blocks.
The upgrade can be interrupted at any point and future invocations will
resume from the point it was interrupted.
The following is a high level overview of the changes:
- Introduce a new database bucket to house the header commitments
- Add serialization code for use when storing and loading the individual
header commitment hashes
- Add full test coverage for new serialization code
- Store the commitment hashes in the db when connecting blocks
- Implement database migration code to retroactively store the
commitment hashes for all applicable historical blocks
- Bump the chain database version to 13
- Support resuming from interrupted upgrades
- Add a new func on the internal header commitment data struct that
returns the v1 header commitment hashes to consolidate the logic
- Update FilterByBlockHash to load the header commitments from the db
and generate the inclusion proof accordingly
This adds logic to avoid hitting the database when a compact filter that
can't possibly be available by first checking the block index to ensure
the requested block is both available and its data is known.
This adds a field to the header commitment data struct to store the hash
of the filter along with the filter so that it can be reused without
needing to recalculate the hash.
starting
