This renames the mempool.Config.RelayNonStd option to AcceptNonStd which
more accurately describes its behavior since the mempool was refactored
into a separate package.
The reasoning for this change is that the mempool is not responsible for
relaying transactions (nor should it be). Its job is to maintain a pool
of unmined transactions that are validated according to consensus and
policy configuration options which are then used to provide a source of
transactions that need to be mined.
Instead, it is the server that is responsible for relaying transactions.
While it is true that the current server code currently only relays txns
that were accepted to the mempool, this does not necessarily have to
be the case. It would be entirely possible (and perhaps even a good
idea as something do in the future), to separate the relay policy from
the mempool acceptance policy (and thus indirectly the mining policy).
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This adds a new definition for the upcoming agenda vote to enable the
lnfeatures as defined by DCP0002 and DCP0003. It does not include any
code to make decisions or bump the versions. It is only the definition.
Decred's serialized format for transactions split the 32-bit version
field into two 16-bit components such that the upper bits are used to
encode a serialization type and the lower 16 bits are the actual
transaction version.
Unfortunately, when this was done, the in-memory transaction struct was
not also updated to hide this complexity, which means that callers
currently have to understand and take special care when dealing with the
version field of the transaction.
Since the main purpose of the wire package is precisely to hide these
details, this remedies the situation by introducing a new field on the
in-memory transaction struct named SerType which houses the
serialization type and changes the Version field back to having the
desired semantics of actually being the real transaction version. Also,
since the maximum version can only be a 16-bit value, the Version field
has been changed to a uint16 to properly reflect this.
The serialization and deserialization functions now deal with properly
converting to and from these fields to the actual serialized format as
intended.
Finally, these changes also include a fairly significant amount of
related code cleanup and optimization along with some bug fixes in order
to allow the transaction version to be bumped as intended.
The following is an overview of all changes:
- Introduce new SerType field to MsgTx to specify the serialization type
- Change MsgTx.Version to a uint16 to properly reflect its maximum
allowed value
- Change the semantics of MsgTx.Version to be the actual transaction
version as intended
- Update all callers that had special code to deal with the previous
Version field semantics to use the new semantics
- Switch all of the code that deals with encoding and decoding the
serialized version field to use more efficient masks and shifts
instead of binary writes into buffers which cause allocations
- Correct several issues that would prevent producing expected
serializations for transactions with actual transaction versions that
are not 1
- Simplify the various serialize functions to use a single func which
accepts the serialization type to reduce code duplication
- Make serialization type switch usage more consistent with the rest of
the code base
- Update the utxoview and related code to use uint16s for the
transaction version as well since it should not care about the
serialization type due to using its own
- Make code more consistent in how it uses bytes.Buffer
- Clean up several of the comments regarding hashes and add some new
comments to better describe the serialization types
This adds a new agenda to mainnet which allows the stake holders to
vote on whether or not they want the developers to begin work on
integrating lightning network support.
This implements a new stake difficulty algorithm along with a voting
agenda for all networks to change and a comprehensive set of tests. It
also implements estimation using the new algorithm for the
estimatestakediff RPC.
The following is an overview of the changes:
- Add new agenda to vote on changing the stake difficulty algorithm to
all networks
- The version on mainnet is version 4
- The version on testnet and simnet is version 5
- Modifies the stake difficulty calculation function to calculate the
difficulty based on the result of the vote
- Modifies the stake difficulty estimation function to calculate the
difficulty based on the result of the vote
- Makes the stake difficulty estimation function concurrent safe
- Calls it directly from the RPC server instead of going through block
manager
- Removes no longer needed code from the block manager
- Generate new version blocks and reject old version blocks after a
super majority has been reached
- New block version on mainnet is version 4
- New block version on testnet and simnet is version 5
- Add tests for the supply estimation used in the new algorithm
- Add tests for the new algorithm calculations
- Add tests for the estimation based on the new algorithm
This renames the chaincfg parameter for the vote choice which represents
an abstaining vote to be named IsAbstain instead of IsIgnore since that
more accurately describes its intent and behavior.
It also updates the RPC server choice result field for isignore to be
named isabstain to match and bumps the major version accordingly.
Finally, it renames other internal variables which make use of the
choice to include the word abstain as well for clarity and renames a
couple of other internal variables.
The test network should allow any scripts and transaction types that are
valid according to consensus rules as opposed to the more strict
standardness rules that apply to mainnet. This allows new script forms
and types to be tested before they go to mainnet.
Also, this is the same setting as the previous test network.
Agendas require some complicated and not always obvious assumptions. In
order to prevent bad agendas from making it into the code we validate
them at init time and panic if an agenda violates assumptions. This
will not adversely impact the user experience but will force developers
to properly design dcrd agendas.
These are the required tests to ensure vote integrity:
* All choices shall be consecutive and enumerated.
* Mask only contains consecutive 1 bits set.
* Ensure IsIgnore and IsNo are never both set to true.
* Ensure there is one and only one IsNo choice set to true.
* Ensure there is one and only one IsIgnore choice set to true.
* Ensure that the abstain choice is 0.
* Ensure that the number of choices does not exceed the mask bits
permissible size.
* Ensure that choice bits match the mask bits.
* Ensure there are no duplicate choice ids.
* Ensure there are no duplicate vote ids.
With this change we can keep the version and the bits in blockNode.
This is used to fix an oversight where we weren't strictly enforcing
the vote version. This also adds tests that validate quorum-1, quorum,
75%-1 and 75% rules during tallying.
Also adjust getvoteinfo totals to only return the specified vote version
totals.
This implements a new voting agenda for the testnet and simnet networks
for increasing the maximum block size from 1MiB to 1.25MB.
The following is an overview of the changes:
- Bump the maximum protocol block size limit to 1.25MB
- Bump the protocol version to 4
- Add wire tests for v3 protocol sizes and update tests for latest
- Update all wire values that are defined in terms of the max block size
to respect the protocol version
- Update the MaxSigOpsPerBlock constant to maintain existing value
regardless of the raised max protocol block size
- Decouple the maximum tx size from the block size by creating a chain
parameter for it with the current sizes so it is not a part of the
hard fork vote
- Add definition for new version 4 stake vote along with agenda to vote
on block size to the testnet and simnet networks
- Convert the MaximumBlockSize chain parameter to a slice in order to
hold the different possible sizes
- Adds a new function that returns the maximum block size based upon the
result of the vote
- Change the existing context-free block size check to enforce the max
protocol size and introduce a context-specific check which enforces a
restricted size based upon the network consensus parameters and the
result of the vote
- Set the max allowed size in generated block templates based upon the
network params and result of the vote
- Generate version 4 blocks and reject version 3 blocks after a super
majority has been reached
Port BIP0009 over from btcd. The following is an overview of the
changes:
- Add new configuration options to the chaincfg package which allows the
rule deployments to be defined per chain
- Implement code to calculate the threshold state as required by voting
- Use threshold state caches that are stored to the database in order
to accelerate startup time
- Remove caches that are invalid due to definition changes in the
params including additions, deletions, and changes to existing
entries
- Verify that PoW and PoS majorities have been reached.
- Add tests for new error type
- Add tests for threshold state
- Deployments are per stake version.
- Add human readable vote/choice infrastructure.
- Add RPC command to interrogate vote status (getvoteinfo).
* Bump block header version to 3.
* Enforce new rules once block version gets to 3.
* Use Voter Versions to control bumping of Stake Version.
* Calculate appropriate stake version based on history and prior
intervals in the mining code.
* Add bunch of UT to validate code.
This is the minimally correct working implementation. This code
desperately needs to memoize the questions that are asked repeatedly.
Fixes#523
This reverts to the correct upstream majority version code which
properly detects when certain thresholds of the network have been
updated in order to determine when to start rejecting old version
blocks and enforcing the rules in the new version block.
Upstream commit a7b35d9f9e.
Also, the merge commit contains necessary decred-specific alterations.
It also has the side effect of correcting a couple of min reduction bugs
even though they can't currently ever be hit due to the min reduction
interval flag being disabled for all networks.
Contains the following commits:
- 711f33450c
- b6b1e55d1e
- Reverted because Travis is already at a more recent version
- bd4e64d1d4
Also, the merge commit contains the necessary decred-specific
alterations, converts all other references to sha to hash to keep with
the spirit of the merged commits, and various other cleanup intended to
bring the code bases more in line with one another.
Bump block version and add stake version to wire. Currently
StakeVersions are unused and once the enforcement logic goes in the
versions will bump again.
This fixes#435
Copy hash_test.go from btcd. This file contains tests for encoding
and decoding of hashes to and from strings, and does not rely on the
hash algorithm.
Profiling indicated that significant time was being spent validating
coinbase outputs, ensuring that they paid to the development
organization's P2SH tax address. This check was more inefficient than
necessary for a couple of reasons:
* The tax address was always decoded from a string to a dcrutil.Address.
* The actual script being validated was always parsed for addresses to
check if they matched the tax address.
Neither of these are needed. To optimize the algorithm, only the
equality of the output script and script version are checked.
The legacy ticket database, which was GOB serialized and stored on
shut down, has been removed. Ticket state information is now held in
a stake node, which acts as a modularized "black box" to contain all
information about the state of the stake system. Stake nodes are now
a component of the blockchain blockNode struct, and are updated with
them.
Stake nodes, like their internal treap primitives, are immutable
objects that are created with their connect and disconnect node
functions. The blockchain database now stores all information about
the stake state of the best node in the block database. The blockchain
makes the assumption that the stake state of the best node is known at
any given time. If the states of former blocks or sidechains must be
evaluated, this can be achieved by iterating backwards along the
blockchain from the best node, and then connecting stake nodes
iteratively if necessary.
Performance improvements with this new module are dramatic. The long
delays on start up and shut down are removed. Blockchain
synchronization time is improved approximately 5-10x on the mainnet
chain. The state of the database is atomic, so unexpected shut downs
should no longer have the ability to disrupt the chain state.
An upgrade path has been added for version 1 blockchain databases.
Users with this blockchain database will automatically update when
they start their clients.
This introduces a new indexing infrastructure for supporting optional
indexes using the new database and blockchain infrastructure along with
two concrete indexer implementations which provide both a
transaction-by-hash and a transaction-by-address index.
The new infrastructure is mostly separated into a package named indexers
which is housed under the blockchain package. In order to support this,
a new interface named IndexManager has been introduced in the blockchain
package which provides methods to be notified when the chain has been
initialized and when blocks are connected and disconnected from the main
chain. A concrete implementation of an index manager is provided by the
new indexers package.
The new indexers package also provides a new interface named Indexer
which allows the index manager to manage concrete index implementations
which conform to the interface.
The following is high level overview of the main index infrastructure
changes:
- Define a new IndexManager interface in the blockchain package and
modify the package to make use of the interface when specified
- Create a new indexers package
- Provides an Index interface which allows concrete indexes to plugin
to an index manager
- Provides a concrete IndexManager implementation
- Handles the lifecycle of all indexes it manages
- Tracks the index tips
- Handles catching up disabled indexes that have been reenabled
- Handles reorgs while the index was disabled
- Invokes the appropriate methods for all managed indexes to allow
them to index and deindex the blocks and transactions
- Implement a transaction-by-hash index
- Makes use of internal block IDs to save a significant amount of
space and indexing costs over the old transaction index format
- Implement a transaction-by-address index
- Makes use of a leveling scheme in order to provide a good tradeoff
between space required and indexing costs
- Supports enabling and disabling indexes at will
- Support the ability to drop indexes if they are no longer desired
The following is an overview of the btcd changes:
- Add a new index logging subsystem
- Add new options --txindex and --addrindex in order to enable the
optional indexes
- NOTE: The transaction index will automatically be enabled when the
address index is enabled because it depends on it
- Add new options --droptxindex and --dropaddrindex to allow the indexes
to be removed
- NOTE: The address index will also be removed when the transaction
index is dropped because it depends on it
- Update getrawtransactions RPC to make use of the transaction index
- Reimplement the searchrawtransaction RPC that makes use of the address
index
- Update sample-btcd.conf to include sample usage for the new optional
index flags
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
This moves several of the chain constants to the Params struct in the
chaincfg package which is intended for that purpose. This is mostly a
backport of the same modifications made in Decred along with a few
additional things cleaned up.
The following is an overview of the changes:
- Comment all fields in the Params struct definition
- Add locals to BlockChain instance for the calculated values based on
the provided chain params
- Rename the following param fields:
- SubsidyHalvingInterval -> SubsidyReductionInterval
- ResetMinDifficulty -> ReduceMinDifficulty
- Add new Param fields:
- CoinbaseMaturity
- TargetTimePerBlock
- TargetTimespan
- BlocksPerRetarget
- RetargetAdjustmentFactor
- MinDiffReductionTime
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
