This is based on https://proposals.decred.org/proposals/c96290a but was
modified in order to deal with realities that were unknown at the time
of the specification draft.
It is large and could not really be broken apart due to the pervasive
use of the isTreasuryEnabled flag. It was primarily authored by
* Marco Peereboom <marco@peereboom.us>
* Dave Collins <davec@conformal.com>
* Matheus Degiovani <opensource@matheusd.com>
With additional contributions from
* Donald Adu-Poku <donald.adu@gmail.com>
* Jamie Holdstock <jholdstock@decred.org>
Major changes:
* Add decentralized treasury agenda, as specified in DCP0006, to all supported
nets.
* Add functions to determine if the decentralized treasury agenda is active at
given block.
* Add new opcode OP_TADD that is a nop in txscript but is used to tag scripts
that credit the treasury account. This opcode is overloaded for treasurybase
and for normal transactions.
* Add new opcode OP_TSPEND that is a nop in txscript but is used to tag scripts
that debit the treasury account.
* Add new opcode OP_TGEN that is a nop in txscript but is used to tag P2PKH and
P2SH outputs in a TSpend transaction.
* Add functions that detect if a transaction is a valid TAdd, TSpend
or treasurybase transaction.
* Add error codes that return specific treasurybase/TAdd/TSpend consensus
violations.
* Modify countSpentOutputs to deal with treasury opcodes accordingly.
* Modify indexBlock to skip treasury transactions that do not have inputs.
* Add IsTreasuryEnabled call to ChainQueryer interface.
* Add treasury logger for debugging and logging the decentralized treasury
subsystem.
* Add IsTreasuryActive flag to BlockConnectedNtfnsData and
BlockDisconnectedNtfnsData.
* Modify OP_SSGEN to allow an optional output that contains votes for a TSpend
transaction hash.
* Add function that returns TSpend votes from an SSGen transaction.
* Modify CalcStakeVoteSubsidy so that treasurybase, unlike coinbase, is always
awarded the full percentage of the assigned block reward.
* Add helper functions to do all TSpend math so that callers don't roll their
own.
* Modify IsCoinBaseTx to not mistake a TSpend transaction as a coinbase.
* Add checkTreasuryBase function that verifies that a treasurybase is properly
constructed and pays the right amount to the treasury account.
* Add functions to calculate treasury balance for the provided block hash/node.
* Add function that verifies if a TSpend has a valid signature.
* Add functions to determine if a TSpend is not overspending.
* Add function to determine if a TSpend has been mined on the provided chain.
* Add functions that count and verifies treasury spend votes.
* Modify connectTransaction and disconnectTransactions to deal with the various
treasury transactions.
* Split CheckTransactionSanity in two functions
checkTransactionSanityContextFree and checkTransactionSanityContextual. This
is done in order to keep the decentralized treasury, which is always
contextual, from infecting the context free checks.
* Modify checkTransactionSanityContextual to recognize and verify treasury
transactions.
* Modify CheckTransactionSanity to deal with treasury transactions.
* Split checkBlockSanity in two functions checkBlockSanityContextFree and
checkBlockSanityContextual. This is done in order to keep the decentralized
treasury, which is always contextual, from infecting the context free checks.
* Modify checkBlockSanityContextual to enforce treasurybase and TAdd consensus
checks.
* Modify checkBlockPositional by unindenting it and adding TSpend consensus
enforcement.
* Modify checkCoinbaseUniqueHeightWithAddress to deal with the removal of the
project subsidy from output 0.
* Add checkCoinbaseUniqueHeightWithTreasuryBase that verifies coinbase and
treasurybase in the provided block.
* Unindent checkBlockContext.
* Modify checkTicketRedeemerCommitments and checkVoteInputs to deal with
potential tspend votes.
* Modify CheckTransactionInputs to skip treasurybase transactions.
* Modify CheckTransactionInputs to deal with TSpend transactions. Ensure the
provided Pi key is valid and that the signature is valid for the transaction.
Ensure that treasury TAdd and TSpend transaction utxo can only be spent after
coinbase maturity.
* Modify CountSigOps to deal with treasury transactions.
* Modify CountP2SHSigOps to deal with treasury transactions.
* Modify getStakeTreeFees to skip treasury transactions. Modify
totalOutputs to subtract ValueIn 0 for TSpend and treasurybase transactions.
* Modify checkTransactionsAndConnect to deal with modified amounts.
* Add tspendChecks function that verifies an entire TSpend transaction
validity at the point of the provided block. It ensures a TSpend is on a TVI.
It ensures the TSpend is in the valid window. It verifies that a TSpend In
and Out amounts match. It ensures a TSpend has the ValueIn amount encoded in
the OP_RETURN in Out 0. It ensures a TSpend has not been mined before on this
chain. It ensures a TSpend has the requisite votes. It ensures a TSpend is
not overspending.
* Modify checkConnectBlock to call checkTreasuryBase and tspendChecks when
treasury agenda is active.
* Add two tables to the database. Table "treasury" records the balance as of
this block and balance changes that occurred in this block which will become
active in CoinbaseMaturity blocks. Table "tspend" records all block hashes
where a TSpend has been mined this is to detect forks and prevent a Tspend
from being mined more than once.
* Modify handleBlockchainNotification to communicate if the treasury agenda is
active and skip treasurybase transaction when needed.
* Add various Treasury parameters to chaincfg params.
* Add hardcoded Tspend signatures in dcr_tmux_simnet_setup.sh.
* Add notifytspend and stoptspend calls to the RPC server. notifytspend
notifies the mempool when a TSpend transaction arrives.
* Modify commit filters V2 to recognize TAdd and TSpend transactions. It was
possible to modify V2 instead of introducing V3 because nothing changes from
the viewpoint of the wallet and treasury opcodes are disallowed prior to
agenda activation.
* Modify AddMemPoolTransaction to skip TSpend transactions that would throw the
fee estimator off.
* Add IsTreasuryAgendaActive, OnTSpendReceived and TSpendMinedOnAncestor to
mempool.Config in order to reject/accept TSpends in the mempool.
* Modify checkPoolDoubleSpend to ignore treasurybase.
* Modify mempool.maybeAcceptTransaction to enforce treasury standardness rules.
Don't allow TSpend transactions prior to stake validation height. Skip
treasurybase and tspend transactions in the orphan test. Ensure a tspend is
in a valid window. Ensure not more than 7 TSpends are active in the mempool.
Ensure TSpend has a well-known Pi key. Ensure The provided Pi key was used to
sign the transaction. Ensure TSpend was not mined in an ancestor block.
Notify subscribers that a valid TSpend was received.
* Add standardCoinbaseOpReturn and standardTreasurybaseOpReturn to create an
OP_RETURN followed by a data push that little endian encodes the height of
the block. Then there are a number of random bytes to ensure that the
transaction hash is always random.
* Modify createCoinbaseTx to create a coinbase that is valid when treasury is
enabled or not. Additionally, alter the transaction version if treasury is
enabled.
* Add createTreasuryBaseTx that creates a standard treasurybase.
* Modify maybeInsertStakeTx to recognize treasurybase and TSpend transactions.
* Modify handleTooFewVoters to call createTreasuryBaseTx when the treasury
agenda is active. Skip copying treasurybase.
* Modify NewBlockTemplate to recognize and deal with treasury transactions.
Skip TSpend transaction if block is not a TVI. Skip TSpend transaction if it
is not in the proper window. Skip TSpend transaction if a TSpend does not
have enough yes votes. Skip TSpend transaction if it overspends the treasury
account. Skip TAdd if there are more than 20 TAdds in the block. Create
treasurybase if required. Insert valid TAdd/TSpend transactions into stake
tree.
* Add TreasuryBalance and IsTreasuryAgendaActive to rpcserver Chain interface.
* Add gettreasurybalance, sendfromtreasury and sendtotreasury calls to RPC
server.
* Add notifytspend and stopnotifytspend to RPC websocket commands.
* Add simnet miner to generate large number of blocks during rpctests without
triggering PoW difficulty increases. This is used to verify various treasury
and tspend conditions during CI/CT.
* Modify RPC voting wallet to also vote on TSpends.
* Add json tests to verify all new opcodes and corner cases in the script
engine.
* Modify isStakeOpcode to recognize treasury opcodes.
* Modify countSigOpsV0 to count TSpends.
* Modify handleStakeOutSign to deal with TSpends.
* Modify SignTxOutput to recognize TSpends.
* Add TSpendSignatureScript that signs a TSpend transaction.
* Add TreasuryAddTy and TreasurySpendTy types to the standard scripts.
* Add isTreasuryAddScript and isTreasurySpendScript functions that recognize
a form of TAdd and TSpend transactions.
* Modify ExtractPkScriptAddrs to deal with TAdd and TSpend outputs.
* Add TxVersionSeqLock = 2 and TxVersionTreasury = 3 to wire. This is
used to discriminate between treasury and non-treasury scripts.
* Rig up all functions that need the isTreasuryEnabledflag directly or
indirectly.
* Shuffle various functions around and export them when they were needed to be
called from other packages.
* Added and modified numerous tests to verify (hopefully) all corner cases that
the decentralized treasury agenda has added.
After recent optimizations, the current next biggest offender of more
allocations than would be expected revealed by profiling is due to the
trace logging closures for the scripting engine execution.
Once upon a time, there was no way to check the current logging level
with the logging infrastructure at the time and thus a logging closure
was used to defer the fairly expensive construction of the trace logging
information until it was actually invoked (meaning tracing is enabled).
However, those closures come at the cost of allocations, and since
script execution is something that happens non-stop during normal
operation, those allocations really add up, as the profiling shows.
As some point, the logging infrastructure was changed out, and it is now
possible to determine the logging level, so this updates the code to
take advantage of that and avoid the closures while still only
performing the fairly expensive construction of trace logging
information when tracing is enabled. In other words, with this change
there is zero cost (other than the conditional check, of course) when
tracing is not enabled.
Finally, this also removes the no longer necessary code related to
creating the logging closures.
This modifies the signature encoding check function to use the new
secp256k1.ModNScalar type for the half order check instead of a big int.
The following benchmark shows a before and after comparison of a typical
signature encoding check:
benchmark old ns/op new ns/op delta
-------------------------------------------------------------------
BenchmarkCheckSignatureEncoding 79.0 46.9 -40.63%
benchmark old allocs new allocs delta
--------------------------------------------------------------------
BenchmarkCheckSignatureEncoding 1 0 -100.00%
benchmark old bytes new bytes delta
--------------------------------------------------------------------
BenchmarkCheckSignatureEncoding 64 0 -100.00%
This PR moves and exports several functions that are independently
useful.
This is one of several PRs that will follow in order to make the
treasury PR a bit smaller.
This modifies the consensus critical function which determines if the
redeem script of a p2sh or stake-tagged p2sh contains stake opcodes to
avoid calling the GetScriptClass function which is only intended
explicitly for working with standard script forms which only apply in
the context of the more restrictive standardness policy rules.
It also renames the function to better indicate its purpose is to
specifically check the redeem script as opposed to the entire signature
script.
Upcoming changes constitute breaking public API changes to the secp256k1
module, therefore, this follows the process for introducing major API
breaks which consists of:
- Bump the major version in the go.mod of the affected module if not
already done since the last release tag
- Add a replacement to the go.mod in the main module if not already done
since the last release tag
- Update all imports in the repo to use the new major version as
necessary
- Make necessary modifications to allow all other modules to use the new
version in the same commit
- Repeat the process for any other modules the require a new major as a
result of consuming the new major(s)
The existing implementation to handle conditional execution makes use of
a stack to track the state of each nested conditional. It is already
fairly efficient in terms of execution costs since it only considers the
most recent conditional stack entry and makes use of pushing OpCondSkip
to essentially track the nesting depth in unexecuted branches, however,
using a stack is less efficient in terms of memory usage than is
actually necessary since there is no need to use a stack at all given
that all that is really needed to provide the necessary behavior is the
current conditional nesting depth and the depth at which branch
execution was disabled (if it has been disabled).
Given the above, this optimizes the txscript conditional execution logic
by replacing the condition stack with two int32 fields to track the
aforementioned cases and updates the conditional execution opcode and
logic accordingly.
This updates the following modules to use the secp256k1/v2 module:
- blockchain
- chaincfg/v2
- dcrutil/v2
- hdkeychain/v2
- mempool/v3
- txscript/v2
- main
The hdkeychain/v3 and txscript/v2 modules both use types from secp256k1
in their public API.
Consequently, in order avoid forcing them to bump their major versions,
secp256k1/v1.0.3 was released with the types redefined in terms of the
secp256k1/v2 module so callers still using v1 of the module that are not
ready to upgrade to the v2 module yet can interoperate by updating to
the latest patch version.
This removes the DefaultScriptVersion constant and updates the code and
tests to use version 0 scripts accordingly.
It is being removed because it is highly error prone since anything that
is working with scripts needs to understand the version it is dealing
with and having a constant that could change that version out from under
them could easily lead to buggy behavior.
This removes a couple of checks for impossible conditions found by the
staticcheck linter. In the case of executeOpcode, bytes are always >= 0
and, similarly for SigCache.Add, uint are always >= 0.
This converts the callback function defined on the internal opcode
struct to accept the opcode and data slice instead of a parsed opcode as
the final step towards removing the parsed opcode struct and associated
supporting code altogether.
It also updates all of the callbacks and tests accordingly and finally
removes the now unused parsedOpcode struct.
This converts the executeOpcode function defined on the engine to accept
an opcode and data slice instead of a parsed opcode as a step towards
removing the parsed opcode struct and associated supporting code altogether.
It also updates all callers accordingly.
This refactors the script engine to store and step through raw scripts
by making using of the new zero-allocation script tokenizer as opposed
to the less efficient method of storing and stepping through parsed
opcodes. It also improves several aspects while refactoring such as
optimizing the disassembly trace, showing all scripts in the trace in
the case of execution failure, and providing additional comments
describing the purpose of each field in the engine.
It should be noted that this is a step towards removing the parsed
opcode struct and associated supporting code altogether, however, in
order to ease the review process, this retains the struct and all
function signatures for opcode execution which make use of an individual
parsed opcode. Those will be updated in future commits.
The following is an overview of the changes:
- Modify internal engine scripts slice to use raw scripts instead of
parsed opcodes
- Introduce a tokenizer to the engine to track the current script
- Remove no longer needed script offset parameter from the engine since
that is tracked by the tokenizer
- Add an opcode index counter for disassembly purposes to the engine
- Update check for valid program counter to only consider the script
index
- Update tests for bad program counter accordingly
- Rework the NewEngine function
- Store the raw scripts
- Setup the initial tokenizer
- Explicitly check against version 0 instead of DefaultScriptVersion
which would break consensus if changed
- Check the scripts parse according to version 0 semantics to retain
current consensus rules
- Improve comments throughout
- Rework the Step function
- Use the tokenizer and raw scripts
- Create a parsed opcode on the fly for now to retain existing
opcode execution function signatures
- Improve comments throughout
- Update the Execute function
- Explicitly check against version 0 instead of DefaultScriptVersion
which would break consensus if changed
- Improve the disassembly tracing in the case of error
- Update the CheckErrorCondition function
- Modify clean stack error message to make sense in all cases
- Improve the comments
- Update the DisasmPC and DisasmScript functions on the engine
- Use the tokenizer
- Optimize construction via the use of strings.Builder
- Modify the subScript function to return the raw script bytes since the
parsed opcodes are no longer stored
- Update the various signature checking opcodes to use the raw opcode
data removal and signature hash calculation functions since the
subscript is now a raw script
- opcodeCheckSig
- opcodeCheckMultiSig
- opcodeCheckSigAlt
This converts the engine's current program counter disasembly to make
use of the standalone disassembly function to remove the dependency on
the parsed opcode struct.
It also updates the tests accordingly.
This converts the checkMinimalDataPush function defined on a parsed
opcode to a standalone function which accepts an opcode and data slice
instead in order to make it more flexible for raw script analysis.
It also updates all callers accordingly.
This converts the isConditional function defined on a parsed opcode to a
standalone function named isOpcodeConditional which accepts an opcode as
a byte instead in order to make it more flexible for raw script
analysis.
It also updates all callers accordingly.
This converts the alwaysIllegal function defined on a parsed opcode to a
standalone function named isOpcodeAlwaysIllegal which accepts an opcode
as a byte instead in order to make it more flexible for raw script
analysis.
It also updates all callers accordingly.
This converts the isDisabled function defined on a parsed opcode to a
standalone function which accepts an opcode as a byte instead in order
to make it more flexible for raw script analysis.
It also updates all callers accordingly.
This moves the check for non push-only pay-to-script-hash signature
scripts before the script parsing logic when creating a new engine
instance to avoid the extra overhead in the error case.
This modifies the check for whether or not a pay-to-script-hash
signature script is a push only script to make use of the new and more
efficient raw script function.
Also, since the script will have already been checked further above when
the ScriptVerifySigPushOnly flags is set, avoid checking it again in
that case.
This converts the isAnyKindOfScriptHash function to analyze the raw
script instead of requiring far less efficient parsed opcodes thereby
significantly optimizing the function.
Since the function relies on isStakeScriptHash to identify a stake
tagged pay-to-script-hash, and is the only consumer of it, this also
converts that function to analyze the raw script and renames it to
isStakeScriptHashScript for more consistent naming.
Finally, the tests are updated accordingly.
The following is a before and after comparison of analyzing a large
script:
benchmark old ns/op new ns/op delta
-------------------------------------------------------------------
BenchmarkIsAnyKindOfScriptHash 101249 3.83 -100.00%
benchmark old allocs new allocs delta
-------------------------------------------------------------------
BenchmarkIsAnyKindOfScriptHash 1 0 -100.00%
benchmark old bytes new bytes delta
-------------------------------------------------------------------
BenchmarkIsAnyKindOfScriptHash 466944 0 -100.00%
This tightens the multisig and pay-to-pubkey standard script
identification functions to use the same strict pubkey requirements as
the consensus rules since standardness rules are generally intended to
be more restrictive than the consensus rules which implies they are at a
minimum at least as restrictive.
The tests are also updated to deal with the additional restriction
accordingly.
This renames the flag that indicates whether or not the script engine is
executing a pay-to-script-hash script pair to a name that more
accurately describes its behavior.
This removes the ScriptVerifyMinimalData flag from the txscript package,
changes the default semantics to always enforce its behavior, and
updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_tests.json
specifically dealt with ensuring equivalency of different ways to encode
the same numbers when the ScriptVerifyMinimalData flag is not set.
Therefore, they are no longer necessary.
A few tests which dealt with equivalency that did not already have
expected failing counterparts were converted to expected failure.
Also, several of the tests which dealt with ensuring the specific
encoding of numeric opcodes is being used have been converted to use
hashes since the minimal data requirements specifically prevent
alternate ways of pushing the same encoding which is necessary for
directly checking equality of the raw bytes.
Finally, the MINIMALDATA indicator to enable the flag in the test data
has been retained for now in order to isolate the logic changes as much
as possible.
This removes the ScriptVerifyDERSignatures flag from the txscript
package, changes the default semantics to always enforce its behavior
and updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_tests.json
specifically dealt with ensuring non-DER-compliant signatures were
handled properly when the ScriptVerifyDERSignatures flag was not set.
Therefore, they are no longer necessary.
Finally, the DERSIG indicator to enable the flag in the test data has
been retained for now in order to keep the logic changes separate.
This removes the ScriptBip16 flag from the txscript package, changes the
default semantics to always enforce its behavior, and updates all
callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so there is
no need to require a flag to conditionally toggle it.
Also, since it is no longer possible to invoke the script engine without
the flag with the clean stack flag, it removes the now unused
ErrInvalidFlags error and associated tests.
It should be noted that the test removed from script_tests.json
specifically dealt with ensuring a signature script that contained
non-data-pushing opcodes was successful when neither the ScriptBip16 or
ScriptVerifySigPushOnly flags were set. Therefore, it is no longer
necessary.
Finally, the P2SH indicator to enable the flag in the test data has been
retained for now in order to keep the logic changes separate.
This converts the majority of script errors from generic errors created
via errors.New and fmt.Errorf to use a concrete type that implements the
error interface with an error code and description.
This allows callers to programmatically detect the type of error via
type assertions and an error code while still allowing the errors to
provide more context.
For example, instead of just having an error the reads "disabled opcode"
as would happen prior to these changes when a disabled opcode is
encountered, the error will now read "attempt to execute disabled opcode
OP_FOO".
While it was previously possible to programmatically detect many errors
due to them being exported, they provided no additional context and
there were also various instances that were just returning errors
created on the spot which callers could not reliably detect without
resorting to looking at the actual error message, which is nearly always
bad practice.
Also, while here, export the MaxStackSize and MaxScriptSize constants
since they can be useful for consumers of the package and perform some
minor cleanup of some of the tests.
This cleans up the code for handling the checksig and checkmultisig
opcodes to explicitly call out any semantics that are likely not
obvious, correct some comments, and improve readability.
It also adds several tests to the reference script tests which exercise
the semantics of the check[multi]sig opcodes including both positive and
negative tests.
Finally, it corrects nearly all of the negative tests related to
signature checking of the script tests which were not properly updated
for the differences introduced by Decred so that they fail for the
intended reasons.
The malformed signatures in the tests were very carefully crafted to be
valid except for the very specific condition being tested. The majority
of the negative tests modified and added can be manually verified by
commenting out the relevant checks in the script engine, although a few
of them will pass because they fail for other reasons. In those cases,
prints can be added to ensure the expected failure path is being hit.
This removes the ScriptVerifyStrictEncoding flag from the txscript
package, changes the default semantics to always enforce its behavior
and updates all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
It should be noted that the tests removed from script_valid.json
specifically dealt with ensuring signatures not compliant with DER
encoding did not cause execution to halt early on invalid signatures
when neither of the ScriptVerifyStrictEncoding or
ScriptVerifyDERSignatures flags were set. Therefore, they are no longer
necessary.
For nearly the same reason, the tx test related to the empty pubkey
tx_valid.json was moved to tx_invalid.json. In particular, an empty
pubkey without ScriptVerifyStrictEncoding simply failed the signature
check and continued execution, while the same condition with the flag
halts execution. Thus, without the flag the final NOT in the script
would allow the script to succeed, while it does not under the strict
encoding rules.
Finally, the STRICTENC indicator to enable the flag in the test data has
been retained for now in order to keep the logic changes separate.
This removes the ScriptVerifyLowS flag from the txscript package,
changes the default semantics to always enforce its behavior and updates
all callers in the repository accordingly.
This change is being made to simplify the script engine code since the
flag has always been active and required by consensus in Decred, so
there is no need to require a flag to conditionally toggle it.
This removes the ScriptStrictMultiSig flag from the txscript package
since it is not used or needed by Decred.
The flag is a holdover from the upstream code which was used to address
a bug that does not exist in Decred.
This modifies the script engine to replace OP_NOP3 with
OP_CHECKSEQUENCEVERIFY and adds a flag to selectively enable its
enforcement.
The new opcode examines the top item on the stack and compares it
against the sequence number of the associated transaction input in order
to allow scripts to conditionally enforce the inclusion of relative time
locks to the transaction.
The following is an overview of the changes:
- Introduce a new flag named ScriptVerifyCheckSequenceVerify to
provide conditional enforcement of the new opcode
- Introduce a constant named OP_CHECKSEQUENCEVERIFY which has the same
value as OP_NOP3 since it is replacing it
- Update opcode to name mappings accordingly
- Abstract the logic that deals with time lock verification since it is
the same for both the new opcode and OP_CHECKLOCKTIMEVERIFY
- Implement the required opcode semantics
- Add tests to ensure the opcode works as expected including when used
both correctly and incorrectly
This reverts the changes related to the CheckSequenceVerify opcode that
were merged from upstream since additional changes are needed and it's
much cleaner to implement all of code related to the sequence locks in
the same PR which will be referenced by the DCP as opposed to being
split up in multiple.
This modifies the script engine to replace OP_UNKNOWN192 with OP_SHA256
along with a flag named ScriptVerifySHA256 to selectively enable its
enforcement.
The new opcode consumes the top item from the data stack, computes its
SHA-256, and pushes the resulting digest back to the data stack.
Since it requires an item on the data stack, execution will terminate
with an error when the stack is empty. This behavior differs from
OP_UNKNOWN192 which does not consume any elements from the data stack
and therefore makes this is hard-forking change when interpreted with
the new semantics due to the ScriptVerifySHA256 flag being set. Code to
selectively enable the opcode based on the result of an agenda vote will
be added in a separate commit.
This also includes tests to ensure the opcode works as expected
including when used both correctly and incorrectly.
Upstream commit a6bf1d9850.
The merge commit modifies all of the encoded transactions in the test
data to use Decred native format and contains some other minor
modifications necessary to integrate with Decred.
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
Contains the following upstream commits:
- ef9c50be57
- eb882f39f8
In addition to merging the fixes in the commits, this also fixes a few
more misspellings that were introduced in the new Decred code.
