This converts the tests for calculating signature hashes to use the
exported function which handles the raw script versus the now deprecated
variant requiring parsed opcodes.
This modifies the CalcSignatureHash function to make use of the new
signature hash calculation function that accepts raw scripts without
needing to first parse them. Consequently, it also doubles as a slight
optimization to the execution time and a significant reduction in the
number of allocations.
In order to convert the CalcScriptHash function and keep the same
semantics, a new function named checkScriptParses is introduced which
will quickly determine if a script can be fully parsed without failure
and return the parse failure in the case it can't.
The following is a before and after comparison of analyzing a large
multiple input transaction:
benchmark old ns/op new ns/op delta
-------------------------------------------------------
BenchmarkCalcSigHash 2792057 2760042 -1.15%
benchmark old allocs new allocs delta
-------------------------------------------------------
BenchmarkCalcSigHash 1691 1068 -36.84%
benchmark old bytes new bytes delta
-------------------------------------------------------
BenchmarkCalcSigHash 521673 438604 -15.92%
This introduces a new function named calcSignatureHashRaw which accepts
the raw script bytes to calculate the script hash versus requiring the
parsed opcode only to unparse them later in order to make it more
flexible for working with raw scripts.
Since there are several places in the rest of the code that currently
only have access to the parsed opcodes, this modifies the existing
calcSignatureHash to first unparse the script before calling the new
function.
Note that the code in the signature hash calculation to remove all
instances of OP_CODESEPARATOR from the script is removed because that is
a holdover from BTC code which does not apply to v0 Decred scripts since
OP_CODESEPARATOR is completely disabled in Decred and thus there can
never actually be one in the script.
Finally, it removes the removeOpcode function and related tests since it
is no longer used.
This converts the DisasmString function to make use of the new
zero-allocation script tokenizer instead of the far less efficient
parseScript thereby significantly optimizing the function.
In order to facilitate this, the opcode disassembly functionality is
split into a separate function called disasmOpcode that accepts the
opcode struct and data independently as opposed to requiring a parsed
opcode. The new function also accepts a pointer to a string builder so
the disassembly can be more efficiently be built.
While here, the comment is modified to explicitly call out the script
version semantics.
The following is a before and after comparison of a large script:
benchmark old ns/op new ns/op delta
----------------------------------------------------------
BenchmarkDisasmString 288729 94157 -67.39%
benchmark old bytes new bytes delta
----------------------------------------------------------
BenchmarkDisasmString 584611 177528 -69.63%
This implements an efficient and zero-allocation script tokenizer that
is exported to both provide a new capability to tokenize scripts to
external consumers of the API as well as to serve as a base for
refactoring the existing highly inefficient internal code.
It is important to note that this tokenizer is intended to be used in
consensus critical code in the future, so it must exactly follow the
existing semantics.
The current script parsing mechanism used throughout the txscript module
is to fully tokenize the scripts into an array of internal parsed
opcodes which are then examined and passed around in order to implement
virtually everything related to scripts.
While that approach does simplify the analysis of certain scripts and
thus provide some nice properties in that regard, it is both extremely
inefficient in many cases, and makes it impossible for external
consumers of the API to implement any form of custom script analysis
without manually implementing a bunch of error prone tokenizing code or,
alternatively, the script engine exposing internal structures.
For example, as shown by profiling the total memory allocations of an
initial sync, the existing script parsing code allocates a total of
around 295.12GB, which equates to around 50% of all allocations
performed. The zero-alloc tokenizer this introduces will allow that to
be reduced to virtually zero.
The following is a before and after comparison of tokenizing a large
script with a high opcode count using the existing code versus the
tokenizer this introduces for both speed and memory allocations:
benchmark old ns/op new ns/op delta
------------------------------------------------------------
BenchmarkScriptParsing 153099 961 -99.37%
benchmark old allocs new allocs delta
------------------------------------------------------------
BenchmarkScriptParsing 1 0 -100.00%
benchmark old bytes new bytes delta
------------------------------------------------------------
BenchmarkScriptParsing 466945 0 -100.00%
The following is an overview of the changes:
- Introduce new error code ErrUnsupportedScriptVersion
- Implement zero-allocation script tokenizer
- Add a full suite of tests to ensure the tokenizer works as intended
and follows the required consensus semantics
- Add an example of using the new tokenizer to count the number of
opcodes in a script
- Update README.md to include the new example
- Update script parsing benchmark to use the new tokenizer
This deprecates the GetMultisigMandN function which should never have
been added since the CalcMultiSigStats function already existed for this
purpose.
While here, redefine the function in terms of CalcMultiSigStats.
This function is only useful for internal consensus purposes within the
script engine and as such should not be exported.
While here, also add a comment to specify to the script version
semantics.
This adds the go 1.11 directive to all of the modules in order to
clearly mark they build and work with that version. Go 1.12 modified
the tools such that tidy will automatically add the new version to
modules that do not already have a directive and that would prevent
builds on Go 1.11 through Go 1.11.3 which is not desirable.
As is already well commented in the code, the sequence number parameter
of the CHECKSEQUENCEVERIFY opcode requires 5 bytes instead of the
standard 4 bytes allowed by math opcodes. This introduces a constant
for the value instead of hardcoding 5 to increase readability and
potentially allow the value to be exported in the future.
As is already well commented in the code, the locktime parameter of the
CHECKLOCKTIMEVERIFY opcode requires 5 bytes instead of the standard 4
bytes allowed by math opcodes. This introduces a constant for the value
instead of hardcoding 5 to increase readability and potentially allow
the value to be exported in the future.
This tightens the multisig and pay-to-pubkey standard script
identification functions to use the same strict pubkey requirements as
the consensus rules since standardness rules are generally intended to
be more restrictive than the consensus rules which implies they are at a
minimum at least as restrictive.
The tests are also updated to deal with the additional restriction
accordingly.
This introduces a new error named ErrCheckSigAltVerify and modifies the
opcodeCheckSigAltVerify handler to use the abstractVerify function along
with the new error. This makes the handling consistent with all other
signature checking verification opcode handlers and ensures the error
both can be programmatically detected as well as be uniquely identified
as compared to a generic verify failure.
This renames the flag that indicates whether or not the script engine is
executing a pay-to-script-hash script pair to a name that more
accurately describes its behavior.
An important (and easy for implementations to miss) aspect of the
CHECKSIG opcodes is that the full signature (signature plus hash type)
that is being checked is first removed from the script prior to
calculating the signature hash against which the signature is verified.
It appears the test in the upstream btcsuite code for this was removed
during the initial Decred port instead of being converted as it should
have been.
Consequently, this converts the relevant test so it is correct for
Decred and adds it to the reference tests. Note that the first of the
two added tests is to ensure the original signature is valid prior to
testing the actual removal condition.
This modifies all of the modules, with the exception of the root module,
to remove all replacement directives from their go.mod files and update
the requirements and module sums accordingly.
While it is nice to be able to build and test directly from each module
directory and have it pull in the latest untagged changes when
developing, having all of the overrides in each module makes it
infeasible to use the module tools to help maintain the modules and thus
makes it quite difficult to ensure they are all independently accurate
for external consumers.
By maintaining all of the overrides in the root module and invoking all
builds and tests from it, the overrides will apply to ensure the latest
code is being built and tested.
This also modifies the tests script used with in CI to run all of the
tests from the root module accordingly.
This bumps the various module versions as follows:
- github.com/decred/dcrd/dcrec/secp256k1@v1.0.1
- github.com/decred/dcrd/dcrjson@v1.1.0
- github.com/decred/dcrd/database@v1.0.3
- github.com/decred/dcrd/blockchain/stake@v1.1.0
- github.com/decred/dcrd/mining@v1.1.0
- github.com/decred/dcrd/certgen@v1.0.2
- github.com/decred/dcrd/connmgr@v1.0.2
- github.com/decred/dcrd/mempool@v1.1.0
In addition, it serves as a base for tagging releases of the following
module versions that have previous been bumped since the last release,
but not yet tagged:
- github.com/decred/dcrd/wire@v1.2.0
- github.com/decred/dcrd/chaincfg@v1.2.0
- github.com/decred/dcrd/dcrutil@v1.2.0
- github.com/decred/dcrd/txscript@v1.0.2
- github.com/decred/dcrd/blockchain@v1.1.0
- github.com/decred/dcrd/hdkeychain@v1.1.1
- github.com/decred/dcrd/peer@v1.1.0
- github.com/decred/dcrd/rpcclient@v1.1.0
Finally, it updates all of the dependencies for every module accordingly,
adds a few missing overrides for transitive dependencies, and tidies up
some of the go module sum files.
This modifies the majority of the tests that make use of chain
parameters and the RPC tests to use the resurrected regression test
network.
It also bumps the affected module versions as follows:
- github.com/decred/dcrd/txscript@v1.0.2
- github.com/decred/dcrd/blockchain/stake@v1.0.3
- github.com/decred/dcrd/mempool@v1.0.2
The blockchain and dcrutil modules are also affected, but since their
version has already been bumped since their last release tags, they are
not bumped again.
This bumps the various module versions as follows:
- github.com/decred/dcrd/addrmgr@v1.0.2
- github.com/decred/dcrd/wire@v1.1.0
- github.com/decred/dcrd/chaincfg@v1.1.1
- github.com/decred/dcrd/connmgr@v1.0.1
- github.com/decred/dcrd/dcrutil@v1.1.1
- github.com/decred/dcrd/database@v1.0.1
- github.com/decred/dcrd/hdkeychain@v1.1.0
- github.com/decred/dcrd/txscript@v1.0.1
- github.com/decred/dcrd/blockchain/stake@v1.0.1
- github.com/decred/dcrd/gcs@v1.0.1
- github.com/decred/dcrd/blockchain@v1.0.1
- github.com/decred/dcrd/mining@v1.0.1
- github.com/decred/dcrd/mempool@v1.0.1
- github.com/decred/dcrd/peer@v1.0.1
- github.com/decred/dcrd/rpcclient@v1.0.1
It also updates all of the dependencies for every module accordingly and
adds a few missing overrides for transitive dependencies.
This modifies the test code for several unit tests to prefer using
another network, such as mainnet or simnet, when not specifically
testing something that requires testnet.
In the cases where where testnet is required, it also now prefers a
local reference over directly accessing the testnet parameters directly
from the chaincfg package.
This is being done because the test network undergoes periodic resets
and by reducing the number of direct references, it simplifies the
process of resetting it.
This further refines the modules to add the following new modules
instead of depending on the entire dcrd module:
- github.com/decred/dcrd/dcrjson@v1.0.0
- github.com/decred/dcrd/blockchain@v1.0.0
- github.com/decred/dcrd/blockchain/stake@v1.0.0
- github.com/decred/dcrd/gcs@v1.0.0
- github.com/decred/dcrd/mining@v1.0.0
- github.com/decred/dcrd/mempool@v1.0.0
- github.com/decred/dcrd/peer@v1.0.0
- github.com/decred/dcrd/rpcclient@v1.0.0
Also, it ensures modules that rely on other modules within the repo are
provided replacements to the latest repo code to ensure builds against
master and continuous integration use the latest code.
- github.com/decred/dcrd/addrmgr
- github.com/decred/dcrd/blockchain
- github.com/decred/dcrd/blockchain/stake
- github.com/decred/dcrd/chaincfg
- github.com/decred/dcrd/connmgr
- github.com/decred/dcrd/database
- github.com/decred/dcrd/dcrec/secp256k1
- github.com/decred/dcrd/dcrjson
- github.com/decred/dcrd/dcrutil
- github.com/decred/dcrd/gcs
- github.com/decred/dcrd/hdkeychain
- github.com/decred/dcrd/mempool
- github.com/decred/dcrd/mining
- github.com/decred/dcrd/peer
- github.com/decred/dcrd/rpcclient
- github.com/decred/dcrd/txscript
- github.com/decred/dcrd/wire
Now that the chaincfg, chainhash, dcrec, edwards, secp256k1, dcrutil,
wire, and slog modules have been defined, update the txscript module to
only depend on them instead of the entire dcrd module.
This removes the unused curve parameter from the ParseSignature and
ParseDERSignature functions of the secp256k1 package and updates all
callers in the repository accordingly.
This modifies the PeekInt function of the stack to accept a maximum
script number length to mirror PopInt for consistency. It also updates
the two callers CLTV and CSV) which were manually performing the same
task with 5 bytes due to PeekInt enforcing 4-byte script nums to use the
modified version accordingly.
It also adds some stack tests for 5-byte encodings on both PopInt and
PeekInt.
This removes the flag to require minimal encoding when create script
numbers since since all callers now call the function with true due to
the recent removal of the minimal data script verification flag from the
script engine and updates the tests accordingly.
