This cleans up the code for handling the substr opcode to explicitly
call out its semantics which are likely not otherwise obvious as well as
improve its readability.
This adds module support for the versioned go toolchain. In particular,
the following packages are defined as modules:
* addrmgr
* blockchain
* certgen
* chaincfg
* connmgr
* database
* dcrjson
* dcrutil
* gcs
* hdkeychain
* mempool
* mining
* peer
* rpcclient
* txscript
* wire
It does not update the travis build environment or README since it is
experimental at this point.
This modifies all packages within the repository to the use the Decred
fork of btclog which has been renamed to slog and updates the dependency
files accordingly.
This removes the ScriptStrictMultiSig flag from the txscript package
since it is not used or needed by Decred.
The flag is a holdover from the upstream code which was used to address
a bug that does not exist in Decred.
Standardness rules are not the same as consensus rules and it is
important to keep clear separation between them, because standardness
rules can and do change, while the consensus rules must not change
without a specific vote, and even then, the old rules must be kept
around for validation of all blocks prior to any changes introduced by a
successful vote.
Prior to this commit, the blockchain code which enforces the consensus
rule that requires the second output of the coinbase transaction to
contain the height of the block in a provably pruneable nulldata script
push was relying on code in txscript/standard.go, which, as the name of
the file suggests, is specifically intended for code related to
standardness checks.
This introduces a new function in the txscript package named
ExtractCoinbaseNullData which does not rely on any of the code related
to standardness checks and modifies the blockchain code accordingly to
make use of the new function instead. It also removes the
standardness-dependent GetNullDataContent function which is no longer
used by anything.
Finally, it adds tests in the txscript package to ensure the new
function has the required semantics.
This modifies the signature hash calculation logic to completely
decouple it from the wire transaction serialization and significantly
optimizes it in the process. It also significantly improves the
comments to specifically call out the semantics.
This change is highly desirable because it has several beneficial
properties:
- The signature hash semantics are much clearer and specific
- There is no longer a need to copy the entire transaction and modify
the relevant portions since the necessary substitutions are made on
the fly
- Allows much faster calculation by serializing directly into byte
slices and avoiding all of the additional error handling logic
dedicated to handling network streams
- Provides the possibility of changing the wire format without breaking
signature hash calculation
- Note that the caching portion still relies on the wire format, but
that can be addressed in future commits since it is merely an
optimization that can also be decoupled
The following is a before and after comparison of signature hash
calculation for both speed and memory allocations for a transaction with
many inputs:
benchmark old ns/op new ns/op delta
------------------------------------------------------------
BenchmarkCalcSigHash 6299714 1551740 -75.37%
benchmark old allocs new allocs delta
------------------------------------------------------------
BenchmarkCalcSigHash 18601 1691 -90.91%
This updates the recently-added JSON-based signature hash test data to
actually test the SigHashAnyOneCanPay flag in the non-fuzzer
transactions as intended.
The flag is 0x80, but the JSON test data specifies the signature hash
type in decimal, so SigHashAll|SigHashAnyOneCanPay, which is 0x81,
should've been specified as 129 decimal instead of 81 decimal. The same
applies to all combinations with the SigHashAnyOneCanPay flag.
This adds a significant number of JSON-based tests with various
transactions, scripts, and hash types, to ensure the signature hash
calculated generates the expected hash in each case.
It specifically tests some specialized transactions such as coinbase,
ticket purchase, vote, and revocation. It also includes carefully
created tests for all combinations of defined hash types along with
triggering an error due to improper use of SigHashSingle.
Finally, it also includes fuzz testing by providing a selection of 100
regular transactions with various numbers of inputs and outputs along
with randomly generated (including undefined) signature hash types and
input indexes. Due to the way that masking of the signature hash type
works, a couple of the fuzzer tests also hit the improper use of
SigHashSingle.
The test data was generated as JSON in order to make it easier for
developers of software in other languages and platforms that need to
deal with generating signature hashes to consume and prove correctness
of their implementation as well.
Putting the test code in the same package makes it easier for forks
since they don't have to change the import paths as much and it also
gets rid of the need for internal_test.go to bridge.
Also, do some light cleanup on a few tests while here.
This removes the SigHashAllValue signature hash type. This is being
done because it is not currently usable without a consensus change due
to a consensus rule which enforces strict signature encoding disallowing
the hash type.
While it would be possible to change the consensus rule in question to
include SigHashAllValue, that would obviously require a consensus vote
since it constitutes a change to the consensus rules. Given that a vote
is required to make any changes in regards to this, it is ideal to
completely change the algorithm altogether to not only address this
issue, but also to address other shortcomings in regards to efficiency
and complexity of the current algorithm in addition to committing to all
input amounts per the aforementioned description.
This moves the logic to calculate the signature hash and the associated
test to separate files. Since there will ultimately be a new signature
algorithm, it makes sense to separate all logic related to signature
hashes for better code organization.
It contains no functional changes.
This is a follow-up to a previous change which removed the ability to
create a dcrutil.Address from a hybrid pubkey. It removes all
secp256k1 hybrid pubkey support from the chainec/dcrec packages since
these pubkeys are not usable when evaluated by OP_CHECKSIG or
OP_CHECKMULTISIG.
While ParsePubKey is called by consensus code and its behavior has
been changed to error when encountering a hybrid secp256k1 pubkey,
this is not a consensus change as every use of ParsePubKey in
consensus code either checks beforehand that the pubkey is compressed
or uncompressed, requires a parsed address to be P2PKH or P2SH, or
checks an address string against the premine ledger (which only
contains P2PKH addresses).
This modifies the recently-added NullDataScript function in several
ways in an effort to make them more consistent with the tests in the
rest of the code base and improve/correct the logic:
- Use the hexToBytes and mustParseShortForm functions
- Consistently format the test errors
- Replace the valid bool flag with an expected error and test against it
- Ensure the returned script type is the expected type in all cases
This modifies the exported CalcSignatureHash function to accept a script
as bytes instead of an array of parsed opcodes which are not available
to callers outside of the package.
While here, it also adds a proper comment for the exported function
since it is intended for more than testing as the previous comment
claimed.
Finally, it updates the tests to use a valid script instead of the
previous impossible to achieve constructed series of parsed opcodes.
This merge commit adds the following code from the
github.com/decred/dcrutil package into a new
github.com/decred/dcrd/dcrutil package:
* Address handling
* Amount type
* AppDataDir func
* bitflags functions
* Block wrapper type
* Hash160 func
* Tx wrapper type
* WIF type
as well as all tests for this code.
The old github.com/decred/dcrutil/hdkeychain package has also been
merged and moved to github.com/decred/dcrd/dcrutil/hdkeychain.
dcrd packages have been updated to use the new packages and the dep
files have been updated for this change.
This modifies the script engine to replace OP_NOP3 with
OP_CHECKSEQUENCEVERIFY and adds a flag to selectively enable its
enforcement.
The new opcode examines the top item on the stack and compares it
against the sequence number of the associated transaction input in order
to allow scripts to conditionally enforce the inclusion of relative time
locks to the transaction.
The following is an overview of the changes:
- Introduce a new flag named ScriptVerifyCheckSequenceVerify to
provide conditional enforcement of the new opcode
- Introduce a constant named OP_CHECKSEQUENCEVERIFY which has the same
value as OP_NOP3 since it is replacing it
- Update opcode to name mappings accordingly
- Abstract the logic that deals with time lock verification since it is
the same for both the new opcode and OP_CHECKLOCKTIMEVERIFY
- Implement the required opcode semantics
- Add tests to ensure the opcode works as expected including when used
both correctly and incorrectly
This reverts the changes related to the CheckSequenceVerify opcode that
were merged from upstream since additional changes are needed and it's
much cleaner to implement all of code related to the sequence locks in
the same PR which will be referenced by the DCP as opposed to being
split up in multiple.
This modifies the way standard verification flags are handled so that it
is possible to selectively enable them based on the result of agenda
votes.
First, it moves the StandardVerifyFlags constant from the txscript
package to the mempool/policy code and rename it to
BaseStandardVerifyFlags. As the TODO in the comment of the moved
constant indicated, these flags are policy related and thus really
belong in policy. Ideally there would be a completely separate policy
package, but since the policy code currently lives in mempool/policy.go,
the constant has been moved there.
Next, it introduces a new function named standardScriptVerifyFlags,
which accepts the chain as an argument and, for now, just returns the
BaseStandardVerifyFlags along with a nil error. This will allow
additional flags to be selectively enabled depending on the result of an
agenda vote.
Finally, it updates the mempool policy struct to require a closure for
obtaining the flags so it can remain decoupled from the chain which in
turn allows easier and more robust unit testing of mempool functionality
since it allows a mocks to be used.
This modifies the script engine to replace OP_UNKNOWN192 with OP_SHA256
along with a flag named ScriptVerifySHA256 to selectively enable its
enforcement.
The new opcode consumes the top item from the data stack, computes its
SHA-256, and pushes the resulting digest back to the data stack.
Since it requires an item on the data stack, execution will terminate
with an error when the stack is empty. This behavior differs from
OP_UNKNOWN192 which does not consume any elements from the data stack
and therefore makes this is hard-forking change when interpreted with
the new semantics due to the ScriptVerifySHA256 flag being set. Code to
selectively enable the opcode based on the result of an agenda vote will
be added in a separate commit.
This also includes tests to ensure the opcode works as expected
including when used both correctly and incorrectly.
Decred updated the semantics of this opcode to use blake256, but did not
rename the opcode to reflect that. This renames the opcode so that it
is clear that a blake256 hash is produced instead of a sha256 hash.
Upstream commit a6bf1d9850.
The merge commit modifies all of the encoded transactions in the test
data to use Decred native format and contains some other minor
modifications necessary to integrate with Decred.
Decred's serialized format for transactions split the 32-bit version
field into two 16-bit components such that the upper bits are used to
encode a serialization type and the lower 16 bits are the actual
transaction version.
Unfortunately, when this was done, the in-memory transaction struct was
not also updated to hide this complexity, which means that callers
currently have to understand and take special care when dealing with the
version field of the transaction.
Since the main purpose of the wire package is precisely to hide these
details, this remedies the situation by introducing a new field on the
in-memory transaction struct named SerType which houses the
serialization type and changes the Version field back to having the
desired semantics of actually being the real transaction version. Also,
since the maximum version can only be a 16-bit value, the Version field
has been changed to a uint16 to properly reflect this.
The serialization and deserialization functions now deal with properly
converting to and from these fields to the actual serialized format as
intended.
Finally, these changes also include a fairly significant amount of
related code cleanup and optimization along with some bug fixes in order
to allow the transaction version to be bumped as intended.
The following is an overview of all changes:
- Introduce new SerType field to MsgTx to specify the serialization type
- Change MsgTx.Version to a uint16 to properly reflect its maximum
allowed value
- Change the semantics of MsgTx.Version to be the actual transaction
version as intended
- Update all callers that had special code to deal with the previous
Version field semantics to use the new semantics
- Switch all of the code that deals with encoding and decoding the
serialized version field to use more efficient masks and shifts
instead of binary writes into buffers which cause allocations
- Correct several issues that would prevent producing expected
serializations for transactions with actual transaction versions that
are not 1
- Simplify the various serialize functions to use a single func which
accepts the serialization type to reduce code duplication
- Make serialization type switch usage more consistent with the rest of
the code base
- Update the utxoview and related code to use uint16s for the
transaction version as well since it should not care about the
serialization type due to using its own
- Make code more consistent in how it uses bytes.Buffer
- Clean up several of the comments regarding hashes and add some new
comments to better describe the serialization types
This cleans up the reference test file to remove references to bitcoin
since it no longer applies to decred and remove a TODO that was recently
completed.
This modifies the signatures of all the transactions in the invalid
transaction data which involve checking signatures to ensure that they
are not failing for the wrong reason.
In order to make it easier for future updates, it introduces three
well-known private keys, 1, 2, and 3, that are used for signing all of
the transactions and adds them to the comments in the test data. The
first one is used for regular CHECKSIG while all three are used for
CHECKMULTISIG.
This removes the tests related to max money checks since txscript does
not do sanity checking on the monetary amounts as that is done by the
consensus rules in blockchain.
While here, update dcrscript -> txscript in the comments to reflect
reality.
As an aside, the existing tests were invalid anyways since the
signatures were invalid and thus the tests were failing for the wrong
reason.
This removes the test for the original Bitcoin bug in CHECKMULTISIG that
required an additional argument since that has been fixed in Decred and
the test therefore no longer applies.
As an aside, the existing test was invalid anyways since the signature
was invalid and thus the test would've failed for the wrong reason.
This replaces the tests for CODESEPARATOR in the invalid transaction
JSON data to be correct for Decred. It is always invalid to pass over
the CODESEPARATOR opcode in Decred as opposed to the Bitcoin behavior
where it changes what is hashed for the signature hash calculation.
As an aside, the previous tests were invalid anyways since they included
a signature that was invalid so the test would've failed for the wrong
reason. For all of these tests it's important for them to pass other
than the specific failure condition they are intended to trigger.
This corrects the hashes for the p2sh scripts in the invalid transaction
JSON data. This helps ensure the transactions are failing for the
intended reason as opposed to the p2sh script never being run because
the script hashes don't match.
decred's consensus rules require address generation
to use a compressed pubkey format, this was hacked
into NewAddressSecpPubKey. This commit removes the
hack and provides NewAddressSecpPubKeyCompressed to
enforce the consensus prerequisite
