This completely reworks the way block index and processing works to use
headers-first semantics and support out of order processing of the
associated block data.
This will ultimately provide a wide variety of benefits as it means the
entire shape of the block tree can be determined from the headers alone
which in turn will allow much more informed decisions to be made,
provides better information regarding sync status and warning
conditions, and allows future work to add invalidation and
reconsideration of arbitrary blocks.
It must be noted that this is a major change to the way the block index
and block handling is done and is deeply integrated with the consensus
rules, so it will need a significant amount of testing and extremely
careful review.
Also of note is that there are a lot of assumptions made in the calling
code in regards to expected error attribution and the state the chain
has at the moment notifications are processed, so all of those semantics
have intentionally been retained, even though they might seem a bit out
of place now, in order to limit the amount of changes introduced at a
time and thus better ensures correctness.
For example, in the future it would probably make more sense to make the
notifications entirely asynchronous and for validation failures to be
reported via those asynchronous notifications. However, before that can
happen, all callers would first need to be updated to ensure they do not
rely on the chain being in the same state it was at the moment the
notification was generated.
In addition to all of the existing full block tests, this introduces
a comprehensive set of processing order tests which exercise all of the
new logic.
High level overview of the changes:
- Introduce tracking for whether a block is fully linked, meaning it
builds on a branch that has block data for all of its ancestors
- Add received order tracking to ensure miners are not able to gain an
advantage in terms of chain selection by only advertising a header
- Add several new pieces of information to the block index:
- Header with the most cumulative work that is not known to be invalid
- Header with the most cumulative work that is known to be invalid
- Map of best chain candidates to aid in efficient selection
- Map of unlinked children for more efficient linking
- Prunable cached chain tips to significantly reduce potential search
space during invalidation
- Introduce a compare function which determines which of two nodes
should be considered better for the purposes of best chain selection
- Add chain tip iteration capabilities with potential filtering
- Mark all descendants invalid due to known invalid ancestor when a
block is marked invalid
- Add ability to determine if a block can currently be validated
- Rework the chain reorganization func to work with an arbitrary target
- Modify the overall chain reorg func to reorg to the block with the
most cumulative work that is valid in the case it is not possible to
reorg to the target block
- Add a new MultiError type for keeping track of multiple errors in the
same call
- Add a new ErrNoBlockData error kind
- Update all cases that only require all of the ancestor block data to
be available to check for that condition instead of the more strict
condition of already being fully validated
- Introduce a new processing lock separate from the overall chain lock
- Add new method and ability to independently accept block headers
- Rework the block processing func to accept blocks out of order so long
as their header is already known valid
- Keep a cache of blocks that recently passed contextual validation
checks
- Cleanup and correct various comments to match reality
- Add comprehensive tests to exercise the new processing logic and best
header tracking (for both invalid and not known invalid)
57b59b4e26