This overhauls the primary code that deals with synchronizing the
blockchain with other peers on the network to use a model based on
header announcements instead of inventory vectors.
Currently, all blocks are discovered via a combination of inventory
vector announcements and the getblocks protocol message, both of which
only include the hash of the blocks in question.
This is not ideal because it means blocks are blindly downloaded based
on those announcements without knowing if they're actually likely to be
something that is useful since there is no further information such as
what blocks they connect to. It also means that extra logic is needed
to deal with orphan blocks (those whose parents are not yet known) such
as caching them, determining known orphan roots, and expiration.
In short, the current method generally ends up wasting bandwidth and
also makes it much more difficult to detect certain classes of malicious
behavior.
The recently added capability of blockchain to process headers
independently from blocks while the block data is added out of order
later opened the door for a much better model that addresses all of the
aforementioned issues as well as paves the way for many other related
enhancements.
The new model discovers and downloads all of the headers prior to any
block data via the getheaders protocol message and then uses those
headers to determine exactly which blocks need to be downloaded to reach
the tip of the best chain. Notably, this means orphan blocks are now a
thing of the past as blocks that do not connect are no longer
downloaded under any circumstance.
Further, the new model also makes use of sendheaders so that all block
announcements are made via the associated block header. This in turn is
used to better determine if an announced block is likely to be useful
prior to downloading it.
It should be noted that the changes herein are intentionally limited to
those necessary to use the new sync model based on headers in an
incremental fashion to help simplify review and make it easier to assert
correctness. There are many more improvements that this model paves the
way to support planned for future commits. For example, syncing from
multiple peers in parallel and improved DoS protection.
The following is a high-level overview of the key features:
- All headers are downloaded and examined prior to downloading any
blocks
- The initial header sync process:
- Detects and recovers from stalled/malicious peers
- The concept of orphan blocks no longer exists
- This means blocks which are not already known to connect are not
downloaded
- All block announcements are handled via header announcements
- Detects and prevents malicious behavior related to orphan headers
- The chain sync process:
- Starts once the headers are downloaded and entails both downloading
blocks as well as verifying them
- Uses the headers to determine the best blocks to download
- Pipelines the requests to increase throughput
- Actively avoids downloading duplicate blocks
- The sync height is dynamically updated as new headers are discovered
accb6c9472