82117d52bf
This implements an Age-Partitioned Bloom Filter (APBF) that is safe for concurrent access. An APBF is a probabilistic data structure suitable for use in processing unbounded data streams where more recent items are more significant than older ones and some false positives are acceptable. It has similar computational costs as traditional Bloom filters and provides space efficiency that is competitive with the current best-known, and more complex, Dictionary approaches. Similar to classic Bloom filters, APBFs have a non-zero probability of false positives that can be tuned via parameters and are free from false negatives up to the capacity of the filter. However, unlike classic Bloom filters, where the false positive rate climbs as items are added until all queries are a false positive, APBFs provide a configurable upper bound on the false positive rate for an unbounded number of additions. The unbounded property is achieved by adding and retiring disjoint segments over time where each segment is a slice of a partitioned Bloom filter. The slices are conceptually aged over time as new items are added by shifting them and discarding the oldest one. While APBFs are useful in a variety of use cases, the primary motivation for adding them to Decred at the current time is for use in more efficiently deduplicating various continuous event streams, such as recently confirmed and rejected transactions and inventory (e.g. transactions, blocks, addresses) other peers are known to have. Currently, a LRU cache is used for this purpose in several places which has a non-trivial amount of overhead. For a concrete example of one particular instance, tracking the known addresses for the maximum default number of 125 peers can currently consume up to around 150 MiB. On the other hand, using the APBF implementation this introduces will, once properly updated, allow that to be reduced to around 3.63 MiB while still maintaining a false positive rate of 1 in a million and exhibiting similar computational performance.
3 lines
169 B
Plaintext
3 lines
169 B
Plaintext
github.com/dchest/siphash v1.2.2 h1:9DFz8tQwl9pTVt5iok/9zKyzA1Q6bRGiF3HPiEEVr9I=
|
|
github.com/dchest/siphash v1.2.2/go.mod h1:q+IRvb2gOSrUnYoPqHiyHXS0FOBBOdl6tONBlVnOnt4=
|