add privacy-preserving hourly aggregate of historical scan stats

README.md

@@ -1,213 +1,316 @@
 ![Frigate logo](https://github.com/sparrowwallet/frigate/raw/refs/heads/master/frigatelogo.png)
 
 # Frigate Electrum Server
- 
-Frigate is an experimental Electrum Server testing Silent Payments scanning with ephemeral client keys.
 
-It has four goals:
-1. To provide a proof of concept implementation of the [Remote Scanner](https://github.com/silent-payments/BIP0352-index-server-specification/blob/main/README.md#remote-scanner-ephemeral) approach discussed in the BIP352 Silent Payments Index Server [Specification](https://github.com/silent-payments/BIP0352-index-server-specification/blob/main/README.md) (WIP).
-2. To propose Electrum RPC protocol methods to request and return Silent Payments information from a server.
-3. To demonstrate an efficient "in database" technique of scanning for Silent Payments transactions.
-4. To demonstrate the use of GPU computation to dramatically decrease scanning time.
-5. _(New)_ To demonstrate that GPU compute can be preferred ahead of CPU compute, greatly reducing the resource impact of scanning while remaining widely deployable.
+Frigate is an Electrum server for [Silent Payments](https://github.com/bitcoin/bips/blob/master/bip-0352.mediawiki) (BIP352).
+It performs Silent Payments scanning server-side using ephemeral client keys, returning discovered transactions to clients over an extension to the Electrum JSON-RPC protocol.
+Scanning runs in-database with optional GPU acceleration, supporting both single-user instances on commodity hardware and multi-user public servers.
 
-## Motivation
+## Features
 
-[BIP 352](https://github.com/bitcoin/bips/blob/master/bip-0352.mediawiki) has proposed that light clients use compact block filters to scan for UTXOs received to a Silent Payments address.
-However, this introduces two significant problems:
+- Server-side Silent Payments scanning with ephemeral (in-RAM) client keys, following the [Remote Scanner](https://github.com/silent-payments/BIP0352-index-server-specification/blob/main/README.md#remote-scanner-ephemeral) approach in the BIP352 Index Server Specification.
+- GPU-accelerated scanning with CUDA (NVIDIA), Metal (Apple), and OpenCL (Intel/AMD/NVIDIA) backends, selected automatically at runtime.
+- In-database EC point computation via a custom [DuckDB extension](https://github.com/sparrowwallet/duckdb-ufsecp-extension) wrapping [UltrafastSecp256k1](https://github.com/shrec/UltrafastSecp256k1).
+- Low-latency mempool ingestion via Bitcoin Core's ZMQ `sequence` publisher.
 
-The first is one of data gravity.
-For any reasonable scan period, the client must download gigabytes of data in tweaks, block filters and finally some of the blocks themselves.
-All this data needs to be downloaded, parsed and potentially saved to avoid downloading it again, requiring significant resources on the client. 
-A client would likely need several gigabytes of data to restore a wallet with historical transactions, which is resource intensive in terms of bandwidth, CPU and storage. 
-Compare this to current Electrum clients which may use just a few megabytes to restore a wallet, and it's easy to see how this approach is unlikely to see widespread adoption - it's just too onerous, particularly for mobile clients.
+## Quick Start
 
-The second problem is the lack of mempool monitoring, which is not supported with compact block filters. 
-Users rely on mempool monitoring to answer the "did you get my transaction?" question.
-The lack of ability to do this can cause user confusion and distrust in the wallet, which education can only go some way in reducing.
+1. Install Bitcoin Core 28 or higher with `txindex=1` in `bitcoin.conf`. (28+ is required for Electrum protocol 1.6; earlier versions work for 1.3–1.5.)
+2. Download the latest Frigate release for your platform from the [Releases](https://github.com/sparrowwallet/frigate/releases) page (`.deb` / `.rpm` for Linux, `.dmg` for macOS, `.zip` for Windows). Releases are signed — see [GPG Key](#gpg-key).
+3. Start Frigate:
+   ```shell
+   /opt/frigate/bin/frigate                            # Linux .deb / .rpm
+   /Applications/Frigate.app/Contents/MacOS/Frigate    # macOS
+   ```
+4. With Bitcoin Core on the same host using default settings, no further configuration is required. Frigate creates `~/.frigate/config.toml` on first start.
+5. Frigate indexes from Taproot activation (mainnet) up to the chain tip before serving Silent Payments queries. Watch the log for `Electrum server listening on tcp://...` for the readiness signal.
+6. (Optional) Verify with the bundled CLI:
+   ```shell
+   /opt/frigate/bin/frigate-cli
+   ```
 
-This project attempts to address these problems using an Electrum protocol styled approach.
-Instead of asking the client to download the required data and perform the scanning, the server performs the scanning locally with an optimized index.
-This is the [Remote Scanner](https://github.com/silent-payments/BIP0352-index-server-specification/blob/main/README.md#remote-scanner-ephemeral) approach discussed in the BIP352 Silent Payments Index Server [Specification](https://github.com/silent-payments/BIP0352-index-server-specification/blob/main/README.md) (WIP).
-It should be noted that both the scan private key and the spend public key must be provided to the server in this approach.
-While this does have a privacy implication, the keys are not stored and only held by the server ephemerally (in RAM) for duration of the client session.
-This is similar to the widely used public Electrum server approach, where the wallet addresses are shared ephemerally with a public server. 
+For non-mainnet networks, pass `-n testnet|testnet4|signet|regtest`.
 
-Finally, although this approach will prove to be satisfactory for single user instances, the computation required is still too onerous for a multi-user instance such as an Electrum public server.
-Payment networks are networks subject to Metcalfe's Law like any other, and Silent Payments is only likely to see widespread real world adoption with free-to-use public servers. 
-For this use case, a further step must be taken to leverage modern GPU computation to dramatically improve performance and allow many simultaneous scanning requests.
+## Configuration
 
-## Approach
+Frigate stores its configuration in `~/.frigate/config.toml` on macOS and Linux, and `%APPDATA%\Frigate\config.toml` on Windows.
+A default configuration file is created on first startup. For indexing, Frigate needs access to the Bitcoin Core RPC, which requires `txindex=1` in `bitcoin.conf`.
 
-The key problem that BIP 352 introduces with respect to scanning is that much of the computation cannot be done generally ahead of time.
-Instead, for every silent payment address, each transaction in the blockchain must be considered separately to determine if it sends funds to that address.
-The computation involves several cryptographic operations, including two resource intensive EC point multiplication operations on _every_ eligible transaction.
-In order to ensure that client keys are ephemeral and not stored, this computation must be done in a reasonable period of time on millions of transactions.
+With Bitcoin Core running on the same machine with default settings, Frigate will connect automatically with no configuration changes required.
 
-This is the key difference between Silent Payments wallets and traditional BIP32 wallets, which can rely on a simple monotonically incrementing derivation path index.
-While Silent Payments provides important advantages in privacy and user experience, this computational burden is the downside that cannot be avoided.
-Any solution addressing the retrieval of Silent Payments transactions will eventually be bounded by the performance of EC point multiplication.
-For best performance and user experience this should be done as efficiently as possible, and therefore as close the source data as possible.
+```toml
+# Frigate configuration
 
-In order to achieve this, Frigate addresses the problem of data gravity directly.
-Like most light client silent payment services, it builds an index of the data that can be pre-computed, generally known as a tweak index.
-This index contains a large number of elements (one for every silent payments eligible transaction in the blockchain) containing a tweak calculated from the public keys used in the transaction inputs.
-Frigate stores this data in a single table with the following schema:
+[core]
+connect = true
+# server = "http://127.0.0.1:8332"
+# authType = "COOKIE"            # COOKIE or USERPASS
+# dataDir = "/home/bitcoin/.bitcoin"
+# auth = "user:password"         # only needed for USERPASS
+# zmqSequenceEndpoint = "tcp://127.0.0.1:28336"   # bitcoind -zmqpubsequence endpoint for low-latency mempool ingestion
+# rpcRequestTimeoutSeconds = 60                   # per-RPC read timeout (raise on slow/remote bitcoind, lower on fast LAN)
+# rpcBatchSize = 100                              # max sub-requests per JSON-RPC array batch (mempool fill)
 
-| Column       | Type         |
-|--------------|--------------|
-| `txid`       | BLOB         |
-| `height`     | INTEGER      |
-| `tweak_key`  | BLOB         |
-| `outputs`    | LIST(BIGINT) |
+[index]
+# startHeight = 0                # default: 709632 on mainnet (Taproot activation), 0 on testnet
+# cacheSize = "10M"              # scriptPubKey cache entries (default: 10M, ~4GB RAM)
 
-The `txid` and `tweak_key` values are 32 and 64 byte BLOBS respectively. 
-The `outputs` value is a list of 8 byte integers, each representing the first 8 bytes of the x-value of the Taproot output public key.
+[scan]
+# batchSize = 300000             # rows per GPU dispatch (reduce if scanning hangs on older GPUs)
+# computeBackend = "AUTO"        # AUTO, GPU, or CPU
+# dbThreads = 4                  # limit DuckDB threads (reduces CPU load when computeBackend = "CPU")
+# memoryLimit = "8GB"            # cap DuckDB memory usage (default: 80% of system RAM)
+# maxLabels = 10                 # maximum number of labels accepted per silent payments subscription
+# maxSubscriptions = 100         # maximum number of silent payments subscriptions per connection
+# metricsEnabled = true          # hourly aggregate scan stats log line (default: true)
 
-On startup, Frigate connects to the configured Bitcoin Core RPC, downloads blocks from the configured block height (or from Taproot activation on mainnet) and adds entries to the table.
-Once it has reached the blockchain tip, it starts a simple (and incomplete) Electrum Server to interface with the client.
-
-The scanning is the interesting part.
-Instead of loading data from the table into the Frigate server application, the database itself performs all the required cryptographic operations.
-To do this, Frigate uses a fast OLAP database called [DuckDB](https://duckdb.org/why_duckdb.html#fast) designed for analytical query workloads.
-It then extends the database with a [custom extension](https://github.com/sparrowwallet/duckdb-ufsecp-extension) wrapping [UltrafastSecp256k1](https://github.com/shrec/UltrafastSecp256k1), a high-performance secp256k1 library with CPU and GPU backends.
-This allows the EC point computation to happen as close to the tweak data as possible.
-
-Conceptually, scanning for silent payments requires computing the Taproot output key for `k = 0` and comparing it to the list of known output keys for each tweak row:
-```sql
-SELECT txid, tweak_key, height FROM tweak WHERE list_contains(outputs, hash_prefix_to_int(secp256k1_ec_pubkey_combine([SPEND_PUBLIC_KEY, secp256k1_ec_pubkey_create(secp256k1_tagged_sha256('BIP0352/SharedSecret', secp256k1_ec_pubkey_tweak_mul(tweak_key, SCAN_PRIVATE_KEY) || int_to_big_endian(0)))]), 1));
-```
-The client can then download the transaction and determine if it does indeed contain outputs it is interested in, including for higher values of `k`.
-
-Frigate performs all of these steps at once using a single scanning function that also includes a further step to scan for change:
-```sql
-SELECT txid, tweak_key, height FROM ufsecp_scan((SELECT txid, height, tweak_key, outputs FROM tweak), SCAN_PRIVATE_KEY, SPEND_PUBLIC_KEY, [CHANGE_TWEAK_KEY]);
-```
-The change tweak is added to the computed P<sub>0</sub> and checked again against the outputs for a match.
-All inputs are provided in little endian format, with public keys in uncompressed little endian x,y format to avoid decompression overhead on each tweak.
-UltrafastSecp256k1 uses batch affine addition and KPlan-optimized scalar multiplication to maximize throughput on CPU, and can leverage CUDA, OpenCl or Metal to offload EC computation to the GPU for significantly higher performance.
-
-## Electrum protocol
-
-The Electrum protocol is by far the most widely used light client protocol for Bitcoin wallets, and support is now almost a requirement for widespread adoption of any wallet technology proposal.
-It is characterised by resource efficiency for the client in terms of bandwidth, CPU and storage, allowing a good user experience on almost any platform. 
-It has however been designed around BIP32 wallets. 
-Silent Payments presents an alternative model, where instead of an incrementing derivation path index (and associated gap limit) transactions must be found through scanning the blockchain.
-As such, new methods are necessary.
-Frigate proposes the following Electrum JSON-RPC methods:
-
-### blockchain.silentpayments.subscribe
-
-**Signature**
-```
-blockchain.silentpayments.subscribe(scan_private_key, spend_public_key, start, labels)
+[server]
+# host = "ssl://xyz.com:50002"   # advertised in server.features; use array for multiple. Omit to advertise nothing.
+# tcp = "tcp://0.0.0.0:50001"    # plaintext listener bind URL; omit (or "") to disable. Default if neither tcp nor ssl is set.
+# ssl = "ssl://0.0.0.0:50002"    # SSL listener bind URL; omit to disable
+# sslCert = "cert.pem"           # PEM certificate (chain allowed); bare filename resolves under Frigate's home dir, or use an absolute path
+# sslKey  = "key.pem"            # PEM-encoded PKCS#8 private key; bare filename resolves under Frigate's home dir, or use an absolute path
+# backendElectrumServer = "tcp://localhost:60001"   # backend must listen on a port distinct from Frigate's tcp/ssl listeners above
 ```
 
-- _scan_private_key_: A 64 character string containing the hex of the scan private key.
-- _spend_public_key_: A 66 character string containing the hex of the spend public key.
-- _start_: (Optional) Block height or timestamp to start scanning from. Values above 500,000,000 are treated as seconds from the start of the epoch.
-- _labels_: (Optional) An array of positive integers specifying additional silent payment labels to scan for. Change (`m = 0`) is always included regardless. To aid in wallet recovery, this parameter should only be used for specialized applications. 
+### Core
 
-**Result**
+Set `connect = false` to run Frigate without connecting to Bitcoin Core.
+This is useful if an index has already been built and you just want to serve queries against it.
+The `authType` can be `COOKIE` (default) or `USERPASS`.
+For cookie authentication, set `dataDir` to the Bitcoin Core data directory if it is not in the default location.
+For user/password authentication, set `auth` to `user:password`.
 
-The silent payment address that has been subscribed.
+When `zmqSequenceEndpoint` is set, Frigate subscribes to Bitcoin Core's ZMQ `sequence` publisher for low-latency mempool ingestion (sub-100ms instead of up to 5s).
+This requires Bitcoin Core to be started with `-zmqpubsequence=tcp://...:<port>` matching this endpoint, and a Bitcoin Core build with ZMQ support compiled in.
+Release binaries from bitcoincore.org include ZMQ support; a from-source build needs the ZeroMQ development library (e.g. `libzmq3-dev`) and an explicit `-DWITH_ZMQ=ON`, since CMake silently disables ZMQ if the library is not found.
+Verify support with `bitcoin-cli getzmqnotifications` — a `-32601 Method not found` error means the binary has no ZMQ support.
 
-**Result Example**
+**Configuring `zmqSequenceEndpoint` is strongly recommended whenever `backendElectrumServer` is configured.**
+Without this configured, the backend may notify the client of a new transaction via scripthash before Frigate's silent payments notification lands — causing wallets to briefly display incorrect amounts.
 
-```
-sp1qqgste7k9hx0qftg6qmwlkqtwuy6cycyavzmzj85c6qdfhjdpdjtdgqjuexzk6murw56suy3e0rd2cgqvycxttddwsvgxe2usfpxumr70xc9pkqwv
+The `rpcRequestTimeoutSeconds` setting controls the per-RPC read timeout against Bitcoin Core (default: 60).
+Raise it for a slow or remote node if you see read timeouts in the log; the default is fine for a co-located Bitcoin Core.
+The `rpcBatchSize` setting caps the number of sub-requests per JSON-RPC array batch used during initial mempool indexing (default: 100); the default suits most deployments.
+
+### Index
+
+Indexing speed is greatly affected by looking up the scriptPubKeys of spent outputs.
+To improve performance, scriptPubKeys are cached to avoid looking them up again with `getrawtransaction`.
+The `cacheSize` limits the number of scriptPubKeys cached during indexing (e.g. `"10M"` for 10 million entries, ~4GB RAM).
+This value can be increased or decreased depending on available RAM.
+
+The DuckDB database is stored in a `db` subfolder in the same directory, in a file called `frigate.duckdb`.
+DuckDB databases can be transferred between different operating systems, and should survive unclean shutdowns.
+
+### Scan
+
+The `computeBackend` setting controls whether historical scanning uses GPU or CPU. Valid values are `AUTO` (default), `GPU`, and `CPU`.
+In `AUTO` mode, the GPU is used if one is detected, otherwise the CPU is used.
+Set to `CPU` to force CPU-only scanning.
+With CPU-only scanning, `dbThreads` can be used to limit the number of DuckDB threads and reduce CPU load.
+The `memoryLimit` setting caps DuckDB's memory usage (e.g. `"8GB"`, `"1024MB"`). DuckDB's default is 80% of system RAM.
+Mempool and incremental block scans always run on the CPU backend, since they are short, latency-sensitive and benefit from being decoupled from the longer-running historical scans.
+
+The `batchSize` setting controls how many transactions are processed per GPU dispatch (default: 300,000).
+If scanning hangs or becomes unstable on certain GPUs (particularly older OpenCL-only GPUs), try reducing this value (e.g. 10,000 to 50,000).
+
+The `maxLabels` setting caps the number of labels accepted per silent payments subscription (default: 10).
+The `maxSubscriptions` setting caps the number of silent payments subscriptions per connection (default: 100).
+Requests exceeding either limit are rejected with a JSON-RPC `-32602 Invalid params` error.
+
+When `metricsEnabled` is true (default), Frigate emits one `Aggregate SP scan stats` log line per hour summarising historical scan throughput across all scans in the window.
+The output is bucketed by result count and duration, rounded, and suppresses any bucket with fewer than ten samples, so no per-client scan information is exposed. Set to `false` to disable the line entirely.
+
+### Server
+
+Listeners are configured as bind URLs in the scheme `tcp://` or `ssl://` followed by a host and port.
+The `tcp` setting is the plaintext listener (default: `tcp://0.0.0.0:50001`), and `ssl` enables a TLS listener (e.g. `ssl://0.0.0.0:50002`).
+Each URL specifies the bind interface — use `127.0.0.1` to restrict a listener to localhost, or `0.0.0.0` to bind all interfaces.
+Both listeners can run simultaneously, or either can be omitted (e.g. set `tcp = ""` to run TLS only).
+If neither `tcp` nor `ssl` is set, Frigate defaults to tcp on `0.0.0.0:50001`.
+
+To enable TLS, set `ssl` and supply a certificate and private key via `sslCert` and `sslKey`.
+Bare filenames are resolved relative to Frigate's home directory (the per-network directory that holds `config.toml`). Absolute paths are used as-is.
+The certificate file may contain a single certificate or a full chain (`fullchain.pem`), and the key file must be an unencrypted PKCS#8 (`-----BEGIN PRIVATE KEY-----`) PEM.
+TLS 1.0, 1.1 and SSLv3 are unconditionally disabled as insecure, and only TLS 1.2 and 1.3 are negotiated.
+
+To generate a self-signed certificate for local or testing use, valid for ten years:
+```shell
+openssl req -x509 -newkey rsa:2048 -keyout ~/.frigate/key.pem -out ~/.frigate/cert.pem -days 3650 -nodes -subj "/CN=localhost"
 ```
 
-### Notifications
+Frigate implements the Silent Payments RPCs natively and proxies all other Electrum requests (including address-related lookups) to a co-located Electrum backend.
+The backend is configured with `backendElectrumServer`, and is intended to point to a server running on the same host.
+Because Frigate occupies the canonical 50001/50002 Electrum ports, a co-located backend (Fulcrum, ElectrumX, electrs, etc.) must be configured to listen on a different port.
+The Electrum protocol from 1.3 to 1.6 is supported — for 1.6, ensure Bitcoin Core 28 or higher.
 
-Once subscribed, the client will receive notifications as results are returned from the scan with the following signature. 
-All historical (`progress` < `1.0`) results **must** be sent before current (up to date) results.
-Once the client has received a notification with `progress` == `1.0`, it should consider the scan complete.
+When `backendElectrumServer` is set, also configure `zmqSequenceEndpoint` under `[core]` — see the [Core](#core) section above for why this pairing matters.
 
-```
-blockchain.silentpayments.subscribe(subscription, progress, history)
+## Usage
+
+The Frigate server may be started as follows:
+```shell
+./bin/frigate
 ```
 
-**Result**
-
-A dictionary with the following key/value pairs:
-
-1. A `subscription` JSON object literal containing details of the current subscription:
-- _address_: The silent payment address that has been subscribed to.
-- _labels_: An array of the labels that are subscribed to (must include `0`).
-- _start_height_: The block height from which the subscription scan was started.
-
-2. A `progress` key/value pair indicating the progress of a historical scan:
-- _progress_: A floating point value between `0.0` and `1.0`. Will be `1.0` for all current (up to date) results.
-
-3. A `history` array of transactions. Confirmed transactions are listed in order by height. Each transaction is a dictionary with the following keys:
-- _height_: The integer height of the block the transaction was confirmed in. For mempool transactions, `0` should be used.
-- _tx_hash_: The transaction hash in hexadecimal.
-- _tweak_key_: The tweak key (`input_hash*A`) for the transaction in compressed format.
-
-**Result Example**
-
-```json
-{
-  "subscription": {
-    "address": "sp1qqgste7k9hx0qftg6qmwlkqtwuy6cycyavzmzj85c6qdfhjdpdjtdgqjuexzk6murw56suy3e0rd2cgqvycxttddwsvgxe2usfpxumr70xc9pkqwv",
-    "labels": [0],
-    "start_height": 882000
-  },
-  "progress": 1.0,
-  "history": [
-    {
-      "height": 890004,
-      "tx_hash": "acc3758bd2a26f869fcc67d48ff30b96464d476bca82c1cd6656e7d506816412",
-      "tweak_key": "0314bec14463d6c0181083d607fecfba67bb83f95915f6f247975ec566d5642ee8"
-    },
-    {
-      "height": 905008,
-      "tx_hash": "f3e1bf48975b8d6060a9de8884296abb80be618dc00ae3cb2f6cee3085e09403",
-      "tweak_key": "024ac253c216532e961988e2a8ce266a447c894c781e52ef6cee902361db960004"
-    },
-    {
-      "height": 0,
-      "tx_hash": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16",
-      "tweak_key": "03aeea547819c08413974e2ab2b12212e007166bb2058f88b009e082b9b4914a58"
-    }
-  ]
-}
+or on macOS:
+```shell
+./Frigate.app/Contents/MacOS/Frigate
 ```
 
-It is recommended that servers implementing this protocol send history results incrementally as the historical scan progresses.
-In addition, a maximum page size of 100 is suggested when sending historical transactions.
-This will avoid transmission issues with large wallets that have many transactions, while providing the client with regular progress updates.
-In the case of block reorgs, the server should rescan all existing subscriptions from the reorg-ed block height and send any history (if found) to the client.
-All found mempool transactions should be sent on the initial subscription, but thereafter previously sent mempool transactions should not be resent.
-
-Clients should retrieve the transactions listed in the history with `blockchain.transaction.get` and subscribe to all owned outputs with `blockchain.scripthash.subscribe`. 
-Electrum wallet functionality then proceeds as normal.
-In other words, the silent payments address subscription is a replacement for the monotonically increasing derivation path index in BIP32 wallets.
-The subscription seeks only to add to the client's knowledge of incoming silent payments transactions.
-The client is responsible for checking the transactions do actually send to addresses it has keys for, and using normal Electrum wallet synchronization techniques to monitor for changes to these addresses.
-The tweak key is provided to allow the client to avoid looking up the scriptPubKeys of spent outputs.
-
-### blockchain.silentpayments.unsubscribe
-
-**Signature**
-```
-blockchain.silentpayments.unsubscribe(scan_private_key, spend_public_key)
+To start with a different network, use the `-n` parameter:
+```shell
+./bin/frigate -n signet
 ```
 
-- _scan_private_key_: A 64 character string containing the hex of the scan private key.
-- _spend_public_key_: A 66 character string containing the hex of the spend public key.
+To change the home directory for config and data storage (default: `~/.frigate/` on Linux/macOS, `%APPDATA%\Frigate` on Windows), use the `-d` parameter:
+```shell
+./bin/frigate -d /var/lib/frigate
+```
 
-**Result**
+The full range of options can be queried with:
+```shell
+./bin/frigate -h
+```
 
-The silent payment address that has been unsubscribed. This should cancel any scans that may be currently running for this address.
+### Frigate CLI
 
-**Result Example**
+Frigate also ships a CLI tool called `frigate-cli` to allow easy access to the Electrum RPC.
+```shell
+./bin/frigate-cli
+```
+
+or on macOS:
+```shell
+./Frigate.app/Contents/MacOS/frigate-cli
+```
+
+It uses similar arguments, for example:
+```shell
+./bin/frigate-cli -n signet
+```
+
+The scan private key and spend public key, along with the start block height or timestamp, can be specified as arguments or are prompted for:
+```shell
+./bin/frigate-cli -s SCAN_PRIVATE_KEY -S SPEND_PUBLIC_KEY -b 890000
+```
+
+```shell
+./bin/frigate-cli
+Enter scan private key: SCAN_PRIVATE_KEY
+Enter spend public key: SPEND_PUBLIC_KEY
+Enter start block height or timestamp (optional, press Enter to skip): 890000
+```
+
+By default the CLI client closes once the initial scan is complete, but it can be configured to `follow` or stay open for incoming updates.
+When in follow mode, results are only printed if transactions are found.
+```shell
+./bin/frigate-cli -f
+```
+
+The full range of options can be queried with:
+```shell
+./bin/frigate-cli -h
+```
+
+## Deployment
+
+### Recommended Topology
+
+The expected production deployment is a single host running Bitcoin Core, Frigate, and a co-located Electrum backend (Fulcrum, electrs, or ElectrumX):
 
 ```
-sp1qqgste7k9hx0qftg6qmwlkqtwuy6cycyavzmzj85c6qdfhjdpdjtdgqjuexzk6murw56suy3e0rd2cgqvycxttddwsvgxe2usfpxumr70xc9pkqwv
+                 ┌─────────────────────────────────────────────┐
+client ─TLS/TCP─►│  Frigate :50001 / :50002                    │
+                 │    │                                        │
+                 │    ├─ Silent Payments RPCs (native)         │
+                 │    └─ all other RPCs ─► Electrum backend    │
+                 │                          :60001 (localhost) │
+                 │                            │                │
+                 │                            ▼                │
+                 │                         Bitcoin Core        │
+                 │                          :8332 RPC          │
+                 │                          :28336 ZMQ         │
+                 └─────────────────────────────────────────────┘
 ```
 
+Frigate occupies the canonical Electrum ports (50001/50002). 
+The backend listens on a non-conflicting port (e.g. 60001) bound to localhost, and all non-silent-payments queries are proxied to it transparently.
+
+### Backend Limits
+
+All Frigate client connections to `backendElectrumServer` appear from a single source IP, so backend per-IP caps (e.g. Fulcrum's `max_clients_per_ip`) can refuse new Frigate clients well below the backend's overall capacity.
+The co-located localhost topology above is handled by default: Fulcrum exempts `127.0.0.1/32` and `::1/128`, and ElectrumX exempts all private addresses (RFC1918, loopback, link-local).
+If Frigate runs on a different host from the backend, on Fulcrum add Frigate's IP to `subnets_to_exclude_from_per_ip_limits`.
+On ElectrumX the exemption set is hardcoded to private addresses, so keep Frigate on the same private network.
+electrs imposes no per-IP limits and requires no tuning.
+
+### Running as a Service
+
+Frigate runs as a long-lived foreground process and integrates straightforwardly with `systemd`. A minimal unit (`/etc/systemd/system/frigate.service`):
+
+```ini
+[Unit]
+Description=Frigate Electrum Server
+After=network-online.target bitcoind.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=frigate
+StateDirectory=frigate
+ExecStart=/opt/frigate/bin/frigate -d /var/lib/frigate
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+```
+
+`StateDirectory=frigate` ensures `/var/lib/frigate` exists and is owned by the service user, so the dedicated `frigate` account does not need a real home directory (`useradd --system --no-create-home frigate`). Adjust `User=` and the binary path to match your install.
+
+### Public-facing TLS
+
+For public deployments, supply a real certificate via `sslCert` and `sslKey` rather than the self-signed example in [Configuration > Server](#server).
+The `host` field under `[server]` is advertised in `server.features` and should match the hostname clients use when validating the certificate.
+TLS termination at a reverse proxy is also supported – in that case, run Frigate with `tcp = "tcp://127.0.0.1:50001"` and terminate TLS upstream.
+
+### Resource Requirements
+
+- **RAM**: indexing uses ~4 GB for the default scriptPubKey cache (`cacheSize = "10M"`). Serving steady-state queries is dominated by DuckDB's memory pool, which defaults to 80% of system RAM and can be capped via `memoryLimit`. 16 GB is a comfortable baseline; 8 GB works with a reduced cache and memory limit.
+- **Disk**: ~18 GB for the DuckDB index, growing slowly over time.
+- **CPU**: any modern x86-64 or ARM64 server. Indexing is I/O- and RPC-bound rather than CPU-bound.
+- **GPU**: optional but strongly recommended for multi-user instances. A single discrete GPU (NVIDIA Ampere or newer, Apple Silicon, or any OpenCL 1.2+ device including integrated GPUs) is sufficient for several simultaneous scans. See [GPU Requirements](#gpu-requirements).
+
+## Operations
+
+### Initial Sync
+
+On first start Frigate indexes from Taproot activation (block 709,632 on mainnet) up to the chain tip. 
+It does not begin serving Silent Payments queries until the index reaches the tip — at that point it logs `Electrum server listening on tcp://...`, which is the readiness signal. 
+Indexing time depends on RPC throughput from Bitcoin Core.
+
+### Logs
+
+Logs are written to `frigate.log` in the per-network data directory (`~/.frigate/` for mainnet, `~/.frigate/<network>/` otherwise) and to stdout. 
+The default appender does not rotate; in production, configure `logrotate` or run under a service manager that captures stdout (e.g. `systemd-journald`).
+
+### Backup
+
+The tweak index can be rebuilt from Bitcoin Core at any time, but a rebuild from Taproot activation is slow. To minimise downtime, snapshot `db/frigate.duckdb` under the data directory while Frigate is stopped (DuckDB files are portable across operating systems and survive unclean shutdowns). 
+The `config.toml` and any TLS material in the data directory are worth including in the same backup.
+
+### Upgrades
+
+Stop Frigate, install the new release, start. 
+The DuckDB file format and the on-disk index schema are stable across Frigate releases.
+
+### Reorgs
+
+Frigate detects chain reorgs via Bitcoin Core and rescans the affected block range automatically. 
+Connected silent-payments subscribers are renotified for the rescanned range; clients reconnecting after a disconnect should pass `start = lastSeenHeight - 100` on resubscribe to cover any reorg they missed.
+
 ## Performance
 
 ### CPU Performance
@@ -250,11 +353,10 @@ Intel Core Ultra 9 285K (24 CPUs):
 
 Higher performance on the longer periods is possible by increasing the number of CPUs.
 Multiple clients conducting simultaneous scans slows each scan linearly, since a single scan already saturates all available CPU cores.
-Further performance improvements to this approach may be achieved by scaling out across [multiple read-only replicas of the database](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/read-scaling/).
 
 ### GPU Performance
 
-GPU performance is significantly higher, and as a result is the default compute backend.
+GPU performance is significantly higher, and as a result is the default compute backend for historical scans.
 
 MacBook M1 Pro (Metal GPU backend):
 
@@ -303,7 +405,7 @@ As EC computation is offloaded to the GPU, CPU overhead is low and normal Electr
 Multiple clients conducting simultaneous scans slows each scan linearly, since a single scan already saturates the available GPUs.
 Using multiple GPUs in the same system is also supported and the workload is scaled across them.
 
-A discrete GPU is not required however. 
+A discrete GPU is not required however.
 Frigate can take advantage of any integrated GPU supported by OpenCL, which in practice includes almost all desktop Intel and AMD chips produced in the last decade.
 This prevents saturation of the CPU, and in the case of weaker CPUs (for example older Intel NUCs) can actually be faster.
 See the section below on enabling the iGPU on Linux.
@@ -340,117 +442,6 @@ The `--clients N` option runs N concurrent clients per scan period to test serve
 python3 benchmark.py --clients 4
 ```
 
-## Configuration
-
-For indexing Frigate will need access to the Bitcoin Core RPC, which will need to have `txindex=1` configured.
-
-By default Frigate stores all configuration in `~/.frigate/config` on macOS and Linux, and `%APPDATA%/Frigate` on Windows.
-An example configuration looks as follows
-```json
-{
-  "coreServer": "http://127.0.0.1:8332",
-  "coreAuthType": "COOKIE",
-  "coreDataDir": "/home/bitcoin/.bitcoin",
-  "coreAuth": "bitcoin:password",
-  "startIndexing": true,
-  "indexStartHeight": 0,
-  "scriptPubKeyCacheSize": 10000000,
-  "batchSize": 300000,
-  "computeBackend": "AUTO",
-  "backendElectrumServer": "tcp://localhost:50001"
-}
-```
-
-Default values for these entries will be set on first startup.
-The value of `coreAuthType` can either be `COOKIE` or `USERPASS`. 
-Configure `coreDataDir` or `coreAuth` respectively to grant RPC access.
-The value of `startIndexing` can be set to `false` if an index has already been built and you just want to execute queries against it without connecting to Bitcoin Core.
-
-Indexing speed is greatly affected by looking up the scriptPubKeys of spent outputs.
-To improve performance, scriptPubKeys are cached to avoid looking them up again with `getrawtransaction`.
-The `scriptPubKeyCacheSize` limits the number of scriptPubKeys cached during indexing. 
-The default value leads to a total application memory size of around 4Gb. 
-This value can be increased or decreased depending on available RAM. 
-
-The DuckDB database is stored in a `db` subfolder in the same directory, in a file called `frigate.duckdb`.
-DuckDB databases can be transferred between different operating systems, and should survive unclean shutdowns.
-
-The `computeBackend` setting controls whether scanning uses GPU or CPU. Valid values are `AUTO` (default), `GPU`, and `CPU`.
-In `AUTO` mode, the GPU is used if one is detected, otherwise the CPU is used and a warning is logged.
-Set to `CPU` to force CPU-only scanning and suppress the warning. 
-With CPU-only scanning, to reduce the CPU load set the number of cores made available to DuckDB with `dbThreads`.
-Valid values are any integer equal to or less than number of CPU cores.
-
-The `batchSize` setting controls how many transactions are processed per GPU dispatch (default: 300,000).
-If scanning hangs or becomes unstable on certain GPUs (particularly older OpenCL-only GPUs), try reducing this value (e.g. 10,000 to 50,000).
-
-Frigate currently only implements a selection of Electrum server RPCs directly.
-Any other requests (including address-related lookups) can be proxied to another Electrum server.
-This server is configured with `backendElectrumServer`, and is intended to be used to point to a server running locally on the same host.
-The Electrum protocol from 1.3 to 1.6 is supported - for 1.6, ensure Bitcoin Core 28 or higher.
-
-## Usage
-
-The Frigate server may be started as follows:
-```shell
-./bin/frigate
-```
-
-or on macOS:
-```shell
-./Frigate.app/Contents/MacOS/Frigate
-```
-
-To start with a different network, use the `-n` parameter:
-```shell
-./bin/frigate -n signet
-```
-
-The full range of options can be queried with:
-```shell
-./bin/frigate -h
-```
-
-### Frigate CLI
-
-Frigate also ships a CLI tool called `frigate-cli` to allow easy access to the Electrum RPC.
-```shell
-./bin/frigate-cli
-```
-
-or on macOS:
-```shell
-./Frigate.app/Contents/MacOS/frigate-cli
-```
-
-It uses similar arguments, for example:
-```shell
-./bin/frigate-cli -n signet
-```
-
-The scan private key and spend public key, along with the start block height or timestamp, can be specified as arguments or are prompted for:
-```shell
-./bin/frigate-cli -s SCAN_PRIVATE_KEY -S SPEND_PUBLIC_KEY -b 890000
-```
-
-```shell
-./bin/frigate-cli
-Enter scan private key: SCAN_PRIVATE_KEY
-Enter spend public key: SPEND_PUBLIC_KEY
-Enter start block height or timestamp (optional, press Enter to skip): 890000
-```
-
-By default the CLI client closes once the initial scan is complete, but it can be configured to `follow` or stay open for incoming updates.
-When in follow mode, results are only printed if transactions are found.
-```shell
-./bin/frigate-cli -f
-```
-
-The full range of options can be queried with:
-```shell
-./bin/frigate-cli -h
-```
-
 ## Enabling Intel iGPU on Linux
 
 Frigate supports GPU-accelerated scanning via OpenCL on Intel integrated GPUs.
@@ -489,6 +480,231 @@ clinfo | grep "Device Name"
 
 You should see your Intel GPU listed (e.g. `Intel(R) UHD Graphics [0x7d67]`).
 
+## How It Works
+
+### Motivation
+
+[BIP 352](https://github.com/bitcoin/bips/blob/master/bip-0352.mediawiki) proposes that light clients use compact block filters to scan for UTXOs received to a Silent Payments address.
+This is workable for full nodes but problematic for light clients: a reasonable scan period requires gigabytes of tweaks, filters and blocks to be downloaded, parsed and cached on the client, far exceeding the few megabytes a traditional Electrum wallet uses to restore. 
+It also offers no mempool monitoring, which users rely on to answer the "did you get my transaction?" question.
+
+Frigate addresses both problems by moving scanning to the server, in line with the [Remote Scanner](https://github.com/silent-payments/BIP0352-index-server-specification/blob/main/README.md#remote-scanner-ephemeral) approach in the BIP352 Index Server Specification.
+The scan private key and spend public key are provided to the server, but not stored - they are held in RAM only for the duration of the client session, analogous to how public Electrum servers see ephemeral wallet addresses today.
+
+### The Tweak Index
+
+On startup Frigate connects to Bitcoin Core, downloads blocks from the configured start height (or Taproot activation on mainnet) and builds a tweak index - a single table containing one row per Silent-Payments-eligible transaction:
+
+| Column       | Type         |
+|--------------|--------------|
+| `txid`       | BLOB         |
+| `height`     | INTEGER      |
+| `tweak_key`  | BLOB         |
+| `outputs`    | LIST(BIGINT) |
+
+The `txid` and `tweak_key` values are 32 and 64 byte BLOBs respectively.
+The `outputs` value is a list of 8 byte integers, each representing the first 8 bytes of the x-value of the Taproot output public key.
+
+### In-Database EC Computation
+
+Every silent-payments scan is bounded by EC point multiplication over the tweak index, so Frigate pushes that computation as close to the rows as possible.
+Instead of loading rows into the Frigate server application, scanning is performed by the database itself.
+Frigate uses [DuckDB](https://duckdb.org/why_duckdb.html#fast), a fast OLAP database designed for analytical query workloads, extended with a [custom extension](https://github.com/sparrowwallet/duckdb-ufsecp-extension) wrapping [UltrafastSecp256k1](https://github.com/shrec/UltrafastSecp256k1) - a high-performance secp256k1 library with CPU and GPU backends.
+
+Conceptually, scanning for Silent Payments requires computing the Taproot output key for `k = 0` and comparing it to the list of known output keys for each tweak row:
+```sql
+SELECT txid, tweak_key, height FROM tweak WHERE list_contains(outputs, hash_prefix_to_int(secp256k1_ec_pubkey_combine([SPEND_PUBLIC_KEY, secp256k1_ec_pubkey_create(secp256k1_tagged_sha256('BIP0352/SharedSecret', secp256k1_ec_pubkey_tweak_mul(tweak_key, SCAN_PRIVATE_KEY) || int_to_big_endian(0)))]), 1));
+```
+The client can then download each matching transaction and determine if it does indeed contain outputs it is interested in, including for higher values of `k`.
+
+Frigate performs all of these steps at once using a single scanning function that also includes a further step to scan for change:
+```sql
+SELECT txid, tweak_key, height FROM ufsecp_scan((SELECT txid, height, tweak_key, outputs FROM tweak), SCAN_PRIVATE_KEY, SPEND_PUBLIC_KEY, [CHANGE_TWEAK_KEY]);
+```
+The change tweak is added to the computed P<sub>0</sub> and checked again against the outputs for a match.
+All inputs are provided in little endian format, with public keys in uncompressed little endian x,y format to avoid decompression overhead on each tweak.
+UltrafastSecp256k1 uses batch affine addition and KPlan-optimized scalar multiplication to maximize throughput on CPU, and can leverage CUDA, OpenCL or Metal to offload EC computation to the GPU for significantly higher performance.
+
+## Electrum Protocol
+
+The Electrum protocol is by far the most widely used light client protocol for Bitcoin wallets, and support is now almost a requirement for widespread adoption of any wallet technology proposal.
+It is characterised by resource efficiency for the client in terms of bandwidth, CPU and storage, allowing a good user experience on almost any platform.
+It has however been designed around BIP32 wallets.
+Silent Payments presents an alternative model, where instead of an incrementing derivation path index (and associated gap limit) transactions must be found through scanning the blockchain.
+As such, new methods are necessary.
+Frigate implements the following Electrum JSON-RPC methods, proposed for inclusion in the Electrum protocol:
+
+### server.features
+
+Servers supporting silent payments advertise the capability by including a `silent_payments` field in the `server.features` response:
+
+```json
+{
+  ...,
+  "silent_payments": [0]
+}
+```
+
+Each integer is a fully-specified silent payments protocol version supported by the server.
+BIP352 defines version 0.
+
+### blockchain.silentpayments.subscribe
+
+**Signature**
+```
+blockchain.silentpayments.subscribe(scan_private_key, spend_public_key, start, labels)
+```
+
+- _scan_private_key_: A 64 character string containing the hex of the scan private key.
+- _spend_public_key_: A 66 character string containing the hex of the spend public key.
+- _start_: (Optional) An integer block height to start scanning from, or a string of the form `"FROM-TO"` to specify a closed range of heights. Integer values above 500,000,000 are treated as seconds from the start of the epoch, matching `nLockTime` convention.
+- _labels_: (Optional) An array of positive integers specifying additional silent payment labels to scan for. Change (`m = 0`) is always included regardless. To aid in wallet recovery, this parameter should only be used for specialized applications.
+
+**Result**
+
+A `subscription` JSON object literal containing details of the current subscription:
+- _address_: The silent payment address that has been subscribed to.
+- _labels_: An array of the labels that are subscribed to (must include `0`).
+- _start_height_: The block height from which the subscription scan was started.
+
+If a subscription for the same scan address already exists on the connection, servers **must not** narrow its coverage.
+If the new request's resolved start height is greater than the existing subscription's `start_height`, the server retains the existing wider start and returns it as `start_height` in the response.
+Clients should treat the returned `start_height` as authoritative.
+On every subscribe (including re-subscribe to the same address), the server **must** re-run the historical scan from the returned `start_height` and emit notifications as usual, so that clients which clear their local cache on re-subscribe are repopulated.
+
+**Result Example**
+
+```json
+{
+  "address": "sp1qqgste7k9hx0qftg6qmwlkqtwuy6cycyavzmzj85c6qdfhjdpdjtdgqjuexzk6murw56suy3e0rd2cgqvycxttddwsvgxe2usfpxumr70xc9pkqwv",
+  "labels": [0],
+  "start_height": 882000
+}
+```
+
+### Notifications
+
+Once subscribed, the client will receive notifications as results are returned from the scan with the following signature.
+Servers **must** send the response to `blockchain.silentpayments.subscribe` before any notification for that subscription.
+The `subscription` object in every notification **must** be byte-identical to the one returned in the original `blockchain.silentpayments.subscribe` response with the same `address`, `labels` and `start_height`.
+It identifies which subscription the notification belongs to, not the range that the server happened to scan to produce it.
+All historical (`progress` < `1.0`) results **must** be sent before current (up to date) results.
+A `progress` of `1.0` indicates the scan is up to date as of this notification.
+The first such notification marks the end of the historical scan; subsequent `1.0` notifications represent live updates as new blocks and mempool transactions are scanned.
+Clients should flush any buffered history to local state on every `1.0` notification.
+
+```
+blockchain.silentpayments.subscribe(subscription, progress, history)
+```
+
+**Body**
+
+A dictionary with the following key/value pairs:
+
+1. A `subscription` JSON object literal containing details of the current subscription:
+- _address_: The silent payment address that has been subscribed to.
+- _labels_: An array of the labels that are subscribed to (must include `0`).
+- _start_height_: The block height from which the subscription scan was started.
+
+2. A `progress` key/value pair indicating the progress of a historical scan:
+- _progress_: A floating point value between `0.0` and `1.0`. Will be `1.0` for all current (up to date) results.
+
+3. A `history` array of transactions. Confirmed transactions are listed in order by height, followed by mempool transactions in arbitary order. Each transaction is a dictionary with the following keys:
+- _height_: The integer height of the block the transaction was confirmed in. For mempool transactions, `0` should be used.
+- _tx_hash_: The transaction hash in hexadecimal.
+- _tweak_key_: The tweak key (`input_hash*A`) for the transaction in compressed format.
+
+**Result Example**
+
+```json
+{
+  "subscription": {
+    "address": "sp1qqgste7k9hx0qftg6qmwlkqtwuy6cycyavzmzj85c6qdfhjdpdjtdgqjuexzk6murw56suy3e0rd2cgqvycxttddwsvgxe2usfpxumr70xc9pkqwv",
+    "labels": [0],
+    "start_height": 882000
+  },
+  "progress": 1.0,
+  "history": [
+    {
+      "height": 890004,
+      "tx_hash": "acc3758bd2a26f869fcc67d48ff30b96464d476bca82c1cd6656e7d506816412",
+      "tweak_key": "0314bec14463d6c0181083d607fecfba67bb83f95915f6f247975ec566d5642ee8"
+    },
+    {
+      "height": 905008,
+      "tx_hash": "f3e1bf48975b8d6060a9de8884296abb80be618dc00ae3cb2f6cee3085e09403",
+      "tweak_key": "024ac253c216532e961988e2a8ce266a447c894c781e52ef6cee902361db960004"
+    },
+    {
+      "height": 0,
+      "tx_hash": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16",
+      "tweak_key": "03aeea547819c08413974e2ab2b12212e007166bb2058f88b009e082b9b4914a58"
+    }
+  ]
+}
+```
+
+It is recommended that servers implementing this protocol send history results incrementally as the historical scan progresses.
+In addition, a maximum page size of 1000 is suggested when sending historical transactions.
+This will avoid transmission issues with large wallets that have many transactions, while providing the client with regular progress updates.
+Servers should also emit a notification with empty `history` at regular intervals (e.g. every 5 seconds) during a historical scan, to keep the client updated on scanning progress.
+In the case of block reorgs, the server should rescan all existing subscriptions from the reorg-ed block height and send any history (if found) to the client.
+All found mempool transactions should be sent on the initial subscription, but thereafter previously sent mempool transactions should not be resent.
+The server **must not** re-notify a previously sent mempool transaction once it confirms in a block - clients track confirmation state via `blockchain.scripthash.subscribe`.
+The silent payments subscription is purely a discovery channel.
+Once a silent payments subscription has reached `progress = 1.0`, servers **must** deliver silent payments notifications before any scripthash notification arising from the same block or mempool event, since the silent payments subscription is how clients learn which scripthashes to subscribe to.
+
+Clients reconnecting with prior history should pass `start = lastSeenHeight - reorgLimit` to limit the rescan to a recent window.
+A reorg limit of 100 blocks is sufficient.
+
+For every transaction announced via this subscription, clients **must** retrieve it with `blockchain.transaction.get` and subscribe to all owned outputs with `blockchain.scripthash.subscribe`.
+The scripthash subscription is the canonical source of confirmation height, reorg recovery, and spend tracking; the silent payments subscription does not duplicate any of these.
+In other words, the silent payments address subscription is a replacement for the monotonically increasing derivation path index in BIP32 wallets.
+The subscription seeks only to add to the client's knowledge of incoming silent payments transactions.
+The client is responsible for checking the transactions do actually send to addresses it has keys for, and using normal Electrum wallet synchronization techniques to monitor for changes to these addresses.
+The tweak key is provided to allow the client to avoid looking up the scriptPubKeys of spent outputs.
+
+### blockchain.silentpayments.unsubscribe
+
+**Signature**
+```
+blockchain.silentpayments.unsubscribe(scan_private_key, spend_public_key)
+```
+
+- _scan_private_key_: A 64 character string containing the hex of the scan private key.
+- _spend_public_key_: A 66 character string containing the hex of the spend public key.
+
+**Result**
+
+The silent payment address that has been unsubscribed. This should cancel any scans that may be currently running for this address.
+
+**Result Example**
+
+```
+sp1qqgste7k9hx0qftg6qmwlkqtwuy6cycyavzmzj85c6qdfhjdpdjtdgqjuexzk6murw56suy3e0rd2cgqvycxttddwsvgxe2usfpxumr70xc9pkqwv
+```
+
+## Scan Audit Mode
+
+Frigate includes a correctness testing mode to verify the database extension will find all outputs for a given wallet.
+In normal operation, the tweak index stores the hash prefixes derived from the taproot output public keys of each eligible transaction.
+In audit mode, the index instead stores the hash prefix of the expected change output key (P<sub>0</sub> = B<sub>spend</sub> + t<sub>0</sub>·G) computed from the provided wallet keys.
+This treats every silent payments eligible transaction as paying to the provided wallet.
+The audit can then be run with the following query:
+```sql
+SELECT (SELECT COUNT(*) FROM tweak) AS total_rows, COUNT(*) AS matched_rows FROM ufsecp_scan((SELECT txid, height, tweak_key, outputs FROM tweak), from_hex('<scan_private_key_little_endian>'), from_hex('<spend_public_key_little_endian_x_y>'), [from_hex('<change_label_key_little_endian_x_y>')]); 
+```
+
+This mode is activated by setting two environment variables before starting Frigate:
+
+```shell
+export FRIGATE_AUDIT_SCAN_KEY=<scan_private_key_hex>
+export FRIGATE_AUDIT_SPEND_KEY=<spend_public_key_hex>
+```
+
+When both variables are set, a warning is logged on startup confirming audit mode is active.
+The index must be rebuilt from scratch when switching between normal and audit modes.
+
 ## Building
 
 To clone this project, use
@@ -530,4 +746,3 @@ Frigate is licensed under the Apache 2 software licence.
 The Frigate release binaries here are signed using [craigraw's GPG key](https://keybase.io/craigraw):  
 Fingerprint: D4D0D3202FC06849A257B38DE94618334C674B40  
 64-bit: E946 1833 4C67 4B40
-

build.gradle

@@ -8,7 +8,7 @@ def os = org.gradle.internal.os.OperatingSystem.current()
 def releaseArch = System.getProperty('os.arch') == 'aarch64' ? 'aarch64' : 'x86_64'
 
 group = 'com.sparrowwallet.frigate'
-version = '1.4.0'
+version = '1.5.3'
 
 repositories {
     mavenCentral()
@@ -31,10 +31,16 @@ dependencies {
         exclude group: 'org.slf4j'
     }
     implementation('com.fasterxml.jackson.core:jackson-databind:2.21.1')
+    implementation('com.fasterxml.jackson.dataformat:jackson-dataformat-toml:2.21.1')
     implementation('org.jcommander:jcommander:3.0')
+    implementation('org.zeromq:jeromq:0.6.0')
     implementation ('org.slf4j:slf4j-api:2.0.17')
+    implementation ('ch.qos.logback:logback-classic:1.5.32') {
+        exclude group: 'org.slf4j'
+    }
     testImplementation platform('org.junit:junit-bom:5.14.1')
     testImplementation 'org.junit.jupiter:junit-jupiter'
+    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
 }
 
 test {
@@ -47,6 +53,9 @@ application {
 
     applicationDefaultJvmArgs = ["-XX:+HeapDumpOnOutOfMemoryError",
                                  "--enable-native-access=duckdb.jdbc,com.sparrowwallet.drongo"]
+    if(os.isMacOsX()) {
+        applicationDefaultJvmArgs += ["-Dapple.awt.UIElement=true"]
+    }
 }
 
 tasks.register('runCli', JavaExec) {
@@ -70,7 +79,11 @@ jlink {
             (osName == nativePlatform && arch == nativeArch) ? null : "glob:/com.sparrowwallet.frigate/native/${osName}/${arch}/**"
         }
     } + ['glob:/com.sparrowwallet.merged.module/META-INF/*']).join(',')
-    options = ['--strip-native-commands', '--strip-java-debug-attributes', '--compress', 'zip-6', '--no-header-files', '--no-man-pages', '--ignore-signing-information', '--exclude-files', '**.png', '--exclude-resources', excludeGlobs]
+    def jlinkOptions = ['--strip-native-commands', '--strip-java-debug-attributes', '--compress', 'zip-6', '--no-header-files', '--no-man-pages', '--ignore-signing-information', '--exclude-files', '**.png', '--exclude-resources', excludeGlobs]
+    if(os.isMacOsX()) {
+        jlinkOptions += ['--add-modules', 'java.desktop']
+    }
+    options = jlinkOptions
     launcher {
         name = 'frigate'
         jvmArgs = ["--enable-native-access=duckdb.jdbc,com.sparrowwallet.drongo"]
@@ -88,7 +101,7 @@ jlink {
         if(os.isWindows()) {
             imageOptions = ['--win-console', '--resource-dir', 'src/main/deploy/package/windows/']
         } else if(os.isMacOsX()) {
-            imageOptions = ['--icon', 'src/main/deploy/package/macos/frigate.icns', '--resource-dir', 'src/main/deploy/package/macos/']
+            imageOptions = ['--icon', 'src/main/deploy/package/macos/frigate.icns', '--resource-dir', 'src/main/deploy/package/macos/', '--java-options', '-Dapple.awt.UIElement=true']
         } else if(os.isLinux()) {
             imageOptions = ['--resource-dir', 'src/main/deploy/package/linux/', '--icon', 'src/main/deploy/package/linux/frigate.png']
             installerOptions = ['--linux-rpm-license-type', 'ASL 2.0', '--linux-deb-maintainer', 'mail@sparrowwallet.com']
@@ -172,4 +185,7 @@ extraJavaModuleInfo {
     module('org.jcommander:jcommander', 'org.jcommander') {
         exports('com.beust.jcommander')
     }
+    module('eu.neilalexander:jnacl', 'jnacl') {
+        exports('com.neilalexander.jnacl.crypto')
+    }
 }

src/main/java/com/sparrowwallet/frigate/Frigate.java

@@ -8,17 +8,28 @@ import com.sparrowwallet.frigate.electrum.ElectrumServerRunnable;
 import com.sparrowwallet.frigate.bitcoind.BitcoindClient;
 import com.sparrowwallet.frigate.index.Index;
 import com.sparrowwallet.frigate.index.IndexQuerier;
+import com.sparrowwallet.drongo.OsType;
 import com.sparrowwallet.frigate.io.Config;
+import com.sparrowwallet.frigate.io.Server;
+import com.sparrowwallet.frigate.io.SslUtil;
 import com.sparrowwallet.frigate.io.Storage;
+import javax.net.ssl.SSLContext;
+import java.net.InetSocketAddress;
+import java.net.UnknownHostException;
+import com.github.arteam.simplejsonrpc.client.exception.JsonRpcException;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.File;
+import java.io.IOException;
+import java.net.BindException;
+import java.net.ConnectException;
+import java.sql.SQLException;
 import java.util.Locale;
 
 public class Frigate {
     public static final String SERVER_NAME = "Frigate";
-    public static final String SERVER_VERSION = "1.4.0";
+    public static final String SERVER_VERSION = "1.5.3";
     public static final String APP_HOME_PROPERTY = "frigate.home";
     public static final String NETWORK_ENV_PROPERTY = "FRIGATE_NETWORK";
     private static final int MAINNET_TAPROOT_ACTIVATION_HEIGHT = 709632;
@@ -28,37 +39,43 @@ public class Frigate {
 
     private Index blocksIndex;
     private Index mempoolIndex;
+    private IndexQuerier indexQuerier;
     private BitcoindClient bitcoindClient;
     private ElectrumServerRunnable electrumServer;
 
     private boolean running;
 
+    private static Object trayManager;
+
     public void start() {
+        getLogger().info("Starting " + SERVER_NAME + " v" + SERVER_VERSION + " on " + Network.get().getName());
+        getLogger().info("Using home directory " + Storage.getFrigateHome().getAbsolutePath());
+
         Runtime.getRuntime().addShutdownHook(new Thread(this::stop));
 
-        Integer startHeight = Config.get().getIndexStartHeight();
+        Config config = Config.get();
+
+        Integer startHeight = config.getIndex().getStartHeight();
         if(startHeight == null) {
             startHeight = Network.get() == Network.MAINNET ? MAINNET_TAPROOT_ACTIVATION_HEIGHT : TESTNET_TAPROOT_ACTIVATION_HEIGHT;
-            Config.get().setIndexStartHeight(startHeight);
         }
 
-        int batchSize = Config.get().getBatchSize();
+        int batchSize = config.getScan().getBatchSize();
 
         blocksIndex = new Index(startHeight, false, batchSize);
         mempoolIndex = new Index(0, true, batchSize);
 
-        Boolean startIndexing = Config.get().isStartIndexing();
-        if(startIndexing == null) {
-            startIndexing = true;
-            Config.get().setStartIndexing(startIndexing);
-        }
-
-        if(startIndexing) {
+        if(config.getCore().shouldConnect()) {
             bitcoindClient = new BitcoindClient(blocksIndex, mempoolIndex);
             bitcoindClient.initialize();
         }
 
-        electrumServer = new ElectrumServerRunnable(bitcoindClient, new IndexQuerier(blocksIndex, mempoolIndex));
+        Config.ServerConfig serverConfig = config.getServer();
+        SSLContext sslContext = serverConfig.isSslEnabled() ? SslUtil.getServerSSLContext(serverConfig.getSslCertFile(), serverConfig.getSslKeyFile()) : null;
+        InetSocketAddress tcpBind = toBindAddress(serverConfig.getTcpServer());
+        InetSocketAddress sslBind = toBindAddress(serverConfig.getSslServer());
+        indexQuerier = new IndexQuerier(blocksIndex, mempoolIndex);
+        electrumServer = new ElectrumServerRunnable(bitcoindClient, indexQuerier, tcpBind, sslBind, sslContext);
         Thread electrumServerThread = new Thread(electrumServer, "Frigate Electrum Server");
         electrumServerThread.setDaemon(false);
         electrumServerThread.start();
@@ -71,6 +88,8 @@ public class Frigate {
     }
 
     public void stop() {
+        getLogger().info(SERVER_NAME + " shutting down...");
+
         if(blocksIndex != null) {
             blocksIndex.close();
         }
@@ -83,6 +102,9 @@ public class Frigate {
         if(electrumServer != null) {
             electrumServer.stop();
         }
+        if(indexQuerier != null) {
+            indexQuerier.close();
+        }
 
         running = false;
     }
@@ -91,10 +113,30 @@ public class Frigate {
         return EVENT_BUS;
     }
 
+    private static InetSocketAddress toBindAddress(Server server) {
+        if(server == null) {
+            return null;
+        }
+
+        try {
+            return server.getInetSocketAddress();
+        } catch(UnknownHostException e) {
+            throw new ConfigurationException("Cannot resolve listener bind host '" + server.getHost() + "': " + e.getMessage(), e);
+        }
+    }
+
     private static Logger getLogger() {
         return LoggerFactory.getLogger(Frigate.class);
     }
 
+    private static void initTray() {
+        if(com.sparrowwallet.frigate.control.TrayManager.isSupported()) {
+            com.sparrowwallet.frigate.control.TrayManager mgr = new com.sparrowwallet.frigate.control.TrayManager();
+            EVENT_BUS.register(mgr);
+            trayManager = mgr;
+        }
+    }
+
     public static void main(String[] argv) {
         Args args = new Args();
         JCommander jCommander = JCommander.newBuilder().addObject(args).programName(SERVER_NAME.toLowerCase(Locale.ROOT)).acceptUnknownOptions(true).build();
@@ -150,7 +192,73 @@ public class Frigate {
             getLogger().info("Using " + Network.get() + " configuration");
         }
 
-        Frigate frigate = new Frigate();
-        frigate.start();
+        try {
+            if(OsType.getCurrent() == OsType.MACOS) {
+                initTray();
+            }
+
+            Frigate frigate = new Frigate();
+            frigate.start();
+        } catch(Exception e) {
+            String message = getOperationalErrorMessage(e);
+            if(message != null) {
+                getLogger().error(message);
+            } else {
+                getLogger().error("Fatal error", e);
+            }
+            System.exit(1);
+        }
+    }
+
+    private static String getOperationalErrorMessage(Exception e) {
+        String configExceptionMessage = null;
+        Throwable current = e;
+        while(current != null) {
+            if(current instanceof ConnectException) {
+                String server = Config.get().getCore().getServer();
+                if(server == null) server = "http://127.0.0.1:8332";
+                return "Cannot connect to Bitcoin Core at " + server + ". Ensure Bitcoin Core is running and the server URL in config.toml is correct, or set connect = false under [core].";
+            }
+            if(current instanceof BindException) {
+                Config.ServerConfig sc = Config.get().getServer();
+                Server tcpServer = sc.getTcpServer();
+                Server sslServer = sc.getSslServer();
+                String msg = current.getMessage();
+                String which;
+                if(tcpServer != null && sslServer != null) {
+                    int sslPort = sslServer.getHostAndPort().getPort();
+                    which = msg != null && msg.contains(Integer.toString(sslPort)) ? "SSL at " + sslServer.getUrl() : "TCP at " + tcpServer.getUrl();
+                } else if(tcpServer != null) {
+                    which = "TCP at " + tcpServer.getUrl();
+                } else {
+                    which = "SSL at " + sslServer.getUrl();
+                }
+                return "Configured Electrum server listener (" + which + ") is already in use. Another Frigate instance may be running, or change the listener under [server] in config.toml.";
+            }
+            if(current instanceof IOException ioe && ioe.getMessage() != null) {
+                String msg = ioe.getMessage();
+                if(msg.contains("Cannot find Bitcoin Core cookie file")) {
+                    return msg + ". Ensure Bitcoin Core is running, or set connect = false under [core] in config.toml.";
+                }
+                if(msg.contains("authentication failed")) {
+                    return "Bitcoin Core authentication failed. Check authType, dataDir, and auth under [core] in config.toml, or set connect = false.";
+                }
+            }
+            if(current instanceof SQLException sql && sql.getMessage() != null) {
+                if(sql.getMessage().contains("Could not set lock")) {
+                    return "Database is locked by another process. Ensure no other Frigate instance is using the same database.";
+                }
+            }
+            if(current instanceof JsonRpcException rpc && rpc.getMessage() != null) {
+                if(rpc.getMessage().contains("-txindex")) {
+                    return "Bitcoin Core requires txindex=1 in bitcoin.conf. Restart Bitcoin Core after adding it.";
+                }
+            }
+            if(current instanceof ConfigurationException && configExceptionMessage == null) {
+                configExceptionMessage = current.getMessage();
+            }
+            current = current.getCause();
+        }
+        return configExceptionMessage;
     }
 }

src/main/java/com/sparrowwallet/frigate/bitcoind/BitcoindClient.java

@@ -1,6 +1,7 @@
 package com.sparrowwallet.frigate.bitcoind;
 
 import com.github.arteam.simplejsonrpc.client.JsonRpcClient;
+import com.github.arteam.simplejsonrpc.client.exception.JsonRpcBatchException;
 import com.github.arteam.simplejsonrpc.client.exception.JsonRpcException;
 import com.sparrowwallet.drongo.Network;
 import com.sparrowwallet.drongo.OsType;
@@ -16,9 +17,20 @@ import com.sparrowwallet.frigate.io.RecentBlocksMap;
 import com.sparrowwallet.frigate.io.Server;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import org.zeromq.SocketType;
+import org.zeromq.ZContext;
+import org.zeromq.ZMQ;
 
 import java.io.File;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
 import java.util.*;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
 import java.util.concurrent.locks.Condition;
 import java.util.concurrent.locks.Lock;
 import java.util.concurrent.locks.ReentrantLock;
@@ -26,18 +38,27 @@ import java.util.concurrent.locks.ReentrantLock;
 public class BitcoindClient {
     private static final Logger log = LoggerFactory.getLogger(BitcoindClient.class);
 
-    public static final int DEFAULT_SCRIPT_PUB_KEY_CACHE_SIZE = 10000000;
     private static final int MAX_REORG_DEPTH = 10;
+    private static final int MIN_GETBLOCK_VERBOSE_VERSION = 250000;
     public static final int MIN_SUBMIT_PACKAGE_VERSION = 280000;
 
+    private static final int FLUSH_BATCH_SIZE = 50;
+    private static final long FLUSH_DEBOUNCE_MS = 100;
+    private static final long ZMQ_MEMPOOL_DIFF_INTERVAL_MS = 30_000;
+    private static final long POLL_MEMPOOL_DIFF_INTERVAL_MS = 5_000;
+    private static final long ZMQ_STALENESS_THRESHOLD_MS = 60_000;
+
+    private final BitcoindTransport bitcoindTransport;
     private final JsonRpcClient jsonRpcClient;
+    private final AtomicLong rpcIdCounter = new AtomicLong();
     private final Timer timer = new Timer(true);
     private final Index blocksIndex;
     private final Index mempoolIndex;
 
     private NetworkInfo networkInfo;
     private String lastBlock;
-    private ElectrumBlockHeader tip;
+    private volatile ElectrumBlockHeader tip;
+    private volatile boolean useGetBlockVerbose;
 
     private Exception lastPollException;
 
@@ -47,60 +68,71 @@ public class BitcoindClient {
 
     private volatile boolean stopped;
 
+    private volatile ZContext zmqContext;
+    private volatile Thread zmqSubscriberThread;
+    private volatile Thread zmqConsumerThread;
+    private volatile long lastZmqMessageMs;
+    private final BlockingQueue<MempoolSeqEvent> zmqQueue = new LinkedBlockingQueue<>(50_000);
+    private final AtomicBoolean pollPending = new AtomicBoolean();
+    private long lastMempoolDiffMs;
+
     private final Map<HashIndex, byte[]> scriptPubKeyCache;
-    private final Set<Sha256Hash> mempoolTxIds = new HashSet<>();
+    private final Set<Sha256Hash> mempoolTxIds = ConcurrentHashMap.newKeySet();
     private final RecentBlocksMap recentBlocksMap = new RecentBlocksMap(MAX_REORG_DEPTH);
 
     public BitcoindClient(Index blocksIndex, Index mempoolIndex) {
-        BitcoindTransport bitcoindTransport;
-
-        Config config = Config.get();
-        Server coreServer = config.getCoreServer();
-        if(coreServer == null) {
-            coreServer = new Server("http://127.0.0.1:" + Network.get().getDefaultPort());
-            Config.get().setCoreServer(coreServer);
-        }
-
-        CoreAuthType coreAuthType = config.getCoreAuthType();
-        if(coreAuthType == null) {
-            coreAuthType = CoreAuthType.COOKIE;
-            Config.get().setCoreAuthType(coreAuthType);
-        }
-
-        File coreDataDir = config.getCoreDataDir();
-        if(coreDataDir == null) {
-            coreDataDir = getDefaultCoreDataDir();
-            Config.get().setCoreDataDir(coreDataDir);
-        }
-
-        String coreAuth = config.getCoreAuth();
-        if(coreAuth == null) {
-            coreAuth = "user:password";
-            Config.get().setCoreAuth(coreAuth);
-        }
-
-        if(coreAuthType == CoreAuthType.COOKIE || coreAuth.length() < 2) {
-            bitcoindTransport = new BitcoindTransport(coreServer, coreDataDir);
-        } else {
-            bitcoindTransport = new BitcoindTransport(coreServer, coreAuth);
-        }
+        this(blocksIndex, mempoolIndex, buildDefaultTransport());
+    }
 
+    BitcoindClient(Index blocksIndex, Index mempoolIndex, BitcoindTransport bitcoindTransport) {
+        this.bitcoindTransport = bitcoindTransport;
         this.jsonRpcClient = new JsonRpcClient(bitcoindTransport);
         this.blocksIndex = blocksIndex;
         this.mempoolIndex = mempoolIndex;
 
-        Integer cacheSize = Config.get().getScriptPubKeyCacheSize();
-        if(cacheSize == null) {
-            cacheSize = DEFAULT_SCRIPT_PUB_KEY_CACHE_SIZE;
-            Config.get().setScriptPubKeyCacheSize(cacheSize);
+        int cacheSize = Config.get().getIndex().getCacheSizeEntries();
+        this.scriptPubKeyCache = Collections.synchronizedMap(lruCache(cacheSize));
+    }
+
+    private static BitcoindTransport buildDefaultTransport() {
+        Config.CoreConfig coreConfig = Config.get().getCore();
+
+        Server coreServer = coreConfig.getServerObj();
+        if(coreServer == null) {
+            coreServer = new Server("http://127.0.0.1:" + Network.get().getDefaultPort());
         }
-        this.scriptPubKeyCache = lruCache(cacheSize);
+
+        CoreAuthType coreAuthType = coreConfig.getAuthTypeEnum();
+        if(coreAuthType == null) {
+            coreAuthType = CoreAuthType.COOKIE;
+        }
+
+        File coreDataDir = coreConfig.getDataDirFile();
+        if(coreDataDir == null) {
+            coreDataDir = getDefaultCoreDataDir();
+        }
+
+        String coreAuth = coreConfig.getAuth();
+        if(coreAuth == null) {
+            coreAuth = "user:password";
+        }
+
+        if(coreAuthType == CoreAuthType.COOKIE || coreAuth.length() < 2) {
+            return new BitcoindTransport(coreServer, coreDataDir);
+        }
+
+        return new BitcoindTransport(coreServer, coreAuth);
     }
 
     public void initialize() {
         networkInfo = getBitcoindService().getNetworkInfo();
 
         BlockchainInfo blockchainInfo = getBitcoindService().getBlockchainInfo();
+        useGetBlockVerbose = networkInfo.version() >= MIN_GETBLOCK_VERBOSE_VERSION && !blockchainInfo.pruned();
+        if(useGetBlockVerbose) {
+            log.debug("Using getblock verbosity=3 (Bitcoin Core {})", networkInfo.version());
+        }
+
         VerboseBlockHeader blockHeader = getBitcoindService().getBlockHeader(blockchainInfo.bestblockhash());
         tip = blockHeader.getBlockHeader();
         timer.schedule(new PollTask(), 5000, 5000);
@@ -130,104 +162,467 @@ public class BitcoindClient {
         }
 
         lastBlock = blockchainInfo.bestblockhash();
-        log.info("Initializing indexes...");
+        Frigate.getEventBus().post(tip);
+
+        blocksIndex.repairOrphanTweaks();
+
+        int markerHeight = blocksIndex.getLastBlockIndexed();
+        byte[] storedHash = blocksIndex.getLastBlockHash();
+        if(markerHeight > 0 && storedHash != null) {
+            String currentHashHex = getBitcoindService().getBlockHash(markerHeight);
+            byte[] currentHash = Sha256Hash.wrap(currentHashHex).getBytes();
+            if(!Arrays.equals(storedHash, currentHash)) {
+                int rollbackTo = Math.max(0, markerHeight - MAX_REORG_DEPTH);
+                log.info("Stored block hash at height {} does not match bitcoind - rolling back to height {} and re-indexing", markerHeight, rollbackTo);
+                blocksIndex.removeFromIndex(rollbackTo + 1);
+            }
+        }
+
+        int startHeight = blocksIndex.getLastBlockIndexed() + 1;
+        int endHeight = tip.height();
+        int blocksToIndex = endHeight - startHeight + 1;
+        if(blocksToIndex > 0) {
+            log.info("Indexing {} blocks ({} to {})...", blocksToIndex, startHeight, endHeight);
+        } else {
+            log.info("Block index is up to date");
+        }
+        long startTime = System.currentTimeMillis();
         updateBlocksIndex();
+        if(blocksToIndex > 0) {
+            long elapsedMs = System.currentTimeMillis() - startTime;
+            double blocksPerSec = blocksToIndex / (elapsedMs / 1000.0);
+            log.info("Indexed {} blocks in {}.{}s ({} blocks/sec)", blocksToIndex, elapsedMs / 1000, String.format(Locale.ROOT, "%03d", elapsedMs % 1000), String.format(Locale.ROOT, "%.1f", blocksPerSec));
+        }
+        blocksIndex.setSteadyState(true);
         updateMempoolIndex();
-    }
+        lastMempoolDiffMs = System.currentTimeMillis();
+        Frigate.getEventBus().post(tip);
 
-    private synchronized void updateBlocksIndex() {
-        BitcoindClientService bitcoindService = getBitcoindService();
-        HexFormat hexFormat = HexFormat.of();
-
-        for(int i = blocksIndex.getLastBlockIndexed() + 1; i <= tip.height(); i++) {
-            String blockHash = getBitcoindService().getBlockHash(i);
-            if(i > tip.height() - MAX_REORG_DEPTH) {
-                recentBlocksMap.put(i, blockHash);
-            }
-            String blockHex = (String)bitcoindService.getBlock(blockHash, 0);
-            Block block = new Block(hexFormat.parseHex(blockHex));
-
-            Map<BlockTransaction, byte[]> eligibleTransactions = new LinkedHashMap<>();
-            Map<HashIndex, Script> spentScriptPubKeys = new HashMap<>();
-            for(Transaction tx : block.getTransactions()) {
-                for(int outputIndex = 0; outputIndex < tx.getOutputs().size(); outputIndex++) {
-                    byte[] scriptPubKeyBytes = tx.getOutputs().get(outputIndex).getScriptBytes();
-                    addtoScriptPubKeyCache(tx.getTxId(), outputIndex, scriptPubKeyBytes);
-                }
-
-                if(!tx.isCoinBase() && containsTaprootOutput(tx)) {
-                    for(TransactionInput txInput : tx.getInputs()) {
-                        HashIndex hashIndex = new HashIndex(txInput.getOutpoint().getHash(), txInput.getOutpoint().getIndex());
-                        spentScriptPubKeys.put(hashIndex, getScriptPubKey(bitcoindService, hexFormat, hashIndex));
-                    }
-
-                    byte[] tweak = SilentPaymentUtils.getTweak(tx, spentScriptPubKeys, false);
-                    if(tweak != null) {
-                        BlockTransaction blkTx = new BlockTransaction(tx.getTxId(), i, block.getBlockHeader().getTimeAsDate(), 0L, tx, block.getHash());
-                        eligibleTransactions.put(blkTx, SilentPaymentUtils.getSecp256k1PubKey(tweak));
-                    }
-                }
-            }
-
-            if(!eligibleTransactions.isEmpty()) {
-                blocksIndex.addToIndex(eligibleTransactions);
+        String zmqEndpoint = Config.get().getCore().getZmqSequenceEndpoint();
+        boolean autoDiscoveryAttempted = false;
+        boolean zmqUnsupported = false;
+        if((zmqEndpoint == null || zmqEndpoint.isBlank()) && isLoopbackBitcoind()) {
+            autoDiscoveryAttempted = true;
+            ZmqDiscovery discovery = discoverZmqSequenceEndpoint();
+            zmqEndpoint = discovery.endpoint();
+            zmqUnsupported = discovery.unsupported();
+        }
+        if(zmqEndpoint != null && !zmqEndpoint.isBlank()) {
+            startZmqSequenceSubscriber(zmqEndpoint);
+        } else {
+            long pollSeconds = POLL_MEMPOOL_DIFF_INTERVAL_MS / 1000;
+            String fix = zmqUnsupported ? "This Bitcoin Core binary was built without ZMQ support — install a release build, or rebuild with ZeroMQ (-DWITH_ZMQ=ON)"
+                    : autoDiscoveryAttempted ? "Add -zmqpubsequence=tcp://127.0.0.1:28336 to bitcoin.conf" : "Enable -zmqpubsequence in bitcoin.conf and set zmqSequenceEndpoint in config.toml to match";
+            if(Config.get().getServer().getBackendElectrumServer() != null) {
+                log.warn("Polling bitcoind every {}s with backendElectrumServer configured — clients may briefly display incorrect amounts for silent payments transactions. {}", pollSeconds, fix);
+            } else {
+                log.warn("Polling bitcoind every {}s, for more responsive updates {}", pollSeconds, fix);
             }
         }
     }
 
-    private synchronized void updateMempoolIndex() {
-        BitcoindClientService bitcoindService = getBitcoindService();
+    private synchronized void updateBlocksIndex() {
         HexFormat hexFormat = HexFormat.of();
 
-        Set<Sha256Hash> currentMempoolTxids = bitcoindService.getRawMempool();
-        Set<Sha256Hash> removedTxids = new HashSet<>(mempoolTxIds);
+        int startHeight = blocksIndex.getLastBlockIndexed() + 1;
+        int maxHeight = tip.height();
+        int totalBlocks = maxHeight - startHeight + 1;
+        long lastLogTime = System.currentTimeMillis();
+
+        for(int i = startHeight; i <= maxHeight; i++) {
+            if(useGetBlockVerbose) {
+                indexBlockVerbose(i, hexFormat);
+            } else {
+                indexBlockLegacy(i, hexFormat);
+            }
+
+            long now = System.currentTimeMillis();
+            if(now - lastLogTime >= 30_000) {
+                int blocksProcessed = i - startHeight + 1;
+                double percent = 100.0 * blocksProcessed / totalBlocks;
+                log.info("Indexing progress: {} / {} blocks ({}%, height {})", blocksProcessed, totalBlocks, String.format(Locale.ROOT, "%.1f", percent), i);
+                lastLogTime = now;
+            }
+        }
+    }
+
+    private void indexBlockVerbose(int height, HexFormat hexFormat) {
+        BitcoindClientService bitcoindService = getBitcoindService();
+        String blockHash = bitcoindService.getBlockHash(height);
+        if(height > tip.height() - MAX_REORG_DEPTH) {
+            recentBlocksMap.put(height, blockHash);
+        }
+
+        VerboseBlock vb = bitcoindService.getVerboseBlock(blockHash, 3);
+
+        Map<BlockTransaction, byte[]> eligibleTransactions = new LinkedHashMap<>();
+        for(VerboseBlock.VerboseTransaction vtx : vb.tx()) {
+            //fast path: populate the scriptPubKey cache and check the taproot-output gate directly from the JSON
+            Sha256Hash txid = Sha256Hash.wrap(vtx.txid());
+            boolean hasTaprootOutput = false;
+            for(VerboseBlock.VerboseVout vout : vtx.vout()) {
+                byte[] spkBytes = hexFormat.parseHex(vout.scriptPubKey().hex());
+                addtoScriptPubKeyCache(txid, vout.n(), spkBytes);
+                if(isP2tr(spkBytes)) {
+                    hasTaprootOutput = true;
+                }
+            }
+            boolean isCoinbase = !vtx.vin().isEmpty() && vtx.vin().getFirst().isCoinbase();
+            if(isCoinbase || !hasTaprootOutput) {
+                continue;
+            }
+
+            //eligible: parse the hex now so getInputPubKeys has witness / scriptSig data (not carried in v3 JSON)
+            Transaction tx = new Transaction(hexFormat.parseHex(vtx.hex()));
+            Map<HashIndex, Script> spentScriptPubKeys = new HashMap<>();
+            boolean missingPrevout = false;
+            for(int k = 0; k < vtx.vin().size(); k++) {
+                VerboseBlock.VerboseVin vin = vtx.vin().get(k);
+                if(vin.prevout() == null || vin.prevout().scriptPubKey() == null || vin.prevout().scriptPubKey().hex() == null) {
+                    missingPrevout = true;
+                    break;
+                }
+                TransactionInput in = tx.getInputs().get(k);
+                HashIndex hi = new HashIndex(in.getOutpoint().getHash(), in.getOutpoint().getIndex());
+                byte[] spkBytes = hexFormat.parseHex(vin.prevout().scriptPubKey().hex());
+                spentScriptPubKeys.put(hi, new Script(spkBytes));
+                addtoScriptPubKeyCache(hi.getHash(), (int)hi.getIndex(), spkBytes);
+            }
+
+            if(missingPrevout) {
+                log.warn("getblock verbosity=3 returned no prevout for an input at height {} - falling back to legacy per-input fetch for this block", height);
+                indexBlockLegacy(height, hexFormat);
+                return;
+            }
+
+            byte[] tweak = SilentPaymentUtils.getTweak(tx, spentScriptPubKeys, false);
+            if(tweak != null) {
+                BlockTransaction blkTx = new BlockTransaction(tx.getTxId(), height, new Date(vb.time() * 1000L), 0L, tx, Sha256Hash.wrap(vb.hash()));
+                eligibleTransactions.put(blkTx, SilentPaymentUtils.getSecp256k1PubKey(tweak));
+            }
+        }
+
+        blocksIndex.addToIndex(height, Sha256Hash.wrap(blockHash).getBytes(), eligibleTransactions);
+    }
+
+    private void indexBlockLegacy(int height, HexFormat hexFormat) {
+        BitcoindClientService bitcoindService = getBitcoindService();
+        String blockHash = bitcoindService.getBlockHash(height);
+        if(height > tip.height() - MAX_REORG_DEPTH) {
+            recentBlocksMap.put(height, blockHash);
+        }
+        String blockHex = (String)bitcoindService.getBlock(blockHash, 0);
+        Block block = new Block(hexFormat.parseHex(blockHex));
+
+        Map<BlockTransaction, byte[]> eligibleTransactions = new LinkedHashMap<>();
+        Map<HashIndex, Script> spentScriptPubKeys = new HashMap<>();
+        for(Transaction tx : block.getTransactions()) {
+            for(int outputIndex = 0; outputIndex < tx.getOutputs().size(); outputIndex++) {
+                byte[] scriptPubKeyBytes = tx.getOutputs().get(outputIndex).getScriptBytes();
+                addtoScriptPubKeyCache(tx.getTxId(), outputIndex, scriptPubKeyBytes);
+            }
+
+            if(!tx.isCoinBase() && containsTaprootOutput(tx)) {
+                for(TransactionInput txInput : tx.getInputs()) {
+                    HashIndex hashIndex = new HashIndex(txInput.getOutpoint().getHash(), txInput.getOutpoint().getIndex());
+                    spentScriptPubKeys.put(hashIndex, getScriptPubKey(bitcoindService, hexFormat, hashIndex));
+                }
+
+                byte[] tweak = SilentPaymentUtils.getTweak(tx, spentScriptPubKeys, false);
+                if(tweak != null) {
+                    BlockTransaction blkTx = new BlockTransaction(tx.getTxId(), height, block.getBlockHeader().getTimeAsDate(), 0L, tx, block.getHash());
+                    eligibleTransactions.put(blkTx, SilentPaymentUtils.getSecp256k1PubKey(tweak));
+                }
+            }
+        }
+
+        blocksIndex.addToIndex(height, Sha256Hash.wrap(blockHash).getBytes(), eligibleTransactions);
+    }
+
+    private synchronized void updateMempoolIndex() {
+        HexFormat hexFormat = HexFormat.of();
+
+        //snapshot before the RPC: if the ZMQ consumer removes a txid (R stream) after this point, this diff at worst re-issues a no-op removeFromIndex;
+        //if an RBF re-broadcast lands in getRawMempool() in the same window, the re-broadcast's own A event re-ingests it - the diff is not what re-indexes an RBF re-broadcast
+        Set<Sha256Hash> knownTxids = new HashSet<>(mempoolTxIds);
+        Set<Sha256Hash> currentMempoolTxids = getBitcoindService().getRawMempool();
+        Set<Sha256Hash> removedTxids = new HashSet<>(knownTxids);
         removedTxids.removeAll(currentMempoolTxids);
         Set<Sha256Hash> addedTxids = new HashSet<>(currentMempoolTxids);
-        addedTxids.removeAll(mempoolTxIds);
+        addedTxids.removeAll(knownTxids);
 
         Map<BlockTransaction, byte[]> eligibleTransactions = new LinkedHashMap<>();
         Map<HashIndex, Script> spentScriptPubKeys = new HashMap<>();
 
-        for(Sha256Hash addedTxid : addedTxids) {
-            try {
-                String txHex = (String)getBitcoindService().getRawTransaction(addedTxid.toString(), false);
-                Transaction tx = new Transaction(hexFormat.parseHex(txHex));
-                for(int outputIndex = 0; outputIndex < tx.getOutputs().size(); outputIndex++) {
-                    byte[] scriptPubKeyBytes = tx.getOutputs().get(outputIndex).getScriptBytes();
-                    addtoScriptPubKeyCache(tx.getTxId(), outputIndex, scriptPubKeyBytes);
+        try {
+            Map<Sha256Hash, String> hexByTxid = fetchRawTxBatch(addedTxids);
+            for(Sha256Hash addedTxid : addedTxids) {
+                String txHex = hexByTxid.get(addedTxid);
+                if(txHex == null) {
+                    continue;
                 }
-
-                if(!tx.isCoinBase() && containsTaprootOutput(tx)) {
-                    for(TransactionInput txInput : tx.getInputs()) {
-                        HashIndex hashIndex = new HashIndex(txInput.getOutpoint().getHash(), txInput.getOutpoint().getIndex());
-                        spentScriptPubKeys.put(hashIndex, getScriptPubKey(bitcoindService, hexFormat, hashIndex));
-                    }
-
-                    byte[] tweak = SilentPaymentUtils.getTweak(tx, spentScriptPubKeys, false);
-                    if(tweak != null) {
-                        BlockTransaction blkTx = new BlockTransaction(tx.getTxId(), 0, null, 0L, tx, null);
-                        eligibleTransactions.put(blkTx, SilentPaymentUtils.getSecp256k1PubKey(tweak));
-                    }
-                }
-            } catch(JsonRpcException e) {
-                //ignore, transaction removed from mempool
+                ingestMempoolTxFromHex(addedTxid, txHex, spentScriptPubKeys, eligibleTransactions, hexFormat);
             }
-        }
 
-        if(!removedTxids.isEmpty()) {
-            mempoolIndex.removeFromIndex(removedTxids);
-        }
-        if(!eligibleTransactions.isEmpty()) {
-            mempoolIndex.addToIndex(eligibleTransactions);
+            if(!removedTxids.isEmpty()) {
+                mempoolIndex.removeFromIndex(removedTxids);
+            }
+            if(!eligibleTransactions.isEmpty()) {
+                mempoolIndex.addToIndex(0, null, eligibleTransactions);
+            }
+        } catch(RuntimeException e) {
+            //nothing was committed to the index - drop the would-be-indexed txids so a later diff retries them
+            eligibleTransactions.keySet().forEach(blkTx -> mempoolTxIds.remove(blkTx.getHash()));
+            throw e;
         }
 
         mempoolTxIds.removeAll(removedTxids);
-        mempoolTxIds.addAll(addedTxids);
+    }
+
+    @SuppressWarnings("unchecked")
+    private Map<Sha256Hash, String> fetchRawTxBatch(Collection<Sha256Hash> txids) {
+        if(txids.isEmpty()) {
+            return Map.of();
+        }
+        PagedBatchRequestBuilder<Sha256Hash, String> builder = PagedBatchRequestBuilder.create(bitcoindTransport, rpcIdCounter)
+                .keysType(Sha256Hash.class).returnType(String.class);
+        for(Sha256Hash txid : txids) {
+            builder.add(txid, "getrawtransaction", txid.toString(), false);
+        }
+        try {
+            return builder.execute();
+        } catch(JsonRpcBatchException e) {
+            return (Map<Sha256Hash, String>)e.getSuccesses();
+        } catch(Exception e) {
+            if(e instanceof RuntimeException re) {
+                throw re;
+            }
+            throw new RuntimeException("Error executing batched getrawtransaction request", e);
+        }
+    }
+
+    private void ingestMempoolTx(Sha256Hash txid, Map<HashIndex, Script> spentScriptPubKeys, Map<BlockTransaction, byte[]> eligibleTransactions, HexFormat hexFormat) {
+        String txHex;
+        try {
+            txHex = (String)getBitcoindService().getRawTransaction(txid.toString(), false);
+        } catch(JsonRpcException e) {
+            //transaction removed from mempool before we could fetch it
+            return;
+        }
+
+        ingestMempoolTxFromHex(txid, txHex, spentScriptPubKeys, eligibleTransactions, hexFormat);
+    }
+
+    /**
+     * Ingest an already-fetched mempool transaction. Idempotent — gated on an atomic insert into {@link #mempoolTxIds},
+     * so a txid already ingested (via ZMQ or the safety-net diff) is skipped.
+     */
+    private void ingestMempoolTxFromHex(Sha256Hash txid, String txHex, Map<HashIndex, Script> spentScriptPubKeys, Map<BlockTransaction, byte[]> eligibleTransactions, HexFormat hexFormat) {
+        if(!mempoolTxIds.add(txid)) {
+            return;
+        }
+
+        try {
+            Transaction tx = new Transaction(hexFormat.parseHex(txHex));
+            for(int outputIndex = 0; outputIndex < tx.getOutputs().size(); outputIndex++) {
+                byte[] scriptPubKeyBytes = tx.getOutputs().get(outputIndex).getScriptBytes();
+                addtoScriptPubKeyCache(tx.getTxId(), outputIndex, scriptPubKeyBytes);
+            }
+
+            if(!tx.isCoinBase() && containsTaprootOutput(tx)) {
+                BitcoindClientService bitcoindService = getBitcoindService();
+                for(TransactionInput txInput : tx.getInputs()) {
+                    HashIndex hashIndex = new HashIndex(txInput.getOutpoint().getHash(), txInput.getOutpoint().getIndex());
+                    spentScriptPubKeys.put(hashIndex, getScriptPubKey(bitcoindService, hexFormat, hashIndex));
+                }
+
+                byte[] tweak = SilentPaymentUtils.getTweak(tx, spentScriptPubKeys, false);
+                if(tweak != null) {
+                    BlockTransaction blkTx = new BlockTransaction(tx.getTxId(), 0, null, 0L, tx, null);
+                    eligibleTransactions.put(blkTx, SilentPaymentUtils.getSecp256k1PubKey(tweak));
+                }
+            }
+        } catch(RuntimeException e) {
+            //transient failure - drop the txid so a later diff re-ingests it
+            mempoolTxIds.remove(txid);
+            throw e;
+        }
+    }
+
+    private boolean isLoopbackBitcoind() {
+        Server coreServer = Config.get().getCore().getServerObj();
+        String host = coreServer != null ? coreServer.getHost() : "127.0.0.1";
+        try {
+            return InetAddress.getByName(host).isLoopbackAddress();
+        } catch(UnknownHostException e) {
+            return false;
+        }
+    }
+
+    private ZmqDiscovery discoverZmqSequenceEndpoint() {
+        try {
+            for(ZmqNotification notification : getBitcoindService().getZmqNotifications()) {
+                if("pubsequence".equals(notification.type()) && notification.address() != null) {
+                    return new ZmqDiscovery(normaliseZmqAddress(notification.address()), false);
+                }
+            }
+        } catch(JsonRpcException e) {
+            if(e.getErrorMessage() != null && e.getErrorMessage().getCode() == -32601) {
+                return new ZmqDiscovery(null, true);
+            }
+            log.debug("Could not auto-discover ZMQ endpoint from Bitcoin Core", e);
+        } catch(Exception e) {
+            log.debug("Could not auto-discover ZMQ endpoint from Bitcoin Core", e);
+        }
+
+        return new ZmqDiscovery(null, false);
+    }
+
+    private static String normaliseZmqAddress(String address) {
+        //wildcard binds aren't usable as connect targets; substitute loopback
+        if(address.startsWith("tcp://0.0.0.0:")) {
+            return "tcp://127.0.0.1:" + address.substring("tcp://0.0.0.0:".length());
+        }
+        if(address.startsWith("tcp://[::]:")) {
+            return "tcp://[::1]:" + address.substring("tcp://[::]:".length());
+        }
+
+        return address;
+    }
+
+    private void startZmqSequenceSubscriber(String endpoint) {
+        if(stopped) {
+            return;
+        }
+
+        zmqContext = new ZContext();
+        zmqSubscriberThread = Thread.ofPlatform().daemon().name("BitcoindZmqSequence").start(() -> {
+            try(ZMQ.Socket socket = zmqContext.createSocket(SocketType.SUB)) {
+                socket.setReconnectIVL(100);
+                socket.setReconnectIVLMax(10_000);
+                socket.subscribe("sequence");
+                socket.connect(endpoint);
+                log.info("Subscribed to ZMQ sequence publisher at {}", endpoint);
+
+                while(!stopped && !Thread.currentThread().isInterrupted()) {
+                    String topic = socket.recvStr();
+                    byte[] body = socket.hasReceiveMore() ? socket.recv() : null;
+                    while(socket.hasReceiveMore()) {
+                        socket.recv();  //drain remaining frames (the per-topic sequence number, plus any added in future bitcoind versions)
+                    }
+                    if(!"sequence".equals(topic) || body == null || body.length < 33) {
+                        continue;
+                    }
+                    lastZmqMessageMs = System.currentTimeMillis();
+
+                    char label = (char)body[32];
+                    if(label == 'A' || label == 'R') {
+                        Sha256Hash txid = Sha256Hash.wrap(Arrays.copyOf(body, 32));
+                        if(!zmqQueue.offer(new MempoolSeqEvent(txid, label == 'R'))) {
+                            log.warn("ZMQ mempool queue full, dropping {} for txid {} (safety-net diff will recover it)", label, txid);
+                        }
+                    } else if(label == 'C' || label == 'D') {
+                        //block connect/disconnect: don't ingest here, just kick an immediate PollTask run - reorg detection, tip update and the forced mempool diff all live there
+                        //pollPending coalesces bursts (testnet3 difficulty-reset storms, multi-block reorgs, post-outage catch-up) down to ~1 in-flight + 1 queued
+                        if(!stopped && pollPending.compareAndSet(false, true)) {
+                            timer.schedule(new PollTask(), 0);
+                        }
+                    }
+                    //'R' fires for non-block removals only (RBF/eviction/expiry); mined txs produce 'C' (block connect) with no per-tx 'R'
+                }
+            } catch(Throwable t) {
+                if(!stopped) {
+                    log.error("Bitcoin Core ZMQ sequence subscriber exited", t);
+                }
+            }
+        });
+
+        zmqConsumerThread = Thread.ofPlatform().daemon().name("BitcoindZmqConsumer").start(this::zmqConsumerLoop);
+
+        timer.schedule(new TimerTask() {
+            @Override
+            public void run() {
+                if(!stopped && lastZmqMessageMs == 0L && Network.get() == Network.MAINNET) {
+                    log.warn("No ZMQ messages received from Bitcoin Core within 60s at {} - verify -zmqpubsequence is configured on this endpoint. " +
+                            "Mempool ingestion latency will be up to {}s until ZMQ messages arrive", endpoint, ZMQ_MEMPOOL_DIFF_INTERVAL_MS / 1000);
+                }
+            }
+        }, 60_000);
+    }
+
+    private void zmqConsumerLoop() {
+        Map<BlockTransaction, byte[]> eligibleTransactions = new LinkedHashMap<>();
+        Set<Sha256Hash> removedTxids = new HashSet<>();
+        Map<HashIndex, Script> spentScriptPubKeys = new HashMap<>();
+        HexFormat hexFormat = HexFormat.of();
+
+        while(!stopped && !Thread.currentThread().isInterrupted()) {
+            try {
+                MempoolSeqEvent first = zmqQueue.poll(1, TimeUnit.SECONDS);
+                if(first == null) {
+                    continue;
+                }
+
+                long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(FLUSH_DEBOUNCE_MS);
+                processSeqEvent(first, spentScriptPubKeys, eligibleTransactions, removedTxids, hexFormat);
+                while(eligibleTransactions.size() + removedTxids.size() < FLUSH_BATCH_SIZE) {
+                    long remaining = deadline - System.nanoTime();
+                    if(remaining <= 0) {
+                        break;
+                    }
+                    MempoolSeqEvent next = zmqQueue.poll(remaining, TimeUnit.NANOSECONDS);
+                    if(next == null) {
+                        break;
+                    }
+                    processSeqEvent(next, spentScriptPubKeys, eligibleTransactions, removedTxids, hexFormat);
+                }
+
+                //add before remove: if a txid somehow ended up in both batches (it shouldn't - see processSeqEvent), the net result is "absent", correct for an A-then-R
+                if(!eligibleTransactions.isEmpty()) {
+                    mempoolIndex.addToIndex(0, null, eligibleTransactions);
+                    eligibleTransactions.clear();
+                }
+                if(!removedTxids.isEmpty()) {
+                    mempoolIndex.removeFromIndex(removedTxids);
+                    removedTxids.clear();
+                }
+                spentScriptPubKeys.clear();
+            } catch(InterruptedException e) {
+                return;
+            } catch(Throwable t) {
+                log.error("Error processing ZMQ mempool transactions", t);
+                eligibleTransactions.keySet().forEach(blkTx -> mempoolTxIds.remove(blkTx.getHash()));
+                eligibleTransactions.clear();
+                mempoolTxIds.addAll(removedTxids);
+                removedTxids.clear();
+                spentScriptPubKeys.clear();
+            }
+        }
+    }
+
+    private void processSeqEvent(MempoolSeqEvent event, Map<HashIndex, Script> spentScriptPubKeys, Map<BlockTransaction, byte[]> eligibleTransactions, Set<Sha256Hash> removedTxids, HexFormat hexFormat) {
+        if(event.removed()) {
+            if(mempoolTxIds.remove(event.txid())) {
+                removedTxids.add(event.txid());
+                eligibleTransactions.keySet().removeIf(blkTx -> blkTx.getHash().equals(event.txid()));
+            }
+        } else {
+            ingestMempoolTx(event.txid(), spentScriptPubKeys, eligibleTransactions, hexFormat);
+        }
     }
 
     public void stop() {
         timer.cancel();
         stopped = true;
+        if(zmqSubscriberThread != null) {
+            zmqSubscriberThread.interrupt();
+        }
+        if(zmqConsumerThread != null) {
+            zmqConsumerThread.interrupt();
+        }
+        if(zmqContext != null) {
+            zmqContext.close();
+        }
     }
 
     public BitcoindClientService getBitcoindService() {
@@ -263,11 +658,15 @@ public class BitcoindClient {
     private class PollTask extends TimerTask {
         @Override
         public void run() {
+            //clear at the start: a 'C' arriving while this run is in flight re-arms pollPending and schedules exactly one follow-up poll (which picks up anything connected during this run)
+            pollPending.set(false);
+
             if(stopped) {
                 timer.cancel();
             }
 
             try {
+                boolean newBlock = false;
                 if(syncing) {
                     BlockchainInfo blockchainInfo = getBitcoindService().getBlockchainInfo();
                     if(blockchainInfo.initialblockdownload() && !isEmptyBlockchain(blockchainInfo)) {
@@ -302,11 +701,13 @@ public class BitcoindClient {
                             log.info("Reorg detected of last block, block height " + tip.height() + " was " + lastBlock + " and now is " + blockhash);
                         }
 
-                        Frigate.getEventBus().post(new BlockReorgEvent(reorgStartHeight));
+                        Frigate.getEventBus().post(new BlockReorgSyncStart(reorgStartHeight));
                         blocksIndex.removeFromIndex(reorgStartHeight + 1);
                         updateBlocksIndex();
+                        Frigate.getEventBus().post(new BlockReorgSyncComplete(reorgStartHeight));
 
                         lastBlock = null;
+                        newBlock = true;
                     }
                 }
 
@@ -319,9 +720,16 @@ public class BitcoindClient {
                     log.debug("New block height " + tip.height());
                     Frigate.getEventBus().post(tip);
                     updateBlocksIndex();
+                    newBlock = true;
                 }
 
-                updateMempoolIndex();
+                boolean zmqHealthy = zmqContext != null && lastZmqMessageMs != 0L && System.currentTimeMillis() - lastZmqMessageMs < ZMQ_STALENESS_THRESHOLD_MS;
+                long mempoolDiffInterval = zmqHealthy ? ZMQ_MEMPOOL_DIFF_INTERVAL_MS : POLL_MEMPOOL_DIFF_INTERVAL_MS;
+                //force the diff when a block was just connected/reorged: a batch of txids left the mempool at once (mined txs get no per-tx 'R'), evict them now rather than up to ZMQ_MEMPOOL_DIFF_INTERVAL_MS later
+                if(newBlock || System.currentTimeMillis() - lastMempoolDiffMs >= mempoolDiffInterval) {
+                    updateMempoolIndex();
+                    lastMempoolDiffMs = System.currentTimeMillis();
+                }
 
                 lastBlock = blockchainInfo.bestblockhash();
             } catch(Exception e) {
@@ -438,15 +846,20 @@ public class BitcoindClient {
         }
 
         // P2TR: 34 bytes - OP_1 <32-byte taproot output>
-        if(length == 34 &&
-                scriptPubKey[0] == (byte) 0x51 &&  // OP_1
-                scriptPubKey[1] == (byte) 0x20) {  // Push 32 bytes
+        if(isP2tr(scriptPubKey)) {
             return ScriptType.P2TR;
         }
 
         return null;
     }
 
+    private static boolean isP2tr(byte[] scriptPubKey) {
+        return scriptPubKey != null
+                && scriptPubKey.length == 34
+                && scriptPubKey[0] == (byte) 0x51   // OP_1
+                && scriptPubKey[1] == (byte) 0x20;  // Push 32 bytes
+    }
+
     private static File getDefaultCoreDataDir() {
         OsType osType = OsType.getCurrent();
         if(osType == OsType.MACOS) {
@@ -471,4 +884,8 @@ public class BitcoindClient {
     public boolean containsSubmitPackage() {
         return networkInfo.version() >= MIN_SUBMIT_PACKAGE_VERSION;
     }
+
+    private record MempoolSeqEvent(Sha256Hash txid, boolean removed) {}
+
+    private record ZmqDiscovery(String endpoint, boolean unsupported) {}
 }

src/main/java/com/sparrowwallet/frigate/bitcoind/BitcoindClientService.java

@@ -8,6 +8,7 @@ import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcParam;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcService;
 import com.sparrowwallet.drongo.protocol.Sha256Hash;
 
+import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
@@ -20,6 +21,9 @@ public interface BitcoindClientService {
     @JsonRpcMethod("getnetworkinfo")
     NetworkInfo getNetworkInfo();
 
+    @JsonRpcMethod("getzmqnotifications")
+    List<ZmqNotification> getZmqNotifications();
+
     @JsonRpcMethod("estimatesmartfee")
     FeeInfo estimateSmartFee(@JsonRpcParam("conf_target") int blocks, @JsonRpcParam("estimate_mode") @JsonRpcOptional String mode);
 
@@ -50,6 +54,9 @@ public interface BitcoindClientService {
     @JsonRpcMethod("getblock")
     Object getBlock(@JsonRpcParam("blockhash") String blockhash, @JsonRpcOptional @JsonRpcParam("verbosity") int verbosity);
 
+    @JsonRpcMethod("getblock")
+    VerboseBlock getVerboseBlock(@JsonRpcParam("blockhash") String blockhash, @JsonRpcParam("verbosity") int verbosity);
+
     @JsonRpcMethod("getrawtransaction")
     Object getRawTransaction(@JsonRpcParam("txid") String txid, @JsonRpcParam("verbose") boolean verbose);
 

src/main/java/com/sparrowwallet/frigate/bitcoind/BitcoindTransport.java

@@ -2,24 +2,27 @@ package com.sparrowwallet.frigate.bitcoind;
 
 import com.github.arteam.simplejsonrpc.client.Transport;
 import com.sparrowwallet.drongo.Network;
+import com.sparrowwallet.frigate.io.Config;
 import com.sparrowwallet.frigate.io.Server;
+import com.sparrowwallet.frigate.io.SslUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.net.ssl.*;
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.SSLSocketFactory;
 import java.io.*;
 import java.net.*;
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
-import java.security.cert.CertificateException;
-import java.security.cert.X509Certificate;
 import java.util.Base64;
 
 public class BitcoindTransport implements Transport {
     private static final Logger log = LoggerFactory.getLogger(BitcoindTransport.class);
     public static final String COOKIE_FILENAME = ".cookie";
 
+    private static final int CONNECT_TIMEOUT_MS = 10_000;
+
     private final Server bitcoindServer;
     private URL bitcoindUrl;
     private File cookieFile;
@@ -54,12 +57,15 @@ public class BitcoindTransport implements Transport {
         HttpURLConnection connection = (HttpURLConnection)bitcoindUrl.openConnection();
 
         if(connection instanceof HttpsURLConnection httpsURLConnection) {
-            SSLSocketFactory sslSocketFactory = getTrustAllSocketFactory();
+            SSLSocketFactory sslSocketFactory = SslUtil.getTrustAllSocketFactory();
             if(sslSocketFactory != null) {
                 httpsURLConnection.setSSLSocketFactory(sslSocketFactory);
             }
         }
 
+        connection.setConnectTimeout(CONNECT_TIMEOUT_MS);
+        connection.setReadTimeout(Config.get().getCore().getRpcRequestTimeoutMillis());
+
         connection.setRequestMethod("POST");
         connection.setRequestProperty("Content-Type", "application/json");
 
@@ -135,29 +141,4 @@ public class BitcoindTransport implements Transport {
         return bitcoindDir;
     }
 
-    private SSLSocketFactory getTrustAllSocketFactory() {
-        TrustManager[] trustAllCerts = new TrustManager[] {
-                new X509TrustManager() {
-                    public X509Certificate[] getAcceptedIssuers() {
-                        return new X509Certificate[0];
-                    }
-
-                    public void checkClientTrusted(X509Certificate[] certs, String authType) throws CertificateException {
-                    }
-
-                    public void checkServerTrusted(X509Certificate[] certs, String authType) throws CertificateException {
-                    }
-                }
-        };
-
-        try {
-            SSLContext sslContext = SSLContext.getInstance("TLS");
-            sslContext.init(null, trustAllCerts, null);
-            return sslContext.getSocketFactory();
-        } catch (Exception e) {
-            log.error("Error creating SSL socket factory", e);
-        }
-
-        return null;
-    }
 }

src/main/java/com/sparrowwallet/frigate/bitcoind/BlockReorgEvent.java

@@ -1,3 +0,0 @@
-package com.sparrowwallet.frigate.bitcoind;
-
-public record BlockReorgEvent(int startHeight) {}

src/main/java/com/sparrowwallet/frigate/bitcoind/BlockReorgSyncComplete.java

@@ -0,0 +1,3 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+public record BlockReorgSyncComplete(int reorgStartHeight) {}

src/main/java/com/sparrowwallet/frigate/bitcoind/BlockReorgSyncStart.java

@@ -0,0 +1,3 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+public record BlockReorgSyncStart(int reorgStartHeight) {}

src/main/java/com/sparrowwallet/frigate/bitcoind/MempoolEntry.java

@@ -1,7 +1,7 @@
 package com.sparrowwallet.frigate.bitcoind;
 
 import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
-import com.sparrowwallet.frigate.index.TxEntry;
+import com.sparrowwallet.frigate.electrum.TxEntry;
 
 @JsonIgnoreProperties(ignoreUnknown = true)
 public record MempoolEntry(int vsize, int ancestorsize, boolean bip125_replaceable, FeesMempoolEntry fees) {

src/main/java/com/sparrowwallet/frigate/bitcoind/PagedBatchRequestBuilder.java

@@ -0,0 +1,150 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.arteam.simplejsonrpc.client.JsonRpcClient;
+import com.github.arteam.simplejsonrpc.client.Transport;
+import com.github.arteam.simplejsonrpc.client.builder.AbstractBuilder;
+import com.github.arteam.simplejsonrpc.client.builder.BatchRequestBuilder;
+import com.github.arteam.simplejsonrpc.client.exception.JsonRpcBatchException;
+import com.github.arteam.simplejsonrpc.core.domain.ErrorMessage;
+import com.google.common.collect.Lists;
+import com.sparrowwallet.frigate.io.Config;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+
+public class PagedBatchRequestBuilder<K, V> extends AbstractBuilder {
+    public static final int DEFAULT_MAX_ATTEMPTS = 3;
+    public static final int RETRY_DELAY_SECS = 1;
+
+    private final AtomicLong counter;
+
+    private final List<Request<K>> requests;
+
+    /**
+     * Type of request ids
+     */
+    private final Class<K> keysType;
+
+    /**
+     * Expected return type for all requests
+     */
+    private final Class<V> returnType;
+
+    public PagedBatchRequestBuilder(Transport transport, ObjectMapper mapper, AtomicLong counter) {
+        this(transport, mapper, new ArrayList<>(), null, null, counter);
+    }
+
+    public PagedBatchRequestBuilder(Transport transport, ObjectMapper mapper,
+                                    List<Request<K>> requests,
+                                    Class<K> keysType, Class<V> returnType,
+                                    AtomicLong counter) {
+        super(transport, mapper);
+        this.requests = requests;
+        this.keysType = keysType;
+        this.returnType = returnType;
+        this.counter = counter;
+    }
+
+    /**
+     * Adds a new request without specifying a return type
+     */
+    public PagedBatchRequestBuilder<K, V> add(K id, String method, Object... params) {
+        requests.add(new Request<>(id, counter == null ? null : counter.incrementAndGet(), method, params));
+        return this;
+    }
+
+    /**
+     * Sets type of request keys.
+     */
+    public <NK> PagedBatchRequestBuilder<NK, V> keysType(Class<NK> keysClass) {
+        return new PagedBatchRequestBuilder<>(transport, mapper, new ArrayList<>(), keysClass, returnType, counter);
+    }
+
+    /**
+     * Sets an expected response type of requests.
+     */
+    public <NV> PagedBatchRequestBuilder<K, NV> returnType(Class<NV> valuesClass) {
+        return new PagedBatchRequestBuilder<>(transport, mapper, requests, keysType, valuesClass, counter);
+    }
+
+    public Map<K, V> execute() throws Exception {
+        return execute(DEFAULT_MAX_ATTEMPTS);
+    }
+
+    /**
+     * Validates, executes the request and processes the response
+     */
+    public Map<K, V> execute(int maxAttempts) throws Exception {
+        Map<K, V> allResults = new HashMap<>();
+        JsonRpcClient client = new JsonRpcClient(transport);
+
+        List<List<Request<K>>> pages = Lists.partition(requests, getPageSize());
+        for(List<Request<K>> page : pages) {
+            if(counter != null) {
+                Map<Long, K> counterIdMap = new HashMap<>();
+                BatchRequestBuilder<Long, V> batchRequest = client.createBatchRequest().keysType(Long.class).returnType(returnType);
+                for(Request<K> request : page) {
+                    counterIdMap.put(request.counterId(), request.id());
+                    batchRequest.add(request.counterId(), request.method(), request.params());
+                }
+
+                try {
+                    Map<Long, V> pageResult = new RetryLogic<Map<Long, V>>(maxAttempts, RETRY_DELAY_SECS, List.of(IllegalStateException.class, IllegalArgumentException.class)).getResult(batchRequest::execute);
+                    for(Map.Entry<Long, V> pageEntry : pageResult.entrySet()) {
+                        allResults.put(counterIdMap.get(pageEntry.getKey()), pageEntry.getValue());
+                    }
+                } catch(JsonRpcBatchException e) {
+                    Map<Object, Object> mappedSuccesses = new HashMap<>();
+                    for(Map.Entry<?, ?> successEntry : e.getSuccesses().entrySet()) {
+                        mappedSuccesses.put(counterIdMap.get((Long)successEntry.getKey()), successEntry.getValue());
+                    }
+                    Map<Object, ErrorMessage> mappedErrors = new HashMap<>();
+                    for(Map.Entry<?, ErrorMessage> errorEntry : e.getErrors().entrySet()) {
+                        mappedErrors.put(counterIdMap.get((Long)errorEntry.getKey()), errorEntry.getValue());
+                    }
+                    throw new JsonRpcBatchException(e.getMessage(), mappedSuccesses, mappedErrors);
+                }
+            } else {
+                BatchRequestBuilder<K, V> batchRequest = client.createBatchRequest().keysType(keysType).returnType(returnType);
+                for(Request<K> request : page) {
+                    if(request.id() instanceof String strReq) {
+                        batchRequest.add(strReq, request.method(), request.params());
+                    } else if(request.id() instanceof Integer intReq) {
+                        batchRequest.add(intReq, request.method(), request.params());
+                    } else {
+                        throw new IllegalArgumentException("Id of class " + request.id().getClass().getName() + " not supported");
+                    }
+                }
+
+                Map<K, V> pageResult = new RetryLogic<Map<K, V>>(maxAttempts, RETRY_DELAY_SECS, List.of(IllegalStateException.class, IllegalArgumentException.class)).getResult(batchRequest::execute);
+                allResults.putAll(pageResult);
+            }
+        }
+
+        return allResults;
+    }
+
+    private int getPageSize() {
+        return Config.get().getCore().getRpcBatchSizeValue();
+    }
+
+    /**
+     * Creates a builder of a JSON-RPC batch request in initial state
+     */
+    public static PagedBatchRequestBuilder<?, ?> create(Transport transport) {
+        return new PagedBatchRequestBuilder<>(transport, new ObjectMapper(), null);
+    }
+
+    /**
+     * Creates a builder of a JSON-RPC batch request in initial state with a counter for request ids
+     */
+    public static PagedBatchRequestBuilder<?, ?> create(Transport transport, AtomicLong counter) {
+        return new PagedBatchRequestBuilder<>(transport, new ObjectMapper(), counter);
+    }
+
+    private record Request<K>(K id, Long counterId, String method, Object[] params) {}
+}

src/main/java/com/sparrowwallet/frigate/bitcoind/RetryLogic.java

@@ -0,0 +1,56 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+import java.util.List;
+import java.util.Random;
+
+/**
+ * Generic retry logic. Delegate must throw the specified exception type to trigger the retry logic.
+ */
+public class RetryLogic<T> {
+    public static interface Delegate<T> {
+        T call() throws Exception;
+    }
+
+    private final int maxAttempts;
+    private final int retryWaitSeconds;
+    @SuppressWarnings("rawtypes")
+    private final List<Class> retryExceptionTypes;
+
+    public RetryLogic(int maxAttempts, int retryWaitSeconds, @SuppressWarnings("rawtypes") Class retryExceptionType) {
+        this(maxAttempts, retryWaitSeconds, List.of(retryExceptionType));
+    }
+
+    public RetryLogic(int maxAttempts, int retryWaitSeconds, @SuppressWarnings("rawtypes") List<Class> retryExceptionTypes) {
+        this.maxAttempts = maxAttempts;
+        this.retryWaitSeconds = Math.max(retryWaitSeconds, 1);
+        this.retryExceptionTypes = retryExceptionTypes;
+    }
+
+    public T getResult(Delegate<T> caller) throws Exception {
+        T result = null;
+        int remainingAttempts = maxAttempts;
+        do {
+            try {
+                return caller.call();
+            } catch(Exception e) {
+                if(retryExceptionTypes.contains(e.getClass())) {
+                    if(--remainingAttempts == 0) {
+                        throw new ServerException("Retries exhausted", e);
+                    } else {
+                        try {
+                            //Sleep with a +/- 2 seconds random wait time to avoid simultaneous retries
+                            Thread.sleep((1000L * (retryWaitSeconds - 1)) + new Random().nextInt(2000));
+                        } catch(InterruptedException ie) {
+                            //ignore
+                        }
+                    }
+                } else {
+                    throw e;
+                }
+            }
+
+        } while(remainingAttempts > 0);
+
+        throw new IllegalStateException("Should be impossible");
+    }
+}

src/main/java/com/sparrowwallet/frigate/bitcoind/ServerException.java

@@ -0,0 +1,19 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+public class ServerException extends Exception {
+    public ServerException() {
+        super();
+    }
+
+    public ServerException(String message) {
+        super(message);
+    }
+
+    public ServerException(Throwable cause) {
+        super(cause);
+    }
+
+    public ServerException(String message, Throwable cause) {
+        super(message, cause);
+    }
+}

src/main/java/com/sparrowwallet/frigate/bitcoind/VerboseBlock.java

@@ -0,0 +1,28 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.util.List;
+
+@JsonIgnoreProperties(ignoreUnknown = true)
+public record VerboseBlock(String hash, int height, long time, List<VerboseTransaction> tx) {
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    public record VerboseTransaction(String txid, @JsonProperty("hex") String hex, List<VerboseVin> vin, List<VerboseVout> vout) {}
+
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    public record VerboseVin(String txid, @JsonProperty("vout") Integer voutIndex, VerbosePrevout prevout) {
+        public boolean isCoinbase() {
+            return txid == null;
+        }
+    }
+
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    public record VerbosePrevout(@JsonProperty("scriptPubKey") VerboseScriptPubKey scriptPubKey) {}
+
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    public record VerboseVout(int n, @JsonProperty("scriptPubKey") VerboseScriptPubKey scriptPubKey) {}
+
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    public record VerboseScriptPubKey(String hex) {}
+}

src/main/java/com/sparrowwallet/frigate/bitcoind/ZmqNotification.java

@@ -0,0 +1,6 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+
+@JsonIgnoreProperties(ignoreUnknown = true)
+public record ZmqNotification(String type, String address, long hwm) {}

src/main/java/com/sparrowwallet/frigate/cli/ElectrumClientService.java

@@ -6,6 +6,7 @@ import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcMethod;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcOptional;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcParam;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcService;
+import com.sparrowwallet.frigate.electrum.SilentPaymentsSubscription;
 
 import java.util.List;
 
@@ -16,5 +17,5 @@ public interface ElectrumClientService {
     List<String> getServerVersion(@JsonRpcParam("client_name") String clientName, @JsonRpcParam("protocol_version") Object protocolVersion);
 
     @JsonRpcMethod("blockchain.silentpayments.subscribe")
-    String subscribeSilentPayments(@JsonRpcParam("scan_private_key") String scanPrivateKey, @JsonRpcParam("spend_public_key") String spendPublicKey, @JsonRpcParam("start") @JsonRpcOptional Long start, @JsonRpcParam("labels") @JsonRpcOptional Integer[] labels);
+    SilentPaymentsSubscription subscribeSilentPayments(@JsonRpcParam("scan_private_key") String scanPrivateKey, @JsonRpcParam("spend_public_key") String spendPublicKey, @JsonRpcParam("start") @JsonRpcOptional Long start, @JsonRpcParam("labels") @JsonRpcOptional Integer[] labels);
 }

src/main/java/com/sparrowwallet/frigate/cli/FrigateCli.java

@@ -9,6 +9,8 @@ import com.sparrowwallet.drongo.Network;
 import com.sparrowwallet.frigate.Frigate;
 import com.sparrowwallet.frigate.electrum.ElectrumServerService;
 import com.sparrowwallet.frigate.electrum.ElectrumTransport;
+import com.sparrowwallet.frigate.electrum.SilentPaymentsSubscription;
+import com.sparrowwallet.frigate.io.Protocol;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -74,7 +76,7 @@ public class FrigateCli implements Thread.UncaughtExceptionHandler {
     }
 
     public void connect() {
-        transport = new ElectrumTransport(server, new SubscriptionService());
+        transport = new ElectrumTransport(server, Protocol.TCP, new SubscriptionService());
         transport.connect();
         reader = Thread.ofVirtual().name("ElectrumServerReadThread").unstarted(new ReadRunnable());
         reader.setUncaughtExceptionHandler(FrigateCli.this);
@@ -85,13 +87,13 @@ public class FrigateCli implements Thread.UncaughtExceptionHandler {
         JsonRpcClient jsonRpcClient = new JsonRpcClient(getTransport());
         ElectrumClientService electrumClientService = jsonRpcClient.onDemand(ElectrumClientService.class);
         electrumClientService.getServerVersion(APP_NAME, ElectrumServerService.MIN_VERSION.get());
-        String address = electrumClientService.subscribeSilentPayments(scanPrivateKey, spendPublicKey, start, labels);
+        SilentPaymentsSubscription subscription = electrumClientService.subscribeSilentPayments(scanPrivateKey, spendPublicKey, start, labels);
 
         try {
-            ScanProgress scanProgress = new ScanProgress(address, !follow, !quiet);
+            ScanProgress scanProgress = new ScanProgress(subscription.address(), !follow, !quiet);
             getEventBus().register(scanProgress);
             if(!quiet) {
-                System.out.println("Scanning address " + address + "...");
+                System.out.println("Scanning address " + subscription.address() + "...");
             }
             scanProgress.waitForCompletion();
             getEventBus().unregister(scanProgress);

src/main/java/com/sparrowwallet/frigate/cli/ScanProgress.java

@@ -4,7 +4,7 @@ import com.google.common.eventbus.Subscribe;
 import com.google.gson.Gson;
 import com.google.gson.GsonBuilder;
 import com.sparrowwallet.frigate.electrum.SilentPaymentsNotification;
-import com.sparrowwallet.frigate.index.TxEntry;
+import com.sparrowwallet.frigate.electrum.SilentPaymentsTxEntry;
 
 import java.util.ArrayList;
 import java.util.List;
@@ -16,7 +16,7 @@ public class ScanProgress {
     private final String address;
     private final boolean canComplete;
     private final boolean showProgress;
-    private final List<TxEntry> results = new ArrayList<>();
+    private final List<SilentPaymentsTxEntry> results = new ArrayList<>();
 
     private volatile boolean isComplete = false;
     private volatile boolean isInitialComplete = false;

src/main/java/com/sparrowwallet/frigate/cli/SubscriptionService.java

@@ -5,7 +5,7 @@ import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcParam;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcService;
 import com.sparrowwallet.frigate.electrum.SilentPaymentsNotification;
 import com.sparrowwallet.frigate.electrum.SilentPaymentsSubscription;
-import com.sparrowwallet.frigate.index.TxEntry;
+import com.sparrowwallet.frigate.electrum.SilentPaymentsTxEntry;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -16,7 +16,7 @@ public class SubscriptionService {
     private static final Logger log = LoggerFactory.getLogger(SubscriptionService.class);
 
     @JsonRpcMethod("blockchain.silentpayments.subscribe")
-    public void silentPaymentsUpdate(@JsonRpcParam("subscription") SilentPaymentsSubscription subscription, @JsonRpcParam("progress") double progress, @JsonRpcParam("history") List<TxEntry> history) {
+    public void silentPaymentsUpdate(@JsonRpcParam("subscription") SilentPaymentsSubscription subscription, @JsonRpcParam("progress") double progress, @JsonRpcParam("history") List<SilentPaymentsTxEntry> history) {
         FrigateCli.getEventBus().post(new SilentPaymentsNotification(subscription, progress, history, null));
     }
 }

src/main/java/com/sparrowwallet/frigate/control/TrayManager.java

@@ -0,0 +1,111 @@
+package com.sparrowwallet.frigate.control;
+
+import com.google.common.eventbus.Subscribe;
+import com.sparrowwallet.drongo.Network;
+import com.sparrowwallet.drongo.OsType;
+import com.sparrowwallet.frigate.Frigate;
+import com.sparrowwallet.frigate.electrum.ElectrumBlockHeader;
+import com.sparrowwallet.frigate.index.SilentPaymentsBlocksIndexUpdate;
+import com.sparrowwallet.frigate.io.Config;
+import com.sparrowwallet.frigate.io.Server;
+import com.sparrowwallet.frigate.io.Storage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.imageio.ImageIO;
+import java.awt.*;
+import java.awt.image.BaseMultiResolutionImage;
+import java.io.IOException;
+import java.text.NumberFormat;
+import java.util.ArrayList;
+import java.util.List;
+
+public class TrayManager {
+    private static final Logger log = LoggerFactory.getLogger(TrayManager.class);
+
+    private final TrayIcon trayIcon;
+    private final PopupMenu popupMenu = new PopupMenu();
+    private final MenuItem statusItem;
+
+    private int tipHeight = -1;
+    private int indexedHeight = -1;
+
+    public TrayManager() {
+        if(!isSupported()) {
+            throw new UnsupportedOperationException("System tray is not supported on this platform.");
+        }
+
+        SystemTray tray = SystemTray.getSystemTray();
+
+        try {
+            List<Image> imgList = new ArrayList<>();
+            imgList.add(ImageIO.read(getClass().getResource("/image/frigate-white-small.png")));
+            imgList.add(ImageIO.read(getClass().getResource("/image/frigate-white-small@2x.png")));
+            imgList.add(ImageIO.read(getClass().getResource("/image/frigate-white-small@3x.png")));
+
+            BaseMultiResolutionImage mrImage = new BaseMultiResolutionImage(imgList.toArray(new Image[0]));
+            this.trayIcon = new TrayIcon(mrImage, "Frigate", popupMenu);
+
+            MenuItem versionItem = new MenuItem(Frigate.SERVER_NAME + " " + Frigate.SERVER_VERSION + " (" + Network.get().getName() + ")");
+            versionItem.setEnabled(false);
+            popupMenu.add(versionItem);
+
+            statusItem = new MenuItem(Config.get().getCore().shouldConnect() ? "Starting..." : "Indexing Disabled");
+            statusItem.setEnabled(false);
+            popupMenu.add(statusItem);
+
+            MenuItem configItem = new MenuItem("Open Config Folder");
+            configItem.addActionListener(e -> {
+                try {
+                    Desktop.getDesktop().open(Storage.getFrigateDir());
+                } catch(IOException ex) {
+                    log.error("Could not open config folder", ex);
+                }
+            });
+            popupMenu.add(configItem);
+
+            MenuItem quitItem = new MenuItem("Quit Frigate");
+            quitItem.addActionListener(e -> System.exit(0));
+            popupMenu.add(quitItem);
+
+            tray.add(trayIcon);
+        } catch(IOException | AWTException e) {
+            log.error("Could not initialize system tray", e);
+            throw new IllegalStateException(e);
+        }
+    }
+
+    @Subscribe
+    public void electrumBlockHeader(ElectrumBlockHeader header) {
+        tipHeight = header.height();
+        indexedHeight = -1;
+        updateStatusItem();
+    }
+
+    @Subscribe
+    public void silentPaymentsBlocksIndexUpdate(SilentPaymentsBlocksIndexUpdate update) {
+        indexedHeight = update.toBlockHeight();
+        updateStatusItem();
+    }
+
+    private void updateStatusItem() {
+        if(tipHeight == -1) {
+            statusItem.setLabel("Starting...");
+        } else if(indexedHeight >= 0 && indexedHeight < tipHeight) {
+            NumberFormat nf = NumberFormat.getIntegerInstance();
+            statusItem.setLabel("Indexing: block " + nf.format(indexedHeight) + " / " + nf.format(tipHeight));
+        } else {
+            Config.ServerConfig sc = Config.get().getServer();
+            Server tcpServer = sc.getTcpServer();
+            Server sslServer = sc.getSslServer();
+            List<String> ports = new ArrayList<>(2);
+            if(tcpServer != null) ports.add(Integer.toString(tcpServer.getHostAndPort().getPort()));
+            if(sslServer != null) ports.add(Integer.toString(sslServer.getHostAndPort().getPort()));
+            statusItem.setLabel("Electrum server on port " + String.join("/", ports));
+        }
+    }
+
+    public static boolean isSupported() {
+        return OsType.getCurrent() == OsType.MACOS && Desktop.isDesktopSupported() && SystemTray.isSupported();
+    }
+}

src/main/java/com/sparrowwallet/frigate/electrum/BackendSubscriptionService.java

@@ -8,11 +8,6 @@ import com.sparrowwallet.frigate.Frigate;
 
 @JsonRpcService
 public class BackendSubscriptionService {
-    @JsonRpcMethod("blockchain.headers.subscribe")
-    public void newBlockHeaderTip(@JsonRpcParam("header") final ElectrumBlockHeader header) {
-        //Nothing required
-    }
-
     @JsonRpcMethod("blockchain.scripthash.subscribe")
     public void scriptHashStatusUpdated(@JsonRpcParam("scripthash") final String scriptHash, @JsonRpcOptional @JsonRpcParam("status") final String status) {
         Frigate.getEventBus().post(new ScriptHashStatus(scriptHash, status));

src/main/java/com/sparrowwallet/frigate/electrum/ElectrumBackendService.java

@@ -6,7 +6,6 @@ import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcMethod;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcOptional;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcParam;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcService;
-import com.sparrowwallet.frigate.index.TxEntry;
 
 import java.util.Collection;
 import java.util.List;

src/main/java/com/sparrowwallet/frigate/electrum/ElectrumNotificationService.java

@@ -4,7 +4,6 @@ import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcMethod;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcOptional;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcParam;
 import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcService;
-import com.sparrowwallet.frigate.index.TxEntry;
 
 import java.util.List;
 
@@ -17,5 +16,5 @@ public interface ElectrumNotificationService {
     void notifyScriptHash(@JsonRpcParam("scripthash") String scriptHash, @JsonRpcOptional @JsonRpcParam("status") String status);
 
     @JsonRpcMethod("blockchain.silentpayments.subscribe")
-    void notifySilentPayments(@JsonRpcParam("subscription") SilentPaymentsSubscription silentPaymentsSubscription, @JsonRpcParam("progress") double progress, @JsonRpcParam("history") List<TxEntry> history);
+    void notifySilentPayments(@JsonRpcParam("subscription") SilentPaymentsSubscription silentPaymentsSubscription, @JsonRpcParam("progress") double progress, @JsonRpcParam("history") List<SilentPaymentsTxEntry> history);
 }

src/main/java/com/sparrowwallet/frigate/electrum/ElectrumNotificationTransport.java

@@ -3,22 +3,17 @@ package com.sparrowwallet.frigate.electrum;
 import com.github.arteam.simplejsonrpc.client.Transport;
 
 import java.io.IOException;
-import java.net.Socket;
-import java.nio.charset.StandardCharsets;
 
 public class ElectrumNotificationTransport implements Transport {
-    private final Socket clientSocket;
+    private final RequestHandler requestHandler;
 
-    public ElectrumNotificationTransport(Socket clientSocket) {
-        this.clientSocket = clientSocket;
+    public ElectrumNotificationTransport(RequestHandler requestHandler) {
+        this.requestHandler = requestHandler;
     }
 
     @Override
     public String pass(String request) throws IOException {
-        byte[] bytes = (request + "\n").getBytes(StandardCharsets.UTF_8);
-        clientSocket.getOutputStream().write(bytes);
-        clientSocket.getOutputStream().flush();
-
+        requestHandler.writeLine(request);
         return "{\"result\":{},\"error\":null,\"id\":1}";
     }
 }

src/main/java/com/sparrowwallet/frigate/electrum/ElectrumServerRunnable.java

@@ -1,80 +1,162 @@
 package com.sparrowwallet.frigate.electrum;
 
+import com.sparrowwallet.frigate.ConfigurationException;
 import com.sparrowwallet.frigate.bitcoind.BitcoindClient;
 import com.sparrowwallet.frigate.index.IndexQuerier;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLServerSocket;
 import java.io.IOException;
+import java.net.InetSocketAddress;
 import java.net.ServerSocket;
 import java.net.Socket;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 
 public class ElectrumServerRunnable implements Runnable {
     private static final Logger log = LoggerFactory.getLogger(ElectrumServerRunnable.class);
-    public static final int DEFAULT_PORT = 57001;
+    private static final int LISTEN_BACKLOG = 50;
+
+    private static final Set<String> ALLOWED_TLS_PROTOCOLS = Set.of("TLSv1.2", "TLSv1.3");
 
     private final BitcoindClient bitcoindClient;
     private final IndexQuerier indexQuerier;
+    private final InetSocketAddress tcpBind;
+    private final InetSocketAddress sslBind;
+    private final List<ServerSocket> serverSockets = new ArrayList<>();
 
-    protected ServerSocket serverSocket = null;
-    protected boolean stopped = false;
-    protected Thread runningThread = null;
+    protected volatile boolean stopped = false;
     protected ExecutorService requestPool = Executors.newThreadPerTaskExecutor(Thread.ofVirtual().name("ElectrumServerRequest-", 0).factory());
 
-    public ElectrumServerRunnable(BitcoindClient bitcoindClient, IndexQuerier indexQuerier) {
+    public ElectrumServerRunnable(BitcoindClient bitcoindClient, IndexQuerier indexQuerier, InetSocketAddress tcpBind, InetSocketAddress sslBind, SSLContext sslContext) {
         this.bitcoindClient = bitcoindClient;
         this.indexQuerier = indexQuerier;
-        openServerSocket();
+        this.tcpBind = tcpBind;
+        this.sslBind = sslBind;
+
+        if(tcpBind == null && sslBind == null) {
+            throw new ConfigurationException("At least one of tcp or ssl must be enabled under [server] in config.toml");
+        }
+
+        if(sslBind != null && sslContext == null) {
+            throw new ConfigurationException("SSL: ssl listener configured but no SSLContext was supplied");
+        }
+
+        openServerSockets(sslContext);
     }
 
-    public int getPort() {
-        return serverSocket.getLocalPort();
+    public InetSocketAddress getTcpBind() {
+        return tcpBind;
+    }
+
+    public InetSocketAddress getSslBind() {
+        return sslBind;
     }
 
     public void run() {
-        synchronized(this) {
-            this.runningThread = Thread.currentThread();
+        StringBuilder banner = new StringBuilder("Electrum server listening on");
+        if(tcpBind != null) banner.append(" tcp://").append(formatBind(tcpBind));
+        if(sslBind != null) banner.append(" ssl://").append(formatBind(sslBind));
+        log.info(banner.toString());
+
+        CountDownLatch done = new CountDownLatch(serverSockets.size());
+        for(ServerSocket ss : serverSockets) {
+            Thread.ofVirtual().name("ElectrumAccept-" + ss.getLocalPort()).start(() -> {
+                try {
+                    acceptLoop(ss);
+                } finally {
+                    done.countDown();
+                }
+            });
         }
 
-        log.info("Electrum server listening on port {}", getPort());
-
-        while(!isStopped()) {
-            Socket clientSocket;
-            try {
-                clientSocket = this.serverSocket.accept();
-            } catch(IOException e) {
-                if(isStopped()) {
-                    break;
-                }
-                throw new RuntimeException("Error accepting client connection", e);
-            }
-            RequestHandler requestHandler = new RequestHandler(clientSocket, bitcoindClient, indexQuerier);
-            this.requestPool.execute(requestHandler);
+        try {
+            done.await();
+        } catch(InterruptedException e) {
+            Thread.currentThread().interrupt();
         }
 
         this.requestPool.shutdown();
     }
 
-    private synchronized boolean isStopped() {
-        return stopped;
+    private void acceptLoop(ServerSocket serverSocket) {
+        while(!stopped) {
+            Socket clientSocket;
+            try {
+                clientSocket = serverSocket.accept();
+            } catch(IOException e) {
+                if(stopped) {
+                    return;
+                }
+                log.error("Error accepting client connection on port " + serverSocket.getLocalPort(), e);
+                return;
+            }
+            RequestHandler requestHandler = new RequestHandler(clientSocket, bitcoindClient, indexQuerier);
+            this.requestPool.execute(requestHandler);
+        }
     }
 
     public synchronized void stop() {
         stopped = true;
-        try {
-            serverSocket.close();
-        } catch(IOException e) {
-            throw new RuntimeException("Error closing server", e);
+        for(ServerSocket ss : serverSockets) {
+            try {
+                ss.close();
+            } catch(IOException e) {
+                log.error("Error closing server socket on port " + ss.getLocalPort(), e);
+            }
         }
     }
 
-    private void openServerSocket() {
+    private void openServerSockets(SSLContext sslContext) {
         try {
-            serverSocket = new ServerSocket(DEFAULT_PORT);
+            if(tcpBind != null) {
+                ServerSocket plain = new ServerSocket(tcpBind.getPort(), LISTEN_BACKLOG, tcpBind.getAddress());
+                serverSockets.add(plain);
+            }
+            if(sslBind != null) {
+                SSLServerSocket sslSocket = (SSLServerSocket)sslContext.getServerSocketFactory().createServerSocket(sslBind.getPort(), LISTEN_BACKLOG, sslBind.getAddress());
+                sslSocket.setNeedClientAuth(false);
+                sslSocket.setEnabledProtocols(restrictedProtocols(sslSocket.getSupportedProtocols()));
+                serverSockets.add(sslSocket);
+            }
         } catch(IOException e) {
+            for(ServerSocket opened : serverSockets) {
+                try {
+                    opened.close();
+                } catch(IOException ignored) {
+                    //ignore
+                }
+            }
+            serverSockets.clear();
             throw new RuntimeException("Cannot open electrum server port", e);
         }
     }
+
+    private static String formatBind(InetSocketAddress addr) {
+        return addr.getAddress().getHostAddress() + ":" + addr.getPort();
+    }
+
+    private static String[] restrictedProtocols(String[] supported) {
+        List<String> enabled = new ArrayList<>(2);
+        Set<String> supportedSet = new HashSet<>(Arrays.asList(supported));
+        for(String p : ALLOWED_TLS_PROTOCOLS) {
+            if(supportedSet.contains(p)) {
+                enabled.add(p);
+            }
+        }
+
+        if(enabled.isEmpty()) {
+            throw new ConfigurationException("SSL: JVM supports neither TLSv1.2 nor TLSv1.3 (supported: " + Arrays.toString(supported) + ")");
+        }
+
+        return enabled.toArray(new String[0]);
+    }
 }

src/main/java/com/sparrowwallet/frigate/electrum/ElectrumServerService.java

@@ -14,13 +14,14 @@ import com.sparrowwallet.drongo.silentpayments.SilentPaymentScanAddress;
 import com.sparrowwallet.frigate.Frigate;
 import com.sparrowwallet.frigate.bitcoind.*;
 import com.sparrowwallet.frigate.index.IndexQuerier;
-import com.sparrowwallet.frigate.index.TxEntry;
+import com.sparrowwallet.frigate.io.Config;
+import com.sparrowwallet.frigate.io.Protocol;
+import com.sparrowwallet.frigate.io.Server;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.lang.ref.WeakReference;
 import java.util.*;
-import java.util.stream.Collectors;
 
 @JsonRpcService
 public class ElectrumServerService {
@@ -28,12 +29,15 @@ public class ElectrumServerService {
     public static final Version MIN_VERSION = new Version("1.4");
     public static final Version MAX_DEFAULT_VERSION = new Version("1.4.2");
     public static final Version MAX_SUBMIT_PACKAGE_VERSION = new Version("1.6");
+    public static final List<Integer> SILENT_PAYMENTS_SUPPORTED_VERSIONS = List.of(0);
+    private static final int METHOD_NOT_FOUND = -32601;
 
     private final BitcoindClient bitcoindClient;
     private final RequestHandler requestHandler;
     private final IndexQuerier indexQuerier;
     private final ElectrumBackendService electrumBackendService;
     private Version protocolVersion;
+    private String genesisHash;
 
     public ElectrumServerService(BitcoindClient bitcoindClient, RequestHandler requestHandler, IndexQuerier indexQuerier, ElectrumTransport backendTransport) {
         this.bitcoindClient = bitcoindClient;
@@ -103,11 +107,50 @@ public class ElectrumServerService {
     @JsonRpcMethod("server.features")
     public ServerFeatures getServerFeatures() {
         checkVersionNegotiated();
+        Map<String, ServerFeatures.HostInfo> ourHosts = buildAdvertisedHosts(Config.get().getServer().getAdvertisedHosts());
+
         if(electrumBackendService != null) {
-            return electrumBackendService.getServerFeatures();
+            try {
+                return electrumBackendService.getServerFeatures().withHosts(ourHosts).withSilentPayments(SILENT_PAYMENTS_SUPPORTED_VERSIONS);
+            } catch(JsonRpcException e) {
+                if(e.getErrorMessage() == null || e.getErrorMessage().getCode() != METHOD_NOT_FOUND) {
+                    throw e;
+                }
+                log.debug("Backend does not support server.features, returning local response");
+            }
         }
 
-        throw new UnsupportedOperationException("Configure backendElectrumServer to use server.features");
+        return new ServerFeatures(ourHosts, getGenesisHash(), "sha256", Frigate.SERVER_NAME + " " + Frigate.SERVER_VERSION,
+                protocolVersion.get(), MIN_VERSION.get(), null, SILENT_PAYMENTS_SUPPORTED_VERSIONS);
+    }
+
+    static Map<String, ServerFeatures.HostInfo> buildAdvertisedHosts(List<Server> servers) {
+        Map<String, ServerFeatures.HostInfo> result = new LinkedHashMap<>();
+        for(Server server : servers) {
+            int port = server.getHostAndPort().getPort();
+            ServerFeatures.HostInfo current = result.get(server.getHost());
+            Integer tcp = current == null ? null : current.tcp_port();
+            Integer ssl = current == null ? null : current.ssl_port();
+            if(server.getProtocol() == Protocol.TCP) {
+                tcp = port;
+            } else if(server.getProtocol() == Protocol.SSL) {
+                ssl = port;
+            }
+            result.put(server.getHost(), new ServerFeatures.HostInfo(tcp, ssl));
+        }
+        return result;
+    }
+
+    private String getGenesisHash() {
+        if(genesisHash == null && bitcoindClient != null) {
+            try {
+                genesisHash = bitcoindClient.getBitcoindService().getBlockHash(0);
+            } catch(Exception e) {
+                log.debug("Could not fetch genesis hash from bitcoind", e);
+            }
+        }
+
+        return genesisHash;
     }
 
     @JsonRpcMethod("server.add_peer")
@@ -201,8 +244,18 @@ public class ElectrumServerService {
     @JsonRpcMethod("blockchain.headers.subscribe")
     public ElectrumBlockHeader subscribeHeaders() {
         checkVersionNegotiated();
-        requestHandler.setHeadersSubscribed(true);
-        return bitcoindClient != null ? bitcoindClient.getTip() : new ElectrumBlockHeader(0, "");
+        if(bitcoindClient == null) {
+            throw new UnsupportedOperationException("Configure coreServer to use blockchain.headers.subscribe");
+        }
+        ElectrumBlockHeader tip = bitcoindClient.getTip();
+        requestHandler.runAfterResponse(() -> {
+            requestHandler.setHeadersSubscribed(true);
+            ElectrumBlockHeader currentTip = bitcoindClient.getTip();
+            if(currentTip != null && !currentTip.equals(tip)) {
+                requestHandler.notifyHeaders(currentTip);
+            }
+        });
+        return tip;
     }
 
     @JsonRpcMethod("server.ping")
@@ -222,9 +275,13 @@ public class ElectrumServerService {
     public String subscribeScriptHash(@JsonRpcParam("scripthash") String scriptHash) {
         checkVersionNegotiated();
         if(electrumBackendService != null) {
-            String status = electrumBackendService.subscribeScriptHash(scriptHash);
             requestHandler.subscribeScriptHash(scriptHash);
-            return status;
+            try {
+                return electrumBackendService.subscribeScriptHash(scriptHash);
+            } catch(RuntimeException e) {
+                requestHandler.unsubscribeScriptHash(scriptHash);
+                throw e;
+            }
         }
 
         throw new UnsupportedOperationException("Configure backendElectrumServer to use blockchain.scripthash.subscribe");
@@ -234,9 +291,8 @@ public class ElectrumServerService {
     public String unsubscribeScriptHash(@JsonRpcParam("scripthash") String scriptHash) {
         checkVersionNegotiated();
         if(electrumBackendService != null) {
-            String status = electrumBackendService.unsubscribeScriptHash(scriptHash);
             requestHandler.unsubscribeScriptHash(scriptHash);
-            return status;
+            return electrumBackendService.unsubscribeScriptHash(scriptHash);
         }
 
         throw new UnsupportedOperationException("Configure backendElectrumServer to use blockchain.scripthash.unsubscribe");
@@ -444,61 +500,138 @@ public class ElectrumServerService {
     }
 
     @JsonRpcMethod("blockchain.silentpayments.subscribe")
-    public String subscribeSilentPayments(@JsonRpcParam("scan_private_key") String scanPrivateKey, @JsonRpcParam("spend_public_key") String spendPublicKey, @JsonRpcParam("start") @JsonRpcOptional Object start, @JsonRpcParam("labels") @JsonRpcOptional Integer[] labels) {
+    public SilentPaymentsSubscription subscribeSilentPayments(@JsonRpcParam("scan_private_key") String scanPrivateKey, @JsonRpcParam("spend_public_key") String spendPublicKey, @JsonRpcParam("start") @JsonRpcOptional Object start, @JsonRpcParam("labels") @JsonRpcOptional Integer[] labels) throws InvalidParamsException {
         checkVersionNegotiated();
-        SilentPaymentScanAddress silentPaymentScanAddress = getSilentPaymentScanAddress(scanPrivateKey, spendPublicKey);
-        Set<Integer> labelSet = getLabels(labels);
-        requestHandler.subscribeSilentPaymentsAddress(silentPaymentScanAddress, labelSet);
+        SilentPaymentScanAddress silentPaymentScanAddress = parseScanAddress(scanPrivateKey, spendPublicKey);
+        Set<Integer> labelSet = parseLabels(labels);
+
+        int maxSubscriptions = Config.get().getScan().getMaxSubscriptions();
+        if(!requestHandler.isSilentPaymentsAddressSubscribed(silentPaymentScanAddress.toString()) && requestHandler.getSilentPaymentsSubscriptionCount() >= maxSubscriptions) {
+            throw new InvalidParamsException("subscription limit reached (" + maxSubscriptions + ") for this connection");
+        }
 
         int[] heightRange = getHeightRange(start);
+        int requestedStart = heightRange[0];
         Integer endHeight = heightRange.length > 1 ? heightRange[1] : null;
-        indexQuerier.startHistoryScan(silentPaymentScanAddress, heightRange[0], endHeight, labelSet, new WeakReference<>(requestHandler));
 
-        return silentPaymentScanAddress.getAddress();
+        int effectiveStart = requestedStart;
+        SilentPaymentAddressSubscription existing = requestHandler.getSilentPaymentsAddressSubscription(silentPaymentScanAddress.toString());
+        if(existing != null && existing.getStartHeight() < requestedStart) {
+            effectiveStart = existing.getStartHeight();
+        }
+
+        int startHeight = effectiveStart;
+        requestHandler.subscribeSilentPaymentsAddress(silentPaymentScanAddress, labelSet, startHeight);
+
+        requestHandler.runAfterResponse(() -> {
+            SilentPaymentAddressSubscription subscription = requestHandler.getSilentPaymentsAddressSubscription(silentPaymentScanAddress.toString());
+            if(subscription == null) {
+                return;
+            }
+            subscription.setActive(true);
+            indexQuerier.startHistoryScan(silentPaymentScanAddress, startHeight, endHeight, subscription, new WeakReference<>(requestHandler), true);
+        });
+
+        return new SilentPaymentsSubscription(silentPaymentScanAddress.getAddress(), labelSet.toArray(new Integer[0]), startHeight);
     }
 
     @JsonRpcMethod("blockchain.silentpayments.unsubscribe")
-    public String unsubscribeSilentPayments(@JsonRpcParam("scan_private_key") String scanPrivateKey, @JsonRpcParam("spend_public_key") String spendPublicKey) {
+    public String unsubscribeSilentPayments(@JsonRpcParam("scan_private_key") String scanPrivateKey, @JsonRpcParam("spend_public_key") String spendPublicKey) throws InvalidParamsException {
         checkVersionNegotiated();
-        SilentPaymentScanAddress silentPaymentScanAddress = getSilentPaymentScanAddress(scanPrivateKey, spendPublicKey);
+        SilentPaymentScanAddress silentPaymentScanAddress = parseScanAddress(scanPrivateKey, spendPublicKey);
         requestHandler.unsubscribeSilentPaymentsAddress(silentPaymentScanAddress);
 
         return silentPaymentScanAddress.getAddress();
     }
 
-    private static SilentPaymentScanAddress getSilentPaymentScanAddress(String scanPrivateKey, String spendPublicKey) {
-        ECKey scanKey = ECKey.fromPrivate(Utils.hexToBytes(scanPrivateKey));
-        ECKey spendKey = ECKey.fromPublicOnly(Utils.hexToBytes(spendPublicKey));
-        return SilentPaymentScanAddress.from(scanKey, spendKey);
+    private static SilentPaymentScanAddress parseScanAddress(String scanPrivateKey, String spendPublicKey) throws InvalidParamsException {
+        byte[] scanBytes;
+        byte[] spendBytes;
+        try {
+            scanBytes = Utils.hexToBytes(scanPrivateKey);
+            spendBytes = Utils.hexToBytes(spendPublicKey);
+        } catch(IllegalArgumentException e) {
+            throw new InvalidParamsException("scan_private_key or spend_public_key is not valid hex", e);
+        }
+        if(scanBytes.length != 32) {
+            throw new InvalidParamsException("scan_private_key must be 32 bytes");
+        }
+        if(spendBytes.length != 33) {
+            throw new InvalidParamsException("spend_public_key must be 33 bytes (compressed)");
+        }
+        try {
+            ECKey scanKey = ECKey.fromPrivate(scanBytes);
+            ECKey spendKey = ECKey.fromPublicOnly(spendBytes);
+            return SilentPaymentScanAddress.from(scanKey, spendKey);
+        } catch(IllegalArgumentException e) {
+            throw new InvalidParamsException("invalid scan/spend key: " + e.getMessage(), e);
+        }
     }
 
-    private int[] getHeightRange(Object start) {
-        if(start instanceof String s && s.contains("-")) {
-            String[] parts = s.split("-", 2);
-            return new int[] { Integer.parseInt(parts[0]), Integer.parseInt(parts[1]) };
+    private int[] getHeightRange(Object start) throws InvalidParamsException {
+        if(start == null) {
+            return new int[] { 0 };
         }
 
-        Long startLong = start instanceof Number n ? n.longValue() : null;
-        int startHeight = 0;
-        if(startLong != null) {
+        if(start instanceof String s) {
+            if(!s.contains("-")) {
+                throw new InvalidParamsException("start string must be of the form 'FROM-TO'");
+            }
+            String[] parts = s.split("-", 2);
+            int from;
+            int to;
+            try {
+                from = Integer.parseInt(parts[0]);
+                to = Integer.parseInt(parts[1]);
+            } catch(NumberFormatException e) {
+                throw new InvalidParamsException("start range must contain integer block heights", e);
+            }
+            if(from < 0 || to < from) {
+                throw new InvalidParamsException("start range must satisfy 0 <= from <= to");
+            }
+            int tip = bitcoindClient != null && bitcoindClient.getTip() != null ? bitcoindClient.getTip().height() : Integer.MAX_VALUE;
+            if(to > tip) {
+                throw new InvalidParamsException("start range 'to' (" + to + ") exceeds tip (" + tip + ")");
+            }
+            return new int[] { from, to };
+        }
+
+        if(start instanceof Number n) {
+            long startLong = n.longValue();
+            if(startLong < 0) {
+                throw new InvalidParamsException("start must be non-negative");
+            }
             if(startLong > Transaction.MAX_BLOCK_LOCKTIME) {
                 if(bitcoindClient == null) {
-                    throw new UnsupportedOperationException("Use a start block height instead of a timestamp when coreServer is not configured");
+                    throw new InvalidParamsException("timestamp start requires coreServer to be configured");
                 }
-                startHeight = bitcoindClient.findBlockByTimestamp(startLong);
-            } else if(startLong > 0) {
-                startHeight = startLong.intValue();
+                return new int[] { bitcoindClient.findBlockByTimestamp(startLong) };
             }
+            return new int[] { (int)startLong };
         }
-        return new int[] { startHeight };
+
+        throw new InvalidParamsException("start must be an integer or 'FROM-TO' string");
     }
 
-    private Set<Integer> getLabels(Integer[] labels) {
-        Set<Integer> labelSet = new HashSet<>();
+    private Set<Integer> parseLabels(Integer[] labels) throws InvalidParamsException {
+        int maxLabels = Config.get().getScan().getMaxLabels();
+        if(labels != null && labels.length > maxLabels) {
+            throw new InvalidParamsException("labels array exceeds " + maxLabels + " entries");
+        }
+
+        SortedSet<Integer> labelSet = new TreeSet<>();
         labelSet.add(0);
         if(labels != null) {
-            labelSet.addAll(Arrays.stream(labels).filter(Objects::nonNull).filter(integer -> integer.compareTo(0) > 0).collect(Collectors.toSet()));
+            for(Integer label : labels) {
+                if(label == null) {
+                    continue;
+                }
+                if(label <= 0) {
+                    throw new InvalidParamsException("label must satisfy 0 < label < 2^31, got " + label);
+                }
+                labelSet.add(label);
+            }
         }
-        return Collections.unmodifiableSet(labelSet);
+        return Collections.unmodifiableSortedSet(labelSet);
     }
 }

src/main/java/com/sparrowwallet/frigate/electrum/ElectrumTransport.java

@@ -1,28 +1,36 @@
 package com.sparrowwallet.frigate.electrum;
 
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
 import com.github.arteam.simplejsonrpc.client.Transport;
 import com.github.arteam.simplejsonrpc.server.JsonRpcServer;
 import com.google.common.net.HostAndPort;
-import com.google.gson.Gson;
+import com.sparrowwallet.frigate.io.Protocol;
+import com.sparrowwallet.frigate.io.SslUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import javax.net.SocketFactory;
+import javax.net.ssl.SSLSocketFactory;
 import java.io.*;
 import java.net.*;
 import java.nio.charset.StandardCharsets;
-import java.util.Objects;
+import java.util.Collections;
+import java.util.LinkedHashSet;
+import java.util.Set;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.Condition;
 import java.util.concurrent.locks.ReentrantLock;
-
-import static com.sparrowwallet.frigate.electrum.ElectrumServerRunnable.DEFAULT_PORT;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
 
 public class ElectrumTransport implements Transport, Closeable {
     private static final Logger log = LoggerFactory.getLogger(ElectrumTransport.class);
 
     private final HostAndPort electrumServer;
+    private final Protocol protocol;
     private Socket socket;
     private String response;
 
@@ -34,12 +42,13 @@ public class ElectrumTransport implements Transport, Closeable {
     private final Condition readingCondition = readLock.newCondition();
 
     private final ReentrantLock clientRequestLock = new ReentrantLock();
-    private boolean running = false;
+    private volatile boolean running = false;
     private volatile boolean reading = true;
-    private boolean closed = false;
+    private volatile boolean closed = false;
     private Exception lastException;
 
-    private static final Gson GSON = new Gson();
+    private static final Pattern ID_PATTERN = Pattern.compile("\"id\"\\s*:\\s*(\\d+)");
+    private static final JsonFactory JSON_FACTORY = new JsonFactory();
 
     private final JsonRpcServer jsonRpcServer = new JsonRpcServer();
     private final Object subscriptionService;
@@ -47,18 +56,31 @@ public class ElectrumTransport implements Transport, Closeable {
     private PrintWriter out;
     private BufferedReader in;
 
-    public ElectrumTransport(HostAndPort electrumServer, Object subscriptionService) {
+    public ElectrumTransport(HostAndPort electrumServer, Protocol protocol, Object subscriptionService) {
         this.electrumServer = electrumServer;
+        this.protocol = protocol;
         this.subscriptionService = subscriptionService;
     }
 
     public void connect() {
         try {
             String host = electrumServer.getHost();
-            int port = electrumServer.hasPort() ? electrumServer.getPort() : DEFAULT_PORT;
+            int port = electrumServer.hasPort() ? electrumServer.getPort() : protocol.getDefaultPort();
 
-            SocketFactory socketFactory = SocketFactory.getDefault();
-            this.socket = socketFactory.createSocket(host, port);
+            SocketFactory socketFactory;
+            if(protocol == Protocol.SSL) {
+                SSLSocketFactory sslSocketFactory = SslUtil.getTrustAllSocketFactory();
+                if(sslSocketFactory == null) {
+                    log.error("Could not create SSL socket factory for Electrum server: " + host);
+                    return;
+                }
+                socketFactory = sslSocketFactory;
+            } else {
+                socketFactory = SocketFactory.getDefault();
+            }
+
+            this.socket = socketFactory.createSocket();
+            this.socket.connect(new InetSocketAddress(host, port));
             this.socket.setSoTimeout(30000); // 30 second timeout for reads
             this.out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), StandardCharsets.UTF_8)));
             this.in = new BufferedReader(new InputStreamReader(socket.getInputStream(), StandardCharsets.UTF_8));
@@ -72,17 +94,20 @@ public class ElectrumTransport implements Transport, Closeable {
 
     @Override
     public String pass(String request) throws IOException {
+        Set<String> sentIdSet = extractIdSet(request);
         clientRequestLock.lock();
         try {
-            Rpc sentRpc = request.startsWith("{") ? GSON.fromJson(request, Rpc.class) : null;
-            Rpc recvRpc;
-            String recv;
-
             writeRequest(request);
+
+            String recv;
+            Set<String> recvIdSet;
             do {
                 recv = readResponse();
-                recvRpc = recv.startsWith("{") ? GSON.fromJson(response, Rpc.class) : null;
-            } while(!Objects.equals(recvRpc, sentRpc));
+                recvIdSet = extractIdSet(recv);
+                if(!sentIdSet.equals(recvIdSet)) {
+                    log.info("Discarding stale response with ids " + recvIdSet + " (expected " + sentIdSet + ")");
+                }
+            } while(!sentIdSet.equals(recvIdSet));
 
             return recv;
         } finally {
@@ -113,21 +138,14 @@ public class ElectrumTransport implements Transport, Closeable {
             }
         }
 
-        try {
-            if(!readLock.tryLock(1, TimeUnit.SECONDS)) {
-                throw new IOException("No response from server");
-            }
-        } catch(InterruptedException e) {
-            throw new IOException("Read thread interrupted");
-        }
-
+        readLock.lock();
         try {
             if(firstRead) {
                 readingCondition.signal();
                 firstRead = false;
             }
 
-            while(reading) {
+            while(reading && running) {
                 try {
                     readingCondition.await();
                 } catch(InterruptedException e) {
@@ -141,6 +159,10 @@ public class ElectrumTransport implements Transport, Closeable {
                 throw new IOException("Error reading response: " + lastException.getMessage(), lastException);
             }
 
+            if(!running) {
+                throw new IOException("Transport closed");
+            }
+
             reading = true;
 
             readingCondition.signal();
@@ -151,60 +173,70 @@ public class ElectrumTransport implements Transport, Closeable {
     }
 
     public void readInputLoop() throws Exception {
+        //Wait for first RPC request before starting to read. The lock must be acquired before
+        //signaling readiness so readResponse() blocks until we reach the atomic await/unlock.
         readLock.lock();
-        readReadySignal.countDown();
-
         try {
-            try {
-                //Don't start reading until first RPC request is sent
+            readReadySignal.countDown();
+            if(running) {
                 readingCondition.await();
-            } catch(InterruptedException e) {
-                Thread.currentThread().interrupt();
             }
+        } catch(InterruptedException e) {
+            Thread.currentThread().interrupt();
+            return;
+        } finally {
+            readLock.unlock();
+        }
 
-            while(running) {
-                try {
-                    String received = readInputStream(in);
-                    if(received.contains("method") && !received.contains("error")) {
-                        //Handle subscription notification
-                        jsonRpcServer.handle(received, subscriptionService);
-                    } else {
-                        //Handle client's response
-                        response = received;
-                        reading = false;
-                        readingCondition.signal();
-                        readingCondition.await();
-                    }
-                } catch(InterruptedException e) {
-                    //Restore interrupt status and continue
-                    Thread.currentThread().interrupt();
-                } catch(Exception e) {
+        while(running) {
+            try {
+                String received = readInputStream(in);
+                if(isNotification(received)) {
+                    jsonRpcServer.handle(received, subscriptionService);
+                } else {
+                    deliverResponse(received);
+                }
+            } catch(InterruptedException e) {
+                //Restore interrupt status and continue
+                Thread.currentThread().interrupt();
+            } catch(Exception e) {
+                if(!closed) {
                     log.trace("Connection error while reading", e);
-                    if(running) {
-                        lastException = e;
-                        reading = false;
-                        readingCondition.signal();
-                        //Allow this thread to terminate as we will need to reconnect with a new transport anyway
-                        running = false;
-                    }
+                }
+                if(running) {
+                    signalException(e);
+                    //Allow this thread to terminate as we will need to reconnect with a new transport anyway
+                    running = false;
                 }
             }
-        } catch(Exception e) {
-            if(!closed) {
-                log.error("Error reading from socket", e);
-            }
-            if(running) {
-                lastException = e;
-                reading = false;
-                readingCondition.signal();
-                //Allow this thread to terminate as we will need to reconnect with a new transport anyway
-                running = false;
+        }
+    }
+
+    private void deliverResponse(String received) throws InterruptedException {
+        readLock.lock();
+        try {
+            response = received;
+            reading = false;
+            readingCondition.signal();
+            while(!reading && running) {
+                readingCondition.await();
             }
         } finally {
             readLock.unlock();
         }
     }
 
+    private void signalException(Exception e) {
+        readLock.lock();
+        try {
+            lastException = e;
+            reading = false;
+            readingCondition.signal();
+        } finally {
+            readLock.unlock();
+        }
+    }
+
     protected String readInputStream(BufferedReader in) throws IOException {
         String response = readLine(in);
 
@@ -229,6 +261,26 @@ public class ElectrumTransport implements Transport, Closeable {
         return null;
     }
 
+    private static boolean isNotification(String json) {
+        try(JsonParser parser = JSON_FACTORY.createParser(json)) {
+            if(parser.nextToken() != JsonToken.START_OBJECT) {
+                return false;
+            }
+            while(parser.nextToken() == JsonToken.FIELD_NAME) {
+                String field = parser.currentName();
+                JsonToken value = parser.nextToken();
+                if("method".equals(field)) {
+                    return value == JsonToken.VALUE_STRING;
+                }
+                parser.skipChildren();
+            }
+            return false;
+        } catch(Exception e) {
+            log.warn("Could not parse JSON-RPC message from backend: " + e.getMessage());
+            return false;
+        }
+    }
+
     public Exception getLastException() {
         return lastException;
     }
@@ -238,6 +290,13 @@ public class ElectrumTransport implements Transport, Closeable {
         running = false;
         closed = true;
 
+        readLock.lock();
+        try {
+            readingCondition.signalAll();
+        } finally {
+            readLock.unlock();
+        }
+
         if(out != null) {
             out.close();
         }
@@ -253,24 +312,15 @@ public class ElectrumTransport implements Transport, Closeable {
         return closed;
     }
 
-    public static class Rpc {
-        public String id;
-
-        @Override
-        public boolean equals(Object o) {
-            if(this == o) {
-                return true;
-            }
-            if(o == null || getClass() != o.getClass()) {
-                return false;
-            }
-            Rpc rpc = (Rpc) o;
-            return Objects.equals(id, rpc.id);
+    private static Set<String> extractIdSet(String json) {
+        if(json == null || json.isEmpty()) {
+            return Collections.emptySet();
         }
-
-        @Override
-        public int hashCode() {
-            return Objects.hash(id);
+        Matcher m = ID_PATTERN.matcher(json);
+        Set<String> ids = new LinkedHashSet<>();
+        while(m.find()) {
+            ids.add(m.group(1));
         }
+        return ids;
     }
 }

src/main/java/com/sparrowwallet/frigate/electrum/InvalidParamsException.java

@@ -0,0 +1,14 @@
+package com.sparrowwallet.frigate.electrum;
+
+import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcError;
+
+@JsonRpcError(code = -32602, message = "Invalid params")
+public class InvalidParamsException extends Exception {
+    public InvalidParamsException(String message) {
+        super(message);
+    }
+
+    public InvalidParamsException(String message, Throwable cause) {
+        super(message, cause);
+    }
+}

src/main/java/com/sparrowwallet/frigate/electrum/RequestHandler.java

@@ -8,9 +8,11 @@ import com.sparrowwallet.drongo.silentpayments.SilentPaymentScanAddress;
 import com.sparrowwallet.frigate.Frigate;
 import com.sparrowwallet.frigate.SubscriptionStatus;
 import com.sparrowwallet.frigate.bitcoind.BitcoindClient;
-import com.sparrowwallet.frigate.bitcoind.BlockReorgEvent;
+import com.sparrowwallet.frigate.bitcoind.BlockReorgSyncStart;
+import com.sparrowwallet.frigate.bitcoind.BlockReorgSyncComplete;
 import com.sparrowwallet.frigate.index.*;
 import com.sparrowwallet.frigate.io.Config;
+import com.sparrowwallet.frigate.io.Server;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -18,11 +20,15 @@ import java.io.*;
 import java.lang.ref.WeakReference;
 import java.net.Socket;
 import java.nio.charset.StandardCharsets;
-import java.util.HashMap;
+import java.util.ArrayDeque;
+import java.util.Deque;
 import java.util.HashSet;
+import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.ReentrantLock;
 import java.util.stream.Collectors;
 
 public class RequestHandler implements Runnable, SubscriptionStatus, Thread.UncaughtExceptionHandler {
@@ -35,14 +41,20 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
     private final Thread reader;
 
     private boolean connected;
-    private boolean headersSubscribed;
-    private final Set<String> scriptHashesSubscribed = new HashSet<>();
-    private final Map<String, SilentPaymentAddressSubscription> silentPaymentsAddressesSubscribed = new HashMap<>();
+    private volatile boolean headersSubscribed;
+    private final Set<String> scriptHashesSubscribed = ConcurrentHashMap.newKeySet();
+    private final Map<String, SilentPaymentAddressSubscription> silentPaymentsAddressesSubscribed = new ConcurrentHashMap<>();
+    private final Deque<Runnable> postResponseTasks = new ArrayDeque<>();
+
+    private final ReentrantLock writeLock = new ReentrantLock();
+    private volatile PrintWriter out;
+    private final ElectrumNotificationService notificationService;
 
     public RequestHandler(Socket clientSocket, BitcoindClient bitcoindClient, IndexQuerier indexQuerier) {
         this.clientSocket = clientSocket;
-        if(Config.get().getBackendElectrumServer() != null) {
-            this.backendTransport = new ElectrumTransport(Config.get().getBackendElectrumServer().getHostAndPort(), new BackendSubscriptionService());
+        Server backendServer = Config.get().getServer().getBackendElectrumServerObj();
+        if(backendServer != null) {
+            this.backendTransport = new ElectrumTransport(backendServer.getHostAndPort(), backendServer.getProtocol(), new BackendSubscriptionService());
             this.reader = Thread.ofVirtual().name("BackendServerReadThread-" + System.identityHashCode(this)).unstarted(new ReadRunnable(backendTransport));
             reader.setUncaughtExceptionHandler(this);
         } else {
@@ -50,6 +62,7 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
             this.reader = null;
         }
         this.electrumServerService = new ElectrumServerService(bitcoindClient, this, indexQuerier, backendTransport);
+        this.notificationService = new JsonRpcClient(new ElectrumNotificationTransport(this)).onDemand(ElectrumNotificationService.class);
     }
 
     public void run() {
@@ -57,15 +70,17 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
         this.connected = true;
 
         try {
-            connectBackendTransport();
-
-            InputStream input  = clientSocket.getInputStream();
+            InputStream input = clientSocket.getInputStream();
             BufferedReader reader = new BufferedReader(new InputStreamReader(input, StandardCharsets.UTF_8));
 
             OutputStream output = clientSocket.getOutputStream();
-            PrintWriter out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(output, StandardCharsets.UTF_8)));
+            this.out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(output, StandardCharsets.UTF_8)));
+
+            connectBackendTransport();
 
             while(true) {
+                postResponseTasks.clear();
+
                 String request = reader.readLine();
                 if(request == null) {
                     break;
@@ -78,11 +93,12 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
                 }
 
                 String response = rpcServer.handle(request, electrumServerService);
-                out.println(response);
-                out.flush();
+                writeLine(response);
+
+                runPostResponseTasks();
             }
         } catch(IOException e) {
-            log.error("Could not communicate with client socket", e);
+            log.debug("Could not communicate with client socket: {}", e.getMessage());
         } finally {
             closeBackendTransport();
             this.connected = false;
@@ -97,6 +113,20 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
         }
     }
 
+    public void writeLine(String line) {
+        writeLock.lock();
+        try {
+            PrintWriter writer = out;
+            if(writer == null) {
+                return;
+            }
+            writer.println(line);
+            writer.flush();
+        } finally {
+            writeLock.unlock();
+        }
+    }
+
     private void connectBackendTransport() {
         if(backendTransport != null) {
             backendTransport.connect();
@@ -148,12 +178,42 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
         return scriptHashesSubscribed.contains(scriptHash);
     }
 
-    public void subscribeSilentPaymentsAddress(SilentPaymentScanAddress silentPaymentsScanAddress, Set<Integer> labelSet) {
-        silentPaymentsAddressesSubscribed.put(silentPaymentsScanAddress.toString(), new SilentPaymentAddressSubscription(silentPaymentsScanAddress, labelSet));
+    public void subscribeSilentPaymentsAddress(SilentPaymentScanAddress silentPaymentsScanAddress, Set<Integer> labelSet, int startHeight) {
+        SilentPaymentAddressSubscription previous = silentPaymentsAddressesSubscribed.get(silentPaymentsScanAddress.toString());
+        if(previous != null) {
+            previous.invalidateInFlightScans();
+        }
+        silentPaymentsAddressesSubscribed.put(silentPaymentsScanAddress.toString(), new SilentPaymentAddressSubscription(silentPaymentsScanAddress, labelSet, startHeight));
     }
 
     public void unsubscribeSilentPaymentsAddress(SilentPaymentScanAddress silentPaymentsScanAddress) {
-        silentPaymentsAddressesSubscribed.remove(silentPaymentsScanAddress.toString());
+        SilentPaymentAddressSubscription previous = silentPaymentsAddressesSubscribed.remove(silentPaymentsScanAddress.toString());
+        if(previous != null) {
+            previous.invalidateInFlightScans();
+        }
+    }
+
+    public SilentPaymentAddressSubscription getSilentPaymentsAddressSubscription(String silentPaymentsAddress) {
+        return silentPaymentsAddressesSubscribed.get(silentPaymentsAddress);
+    }
+
+    public void runAfterResponse(Runnable task) {
+        postResponseTasks.add(task);
+    }
+
+    private void runPostResponseTasks() {
+        while(!postResponseTasks.isEmpty()) {
+            Runnable task = postResponseTasks.poll();
+            try {
+                task.run();
+            } catch(Exception e) {
+                log.error("Error running post-response task", e);
+            }
+        }
+    }
+
+    public int getSilentPaymentsSubscriptionCount() {
+        return silentPaymentsAddressesSubscribed.size();
     }
 
     @Override
@@ -170,18 +230,18 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
     @Subscribe
     public void newBlock(ElectrumBlockHeader electrumBlockHeader) {
         if(isHeadersSubscribed()) {
-            ElectrumNotificationTransport electrumNotificationTransport = new ElectrumNotificationTransport(clientSocket);
-            JsonRpcClient jsonRpcClient = new JsonRpcClient(electrumNotificationTransport);
-            jsonRpcClient.onDemand(ElectrumNotificationService.class).notifyHeaders(electrumBlockHeader);
+            notifyHeaders(electrumBlockHeader);
         }
     }
 
+    void notifyHeaders(ElectrumBlockHeader electrumBlockHeader) {
+        notificationService.notifyHeaders(electrumBlockHeader);
+    }
+
     @Subscribe
     public void scriptHashStatus(ScriptHashStatus scriptHashStatus) {
         if(isScriptHashSubscribed(scriptHashStatus.scriptHash())) {
-            ElectrumNotificationTransport electrumNotificationTransport = new ElectrumNotificationTransport(clientSocket);
-            JsonRpcClient jsonRpcClient = new JsonRpcClient(electrumNotificationTransport);
-            jsonRpcClient.onDemand(ElectrumNotificationService.class).notifyScriptHash(scriptHashStatus.scriptHash(), scriptHashStatus.status());
+            notificationService.notifyScriptHash(scriptHashStatus.scriptHash(), scriptHashStatus.status());
         }
     }
 
@@ -189,28 +249,24 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
     public void silentPaymentsNotification(SilentPaymentsNotification notification) {
         if(isSilentPaymentsAddressSubscribed(notification.subscription().address()) && notification.status() == this) {
             SilentPaymentAddressSubscription subscription = silentPaymentsAddressesSubscribed.get(notification.subscription().address());
-            subscription.setHighestBlockHeight(notification.history().stream().mapToInt(TxEntry::getHeight).max().orElse(subscription.getHighestBlockHeight()));
+            if(!subscription.isActive()) {
+                return;
+            }
+            notification.history().stream().mapToInt(SilentPaymentsTxEntry::getHeight).filter(h -> h > 0).max().ifPresent(subscription::accumulateMaxBlockHeight);
             subscription.getMempoolTxids().addAll(notification.history().stream().filter(txEntry -> txEntry.height <= 0).map(txEntry -> Sha256Hash.wrap(txEntry.tx_hash)).collect(Collectors.toSet()));
 
-            try {
-                ElectrumNotificationTransport electrumNotificationTransport = new ElectrumNotificationTransport(clientSocket);
-                JsonRpcClient jsonRpcClient = new JsonRpcClient(electrumNotificationTransport);
-                jsonRpcClient.onDemand(ElectrumNotificationService.class).notifySilentPayments(notification.subscription(), notification.progress(), notification.history());
-            } catch(IllegalStateException e) {
-                if(e.getCause() instanceof java.io.IOException) {
-                    log.debug("Client disconnected before notification could be sent");
-                } else {
-                    throw e;
-                }
-            }
+            List<SilentPaymentsTxEntry> deliverable = notification.history().stream()
+                    .filter(txEntry -> txEntry.height <= 0 || !subscription.getMempoolTxids().contains(Sha256Hash.wrap(txEntry.tx_hash))).toList();
+
+            notificationService.notifySilentPayments(notification.subscription(), notification.progress(), deliverable);
         }
     }
 
     @Subscribe
     public void silentPaymentsBlocksIndexUpdate(SilentPaymentsBlocksIndexUpdate update) {
         for(SilentPaymentAddressSubscription subscription : silentPaymentsAddressesSubscribed.values()) {
-            if(update.fromBlockHeight() > subscription.getHighestBlockHeight()) {
-                electrumServerService.getIndexQuerier().startHistoryScan(subscription.getAddress(), update.fromBlockHeight(), null, subscription.getLabels(), new WeakReference<>(this), false);
+            if(subscription.isActive() && !subscription.isPendingHistoricalRescan() && update.fromBlockHeight() > subscription.getHighestBlockHeight()) {
+                electrumServerService.getIndexQuerier().startHistoryScan(subscription.getAddress(), update.fromBlockHeight(), null, subscription, new WeakReference<>(this), false);
             }
         }
     }
@@ -218,21 +274,31 @@ public class RequestHandler implements Runnable, SubscriptionStatus, Thread.Unca
     @Subscribe
     public void silentPaymentsMempoolIndexAdded(SilentPaymentsMempoolIndexAdded added) {
         for(SilentPaymentAddressSubscription subscription : silentPaymentsAddressesSubscribed.values()) {
-            electrumServerService.getIndexQuerier().startMempoolScan(subscription.getAddress(), null, null, subscription.getLabels(), new WeakReference<>(this));
+            if(subscription.isActive()) {
+                electrumServerService.getIndexQuerier().startMempoolScan(subscription.getAddress(), null, null, added.getTxids(), subscription, new WeakReference<>(this));
+            }
         }
     }
 
     @Subscribe
-    public void silentPaymentsMempoolIndexRemoved(SilentPaymentsMempoolIndexRemoved removed) {
+    public void blockReorgSyncStart(BlockReorgSyncStart event) {
+        int reorgPoint = event.reorgStartHeight() - 1;
         for(SilentPaymentAddressSubscription subscription : silentPaymentsAddressesSubscribed.values()) {
-            subscription.getMempoolTxids().removeAll(removed.getTxids());
+            subscription.invalidateInFlightScans();
+            subscription.accumulateMinBlockHeight(reorgPoint);
+            if(subscription.isActive() && !subscription.isHistoricalComplete()) {
+                subscription.markPendingHistoricalRescan();
+            }
         }
     }
 
     @Subscribe
-    public void blockReorgEvent(BlockReorgEvent event) {
+    public void blockReorgSyncComplete(BlockReorgSyncComplete event) {
         for(SilentPaymentAddressSubscription subscription : silentPaymentsAddressesSubscribed.values()) {
-            subscription.setHighestBlockHeight(event.startHeight() - 1);
+            if(subscription.isActive() && subscription.consumePendingHistoricalRescan()) {
+                int scanFrom = subscription.getHighestBlockHeight() + 1;
+                electrumServerService.getIndexQuerier().startHistoryScan(subscription.getAddress(), scanFrom, null, subscription, new WeakReference<>(this), true);
+            }
         }
     }
 

src/main/java/com/sparrowwallet/frigate/electrum/ServerFeatures.java

@@ -1,7 +1,31 @@
 package com.sparrowwallet.frigate.electrum;
 
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonInclude;
+
+import java.util.List;
 import java.util.Map;
 
-public record ServerFeatures(Map<String, HostInfo> hosts, String genesis_hash, String hash_function, String server_version, String protocol_max, String protocol_min, Integer pruning) {
-    public record HostInfo(Integer tcp_port, Integer ssl_port) {}
+@JsonInclude(JsonInclude.Include.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown = true)
+public record ServerFeatures(Map<String, HostInfo> hosts, String genesis_hash, String hash_function, String server_version, String protocol_max, String protocol_min, Integer pruning, List<Integer> silent_payments) {
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    public record HostInfo(Integer tcp_port, Integer ssl_port) {
+        //Some non-conformant Electrum servers return hosts as {"host": <port>} instead of
+        //{"host": {"tcp_port": <port>, "ssl_port": <port>}}. Accept that shape and treat the
+        //bare number as the tcp_port.
+        @JsonCreator(mode = JsonCreator.Mode.DELEGATING)
+        public static HostInfo fromTcpPort(int tcpPort) {
+            return new HostInfo(tcpPort, null);
+        }
+    }
+
+    public ServerFeatures withSilentPayments(List<Integer> versions) {
+        return new ServerFeatures(hosts, genesis_hash, hash_function, server_version, protocol_max, protocol_min, pruning, versions);
+    }
+
+    public ServerFeatures withHosts(Map<String, HostInfo> hosts) {
+        return new ServerFeatures(hosts, genesis_hash, hash_function, server_version, protocol_max, protocol_min, pruning, silent_payments);
+    }
 }

src/main/java/com/sparrowwallet/frigate/electrum/SilentPaymentAddressSubscription.java

@@ -3,18 +3,69 @@ package com.sparrowwallet.frigate.electrum;
 import com.sparrowwallet.drongo.protocol.Sha256Hash;
 import com.sparrowwallet.drongo.silentpayments.SilentPaymentScanAddress;
 
-import java.util.HashSet;
 import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BooleanSupplier;
 
 public class SilentPaymentAddressSubscription {
     private final SilentPaymentScanAddress address;
     private final Set<Integer> labels;
-    private int highestBlockHeight;
-    private final Set<Sha256Hash> mempoolTxids = new HashSet<>();
+    private final int startHeight;
+    private final AtomicInteger highestBlockHeight = new AtomicInteger();
+    private final Set<Sha256Hash> mempoolTxids = ConcurrentHashMap.newKeySet();
+    private volatile boolean active;
+    private volatile boolean historicalComplete;
+    private final AtomicBoolean pendingHistoricalRescan = new AtomicBoolean(false);
+    private final AtomicLong scanEpoch = new AtomicLong();
 
-    public SilentPaymentAddressSubscription(SilentPaymentScanAddress address, Set<Integer> labels) {
+    public SilentPaymentAddressSubscription(SilentPaymentScanAddress address, Set<Integer> labels, int startHeight) {
         this.address = address;
         this.labels = labels;
+        this.startHeight = startHeight;
+    }
+
+    public int getStartHeight() {
+        return startHeight;
+    }
+
+    public boolean isActive() {
+        return active;
+    }
+
+    public void setActive(boolean active) {
+        this.active = active;
+    }
+
+    public BooleanSupplier captureScanCancellation() {
+        long captured = scanEpoch.get();
+        return () -> scanEpoch.get() != captured;
+    }
+
+    public void invalidateInFlightScans() {
+        scanEpoch.incrementAndGet();
+    }
+
+    public boolean isHistoricalComplete() {
+        return historicalComplete;
+    }
+
+    public void markHistoricalComplete() {
+        historicalComplete = true;
+    }
+
+    public void markPendingHistoricalRescan() {
+        pendingHistoricalRescan.set(true);
+    }
+
+    public boolean isPendingHistoricalRescan() {
+        return pendingHistoricalRescan.get();
+    }
+
+    public boolean consumePendingHistoricalRescan() {
+        return pendingHistoricalRescan.getAndSet(false);
     }
 
     public SilentPaymentScanAddress getAddress() {
@@ -26,11 +77,15 @@ public class SilentPaymentAddressSubscription {
     }
 
     public int getHighestBlockHeight() {
-        return highestBlockHeight;
+        return highestBlockHeight.get();
     }
 
-    public void setHighestBlockHeight(int highestBlockHeight) {
-        this.highestBlockHeight = highestBlockHeight;
+    public void accumulateMaxBlockHeight(int candidate) {
+        highestBlockHeight.accumulateAndGet(candidate, Math::max);
+    }
+
+    public void accumulateMinBlockHeight(int candidate) {
+        highestBlockHeight.accumulateAndGet(candidate, Math::min);
     }
 
     public Set<Sha256Hash> getMempoolTxids() {

src/main/java/com/sparrowwallet/frigate/electrum/SilentPaymentsNotification.java

@@ -1,10 +1,9 @@
 package com.sparrowwallet.frigate.electrum;
 
 import com.sparrowwallet.frigate.SubscriptionStatus;
-import com.sparrowwallet.frigate.index.TxEntry;
 
 import java.util.List;
 
-public record SilentPaymentsNotification(SilentPaymentsSubscription subscription, double progress, List<TxEntry> history, SubscriptionStatus status) {
+public record SilentPaymentsNotification(SilentPaymentsSubscription subscription, double progress, List<SilentPaymentsTxEntry> history, SubscriptionStatus status) {
 
 }

src/main/java/com/sparrowwallet/frigate/electrum/SilentPaymentsTxEntry.java

@@ -0,0 +1,59 @@
+package com.sparrowwallet.frigate.electrum;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+
+import java.util.Objects;
+
+@JsonInclude(JsonInclude.Include.NON_NULL)
+public class SilentPaymentsTxEntry implements Comparable<SilentPaymentsTxEntry> {
+    public int height;
+    public String tx_hash;
+    public String tweak_key;
+
+    public SilentPaymentsTxEntry() {
+    }
+
+    public SilentPaymentsTxEntry(int height, String tx_hash, String tweak_key) {
+        this.height = height;
+        this.tx_hash = tx_hash;
+        this.tweak_key = tweak_key;
+    }
+
+    public int getHeight() {
+        return height;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if(this == o) {
+            return true;
+        }
+        if(!(o instanceof SilentPaymentsTxEntry that)) {
+            return false;
+        }
+
+        return height == that.height && Objects.equals(tx_hash, that.tx_hash) && Objects.equals(tweak_key, that.tweak_key);
+    }
+
+    @Override
+    public int hashCode() {
+        int result = height;
+        result = 31 * result + Objects.hashCode(tx_hash);
+        result = 31 * result + Objects.hashCode(tweak_key);
+        return result;
+    }
+
+    @Override
+    public int compareTo(SilentPaymentsTxEntry o) {
+        if(height <= 0 && o.height > 0) {
+            return 1;
+        }
+        if(height > 0 && o.height <= 0) {
+            return -1;
+        }
+        if(height != o.height) {
+            return height - o.height;
+        }
+        return tx_hash.compareTo(o.tx_hash);
+    }
+}

src/main/java/com/sparrowwallet/frigate/electrum/TxEntry.java

@@ -1,4 +1,4 @@
-package com.sparrowwallet.frigate.index;
+package com.sparrowwallet.frigate.electrum;
 
 import com.fasterxml.jackson.annotation.JsonInclude;
 import com.sparrowwallet.drongo.protocol.Transaction;
@@ -10,7 +10,6 @@ public class TxEntry implements Comparable<TxEntry> {
     public int height;
     private transient int index;
     public String tx_hash;
-    public String tweak_key;
     public Long fee;
 
     public TxEntry() {
@@ -20,15 +19,6 @@ public class TxEntry implements Comparable<TxEntry> {
         this.height = height;
         this.index = index;
         this.tx_hash = tx_hash;
-        this.tweak_key = null;
-        this.fee = null;
-    }
-
-    public TxEntry(int height, int index, String tx_hash, String tweak_key) {
-        this.height = height;
-        this.index = index;
-        this.tx_hash = tx_hash;
-        this.tweak_key = tweak_key;
         this.fee = null;
     }
 
@@ -36,7 +26,6 @@ public class TxEntry implements Comparable<TxEntry> {
         this.height = height;
         this.index = index;
         this.tx_hash = tx_hash;
-        this.tweak_key = null;
         this.fee = btcFee > 0.0 ? (long)(btcFee * Transaction.SATOSHIS_PER_BITCOIN) : null;
     }
 

src/main/java/com/sparrowwallet/frigate/index/DuckDBReadPool.java

@@ -29,8 +29,11 @@ public class DuckDBReadPool {
         this.masterConnection = (DuckDBConnection)DriverManager.getConnection(connectionUrl, props);
 
         try(Statement stmt = masterConnection.createStatement()) {
-            if(Config.get().getDbThreads() != null) {
-                stmt.execute("SET threads = '" + Config.get().getDbThreads() + "'");
+            if(Config.get().getScan().getDbThreads() != null) {
+                stmt.execute("SET threads = '" + Config.get().getScan().getDbThreads() + "'");
+            }
+            if(Config.get().getScan().getMemoryLimit() != null) {
+                stmt.execute("SET memory_limit = '" + Config.get().getScan().getMemoryLimit() + "'");
             }
 
             File ufsecpExtensionFile = Storage.getUfsecpExtensionFile();

src/main/java/com/sparrowwallet/frigate/index/HistoricalScanMetrics.java

@@ -0,0 +1,95 @@
+package com.sparrowwallet.frigate.index;
+
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicIntegerArray;
+
+/**
+ * Privacy-preserving aggregate counters for historical scan throughput.
+ */
+class HistoricalScanMetrics {
+    private static final int MIN_SAMPLES_PER_BUCKET = 10;
+    private static final int COUNT_ROUNDING = 10;
+
+    static final String[] RESULT_LABELS = {"0", "1-10", "11-100", "101-1000", "1001-10000", "10001+"};
+    static final String[] DURATION_LABELS = {"0-100ms", "100-500ms", "500ms-2s", "2-10s", "10-60s", "60s+"};
+
+    private final AtomicIntegerArray resultBuckets = new AtomicIntegerArray(RESULT_LABELS.length);
+    private final AtomicIntegerArray durationBuckets = new AtomicIntegerArray(DURATION_LABELS.length);
+
+    record Snapshot(int[] results, int[] durations) {}
+
+    void record(int resultCount, long durationMillis) {
+        resultBuckets.incrementAndGet(resultBucket(resultCount));
+        durationBuckets.incrementAndGet(durationBucket(durationMillis));
+    }
+
+    //a record() concurrent with snapshotAndReset() may have its two increments split across windows. The per-bucket drift is at
+    //most one sample per emission and never crosses subscription identity, so it is fine for an aggregate stat.
+    Snapshot snapshotAndReset() {
+        int[] results = new int[RESULT_LABELS.length];
+        int[] durations = new int[DURATION_LABELS.length];
+        for(int i = 0; i < results.length; i++) {
+            results[i] = resultBuckets.getAndSet(i, 0);
+        }
+        for(int i = 0; i < durations.length; i++) {
+            durations[i] = durationBuckets.getAndSet(i, 0);
+        }
+        return new Snapshot(results, durations);
+    }
+
+    Optional<String> format(Snapshot snapshot) {
+        String resultsStr = formatHist(snapshot.results(), RESULT_LABELS);
+        String durationsStr = formatHist(snapshot.durations(), DURATION_LABELS);
+        if(resultsStr.isEmpty() && durationsStr.isEmpty()) {
+            return Optional.empty();
+        }
+
+        StringBuilder sb = new StringBuilder("Aggregate SP scan stats (1h window):");
+        if(!resultsStr.isEmpty()) {
+            sb.append(" results [").append(resultsStr).append("]");
+        }
+        if(!durationsStr.isEmpty()) {
+            sb.append(" duration [").append(durationsStr).append("]");
+        }
+        return Optional.of(sb.toString());
+    }
+
+    static int resultBucket(int n) {
+        if(n <= 0) return 0;
+        if(n <= 10) return 1;
+        if(n <= 100) return 2;
+        if(n <= 1000) return 3;
+        if(n <= 10000) return 4;
+        return 5;
+    }
+
+    static int durationBucket(long ms) {
+        if(ms < 100) return 0;
+        if(ms < 500) return 1;
+        if(ms < 2000) return 2;
+        if(ms < 10000) return 3;
+        if(ms < 60000) return 4;
+        return 5;
+    }
+
+    private static String formatHist(int[] counts, String[] labels) {
+        StringBuilder sb = new StringBuilder();
+        for(int i = 0; i < counts.length; i++) {
+            int rounded = roundCount(counts[i]);
+            if(rounded > 0) {
+                if(sb.length() > 0) {
+                    sb.append(", ");
+                }
+                sb.append(labels[i]).append(":").append(rounded);
+            }
+        }
+        return sb.toString();
+    }
+
+    private static int roundCount(int raw) {
+        if(raw < MIN_SAMPLES_PER_BUCKET) {
+            return 0;
+        }
+        return ((raw + COUNT_ROUNDING / 2) / COUNT_ROUNDING) * COUNT_ROUNDING;
+    }
+}

src/main/java/com/sparrowwallet/frigate/index/Index.java

@@ -12,6 +12,7 @@ import com.sparrowwallet.frigate.Frigate;
 import com.sparrowwallet.frigate.SubscriptionStatus;
 import com.sparrowwallet.frigate.electrum.SilentPaymentsNotification;
 import com.sparrowwallet.frigate.electrum.SilentPaymentsSubscription;
+import com.sparrowwallet.frigate.electrum.SilentPaymentsTxEntry;
 import com.sparrowwallet.frigate.io.ComputeBackend;
 import com.sparrowwallet.frigate.io.Config;
 import com.sparrowwallet.frigate.io.Storage;
@@ -23,31 +24,51 @@ import org.slf4j.LoggerFactory;
 
 import java.io.File;
 import java.lang.ref.WeakReference;
+import java.math.BigInteger;
 import java.sql.*;
 import java.util.*;
 import java.util.concurrent.*;
-import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BooleanSupplier;
 import java.util.stream.Collectors;
 
 public class Index {
     private static final Logger log = LoggerFactory.getLogger(Index.class);
     public static final String DEFAULT_DB_FILENAME = "frigate.duckdb";
     private static final String TWEAK_TABLE = "tweak";
+    private static final String INDEXED_BLOCK_TABLE = "indexed_block";
     public static final int HISTORY_PAGE_SIZE = 100;
 
+    private static final String AUDIT_SCAN_KEY_ENV = "FRIGATE_AUDIT_SCAN_KEY";
+    private static final String AUDIT_SPEND_KEY_ENV = "FRIGATE_AUDIT_SPEND_KEY";
+
     private final DbManager dbManager;
-    private volatile int lastBlockIndexed = -1;
+    private final AtomicInteger lastBlockIndexed = new AtomicInteger(-1);
     private final int batchSize;
+    private final ECKey auditScanKey;
+    private final ECKey auditSpendKey;
+    private volatile boolean steadyState = false;
 
     public Index(int startHeight, boolean inMemory, int batchSize) {
-        lastBlockIndexed = Math.max(lastBlockIndexed, startHeight - 1);
+        lastBlockIndexed.accumulateAndGet(startHeight - 1, Math::max);
         this.batchSize = batchSize;
 
+        String scanKeyHex = System.getenv(AUDIT_SCAN_KEY_ENV);
+        String spendKeyHex = System.getenv(AUDIT_SPEND_KEY_ENV);
+        if(scanKeyHex != null && spendKeyHex != null) {
+            this.auditScanKey = ECKey.fromPrivate(Utils.hexToBytes(scanKeyHex));
+            this.auditSpendKey = ECKey.fromPublicOnly(Utils.hexToBytes(spendKeyHex));
+            log.warn("Scan audit mode enabled — output prefixes will be computed for the provided wallet keys");
+        } else {
+            this.auditScanKey = null;
+            this.auditSpendKey = null;
+        }
+
         if(inMemory) {
             dbManager = new MemoryDbManager();
         } else {
-            String dbUrl = Config.get().getDbUrl();
-            List<String> readDbUrls = Config.get().getReadDbUrls();
+            String dbUrl = Config.get().getDatabase().getUrl();
+            List<String> readDbUrls = Config.get().getDatabase().getReadUrls();
             if(dbUrl != null && readDbUrls != null && !readDbUrls.isEmpty()) {
                 dbManager = new ScalingDbManager(dbUrl, readDbUrls);
             } else if(dbUrl == null) {
@@ -61,9 +82,12 @@ public class Index {
         try {
             dbManager.executeWrite(connection -> {
                 try(Statement stmt = connection.createStatement()) {
-                    return stmt.execute("CREATE TABLE IF NOT EXISTS " + TWEAK_TABLE + " (txid BLOB NOT NULL, height INTEGER NOT NULL, tweak_key BLOB NOT NULL, outputs BIGINT[])");
+                    stmt.execute("CREATE TABLE IF NOT EXISTS " + TWEAK_TABLE + " (txid BLOB NOT NULL, height INTEGER NOT NULL, tweak_key BLOB NOT NULL, outputs BIGINT[])");
+                    stmt.execute("CREATE TABLE IF NOT EXISTS " + INDEXED_BLOCK_TABLE + " (height INTEGER NOT NULL, block_hash BLOB NOT NULL, singleton BOOLEAN PRIMARY KEY DEFAULT true CHECK (singleton))");
+                    return true;
                 }
             });
+            seedIndexedBlockIfEmpty();
         } catch(Exception e) {
             throw new ConfigurationException("Error initialising index", e);
         }
@@ -73,8 +97,32 @@ public class Index {
         }
     }
 
+    private void seedIndexedBlockIfEmpty() throws SQLException, InterruptedException {
+        dbManager.executeWrite(connection -> {
+            try(Statement stmt = connection.createStatement();
+                ResultSet rs = stmt.executeQuery("SELECT COUNT(*) FROM " + INDEXED_BLOCK_TABLE)) {
+                if(rs.next() && rs.getInt(1) == 0) {
+                    try(ResultSet maxRs = stmt.executeQuery("SELECT MAX(height) FROM " + TWEAK_TABLE)) {
+                        if(maxRs.next() && maxRs.getObject(1) != null) {
+                            int maxHeight = maxRs.getInt(1);
+                            if(maxHeight > 0) {
+                                try(PreparedStatement ins = connection.prepareStatement("INSERT INTO " + INDEXED_BLOCK_TABLE + " (height, block_hash) VALUES (?, ?)")) {
+                                    ins.setInt(1, maxHeight);
+                                    ins.setBytes(2, new byte[0]);
+                                    ins.executeUpdate();
+                                }
+                            }
+                        }
+                    }
+                }
+
+                return true;
+            }
+        });
+    }
+
     private void checkGpuBackend() {
-        ComputeBackend computeBackend = Config.get().getComputeBackend();
+        ComputeBackend computeBackend = Config.get().getScan().getComputeBackendEnum();
         if(computeBackend == ComputeBackend.CPU) {
             return;
         }
@@ -124,72 +172,149 @@ public class Index {
         dbManager.close();
     }
 
-    public int getLastBlockIndexed() {
-        try {
-            return dbManager.executeRead(connection -> {
-                try(PreparedStatement statement = connection.prepareStatement("SELECT MAX(height) from " + TWEAK_TABLE)) {
-                    ResultSet resultSet = statement.executeQuery();
-                    return resultSet.next() ? Math.max(lastBlockIndexed, resultSet.getInt(1)) : lastBlockIndexed;
-                }
-            });
-        } catch(Exception e) {
-            log.error("Error getting last block indexed", e);
-            return lastBlockIndexed;
-        }
+    public void setSteadyState(boolean steadyState) {
+        this.steadyState = steadyState;
     }
 
-    public void addToIndex(Map<BlockTransaction, byte[]> transactions) {
+    public void repairOrphanTweaks() {
         if(dbManager.isShutdown()) {
             return;
         }
 
-        int fromBlockHeight = lastBlockIndexed;
         try {
-            lastBlockIndexed = dbManager.executeWrite(connection -> {
-                DuckDBConnection duckDBConnection = (DuckDBConnection)connection;
-                try(DuckDBAppender appender = duckDBConnection.createAppender(DuckDBConnection.DEFAULT_SCHEMA, TWEAK_TABLE)) {
-                    int blockHeight = -1;
-
-                    for(BlockTransaction blkTx : transactions.keySet()) {
-                        appender.beginRow();
-                        appender.append(blkTx.getTransaction().getTxId().getBytes());
-                        appender.append(blkTx.getHeight());
-                        appender.append(transactions.get(blkTx));
-
-                        List<TransactionOutput> outputs = blkTx.getTransaction().getOutputs();
-                        List<Long> hashPrefixes = new ArrayList<>();
-                        for(TransactionOutput output : outputs) {
-                            if(ScriptType.P2TR.isScriptType(output.getScript())) {
-                                long hashPrefix = getHashPrefix(ScriptType.P2TR.getPublicKeyFromScript(output.getScript()).getPubKey(), 1);
-                                hashPrefixes.add(hashPrefix);
-                            }
-                        }
-                        appender.append(hashPrefixes.stream().mapToLong(Long::longValue).toArray());
-                        appender.endRow();
-
-                        blockHeight = Math.max(blockHeight, blkTx.getHeight());
-                    }
-
-                    if(blockHeight <= 0 && lastBlockIndexed < 0) {
-                        log.info("Indexed " + transactions.size() + " mempool transactions");
-                    } else if(blockHeight > 0) {
-                        log.info("Indexed " + transactions.size() + " transactions to block height " + blockHeight);
-                    }
-
-                    return blockHeight;
+            int deleted = dbManager.executeWrite(connection -> {
+                try(PreparedStatement ps = connection.prepareStatement("DELETE FROM " + TWEAK_TABLE + " WHERE height > (SELECT height FROM " + INDEXED_BLOCK_TABLE + ")")) {
+                    return ps.executeUpdate();
                 }
             });
+            if(deleted > 0) {
+                log.info("Removed {} orphan tweak rows above the indexed-block marker (interrupted shutdown recovery)", deleted);
+            }
+        } catch(Exception e) {
+            log.error("Error repairing orphan tweak rows", e);
+        }
+    }
 
-            if(lastBlockIndexed <= 0) {
+    public int getLastBlockIndexed() {
+        try {
+            return dbManager.executeRead(connection -> {
+                try(PreparedStatement statement = connection.prepareStatement("SELECT height FROM " + INDEXED_BLOCK_TABLE)) {
+                    ResultSet resultSet = statement.executeQuery();
+                    return resultSet.next() ? Math.max(lastBlockIndexed.get(), resultSet.getInt(1)) : lastBlockIndexed.get();
+                }
+            });
+        } catch(Exception e) {
+            log.error("Error getting last block indexed", e);
+            return lastBlockIndexed.get();
+        }
+    }
+
+    public byte[] getLastBlockHash() {
+        try {
+            return dbManager.executeRead(connection -> {
+                try(PreparedStatement statement = connection.prepareStatement("SELECT block_hash FROM " + INDEXED_BLOCK_TABLE)) {
+                    ResultSet resultSet = statement.executeQuery();
+                    if(!resultSet.next()) {
+                        return null;
+                    }
+                    byte[] hash = resultSet.getBytes(1);
+                    return (hash == null || hash.length == 0) ? null : hash;
+                }
+            });
+        } catch(Exception e) {
+            log.error("Error getting last block hash", e);
+            return null;
+        }
+    }
+
+    public void addToIndex(int height, byte[] blockHash, Map<BlockTransaction, byte[]> transactions) {
+        if(dbManager.isShutdown()) {
+            return;
+        }
+
+        int fromBlockHeight = lastBlockIndexed.get();
+        try {
+            int newLastBlockIndexed = dbManager.executeWrite(connection -> {
+                if(!transactions.isEmpty()) {
+                    DuckDBConnection duckDBConnection = (DuckDBConnection)connection;
+                    try(DuckDBAppender appender = duckDBConnection.createAppender(DuckDBConnection.DEFAULT_SCHEMA, TWEAK_TABLE)) {
+                        int blockHeight = -1;
+
+                        for(BlockTransaction blkTx : transactions.keySet()) {
+                            appender.beginRow();
+                            appender.append(blkTx.getTransaction().getTxId().getBytes());
+                            appender.append(blkTx.getHeight());
+                            appender.append(transactions.get(blkTx));
+
+                            List<Long> hashPrefixes = new ArrayList<>();
+                            if(auditScanKey != null) {
+                                long hashPrefix = getAuditHashPrefix(transactions, blkTx);
+                                hashPrefixes.add(hashPrefix);
+                            } else {
+                                List<TransactionOutput> outputs = blkTx.getTransaction().getOutputs();
+                                for(TransactionOutput output : outputs) {
+                                    if(ScriptType.P2TR.isScriptType(output.getScript())) {
+                                        long hashPrefix = getHashPrefix(ScriptType.P2TR.getPublicKeyFromScript(output.getScript()).getPubKey(), 1);
+                                        hashPrefixes.add(hashPrefix);
+                                    }
+                                }
+                            }
+                            appender.append(hashPrefixes.stream().mapToLong(Long::longValue).toArray());
+                            appender.endRow();
+
+                            blockHeight = Math.max(blockHeight, blkTx.getHeight());
+                        }
+
+                        if(blockHeight <= 0 && lastBlockIndexed.get() < 0) {
+                            log.info("Indexed " + transactions.size() + " mempool transactions");
+                        } else if(blockHeight > 0) {
+                            String msg = "Indexed " + transactions.size() + " transactions to block height " + blockHeight;
+                            if(steadyState) {
+                                log.info(msg);
+                            } else {
+                                log.debug(msg);
+                            }
+                        }
+                    }
+                }
+
+                if(height > 0 && blockHash != null) {
+                    try(PreparedStatement ps = connection.prepareStatement("INSERT INTO " + INDEXED_BLOCK_TABLE + " (height, block_hash) VALUES (?, ?) " +
+                            "ON CONFLICT (singleton) DO UPDATE SET height = excluded.height, block_hash = excluded.block_hash")) {
+                        ps.setInt(1, height);
+                        ps.setBytes(2, blockHash);
+                        ps.executeUpdate();
+                    }
+                }
+
+                return height;
+            });
+            lastBlockIndexed.set(newLastBlockIndexed);
+
+            if(transactions.isEmpty()) {
+                //empty block: marker advanced, but nothing to notify on
+            } else if(newLastBlockIndexed <= 0) {
                 Frigate.getEventBus().post(new SilentPaymentsMempoolIndexAdded(transactions.keySet().stream().map(blkTx -> blkTx.getTransaction().getTxId()).collect(Collectors.toSet())));
             } else {
-                Frigate.getEventBus().post(new SilentPaymentsBlocksIndexUpdate(fromBlockHeight + 1, lastBlockIndexed, transactions.size()));
+                Frigate.getEventBus().post(new SilentPaymentsBlocksIndexUpdate(fromBlockHeight + 1, newLastBlockIndexed, transactions.size()));
             }
         } catch(Exception e) {
             log.error("Error adding to index", e);
         }
     }
 
+    private long getAuditHashPrefix(Map<BlockTransaction, byte[]> transactions, BlockTransaction blkTx) {
+        byte[] tweakKeyBytes = transactions.get(blkTx);
+        ECKey tweakKey = ECKey.fromPublicOnly(compressRawKey(tweakKeyBytes));
+        ECKey sharedSecret = tweakKey.multiply(auditScanKey.getPrivKey(), true);
+        byte[] ser37 = new byte[37];
+        System.arraycopy(sharedSecret.getPubKey(true), 0, ser37, 0, 33);
+        byte[] t_k = Utils.taggedHash("BIP0352/SharedSecret", ser37);
+        ECKey tkG = ECKey.fromPublicOnly(ECKey.publicKeyFromPrivate(new BigInteger(1, t_k), true));
+        ECKey P0 = auditSpendKey.add(tkG, true);
+        return getHashPrefix(P0.getPubKeyXCoord(), 0);
+    }
+
     public void removeFromIndex(int startHeight) {
         if(dbManager.isShutdown()) {
             return;
@@ -197,11 +322,30 @@ public class Index {
 
         try {
             dbManager.executeWrite(connection -> {
-                try(PreparedStatement statement = connection.prepareStatement("DELETE FROM " + TWEAK_TABLE + " WHERE height >= ?")) {
-                    statement.setInt(1, startHeight);
-                    return statement.execute();
+                boolean prevAutoCommit = connection.getAutoCommit();
+                connection.setAutoCommit(false);
+                try {
+                    try(PreparedStatement deleteTweak = connection.prepareStatement("DELETE FROM " + TWEAK_TABLE + " WHERE height >= ?")) {
+                        deleteTweak.setInt(1, startHeight);
+                        deleteTweak.execute();
+                    }
+                    //hash applied to the original (higher) marker height — clear it so the startup hash check is a no-op until the first re-indexed block writes a real hash
+                    try(PreparedStatement updateMarker = connection.prepareStatement("INSERT INTO " + INDEXED_BLOCK_TABLE + " (height, block_hash) VALUES (?, ?) " +
+                            "ON CONFLICT (singleton) DO UPDATE SET height = excluded.height, block_hash = excluded.block_hash")) {
+                        updateMarker.setInt(1, Math.max(0, startHeight - 1));
+                        updateMarker.setBytes(2, new byte[0]);
+                        updateMarker.executeUpdate();
+                    }
+                    connection.commit();
+                    return true;
+                } catch(SQLException e) {
+                    connection.rollback();
+                    throw e;
+                } finally {
+                    connection.setAutoCommit(prevAutoCommit);
                 }
             });
+            lastBlockIndexed.accumulateAndGet(startHeight - 1, Math::min);
         } catch(Exception e) {
             log.error("Error removing from index", e);
         }
@@ -231,62 +375,72 @@ public class Index {
         }
     }
 
-    public List<TxEntry> getHistoryAsync(SilentPaymentScanAddress scanAddress, SilentPaymentsSubscription subscription, Integer startHeight, Integer endHeight, WeakReference<SubscriptionStatus> subscriptionStatusRef) {
-        ConcurrentLinkedQueue<TxEntry> queue = new ConcurrentLinkedQueue<>();
+    public List<SilentPaymentsTxEntry> getHistoryAsync(SilentPaymentScanAddress scanAddress, SilentPaymentsSubscription subscription, Integer startHeight, Integer endHeight, Set<Sha256Hash> mempoolTxids, WeakReference<SubscriptionStatus> subscriptionStatusRef, BooleanSupplier cancelled, boolean isHistorical) {
+        if(mempoolTxids != null && mempoolTxids.isEmpty()) {
+            return Collections.emptyList();
+        }
+
+        ConcurrentLinkedQueue<SilentPaymentsTxEntry> queue = new ConcurrentLinkedQueue<>();
         byte[] scanKeyBytes = Utils.reverseBytes(scanAddress.getScanKey().getPrivKeyBytes());
 
         try {
             dbManager.executeRead(connection -> {
-                String sql = getSql(subscription, startHeight, endHeight);
+                String sql = getSql(subscription, startHeight, endHeight, mempoolTxids, isHistorical);
 
                 try(DuckDBPreparedStatement statement = connection.prepareStatement(sql).unwrap(DuckDBPreparedStatement.class)) {
-                    if(isUnsubscribed(scanAddress, subscriptionStatusRef)) {
+                    if(isUnsubscribed(scanAddress, subscriptionStatusRef) || cancelled.getAsBoolean()) {
                         return false;
                     }
 
-                    bindParameters(statement, scanAddress, subscription, startHeight, endHeight);
+                    Long totalRows = isHistorical ? getInputRowCount(connection, startHeight, endHeight) : null;
+                    bindParameters(statement, scanAddress, subscription, startHeight, endHeight, mempoolTxids, isHistorical, totalRows);
 
-                    try(ScheduledThreadPoolExecutor queryProgressExecutor = new ScheduledThreadPoolExecutor(1, r -> {
-                        ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat("IndexQueryProgress-%d").build();
-                        Thread t = namedThreadFactory.newThread(r);
-                        t.setDaemon(true);
-                        return t;
-                    })) {
-                        queryProgressExecutor.scheduleAtFixedRate(() -> {
-                            try {
-                                if(dbManager.isShutdown() || isUnsubscribed(scanAddress, subscriptionStatusRef)) {
-                                    statement.cancel();
-                                    queryProgressExecutor.shutdownNow();
-                                    return;
-                                }
-
-                                double progress = pollScanProgress(scanKeyBytes);
-
-                                List<TxEntry> history = new ArrayList<>();
-                                TxEntry entry;
-                                while((entry = queue.poll()) != null) {
-                                    history.add(entry);
-                                    if(history.size() >= HISTORY_PAGE_SIZE) {
-                                        Frigate.getEventBus().post(new SilentPaymentsNotification(subscription, progress, new ArrayList<>(history), subscriptionStatusRef.get()));
-                                        history.clear();
+                    if(isHistorical) {
+                        try(ScheduledThreadPoolExecutor queryProgressExecutor = new ScheduledThreadPoolExecutor(1, r -> {
+                            ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat("IndexQueryProgress-%d").build();
+                            Thread t = namedThreadFactory.newThread(r);
+                            t.setDaemon(true);
+                            return t;
+                        })) {
+                            queryProgressExecutor.scheduleAtFixedRate(() -> {
+                                try {
+                                    if(queryProgressExecutor.isShutdown() || dbManager.isShutdown() || isUnsubscribed(scanAddress, subscriptionStatusRef) || cancelled.getAsBoolean()) {
+                                        statement.cancel();
+                                        queryProgressExecutor.shutdownNow();
+                                        return;
+                                    }
+
+                                    double progress = pollScanProgress(scanKeyBytes);
+
+                                    if(queryProgressExecutor.isShutdown()) {
+                                        return;
+                                    }
+
+                                    List<SilentPaymentsTxEntry> history = new ArrayList<>();
+                                    SilentPaymentsTxEntry entry;
+                                    while((entry = queue.poll()) != null) {
+                                        history.add(entry);
+                                        if(history.size() >= HISTORY_PAGE_SIZE) {
+                                            Frigate.getEventBus().post(new SilentPaymentsNotification(subscription, progress, new ArrayList<>(history), subscriptionStatusRef.get()));
+                                            history.clear();
+                                        }
                                     }
-                                }
-                                if(!history.isEmpty() || queryProgressExecutor.getTaskCount() % 5 == 0) {
                                     Frigate.getEventBus().post(new SilentPaymentsNotification(subscription, progress, new ArrayList<>(history), subscriptionStatusRef.get()));
                                     history.clear();
+                                } catch(Exception e) {
+                                    log.error("Error getting query progress", e);
                                 }
-                            } catch(Exception e) {
-                                log.error("Error getting query progress", e);
-                            }
-                        }, 1, 1, TimeUnit.SECONDS);
+                            }, 5, 5, TimeUnit.SECONDS);
 
-                        ResultSet resultSet = statement.executeQuery();
-                        while(resultSet.next()) {
-                            byte[] txid = resultSet.getBytes(1);
-                            byte[] tweak_key = compressRawKey(resultSet.getBytes(2));
-                            int height = resultSet.getInt(3);
-                            queue.offer(new TxEntry(height, 0, Utils.bytesToHex(txid), Utils.bytesToHex(tweak_key)));
+                            try {
+                                drainResultSet(statement.executeQuery(), queue);
+                            } finally {
+                                //interrupt any parked progress poll - the implicit close() doesn't, which deadlocks against a queued writer
+                                queryProgressExecutor.shutdownNow();
+                            }
                         }
+                    } else {
+                        drainResultSet(statement.executeQuery(), queue);
                     }
                 }
 
@@ -304,12 +458,12 @@ public class Index {
             return Collections.emptyList();
         }
 
-        if(isUnsubscribed(scanAddress, subscriptionStatusRef)) {
+        if(isUnsubscribed(scanAddress, subscriptionStatusRef) || cancelled.getAsBoolean()) {
             return Collections.emptyList();
         }
 
-        List<TxEntry> history = new ArrayList<>();
-        TxEntry entry;
+        List<SilentPaymentsTxEntry> history = new ArrayList<>();
+        SilentPaymentsTxEntry entry;
         while((entry = queue.poll()) != null) {
             history.add(entry);
         }
@@ -317,46 +471,39 @@ public class Index {
         return history;
     }
 
-    private String getSql(SilentPaymentsSubscription subscription, Integer startHeight, Integer endHeight) {
+    private void drainResultSet(ResultSet resultSet, ConcurrentLinkedQueue<SilentPaymentsTxEntry> queue) throws SQLException {
+        while(resultSet.next()) {
+            byte[] txid = resultSet.getBytes(1);
+            byte[] tweak_key = compressRawKey(resultSet.getBytes(2));
+            int height = resultSet.getInt(3);
+            queue.offer(new SilentPaymentsTxEntry(height, Utils.bytesToHex(txid), Utils.bytesToHex(tweak_key)));
+        }
+    }
+
+    private String getSql(SilentPaymentsSubscription subscription, Integer startHeight, Integer endHeight, Set<Sha256Hash> mempoolTxids, boolean isHistorical) {
         String labelsStr = "[" + String.join(", ", Collections.nCopies(subscription.labels().length, "?")) + "]";
 
-        String sql = "SELECT txid, tweak_key, height FROM ufsecp_scan((SELECT txid, height, tweak_key, outputs FROM " + TWEAK_TABLE;
+        String sql = "SELECT txid, tweak_key, height FROM ufsecp_scan((SELECT txid, height, tweak_key, outputs FROM " + TWEAK_TABLE
+                + buildWhereClause(startHeight, endHeight, mempoolTxids)
+                + "), ?, ?, " + labelsStr + ", batch_size := ?";
 
-        if(startHeight != null || endHeight != null) {
-            sql += " WHERE ";
-        }
-
-        if(startHeight != null) {
-            sql += "height >= ?";
-            if(endHeight != null) {
-                sql += " AND ";
-            }
-        }
-
-        if(endHeight != null) {
-            sql += "height <= ?";
-        }
-
-        sql += "), ?, ?, " + labelsStr + ", batch_size := ?";
-
-        ComputeBackend computeBackend = Config.get().getComputeBackend();
-        if(computeBackend != ComputeBackend.AUTO) {
+        ComputeBackend backend = resolveBackend(isHistorical);
+        if(backend != ComputeBackend.AUTO) {
             sql += ", backend := ?";
         }
 
+        if(isHistorical) {
+            sql += ", total_rows := ?";
+        }
+
         sql += ") ORDER BY height";
 
         return sql;
     }
 
-    private void bindParameters(DuckDBPreparedStatement statement, SilentPaymentScanAddress scanAddress, SilentPaymentsSubscription subscription, Integer startHeight, Integer endHeight) throws SQLException {
-        int index = 1;
-        if(startHeight != null) {
-            statement.setInt(index++, startHeight);
-        }
-        if(endHeight != null) {
-            statement.setInt(index++, endHeight);
-        }
+    private void bindParameters(DuckDBPreparedStatement statement, SilentPaymentScanAddress scanAddress, SilentPaymentsSubscription subscription, Integer startHeight, Integer endHeight, Set<Sha256Hash> mempoolTxids, boolean isHistorical, Long totalRows) throws SQLException {
+        int index = bindTweakHeightFilter(statement, 1, startHeight, endHeight);
+        index = bindTweakTxidsFilter(statement, index, mempoolTxids);
         statement.setBytes(index++, Utils.reverseBytes(scanAddress.getScanKey().getPrivKeyBytes()));
         statement.setBytes(index++, SilentPaymentUtils.getSecp256k1PubKey(scanAddress.getSpendKey()));
         for(Integer label : subscription.labels()) {
@@ -364,10 +511,91 @@ public class Index {
         }
         statement.setInt(index++, batchSize);
 
-        ComputeBackend computeBackend = Config.get().getComputeBackend();
-        if(computeBackend != ComputeBackend.AUTO) {
-            statement.setString(index, computeBackend.toSqlValue());
+        ComputeBackend backend = resolveBackend(isHistorical);
+        if(backend != ComputeBackend.AUTO) {
+            statement.setString(index++, backend.toSqlValue());
         }
+
+        if(totalRows != null) {
+            statement.setLong(index, totalRows);
+        }
+    }
+
+    private static String buildWhereClause(Integer startHeight, Integer endHeight, Set<Sha256Hash> mempoolTxids) {
+        String heightClause = tweakHeightFilter(startHeight, endHeight);
+        String txidsClause = tweakTxidsFilter(mempoolTxids);
+        if(heightClause.isEmpty() && txidsClause.isEmpty()) {
+            return "";
+        }
+        if(heightClause.isEmpty()) {
+            return " WHERE " + txidsClause;
+        }
+        if(txidsClause.isEmpty()) {
+            return " WHERE " + heightClause;
+        }
+        return " WHERE " + heightClause + " AND " + txidsClause;
+    }
+
+    private static String tweakHeightFilter(Integer startHeight, Integer endHeight) {
+        if(startHeight == null && endHeight == null) {
+            return "";
+        }
+        StringBuilder sql = new StringBuilder();
+        if(startHeight != null) {
+            sql.append("height >= ?");
+            if(endHeight != null) {
+                sql.append(" AND ");
+            }
+        }
+        if(endHeight != null) {
+            sql.append("height <= ?");
+        }
+        return sql.toString();
+    }
+
+    private static String tweakTxidsFilter(Set<Sha256Hash> mempoolTxids) {
+        if(mempoolTxids == null) {
+            return "";
+        }
+        return "txid IN (SELECT unnest(?))";
+    }
+
+    private static int bindTweakHeightFilter(PreparedStatement statement, int startIndex, Integer startHeight, Integer endHeight) throws SQLException {
+        int idx = startIndex;
+        if(startHeight != null) {
+            statement.setInt(idx++, startHeight);
+        }
+        if(endHeight != null) {
+            statement.setInt(idx++, endHeight);
+        }
+        return idx;
+    }
+
+    private static int bindTweakTxidsFilter(PreparedStatement statement, int startIndex, Set<Sha256Hash> mempoolTxids) throws SQLException {
+        if(mempoolTxids == null) {
+            return startIndex;
+        }
+        Object[] arr = mempoolTxids.stream().map(Sha256Hash::getBytes).toArray();
+        Array array = statement.getConnection().createArrayOf("BLOB", arr);
+        statement.setArray(startIndex, array);
+        return startIndex + 1;
+    }
+
+    private long getInputRowCount(Connection connection, Integer startHeight, Integer endHeight) throws SQLException {
+        String sql = "SELECT COUNT(*) FROM " + TWEAK_TABLE + buildWhereClause(startHeight, endHeight, null);
+        try(PreparedStatement countStmt = connection.prepareStatement(sql)) {
+            bindTweakHeightFilter(countStmt, 1, startHeight, endHeight);
+            ResultSet rs = countStmt.executeQuery();
+            return rs.next() ? rs.getLong(1) : 0L;
+        }
+    }
+
+    private static ComputeBackend resolveBackend(boolean isHistorical) {
+        if(!isHistorical) {
+            return ComputeBackend.CPU;
+        }
+
+        return Config.get().getScan().getComputeBackendEnum();
     }
 
     private static boolean isUnsubscribed(SilentPaymentScanAddress scanAddress, WeakReference<SubscriptionStatus> subscriptionStatusRef) {

src/main/java/com/sparrowwallet/frigate/index/IndexQuerier.java

@@ -5,8 +5,14 @@ import com.sparrowwallet.drongo.protocol.Sha256Hash;
 import com.sparrowwallet.drongo.silentpayments.SilentPaymentScanAddress;
 import com.sparrowwallet.frigate.Frigate;
 import com.sparrowwallet.frigate.SubscriptionStatus;
+import com.sparrowwallet.frigate.electrum.SilentPaymentAddressSubscription;
 import com.sparrowwallet.frigate.electrum.SilentPaymentsNotification;
 import com.sparrowwallet.frigate.electrum.SilentPaymentsSubscription;
+import com.sparrowwallet.frigate.electrum.SilentPaymentsTxEntry;
+import com.sparrowwallet.frigate.io.Config;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.lang.ref.WeakReference;
 import java.util.ArrayList;
@@ -14,17 +20,37 @@ import java.util.List;
 import java.util.Set;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BooleanSupplier;
 
 public class IndexQuerier {
+    private static final Logger log = LoggerFactory.getLogger(IndexQuerier.class);
+
     public static final double PROGRESS_COMPLETE = 1.0d;
 
     private final Index blocksIndex;
     private final Index mempoolIndex;
+    private final HistoricalScanMetrics metrics;
+    private final ScheduledExecutorService metricsExecutor;
 
     public IndexQuerier(Index blocksIndex, Index mempoolIndex) {
         this.blocksIndex = blocksIndex;
         this.mempoolIndex = mempoolIndex;
+
+        if(Config.get().getScan().isMetricsEnabled()) {
+            this.metrics = new HistoricalScanMetrics();
+            this.metricsExecutor = Executors.newSingleThreadScheduledExecutor(r -> {
+                Thread t = new ThreadFactoryBuilder().setNameFormat("IndexQueryMetrics-%d").build().newThread(r);
+                t.setDaemon(true);
+                return t;
+            });
+            this.metricsExecutor.scheduleAtFixedRate(this::emitMetrics, 1, 1, TimeUnit.HOURS);
+        } else {
+            this.metrics = null;
+            this.metricsExecutor = null;
+        }
     }
 
     private final ExecutorService queryPool = Executors.newFixedThreadPool(10, r -> {
@@ -34,36 +60,65 @@ public class IndexQuerier {
         return t;
     });
 
-    public void startHistoryScan(SilentPaymentScanAddress scanAddress, Integer startHeight, Integer endHeight, Set<Integer> labelSet, WeakReference<SubscriptionStatus> subscriptionStatusRef) {
-        startHistoryScan(scanAddress, startHeight, endHeight, labelSet, subscriptionStatusRef, true);
+    private void emitMetrics() {
+        try {
+            metrics.format(metrics.snapshotAndReset()).ifPresent(log::info);
+        } catch(Throwable t) {
+            log.warn("Failed to emit scan metrics", t);
+        }
     }
 
-    public void startHistoryScan(SilentPaymentScanAddress scanAddress, Integer startHeight, Integer endHeight, Set<Integer> labelSet, WeakReference<SubscriptionStatus> subscriptionStatusRef, boolean postIfEmpty) {
-        queryPool.submit(() -> {
-            SilentPaymentsSubscription subscription = new SilentPaymentsSubscription(scanAddress.toString(), labelSet.toArray(new Integer[0]), startHeight == null ? 0 : startHeight);
-            List<TxEntry> history = blocksIndex.getHistoryAsync(scanAddress, subscription, startHeight, endHeight, subscriptionStatusRef);
-            List<TxEntry> mempoolHistory = getMempoolHistory(scanAddress, subscriptionStatusRef, subscription);
-            history.addAll(mempoolHistory);
+    public void close() {
+        if(metricsExecutor != null) {
+            metricsExecutor.shutdownNow();
+        }
+    }
 
-            if(postIfEmpty || !history.isEmpty()) {
-                Frigate.getEventBus().post(new SilentPaymentsNotification(subscription, PROGRESS_COMPLETE, new ArrayList<>(history), subscriptionStatusRef.get()));
+    public void startHistoryScan(SilentPaymentScanAddress scanAddress, Integer startHeight, Integer endHeight, SilentPaymentAddressSubscription subscription, WeakReference<SubscriptionStatus> subscriptionStatusRef, boolean isHistorical) {
+        BooleanSupplier cancelled = subscription.captureScanCancellation();
+        queryPool.submit(() -> {
+            long startMillis = isHistorical ? System.currentTimeMillis() : 0L;
+            try {
+                SilentPaymentsSubscription notificationSubscription = new SilentPaymentsSubscription(scanAddress.toString(), subscription.getLabels().toArray(new Integer[0]), subscription.getStartHeight());
+                List<SilentPaymentsTxEntry> history = blocksIndex.getHistoryAsync(scanAddress, notificationSubscription, startHeight, endHeight, null, subscriptionStatusRef, cancelled, isHistorical);
+                List<SilentPaymentsTxEntry> mempoolHistory = getMempoolHistory(scanAddress, null, subscriptionStatusRef, notificationSubscription, cancelled);
+                history.addAll(mempoolHistory);
+                long scanDurationMillis = isHistorical ? System.currentTimeMillis() - startMillis : 0L;
+
+                boolean wasCancelled = cancelled.getAsBoolean();
+                if(!wasCancelled && (isHistorical || !history.isEmpty())) {
+                    Frigate.getEventBus().post(new SilentPaymentsNotification(notificationSubscription, PROGRESS_COMPLETE, new ArrayList<>(history), subscriptionStatusRef.get()));
+                }
+                if(!wasCancelled && isHistorical) {
+                    subscription.markHistoricalComplete();
+                    if(metrics != null) {
+                        metrics.record(history.size(), scanDurationMillis);
+                    }
+                }
+            } catch(Throwable t) {
+                log.error("History scan task failed for " + scanAddress + " (start=" + startHeight + ", end=" + endHeight + ", isHistorical=" + isHistorical + ")", t);
             }
         });
     }
 
-    public void startMempoolScan(SilentPaymentScanAddress scanAddress, Integer startHeight, Integer endHeight, Set<Integer> labelSet, WeakReference<SubscriptionStatus> subscriptionStatusRef) {
+    public void startMempoolScan(SilentPaymentScanAddress scanAddress, Integer startHeight, Integer endHeight, Set<Sha256Hash> mempoolTxids, SilentPaymentAddressSubscription subscription, WeakReference<SubscriptionStatus> subscriptionStatusRef) {
+        BooleanSupplier cancelled = subscription.captureScanCancellation();
         queryPool.submit(() -> {
-            SilentPaymentsSubscription subscription = new SilentPaymentsSubscription(scanAddress.toString(), labelSet.toArray(new Integer[0]), startHeight == null ? 0 : startHeight);
-            List<TxEntry> mempoolHistory = getMempoolHistory(scanAddress, subscriptionStatusRef, subscription);
+            try {
+                SilentPaymentsSubscription notificationSubscription = new SilentPaymentsSubscription(scanAddress.toString(), subscription.getLabels().toArray(new Integer[0]), subscription.getStartHeight());
+                List<SilentPaymentsTxEntry> mempoolHistory = getMempoolHistory(scanAddress, mempoolTxids, subscriptionStatusRef, notificationSubscription, cancelled);
 
-            if(!mempoolHistory.isEmpty()) {
-                Frigate.getEventBus().post(new SilentPaymentsNotification(subscription, PROGRESS_COMPLETE, new ArrayList<>(mempoolHistory), subscriptionStatusRef.get()));
+                if(!cancelled.getAsBoolean() && !mempoolHistory.isEmpty()) {
+                    Frigate.getEventBus().post(new SilentPaymentsNotification(notificationSubscription, PROGRESS_COMPLETE, new ArrayList<>(mempoolHistory), subscriptionStatusRef.get()));
+                }
+            } catch(Throwable t) {
+                log.error("Mempool scan task failed for " + scanAddress, t);
             }
         });
     }
 
-    private List<TxEntry> getMempoolHistory(SilentPaymentScanAddress scanAddress, WeakReference<SubscriptionStatus> subscriptionStatusRef, SilentPaymentsSubscription subscription) {
-        List<TxEntry> mempoolHistory = mempoolIndex.getHistoryAsync(scanAddress, subscription, null, null, subscriptionStatusRef);
+    private List<SilentPaymentsTxEntry> getMempoolHistory(SilentPaymentScanAddress scanAddress, Set<Sha256Hash> mempoolTxids, WeakReference<SubscriptionStatus> subscriptionStatusRef, SilentPaymentsSubscription notificationSubscription, BooleanSupplier cancelled) {
+        List<SilentPaymentsTxEntry> mempoolHistory = mempoolIndex.getHistoryAsync(scanAddress, notificationSubscription, null, null, mempoolTxids, subscriptionStatusRef, cancelled, false);
         SubscriptionStatus subscriptionStatus = subscriptionStatusRef.get();
         if(subscriptionStatus != null && subscriptionStatus.getSilentPaymentsMempoolTxids(scanAddress.toString()) != null) {
             mempoolHistory.removeIf(txEntry -> subscriptionStatus.getSilentPaymentsMempoolTxids(scanAddress.toString()).contains(Sha256Hash.wrap(txEntry.tx_hash)));

src/main/java/com/sparrowwallet/frigate/index/MemoryDbManager.java

@@ -63,8 +63,11 @@ public class MemoryDbManager implements DbManager {
 
         Properties duckDbProperties = new Properties();
         duckDbProperties.setProperty("allow_unsigned_extensions", "true");
-        if(Config.get().getDbThreads() != null) {
-            duckDbProperties.setProperty("threads", Config.get().getDbThreads().toString());
+        if(Config.get().getScan().getDbThreads() != null) {
+            duckDbProperties.setProperty("threads", Config.get().getScan().getDbThreads().toString());
+        }
+        if(Config.get().getScan().getMemoryLimit() != null) {
+            duckDbProperties.setProperty("memory_limit", Config.get().getScan().getMemoryLimit());
         }
 
         connection = DriverManager.getConnection(DbManager.DB_PREFIX + "memory:", duckDbProperties);

src/main/java/com/sparrowwallet/frigate/index/SingleDbManager.java

@@ -31,6 +31,7 @@ public class SingleDbManager extends AbstractDbManager {
         this.inWriteMode = false;
     }
 
+    //Operations passed to executeRead must not await an operation that itself calls executeRead on another thread
     public <T> T executeRead(DbManager.ReadOperation<T> operation) throws SQLException, InterruptedException {
         if(shutdown) {
             throw new SQLException("Connection manager is shutting down");

src/main/java/com/sparrowwallet/frigate/io/Config.java

@@ -1,236 +1,689 @@
 package com.sparrowwallet.frigate.io;
 
-import com.google.gson.*;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonSetter;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.toml.TomlMapper;
+import com.sparrowwallet.frigate.ConfigurationException;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.*;
-import java.lang.reflect.Type;
+import java.nio.file.Files;
+import java.util.ArrayList;
 import java.util.List;
 
+@JsonIgnoreProperties(ignoreUnknown = true)
+@JsonInclude(JsonInclude.Include.NON_EMPTY)
 public class Config {
     private static final Logger log = LoggerFactory.getLogger(Config.class);
 
-    public static final String CONFIG_FILENAME = "config";
+    public static final String TOML_CONFIG_FILENAME = "config.toml";
+    public static final String JSON_CONFIG_FILENAME = "config";
 
-    private Server coreServer;
-    private CoreAuthType coreAuthType;
-    private File coreDataDir;
-    private String coreAuth;
-    private Boolean startIndexing;
-    private Integer indexStartHeight;
-    private Integer scriptPubKeyCacheSize;
-    private Integer dbThreads;
-    private String dbUrl;
-    private List<String> readDbUrls;
-    private int batchSize = 300000;
-    private ComputeBackend computeBackend;
-    private Server backendElectrumServer;
+    private CoreConfig core;
+    private IndexConfig index;
+    private ScanConfig scan;
+    private ServerConfig server;
+    private DatabaseConfig database;
 
     private static Config INSTANCE;
 
-    private static Gson getGson() {
-        GsonBuilder gsonBuilder = new GsonBuilder();
-        gsonBuilder.registerTypeAdapter(File.class, new FileSerializer());
-        gsonBuilder.registerTypeAdapter(File.class, new FileDeserializer());
-        gsonBuilder.registerTypeAdapter(Server.class, new ServerSerializer());
-        gsonBuilder.registerTypeAdapter(Server.class, new ServerDeserializer());
-        return gsonBuilder.setPrettyPrinting().disableHtmlEscaping().create();
+    private static final TomlMapper tomlMapper = TomlMapper.builder().disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES).build();
+
+    public Config() {
+        core = new CoreConfig();
+        index = new IndexConfig();
+        scan = new ScanConfig();
+        server = new ServerConfig();
+        database = new DatabaseConfig();
     }
 
-    private static File getConfigFile() {
-        File sparrowDir = Storage.getFrigateDir();
-        return new File(sparrowDir, CONFIG_FILENAME);
+    private static File getTomlConfigFile() {
+        return new File(Storage.getFrigateDir(), TOML_CONFIG_FILENAME);
+    }
+
+    private static File getJsonConfigFile() {
+        return new File(Storage.getFrigateDir(), JSON_CONFIG_FILENAME);
     }
 
     private static Config load() {
-        File configFile = getConfigFile();
-        if(configFile.exists()) {
-            try {
-                Reader reader = new FileReader(configFile);
-                Config config = getGson().fromJson(reader, Config.class);
-                reader.close();
+        File tomlFile = getTomlConfigFile();
+        File jsonFile = getJsonConfigFile();
 
+        if(tomlFile.exists()) {
+            try {
+                Config config = tomlMapper.readValue(tomlFile, Config.class);
                 if(config != null) {
                     return config;
                 }
             } catch(Exception e) {
-                log.error("Error opening " + configFile.getAbsolutePath(), e);
-                //Ignore and assume no config
+                log.error("Error reading " + tomlFile.getAbsolutePath(), e);
+            }
+        } else if(jsonFile.exists()) {
+            try {
+                Config config = migrateFromJson(jsonFile);
+                if(config != null) {
+                    saveToml(config, tomlFile);
+                    File backupFile = new File(jsonFile.getPath() + ".bak");
+                    jsonFile.renameTo(backupFile);
+                    log.info("Migrated config from JSON to TOML (backup: {})", backupFile.getName());
+                    return config;
+                }
+            } catch(Exception e) {
+                log.error("Error migrating " + jsonFile.getAbsolutePath(), e);
+            }
+        } else {
+            try {
+                writeDefaultConfig(tomlFile);
+            } catch(Exception e) {
+                log.error("Error writing default config", e);
             }
         }
 
         return new Config();
     }
 
+    private static Config migrateFromJson(File jsonFile) throws IOException {
+        ObjectMapper jsonMapper = new ObjectMapper();
+        jsonMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+        JsonNode root = jsonMapper.readTree(jsonFile);
+
+        Config config = new Config();
+
+        if(root.has("coreServer")) {
+            config.getCore().setServer(root.get("coreServer").asText());
+        }
+        if(root.has("coreAuthType")) {
+            config.getCore().setAuthType(root.get("coreAuthType").asText());
+        }
+        if(root.has("coreDataDir")) {
+            config.getCore().setDataDir(root.get("coreDataDir").asText());
+        }
+        if(root.has("coreAuth")) {
+            config.getCore().setAuth(root.get("coreAuth").asText());
+        }
+        if(root.has("startIndexing")) {
+            config.getCore().setConnect(root.get("startIndexing").asBoolean());
+        }
+        if(root.has("indexStartHeight")) {
+            config.getIndex().setStartHeight(root.get("indexStartHeight").asInt());
+        }
+        if(root.has("scriptPubKeyCacheSize")) {
+            int oldSize = root.get("scriptPubKeyCacheSize").asInt();
+            config.getIndex().setCacheSize(formatCacheSize(oldSize));
+        }
+
+        if(root.has("batchSize")) {
+            config.getScan().setBatchSize(root.get("batchSize").asInt());
+        }
+        if(root.has("computeBackend")) {
+            config.getScan().setComputeBackend(root.get("computeBackend").asText());
+        }
+        if(root.has("dbThreads")) {
+            config.getScan().setDbThreads(root.get("dbThreads").asInt());
+        }
+
+        if(root.has("backendElectrumServer")) {
+            config.getServer().setBackendElectrumServer(root.get("backendElectrumServer").asText());
+        }
+
+        if(root.has("dbUrl")) {
+            config.getDatabase().setUrl(root.get("dbUrl").asText());
+        }
+        if(root.has("readDbUrls")) {
+            List<String> urls = new java.util.ArrayList<>();
+            root.get("readDbUrls").forEach(node -> urls.add(node.asText()));
+            config.getDatabase().setReadUrls(urls);
+        }
+
+        return config;
+    }
+
+    private static String formatCacheSize(int size) {
+        if(size >= 1_000_000 && size % 1_000_000 == 0) {
+            return (size / 1_000_000) + "M";
+        } else if(size >= 1_000 && size % 1_000 == 0) {
+            return (size / 1_000) + "K";
+        }
+        return String.valueOf(size);
+    }
+
+    static int parseCacheSize(String value) {
+        if(value == null || value.isEmpty()) {
+            return 10_000_000;
+        }
+        String s = value.trim().toUpperCase();
+        if(s.endsWith("M")) {
+            return (int) (Double.parseDouble(s.substring(0, s.length() - 1)) * 1_000_000);
+        } else if(s.endsWith("K")) {
+            return (int) (Double.parseDouble(s.substring(0, s.length() - 1)) * 1_000);
+        }
+        return Integer.parseInt(s);
+    }
+
+    private static void writeDefaultConfig(File tomlFile) {
+        try(InputStream is = Config.class.getResourceAsStream("/config.toml.default")) {
+            if(is != null) {
+                Storage.createOwnerOnlyFile(tomlFile);
+                Files.copy(is, tomlFile.toPath(), java.nio.file.StandardCopyOption.REPLACE_EXISTING);
+            }
+        } catch(IOException e) {
+            log.error("Error writing default config", e);
+        }
+    }
+
+    private static void saveToml(Config config, File tomlFile) {
+        try {
+            if(!tomlFile.exists()) {
+                Storage.createOwnerOnlyFile(tomlFile);
+            }
+            tomlMapper.writeValue(tomlFile, config);
+        } catch(IOException e) {
+            log.error("Error writing config", e);
+        }
+    }
+
     public static synchronized Config get() {
         if(INSTANCE == null) {
             INSTANCE = load();
+            INSTANCE.getServer().getAdvertisedHosts();
         }
-
         return INSTANCE;
     }
 
-    public Server getCoreServer() {
-        return coreServer;
+    public CoreConfig getCore() {
+        if(core == null) {
+            core = new CoreConfig();
+        }
+        return core;
     }
 
-    public void setCoreServer(Server coreServer) {
-        this.coreServer = coreServer;
-        flush();
+    public IndexConfig getIndex() {
+        if(index == null) {
+            index = new IndexConfig();
+        }
+        return index;
     }
 
-    public CoreAuthType getCoreAuthType() {
-        return coreAuthType;
+    public ScanConfig getScan() {
+        if(scan == null) {
+            scan = new ScanConfig();
+        }
+        return scan;
     }
 
-    public void setCoreAuthType(CoreAuthType coreAuthType) {
-        this.coreAuthType = coreAuthType;
-        flush();
+    public ServerConfig getServer() {
+        if(server == null) {
+            server = new ServerConfig();
+        }
+        return server;
     }
 
-    public File getCoreDataDir() {
-        return coreDataDir;
+    public DatabaseConfig getDatabase() {
+        if(database == null) {
+            database = new DatabaseConfig();
+        }
+        return database;
     }
 
-    public void setCoreDataDir(File coreDataDir) {
-        this.coreDataDir = coreDataDir;
-        flush();
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    @JsonInclude(JsonInclude.Include.NON_NULL)
+    public static class CoreConfig {
+        public static final int DEFAULT_RPC_REQUEST_TIMEOUT_SECONDS = 60;
+        public static final int DEFAULT_RPC_BATCH_SIZE = 100;
+
+        private Boolean connect;
+        private String server;
+        private String authType;
+        private String dataDir;
+        private String auth;
+        private String zmqSequenceEndpoint;
+        private Integer rpcRequestTimeoutSeconds;
+        private Integer rpcBatchSize;
+
+        public Boolean getConnect() {
+            return connect;
+        }
+
+        public void setConnect(Boolean connect) {
+            this.connect = connect;
+        }
+
+        @JsonIgnore
+        public boolean shouldConnect() {
+            return connect == null || connect;
+        }
+
+        public String getServer() {
+            return server;
+        }
+
+        public void setServer(String server) {
+            this.server = server;
+        }
+
+        public String getAuthType() {
+            return authType;
+        }
+
+        public void setAuthType(String authType) {
+            this.authType = authType;
+        }
+
+        @JsonIgnore
+        public CoreAuthType getAuthTypeEnum() {
+            if(authType == null) {
+                return null;
+            }
+            try {
+                return CoreAuthType.valueOf(authType);
+            } catch(Exception e) {
+                return null;
+            }
+        }
+
+        public String getDataDir() {
+            return dataDir;
+        }
+
+        public void setDataDir(String dataDir) {
+            this.dataDir = dataDir;
+        }
+
+        @JsonIgnore
+        public File getDataDirFile() {
+            return dataDir != null ? new File(dataDir) : null;
+        }
+
+        public String getAuth() {
+            return auth;
+        }
+
+        public void setAuth(String auth) {
+            this.auth = auth;
+        }
+
+        @JsonIgnore
+        public Server getServerObj() {
+            return server != null ? Server.fromString(server) : null;
+        }
+
+        public String getZmqSequenceEndpoint() {
+            return zmqSequenceEndpoint;
+        }
+
+        public void setZmqSequenceEndpoint(String zmqSequenceEndpoint) {
+            this.zmqSequenceEndpoint = zmqSequenceEndpoint;
+        }
+
+        public Integer getRpcRequestTimeoutSeconds() {
+            return rpcRequestTimeoutSeconds;
+        }
+
+        public void setRpcRequestTimeoutSeconds(Integer rpcRequestTimeoutSeconds) {
+            this.rpcRequestTimeoutSeconds = rpcRequestTimeoutSeconds;
+        }
+
+        @JsonIgnore
+        public int getRpcRequestTimeoutMillis() {
+            int seconds = rpcRequestTimeoutSeconds != null ? rpcRequestTimeoutSeconds : DEFAULT_RPC_REQUEST_TIMEOUT_SECONDS;
+            return seconds * 1000;
+        }
+
+        public Integer getRpcBatchSize() {
+            return rpcBatchSize;
+        }
+
+        public void setRpcBatchSize(Integer rpcBatchSize) {
+            this.rpcBatchSize = rpcBatchSize;
+        }
+
+        @JsonIgnore
+        public int getRpcBatchSizeValue() {
+            return rpcBatchSize != null && rpcBatchSize > 0 ? rpcBatchSize : DEFAULT_RPC_BATCH_SIZE;
+        }
     }
 
-    public String getCoreAuth() {
-        return coreAuth;
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    @JsonInclude(JsonInclude.Include.NON_NULL)
+    public static class IndexConfig {
+        private Integer startHeight;
+        private String cacheSize;
+
+        public Integer getStartHeight() {
+            return startHeight;
+        }
+
+        public void setStartHeight(Integer startHeight) {
+            this.startHeight = startHeight;
+        }
+
+        public String getCacheSize() {
+            return cacheSize;
+        }
+
+        public void setCacheSize(String cacheSize) {
+            this.cacheSize = cacheSize;
+        }
+
+        @JsonIgnore
+        public int getCacheSizeEntries() {
+            return Config.parseCacheSize(cacheSize);
+        }
     }
 
-    public void setCoreAuth(String coreAuth) {
-        this.coreAuth = coreAuth;
-        flush();
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    @JsonInclude(JsonInclude.Include.NON_NULL)
+    public static class ScanConfig {
+        public static final int DEFAULT_BATCH_SIZE = 300_000;
+        public static final int DEFAULT_MAX_LABELS = 10;
+        public static final int DEFAULT_MAX_SUBSCRIPTIONS = 100;
+
+        private Integer batchSize;
+        private String computeBackend;
+        private Integer dbThreads;
+        private String memoryLimit;
+        private Integer maxLabels;
+        private Integer maxSubscriptions;
+        private Boolean metricsEnabled;
+
+        public int getBatchSize() {
+            return batchSize != null ? batchSize : DEFAULT_BATCH_SIZE;
+        }
+
+        public void setBatchSize(int batchSize) {
+            this.batchSize = batchSize;
+        }
+
+        public int getMaxLabels() {
+            return maxLabels != null ? maxLabels : DEFAULT_MAX_LABELS;
+        }
+
+        public void setMaxLabels(Integer maxLabels) {
+            this.maxLabels = maxLabels;
+        }
+
+        public int getMaxSubscriptions() {
+            return maxSubscriptions != null ? maxSubscriptions : DEFAULT_MAX_SUBSCRIPTIONS;
+        }
+
+        public void setMaxSubscriptions(Integer maxSubscriptions) {
+            this.maxSubscriptions = maxSubscriptions;
+        }
+
+        @JsonIgnore
+        public ComputeBackend getComputeBackendEnum() {
+            if(computeBackend == null) {
+                return ComputeBackend.AUTO;
+            }
+            try {
+                return ComputeBackend.valueOf(computeBackend);
+            } catch(Exception e) {
+                return ComputeBackend.AUTO;
+            }
+        }
+
+        public String getComputeBackend() {
+            return computeBackend;
+        }
+
+        public void setComputeBackend(String computeBackend) {
+            this.computeBackend = computeBackend;
+        }
+
+        public Integer getDbThreads() {
+            return dbThreads;
+        }
+
+        public void setDbThreads(Integer dbThreads) {
+            this.dbThreads = dbThreads;
+        }
+
+        public String getMemoryLimit() {
+            return memoryLimit;
+        }
+
+        public void setMemoryLimit(String memoryLimit) {
+            this.memoryLimit = memoryLimit;
+        }
+
+        public boolean isMetricsEnabled() {
+            return metricsEnabled == null || metricsEnabled;
+        }
+
+        public void setMetricsEnabled(Boolean metricsEnabled) {
+            this.metricsEnabled = metricsEnabled;
+        }
     }
 
-    public Boolean isStartIndexing() {
-        return startIndexing;
-    }
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    @JsonInclude(JsonInclude.Include.NON_NULL)
+    public static class ServerConfig {
+        private List<String> host;
+        private String tcp;
+        private String ssl;
+        private String sslCert;
+        private String sslKey;
+        private String backendElectrumServer;
 
-    public void setStartIndexing(Boolean startIndexing) {
-        this.startIndexing = startIndexing;
-        flush();
-    }
+        @JsonIgnore
+        private List<Server> advertisedHostsCache;
 
-    public Integer getIndexStartHeight() {
-        return indexStartHeight;
-    }
+        public Object getHost() {
+            if(host == null || host.isEmpty()) {
+                return null;
+            }
+            return host.size() == 1 ? host.getFirst() : host;
+        }
 
-    public void setIndexStartHeight(Integer indexStartHeight) {
-        this.indexStartHeight = indexStartHeight;
-        flush();
-    }
-
-    public Integer getScriptPubKeyCacheSize() {
-        return scriptPubKeyCacheSize;
-    }
-
-    public void setScriptPubKeyCacheSize(Integer scriptPubKeyCacheSize) {
-        this.scriptPubKeyCacheSize = scriptPubKeyCacheSize;
-        flush();
-    }
-
-    public Integer getDbThreads() {
-        return dbThreads;
-    }
-
-    public void setDbThreads(Integer dbThreads) {
-        this.dbThreads = dbThreads;
-        flush();
-    }
-
-    public String getDbUrl() {
-        return dbUrl;
-    }
-
-    public void setDbUrl(String dbUrl) {
-        this.dbUrl = dbUrl;
-        flush();
-    }
-
-    public List<String> getReadDbUrls() {
-        return readDbUrls;
-    }
-
-    public void setReadDbUrls(List<String> readDbUrls) {
-        this.readDbUrls = readDbUrls;
-        flush();
-    }
-
-    public int getBatchSize() {
-        return batchSize;
-    }
-
-    public void setBatchSize(int batchSize) {
-        this.batchSize = batchSize;
-        flush();
-    }
-
-    public ComputeBackend getComputeBackend() {
-        return computeBackend != null ? computeBackend : ComputeBackend.AUTO;
-    }
-
-    public void setComputeBackend(ComputeBackend computeBackend) {
-        this.computeBackend = computeBackend;
-        flush();
-    }
-
-    public Server getBackendElectrumServer() {
-        return backendElectrumServer;
-    }
-
-    public void setBackendElectrumServer(Server backendElectrumServer) {
-        this.backendElectrumServer = backendElectrumServer;
-        flush();
-    }
-
-    private synchronized void flush() {
-        Gson gson = getGson();
-        try {
-            File configFile = getConfigFile();
-            if(!configFile.exists()) {
-                Storage.createOwnerOnlyFile(configFile);
+        @JsonSetter("host")
+        public void setHost(JsonNode node) {
+            if(node == null || node.isNull()) {
+                this.host = null;
+                return;
             }
 
-            Writer writer = new FileWriter(configFile);
-            gson.toJson(this, writer);
-            writer.flush();
-            writer.close();
-        } catch (IOException e) {
-            //Ignore
+            List<String> entries = new ArrayList<>();
+            if(node.isTextual()) {
+                String value = node.asText();
+                if(!value.isEmpty()) {
+                    entries.add(value);
+                }
+            } else if(node.isArray()) {
+                for(JsonNode item : node) {
+                    if(!item.isTextual()) {
+                        throw new ConfigurationException("Each entry in 'host' under [server] must be a string (got: " + item + ")");
+                    }
+                    String value = item.asText();
+                    if(!value.isEmpty()) {
+                        entries.add(value);
+                    }
+                }
+            } else {
+                throw new ConfigurationException("'host' under [server] must be a string or array of strings (got: " + node + ")");
+            }
+
+            this.host = entries.isEmpty() ? null : entries;
+        }
+
+        @JsonIgnore
+        public List<Server> getAdvertisedHosts() {
+            if(advertisedHostsCache != null) {
+                return advertisedHostsCache;
+            }
+            if(host == null || host.isEmpty()) {
+                advertisedHostsCache = List.of();
+                return advertisedHostsCache;
+            }
+
+            List<Server> result = new ArrayList<>();
+            for(String entry : host) {
+                Protocol protocol = Protocol.getProtocol(entry);
+                if(protocol == null) {
+                    Server tcpBind = getTcpServer();
+                    if(tcpBind != null) {
+                        result.add(Server.fromString(Protocol.TCP.toUrlString(entry, tcpBind.getHostAndPort().getPort())));
+                    }
+                    Server sslBind = getSslServer();
+                    if(sslBind != null) {
+                        result.add(Server.fromString(Protocol.SSL.toUrlString(entry, sslBind.getHostAndPort().getPort())));
+                    }
+                } else if(protocol == Protocol.TCP || protocol == Protocol.SSL) {
+                    result.add(parseListener("host", entry, protocol));
+                } else {
+                    throw new ConfigurationException("Host advertisement '" + entry + "' under [server] must use tcp:// or ssl:// scheme");
+                }
+            }
+
+            advertisedHostsCache = List.copyOf(result);
+            return advertisedHostsCache;
+        }
+
+        public String getTcp() {
+            return tcp;
+        }
+
+        public void setTcp(String tcp) {
+            this.tcp = tcp;
+        }
+
+        @JsonSetter("port")
+        public void setLegacyPort(Integer port) {
+            if(port != null && this.tcp == null) {
+                this.tcp = "tcp://0.0.0.0:" + port;
+            }
+        }
+
+        public String getSsl() {
+            return ssl;
+        }
+
+        public void setSsl(String ssl) {
+            this.ssl = ssl;
+        }
+
+        @JsonIgnore
+        public Server getTcpServer() {
+            if(tcp == null && ssl == null) {
+                return parseListener("tcp", "tcp://0.0.0.0:" + Protocol.TCP.getDefaultPort(), Protocol.TCP);
+            }
+            if(tcp == null || tcp.isEmpty()) {
+                return null;
+            }
+            return parseListener("tcp", tcp, Protocol.TCP);
+        }
+
+        @JsonIgnore
+        public Server getSslServer() {
+            if(ssl == null || ssl.isEmpty()) {
+                return null;
+            }
+            return parseListener("ssl", ssl, Protocol.SSL);
+        }
+
+        @JsonIgnore
+        public boolean isTcpEnabled() {
+            return getTcpServer() != null;
+        }
+
+        @JsonIgnore
+        public boolean isSslEnabled() {
+            return getSslServer() != null;
+        }
+
+        private static Server parseListener(String key, String url, Protocol expected) {
+            Server server;
+            try {
+                server = Server.fromString(url);
+            } catch(IllegalArgumentException e) {
+                throw new com.sparrowwallet.frigate.ConfigurationException("Invalid " + key + " listener URL '" + url + "' under [server] in config.toml: " + e.getMessage(), e);
+            }
+
+            if(server.getProtocol() != expected) {
+                throw new com.sparrowwallet.frigate.ConfigurationException("Listener '" + key + "' must use the " + expected.toUrlString() + " scheme (got '" + url + "')");
+            }
+
+            if(!server.getHostAndPort().hasPort()) {
+                throw new com.sparrowwallet.frigate.ConfigurationException("Listener '" + key + "' must specify an explicit port: '" + url + "'");
+            }
+
+            return server;
+        }
+
+        public static final String DEFAULT_SSL_CERT = "cert.pem";
+        public static final String DEFAULT_SSL_KEY = "key.pem";
+
+        public String getSslCert() {
+            return sslCert;
+        }
+
+        public void setSslCert(String sslCert) {
+            this.sslCert = sslCert;
+        }
+
+        @JsonIgnore
+        public File getSslCertFile() {
+            return resolveFrigateDirPath(sslCert, DEFAULT_SSL_CERT);
+        }
+
+        public String getSslKey() {
+            return sslKey;
+        }
+
+        public void setSslKey(String sslKey) {
+            this.sslKey = sslKey;
+        }
+
+        @JsonIgnore
+        public File getSslKeyFile() {
+            return resolveFrigateDirPath(sslKey, DEFAULT_SSL_KEY);
+        }
+
+        private static File resolveFrigateDirPath(String configured, String defaultName) {
+            String value = (configured == null || configured.isEmpty()) ? defaultName : configured;
+            File file = new File(value);
+            return file.isAbsolute() ? file : new File(Storage.getFrigateDir(), value);
+        }
+
+        public String getBackendElectrumServer() {
+            return backendElectrumServer;
+        }
+
+        public void setBackendElectrumServer(String backendElectrumServer) {
+            this.backendElectrumServer = backendElectrumServer;
+        }
+
+        @JsonIgnore
+        public Server getBackendElectrumServerObj() {
+            return backendElectrumServer != null ? Server.fromString(backendElectrumServer) : null;
         }
     }
 
-    private static class FileSerializer implements JsonSerializer<File> {
-        @Override
-        public JsonElement serialize(File src, Type typeOfSrc, JsonSerializationContext context) {
-            return new JsonPrimitive(src.getAbsolutePath());
-        }
-    }
+    @JsonIgnoreProperties(ignoreUnknown = true)
+    @JsonInclude(JsonInclude.Include.NON_NULL)
+    public static class DatabaseConfig {
+        private String url;
+        private List<String> readUrls;
 
-    private static class FileDeserializer implements JsonDeserializer<File> {
-        @Override
-        public File deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException {
-            return new File(json.getAsJsonPrimitive().getAsString());
+        public String getUrl() {
+            return url;
         }
-    }
 
-    private static class ServerSerializer implements JsonSerializer<Server> {
-        @Override
-        public JsonElement serialize(Server src, Type typeOfSrc, JsonSerializationContext context) {
-            return new JsonPrimitive(src.toString());
+        public void setUrl(String url) {
+            this.url = url;
         }
-    }
 
-    private static class ServerDeserializer implements JsonDeserializer<Server> {
-        @Override
-        public Server deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException {
-            return Server.fromString(json.getAsJsonPrimitive().getAsString());
+        public List<String> getReadUrls() {
+            return readUrls;
+        }
+
+        public void setReadUrls(List<String> readUrls) {
+            this.readUrls = readUrls;
         }
     }
 }
-

src/main/java/com/sparrowwallet/frigate/io/JsonRpcErrorFilter.java

@@ -0,0 +1,28 @@
+package com.sparrowwallet.frigate.io;
+
+import ch.qos.logback.classic.Level;
+import ch.qos.logback.classic.Logger;
+import ch.qos.logback.classic.turbo.TurboFilter;
+import ch.qos.logback.core.spi.FilterReply;
+import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcError;
+import org.slf4j.Marker;
+
+public class JsonRpcErrorFilter extends TurboFilter {
+    private static final String JSON_RPC_SERVER_LOGGER = "com.github.arteam.simplejsonrpc.server.JsonRpcServer";
+
+    @Override
+    public FilterReply decide(Marker marker, Logger logger, Level level, String format, Object[] params, Throwable t) {
+        if(level == Level.ERROR && t != null && JSON_RPC_SERVER_LOGGER.equals(logger.getName()) && rootCauseHasJsonRpcError(t)) {
+            return FilterReply.DENY;
+        }
+        return FilterReply.NEUTRAL;
+    }
+
+    private static boolean rootCauseHasJsonRpcError(Throwable t) {
+        Throwable cause = t;
+        while(cause.getCause() != null && cause.getCause() != cause) {
+            cause = cause.getCause();
+        }
+        return cause.getClass().isAnnotationPresent(JsonRpcError.class);
+    }
+}

src/main/java/com/sparrowwallet/frigate/io/Server.java

@@ -2,6 +2,9 @@ package com.sparrowwallet.frigate.io;
 
 import com.google.common.net.HostAndPort;
 
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.net.UnknownHostException;
 import java.util.Arrays;
 
 public class Server {
@@ -53,6 +56,11 @@ public class Server {
         return getHostAndPort().getHost();
     }
 
+    public InetSocketAddress getInetSocketAddress() throws UnknownHostException {
+        HostAndPort hostAndPort = getHostAndPort();
+        return new InetSocketAddress(InetAddress.getByName(hostAndPort.getHost()), hostAndPort.getPort());
+    }
+
     public boolean isOnionAddress() {
         return Protocol.isOnionAddress(getHostAndPort());
     }

src/main/java/com/sparrowwallet/frigate/io/SslUtil.java

@@ -0,0 +1,156 @@
+package com.sparrowwallet.frigate.io;
+
+import com.sparrowwallet.frigate.ConfigurationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.security.KeyFactory;
+import java.security.KeyStore;
+import java.security.PrivateKey;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.CertificateFactory;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.PKCS8EncodedKeySpec;
+import java.util.Base64;
+import java.util.Collection;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public final class SslUtil {
+    private static final Logger log = LoggerFactory.getLogger(SslUtil.class);
+
+    private static final Pattern PEM_BLOCK = Pattern.compile("-----BEGIN ([A-Z0-9 ]+?)-----\\s*([A-Za-z0-9+/=\\s]+?)-----END \\1-----", Pattern.DOTALL);
+    private static final List<String> KEY_FACTORY_ALGORITHMS = List.of("RSA", "EC", "DSA");
+
+    private SslUtil() {}
+
+    public static SSLSocketFactory getTrustAllSocketFactory() {
+        TrustManager[] trustAllCerts = new TrustManager[] {
+                new X509TrustManager() {
+                    public X509Certificate[] getAcceptedIssuers() {
+                        return new X509Certificate[0];
+                    }
+
+                    public void checkClientTrusted(X509Certificate[] certs, String authType) throws CertificateException {
+                    }
+
+                    public void checkServerTrusted(X509Certificate[] certs, String authType) throws CertificateException {
+                    }
+                }
+        };
+
+        try {
+            SSLContext sslContext = SSLContext.getInstance("TLS");
+            sslContext.init(null, trustAllCerts, null);
+            return sslContext.getSocketFactory();
+        } catch(Exception e) {
+            log.error("Error creating SSL socket factory", e);
+        }
+
+        return null;
+    }
+
+    public static SSLContext getServerSSLContext(File certFile, File keyFile) {
+        if(!certFile.isFile()) {
+            throw new ConfigurationException("SSL: certificate file not found: " + certFile.getAbsolutePath());
+        }
+        if(!keyFile.isFile()) {
+            throw new ConfigurationException("SSL: private key file not found: " + keyFile.getAbsolutePath());
+        }
+
+        X509Certificate[] chain = readCertificateChain(certFile);
+        PrivateKey privateKey = readPrivateKey(keyFile);
+
+        try {
+            KeyStore keyStore = KeyStore.getInstance("PKCS12");
+            keyStore.load(null, new char[0]);
+            keyStore.setKeyEntry("frigate", privateKey, new char[0], chain);
+
+            KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
+            kmf.init(keyStore, new char[0]);
+
+            SSLContext sslContext = SSLContext.getInstance("TLS");
+            sslContext.init(kmf.getKeyManagers(), null, null);
+            return sslContext;
+        } catch(Exception e) {
+            throw new ConfigurationException("SSL: failed to initialise TLS context: " + e.getMessage(), e);
+        }
+    }
+
+    private static X509Certificate[] readCertificateChain(File certFile) {
+        try(FileInputStream fis = new FileInputStream(certFile);
+            BufferedInputStream bis = new BufferedInputStream(fis)) {
+            CertificateFactory cf = CertificateFactory.getInstance("X.509");
+            Collection<? extends Certificate> certs = cf.generateCertificates(bis);
+
+            if(certs.isEmpty()) {
+                throw new ConfigurationException("SSL: no certificates found in " + certFile.getAbsolutePath());
+            }
+
+            X509Certificate[] chain = new X509Certificate[certs.size()];
+            int i = 0;
+            for(Certificate c : certs) {
+                chain[i++] = (X509Certificate)c;
+            }
+
+            return chain;
+        } catch(IOException | CertificateException e) {
+            throw new ConfigurationException("SSL: failed to parse certificate " + certFile.getAbsolutePath() + ": " + e.getMessage(), e);
+        }
+    }
+
+    private static PrivateKey readPrivateKey(File keyFile) {
+        String pem;
+        try {
+            pem = Files.readString(keyFile.toPath(), StandardCharsets.UTF_8);
+        } catch(IOException e) {
+            throw new ConfigurationException("SSL: failed to read private key " + keyFile.getAbsolutePath() + ": " + e.getMessage(), e);
+        }
+
+        Matcher m = PEM_BLOCK.matcher(pem);
+        if(!m.find()) {
+            throw new ConfigurationException("SSL: no PEM block found in " + keyFile.getAbsolutePath());
+        }
+        String label = m.group(1).trim();
+        if(!"PRIVATE KEY".equals(label)) {
+            throw new ConfigurationException("SSL: unsupported key format '" + label + "' in " + keyFile.getAbsolutePath()
+                    + ". Only unencrypted PKCS#8 ('-----BEGIN PRIVATE KEY-----') is supported. Convert with: "
+                    + "openssl pkcs8 -topk8 -nocrypt -in <key> -out <pkcs8-key>");
+        }
+
+        byte[] der;
+        try {
+            der = Base64.getMimeDecoder().decode(m.group(2));
+        } catch(IllegalArgumentException e) {
+            throw new ConfigurationException("SSL: malformed base64 in private key " + keyFile.getAbsolutePath(), e);
+        }
+
+        PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(der);
+        InvalidKeySpecException lastException = null;
+        for(String algorithm : KEY_FACTORY_ALGORITHMS) {
+            try {
+                return KeyFactory.getInstance(algorithm).generatePrivate(spec);
+            } catch(InvalidKeySpecException e) {
+                lastException = e;
+            } catch(Exception e) {
+                throw new ConfigurationException("SSL: failed to load private key " + keyFile.getAbsolutePath() + ": " + e.getMessage(), e);
+            }
+        }
+
+        throw new ConfigurationException("SSL: unrecognised private key algorithm in " + keyFile.getAbsolutePath() + " (tried " + KEY_FACTORY_ALGORITHMS + ")", lastException);
+    }
+}

src/main/java/module-info.java

@@ -2,19 +2,26 @@ module com.sparrowwallet.frigate {
     requires com.sparrowwallet.drongo;
     requires duckdb.jdbc;
     requires com.fasterxml.jackson.annotation;
+    requires com.fasterxml.jackson.databind;
+    requires com.fasterxml.jackson.dataformat.toml;
     requires simple.json.rpc.core;
     requires simple.json.rpc.client;
     requires simple.json.rpc.server;
     requires com.google.gson;
     requires com.google.common;
     requires org.jcommander;
+    requires org.zeromq.jeromq;
     requires org.slf4j;
+    requires ch.qos.logback.core;
+    requires ch.qos.logback.classic;
     requires java.sql;
+    requires static java.desktop;
     exports com.sparrowwallet.frigate;
     exports com.sparrowwallet.frigate.io;
     exports com.sparrowwallet.frigate.bitcoind;
     exports com.sparrowwallet.frigate.electrum;
     exports com.sparrowwallet.frigate.index;
     exports com.sparrowwallet.frigate.cli;
-    opens com.sparrowwallet.frigate.io;
+    opens com.sparrowwallet.frigate.control to com.google.common;
+    opens com.sparrowwallet.frigate.io to com.fasterxml.jackson.databind;
 }

src/main/resources/config.toml.default

@@ -0,0 +1,36 @@
+# Frigate configuration
+# See https://github.com/sparrowwallet/frigate for documentation
+
+[core]
+connect = true
+# server = "http://127.0.0.1:8332"
+# authType = "COOKIE"            # COOKIE or USERPASS
+# dataDir = "/home/bitcoin/.bitcoin"
+# auth = "user:password"         # only needed for USERPASS
+# zmqSequenceEndpoint = "tcp://127.0.0.1:28336"   # bitcoind -zmqpubsequence endpoint for low-latency mempool ingestion
+# rpcRequestTimeoutSeconds = 60                   # per-RPC read timeout (raise on slow/remote bitcoind, lower on fast LAN)
+# rpcBatchSize = 100                              # max sub-requests per JSON-RPC array batch (mempool fill)
+
+[index]
+# startHeight = 0                # default: 709632 on mainnet (Taproot activation), 0 on testnet
+# cacheSize = "10M"              # scriptPubKey cache entries (default: 10M, ~4GB RAM)
+
+[scan]
+# batchSize = 300000             # rows per GPU dispatch (reduce if scanning hangs on older GPUs)
+# computeBackend = "AUTO"        # AUTO, GPU, or CPU (AUTO prefers GPU over CPU)
+# dbThreads = 4                  # limit DuckDB threads (reduces CPU load when computeBackend = "CPU")
+# memoryLimit = "8GB"            # cap DuckDB memory usage (default: 80% of system RAM)
+# maxLabels = 10                 # maximum number of labels accepted per silent payments subscription
+# maxSubscriptions = 100         # maximum number of silent payments subscriptions per connection
+
+[server]
+# host = "ssl://xyz.com:50002"   # advertised in server.features; use array for multiple. Omit to advertise nothing.
+# tcp = "tcp://0.0.0.0:50001"    # plaintext listener bind URL; omit (or set to "") to disable. Default if neither tcp nor ssl is set.
+# ssl = "ssl://0.0.0.0:50002"    # SSL listener bind URL; omit to disable
+# sslCert = "cert.pem"           # PEM certificate (chain allowed); bare filename resolves under Frigate's home dir, or use an absolute path
+# sslKey  = "key.pem"            # PEM-encoded PKCS#8 private key; bare filename resolves under Frigate's home dir, or use an absolute path
+# backendElectrumServer = "tcp://localhost:60001"   # backend must listen on a port distinct from Frigate's tcp/ssl listeners above
+
+# [database]
+# url = "jdbc:duckdb:/custom/path/frigate.duckdb"
+# readUrls = ["jdbc:duckdb:/replica1/frigate.duckdb"]

src/main/resources/logback.xml

@@ -1,6 +1,8 @@
 <configuration>
     <statusListener class="ch.qos.logback.core.status.NopStatusListener" />
 
+    <turboFilter class="com.sparrowwallet.frigate.io.JsonRpcErrorFilter" />
+
     <logger name="sun.net.www.protocol.http.HttpURLConnection" level="INFO" />
     <logger name="com.github.arteam.simplejsonrpc.server.JsonRpcServer" level="INFO" />
 

src/test/java/com/sparrowwallet/frigate/bitcoind/VerboseBlockTest.java

@@ -0,0 +1,128 @@
+package com.sparrowwallet.frigate.bitcoind;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class VerboseBlockTest {
+    private static final String V3_JSON = """
+            {
+              "hash": "0000000000000000000111111111111111111111111111111111111111111111",
+              "confirmations": 1,
+              "height": 800000,
+              "time": 1690000000,
+              "previousblockhash": "0000000000000000000222222222222222222222222222222222222222222222",
+              "tx": [
+                {
+                  "txid": "aaaa000000000000000000000000000000000000000000000000000000000000",
+                  "hash": "aaaa000000000000000000000000000000000000000000000000000000000000",
+                  "hex": "01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff03520101ffffffff0100000000000000000000000000",
+                  "vin": [
+                    {
+                      "coinbase": "520101",
+                      "sequence": 4294967295
+                    }
+                  ],
+                  "vout": [
+                    {
+                      "value": 0.0,
+                      "n": 0,
+                      "scriptPubKey": {
+                        "asm": "",
+                        "hex": "",
+                        "type": "nonstandard"
+                      }
+                    }
+                  ]
+                },
+                {
+                  "txid": "bbbb000000000000000000000000000000000000000000000000000000000000",
+                  "hex": "0200000001cccc00000000000000000000000000000000000000000000000000000000000000000000ffffffff0100000000000000002251200000000000000000000000000000000000000000000000000000000000000abc00000000",
+                  "vin": [
+                    {
+                      "txid": "cccc000000000000000000000000000000000000000000000000000000000000",
+                      "vout": 0,
+                      "scriptSig": {"asm": "", "hex": ""},
+                      "sequence": 4294967295,
+                      "prevout": {
+                        "generated": false,
+                        "height": 799999,
+                        "value": 0.001,
+                        "scriptPubKey": {
+                          "asm": "OP_1 abcd",
+                          "hex": "5120000000000000000000000000000000000000000000000000000000000000abcd",
+                          "address": "bc1pdummy",
+                          "type": "witness_v1_taproot"
+                        }
+                      }
+                    }
+                  ],
+                  "vout": [
+                    {
+                      "value": 0.0009,
+                      "n": 0,
+                      "scriptPubKey": {
+                        "hex": "5120000000000000000000000000000000000000000000000000000000000000abc0",
+                        "type": "witness_v1_taproot"
+                      }
+                    }
+                  ]
+                }
+              ]
+            }
+            """;
+
+    @Test
+    public void deserialisesV3Block() throws Exception {
+        ObjectMapper mapper = new ObjectMapper();
+        VerboseBlock vb = mapper.readValue(V3_JSON, VerboseBlock.class);
+
+        assertEquals("0000000000000000000111111111111111111111111111111111111111111111", vb.hash());
+        assertEquals(800000, vb.height());
+        assertEquals(1690000000L, vb.time());
+        assertEquals(2, vb.tx().size());
+    }
+
+    @Test
+    public void coinbaseInputHasNoTxid() throws Exception {
+        ObjectMapper mapper = new ObjectMapper();
+        VerboseBlock vb = mapper.readValue(V3_JSON, VerboseBlock.class);
+
+        VerboseBlock.VerboseTransaction coinbase = vb.tx().get(0);
+        VerboseBlock.VerboseVin vin = coinbase.vin().get(0);
+        assertTrue(vin.isCoinbase());
+        assertNull(vin.txid());
+        assertNull(vin.prevout());
+    }
+
+    @Test
+    public void nonCoinbaseInputCarriesPrevoutScriptPubKey() throws Exception {
+        ObjectMapper mapper = new ObjectMapper();
+        VerboseBlock vb = mapper.readValue(V3_JSON, VerboseBlock.class);
+
+        VerboseBlock.VerboseTransaction spending = vb.tx().get(1);
+        VerboseBlock.VerboseVin vin = spending.vin().get(0);
+        assertFalse(vin.isCoinbase());
+        assertEquals("cccc000000000000000000000000000000000000000000000000000000000000", vin.txid());
+        assertEquals(0, vin.voutIndex().intValue());
+        assertNotNull(vin.prevout());
+        assertNotNull(vin.prevout().scriptPubKey());
+        assertEquals("5120000000000000000000000000000000000000000000000000000000000000abcd", vin.prevout().scriptPubKey().hex());
+    }
+
+    @Test
+    public void outputCarriesScriptPubKey() throws Exception {
+        ObjectMapper mapper = new ObjectMapper();
+        VerboseBlock vb = mapper.readValue(V3_JSON, VerboseBlock.class);
+
+        VerboseBlock.VerboseTransaction spending = vb.tx().get(1);
+        VerboseBlock.VerboseVout vout = spending.vout().get(0);
+        assertEquals(0, vout.n());
+        assertEquals("5120000000000000000000000000000000000000000000000000000000000000abc0", vout.scriptPubKey().hex());
+    }
+}

src/test/java/com/sparrowwallet/frigate/electrum/AdvertisedHostsTest.java

@@ -0,0 +1,43 @@
+package com.sparrowwallet.frigate.electrum;
+
+import com.sparrowwallet.frigate.io.Server;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+
+public class AdvertisedHostsTest {
+    @Test
+    public void emptyInputYieldsEmptyMap() {
+        Map<String, ServerFeatures.HostInfo> hosts = ElectrumServerService.buildAdvertisedHosts(List.of());
+        Assertions.assertTrue(hosts.isEmpty());
+    }
+
+    @Test
+    public void tcpAndSslOnSameHostMerge() {
+        Map<String, ServerFeatures.HostInfo> hosts = ElectrumServerService.buildAdvertisedHosts(List.of(Server.fromString("tcp://example.com:50001"), Server.fromString("ssl://example.com:50002")));
+        Assertions.assertEquals(1, hosts.size());
+        ServerFeatures.HostInfo info = hosts.get("example.com");
+        Assertions.assertEquals(50001, info.tcp_port());
+        Assertions.assertEquals(50002, info.ssl_port());
+    }
+
+    @Test
+    public void differentHostsRemainSeparate() {
+        Map<String, ServerFeatures.HostInfo> hosts = ElectrumServerService.buildAdvertisedHosts(List.of(Server.fromString("tcp://example.com:50001"), Server.fromString("ssl://onion.example:443")));
+        Assertions.assertEquals(2, hosts.size());
+        Assertions.assertEquals(50001, hosts.get("example.com").tcp_port());
+        Assertions.assertNull(hosts.get("example.com").ssl_port());
+        Assertions.assertEquals(443, hosts.get("onion.example").ssl_port());
+        Assertions.assertNull(hosts.get("onion.example").tcp_port());
+    }
+
+    @Test
+    public void singleProtocolLeavesOtherNull() {
+        Map<String, ServerFeatures.HostInfo> hosts = ElectrumServerService.buildAdvertisedHosts(List.of(Server.fromString("ssl://example.com:443")));
+        ServerFeatures.HostInfo info = hosts.get("example.com");
+        Assertions.assertNull(info.tcp_port());
+        Assertions.assertEquals(443, info.ssl_port());
+    }
+}

src/test/java/com/sparrowwallet/frigate/electrum/ServerFeaturesTest.java

@@ -0,0 +1,33 @@
+package com.sparrowwallet.frigate.electrum;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class ServerFeaturesTest {
+    private static final ObjectMapper MAPPER = new ObjectMapper();
+
+    @Test
+    public void hostInfoDeserializesObjectShape() throws Exception {
+        ServerFeatures.HostInfo info = MAPPER.readValue("{\"tcp_port\":50001,\"ssl_port\":50002}", ServerFeatures.HostInfo.class);
+        Assertions.assertEquals(50001, info.tcp_port());
+        Assertions.assertEquals(50002, info.ssl_port());
+    }
+
+    @Test
+    public void hostInfoDeserializesBareIntegerAsTcpPort() throws Exception {
+        ServerFeatures.HostInfo info = MAPPER.readValue("40001", ServerFeatures.HostInfo.class);
+        Assertions.assertEquals(40001, info.tcp_port());
+        Assertions.assertNull(info.ssl_port());
+    }
+
+    @Test
+    public void serverFeaturesDeserializesMixedHostsMap() throws Exception {
+        String json = "{\"hosts\":{\"a.example.com\":{\"tcp_port\":50001,\"ssl_port\":50002},\"b.example.com\":40001}}";
+        ServerFeatures features = MAPPER.readValue(json, ServerFeatures.class);
+        Assertions.assertEquals(50001, features.hosts().get("a.example.com").tcp_port());
+        Assertions.assertEquals(50002, features.hosts().get("a.example.com").ssl_port());
+        Assertions.assertEquals(40001, features.hosts().get("b.example.com").tcp_port());
+        Assertions.assertNull(features.hosts().get("b.example.com").ssl_port());
+    }
+}

src/test/java/com/sparrowwallet/frigate/electrum/SilentPaymentsTxEntryTest.java

@@ -0,0 +1,24 @@
+package com.sparrowwallet.frigate.electrum;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class SilentPaymentsTxEntryTest {
+    @Test
+    public void serializedFormHasNoFeeField() throws Exception {
+        SilentPaymentsTxEntry entry = new SilentPaymentsTxEntry(800000, "abcd", "0011");
+        String json = new ObjectMapper().writeValueAsString(entry);
+        Assertions.assertFalse(json.contains("\"fee\""));
+        Assertions.assertTrue(json.contains("\"tweak_key\""));
+        Assertions.assertTrue(json.contains("\"tx_hash\""));
+        Assertions.assertTrue(json.contains("\"height\""));
+    }
+
+    @Test
+    public void nullTweakKeyIsOmitted() throws Exception {
+        SilentPaymentsTxEntry entry = new SilentPaymentsTxEntry(800000, "abcd", null);
+        String json = new ObjectMapper().writeValueAsString(entry);
+        Assertions.assertFalse(json.contains("\"tweak_key\""));
+    }
+}

src/test/java/com/sparrowwallet/frigate/electrum/TxEntryTest.java

@@ -0,0 +1,22 @@
+package com.sparrowwallet.frigate.electrum;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TxEntryTest {
+    @Test
+    public void scripthashEntryHasNoTweakKeyField() throws Exception {
+        TxEntry entry = new TxEntry(800000, 0, "abcd");
+        String json = new ObjectMapper().writeValueAsString(entry);
+        Assertions.assertFalse(json.contains("\"tweak_key\""));
+        Assertions.assertTrue(json.contains("\"tx_hash\""));
+    }
+
+    @Test
+    public void nullFeeIsOmitted() throws Exception {
+        TxEntry entry = new TxEntry(800000, 0, "abcd");
+        String json = new ObjectMapper().writeValueAsString(entry);
+        Assertions.assertFalse(json.contains("\"fee\""));
+    }
+}

src/test/java/com/sparrowwallet/frigate/index/HistoricalScanMetricsTest.java

@@ -0,0 +1,147 @@
+package com.sparrowwallet.frigate.index;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class HistoricalScanMetricsTest {
+    @Test
+    public void resultBucketBoundaries() {
+        assertEquals(0, HistoricalScanMetrics.resultBucket(0));
+        assertEquals(1, HistoricalScanMetrics.resultBucket(1));
+        assertEquals(1, HistoricalScanMetrics.resultBucket(10));
+        assertEquals(2, HistoricalScanMetrics.resultBucket(11));
+        assertEquals(2, HistoricalScanMetrics.resultBucket(100));
+        assertEquals(3, HistoricalScanMetrics.resultBucket(101));
+        assertEquals(3, HistoricalScanMetrics.resultBucket(1000));
+        assertEquals(4, HistoricalScanMetrics.resultBucket(1001));
+        assertEquals(4, HistoricalScanMetrics.resultBucket(10000));
+        assertEquals(5, HistoricalScanMetrics.resultBucket(10001));
+    }
+
+    //pair each sample input with the label the log line will use, so a future edit that changes a boundary without updating
+    //the corresponding label (or vice versa) fails here instead of silently producing a misleading aggregate line.
+    @Test
+    public void resultLabelsMatchBoundaries() {
+        assertEquals("0", HistoricalScanMetrics.RESULT_LABELS[HistoricalScanMetrics.resultBucket(0)]);
+        assertEquals("1-10", HistoricalScanMetrics.RESULT_LABELS[HistoricalScanMetrics.resultBucket(5)]);
+        assertEquals("11-100", HistoricalScanMetrics.RESULT_LABELS[HistoricalScanMetrics.resultBucket(50)]);
+        assertEquals("101-1000", HistoricalScanMetrics.RESULT_LABELS[HistoricalScanMetrics.resultBucket(500)]);
+        assertEquals("1001-10000", HistoricalScanMetrics.RESULT_LABELS[HistoricalScanMetrics.resultBucket(5000)]);
+        assertEquals("10001+", HistoricalScanMetrics.RESULT_LABELS[HistoricalScanMetrics.resultBucket(50000)]);
+    }
+
+    @Test
+    public void durationLabelsMatchBoundaries() {
+        assertEquals("0-100ms", HistoricalScanMetrics.DURATION_LABELS[HistoricalScanMetrics.durationBucket(50)]);
+        assertEquals("100-500ms", HistoricalScanMetrics.DURATION_LABELS[HistoricalScanMetrics.durationBucket(250)]);
+        assertEquals("500ms-2s", HistoricalScanMetrics.DURATION_LABELS[HistoricalScanMetrics.durationBucket(1000)]);
+        assertEquals("2-10s", HistoricalScanMetrics.DURATION_LABELS[HistoricalScanMetrics.durationBucket(5000)]);
+        assertEquals("10-60s", HistoricalScanMetrics.DURATION_LABELS[HistoricalScanMetrics.durationBucket(30000)]);
+        assertEquals("60s+", HistoricalScanMetrics.DURATION_LABELS[HistoricalScanMetrics.durationBucket(120000)]);
+    }
+
+    @Test
+    public void durationBucketBoundaries() {
+        assertEquals(0, HistoricalScanMetrics.durationBucket(0));
+        assertEquals(0, HistoricalScanMetrics.durationBucket(99));
+        assertEquals(1, HistoricalScanMetrics.durationBucket(100));
+        assertEquals(1, HistoricalScanMetrics.durationBucket(499));
+        assertEquals(2, HistoricalScanMetrics.durationBucket(500));
+        assertEquals(2, HistoricalScanMetrics.durationBucket(1999));
+        assertEquals(3, HistoricalScanMetrics.durationBucket(2000));
+        assertEquals(3, HistoricalScanMetrics.durationBucket(9999));
+        assertEquals(4, HistoricalScanMetrics.durationBucket(10000));
+        assertEquals(4, HistoricalScanMetrics.durationBucket(59999));
+        assertEquals(5, HistoricalScanMetrics.durationBucket(60000));
+    }
+
+    @Test
+    public void thresholdSuppressesBelowTen() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        for(int i = 0; i < 9; i++) {
+            m.record(5, 200);
+        }
+        assertTrue(m.format(m.snapshotAndReset()).isEmpty());
+    }
+
+    @Test
+    public void exactlyTenEmitsAsTen() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        for(int i = 0; i < 10; i++) {
+            m.record(5, 200);
+        }
+        String line = m.format(m.snapshotAndReset()).orElseThrow();
+        assertTrue(line.contains("1-10:10"), line);
+        assertTrue(line.contains("100-500ms:10"), line);
+    }
+
+    @Test
+    public void fourteenRoundsDownToTen() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        for(int i = 0; i < 14; i++) {
+            m.record(5, 200);
+        }
+        assertTrue(m.format(m.snapshotAndReset()).orElseThrow().contains("1-10:10"));
+    }
+
+    @Test
+    public void fifteenRoundsUpToTwenty() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        for(int i = 0; i < 15; i++) {
+            m.record(5, 200);
+        }
+        assertTrue(m.format(m.snapshotAndReset()).orElseThrow().contains("1-10:20"));
+    }
+
+    @Test
+    public void snapshotResetsCounters() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        for(int i = 0; i < 20; i++) {
+            m.record(5, 200);
+        }
+        m.snapshotAndReset();
+        assertTrue(m.format(m.snapshotAndReset()).isEmpty());
+    }
+
+    //spread result counts across all five non-zero buckets so no result bucket reaches threshold, while every duration lands in
+    //the same bucket so it does — verifies one-sided emission.
+    @Test
+    public void onlyOneHistogramEmittedWhenOnlyOneClearsThreshold() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        int[] resultsPerBucket = {0, 0, 5, 5, 50, 50, 500, 500, 5000, 5000};
+        for(int n : resultsPerBucket) {
+            m.record(n, 200);
+        }
+        String line = m.format(m.snapshotAndReset()).orElseThrow();
+        assertFalse(line.contains("results ["), line);
+        assertTrue(line.contains("duration ["), line);
+    }
+
+    @Test
+    public void emptyWhenNoRecords() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        assertTrue(m.format(m.snapshotAndReset()).isEmpty());
+    }
+
+    //mechanical guardrail: future edits to the format must not introduce summary statistics, relative timestamps, or absolute
+    //ISO timestamps. The forbidden tokens are bare (no colons) so "last value: 123ms" or "min 100ms" both trip the check.
+    @Test
+    public void formattedLineHasNoForbiddenSubstrings() {
+        HistoricalScanMetrics m = new HistoricalScanMetrics();
+        for(int i = 0; i < 20; i++) {
+            m.record(50, 1500);
+        }
+        String line = m.format(m.snapshotAndReset()).orElseThrow();
+        String lower = line.toLowerCase();
+        for(String forbidden : new String[]{"min", "max", "avg", "mean", "last", "ago"}) {
+            assertFalse(lower.contains(forbidden), "line must not contain '" + forbidden + "': " + line);
+        }
+        assertFalse(line.matches(".*\\d{4}-\\d{2}-\\d{2}.*"), "line must not contain ISO date: " + line);
+        assertFalse(line.matches(".*\\d{2}:\\d{2}:\\d{2}.*"), "line must not contain ISO time: " + line);
+    }
+}

src/test/java/com/sparrowwallet/frigate/index/SingleDbManagerTest.java

@@ -0,0 +1,63 @@
+package com.sparrowwallet.frigate.index;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class SingleDbManagerTest {
+    //regression for the three-way DB deadlock: once a writer holds the writerWaiting permit, any new executeRead parks in wait().
+    //The implicit close() in try-with-resources does shutdown+awaitTermination, which doesn't interrupt - shutdownNow is the load-bearing call.
+    @Test
+    public void testShutdownNowInterruptsExecuteReadParkedBehindWriter() throws Exception {
+        SingleDbManager dbManager = new SingleDbManager(DbManager.DB_PREFIX);
+        CountDownLatch writerHoldingPermit = new CountDownLatch(1);
+        CountDownLatch writerMayRelease = new CountDownLatch(1);
+
+        //writer takes the writerWaiting permit + write lock and holds them until the test releases it
+        Thread writer = new Thread(() -> {
+            try {
+                dbManager.executeWrite(conn -> {
+                    writerHoldingPermit.countDown();
+                    try {
+                        writerMayRelease.await();
+                    } catch(InterruptedException e) {
+                        Thread.currentThread().interrupt();
+                    }
+                    return null;
+                });
+            } catch(Exception ignored) {
+            }
+        }, "deadlock-test-writer");
+
+        try {
+            writer.start();
+            writerHoldingPermit.await();
+
+            //schedule a task that will park in SingleDbManager.executeRead.wait() (writerWaiting.availablePermits()==0)
+            ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1);
+            executor.scheduleAtFixedRate(() -> {
+                try {
+                    dbManager.executeRead(conn -> null);
+                } catch(Exception ignored) {
+                }
+            }, 0, 100, TimeUnit.MILLISECONDS);
+
+            //give the task time to fire and park
+            Thread.sleep(200);
+
+            //the load-bearing claim of the fix: shutdownNow interrupts the parked executeRead so the executor terminates promptly
+            executor.shutdownNow();
+            boolean terminated = executor.awaitTermination(2, TimeUnit.SECONDS);
+
+            assertTrue(terminated, "shutdownNow should interrupt the parked executeRead and let the executor terminate");
+        } finally {
+            writerMayRelease.countDown();
+            writer.join(2000);
+            dbManager.close();
+        }
+    }
+}

src/test/java/com/sparrowwallet/frigate/io/JsonRpcErrorFilterTest.java

@@ -0,0 +1,48 @@
+package com.sparrowwallet.frigate.io;
+
+import ch.qos.logback.classic.Level;
+import ch.qos.logback.classic.Logger;
+import ch.qos.logback.classic.LoggerContext;
+import ch.qos.logback.core.spi.FilterReply;
+import com.github.arteam.simplejsonrpc.core.annotation.JsonRpcError;
+import org.junit.jupiter.api.Test;
+import org.slf4j.LoggerFactory;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class JsonRpcErrorFilterTest {
+    @Test
+    public void filterRegisteredFromLogbackConfig() {
+        LoggerContext ctx = (LoggerContext) LoggerFactory.getILoggerFactory();
+        assertTrue(ctx.getTurboFilterList().stream().anyMatch(f -> f instanceof JsonRpcErrorFilter), "JsonRpcErrorFilter must be loaded from logback.xml");
+    }
+
+    @Test
+    public void deniesAnnotatedRootCause() {
+        JsonRpcErrorFilter filter = new JsonRpcErrorFilter();
+        Logger logger = (Logger) LoggerFactory.getLogger("com.github.arteam.simplejsonrpc.server.JsonRpcServer");
+        Throwable cause = new AnnotatedException();
+        Throwable wrapped = new RuntimeException(cause);
+        assertEquals(FilterReply.DENY, filter.decide(null, logger, Level.ERROR, "Error while processing", null, wrapped));
+    }
+
+    @Test
+    public void neutralWhenNoAnnotation() {
+        JsonRpcErrorFilter filter = new JsonRpcErrorFilter();
+        Logger logger = (Logger) LoggerFactory.getLogger("com.github.arteam.simplejsonrpc.server.JsonRpcServer");
+        Throwable wrapped = new RuntimeException(new IllegalStateException("boom"));
+        assertEquals(FilterReply.NEUTRAL, filter.decide(null, logger, Level.ERROR, "Error while processing", null, wrapped));
+    }
+
+    @Test
+    public void neutralForOtherLoggers() {
+        JsonRpcErrorFilter filter = new JsonRpcErrorFilter();
+        Logger logger = (Logger) LoggerFactory.getLogger("some.other.Logger");
+        assertEquals(FilterReply.NEUTRAL, filter.decide(null, logger, Level.ERROR, "msg", null, new AnnotatedException()));
+    }
+
+    @JsonRpcError(code = -1, message = "test")
+    private static class AnnotatedException extends Exception {
+    }
+}

src/test/java/com/sparrowwallet/frigate/io/ServerConfigHostTest.java

@@ -0,0 +1,103 @@
+package com.sparrowwallet.frigate.io;
+
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.dataformat.toml.TomlMapper;
+import com.sparrowwallet.frigate.ConfigurationException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+
+public class ServerConfigHostTest {
+    private static final TomlMapper MAPPER = TomlMapper.builder().disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES).build();
+
+    @Test
+    public void unsetHostAdvertisesNothing() throws Exception {
+        Config.ServerConfig config = parse("");
+        Assertions.assertTrue(config.getAdvertisedHosts().isEmpty());
+    }
+
+    @Test
+    public void emptyStringHostAdvertisesNothing() throws Exception {
+        Config.ServerConfig config = parse("host = \"\"");
+        Assertions.assertTrue(config.getAdvertisedHosts().isEmpty());
+    }
+
+    @Test
+    public void bareHostnameExpandsToBindPorts() throws Exception {
+        Config.ServerConfig config = parse("""
+                host = "example.com"
+                tcp = "tcp://0.0.0.0:50001"
+                ssl = "ssl://0.0.0.0:50002"
+                """);
+        List<Server> advertised = config.getAdvertisedHosts();
+        Assertions.assertEquals(2, advertised.size());
+        Assertions.assertEquals("tcp://example.com:50001", advertised.get(0).getUrl());
+        Assertions.assertEquals("ssl://example.com:50002", advertised.get(1).getUrl());
+    }
+
+    @Test
+    public void bareHostnameWithOnlyTcpBindAdvertisesTcpOnly() throws Exception {
+        Config.ServerConfig config = parse("""
+                host = "example.com"
+                tcp = "tcp://0.0.0.0:50001"
+                """);
+        List<Server> advertised = config.getAdvertisedHosts();
+        Assertions.assertEquals(1, advertised.size());
+        Assertions.assertEquals(Protocol.TCP, advertised.getFirst().getProtocol());
+        Assertions.assertEquals(50001, advertised.getFirst().getHostAndPort().getPort());
+    }
+
+    @Test
+    public void urlFormUsesExplicitPort() throws Exception {
+        Config.ServerConfig config = parse("""
+                host = "ssl://example.com:443"
+                tcp = "tcp://0.0.0.0:50001"
+                """);
+        List<Server> advertised = config.getAdvertisedHosts();
+        Assertions.assertEquals(1, advertised.size());
+        Assertions.assertEquals(Protocol.SSL, advertised.getFirst().getProtocol());
+        Assertions.assertEquals(443, advertised.getFirst().getHostAndPort().getPort());
+    }
+
+    @Test
+    public void arrayFormAdvertisesEachEntry() throws Exception {
+        Config.ServerConfig config = parse("""
+                host = ["tcp://example.com:80", "ssl://example.com:443"]
+                """);
+        List<Server> advertised = config.getAdvertisedHosts();
+        Assertions.assertEquals(2, advertised.size());
+        Assertions.assertEquals("tcp://example.com:80", advertised.get(0).getUrl());
+        Assertions.assertEquals("ssl://example.com:443", advertised.get(1).getUrl());
+    }
+
+    @Test
+    public void mixedArrayAcceptsBareAndUrl() throws Exception {
+        Config.ServerConfig config = parse("""
+                host = ["example.com", "ssl://onion.example:443"]
+                tcp = "tcp://0.0.0.0:50001"
+                """);
+        List<Server> advertised = config.getAdvertisedHosts();
+        Assertions.assertEquals(2, advertised.size());
+        Assertions.assertEquals("tcp://example.com:50001", advertised.get(0).getUrl());
+        Assertions.assertEquals("ssl://onion.example:443", advertised.get(1).getUrl());
+    }
+
+    @Test
+    public void httpSchemeIsRejected() throws Exception {
+        Config.ServerConfig config = parse("host = \"http://example.com:80\"");
+        Assertions.assertThrows(ConfigurationException.class, config::getAdvertisedHosts);
+    }
+
+    @Test
+    public void urlWithoutPortIsRejected() throws Exception {
+        Config.ServerConfig config = parse("host = \"tcp://example.com\"");
+        Assertions.assertThrows(ConfigurationException.class, config::getAdvertisedHosts);
+    }
+
+    private static Config.ServerConfig parse(String tomlBody) throws Exception {
+        String toml = "[server]\n" + tomlBody;
+        Config config = MAPPER.readValue(toml, Config.class);
+        return config.getServer();
+    }
+}