In this commit, we introduce a new ValidatePayReqAmt helper function
in the lnrpc package that consolidates the logic for validating a
caller-specified payment amount against a BOLT11 invoice amount. The
helper handles three cases: zero-amount invoices (caller must specify
an amount), fixed-amount invoices with no caller override (use invoice
amount), and fixed-amount invoices with a caller override (allow
overpayment, reject underpayment).
This helper will be used in the next commit to fix a bug where
amt_msat was silently ignored when paying fixed-amount invoices.
Previously, the SendPaymentSync path in rpcserver.go silently ignored
the amt_msat field when paying a BOLT11 invoice that already contained
a fixed amount, and the SendPaymentV2 path in router_backend.go
rejected the request outright with "amount must not be specified when
paying a non-zero amount invoice". Both behaviors prevented users from
intentionally overpaying an invoice, which is permitted by the BOLT
spec.
In this commit, we replace the duplicated amount validation logic in
both extractPaymentIntent (rpcserver.go) and
extractIntentFromSendRequest (router_backend.go) with calls to the new
ValidatePayReqAmt helper. When a caller specifies amt_msat alongside a
fixed-amount invoice, the payment now proceeds as long as the
specified amount is at least the invoice amount. Underpayment is still
rejected with a clear error message.
Fixes https://github.com/lightningnetwork/lnd/issues/10541
In this commit, we add an integration test that verifies a payer can
overpay a fixed-amount BOLT11 invoice by specifying a larger amt_msat
in the SendPaymentV2 request. The test creates a 1000 sat invoice,
pays it with 1100 sat (1,100,000 msat), and asserts that both the
sender's payment record and the receiver's invoice reflect the
overpaid amount.
To make sure we don't cause force-closures because of commit sig
mismatches, we add a test case to verify that the retransmitted HTLC
matches the original HTLC.
We update the lightning channel state machine in some key areas. If the
noop TLV is set in the update_add_htlc custom records then we change the
entry type to noop. When settling the HTLC if the type is noop we credit
the satoshi amount back to the sender.
We add a new update type to the payment descriptor to describe this new
type of htlc. This type of HTLC will only end up being set if explicitly
signalled by external software.
We check the context of the payment lifecycle at the beginning of
the `resumepayment` loop. This will make sure we have always the
latest state of the payment before deciding on the next steps in
the function `decideNextStep`.
This demonstrates that the "imported" account behaves differently for a
wallet that's watch-only vs. normal. The testTaprootImportTapscriptFullKeyFundPsbt
test case succeeds in a normal run but fails in a remote-signing setup.
For some reason, an imported tapscript address ends up in the "default"
account when remote-signing but in the "imported" account for a normal
wallet.
In this commit, we add a new atomic bool to only permit a single gossip
backlog goroutine per peer. If we get a new reuqest that needs a backlog
while we're still processing the other, then we'll drop that request.
In this commit, we add detailed documentation to help node operators
understand and configure the gossip rate limiting system effectively.
The new guide addresses a critical knowledge gap that has led to
misconfigured nodes experiencing synchronization failures.
The documentation covers the token bucket algorithm used for rate
limiting, providing clear formulas and examples for calculating
appropriate values based on node size and network position. We include
specific recommendations ranging from 50 KB/s for small nodes to
1 MB/s for large routing nodes, with detailed calculations showing
how these values are derived.
The guide explains the relationship between rate limiting and other
configuration options like num-restricted-slots and the new
filter-concurrency setting. We provide troubleshooting steps for
common issues like slow initial sync and peer disconnections, along
with debug commands and log patterns to identify problems.
Configuration examples are provided for conservative, balanced, and
performance-oriented setups, giving operators concrete starting points
they can adapt to their specific needs. The documentation emphasizes
the importance of not setting rate limits too low, warning that values
below 50 KB/s can cause synchronization to fail entirely.
In this commit, we complete the integration of the configurable gossip
filter concurrency by wiring the new FilterConcurrency configuration
through all layers of the application.
The changes connect the gossip.filter-concurrency configuration option
from the command-line interface through the server initialization code
to the gossiper and sync manager. This ensures that operators can
actually use the new configuration option to tune their node's
concurrent gossip filter processing capacity based on their specific
requirements and available resources.
In this commit, we complete the integration of the asynchronous
timestamp range queue by modifying ProcessRemoteAnnouncement to use
the new queuing mechanism instead of calling ApplyGossipFilter
synchronously.
This change ensures that when a peer sends a GossipTimestampRange
message, it is queued for asynchronous processing rather than
blocking the gossiper's main message processing loop. The modification
prevents the peer's readHandler from blocking on potentially slow
gossip filter operations, maintaining connection stability during
periods of high synchronization activity.
If the queue is full when attempting to enqueue a message, we log
a warning but return success to prevent peer disconnection. This
design choice prioritizes connection stability over guaranteed
delivery of every gossip filter request, which is acceptable since
peers can always resend timestamp range messages if needed.
In this commit, we introduce an asynchronous processing queue for
GossipTimestampRange messages in the GossipSyncer. This change addresses
a critical issue where the gossiper could block indefinitely when
processing timestamp range messages during periods of high load.
Previously, when a peer sent a GossipTimestampRange message, the
gossiper would synchronously call ApplyGossipFilter, which could block
on semaphore acquisition, database queries, and rate limiting. This
synchronous processing created a bottleneck where the entire peer
message processing pipeline would stall, potentially causing timeouts
and disconnections.
The new design adds a timestampRangeQueue channel with a capacity of 1
message and a dedicated goroutine for processing these messages
asynchronously. This follows the established pattern used for other
message types in the syncer. When the queue is full, we drop messages
and log a warning rather than blocking indefinitely, providing graceful
degradation under extreme load conditions.
In this commit, we make the gossip filter semaphore capacity configurable
through a new FilterConcurrency field. This change allows node operators
to tune the number of concurrent gossip filter applications based on
their node's resources and network position.
The previous hard-coded limit of 5 concurrent filter applications could
become a bottleneck when multiple peers attempt to synchronize
simultaneously. By making this value configurable via the new
gossip.filter-concurrency option, operators can increase this limit
for better performance on well-resourced nodes or maintain conservative
values on resource-constrained systems.
We keep the default value at 5 to maintain backward compatibility and
avoid unexpected resource usage increases for existing deployments. The
sample configuration file is updated to document this new option.
On startup, we currently blindly overwrite our node announcement info
for our source node. This includes overwriting the existing LastUpdate
timestamp. This can cause an issue with our new SQL backends in the case
that the new timestamp is not greater than the previously written
timestamp. So here, we first fetch the source node and make sure to use
a timestamp that is greater than the previosly set one.
Replace all usages of the "github.com/go-errors/errors" and
"github.com/pkg/errors" packages with the standard lib's "errors"
package. This ensures that error wrapping and `errors.Is` checks will
work as expected.
If a method returns an error, we should assume all other parameters to
be nil unless the documentation explicitly says otherwise. So here, we
fix a log line where a dereference is made to an object that will be nil
due to an error being returned.
If there's an error occured when updating the peer's status after the
channel status is changed, we now make sure we log the error instead of
letting it interrupt the channel open/close flow.
For outbound connections, since they are initialized by the users, we
can relax on the restriction. A future global limit may be added - as
for now, we will let them to be managed by the users.
When the callback is called in `scheduledPeerConnection`, it is
referencing the old `access` variable which was created when the peer
was first connected. However, if this peer opens a channel with us and
goes offline, or another inbound connection is made from this peer, we
may still use the old `access` value. To fix it, we need to make sure we
always get the fresh perm by calling `assignPeerPerms` inside
`peerConnected`.
When a peer already has a connection with us, there's no need to check
for available slots as we will either close the old conn or refuse the
new conn.
This commit removes the `batchReplayBkt` as its only effect is to allow
reforwarding htlcs during startup.
Normally for every incoming htlc added, their shared secret is used as
the key to be saved into the `sharedHashBucket`, which will be used for
check for replays. In addition, the fwdPkg's ID, which is SCID+height is
also saved to the bucket `batchReplayBkt`. Since replays of HTLCs cannot
happen at the same commitment height, when a replay happens,
`batchReplayBkt` simply doesn't have this info, and we again rely on
`sharedHashBucket` to detect it. This means most of the time the
`batchReplayBkt` is a list of SCID+height with empty values.
The `batchReplayBkt` was previously used as a mechanism to check for
reforwardings during startup - when reforwarding htlcs, it quries this
bucket and finds an empty map, knowing this is a forwarding and skips
the check in `sharedHashBucket`. Given now we use a bool flag to
explicitly skip the replay check, this bucket is no longer useful.
We now rely on the forwarding package's state to decide whether a given
packet is a reforwarding or not. If we know it's a reforwarding packet,
there's no need to check for replays in the `sharedHashes` bucket, which
behaves the same as if we are querying the `batchReplayBkt`.
In this commit, we add an option to allow a conf req caller to receive
the full block. This is useful if the caller wants to be able to create
an SPV proof.
This commit introduces a new generic type assertion function
`assertState` to the state machine tests. This function asserts that the
state machine is currently in the expected state type and returns the
state cast to that type. This allows us to directly access the fields of
the state without having to perform a type assertion manually.
In this commit, we add a new ConfMapper which is useful for state
machines that want to project some of the conf attributes into a new
event to be sent post conf.
There's no guarantee that the `b.beat` is initialized when the
dispatcher shuts down, especially with the case in the remote signer
where no chainbackend is created.
Previously when deciding whether a UTXO is spent or not, we accept a
height hint as the starting block to look for the spending tx in the
rescan process. When it's already found spent before the rescan starts,
we will update the height hint to be the spent height, only if the
latter is greater. This means if the user-specified hint is greater than
the actual spending height, this UTXO will never be found as spent. We
now fix it by always using the spent height as the hint.
We add logging to we can draw conclusions how long the processing
of gossip message last and potentially see whether the syncer
buffer channel size is a bottleneck in processing.
In this commit, we start to signal the prod bit for the rbf coop close
feature. We keep our signaling of the staging bit in place to ensure
the protocol continues to work between those nodes in the wild that are
still signaling the bit.
Fixes https://github.com/lightningnetwork/lnd/issues/9852
This commit introduces a new `IsRunning()` method to the `StateMachine`.
This method allows callers to safely query whether the state machine
is currently active after it has been started and before it has been
stopped.
To ensure thread-safety, the internal `running` status flag is
implemented using `atomic.Bool` (from `sync/atomic`). Without atomic
operations, concurrent accesses to a simple boolean flag from different
goroutines (e.g., one goroutine calling `IsRunning()` while another
executes `Start()` or `Stop()`) could lead to stale reads or data races.
Since we have not removed all call-sites that make use of this
parameter, we can remove it. This helps hide DB-specific details from
the interface we will introduce for the graph store.
In preparation for creating a clean interface for the graph store, we
want to hide anything that is DB specific from the exposed methods on
the interface. Currently the `ForEachNodeChannel` and the
`FetchOtherNode` methods of the `KVStore` expose a `kvdb.RTx` parameter
which is bbolt specific. There is only one call-site of
`ForEachNodeChannel` actually makes use of the passed `kvdb.RTx`
parameter, and that is in the `establishPersistentConnections` method of
the `server` which then passes the tx parameter to `FetchOtherNode`.
So to clean-up the interface such that the `kvdb.RTx` is no longer
exposed: we instead create one new method called
`ForEachSourceNodeChannel` which can be used to replace the above
mentioned call-site. So as of this commit, all the remaining call-site
of `ForEachNodeChannel` pass in a nil param for `kvdb.RTx` - meaning we
can remove the parameter in a future commit.
Later on we will create an interface for the persisted graph data. We
want this interface to be as small and as neat as possible. In
preparation for this, we remove this unused `Wipe` method.
The `GraphSource` interface in the `autopilot` package is directly
implemented by the `graphdb.KVStore` and so we will eventually thread
contexts through to this interface. So in this commit, we start updating
the autopilot system to thread contexts through in preparation for
passing the context through to any calls made to the GraphSource.
Two context.TODOs are added here which will be addressed in follow up
commits.
Replace all tests calls to the private `forEachNode` method on the
`KVStore` with the exported ForEachNode method. This is in preparation
for having the tests run against an abstract DB backend.
Remove the kvdb.Backend parameter from the `createChannelEdge` helper.
This is all in preparation for having the unit tests run against any DB
backend.
This commit cleans up the graph test code by removing unused kvdb type
parameters from the `createTextVertex` and `createLightningNode` helper
methods. We also pass in the testing parameter now so that we dont need
to check the error each time we call `createTestVertex`.
For any method that takes a context that has a select that listens on
the systems quit channel, we should also listen on the ctx since we
should not need to worry about if this context is derived internally or
externally.
With this, we move a context.TODO() out of the gossiper and into the
brontide package - this will be removed in a future PR which focuses on
threading contexts through that code.
The `GossiperSyncer` makes various calls to the `ChannelGraphTimeSeries`
interface which threads through to the graph DB. So in preparation for
threading context through to all the methods on that interface, we
update the GossipSyncer accordingly by passing contexts through.
Two `context.TODO()`s are added in this commit. They will be removed in
the upcoming commits.
Pass the parent LND context to the gossiper, let it derive a child
context that gets cancelled on Stop. Pass the context through to any
methods that will eventually thread it through to any graph DB calls.
One `context.TODO()` is added here - this will be removed in the next
commit.
NOTE: for any internal methods that the context gets passed to, if those
methods already listen on the gossiper's `quit` channel, then then don't
need to also listen on the passed context's Done() channel because the
quit channel is closed at the same time that the context is cancelled.
In preparation for starting to thread a single parent context through
LND, we update the main `server.Start` method to take a context so that
it can later pass it to any subsytem's Start method it calls. We also
pass the context to `newServer` since it makes some calls that will
eventually reach the DB (for example the graph db).
We want `context.TODO()` to be high signal in the code-base. It should
signal clearly that work is required to thread parent context through to
the call-site. So to keep the signal-to-noise ratio high, we remove any
context.TODO() calls from tests since these will never need to be
replace by a parent context.
After this commit, there is only a single context.TODO() left in the
code-base.
We want `context.TODO()` to be high signal in the code-base. It should
signal clearly that work is required to thread parent context through to
the call-site. So to keep the signal-to-noise ratio high, we remove any
context.TODO() calls from tests since these will never need to be
replace by a parent context.
The `lncli listchannels` command was changed and might cause breaking
changes for people relying on the `chan_id` return value in their
automation scripts.
In this commit, we add a new CLI option to control if we D/C on slow
pongs or not. Due to the existence of head-of-the-line blocking at
various levels of abstraction (app buffer, slow processing, TCP kernel
buffers, etc), if there's a flurry of gossip messages (eg: 1K channel
updates), then even with a reasonable processing latency, a peer may
still not read our ping in time.
To give users another option, we add a flag that allows users to disable
this behavior. The default remains.
In this commit, we ensure that any topology update is forced to go via
the `handleTopologySubscriptions` handler so that client subscriptions
and updates are handled correctly and in the correct order.
This removes a bug that could result from a client missing a
notification about a channel being closed if the client is subscribed
and shortly after, `PruneGraph` is called which would notify all
subscribed clients and possibly do so before the client subscription has
actually been persisted.
We remove the mutex that was previously held between DB calls and calls
that update the graphCache. This is done so that the underlying DB calls
can make use of any batch requests which they currently cannot since the
mutex prevents multiple requests from calling the methods at once.
The reason the cacheMu was originally added here was during a code
refactor that moved the `graphCache` out of the `KVStore` and into the
`ChannelGraph` and the aim was then to have a best effort way of
ensuring that updates to the DB and updates to the graphCache were as
consistent/atomic as possible.
In this commit, we update the `tlv` package version which includes type
constraints on the `tlv.SizeBigSize` method parameter. This exposes a
bug in the MilliSatoshi Record method which is fixed here.
This was not caught in tests before since currently only
our TLV encoding code makes use of this SizeFunc (so we would write 0
size to disk) but then when we read the bytes from disk and decode, we
dont use the SizeFunc and our MilliSatoshi decode method makes direct
use of the `tlv.DBigSize` function which _currently does not make use of
the `l` length variable passed to it_. So it currently does correctly
read the data.
It can take some time to unmarshal large mission control data sets such
that the macaroon can expire during that phase. We postpone the
connection establishment to give the user more time to answer the
prompt.
Mission control may have outdated success/failure amounts for node pairs
that have channels with differing capacities. In that case we assume to
still find the liquidity as before and rescale the amounts to the
according range.
We skip the evaluation of probabilities when the amount is lower than
the last success amount, as the probability would be evaluated to 1 in
that case.
If the success and fail amounts indicate that a channel doesn't obey a
bimodal distribution, we fall back to a uniform/linear success
probability model. This also helps to avoid numerical normalization
issues with the bimodal model.
This is achieved by adding a very small summand to the balance
distribution P(x) ~ exp(-x/s) + exp((x-c)/s), 1/c that helps to
regularize the probability distribution. The distribution becomes finite
for intermediate balances where the exponentials would be evaluated to
an exact zero (float) otherwise. This regularization is effective in
edge cases and leads to falling back to a uniform model should the
bimodal model fail.
This affects the normalization to be s * (-2 * exp(-c/s) + 2 + 1/s) and
the primitive function to receive an extra term x/(cs).
The previously added fuzz seed is expected to be resolved with this.
This test demonstrates an error found in a fuzz test by adding a
previously found seed, which will be fixed in an upcoming commit.
The following fuzz test is expected to fail:
go test -v -fuzz=Prob ./routing/
The bimodal model doesn't depend on the unit, which is why updating to
more realistic values doesn't require changes in tests.
We pin the scale in the fuzz test to not invalidate the corpus.
We can remove paymentFailureInfo since we can gate the result
interpretation on the source index, meaning that if we don't have a
source index, we deal with an unknown payment outcome because we
couldn't pinpoint the failing hop.
Previously we'd skip returning immature inputs to stay backwards
compatible. This has the downside as the node operator cannot take
actions on this input, like some customized batching strategy via
`bumpfee` RPC.
After this change, it means when calling `bumpfee` RPC, it will now
update the params for the immature input instead of returning an error
saying `output does not belong to wallet`.
We also add a `MaturityHeight` field so the user knows when this input
will be swept.
Make sure that an address imported to LND via ImportTapscript or ImportPublicKey
can be used as a delivery address in coop close.
New test cases:
- P2TR address imported with ImportTapscript
- P2TR address imported with ImportPublicKey
- P2WPKH address imported with ImportPublicKey
Safeguard against https://github.com/lightninglabs/loop/issues/923
More sub-tests are needed in "coop close with external delivery" itest, which
would break the limit for 50 blocks mined. Turning it into prefixed itest.
Also used new exclusion approach in test "remote signer" where all the prefixed
tests are excluded.
An address imported using ImportTapscript doesn't provide a private key
so it can't satisfy waddrmgr.ManagedPubKeyAddress interface. So we don't
return an error for imported addresses.
Fix https://github.com/lightninglabs/loop/issues/923
The MW interceptor works via a stream & sends requests to the
interceptor via this stream. Sometimes, the interceptor might want to
work with grpc metadata pairs that were sent via the context of the
original request though and so in this commit, we ensure that for each
request sent via the stream to the interceptr, the grpc metadata is sent
along explicitly as well. This is done by adding a new `metadata_pairs`
field to the RPCMiddlewareRequest proto message.
Whether we should let the aux bandwidth manager decide what the
bandwidth of a channel is should also depend on whether the HTLC is a
custom HTLC, not just the channel.
This commit adjusts the backwards compatability test in 2 ways:
1) we first set up the bitcoin backend before waiting for our network
nodes to start. This is so that the nodes can sync to chain and
become fully active during the `wait_for_nodes` call.
2) Then, in the `wait_for_node` helper, we use the status server
response to determine if a node is ready instead of just waiting for
`GetInfo` to return a response. This is because waiting for
SERVER_ACTIVE is a more accurate signal that the node is ready to
receive RPC calls.
The itest framework now handles the closure of any channels opened
during the test and so we dont need to manually close them. The manual
closure code in the route blinding tests were resulting in the
occasional test flake.
In order to get more precise bandwidth reports, we also need to provide
this method with the latest htlc view. Since aux data is committed to in
the channel commitment, some uncommited HTLCs may not be accounted for,
so we need to manually provide them via the HTLC view.
Ensure shutdown assertions through log inspection are only performed
when the harness node has an active log file handle. This avoids
errors during shutdown when log file output is disabled.
For legacy payments, the hash field will be nil, and we need to use the
payment identifier instead. We have multiple ways to fix this:
A trivial solution is we can simply call `sharder.GetHash` in
`collectResult`, and pass this hash to `attempt.Circuit()`, which ends
up multiple methods taking the hash. This is bad as it's confusing why
the methods of `HTLCAttempt` need to take another hash value, while
itself already has the info via `HTLCAttempt.Hash`. We don't want an
exceptional case to influence our main flow.
We can then patch the field `HTLCAttempt.Hash`, and set it to the
payment hash if it's nil, which can be done in `collectResult`. This is
also less optimal as it means every htlc attempts, either legacy or not,
now need to bear this context.
The best way to do this is to patch the field in
`reloadInflightAttempts`. As we are sure any new payments made won't be
legacy, and the only source of legacy payments comes from reloading
existing payments.
Adds validation to ensure that MPP parameters are compatible
with the payment amount before attempting the payment. This
prevents payments from entering a path finding loop that
would eventually timeout.
Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
In an attempt to fix the "out of disk space" build error during release
builds, we first disable using Golang caches (which can be quite large)
and then remove a bunch of pre-installed tools and their caches to
provide some additional disk storage.
In this commit, we make a change that enables the `auto_vacuum =
incremental` pragma for SQLite databases, but only when the database
file is first created. Incremental auto-vacuum allows SQLite to reclaim
unused space within the database file over time, preventing indefinite
growth.
In this commit, we test all the combinations of rbf close and taproot
chans. This ensures that the downgrade logic works properly.
Along the way we refactor the tests slightly, and also split them up, as
running all the combos back to back mines more than 50 blocks in a test,
which triggers an error in the itest sanity checks.
In this commit, we implement logic to downgrade to the legacy coop close
for taproot channels. Before this commit, we wouldn't allow nodes to
start up with both the taproot flag and the rbf flag activated.
In the future, once we implement the spec updates, we'll add support for
this combo, and can revert parts of this commit.
Previously, if a DB version wasn't available, we re-ran all schema
migrations without verifying the schema version. However, setting a
base schema version is essential because some earlier migrations were
not idempotent. This commit addresses the issue by using the current
schema version provided by sqlc as the base.
In this commit, we fix a flake in the
`TestRbfChannelFlushingTransitions/early_offer` test. The fix is simple:
this is actually an "iteration", as we have a self transition to the
ChannelNegotiation state first. We also don't need to send the
remoteOffer, so we set `sendInit` to false. The offer still needs to be
passed in to ensure that the assertions work however.
In this commit, we reduce log spam a bit during state transitions. We
only log the type of the event when sending the event, as we'll print
the same event when creating the queue "applying", and later when
processing". Next we remove the "applying" log as that first event will
always be logged twice.
The combo of these to changes makes the logs much easier to follow.
Fixes a bug introduced by #9605, fixes#9648.
We return a specific error in the RPC permission interceptor for the
case where the wallet is already unlocked or is still locked.
We need to catch those errors correctly to give the user a bit more
context on what to do.
We now return the next retry fee rate in `TxFailed` event in
`TxPublisher`. When handling the event, `UtxoSweeper` will update the
inputs to make sure the starting fee rate is set before attempting the
next sweep.
We now always create the sweeping tx even though the budget cannot be
covered so we don't miss the deadline. Note that the fee bump will fail
once the provided wallet input cannot cover the increase fees, which is
fine as these inputs will be marked as failed and be retried again in
the next block. When that happens, if there are new wallet UTXOs, a new
batch will be created to perform the fee bump.
In this commit, we revamp the old message based rate limiting. First, we
move to meter by bytes/s instead of messages/s. The old logic had an
error in that it limited groups of message replies, instead of each
message. With this new approach, we'll use the newly added
SerializedSize method to implement fine grained bandwidth metering.
We need to pick two values, the burst rate, and the msg bytes rate. The
burst rate is the max amt that can be sent in a given period of time. We
need to set this above 65 KB, or the max msg limit, otherwise no
messages can be sent. The bucket starts with this many tokens (bytes).
As those are depleted, the amount of tokens is refilled at the msg
bytes rate.
As conservative values, we've chosen 200 KB as the burst rate, and 100
KB/s as the limit.
We plan to later on add an option for a remote graph source which will
be managed from the ChannelGraph. In such a set-up, a node would rely on
the remote graph source for graph updates instead of from gossip sync.
In this scenario, however, our topology subscription logic should still
notify clients of all updates and so it makes more sense to have the
logic as part of the ChannelGraph so that we can send updates we receive
from the remote graph.
The test as it stands today does not make sense as it adds a
Partial/Shell node to the graph via AddLightningNode which will never
happen since this is only ever triggered by the gossiper which only
calls the method with a full node announcement. Shell/Partial nodes are
only ever added via AddChannelEdge which will insert a partial node if
we are adding a channel edge which has node pub keys that we dont have a
node entry for. So we adjust the test to use this more accurate flow.
We do this in preparation for moving channel cache population logic out
of the constructor and into the Start method. We also will later on
(when topology subscription is moved to the ChannelGraph), have a
goroutine that will need to be kicked off and stopped.
Here, we move the graph cache writes for AddLightningNode,
DeleteLightningNode, AddChannelEdge and MarkEdgeLive to the
ChannelGraph. Since these are writes, the cache is only updated if the
DB write is successful.
This commit moves the graph cache checks for FetchNodeFeatures,
ForEachNodeDirectedChannel, GraphSession and ForEachNodeCached from the
KVStore to the ChannelGraph. Since the ChannelGraph is currently just a
pass-through for any of the KVStore methods, all that needs to be done
for calls to go via the ChannelGraph instead directly to the KVStore is
for the ChannelGraph to go and implement those methods.
In this commit, we let the ChannelGraph be responsible for populating
the graphCache and then passing it to the KVStore. This is a first step
in moving the graphCache completely out of the KVStore layer.
And use this struct to pass NewChannelGraph anything it needs to be able
to init the KVStore that it houses. This will allow us to add
ChannelGraph specific options.
Since we have renamed a file housing some very old code, the linter has
now run on all this code for the first time. So we gotta do some
clean-up work here to make it happy.
In this commit, we rename the existing ChannelGraph struct to KVStore to
better reflect its responsibilities as the CRUD layer. We then introduce
a new ChannelGraph struct which will eventually be the layer above the
CRUD layer in which we will handle cacheing and topology subscriptions.
For now, however, it houses only the KVStore. This means that all calls
to the KVStore will now go through this layer of indirection first. This
will allow us to slowly move the graph Cache management out of the
KVStore and into the new ChannelGraph layer.
We introduce the new ChannelGraph and rename the old one in the same
commit so that all existing call-sites don't need to change at all :)
Rename it to kv_store.go so that we can re-use the graph.go file name
later on. We will use it to house the _new_ ChannelGraph when the
existing ChannelGraph is renamed to more clearly reflect its
responsibilities as the CRUD layer.
With what we added in the prior commit, we can significantly shrink the
size of this test. We also make it easier to extend in the future, as
this will fail if a new message is added, that doesn't have the needed
methods, as long as MsgEnd is updated.
In this commit, we add a new `TestMessage` interface for use in property
tests. With this, we'll be able to generate a random instance of a given
message, using the rapid byte stream. This can also eventually be useful
for fuzzing.
This commit makes sure when processing resolutions, e.g, settling
invoices, when the link is already broken, the process would exit with
an error. This fixes the issue we found in the itest, where an
unexpected empty remote pending commitment was created although the
remote peer is already offline.
In this commit, we add a new field `LockedIn` on HTLCs so it can be used
to decide whether an HTLC found on the local commitment has been
committed on the remote commitment.
This fixes an issue in the itests in the restart case. We'd see an error
like:
```
2025-03-12 23:41:10.754 [ERR] PFSM state_machine.go:661: FSM(rbf_chan_closer(2f20725d9004f7fda7ef280f77dd8d419fd6669bda1a5231dd58d6f6597066e0:0)): Unable to apply event err="invalid state transition: received *chancloser.SendOfferEvent while in ClosingNegotiation(local=LocalOfferSent(proposed_fee=0.00000193 BTC), remote=ClosePending(txid=07229915459cb439bdb8ad4f5bf112dc6f42fca0192ea16a7d6dd05e607b92ae, party=Remote, fee_rate=1 sat/vb))"
```
We resolve this by waiting to send in the new request unil the old one
has been completed.
The itest has both sides try to close multiple times, each time with
increasing fee rates. We also test the reconnection case, bad RBF
updates, and instances where the local party can't actually pay for
fees.
In this commit, we alter the existing co-op close flow to enable RBF
bumps after re connection. With the new RBF close flow, it's possible
that after a success round _and_ a re connection, either side wants to do
another fee bump. Typically we route these requests through the switch,
but in this case, the link no longer exists in the switch, so any
requests to fee bump again would find that the link doesn't exist.
In this commit, we implement a work around wherein if we have an RBF
chan closer active, and the link isn't in the switch, then we just route
the request directly to the chan closer via the peer. Once we have the
chan closer, we can use the exact same flow as prior.
We'll properly handle a protocol error due to user input by halting, and
sending the error back to the user.
When a user goes to issue a new update, based on which state we're in,
we'll either kick off the shutdown, or attempt a new offer. This matches
the new spec update where we'll only send `Shutdown` once per
connection.
In this commit, we extend `CloseChannelAssertPending` with new args that
returns the raw close status update (as we have more things we'd like to
assert), and also allows us to pass in a custom fee rate.
In this commit, we fully integrate the new RBF close state machine into
the peer.
For the restart case after shutdown, we can short circuit the existing
logic as the new FSM will handle retransmitting the shutdown message
itself, and doesn't need to delegate that duty to the link.
Unlike the existing state machine, we're able to restart the flow to
sign a coop close with a new higher fee rate. In this case, we can now
send multiple updates to the RPC caller, one for each newly singed coop
close transaction.
To implement the async flush case, we'll launch a new goroutine to wait
until the state machine reaches the `ChannelFlushing` state, then we'll
register the hook. We don't do this at start up, as otherwise the
channel may _already_ be flushed, triggering an invalid state
transition.
In this commit, we add a new composite chanCloserFsm type. This'll allow
us to store a single value that might be a negotiator or and rbf-er.
In a follow up commit, we'll use this to conditionally create the new
rbf closer.
In the next commit, we'll start checking feature bits to decide how to
init the chan closer.
In the future, we can move the current chan closer handling logic into
an `MsgEndpoint`, which'll allow us to get rid of the explicit chan
closer map and direct handling.
In this commit, we add an upfront check for `SendWhen` predicates before
deciding to launch a goroutine. This ensures that when a message comes
along that is already ready to send, we do the send in a synchronous
manner.
In this commit, we update the RBF state machine to handle early offer
cases. This can happen if after we send out shutdown (to kick things
off), the remote party sends their offer early. This can also happen if
their outgoing shutdown (to ACK ours) was delayed for w/e reason, and we
get their offer first.
The alternative was to modify the state machine itself, but we feel that
handling this early case is better in line with the Robustness principle.
In this commit, we implement a special case for OP_RETURN scripts
outlined in the spec. If a party decides that its output will be too
small even after the dust check, then they can opt to set it to zero by sending an `OP_RETURN` as their script.
In this commit, we implement the latest version of the RBF loop as
described in the spec. We remove the self loop back based on sending or
receiving shutdown. Instead, from the ClosePending state, we can trigger
a new loop by sending SendOfferEvent (we bump), or OfferReceivedEvent
(they bump).
We also update the rbf state machine w/ the new close addr logic. This
log ensures that the remote party always sends our current address, and
that if they send a new address, we'll update our view of it, and
counter sign the correct transaction.
We also add a CloseErr state. With this new state, we can ensure that
we're able to properly report errors back to the RPC client, and also
optionally force a reconnection or send a warning to the remote party.
In this commit, we update `chooseDeliveryScript` to generate a new
script if needed. This allows us to fold in a few other lines that
always followed this function into this expanded function.
The tests have been updated accordingly.
We don't return an error on broadcast fail as the broadcast might have
failed due to insufficient fees, or inability to be replaced, which may
happen when one side attempts to unnecessarily bump their coop close
fee.
In this commit, we add an implementation of a new interface the rbf coop
state machine needs. We take care to accept interfaces everywhere, to
make this easier to test, and decouple from the concrete types we'll end
up using elsewhere.
In this commit, we change LinkDirection to be a type alias. This makes
creating wrapper structs/interfaces easier, as we don't need to leak
htlcswitch.LinkDirection, instead we can accept a bool.
Previously we'd restart Alice and then restart Bob, which means once
Alice is shut down and started again before we shut down Bob, Bob will
attempt to connect Alice since the connection is permanent, which could
put the node in a weird state. We now make sure both nodes are shut down
first, then bring them back online to avoid the above case. We may,
however, create another test in the future to check the above case if needed.
In future releases we will use this flag to make potentially failing kv
to native SQL migrations optional. For this reason it is better to
rename it to avoid confusion.
The `codes.Unimplemented` is only returned when the RPCs are not built.
When the wallet is in an unexpected state, `codes.Unknown` is returned
instead, so we need to catch it properly to make sure we return the
right error msg.
This modifies the various channelnotifier notification functions
to instead hit the server and then call the notification routine.
This allows us to accurately modify the server's maps.
Here we introduce the access manager which has caches that will
determine the access control status of our peers. Peers that have
had their funding transaction confirm with us are protected. Peers
that only have pending-open channels with us are temporary access
and can have their access revoked. The rest of the peers are granted
restricted access.
We introduce a new func FetchPermAndTempPeers that returns two maps.
The first map indicates the nodes that will have "protected" access
to the server. The second map indicates the nodes that have
"temporary" access to the server. This will be used in a future
commit in the server.go code.
Previously, we would set the state of the syncer after sending the msg,
which has the following flow,
1. In state `queryNewChannels`, we send the msg `QueryShortChanIDs`.
2. Once the msg is sent, we change to state `waitingQueryChanReply`.
But there's no guarantee the remote won't reply back inbetween the two
step. When that happens, our syncer would still be in state
`queryNewChannels`, causing the following error,
```
[ERR] DISC gossiper.go:873: Process query msg from peer [Alice] got unexpected msg *lnwire.ReplyShortChanIDsEnd received in state queryNewChannels
```
To fix it, we now make sure the state is updated before sending the msg.
After merging #9581, the flake in the coverage unit test should be gone.
All we have to do is update the submodule version to the fixed one
(since during unit tests the module is used not the physical directory
on disk).
This commit adds an integration test that
verifies the funding timeout behavior in the
funding manager, in dev/integration test.
Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
Nudging test authors towards not mining too many blocks makes sense,
especially in lnd where we have a lot of integration tests.
But the lntest package is also used in other projects where this
restriction might lead to large refactors.
To be able to stage those refactors we also want to allow this limit to
be configurable if lntest is used as a library.
This commit adds a test to demonstrate that if we receive two identical
updates (which can happen if we get the same update from two peers in
quick succession), then our rate limiting logic will be hit early as
both updates might be counted towards the rate limit. This will be fixed
in an upcoming commit.
Before this commit, it was possible for a request to be sent on the
`chanWatchRequests` channel in `WaitForChannelPolicyUpdate` and then for
the `ticker.C` case to select _before_ the `eventChan` select gets
triggered when the `topologyWatcher` closes the `eventChan` in its call
to `handlePolicyUpdateWatchRequest`. This could lead to a "close of a
closed channel" panic.
To fix this, this commit ensures that we only move on to the next
iteration of the select statement in `WaitForChannelPolicyUpdate` once
the request sent on `chanWatchRequests` has been fully handled.
In this commit we document an unexpected behavior found when connecting
a bitcoind node to a btcd node. We mitigate this in our test by
reconnecting the nodes when the connection is broken. We also limit the
connection made from `bitcoind` to be v1 only.
We change how the `bitcoind` node connects to the miner by creating a
temp RPC client and use it to connect to the miner. In addition we also
assert the connection is made.
This commit adds a script to hunt flakes for a specific unit test with
trace logs. Also rename the make commands to make them more clear on
whether it's a unit test, itest, or paralleled itest.
The exposed AddNode, AddEdge and UpdateEdge methods of the Builder are
currently synchronous since even though they pass messages to the
network handler which spins off the handling in a goroutine, the public
methods still wait for a response from the handling before returning.
The only part that is actually done asynchronously is the topology
notifications.
We previously tried to simplify things in [this
commit](d757b3bcfc)
but we soon realised that there was a reason for sending the messages to
the central/synchronous network handler first: it was to ensure
consistency for topology clients: ie, the ordering between when there is
a new topology client or if it is cancelled needs to be consistent and
handled synchronously with new network updates. So for example, if a new
update comes in right after a topology client cancels its subscription,
then it should _not_ be notified. Similariy for new subscriptions. So
this commit was reverted soon after.
We can, however, still simplify things as is done in this commit by
noting that _only topology subscriptions and notifications_ need to be
handled separately. The actual network updates do not need to. So that
is what is done here.
This refactor will make moving the topology subscription logic to a new
subsystem later on much easier.
Turns out that actions/setup-go starting with @v4 also adds caching.
With that, our cache size on disk has almost doubled, leading to the
GitHub runner running out of space in certain situation.
We fix that by disabling the automated caching since we already have our
own, custom-tailored version.
For the lncli cmd we now always initiate the coop close even if
there are active HTLCs on the channel. In case HTLCs are on the
channel and the coop close is initiated LND handles the closing
flow in the background and the lncli cmd will block until the
transaction is broadcasted to the mempool. In the background LND
disallows any new HTLCs and waits until all HTLCs are resolved
before kicking of the negotiation process.
Moreover if active HTLCs are present and the no_wait param is not
set the error msg is now highlightning it so the user can react
accordingly.
Before this commit, the only error returned from `IsOurTx` is when the
root bucket was not created. In that case, we should consider the tx to
be not found in our db, since technically our db is empty.
A future PR may consider treating our wallet as the single source of
truth and query the wallet instead to check for past sweeping txns.
Previously, when a given input is found spent in the mempool, we'd mark
it as Published and never offer it to the fee bumper. This is dangerous
as the input will never be fee bumped. We now fix it by always
initializing the input with state Init, and only use mempool to check
for fee and fee rate.
This changes the current restart behavior - as previously when a
sweeping tx is broadcast, the node shuts down, when it starts again, the
input will be offered to the sweeper again, but not to the fee bumper,
which means the sweeping tx will stay in the mempool with the last-tried
fee rate. After this change, after a restart, the input will be swept
again, and the fee bumper will monitor its status. The restart will also
behave like a fee bump if there's already an existing sweeping tx in the
mempool.
This commit handles the case when the input is missing during the RBF
process, which could happen when the bumped tx has inputs being spent by
a third party. Normally we should be able to catch the spend early via
the spending notification and never attempt to fee bump the record.
However, due to the possible race between block notification and spend
notification, this cannot be guaranteed. Thus, we need to handle the
case during the RBF when seeing a `ErrMissingInputs`, which can only
happen when the inputs are spent by others.
This commit refactors `handleInitialTxError` and `createAndCheckTx` to
take a `monitorRecord` param, which prepares for the following commit
where we start handling missing inputs.
This is a minor refactor so the `createAndPublishTx` flow becomes more
clear, also prepares for the following commit where we start to handle
missing inputs.
We now rename "third party" to "unknown" as the inputs can be spent via
an older sweeping tx, a third party (anchor), or a remote party (pin).
In fee bumper we don't have the info to distinguish the above cases, and
leave them to be further handled by the sweeper as it has more context.
This commit refactors the `processRecords` to always handle the inputs
spent when processing the records. We now make sure to handle unknown
spends for all backends (previously only neutrino), and rely solely on
the spending notification to give us the onchain status of inputs.
This commit restricts the graph CRUD interface such that one can only
add a proof to a channel announcement and not update any other fields.
This pattern is better suited for SQL land too.
This commit fixes a bug that would check that the passed `edge` argument
of UpdateChannelEdge is nil but it should actually be checking if the
`edges` bucket is nil.
In this commit, we use the available kvdb `View` method directly for
starting a graph session instead of manually creating and commiting the
transaction. Note this changes behaviour since failed tx create/commit
will now error instead of just log.
In this commit, we add a `GraphSession` method to the `ChannelGraph`.
This method provides a caller with access to a `NodeTraverser`. This is
used by pathfinding to create a graph "session" overwhich to perform a
set of queries for a pathfinding attempt. With this refactor, we hide
details such as DB transaction creation and transaction commits from the
caller. So with this, pathfinding does not need to remember to "close
the graph session". With this commit, the `graphsession` package may be
completely removed.
In preparation for the next commit where we will start hiding underlying
graph details such as that a graph session needs to be "closed" after
pathfinding is done with it, we refactor things here so that the main
pathfinding logic is done in a call-back.
The `graphsession.NewRoutingGraph` method was used to create a
RoutingGraph instance with no consistent read transaction across calls.
But now that the ChannelGraph directly implements this, we can remove
The NewRoutingGraph method.
In preparation for having the ChannelGraph directly implement the
`routing.Graph` interface, we rename the `ForEachNodeChannel` method to
`ForEachNodeDirectedChannel` since the ChannelGraph already uses the
`ForEachNodeChannel` name and the new name is more appropriate since the
ChannelGraph currently has a `ForEachNodeDirectedChannelTx` method which
passes the same DirectedChannel type to the given call-back.
Add the `Tx` suffix to both ForEachNodeDirectedChannelTx and
FetchNodeFeatures temporarily so that we free up the original names for
other use. The renamed methods will be removed or unexported in an
upcoming commit. The aim is to have no exported methods on the
ChannelGraph that accept a kvdb.RTx as a parameter.
For consistency in the graphsessoin.graph interface, we let the
FetchNodeFeatures method take a read transaction just like the
ForEachNodeDirectedChannel. This is nice because then all calls in the
same pathfinding transaction use the same read transaction.
We only append the chan_id and the human readable scid
for the commands `listchannels` and `closedchannels`. This
ensures that other commands like `buildroute` are not affected
by this change so their output can be piped into other cmds.
For some cmds it is not very practical to replace the json output
because we might pipe it into other commands. For example when
creating the route we want to pipe the result of buildroute into
sendtoRoute.
This commit is a pure refactor. We move the transaction validation
(existence, spentness, correctness) from the `graph.Builder` to the
gossiper since this is where all protocol level checks should happen.
All tests involved are also updated/moved.
As we move the funding transaction validation logic out of the builder
and into the gossiper, we want to ensure that the behaviour stays
consistent with what we have today. So we should aquire this lock before
performing any expensive checks such as building the funding tx or
valdating it.
In preparation for an upcoming commit which will move all channel
funding tx validation to the gossiper, we first move the helper method
which helps build the expected funding transaction script based on the
fields in the channel announcement. We will still want this script later
on in the builder for updating the ChainView though, and so we pass this
field along with the ChannelEdgeInfo. With this change, we can remove
the TapscriptRoot field from the ChannelEdgeInfo since the only reason
it was there was so that the builder could reconstruct the full funding
script.
Here, we add a new fundingTxOption modifier which will configure how we
set-up expected calls to the mock Chain once we have moved funding tx
logic to the gossiper. Note that in this commit, these modifiers don't
yet do anything.
This is in preparation for the commit where we move across all the
funding tx validation so that we can test that we are correctly updating
the zombie index.
The `graph.Builder`'s `addZombieEdge` method is currently called during
funding transaction validation for the case where the funding tx is not
found. In preparation for moving this code to the gossiper, we export
the method and add it to the ChannelGraphSource interface so that the
gossiper will be able to call it later on.
Sometimes only the "Send coverage" step of a CI job will fail. This
commit turns this step into a "best effort" step instead so that it
does not block a CI job from passing.
It can, for example, often happen that a single job is re-run from the
GH UI, it then passes but the "Send coverage" step fails due to the
"Can't add a job to a build that is already closed." error meaning that
the only way to get the CI step to pass is to re-push and retrigger a
full CI run.
This commit changes `GetInfo` to include `blockbeatDispatcher`'s current
state when deciding whether the system is synced to chain. Previously we
check the best height against the wallet and the channel graph, we
should also do this to the blockbeat dispatcher to make sure the
internal consumers are also synced to the best block.
With the previous commit, the AddNode method was removed and since that
was the only method making use of the ForEachChannel on the
GraphCacheNode interface, we can remove that method. Since the only two
methods left just expose the node's pub key and features, it really is
not required anymore and so the entire thing can be removed along with
the implementation of it.
The AddNode method on the GraphCache calls `AddNodeFeatures` underneath
and then iterates through all the node's persisted channels and adds
them to the cache too via `AddChannel`.
This is, however, not required since at the time the cache is populated
in `NewChannelGraph`, the cache is populated will all persisted nodes
and all persisted channels. Then, once any new channels come in, via
`AddChannelEdge`, they are added to the cache via AddChannel. If any new
nodes come in via `AddLightningNode`, then currently the cache's AddNode
method is called which the both adds the node and again iterates through
all persisted channels and re-adds them to the cache. This is definitely
redundent since the initial cache population and updates via
AddChannelEdge should keep the cache fresh in terms of channels.
So we remove this for 2 reasons: 1) to remove the redundent DB calls and
2) this requires a kvdb.RTx to be passed in to the GraphCache calls
which will make it hard to extract the cache out of the CRUD layer
and be used more generally.
The AddNode method made sense when the cache was first added in the
code-base
[here](369c09be61 (diff-ae36bdb6670644d20c4e43f3a0ed47f71886c2bcdf3cc2937de24315da5dc072R213))
since then during graph cache population, nodes and channels would be
added to the cache in a single DB transaction. This was, however,
changed [later
on](352008a0c2)
to be done in 2 separate DB calls for efficiency reasons.
Define a new GraphSource interface that describes the access required by
the autopilot server. Let its constructor take this interface instead of
a raw pointer to the graphdb.ChannelGraph.
In this commit, a new NodeRTx interface is added which represents
consistent access to a persisted models.LightningNode. The
ForEachChannel method of the interface gives the caller access to the
node's channels under the same read transaction (if any) that was used
to fetch the node in the first place. The FetchNode method returns
another NodeRTx which again will have the same underlying read
transaction.
The main point of this interface is to provide this consistent access
without needing to expose the `kvdb.RTx` type as a method parameter.
This will then make it much easier in future to add new implementations
of this interface that are backed by other databases (or RPC
connections) where the `kvdb.RTx` type does not apply.
We will make use of the new interface in the `autopilot` package in
upcoming commits in order to remove the `autopilot`'s dependence on the
pointer to the `*graphdb.ChannelGraph` which it has today.
Introduce a new type for testing code so the main databaseChannelGraph
type does not need to make various write calls to the
`graphdb.ChannelGraph` but the new testDBGraph type still can for tests.
This is a pure code move commit where we move any code that is only ever
used by tests to test files. Many of the calls to the
graphdb.ChannelGraph pointer are only coming from tests code.
We need to make sure the links are recreated before calling the
interceptor, otherwise we'd get the following error, causing the test to
fail.
```
2025-02-07 15:01:38.991 [ERR] RPCS interceptor.go:624: [/routerrpc.Router/HtlcInterceptor]: fwd (Chan ID=487:3:0, HTLC ID=0) not found
```
This commit refactors `collectResultAsync` such that this method is now
only responsible for collecting results from the switch. The method
`decideNextStep` is expanded to process these results in the same
goroutine where we fetch the payment from db, to make sure the lifecycle
loop always have a consistent view of a given payment.
In preparation for adding more modifiers. We want to later add a
modifier that will tweak the errors returned by the mock chain once
funding transaction validation has been moved to the gossiper.
This is in preparation for moving the funding transaction validation
code to the gossiper from the graph.Builder since then the gossiper will
start making GetBlockHash/GetBlock and GetUtxo calls.
Convert a bunch of the helper functions to instead be methods on the
testCtx type. This is in preparation for adding a mockChain to the
testCtx that these helpers can then use to add blocks and utxos to.
See `notifications_test.go` for an idea of what we are trying to emulate
here. Once the funding tx code has moved to the gossiper, then the logic
in `notifications_test.go` will be removed.
In preparation for moving funding transaction validiation from the
Builder to the Gossiper in later commit, we first convert these graph
Error Codes to normal error variables. This will help make the later
commit a pure code move.
The point of the `graph.Builder`'s `networkHandler` goroutine is to
ensure that certain requests are handled in a synchronous fashion.
However, any requests received on the `networkUpdates` channel, are
currently immediately handled in a goroutine which calls
`handleNetworkUpdate` which calls `processUpdate` before doing topology
notifications. In other words, there is no reason for these
`networkUpdates` to be handled in the `networkHandler` since they are
always handled asynchronously anyways. This design is most likely due to
the fact that originally the gossiper and graph builder code lived in
the same system and so the pattern was copied across.
So in this commit, we just remove the complexity. The only part we need
to spin off in a goroutine is the topology notifications.
In this commit, we remove the `processUpdate` method which handles each
announement type (node, channel, channel update) in a separate switch
case. Each of these cases currently has a non-trivial amount of code.
This commit creates separate methods for each message type we want to
handle instead. This removes a level of indentation and will make things
easier to review when we start editing the code for each handler.
The `netann` package is a more appropriate place for this code to live.
Also, once the funding transaction code is moved out of the
`graph.Builder`, then no `lnwire` validation will occur in the `graph`
package.
We can use the built-in `min` and `max` functions since Go 1.21.
Reference: https://go.dev/ref/spec#Min_and_max
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This way we can greatly simplify the method signatures, also paving the
upcoming changes where we wanna make it clear when updating the
monitorRecord, we only touch a portion of it.
This commit shortens the function signature of `storeRecord`, also makes
sure we don't call `t.records.Store` directly but always using
`storeRecord` instead so it's easier to trace the record creation.
In this commit, we warn users about the removal
of RPCs `SendToRoute`, `SendToRouteSync`, `SendPayment`,
and `SendPaymentSync` in the next release 0.20.
We now cancel all HTLCs of an AMP invoice as soon as it expires.
Otherwise because we mark the invoice as cancelled we would not
allow accepted HTLCs to be resolved via the invoiceEventLoop.
We always fetch the HTLCs for the specific setID, so there is no
need to keep this code. In earlier versions we would call the
UpdateInvoice method with `nil` for the setID therefore we had
to lookup the AMPState. However this was error prune because in
case one partial payment times-out the AMPState would change to
cancelled and that could lead to not resolve HTLCs.
We introduce a new specific fail resolution error when the
external HTLC interceptor denies the incoming HTLC. Moreover
we introduce a new traffic shaper method which moves the
implementation of asset HTLC to the external layers.
Moreover itests are adopted to reflect this new change.
We make sure that HTLCs which have already been decided upon
are resolved before before allowing the external interceptor to
potentially cancel them back. This makes the implementation for
the external HTLC interceptor more streamlined.
When mining lots of blocks in the itest, the subsystems can be out of
sync in terms of the best block height. We now assert the num of pending
sweeps on Alice's node to give her more time to sync the blocks,
essentially behaving like a `time.Sleep` as in reality, the blocks would
never be generated this fast.
In this commit, we add the --no-backup-archive with a default
as false. When set to true then previous channel backup file will
not be archived but replaced. We also modify TestUpdateAndSwap
test to make sure the new behaviour works as expected.
In this commit, we first check if a previous backup file exists,
if it does we copy it to archive folder before replacing it with
a new backup file. We also added a test for archiving chan backups.
This commit does two things:
- removes the concept of allow / deny. Having this in place was a
minor optimization and removing it makes the solution simpler.
- changes the job dependency tracking to track sets of abstact
parent jobs rather than individual parent jobs.
As a note, the purpose of the ValidationBarrier is that it allows us
to launch gossip validation jobs in goroutines while still ensuring
that the validation order of these goroutines is adhered to when it
comes to validating ChannelAnnouncement _before_ ChannelUpdate and
_before_ NodeAnnouncement.
This omits calls to InitJobDependencies, SignalDependants, and
WaitForDependants. These changes have been made here because
the router / builder code does not actually need job dependency
management. Calls to the builder code (i.e. AddNode, AddEdge,
UpdateEdge) are all blocking in the gossiper. This, combined
with the fact that child jobs are run after parent jobs in the
gossiper, means that the calls to the router will happen in the
proper dependency order.
This commit introduces the functionality to set a tombstone key
in the invoice bucket after the migration to the native SQL
database. The tombstone prevents the user from switching back
to the KV invoice database, ensuring data consistency and
avoiding potential issues like lingering invoices or partial
state in KV tables. The tombstone is checked on startup to
block any manual overrides that attempt to revert the migration.
Previously, we applied replacements to our schema definitions
to make them compatible with both SQLite and Postgres backends,
as the files were not fully compatible with either.
With this change, the only replacement required for SQLite has
been moved to the generator script. This adjustment ensures
compatibility by enabling auto-incrementing primary keys that
are treated as 64-bit integers by sqlc.
Previously we'd recalculate the paid amount by summing amounts of
settled HTLCs. This approach while correct would stop the SQL migration
process as some KV invoices may have incorrectly stored paid amounts.
The current sqlc GetInvoice query experiences incremental slowdowns during
the migration of large invoice databases, primarily due to its complex
predicate set. For this specific use case, a streamlined GetInvoiceByHash
function provides a more efficient solution, maintaining near-constant
lookup times even with extensive table sizes.
Certain invoices may not have a deterministic payment hash. For such
invoices we still store the payment hashes in our KV database, but we do
not have a sufficient index to retrieve them. This PR adds such index to
the SQL database that will be used during migration to retrieve payment
hashes.
Previously we intentially did not set settled_at and settle_index when
inserting a new invoice as those fields are set when we settle an
invoice through the usual invoice update. As migration requires that we
set these nullable fields, we can safely add them.
This commit separates the execution of SQL and in-code migrations
from their construction. This change is necessary because,
currently, the SQL schema is migrated during the construction
phase in the lncfg package. However, migrations are typically
executed when individual stores are constructed within the
configuration builder.
This commit introduces support for custom, in-code migrations, allowing
a specific Go function to be executed at a designated database version
during sqlc migrations. If the current database version surpasses the
specified version, the migration will be skipped.
We need to make sure Caarol finishes settling her invoice with Bob
before shutting down, so we make sure `AssertHTLCNotActive` on Bob
happens before shutting node Carol.
We add a new config option to set the `ProofMatureDelta` so the users
can tune their graphs based on their own preference over the num of
confs found in the announcement signatures.
The mocked peer used here blocks on `sendToPeer`, which is not the
behavior of the `SendMessageLazy` of `lnpeer.Peer`. To reflect the
reality, we now make sure the `sendToPeer` is non-blocking in the tests.
This commit fixes the following race,
1. syncer(state=syncingChans) sends QueryChannelRange
2. remote peer replies ReplyChannelRange
3. ProcessQueryMsg fails to process the remote peer's msg as its state
is neither waitingQueryChanReply nor waitingQueryRangeReply.
4. syncer marks its new state waitingQueryChanReply, but too late.
The historical sync will now fail, and the syncer will be stuck at this
state. What's worse is it cannot forward channel announcements to other
connected peers now as it will skip the broadcasting during initial
graph sync.
This is now fixed to make sure the following two steps are atomic,
1. syncer(state=syncingChans) sends QueryChannelRange
2. syncer marks its new state waitingQueryChanReply.
Added scid_str as a string representation of chan_id,
replacing chan_id with scid, and including chan_id(BOLT02)
in lncli listchannels output.
Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
Add the `AltString` method for `ShortChannelID` to produce a
human-readable format with 'x' as a separator
(block x transaction x output).
Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
This commit updates the fn dep to the version containing the updates to
the ContextGuard implementation. Only the htlcswitch/link uses the guard
at the moment so this is updated to make use of the new implementation.
This bump includes a fix which prevents attribute value quoting if the
value string contains a newline character. This is so that if we call
spew.DumpS(), the output will stay nicely formatted.
The update also includes a couple more Hex helpers which we can make use
of now.
Simplifies context cancellation handling by using context.AfterFunc instead of a
goroutine to wait for context cancellation. This approach avoids the overhead of
a goroutine during the waiting period.
For ctxQuitUnsafe, since g.quit is closed only in the Quit method (which also
cancels all associated contexts), waiting on context cancellation ensures the
same behavior without unnecessary dependency on g.quit.
Added a test to ensure that the Create method does not launch any goroutines.
Removed 'cancel' argument, because it is called only in case the context has
already expired and the only action that cancel function did was cancelling the
context.
We sometimes see `timeout waiting for UTXOs` error from bitcoind-related
itests due to the chain backend not synced to the miner. We now assert
it's synced before continue.
This commit removes the panic used in checking the shutdown log.
Instead, the error is returned and asserted in `shutdownAllNodes` so
it's easier to check which node failed in which test. We also catch all
the errors returned from `StopDaemon` call to properly access the
shutdown behavior.
Previous splitting logic simply put all the remainder in the last
tranche, which could make the last tranche run significantly more test
cases. We now change it so the remainder is evened out across tranches.
Most of the time we only need to fund the node with given number of
UTXOs without concerning the amount, so we add the more efficient
funding method as it mines a single block in the end.
To make the CI indicative, we now starting tracking the flaky tests
found when running on Windows. As a starting point, rather than ignore
the windows CI entirely, we now identify there are cases where lnd can
be buggy when running in windows.
We should fix the tests in the future, otherwise the windows build
should be deleted.
This has been seen in the itest which can lead to the node startup
failure,
```
2024-11-20 18:55:15.727 [INF] RPCS: Max websocket clients exceeded [25] - disconnecting client 127.0.0.1:57224
```
We need to mine an empty block as the tx may already have entered the
mempool. This should be fixed once we start using the sweeper to handle
the justice tx.
Soemtimes the node may be hanging for a while without being noticed,
causing failures in its following tests, thus making the debugging
extrememly difficult. We now assert the node has been shut down from the
logs to assert the shutdown process behaves as expected.
Since we don't have standby nodes anymore, we don't need to close the
channels when the test finishes. Previously we would do so to make sure
the standby nodes have a clean state for the next test case, which is no
longer relevant.
This commit removes the usage of the standby nodes and uses
`CreateSimpleNetwork` when applicable. Also introduces a helper method
`NewNodeWithCoins` to quickly start a node with funds.
Prepares the upcoming refactor. We now never call `ht.Alice` directly,
instead, we always init `alice := ht.Alice` so it's easier to see how
they are removed in a following commit.
This commit adds a new flag to shuffle all the test cases before running
them so tests which require lots of blocks to be mined are less likely
to be run in the same tranch.
The other benefit is this approach provides a more efficient way to
figure which tests are broken since all the differnet backends are
running different tranches in their builds, we can identify more failed
tests in one push.
The method `AssertActiveHtlcs` is now removed due to it's easy to be
misused. To assert a given htlc, use `AssertOutgoingHTLCActive` and
`AssertIncomingHTLCActive` instead for ensuring the HTLC exists in the
right direction. Although often the case `AssertNumActiveHtlcs` would be
enough as it implicitly checks the forwarding behavior for an
intermediate node by asserting there are always num_payment*2 HTLCs.
This commit simplfies the test since we only test the preimage
extraction logic in the htlc timeout resolver, there's no need to test
it for all different channel types as the resolver is made to be
oblivious about them.
Starting from this commit, we begin the process of flattening the
multi-hop itests to make them easier to be maintained. The tests are
refactored into their own test cases, with each test focusing on testing
one channel type. This is necessary to save effort for future
development.
These tests are also updated to reflect the new `blockbeat` behavior.
We need to check `dispatched` before sending conf details, otherwise the
channel `ntfn.Event.Confirmed` will be blocking, which is the leftover
from #9275.
This commit adds a new method to enable us resending the blockbeat in
`ChainArbitrator`, which is needed for the channel restore as the chain
watcher and channel arbitrator are added after the start of the chain
arbitrator.
This commit moves the creation of the spending notification from `Start`
to `newChainWatcher` so we subscribe the spending event before handling
the block, which is needed to properly handle the blockbeat.
We need to offer the outgoing htlc one block earlier to make sure when
the expiry height hits, the sweeper will not miss sweeping it in the
same block. This also means the outgoing contest resolver now only does
one thing - watch for preimage spend till height expiry-1, which can
easily be moved into the timeout resolver instead in the future.
In this commit, we break the old `launchResolvers` into two steps - step
one is to launch the resolvers synchronously, and step two is to
actually waiting for the resolvers to be resolved. This is critical as
in the following commit we will require the resolvers to be launched at
the same blockbeat when a force close event is sent by the chain watcher.
When calling `NotifyExitHopHtlc` it is allowed to pass a chan to
subscribe to the HTLC's resolution when it's settled. However, this
method will also return immediately if there's already a resolution,
which means it behaves like a notifier and a getter. If the caller
decides to only use the getter to do a non-blocking lookup, it can pass
a nil subscriber chan to bypass the notification.
This commit breaks the `Resolve` into two parts - the first part is
moved into a `Launch` method that handles sending sweep requests, and
the second part remains in `Resolve` which handles waiting for the
spend. Since we are using both utxo nursery and sweeper at the same
time, to make sure this change doesn't break the existing behavior, we
implement the `Launch` as following,
- zero-fee htlc - handled by the sweeper
- direct output from the remote commit - handled by the sweeper
- legacy htlc - handled by the utxo nursery
This commit breaks the `Resolve` into two parts - the first part is
moved into a `Launch` method that handles sending sweep requests, and
the second part remains in `Resolve` which handles waiting for the
spend. Since we are using both utxo nursery and sweeper at the same
time, to make sure this change doesn't break the existing behavior, we
implement the `Launch` as following,
- zero-fee htlc - handled by the sweeper
- direct output from the remote commit - handled by the sweeper
- legacy htlc - handled by the utxo nursery
We will use this and its following commits to break the original
`Resolve` methods into two parts - the first part is moved to a new
method `Launch`, which handles sending a sweep request to the sweeper.
The second part remains in `Resolve`, which is mainly waiting for a
spending tx.
Breach resolver currently doesn't do anything in its `Launch` since the
sweeping of justice outputs are not handled by the sweeper yet.
In this commit we start to break up the starting process into smaller
pieces, which is needed in the following commit to initialize blockbeat
consumers.
This `immediate` flag was added as a hack so during a restart, the
pending resolvers would offer the inputs to the sweeper and ask it to
sweep them immediately. This is no longer need due to `blockbeat`, as
now during restart, a block is always sent to all subsystems via the
flow `ChainArb` -> `ChannelArb` -> resolvers -> sweeper. Thus, when
there are pending inputs offered, they will be processed by the sweeper
immediately.
The sweeper can handle the waiting so there's no need to wait for blocks
inside the resolvers. By offering the inputs prior to their mature
heights also guarantees the inputs with the same deadline are
aggregated.
This commit removes the hack introduced in #4851. Previously we had this
issue because the chain notifier was stopped before the sweeper, which
was changed a while back and we now always stop the chain notifier last.
In addition, since we no longer subscribe to the block epoch chan
directly, this issue can no longer happen.
This commit removes the independent block subscriptions in `UtxoSweeper`
and `TxPublisher`. These subsystems now listen to the `BlockbeatChan`
for new blocks.
In this commit, a minimal implementation of `Blockbeat` is added to
synchronize block heights, which will be used in `ChainArb`, `Sweeper`,
and `TxPublisher` so blocks are processed sequentially among them.
This commit inits the package `chainio` and defines the interface
`Blockbeat` and `Consumer`. The `Consumer` must be implemented by other
subsystems if it requires block epoch subscription.
Previously in `markInputFailed`, we'd remove all inputs under the same
group via `removeExclusiveGroup`. This is wrong as when the current
sweep fails for this input, it shouldn't affect other inputs.
This commit adds a new method `handleInitialBroadcast` to handle the
initial broadcast. Previously we'd broadcast immediately inside
`Broadcast`, which soon will not work after the `blockbeat` is
implemented as the action to publish is now always triggered by a new
block. Meanwhile, we still keep the option to bypass the block trigger
so users can broadcast immediately by setting `Immediate` to true.
This commit changes how inputs are handled upon receiving a bump result.
Previously the inputs are taken from the `BumpResult.Tx`, which is now
instead being handled locally as we will remember the input set when
sending the bump request, and handle this input set when a result is
received.
With this we allow the linter to only look at recent changes, since
everything between that old commit and this most recent one has been
linted correctly anyway.
The fn.WriteFile writes data like os.WriteFile but in sync mode.
Also adds a behaviour flag that enables removal of file on error.
Co-authored-by: Maurice Poirrier Chuden <mauricepoirrier@gmail.com>
Co-authored-by: Greg Weber <1183+gregwebs@users.noreply.github.com>
When notifying the invoice registry for an exit hop htlc we also want to
include its custom records. The channelLink, the other caller of this
method, already populates this field. So we make sure the contest
resolver does so too.
In a rare case when the critical log is triggered when using postgres as
db backend, the `payment` could be nil cause the server is shutting
down, causing the payment fetching to return an error. We now cache its
state before fetching it from the db.
We have two sources which can call `handlePacketSettle`, either through
the link's `<-s.htlcPlex`, or the `<-s.resolutionMsgs`, which means the
`closeCircuit` could be call twice. Previously we already caught this
case inside `closeCircuit`, in that we would return a nil circuit upon
seeing `ErrUnknownCircuit`, indicating the circuit was removed. However,
we still need to account the case when the circuit is the process of
being closed, which is now fixed as we will ignore when seeing
`ErrCircuitClosing`.
In this commit, the ContextGuard struct is re-worked such that the
context that its new main WithCtx method provides is cancelled in sync
with a parent context being cancelled or with it's quit channel being
cancelled. Tests are added to assert the behaviour. In order for the
close of the quit channel to be consistent with the cancelling of the
derived context, the quit channel _must_ be contained internal to the
ContextGuard so that callers are only able to close the channel via the
exposed Quit method which will then take care to first cancel any
derived context that depend on the quit channel before returning.
To make this itest work reliably with multiple parallel SQL
transactions, we need to count both the settle and final HTLC
events. Otherwise, sometimes the final events from earlier
forwards are counted before the forward events from later
forwards, causing a miscount of the settle events. If we
expect both the settle and final event for each forward,
we don't miscount.
In this commit, we add a rapid derived fuzz test for the HtlcAuxBlob
test. This uses the rapid (randomized property testing) into Go's built
in fuzzer. This wrapper will use the fuzz stream, and pass that into
rapid where the stream is used to make structured test inputs which are
tested against the existing properties.
This can be done more widely in the codebase, we pick a simple example
to port first before tackling others.
This is a tradeoff of disk space (and with that cache size) and
compilation speed. Because we're still running into disk full errors
with the full build cache, we remove it for the cross compile step.
Which means we'll do more work each time.
In this commit, we enable a custom payer for the rbf coop close. This
allows us to ensure that the party that started one side of the close
flow pays the fees.
In this commit, we update the core coop close logic with the new custom
payer param. We also expand the existing unit tests to ensure that the
fee is deducted from the proper party.
This preps us for an upcoming change to the rbf coop state machine where
either party can pay for the channel fees. We also add a new test to
make sure the new function adheres to some key properties.
In this commit, we add the state transitions for the new protofsm based
RBF chan closer. The underlying protocol is a new asymmetric co-op close
process, wherein either side can initiate a chan closer, and use their
settled funds to pay for fees within the channel.
In this commit, we add the ability to specify a custom sequence for a
co-op close tx. This'll come in handy later as the new co-op close
process allows a party to set a custom sequence.
In this commit, we add the initial set of states for the new protofsm
based rbf chan closer. A diagram outlining the new states and their
transitions can be found here:
https://gist.github.com/Roasbeef/acc4ff51b9dff127230228a05553cdfe.
Unlike the existing co-op close process, this co-op close can be
restarted at anytime if either side sends a shutdown message. From
there, we'll each obtain a new RBF'd version that can be re-broadcasted.
This commit creates the set of states, along with the environment that
our state machine will use to drive itself forward.
When reporting an error or a success case of a payment to a
blinded paths, the amounts to forward for intermediate hops
are set to 0 so we need to use the receiver amount instead.
Fixes a bug and makes the function more robust. Before
we would always return the encrypted data size of last hop
of the last path. Now we return the greatest last hop payload
not always the one of the last path.
To be able to do MPP payment to multiple blinded routes we need
to add a constant dummy hop as a final hop to every blined path.
This is used when sending or querying a blinded path, to let the
pathfinder be able to send MPP payments over different blinded
routes. For this dummy final hop we use a NUMS key so that we
are sure no other node can use this blinded pubkey either in a
normal or blinded route.
Moreover this helps us handling the mission control data for
blinded paths correctly because we always consider the blinded
pubkey pairs which are registered with mission control when
a payment to a blinded path fails.
For calculating the available auxiliary bandwidth of a channel, we need
access to the inbound custom wire records of the HTLC packet, which
might contain auxiliary information about the worth of the HTLC packet
apart from the BTC value being transported.
With this commit we move the traffic shaper definition from the routing
package to the HTLC switch package as a preparation for being able to
use it there as well.
At the same time we rename it to AuxTrafficShaper to be more in line
with the other auxiliary components.
Since we now have a scripted check that makes sure the Golang version is
synced throughout all files, it is important that the main version in
the Makefile is changed, then all other versions can be detected by that
script.
Find and replace all nolint instances refering to the `lll` linter and
replace with `ll` which is the name of our custom version of the `lll`
linter which can be used to ignore log lines during linting.
The next commit will do the configuration of the custom linter and
disable the default one.
Add this file both to the main LND directory so that devs can use it for
local linter runs and also add it to the `tools` directory so that the
docker environment used to run the linter in CI has access to it. A
custom linter binary can be built via `golangci-lint custom`. This will
pull in and register all the plugins listed in the new config file when
building the new binary. The new binary can then be run using
`custom-gcl run`.
This commit introduces the `LLPlugin` type and converts the existing lll
code such that the LLPlugin implements the register.LinterPlugin
interface. This will allow us to plug it into golangci-linter as a
plugin.
In this commit, we copy the implementation of the golangci-lint lll
linter which we currently use in our CI flow during the `make lint`
check. This commit copies the code mostly as is (the only exception
being the line which trims white space before checking if a line starts
with "import"), and formats it to fit our codebase guidelines. A test is
also added so that we can be sure that the following commits which
adjust the implementation have the intended results.
The custom linter is put into its own module as this is requied by the
`golangci-lint custom` when building the custom linter binary which
includes the plugin.
and the same for ChannelStateDB.FetchChannel. Most of the calls to these
methods provide a `nil` Tx anyways. The only place that currently
provides a non-nil tx is in the `localchans.Manager`. It takes the
transaction provided to the `ForAllOutgoingChannels` callback and passes
it to it's `updateEdge` method. Note, however, that the
`ForAllOutgoingChannels` call is a call to the graph db and the call to
`updateEdge` is a call to the `ChannelStateDB`. There is no reason that
these two calls need to happen under the same transaction as they are
reading from two completely disjoint databases. And so in the effort to
completely split untangle the relationship between the two databases, we
now dont use the same transaction for these two calls.
Before this commit, the `(channeldb.DB).AddrsForNode` method treats the
results from the channel db and the graph db slightly differently. It
errors out if the channel db is unaware of the node in question but does
not error out if the graph is unaware of the node. So this commit
changes the logic slightly so that a "node-unknown" error from either
backing source is not seen as an error.
In this commit, we implement a version of the AddrSource interface which
merges the results of a set of AddrSource implementations. This will
later be used to merge the results of the channel db and graph db.
Our aim is to completely remove the `channeldb.DB`'s direct dependence
on the `graphdb.ChannelGraph` pointer. The only place where it still
depends on this pointer is in the `(DB).AddrsForNode(..)` method where
it queries both the channel DB and the graph db for the known addresses
for the node in question and then combines the results. So, to separate
these out, we will define an AddrsForNodes interface in this commit
which we will later let both the ChannelGraph and channeldb.DB both
implement and we will merge these results outside of the channeldb
package.
All this commit does is to unify the `AddrSource` interface since this
has been defined separately in a couple of places.
We also now use the graph DB's own optional functions. An instance of
the graph is currently still passed to the channeldb's
`CreateWithBackend` function. This will be removed in a later commit
once the two have been completely disjoint.
All the structs defined in the `channeldb/models` package are graph
related. So once we move all the graph CRUD code to the graph package,
it makes sense to have the schema structs there too. So this just moves
the `models` package over to `graph/db/models`.
We have the same helpers for writing and reading a wire.Outpoint type
defined separately in a couple places. We will want to use these from
the graph db package soon though so instead of defining them again
there, this commit unifies things and creates a single exported set of
helpers. The next commit will make use of these.
In preparation for moving the graph related schema structs to the graph
package in an upcoming commit, we move these methods to the graph
package. The structs we will move make use of these methods but we still
import them from channeldb so as to keep the ReadElement and
WriteElement helpers working as they do today.
If feeurl is not provided and LND is running in bitcoind or btcd mode, use
fee estimator provided by bitcoind or btcd instead of using static fee estimator
(which always returns 50 sats/vbyte).
This enables simulating high feerate environment in regtest and simnet not only
via feeurl, but also by filling mempool with high feerate transactions, which is
closer to what happens in mainnet.
Also skip creating bitcoind or btcd fee estimator, if feeurl is provided. Before
LND logged both "Initializing bitcoind backed fee estimator" and "Using external
fee estimator" if running in bitcoind mode with feeurl provided, but actually
feeurl was used, so this is not a change of behavior.
In this commit, we fix an existing race in the new `protofsm` state
machine executor tests.
The race would appear as such:
```
--- FAIL: TestStateMachineMsgMapper (0.00s)
state_machine_test.go:165:
Error Trace:/home/runner/work/lnd/lnd/protofsm/state_machine_test.go:165
/home/runner/work/lnd/lnd/protofsm/state_machine_test.go:451
Error: Object expected to be of type *protofsm.dummyStateStart, but was *protofsm.dummyStateFin
Test: TestStateMachineMsgMapper
FAIL
FAILgithub.com/lightningnetwork/lnd/protofsm0.116s
FAIL
```
This race condition was triggered as before we would start the state
machine _then_ register for notifications. In `Start` we emit the
starting event, then enter the main loop. If that event gets emitted
before our subscription, then we'll miss the event, as the terminal
event will be the only one received.
We fix this by registering for the events before the daemon has started.
This change simplifies some of the quiescer responsibilities in
favor of making the link check whether or not it has a clean state
to be able to send or receive an stfu. This change was made on the
basis that the only use the quiescer makes of this information is
to assess that it is or is not zero. Further the difficulty of
checking this condition in the link is barely more burdensome than
selecting the proper information to pass to the quiescer anyway.
In this commit we implement a noop quiescer that we will use when
the feature hasn't been negotiated. This will make it far easier to
manage quiescence operations without having a number of if statements
in the link logic.
In this commit we defer processRemoteAdds using a new mechanism on
the quiescer where we capture a closure that needs to be run. We
do this because we need to avoid the scenario where we send back
immediate resolutions to the newly added HTLCs when quiescent as
it is a protocol violation. It is not enough for us to simply defer
sending the messages since the purpose of quiescence itself is to
have well-defined and agreed upon channel state. If, for whatever
reason, the node (or connection) is restarted between when these
hooks are captured and when they are ultimately run, they will
be resolved by the resolveFwdPkgs logic when the link comes back
up.
In a future commit we will explicitly call the quiescer's resume
method when it is OK for htlc traffic to commence.
Before we have sufficient signaling in the network to relay this
signal, set a zero value experimental endorsement value on the sender's
outgoing htlc. Once the network is relaying this signal and a flag day
has been set, we'll be able to set a non-zero value here.
In this commit, we test for different values of index_offset and
max_transactions settings when getting transactions to make sure
the right number of transactions are returned along with the right
first and last indices.
In this commit we introduce first and last indices for the
tranasctions returned which can be used to seek for further
transactions in a pagination style.
Signed-off-by: Abdullahi Yunus <abdoollahikbk@gmail.com>
This commit adds index_offset and max_transactions to the list
of parameters passed during gettransactions call. index_offset
specify transactions to skip and max_transactions the total
transactions returned
Previously, the error message had one missing space:
"... either --bitcoin.mainnet, or bitcoin.testnet,bitcoin.simnet, ..."
I added a space after "bitcoin.testnet,".
Previously, "lnd --debuglevel show" complained that bitcoin backend is not
configured:
$ lnd --debuglevel show
failed to load config: ValidateConfig: either --bitcoin.mainnet, or ...
To make it working, the user had to specify --bitcoin.mainnet and configure one
of the backends, e.g. "--bitcoin.node neutrino".
With this fix, the command works without any additional flags.
We now make it possible to get the current db version of the
wtclient.db. Moreover we can now fetch the latest available
migration version for the client db. This allows us to compare
whether the client.db has all the expected migrations applied.
This commit adds a new state to the `ConfNtfn` struct to start tracking
the number of confs left to be notified to avoid sending duplicate
notifications.
This commit fixes a bug where the confirmation details may be missed.
When the same tx is subscribed via `RegisterConfirmationsNtfn`, we will
put them into the same set and notify the whole set. However, this logic
is missing when performing the rescan - once the confirmation detail is
found, we only notify the current subscriber. Later we will skip
notifying other subscribers in `UpdateConfDetails` due to the
`confSet.details != nil` check. We now fix it by immediately notify all
the subscribers when the confirmation detail is found during the rescan.
This commit changes the order of notifications when a relevant tx is
found in a block and now we will always notify the tx subscribers before
notifying the block, which has implications in the upcoming blockbeat.
When a block notification is subscribed via `RegisterBlockEpochNtfn` and
a confirm or spend is subscribed via `RegisterConfirmationsNtfn` or
`RegisterSpendNtfn`, we would always notify the block first before the
tx, causing the subsystem to think there's no relevant txns found in
this block, while the notifications are sent later. We now fix it by
always sending the txns notifications first, so the subsystem can
receive the txns, process them, then attempt to advance its state based
on the block received.
In this commit, we update the execution logic to allow multiple internal
events to be emitted. This is useful to handle potential out of order
state transitions, as they can be cached, then emitted once the relevant
pre-conditions have been met.
In this commit, we add the SpendMapper which allows callers to create
custom spent events. Before this commit, the caller would be able to
have an event sent to them in the case a spend happens, but that event
wouldn't have any of the relevant spend details.
With this new addition, the caller can specify how to take a generic
spend event, and transform it into the state machine specific spend
event.
In this commit, we add the ability for the state machine to consume wire
messages. This'll allow the creation of a new generic message router
that takes the place of the current peer `readHandler` in an upcoming
commit.
In this commit, we add an optional daemon event that can be specified to
dispatch during init. This is useful for instances where before we
start, we want to make sure we have a registered spend/conf notification
before normal operation starts.
We also add new unit tests to cover this, and the prior spend/conf event
additions.
In this PR, we create a new package, `protofsm` which is intended to
abstract away from something we've done dozens of time in the daemon:
create a new event-drive protocol FSM. One example of this is the co-op
close state machine, and also the channel state machine itself.
This packages picks out the common themes of:
* clear states and transitions between them
* calling out to special daemon adapters for I/O such as transaction
broadcast or sending a message to a peer
* cleaning up after state machine execution
* notifying relevant callers of updates to the state machine
The goal of this PR, is that devs can now implement a state machine
based off of this primary interface:
```go
// State defines an abstract state along, namely its state transition function
// that takes as input an event and an environment, and returns a state
// transition (next state, and set of events to emit). As state can also either
// be terminal, or not, a terminal event causes state execution to halt.
type State[Event any, Env Environment] interface {
// ProcessEvent takes an event and an environment, and returns a new
// state transition. This will be iteratively called until either a
// terminal state is reached, or no further internal events are
// emitted.
ProcessEvent(event Event, env Env) (*StateTransition[Event, Env], error)
// IsTerminal returns true if this state is terminal, and false otherwise.
IsTerminal() bool
}
```
With their focus being _only_ on each state transition, rather than all
the boiler plate involved (processing new events, advancing to
completion, doing I/O, etc, etc).
Instead, they just make their states, then create the state machine
given the starting state and env. The only other custom component needed
is something capable of mapping wire messages or other events from the
"outside world" into the domain of the state machine.
The set of types is based on a pseudo sum type system wherein you
declare an interface, make the sole method private, then create other
instances based on that interface. This restricts call sites (must pass
in that interface) type, and with some tooling, exhaustive matching can
also be enforced via a linter.
The best way to get a hang of the pattern proposed here is to check out
the tests. They make a mock state machine, and then use the new executor
to drive it to completion. You'll also get a view of how the code will
actually look, with the focus being on the: input event, current state,
and output transition (can also emit events to drive itself forward).
In this commit, we complete a recently added feature by ensuring that
even if we go through the RBF loop to create a txn, that we still
populate the `outpointToIndex` map. Unit tests have been updated to
ensure this is always set as expected.
This commit fixes a case where the peer blocks the shutdown process.
During the shutdown, the server will call `s.DisconnectPeer`, which
calls `peer.Disconnect`. Because the peer never enters `peer.Start` via
`s.peerInitializer`, the `startReady` chan will not be closed, causing
`peer.Disconnect` to hang forever. We now fix it by only block on
`startReady` when the peer is started.
In this commit, we start to print the PID of node's we created within an
itest. This is useful when debugging itests with `dlv` by attaching to a
given node. By printing out the PID, we facilitate such debugging
attempts.
In this commit, we update the `Budget()` call to factor in the
`extraBudget` value. Otherwise, when we go to intialize the fee
function, we won't factor in the extra budget, and will determine that
we can't broadcast/bump.
In this commit, we expand the `NotifyBroadcast` to include an outpoint
index. This is useful as it indicates the index of a given required tx
out input.
With this commit, we update all the resolvers to pass in the new htlc
resolution blobs. Along the way, we remove the old blocking guard on
this resolution logic for HTLCs with blobs.
In this commit, we add the set of HtlcBlobs to the taprootBriefcase
struct. This new field will store all the resolution blobs for a given
HTLC. We also add some new property based tests along the way for
adequate test coverage.
In this commit, we add a new method to obtain an option of a preimage to
the input.Input struct. This is useful for callers that have an Input,
and want to optionally obtain the preimage.
In this commit, we populate the resolution blobs for the incoming and
outgoing HTLCs. We take care to populate the AuxSigDesc with the correct
information, as we need to pass along the second-level aux signature and
also sign desc along with it.
Similar to the other blobs we have for the commitment output force close
resolution, these blobs will be used to ensure that we have everything
needed to sweep aux HTLCs.
In this commit, we add some additional attributes to the ResolutionReq
struct. These will be used to make sure that we can properly handle all
the HTLC variants, on chain.
The `AuxSigDesc` will be used to communicate if an HTLC needs to go to
the second level or not. It contains the second-level sig information
needed to finalize a broadcast to the second level.
The prefixing is done every time the harness is used, so it may as well
reside in the harness itself.
Since we already pass an empty message of the correct type, we can use
that message to get the required prefix bytes.
There are several fuzz targets that can't use the standard require.Equal
check for various reasons. By adapting the harness to accept a custom
equality function, we can reduce code duplication in these targets.
This slightly more descriptive name distinguishes the wire message
harness from the onion failure harness while also obviating the
repetitive comments at every call site.
If a node contains a channel, but doesn't have a corresponding edge in
the graph database, updating the channel policy would fail. In this
commit the edge is recreated if the channel exists. This ensures a node
can recover from a missing edge in the graph database by calling
updatechanpolicy.
This commit replaces `AssertTopologyChannelOpen` with
`AssertChannelInGraph`, which asserts a given channel edge is found.
`AssertTopologyChannelOpen` only asserts a given edge has been received
via the topology subscription, while we need to make sure the channel is
in the graph before continuing our tests.
The new Fee TLV is not included in any other messages within the lnwire
package, so it currently has no fuzzing coverage. This fuzz target
directly tests the encoding/decoding of the TLV to get some coverage.
We will now cancel dust htlcs on the local/remote commits after
we decided to go onchain. This can be done because dust cannot
be enforced onchain and therefore there is no way to also reveil
the preimage onchain.
We distinguish between dangling and dust htlcs. This does not
change any logic but only introduces new types to later act on them
differently when we begin to fail dust htlcs earlier in a later
commit.
Refactor the part where we are failing back the incoming htlc
when the channel of the corresponding outgoing htlc is force
closed. We do this because in furture commits we separate the
logic when we fail back the incoming htlc (abandonForward).
Right now we fail abandon dust forwards and non-dust forwards
only when the commitment transaction is confirmed. Later we will
move the canceling of the upstream htlc when the commitment
transaction is broadcasted instead of waiting until the commitment
tx is confirmed. The reason for that is that dust cannot be enforced
onchain anyways so there is no reason to wait.
Because we need to remain backwards compatible with the old
`dust-threshold` value we set the default value for
`channel-max-fee-exposure` later in the program flow. Given
the restrictions of the sample config check we need to exclude
this value from the check.
Replace ambigious config value "dust-treshold" with a more clear
"channel-max-fee-exposure" exposure value. The old value is
deprecated and will be removed in the near future.
In this commit, we update an existing migration which at the time of
writing has not been included in a release. We update it so that it
converts the format used for MissionControl result encoding to use pure
TLV instead. The 3 structs that have been updated are: `mcHop`,
`mcRoute` and `paymentResult`.
So that we can use it in TLV encoding.
Also add this to the codec for channeldb migration 32 since we will be
using it there in an upcoming adjustment commit.
Add the TrueBoolean type along with its Record method. Also update the
Millisatoshi type with a Record method. Both of these will be used in an
upcoming commit which adjusts a mission control migration to use pure
TLV types.
Add a function `OptionFromPtr` that constructs an `Option` from a
pointer. The function signature is: `OptionFromPtr : *A -> Option[A]`.
This utility has proven useful in the taproot-assets project.
If the delivery address is P2TR, function InternalKeyForAddr checks its
existance in the wallet to return internal key for it in case it is a custom
taproot channel. It used to return the error returned by wallet.AddressInfo.
The error is now ignored if it is ErrAddressNotFound error. This fixes
"lncli closechannel --delivery_addr <external p2tr address" case.
This commit fixes the display of the amount when
selecting utxos for the sendcoins command and
combining it with the `-sweepall` flag. Prior this
would show the full balance of the wallet. Now it
shows the total amount of the selected utxos.
This commit updates `scripts/release.sh` to include a check for the
correct Go version before executing the release build. This ensures that
the release binaries are built with the specified Go version,
maintaining consistency and integrity for developer signatures.
Since most of our projects will use the same handler duo and apply the
config options the same way, let's make our lives easier and add a
default handler constructor.
This commit adds config options so that users can for both the console
logger and the file logger set the following things:
- disable the logger
- omit timestamps from log lines.
Start using the new slog handlers. With this commit we also remove the
need for the LogWriter since we let our LogRotator implement io.Writer
and pass that in to our log file handler.
This commit then also adds an implementation of the btclog.Handler
interface called `handlerSets` which basically lets us have a Handler
backed by many Handlers (we need one for the stdout and one for our log
file).
These are two separate concerns. So this commit splits them up and just
passes a LogWriter from the one to the other. This will become cleaner
in an upcoming commit where the Rotator will implement io.Writer and
there will no longer be a need for LogWriter.
Previously, LND log used to have such records:
[WRN] PEER: Peer(...): Link=%!v(MISSING) is not active
Now the short channel ID is printed there instead of "%!v(MISSING)".
During testing of this PR I figured out the the htlc interceptor
was not shutdown and started properly in terms of closing the
quit channel but also in terms of the relevant logs.
Describe the basics of the LND profiler so that node runners can
easily provide profile dumps to analyse faulty behaviour more
quickly.
Also update the description of LND's logging system.
In this commit, we make sig job handling when singing a next commitment
non-blocking by allowing the shutdown of a channel link to prevent
further waiting on sig jobs by the channel state machine. This addresses
possible cases where the aux signer may be shut down via a separate quit
signal, so the state machine could block indefinitely on receiving an
update on a sig job.
This is a requirement for replacing the quit channel with a Context.
The Done() channel of a Context is always recv-only, so all users of
that channel must not expect a bidirectional channel.
This adds the ContextGuard struct from the taproot-assets repository
that has been in use there for a while.
The context guard allows for easy creation of contexts that depend on a
central wait group and quit channel.
A context can either be created with a timeout and quit, meaning it will
cancel either on reaching the timeout or when the central quit channel
is closed.
Or a context can be created to block and use a timeout, meaning it will
_not_ cancel on quit but rather block the shutdown until it is completed
(or times out).
The third way is to create a context that just cancels on quit with no
timeout.
This commit adds a new --mac_root_key flag to both the lncli create and
lncli createwatchonly commands that allows the user to specify the
macaroon root key that should be used when creating the macaroon
database on wallet initialization.
This allows for deterministic wallet initialization and baking of
macaroons before the wallet is initialized.
For pure TLV messages such as Gossip 1.75 and Bolt12 messages, there are
defined TLV ranges for unsigned data vs signed data. The signed ranges are
the inclusive ranges of 0 to 159 and 1000000000 to 2999999999.
What we do here in this commit is: 1) update the number of main types
generated so that they include type 160 which is the first type in the
un-signed range and we will make use of this type for signature fields
in gossip messages. Next, we add two marker types so that we can define
the rest of the ranges.
Here we return the balance deltas from evaluateHTLCView rather than
passing in references to variables that will be modified. It is a
far cleaner and compositional approach which allows readers of this
code to more effectively reason about the code without having to
keep the whole codebase in their head.
This commit observes that processAddEntry and processRemoveEntry
are only invoked at a single call-site. Here we inline them at their
call-sites, which will unlock further simplifications of the code
that will allow us to remove pointer mutations in favor of explicit
expression oriented programming.
We also delete the tests associated with these functions, the overall
functionality is implicitly tested by the TestEvaluateHTLCView tests.
We had four for-loops in evaluateHTLCView that were exact mirror
images of each other. By making use of the new ChannelParty and
Dual facilities introduced in prior commits, we consolidate these
into two for-loops.
This commit simplifies how we compute the commitment fee rate based
off of the live updates. Prior to this commit we processed all of
the FeeUpdate paymentDescriptors of both ChannelParty's. Now we only
process the last FeeUpdate of the OpeningParty
This commit moves the collection of updates behind a Dual structure.
This allows us in a later commit to index into it via a ChannelParty
parameter which will simplify the loops in evaluateHTLCView.
This commit removes another raw boolean value and replaces it with
a more clear type/name. This will also assist us when we later try
and consolidate the logic of evaluateHTLCView into a single
coherent computation.
In line with previous commits we are progressively removing the
mutateState argument from this call stack for a more principled
software design approach.
NOTE FOR REVIEWERS:
We take a naive approach to updating the tests here and simply
take the functionality we are removing from evaluateHTLCView and
run it directly after the function in the test suite.
It's possible that we should instead remove this from the test
suite altogether but I opted to take a more conservative approach
with respect to reducing the scope of tests. If you have opinions
here, please make them known.
The field is optional. It stores inputs needed to produce signed commit tx using
chantools scbforceclose, which calls function GetSignedCommitTx. New backups
have this field filled if commit tx is available (for all cases except when DLP
is active). If a backup has this data, the field is filled from it, otherwise it
is kept empty.
Modified test function genRandomOpenChannelShell to cover new types of channels
(simple taproot channel and custom channel) and to cover combinations of bits.
Make sure that TapscriptRoot field is properly packed and unpacked.
Previous to this change taproot assets channels and simple taproot channels were
considered the same in the context of chanbackup package, since they stored the
same data. In the following commits we are adding the data needed to produce a
signed commitment transaction from a SCB file and in order to do that we need to
add more fields and a custom channel gets one additional field (TapscriptRoot)
compared to a simple taproot channel. So now we have to distinguish these kinds
of channels in chanbackup package.
See PR https://github.com/lightningnetwork/lnd/pull/8183 for more details.
This pure function creates signed commit transaction, using various
inputs passed as struct TaprootSignedCommitTxInputs and a signer.
This is needed to be able to store the inputs without a signature
in SCB and sign the transaction in chantools scbforceclose.
See https://github.com/lightningnetwork/lnd/pull/8183/files#r1423959791
In this commit we observe that the previous commit reduced the role
of this function to a single assignment statement with numerous newly
irrelevant parameters. This commit makes the choice of inlining it at
the two call-sites within evaluateHTLCView and removing the funciton
definition entirely. This also allows us to drop a huge portion of
newly irrelevant test code.
This commit redoes the API and semantics of processFeeUpdate to make
it consistent with the semantics of it's sister functions. This is
part of an ongoing series of commits to remove mutateState arguments
pervasively from the codebase.
As with the previous commit this makes state mutation the caller's
responsibility. This temporarily increases code duplication at the
call-sites, but this will unlock other refactor opportunities.
This commit begins the process of moving towards a more principled
means of state tracking. We eliminate the mutateState argument from
processAddEntry and processRemoveEntry and move the responsibility
of mutating said state to the call-sites.
The current call-sites of these functions still have their *own*
mutateState argument which will be eliminated during upcoming commits.
However, following the principle of micro-commits I opted to break
these changes up to make review simpler.
The purpose of this commit is to begin the process of packing
symmetric fields into the newly introduced Dual structure. The
reason for this is that the Dual structure has a handy indexing
method where we can supply a ChannelParty and get back a value.
This will cut down on the amount of branching code in the main
lines of the codebase logic, making it easier to follow what is
going on.
fix the signature of the Encode method so that it does in fact implement
the Serializable interface defined in the lnwire21 package. This differs
slightly from the interface defined in the main lnwire package.
Similar to the sweeper, when we're about to make a new breach
transaction, we ask the sweeper for a new change address, if it has one.
Then when we go to publish, we notify broadcast.
In this commit, we start to use the AuxSweeper (if present) to obtain a
new extra change addr we should add to the sweeping transaction. With
this, we'll take the set of inputs and our change addr, and then maybe
gain a new change addr to add to the sweep transaction.
The extra change addr will be treated as an extra required tx out,
shared across all the relevant inputs. This'll also be used in
NeedWalletInput to make sure that we add an extra input if needed to be
able to pay for the change addr.
In this commit, we add a new AuxSweeper interface. This'll take a set of
inputs, and a change addr for the sweep transaction, then optionally
return a new sweep output to be added to the sweep transaction.
We also add a new NotifyBroadcast method. This'll be used to notify
that we're _about_ to broadcast a sweeping transaction. The set of
inputs is passed in, which allows the caller to prepare for the ultimate
broadcast of the sweeping transaction.
We also add ExtraTxOut to BumpRequest pass fees to NotifyBroadcast. This
allows the callee to know the total fee of the sweeping transaction.
In this commit, we refactor all the other constructors for the input to
use MakeBaseInput. We also add a new set of functional options as well.
This'll be useful later on to ensure that new options are properly
applied to all the input types.
This will be used by external callers to modify the way we resolve
contracts on chain. For a given contract, we'll store an extra "blob",
that will later be presented during the sweeping phase.
This commit was added to the 0-19-staging branch recently and therefore
didn't make it into a previous part yet. So it's unrelated to the
changes in this part but is required for the whole custom channel saga.
The package provides type GoroutineManager which is used to launch goroutines
until context expires or the manager is stopped. Stop method blocks until all
started goroutines stop.
Original code by Andras https://go.dev/play/p/HhRpE-K2lA0
Adjustments and tests by Boris.
This document contained a link to a MuSig2 draft BIP, which is no longer available. I have updated the link with the link to BIP327 in the bitcoin/bips repository.
Update musig2.md
i've removed the extra spaces and standardized the formatting for this section
This commit uses the new SetBit helper for setting the Bolt11 route
blinding required bit in a generated invoice. This helper will take care
of setting the dependent bits of the feature bit (namely the route
blinding and TLV feature bits and will use the required variant of
both).
This commit adds a new SetBit helper function in the features package
along with a test for it. SetBit sets the given feature bit on the
given feature bit vector along with any of its dependencies. If the bit
is required, then all the dependencies are also set to required,
otherwise, the optional dependency bits are set. Existing bits are only
upgraded from optional to required but never downgraded from required to
optional.
This commit renames the previous MissionControl to MissionController and
the previous MissionController interface to MissionControlQuerier. This
is done because soon the (new) MissionController will back multiple
namespaced MissionControl instances. For now, it just houses a single
MissionControl in the default namespace.
This commit also replaces the MissionControl's `now` function with a
`clock.Clock`.
In this commit, the mission control store is migrated such that all
existing pairs which are currently stored directly in the top level
results bucket are now instead moved to a "default" namespace bucket.
Note that this migration is not yet invoked in this commit. The
migration will be invoked in the same commit that starts writing and
reading the new format.
So that `missionControlStore` can be unaware of the backing DB structure
it is writing to. In an upcoming commit when we change mission control
to write to namespaced buckets instead, we then only need to update the
`namespacedDB` implementation.
Prior to this commit, lnd could become unresponsive to shutdown signals
during the `IsSynced` check. Since the `IsSynced` check can occasionally
take a long time to complete, this could result in lnd failing to shut
down promptly.
In this commit, we remove the global application level lock from the
postgres backend. This lock prevents multiple write transactions from
happening at the same time, and will also block a writer if a read is on
going. Since this lock was added, we know always open DB connections
with the strongest level of concurrency control available:
`LevelSerializable`. In concert with the new auto retry logic, we ensure
that if db transactions conflict (writing the same key/row in this
case), then the tx is retried automatically.
Removing this lock should increase perf for the postgres backend, as now
concurrent write transactions can proceed, being serialized as needed.
Rather then trying to handle concurrency at the application level, we'll
set postgres do its job, with the application only needing to retry as
necessary.
Add a new mcRoute type that houses the data about a route that MC
actually uses. Then add a migration (channeldb/migration32) that
migrates the existing store from its current serialisation to the new,
more minimal serialisation.
In preparation for the commit which will add the main logic for
migration 32 (which will migrate the MC store to use a more minimal
encoding), this commit just adds some of the code that the migration
will need to the package.
We add the new custom records encoding/decoding logic to the "frozen"
lnwire21 package. We can do this because nothing uses this logic yet. If
the custom records logic changes, the changes should _not_ be added to
the lnwire21 version.
To prevent the need to copy the entire onion_error.go file for a new
Mission Control migration, this commit just updates the existing
lnwire21/onion_error.go file with the new CodeInvalidBlinding code. The
lnwire21 should not really ever be updated but adding a new code should
be fine as it does not affect old migrations since this is a new code.
This commit adds an immutable Filter method to the linked List API.
This is useful because there are several instances wherein we iterate
through the linked List and only process a subset of it in some way
or another.
With this commit we populate additional information about the close
outputs (including potential custom channel data) in the close update
RPC message.
This will allow custom channels to find out how the additional close
outputs look like on chain and what data they might commit to.
We also hook up the aux custom data formatter, so it can format the
custom channel data to JSON.
In this commit, we move to add the internal key to the delivery addr. This way, we give the aux chan closer the extra information it may need to properly augment the normal co-op close process.
As is customary when preparing for the next major (or minor) release, we
bump the version to .99 to allow us to set the minimum required version
to something like v0.18.4-beta in lndclient and it would still accept
the master branch (even if that target version hasn't been
released/tagged yet).
This commit introduces a basic integration test for the invoice
HTLC modifier. The test covers scenarios where an invoice is settled with a
payment that is less than the invoice amount, facilitated by the invoice
HTLC modifier.
With this commit we encode the custom records as a TLV stream into the
custom channel data field of the invoice HTLC.
This allows the custom data parser to parse those records and replace it
with human-readable JSON on the RPC interface.
This commit enhances the itest LND node harness to include support for
the new `HtlcModifier` RPC endpoint.
At the same time we move another method to the correct file.
This commit introduces a singleton invoice HTLC modifier RPC server and
an endpoint to activate it. The server interfaces with the internal
invoice HTLC modifier interpreter, handling the marshalling between RPC
types and internal formats.
This commit updates the invoice registry to utilize the settlement
interceptor during the invoice settlement routine. It allows the
interceptor to capture the invoice, providing interception clients an
opportunity to determine the settlement outcome.
This commit introduces a new invoice htlc interceptor service
that intercepts invoice HTLCs during their settlement phase. It forwards
HTLCs to a subscribed client to determine their settlement outcomes.
This commit also introduces an interface to facilitate integrating the
interceptor with other packages.
This commit adds an optional data parser that can inspect and in-place
format custom data of certain RPC messages.
We don't add an implementation of the interface itself, as that will be
provided by external components when packaging up lnd as a bundle with
other software.
To make sure we attempt to read the results of the sig batches in the
same order they're processed, we sort them _before_ submitting them to
the batch processor.
Otherwise it might happen that we try to read on a result channel that
was never sent on because we aborted due to an error.
We also use slices.SortFunc now which doesn't use reflection and might
be slightly faster.
In this commit, we start to use the new AuxSigner to obtain+verify aux sigs for all second level HTLCs. This is similar to the existing SigPool, but we'll only attempt to do this if the AuxSigner is present (won't be for most channels).
This commit makes an lnwallet.BlockChainIO available to the gossiper and
uses it to construct a helper that can be used to fetch the pk script
for a given SCID. This will be used for channel announcement
verification in an upcoming commit.
Add new FirstBlockHeight and BlockRange TLV fields to the
GossipTimestampRange message. This will be used to query for Gossip 1.75
messages which are timestamped using block height instead of Unix
timestamps.
In preparation for Gossip 1.75, we add new TLV's to the `ChannelReady`
message. Namely: `AnnouncementBitcoinNonce` and `AnnouncementNodeNonce`.
These will be used to exchange nones required for producing the partial
signature to be send in the `AnnouncementSignatures2` message.
The type numbers for these new fields are even because if they are set,
then a peer is expecting its peer to understand gossip 1.75 and the new
fields.
In preparation for adding the new ChannelAnnouncement2 message along
with a ChannelAnnouncement interface, we rename the existing message to
ChannelAnnouncement1.
In preparation for adding a new message, AnnounceSignatures2 along with
an AnnounceSignatures interface, we rename the existing message to
AnnounceSignatures1.
Due to a recent refactor, the HTLCs are no longer an exported type.
Custom channels need access to those updates, so we provide them in a
read-only manner.
In this commit, we add a new aux signer interface that's meant to mirror the SigPool. If present, this'll be used to (maybe) obtain signatures for second level HTLCs for certain classes of custom channels.
For the initiator, once we get the signal that the PSBT has been
finalized, we'll call into the aux funder to get the funding desc. For
the responder, once we receive the funding_created message, we'll do the
same.
We now also have local+remote aux leaves for the commitment transaction.
Some old TODO comments that in retrospect aren't required anymore are
removed as well.
In this commit, we modify the aux funding work flow slightly. We won't
be able to generate the full AuxFundingDesc until both sides has
sent+received funding params. So we'll now only attempt to bind the
tapscript root as soon as we send+recv the open_channel message.
We'll now also make sure that we pass the tapscript root all the way
down into the musig2 session creation.
In this commit, we make a new `AuxFundingController` interface capable of processing messages off the wire. In addition, we can use it to abstract away details w.r.t how we obtain a `AuxFundingDesc` for a given channel.
We'll now use this whenever we get a channel funding request, to make sure we pass along the custom state that a channel may require.
This struct will house all the information we'll need to do a class of custom channels that relies primarily on adding additional items to the tapscript root of the HTLC/commitment/funding outputs.
In this commit, we bring the timeout resolver more in line with the
success resolver by using the sweeper to handle the HTLC offered remote
timeout outputs. These are outputs that we can sweep directly from the
remote party's commitment transaction when they broadcast their version
of the commitment transaction.
With this change, we slim down the scope slightly by only doing this for
anchor channels. Non-anchor channels will continue to use the
utxonursery for this output type for now.
The objective of this commit is to make paymentDescriptor a private
data structure so we can quarantine it to the lnwallet package.
To accomplish this we had to prevent it from leaking out via the
arguments or return values of the public functions in lnwallet.
This naturally had consequences for the htlcswitch package as we
choose other mechanisms for tracking the data that paymentDescriptor
was responsible for.
Astoundingly, this was highly successful and allowed us to remove
a ton of redundant code. The diff for this commit represents a
substantial reduction in total lines of code as well as extraneous
arguments and return values from key functions.
This also sets the stage for future commits where we actually will
be attempting to rid lnwallet of paymentDescriptor completely.
probabilityFormula() is expected to return an error if capacity is 0, so
we should exclude that case from fuzzing.
Previously it was attempted to avoid this case by seeding the corpus
with an input that had capacity 1. That is not an effective solution
since the fuzzer can still generate an input with capacity 0.
Currently if an incorrect number of timestamps is given, we fail later
on in the GossipSyncer. It makes more sense to fail right away, since we
already do that for incorrect SCID formats (e.g., unsorted or duplicate
SCIDs). There is already a matching check in Encode for incorrect number
of timestamps, so adding this check to Decode makes things symmetric.
This is part of a systematic removal of PaymentDescriptor from the public
API of the lnwallet package. This marks the last change needed before we
make the PaymentDescriptor structure private.
Here we add a function that is capable of recovering LogUpdates from
paymentDescriptors and we refactor the lnwallet code to use this
rather than doing JIT inline construction of the LogUpdates.
In this commit we track the ChannelID on the PaymentDescriptor.
This will be useful in upcoming commits that need to be able to
reconstruct lnwire.Message values from PaymentDescriptors as the
Messages that are exchanged to update channel state all include
the ChannelID.
In this commit we introduce a convenience method to LightningChannel
to allow us to quickly grab the channel id. This will be important
in upcoming commits where we need to remember the ChannelID to
reconstruct update messages.
We might be trying to send an invoice amount that's greater than the size of the channel, but once you factor in the custom channel logic, an actual HTLC can be sent over the channel to pay that larger payment.
As a result, we'll skip over this check if a have a custom HTLC.
With this commit we make sure the first hop custom records aren't lost
on restart/resume of a payment, so we persist it as part of the
PaymentCreationInfo struct.
This commit extends the forward HTLC intercept response with fields that
can be used in conjunction with a `ResumeModified` action to modify the
intercepted HTLC p2p message.
Introduce `ResumeModified` action to resume standard behavior of a p2p
message with optional modifications as specified by the client during
interception.
- Introduce the field `CustomRecords` to the type `UpdateFulfillHtlc`.
- Encode and decode the new field into the `ExtraData` field of the
`update_fulfill_htlc` wire message.
- Empty `ExtraData` field is set to `nil`.
- Introduce the field `CustomRecords` to the type `UpdateAddHtlc`.
- Encode and decode the new field into the `ExtraData` field of
the `update_add_htlc` wire message.
Because we restrict custom SCID aliases to be in a specific range, we
export the range start and end values so a user of the RPCs we're going
to add in the next commits can adjust their values to fit within the
range.
With the new RPC calls that we are going to add in the next commits, it
will be possible for users to add (local only, non-gossipped) SCID
aliases for channels. Since those will be in the same range as the ones
given out by RequestAlias, we need to make sure that when we generate a
new one that it doesn't collide with an already existing one.
In this commit, we expand some of the existing chan sync tests to cover
taproot channels (the others already did). Along the way, we always
assert that the `PartialSig` is populated on retransmission. In
addition, we now send the new commit sig rather than the existing
in-memory one to test the new logic that re-signs the commitment.
In this commit, we fix an existing bug with the taproot channel type
that can cause force closes if a peer disconnects while attempting to
send the commitment signature.
Before this commit, since the `PartialSig` we send is never committed to
disk, the version read wouldn't contain the musig2 partial sig. We never
write these signatures to disk, as each time we make a new session, we
need to generate fresh nonces to avoid nonce-reuse.
Due to the above interaction, if we went to re-send a signature after a
disconnection, the `CommitSig` message we sent wouldn't actually contain
a `PartialSigWithNonce`, causing a protocol error.
Previously, the SQL implementation of the invoice query simply
converted the start and end timestamps to time and used them
in SQL queries to check for inclusivity. However, this logic
failed when the start and end timestamps were equal.
This commit addresses and corrects this issue.
Previously SQL invoice updater ignored the set ID hint when updating an
AMP invoice resulting in update subscriptions returning all of the AMP
state as well as all AMP HTLCs. This commit synchornizes behavior with
the KV implementation such that we now only return relevant AMP state
and HTLCs when updating an AMP invoice.
When fetching an AMP invoice we sometimes filter HTLCs to selected set
IDs, however we always kept the full AMP state which is irrelevant as it
contains state for all AMP payments. This was a side effect of
UpdateInvoice needing to serialize the whole invoice when storing after
an update but it is an unwanted "feature" as users will need to filter
to relevant set when listing an AMP payment or subsribing to an update.
This commit squashes the below operations for a net result where
we have an expanded capability of assessing pending updates. This
is made possible by packing the components into Duals in the prior
commits. We squash the operations to simplify review.
htlcswitch+lnwallet: rename PendingLocalUpdateCount
lnwallet: complete pending update queries API for LightningChannel
lnwallet+htlcswitch: consolidate NumPendingUpdates using ChannelParty
This commit makes the observation that we can cleanly define the
NumPendingUpdates function using a single expression by taking
advantage of the relevant fields being properly packed into Duals.
This is yet another commit that packs a symmetric structure into
a Dual. This is the last one needed for the time being to consolidate
Num{X}UpdatesPendingOn{Y} functions into a single one.
This commit, like the last one packs the update logs into a symmetric
Dual structure. This will allow us to index into them more concisely
in higher order logic.
This commit packs the LightningChannel's localCommitmentChain and
remoteCommitmentChain into a Dual structure for better symmetric
access. This will be leveraged by an upcoming commit where we want
to more concisely express how we compute the number of pending
updates.
In this commit, we start to thread thru the new aux tap leaf structures to all relevant areas. This includes: commitment outputs, resolution creation, breach handling, and also HTLC scripts.
This may be useful for custom channel types that base everything off the index (a global value) rather than the output index (can change with each state).
In this commit, we add a TLV blob to the PaymentDescriptor struct. We also now thread through this value from the UpdateAddHTLC message to the PaymentDescriptor mapping, and the other way around.
In this commit, we add a new AuxLeafStore which can be used to dynamically fetch the latest aux leaves for a given state. This is useful for custom channel types that will store some extra information in the form of a custom blob, then will use that information to derive the new leaf tapscript leaves that may be attached to reach state.
In this commit, we also add the custom TLV blob to the internal commitment struct that we use within the in-memory commitment linked list.
This'll be useful to ensure that we're tracking the current blob for our in memory commitment for when we need to write it to disk.
In this commit, for each channel, we'll now start to store an optional custom blob. This can be used to store extra information for custom channels in an opauqe manner.
In this commit, we update all the taproot scripts to also accept an
optional aux leaf. This aux leaf can be used to add more redemption
paths for advanced channels, or just as an extra commitment space.
With this commit, the channel is now aware of if it's a musig2 channel, that also has a tapscript root. We'll need to always pass in the tapscript root each time we: make the funding output, sign a new state, and also verify a new state.
This isn't hooked up yet to the funding manager, but with this commit, we can now start to write internal unit tests that handle musig2 channels with a tapscript root.
In this commit, we consolidate the root bucket TLVs into a new struct.
This makes it easier to see all the new TLV fields at a glance. We also
convert TLV usage to use the new type param based APis.
This commit hooks up the banman to the gossiper:
- peers that are banned and don't have a channel with us will get
disconnected until they are unbanned.
- peers that are banned and have a channel with us won't get
disconnected, but we will ignore their channel announcements until
they are no longer banned. Note that this only disables gossip of
announcements to us and still allows us to open channels to them.
This commit introduces a ban manager that marks peers as banned if
they send too many invalid channel announcements to us. Expired
entries are purged after a certain period of time (currently 48 hours).
This commit adds the ability to store closed channels by scid in
the database. This will allow the gossiper to ignore channel
announcements for closed channels without having to do any
expensive validation.
This commit fixes the instantiation of the BlindingPoint member of
PaymentDescriptor during the conversion from persisted LogUpdates.
Previously, the blinding point was not set correctly. The test from the
previous commit is also updated to now assert that this behaviour is now
correct.
This commit adds a new route blinding itest that demonstrates that the
reloading and re-forwarding of an UpdateAddHTLC message on restart
currently is done incorrectly for a blinded path payment. This is due to
the fact that the blinding point member is not currently set correctly.
This is fixed in the next commit which will also change the test to
assert that the behaviour is now correct.
Provide sample of command to create macaroon with similar rights as admin.macaroon to be used with 3rd party applications.
Co-Authored-By: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
We want to export some of our CLI code to re-use in other projects. But
in Golang you cannot import code from a `main` package.
So we need to move the actual code into its own package and only have
the `func main()` in the `main` package.
We will not add a buffer to the chan policy for blinded paths in case
the sender amount violates the minHTLC restriction in the first place.
Moreover we disgard a route fast if the payment amount is smaller than
the minHTLC along the route.
This lets us get rid of the mutex usage there. We also shift the algo slightly to increment by 1, then use that as the next value, which plays nicer with the atomics.
This commit introduces the `CustomRecords` type in the `lnwire` package,
designed to hold arbitrary byte slices. Each entry in this map can
associate with TLV type values that are greater than or equal to 65536.
Removes a check where we would NOT allow to create a blinded invoice
with an expiry (invoice expiry in seconds considered as block time)
lower than the min_final_ctlv_delta.
In this commit, we rename the files as assembler.go houses the primary
interfaces/abstractions of the package. In the rest of the codebase,
this file is near uniformly called interface.go, so we rename the file
to make the repo more digestible at a scan.
ChanUpdate timestamps are now restircted so that they cannot be
more than two weeks into the future. Moreover channels with both
timestamps in the ReplyChannelRange msg either too far in the past
or too far in the future are not queried.
Moreover fix unitests.
In this commit, we fix a bug that would cause a global message router to
be stopped anytime a peer disconnected. The global msg router only
allows `Start` to be called once, so afterwards, no messages would
properly be routed.
With this commit, we allow the `MsgRouter` to be available in the
`ImplementationCfg`. With this, programs outside of lnd itself are able
to now hook into the message processing flow to direct handle custom
messages, and even normal wire messages.
Over time with this, we should be able to significantly reduce the size
of the peer.Brontide struct as we only need all those deps as the peer
needs to recognize and handle each incoming wire message itself.
In this commit, we add a new abstract message router. Over time, the
goal is that this message router replaces the logic we currently have in
the readHandler (the giant switch for each message).
With this new abstraction, can reduce the responsibilities of the
readHandler to *just* reading messages off the wire and handing them off
to the msg router. The readHandler no longer needs to know *where* the
messages should go, or how they should be dispatched.
This will be used in tandem with the new `protofsm` module in an
upcoming PR implementing the new rbf-coop close.
Initially in lnd, we didn't store the extra TLV data that could be
dangling off of gossip messages. This was fixed initially in lnd v0.5
with this PR: https://github.com/lightningnetwork/lnd/pull/1825.
Within the PR, we incorrect set the `ExtraOpaqueData` (extra TLV blob)
of the `ChannelAnnouncement` to the value stored in `edge`, which is
actually our channel update. As 6-ish years ago we didn't yet have
anything that used the TLV gossip fields, this went unnoticed.
Fast forward to 2024, we shipped an experimental version of inbounbd
fees. This starts to store additional data in the `ExtraOpaqueData`
field, the TLV for the inbound fee. Initially, everything is valid when
the first `ChannelAnnouncement` is sent, but as soon as a user attempts
to set an inbound fee policy, we'd incorrectly swap in that new
serialized TLV for the _channel announcement_:
841e24399c (diff-1eda595bbebe495bd74a6a0431c46b66cb4e8b53beb311067c010feac2665dcbR2560).
Since we're just trying to generate a new `channel_update`, we don't
also regenerate the signature for the `channel_announcement` message. As
a result, we end up storing a `channel_announcement` with an invalid sig
on disk, continuing to broadcast that to peers.
Add a new bumpforceclosefee rpc endpoint to the wallet server.
Move the logic from the lncli level to the wallet server rpc level.
This is more in line with a proper client-server design.
wallet lncli: use new bumpforceclosefee endpoint.
Besides using the new bumpforceclosefee rpc endpoint we also enable the
bumping of taproot anchor channels.
This commit removes the call toe `ListLeasedOutputs` in `LeaseOutput` -
the returned info from `ListLeasedOutputs` can easily be accessed via
`FetchInputInfo` and this info is only used at one callsite.
`ListLeasedOutputs` then becomes unnecessary here, plus it's very slow
and needs to be refactored in `btcwallet` instead.
This commit replaces the usage of `FetchInputInfo` with
`FetchOutpointInfo` and `FetchDerivationInfo` to remove unncessary
fetching of the derivation path.
In this commit, we add a model for the linear fee function we use in lnd
for fee bumping. This models can be used to reproduce the issue reported
in https://github.com/lightningnetwork/lnd/issues/8741, and can also be
shown that that bug fix resolves a counter example found by the model
checker.
@ -11,8 +11,9 @@ Steps for reviewers to follow to test the change.
- [ ] Bug fixes contain tests triggering the bug to prevent regressions.
### Code Style and Documentation
- [ ] The change obeys the [Code Documentation and Commenting](https://github.com/lightningnetwork/lnd/blob/master/docs/code_contribution_guidelines.md#CodeDocumentation) guidelines, and lines wrap at 80.
- [ ] Commits follow the [Ideal Git Commit Structure](https://github.com/lightningnetwork/lnd/blob/master/docs/code_contribution_guidelines.md#IdealGitCommitStructure).
- [ ] The change is not [insubstantial](https://github.com/lightningnetwork/lnd/blob/master/docs/code_contribution_guidelines.md#substantial-contributions-only). Typo fixes are not accepted to fight bot spam.
- [ ] The change obeys the [Code Documentation and Commenting](https://github.com/lightningnetwork/lnd/blob/master/docs/development_guidelines.md#code-documentation-and-commenting) guidelines, and lines wrap at 80.
- [ ] Commits follow the [Ideal Git Commit Structure](https://github.com/lightningnetwork/lnd/blob/master/docs/development_guidelines.md#ideal-git-commit-structure).
- [ ] Any new logging statements use an appropriate subsystem and logging level.
- [ ] Any new lncli commands have appropriate tags in the comments for the rpc in the proto file.
- [ ] [There is a change description in the release notes](https://github.com/lightningnetwork/lnd/tree/master/docs/release-notes), or `[skip ci]` in the commit message for small changes.
go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=${{ env.RELEASE_VERSION }}" ./cmd/lnd
go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=${{ env.RELEASE_VERSION }}" ./cmd/lncli
```
The `-mod=vendor` flag tells the `go build` command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.
#? flakehunter-unit: Run the unit tests continuously until one fails
flakehunter-unit:
@$(call print, "Flake hunting unit test.")
scripts/unit-test-flake-hunter.sh ${pkg}${case}
#? flakehunter-parallel: Run the integration tests continuously until one fails, running up to ITEST_PARALLELISM test tranches in parallel (default 4)
flakehunter-parallel:
#? flakehunter-itest-parallel: Run the integration tests continuously until one fails, running up to ITEST_PARALLELISM test tranches in parallel (default 4)
flakehunter-itest-parallel:
@$(call print, "Flake hunting ${backend} integration tests in parallel.")
while[$$? -eq 0];do make itest-parallel tranches=1parallel=${ITEST_PARALLELISM}icase='${icase}'backend='${backend}';done
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
