With recent changes, particularly using postgres, we can finish the HTLC before
we force close, so the test fails. Insert an explicit hangup here so we reliably
get the result we expect:
```
@pytest.mark.parametrize("anchors", [False, True])
@unittest.skipIf(TEST_NETWORK != 'regtest', 'elementsd anchors not supportd')
def test_anchorspend_using_to_remote(node_factory, bitcoind, anchors):
"""Make sure we can use `to_remote` output of previous close to spend anchor"""
# Try with old output from both anchor and non-anchor channel.
l4_opts = {}
if anchors is False:
l4_opts['dev-force-features'] = "-23"
l1, l2, l3, l4 = node_factory.get_nodes(4, opts=[{},
{},
{'disconnect': ['-WIRE_UPDATE_FULFILL_HTLC']},
l4_opts])
# Give l2 some funds, from a to-remote output. It will have to spend
# this to use anchor.
node_factory.join_nodes([l4, l2])
# l4 unilaterally closes, l2 gets to-remote with its output.
l4.rpc.pay(l2.rpc.invoice(100000000, 'test', 'test')['bolt11'])
> wait_for(lambda: only_one(l4.rpc.listpeerchannels()['channels'])['htlcs'] != [])
tests/test_closing.py:4183:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
success = <function test_anchorspend_using_to_remote.<locals>.<lambda> at 0x7f8293e7fe20>
timeout = 180
def wait_for(success, timeout=TIMEOUT):
start_time = time.time()
interval = 0.25
while not success():
time_left = start_time + timeout - time.time()
if time_left <= 0:
> raise ValueError("Timeout while waiting for {}".format(success))
E ValueError: Timeout while waiting for <function test_anchorspend_using_to_remote.<locals>.<lambda> at 0x7f8293e7fe20>
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also gets some new timemono helpers, but we don't use them (yet).
Changelog-Fixed: Compilation on FreeBSD.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Closes: https://github.com/ElementsProject/lightning/issues/7758
Apparently clboss gives us \u UTF codes. We don't support that (use UTF-8 directly)
```
126 ../sysdeps/x86_64/multiarch/strlen-vec.S: No such file or directory.
(gdb) bt
label=label@entry=0x63e2f9604db9 "char *[]") at ccan/ccan/tal/str/str.c:137
complete=complete@entry=0x7ffe9090b0f6, destroyed=destroyed@entry=0x7ffe9090b0f7) at lightningd/plugin.c:773
```
Reported-by: Ken Sedgwick
Fixes: #8338
Changelog-None: broken in this release
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Checking a signature is a CPU-intensive operation that should be performed only
if gossmap doesn't already have the channel announcement in question and we're
not already checking for the announcement's UTxO.
Changelog-Fixed: `gossipd` doesn't waste CPU cycles checking signatures on channel announcements that are already known
Issue: https://github.com/ElementsProject/lightning/issues/7972
Ubuntu 20.04 reached its end-of-life (EOL) on April 15, 2025, and GitHub Actions subsequently removed the Ubuntu 20.04 LTS runner. As a result, we must upgrade all workflow runners to at least Ubuntu 22.04. Currently, this update has only been applied to pypi.yml for core-lightning.
Changelog-None.
The reproducible build is currently failing on Ubuntu Noble, causing the daily `Repro Build Nightly` GitHub action to fail and triggering email notifications.
This update resolves the issue by adjusting the default sqlite3 version and checksums to match the packages now available in Noble, allowing the image to build successfully.
Changelog-None.
We asserted that a node would be connected, but now we always call the
chanbackup peer_connected hook, it might not be that fast:
```
2025-05-17T08:37:58.7193306Z def test_funding_fail(node_factory, bitcoind):
...
2025-05-17T08:37:58.7204037Z # Restart l2 without ridiculous locktime.
2025-05-17T08:37:58.7204546Z del l2.daemon.opts['watchtime-blocks']
2025-05-17T08:37:58.7204990Z l2.restart()
2025-05-17T08:37:58.7206578Z l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
2025-05-17T08:37:58.7207072Z
2025-05-17T08:37:58.7207522Z # We don't have enough left to cover fees if we try to spend it all.
2025-05-17T08:37:58.7208203Z with pytest.raises(RpcError, match=r'not afford'):
2025-05-17T08:37:58.7208761Z l1.rpc.fundchannel(l2.info['id'], funds)
2025-05-17T08:37:58.7209203Z
2025-05-17T08:37:58.7209598Z # Should still be connected (we didn't contact the peer)
2025-05-17T08:37:58.7210243Z assert only_one(l1.rpc.listpeers()['peers'])['connected']
2025-05-17T08:37:58.7210875Z l2.daemon.wait_for_log('Handed peer, entering loop')
2025-05-17T08:37:58.7211439Z > assert only_one(l2.rpc.listpeers()['peers'])['connected']
2025-05-17T08:37:58.7211912Z E assert False
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It can definitely happen (it's not dead yet):
```
2025-05-17T08:44:48.9829150Z lightningd-2 2025-05-17T08:36:43.597Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#3: gossipd gave channel_update in CGOSSIP_CHANNEL_ANNOUNCED_DYING? update=01027cf5d46590c45eb608a7e07a7b123c998ac12a655ce11004c9d9bb59f83698e8760d7b992df0dcf925f6732503de55d596aadaa35f0a8837cee58e0769172d2a06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006f00000100006828493f010200060000000000000000000000010000000a000000003b023380
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We picked one node and iterated. This means we would only sent to 1,
not 2 nodes if we picked the last (common if there were only a few
peers). But it also means we would generally send to the same pair of
nodes, which isn't great for redundancy.
Rework to be more random.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we've make it only on existing channels, and not have to call
listdatastore every time, that means we can safely turn it on by
default.
Changelog-Added: Protocol: we now offer peer storage to any peers who create a channel.
Changelog-Deprecated: Config: `--experimental-peer-storage` (it's now the default).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When peer backup is enabled by default, it puts things in the datastore, breaking
this assumption. Narrow the test to examine the specific funder directory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They will in fact get truncated, and never restore. Large nodes should be using some real
backup strategy!
For this reason, we split "peer_backup" support into send vs store support, so we can
turn off send of our own without disabling storing/sending theirs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We already get the connected hook, so in there we can add to a hash
table of suitable peers. Rather than subscribe to disconnection, we
simply remove the peer if a sendcustommsg fails.
This does make after_send_scb_single() a bit more complex, since it
needs this specific node_id: we use a `struct info` per node and a
pointer to a shared "idx" reference counter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By keeping a local hash table, we won't have to look up every time.
We still write to the datastore when it changes, and we need to
initialize it at plugin start.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Eclair only sets `batch_size` and `funding_txid` when its a batch larger than 1.
Adjust how we send and what we expect to receive to match this.
Changelog-None
And we timeout in:
l2.wait_for_channel_onchain(l1.info['id'])
Because the unilateral wins. This is because we tell closingd to
negotiate with a min feerate of 3750, but when it does and we check it,
our min has jumped, lightningd rejects it:
... That's below our min XXX for weight 772 at feerate 7000
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The spec used to say you had to wait for channel to be ready, *and* 6
depth before exchanging signatures. Now the 6 depth requirement is only
on the actual announcing of the channel: you can send sigs any time.
This means our state machine goes from:
NOT_USABLE -> NOT_DEEP_ENOUGH -> NEED_PEER_SIGS -> ANNOUNCED
to:
NOT_USABLE -> NEED_PEER_SIGS -> NOT_DEEP_ENOUGH -> ANNOUNCED
However, this revealed that our state machine is insufficient, so
rework it to be more general and understandable. In particular,
check for unexpected state transitions, and thus document them.
Note that cg->sent_sigs replaces channel->replied_to_announcement_sigs,
too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now exchange `announcement_signatures` as soon as we're ready, rather than waiting for 6 blocks (as per recent BOLT update)
They can all call get_block_height(); the extra argument confused me and
I thought they were called before the block height was actually updated.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want a more fine-grained approach, so we now have:
1. update_channel_update() - returns true if it changed.
2. channel_should_enable() - should this channel be marked enabled.
3. arm_refresh_timer() - start a refresh timer for the channel_update.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This could always happen, but now we're more correct about send it, it's
a possible race that I've seen in CI.
We can never get a PONG (that's now handled by connectd directly), but we
can definitely see this:
```
2025-05-15T02:29:14.7403081Z # Even with onchain anchor channel, it still keeps reserve (just in case!).
2025-05-15T02:29:14.7403766Z > l1.rpc.close(l2.info['id'])
2025-05-15T02:29:14.7404069Z
2025-05-15T02:29:14.7404248Z tests/test_opening.py:2536:
...
2025-05-15T02:29:14.7413090Z > return self.sock.recv(length)
2025-05-15T02:29:14.7413353Z E Failed: Timeout >1800.0s
...
2025-05-15T02:29:14.8097229Z lightningd-1 2025-05-15T01:50:59.538Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-closingd-chan#1: billboard perm: Expected closing_signed: 0103fad81b49d5e367cd785e3088b9acf356a7984a17a1ccda86d09da41438840b08000067000001000078c3314666731e339c0b8434f7824797a084ed7ca3655991a672da068e2c44cb53b57b53a296c133bc879109a8931dc31e6913a4bda3d58559b99b95663e6d52679cabaccc6e39dbd95a4dac90e75a258893c3aa3f733d1b8890174d5ddea8003cadffe557773c54d2c07ca1d535c4bf85885f879ae466c16a516e8ffcfec174
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Calling the funding tx an anchor is pre-spec terminology, which is now
confusing; let's rename the variable.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When the command string run through $(call VERBOSE,…) contains a single-quote
character followed by shell metacharacters, the shell executing the VERBOSE
template's $(ECHO) command will attempt to interpret those metacharacters. This
could be disastrous, such as if someone were to put in a Makefile recipe:
$(call VERBOSE, , $(ECHO) 'Will not `rm -rf ~`')
When run with V=1, which causes VERBOSE to be defined as…
VERBOSE = $(ECHO) '$(2)'; $(2)
…Make would evaluate the above call into:
echo 'echo 'Will not `rm -rf ~`''; echo 'Will not `rm -rf ~`'
And oops, there goes the neighborhood.
The real-world motivating case for this fix is the sed call in the recipe for
doc/index.rst in doc/Makefile. It contains a sed expression enclosed in single
quotes, containing parentheses. When run through VERBOSE with V=1, the single
quotes around the sed expression actually escape _out_ of the single-quoted
string that is intended to be the whole command line, and you get:
/bin/sh: -c: line 1: syntax error near unexpected token `('
The fix is for VERBOSE to escape any single quotes embedded in the command line
argument when echoing it:
VERBOSE = $(ECHO) '$(subst ','\'',$(2))'; $(2)
Note that this is still wrong, as it will not do the right thing if $(2) happens
to begin with a hyphen, but I didn't want to introduce a new "PRINTF" variable
(or do something unsavory like calling cat with a here-doc) to squash a bug that
currently has no known manifestations.
Changelog-None
Lowdown requires a blank line before all preformatted blocks, or it doesn't
recognize them. `tools/md2man.sh` contained some ad-hoc efforts at fixing up
some locations where these required blank lines are absent from the output of
`tools/fromschema.py`, but it missed some. Instead of playing Whack-a-Mole, use
a blanket sed expression to ensure that a blank line precedes _every_ opening
```.
`esc_underscores(…)` in `tools/fromschema.py` did not work correctly on strings
containing an odd number of backticks, notably the ``` delimiters surrounding
preformatted text blocks. Specifically, it was dropping the last backtick since
none of the alternatives in the regex matched it. Add a new alternative that
matches a whole preformatted block as a single unit.
`output_member(…)` in `tools/fromschema.py` was passing each line of a member's
description through `esc_underscores(…)` individually, but that breaks
preformatted text blocks that are naturally multi-line and leads to mistakenly
escaping underscores inside such blocks. Rewrite the code to make use of the
`outputs(…)` utility function that joins all the provided lines together before
passing the whole text through `esc_underscores(…)`.
Drive-by fix a couple of flubbed preformatted blocks in schemas.
[ Added shellcheck suppression for md2man.sh --RR ]
Changelog-None
If `wally_psbt_extract` fails the function returns early leaking psbt, if it was `taken()`.
Throw it on tmpctx so its not leaked in this case.
Changelog-None
Fix a typo where the commit sig message ordering was not handled correctly for the first element.
We need to use msg_batch[0] to get the first post-sorted result instead of the original msg.
Changelog-None
It would be wrong to omit those prior to the last, and I want to see
what test this was supposedly breaking...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously, the code utilized JSON-encoded text to pass log messages to the
log_() function, resulting in new lines being printed to the log as '\n'. This
commit addresses this issue by unescaping and splitting the log message into
lines, with each non-empty line being logged separately.
It is deemed acceptable to pass non-printable characters to log_() since they
are replaced with "?" in the logv() function, invoked by log_(). Therefore,
leaving lines JSON unescaped is permissible.
It's important to note that only the last log_() call may have call_notifier
set to true. This adjustment is crucial to prevent Python tracebacks complaining
about a broken pipe after lightningd exits.
Fix https://github.com/ElementsProject/lightning/issues/4912
For reference, the log of lightningd failing without a backend:
./lightningd/lightningd
2024-02-08T20:33:17.642Z INFO lightningd: v23.11-257-g968d6d6-modded
Could not connect to bitcoind using bitcoin-cli. Is bitcoind running?
Make sure you have bitcoind running and that bitcoin-cli is able to connect to bitcoind.
You can verify that your Bitcoin Core installation is ready for use by running:
$ bitcoin-cli echo 'hello world'
2024-02-08T20:33:18.227Z **BROKEN** plugin-bcli: Could not connect to bitcoind using bitcoin-cli. Is bitcoind running?
2024-02-08T20:33:18.227Z **BROKEN** plugin-bcli: Make sure you have bitcoind running and that bitcoin-cli is able to connect to bitcoind.
2024-02-08T20:33:18.227Z **BROKEN** plugin-bcli: You can verify that your Bitcoin Core installation is ready for use by running:
2024-02-08T20:33:18.227Z **BROKEN** plugin-bcli: $ bitcoin-cli echo 'hello world'
2024-02-08T20:33:18.227Z INFO plugin-bcli: Killing plugin: exited before we sent init
The Bitcoin backend died.
Changelog-Changed: Plugins: log messages containing \n are now split into multiple log lines, for neatness, and " is no longer turned into \".
This updates all reckless-installed plugins with `reckless update` or
update individual plugins by passing the plugin names as arguments.
The metadata stored with the installed plugin is used to find the
plugin from the appropriate source (the same source is used as when
originally installed.)
Changelog-Added: Reckless: `reckless update` updates all reckless-installed plugins.
This allows installing a local plugin directly without having
to modify reckless sources.
Changelog-changed: Reckless can be passed a local file or directory for installation.
channeld complains if it sees blocks going backwards, but that's
actually expected: allow those log messages:
```
2025-05-13T10:28:56.3283346Z _______________ ERROR at teardown of test_v2_replay_bookkeeping ________________
...
2025-05-13T10:28:56.3301378Z request.node.has_errors = True
2025-05-13T10:28:56.3301833Z > raise ValueError(str(errors))
2025-05-13T10:28:56.3302268Z E ValueError:
2025-05-13T10:28:56.3303425Z E Node errors:
2025-05-13T10:28:56.3303923Z E - lightningd-1: had BROKEN messages
2025-05-13T10:28:56.3304404Z E Global errors:
...
2025-05-13T10:28:56.4619742Z lightningd-1 2025-05-13T10:24:38.112Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 109 less than last 111
2025-05-13T10:28:56.4620188Z lightningd-2 2025-05-13T10:24:38.112Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#2: peer_in WIRE_SHUTDOWN
2025-05-13T10:28:56.4620600Z lightningd-1 2025-05-13T10:24:38.112Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: Trying commit
2025-05-13T10:28:56.4621119Z lightningd-2 2025-05-13T10:24:38.112Z INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN
2025-05-13T10:28:56.4621623Z lightningd-1 2025-05-13T10:24:38.112Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 109 less than last 111
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to wait for exiting startup mode *before* closing channel, so it doesn't ignore
the disconnection of l3, otherwise we don't get a disabled update, and timeout waiting
at line 330:
wait_for(lambda: not l1.is_channel_active(chanid2))
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
2025-05-12T04:37:57.4824338Z else:
2025-05-12T04:37:57.4824626Z # It will forget the older one.
2025-05-12T04:37:57.4825508Z > l2.daemon.wait_for_log(r"UNUSUAL {}-chan#1: Forgetting channel: It has been {} blocks without the funding transaction ".format(l1.info['id'], blocks + 1))
```
This fails because l3 can still transmit, so the second channel can confirm in time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We may introduce high capacity channels in askrene to represent problems
with multiple destinations (eg. multiple blinded paths) or to solve
self-payments. The integer computation of the deliverable amount through
these channels (I tested this with a channel with 21M bitcoin) would
fail due to an integer overflow in the function `amount_msat_sub_fee`,
21M BTC = 21 x 10^17 msat, which overflows u64 integers when multiplied
by 10^6. We have fixed `amount_msat_sub_fee` operations, so that it
doesn't overflow.
Changelog-Fixed: askrene: fixed routing in high capacity channels.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Change order of operations to ensure that askrene-inform-channel with
inform=constrained computes the correct upper bound on the liquidity:
upperBound = reserves + amount - 1
With the previous order of operations in the case amount=0 we would get:
upperBound = reserves
instead of
upperBound = reserves - 1
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-Added: sendonion: a new paramter total_amount_msat to make MPP payments with sendpay and sendonion compatible.
[ Reordered to put new parameter at the end --RR ]
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
In some cases the CI is so quick and / or slow that the mempool doesn’t have the transaction by the time we’re looking for it.
Move the mempool check into a wait_for.
We do this by adding a specific txid the tx_abort applies to and performing checks based on that.
If the txid is NULL or unrecognized than no inflights are dropped from DB. If we recognize it than we do the check to see if we signed it and, if not, we let lightningd remove it from DB.
New reestablish behvaior allows splices to resume on reestablish if only commit sigs were exchanged.
Update the `test_commit_crash_splice` to reflect this (and turn it back on).
A splice where reestablish happens at the right moment causes an infinite loop of announcement signatures being sent back and forth.
Limit the announcement sigs we send in response to announcement sigs to once per channel session.
ChangelogNone
Allow user’s to RBF existing splices. For now this is done by simple executing an additional splice command, in the future this will can also be done with dedicated RPCs.
Changelog-Added: Enabled the ability to RBF splices
The result of splice_signed can fail for many reasons that are non-critical (already in mempool for instance).
Don’t abort channels in this case as that causes a force close.
Other implementations are sending commit_sig batches in different orders. We add support for them being in any order by ordering the batch of messages after receiving them.
Changelog-Changed: Increase interop compatability by loosening requirement that commitment signed messages be received in a particular order and sorting them internally.
The value of `our_funds` inlightningd is the funds added to the channel during creation. Splicing is a quasi-creation event. This change makes our pending funds be considered funding funds at the moment of splice confirmation.
Changelog-None
`bitcoin_tx_with_psbt` would somewhat opaquely steal the passed `psbt` value.
This caused a bug where code made a `bitcoin_tx` using a psbt without realizing the value was stolen. Because the resulting `bitcoin_tx` was placed in tmpctx it was not immediately clear that using `psbt` afterwards was an error until the tmpctx was cleared — creating a valgrind backtrace far from the actual issue.
Switching to the routine to using TAKES and adding documentation in the header, makes it explicitly clear which operation the user is doing — helping prevent future regressions of this kind.
Changelog-None
PSBT changeset routines were using linearize_output which mutated the memory of the objects it was comparing.
This commit fixes that and also cleans up the memory usage to be more clear and more guarentee there is no memory corruption.
Changelog-None
Check that the peer sent the correct txid in their `splice_locked` message.
We have to check this later on in `check_mutal_splice_locked` so we store the value in `splice_state`
A new case where `splice_locked` must be sent again on reestablish.
This handles the case where `splice_locked` did not complete locally or remotely and must be resumed.
Interop testing with Eclair revealed an issue with remote funding key rotation.
This searches for the funding output using the rotated remote funding pubkey instead of the furrent funding pubkey.
Also update the variable name to be more clear which this represents.
Changelog-Changed: Interop fixes for compatability with Eclair
Add preapprove_check capabilities:
WIRE_HSMD_PREAPPROVE_INCOICE_CHECK and
WIRE_HSMD_PREAPPROVE_KEYSEND_CHECK to the capabilities array
if dev_no_preapprove_check is not set.
Do not assume those occupy the last two slots in the array.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
signmessagewithkey: allows to sign a message with a key associated with
one bitcoin address in our wallet.
Changelog-Added: add a new rpc command signmessagewithkey to sign input messages with keys from our wallet.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-Added: HSMD: add new wire API to sign messages with bitcoin wallet keys according to BIP137.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Make sure we are sending bogus channel reestablish after recovering from
emergency.recover file and peer storage backup.
Key Changes:
- Add wait_for_log() with appropriate debug statements
Unfortunately LND does not force close the channels on receiving an error,
they blame us for this behaviour (abb1e3463f/htlcswitch/link.go (L2119))
To fix this we will send them a Bogus Channel Reestablish with 0 commitment_number and invalid last_per_commit_secret.
Key Changes:
- In connect_activate_subd, if we detect a stub channel, we serialize and send a bogus channel_reestablish message.
Changelog-Fixed: Fixing LND's non responsive behaviour on receiving an error.
Suggested-by: @NicolasDorier
Closes: https://github.com/ElementsProject/lightning/issues/8078
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we no longer try asking for blocks from bitcoind's peers if they don't store blocks.
We can fail the fundchannel because of a reconnect race: funder tells
us to reconnect so fast that we are still cleaning up from last time.
We deliberately defer clean up, to give the subds a chance to process any
final messages. However, on reconnection we force them to exit immediately:
but this causes new `connect` commands to see the exit and fail.
The workaround is to do this cleanup when a `connect` command is
issued (as well as the current case, which covers an automatic
reconnection or an incoming reconnection)
```
2025-05-09T00:40:37.1769508Z def test_reconnect_signed(node_factory):
2025-05-09T00:40:37.1770273Z # This will fail *after* both sides consider channel opening.
2025-05-09T00:40:37.1770850Z disconnects = ['<WIRE_FUNDING_SIGNED']
2025-05-09T00:40:37.1771298Z if EXPERIMENTAL_DUAL_FUND:
2025-05-09T00:40:37.1771735Z disconnects = ['<WIRE_COMMITMENT_SIGNED']
2025-05-09T00:40:37.1772155Z
2025-05-09T00:40:37.1772598Z l1 = node_factory.get_node(may_reconnect=True, disconnect=disconnects)
2025-05-09T00:40:37.1773210Z l2 = node_factory.get_node(may_reconnect=True)
2025-05-09T00:40:37.1773632Z
2025-05-09T00:40:37.1773917Z l1.fundwallet(2000000)
2025-05-09T00:40:37.1774268Z
2025-05-09T00:40:37.1774611Z l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
2025-05-09T00:40:37.1775151Z > l1.rpc.fundchannel(l2.info['id'], CHANNEL_SIZE)
2025-05-09T00:40:37.1775388Z
2025-05-09T00:40:37.1775485Z tests/test_connection.py:667:
...
2025-05-09T00:40:37.1799527Z > raise RpcError(method, payload, resp['error'])
2025-05-09T00:40:37.1800993Z E pyln.client.lightning.RpcError: RPC call failed: method: fundchannel, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'amount': 50000, 'announce': True}, error: {'code': -1, 'message': 'Disconnected', 'data': {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'method': 'openchannel_update'}}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We cannot add new parameters in the middle, since we accept parameters by JSON
array as well as by dicts. In fact, this broke tests, but due to unrelated
breakage in the GitHub "Automerge" functionality, it got applied as
556e38c838 ("askrene-bias-channel: bias call add
up.").
Also add tests, and a better Changelog line.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `askrene-bias-channel` now has a `relative` option to add, rather than replace, a channel bias.
We need to make sure the close tx is generated where we expect, so the following code
doesn't wait foreverL
```
2025-05-08T04:31:31.1071194Z > assert l2.rpc.wait('htlcs', 'deleted', 3)['deleted'] == 5
2025-05-08T04:31:31.1071518Z
2025-05-08T04:31:31.1071645Z tests/test_misc.py:3398:
...
2025-05-08T04:31:31.1079841Z E Failed: Timeout >1800.0s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't care about data persistence, and this suppresses all sync calls (speeding
sqlite3 specifically).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This significantly slows down the cycle when we have flakes (it's good if
the tree is completely broken though!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This speeds logging slightly, but most of the time is actually
spent in fflush() (which we need).
Result (10 runs, eatmydata):
FAILED tests/test_connection.py::test_bench - assert 24.086638-25.347475(24.6901+/-0.4) == 0
This is an 8% performance improvement (over the entire series), which is not bad.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to iterate through all plugins, making
our "do we even have to construct this notification" optimization
much more efficient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If nobody is subscribed, have notify_start return NULL and the caller
can skip serialization. This is particularly useful for the "log"
notification which can get called a lot.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our CORK logic was wrong, and it's better to use Nagle anyway:
we disable Nagle just before sending timing-critical messages.
Time for 100 (failed) payments:
Before:
148.8573575
After:
10.7356977
Note this revealed a timing problem in test_reject_invalid_payload: we would
miss the cause of the sendonion failure, and waitsendpay would be called
*after* it had failed, so simply returns "Payment failure reason unknown".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: Removed 200ms latency from sending commit/revoke messages.
We need to make sure the lease starts a the block we expect:
```
# Note that l3 has the whole lease delay (minus blocks already mined)
_, _, l3blocks = l3.wait_for_onchaind_tx('OUR_DELAYED_RETURN_TO_WALLET',
'OUR_UNILATERAL/DELAYED_OUTPUT_TO_US')
> assert l3blocks == 4032 - 6 - 2 - 1
E assert 4022 == (((4032 - 6) - 2) - 1)
tests/test_closing.py:985: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can fail to connect because l3 connects to l2 at the same time l2 connects to l3:
```
# The plugin in l2 fixes up the connection, so this works!
> l1.rpc.fetchinvoice(offer['bolt12'])
tests/test_connection.py:4792:
...
elif "error" in resp:
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: fetchinvoice, payload: {'offer': 'lno1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcgqgqjczs7w3jhxazldahxjmmwd4jhxumpvaj47en0wfmkzunytanxz6tvzcssxhftzxfdlwsnfcgw2sy8t5mxa0ytcdfat2nkdwqvpy9nnsa9mzza'}, error: {'code': 1005, 'message': 'Timeout waiting for response'}
```
From logs:
```
lightningd-2 2025-05-07T21:20:54.367Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: peer_out WIRE_INIT
lightningd-2 2025-05-07T21:20:54.367Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: Connect IN
``
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Presumably we want to wait for htlcs to be all resolved here. With the speedup,
this failed under Postgres:
```
> wait_for(lambda: only_one(l2.rpc.listpeerchannels(l3.info['id'])['channels'])['htlcs'] != [])
tests/test_closing.py:4191:
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Once we speed things up, this can receive COMMITMENT_SIGNED, and hence succeed. Let's
drop on receive, not send!
```
2025-05-06T06:07:51.5997333Z # Peers try RBF, break on initial COMMITMENT_SIGNED
2025-05-06T06:07:51.5997855Z bump = l1.rpc.openchannel_bump(chan_id, chan_amount, initpsbt['psbt'])
2025-05-06T06:07:51.5998214Z > with pytest.raises(RpcError):
2025-05-06T06:07:51.5998542Z E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Once we speed things up, we can *receive* commitment_signed before we disconnect
due to sending ours: in this case, we *will* have a scratch tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
DNS seeds have been down/offline for a while, and this code (which
blocks!) has been a source of trouble. We should probably use a
canned set of "known nodes" if we want to bootstrap.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we no longer use DNS seeds for peer lookup fallbacks.
Fixes: https://github.com/ElementsProject/lightning/issues/7913
From grubles' logs:
```
2025-01-06T15:30:31.449Z DEBUG lightningd: attempting connection to 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923 for additional gossip
2025-01-06T15:30:31.449Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Adding 0 addresses to important peer
2025-01-06T15:30:31.449Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:31.449Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.037Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.037Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.428Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.428Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.680Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.681Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.468Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.469Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.471Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.471Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.935Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.935Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:34.125Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:34.125Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.496Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.497Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.623Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.623Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.751Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.751Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.892Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.892Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
```
We promised to wait 300 seconds!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The channel bias feature is not being used yet by any plugin, so this
hopefully doesn't break any working code.
When askrene-bias-channel is called the bias quantity is added on top of
any previous biased already present on that channel instead of
overwriting it.
Changelog-Changed: askrene-bias-channel: bias call add up.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Reported-by: daywalker90
Changelog-Deprecated: JSON-RPC: channel_state_changed notification field `old_state` value "unknown" (it will be omitted, instead)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
fundwallet generates a block: if l2 sees one block and not the other, it can
consider the time on the first fundchannel to be one block earlier than expected:
```
2025-05-05T21:05:33.6260600Z E TimeoutError: Unable to find "[re.compile('UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Forgetting channel: It has been 51 blocks without the funding transaction ')]" in logs.
...
2025-05-05T21:05:33.7179461Z lightningd-2 2025-05-05T20:44:26.141Z UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Forgetting channel: It has been 52 blocks without the funding transaction 5a18d113f53df8b205ab679924babde4068f2d1876d34edc43701bf92b8ff13f getting deeply confirmed. We are fundee and can forget channel without loss of funds.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is inspired by a patch from @whitslack, which overlapped with this series.
Most importantly, there was only one call to bitcoin_tx_simple_input_weight(),
and it is better to be explicit with that one.
This also changes our funder calculation to assume our own input is taproot,
which it is likely to be given we've defaulted to taproot for outputs for
change addresses since 23.08.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
One in 256 times, we will grind a signature to 70 bytes (or shorter). This breaks
our feerate tests. Unfortunately the grinding is deterministic, so there doesn't
seem to be a way to avoid it. So we add a log message, and then we skip the
feerate test if it happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is legal! And we actually do this in tests, but we didn't check the psbt
was spendable (the next patch does, indirectly, by testing feerate):
```
# Discard prep4 and get all funds again
l1.rpc.txdiscard(prep5['txid'])
# You can have one which is all, but not two.
prep5 = l1.rpc.txprepare([{addr: Millisatoshi(amount * 3 * 1000)},
{addr: 'all'}])
# Feerate should be ~ as we asked for
> assert normal_feerate_perkw - 1 < feerate_from_psbt(bitcoind, l1, prep5['psbt']) < normal_feerate_perkw + 1
E AssertionError: assert (7500 - 1) < -1091803.9574935874
```
Changelog-Fixed: JSON-RPC: `txprepare` with `all` as well as one or more non-all amount fields now produces a valid PSBT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we've fixed various underlying weight mis-estimation, we can do a few final tweaks and
test that anchors work as intended.
1. We assumed a p2wpkh output, but modern change is p2tr.
2. We handed `2` instead of `1` to bitcoin_tx_core_weight (which doesn't matter, but was weird).
3. Make change calculation clearer. I'm not sure the previous one was wrong, but it was harder
to understand.
4. Fix the test and make it clearly test that we are aiming for (and achieving) the right feerate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: anchors' fees are now much closer to the feerate targets.
We need one byte for the number of witness elements. Some callers added it themselves,
but it's always needed. So document and fix the callers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't add the weight of the two sigs! The BOLT defines that to be a worst-case 73 byte sig,
but that turns out to be an overestimate (and this is not required for consensus) so we assume
everyone grinds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use this for anchors, in which case we have a minimum value for
change. If we don't take this into account, we end up with a lower
feerate once we actually create the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We previously treated it as a P2WPKH, which is wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: wallet: fees are much closer to target feerate when doing txprepare/fundchannel.
Due to a bug elsewhere I actually triggered this path, and it
broadcast the tx anyway, *then* closed the channel. We should abandon
the channel if we can, instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To actually evaluate spend cost, we need to know whether it's taproot or not.
Using an enum (rather than making callers examine the script) means we can
ensure all cases are handled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The renaming makes it clear that it's HSM specific.
And it has no pointers, so we can have an array instead of an array of pointers.
I tested this hadn't accidentally changed the wire format by disabling
version checks and using an old hsmd with the altered daemons and
running the test suite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is such a simple struct that we can actually define it in csv.
This prevents us from accidentally breaking the ABI in future.
I tested this hadn't accidentally changed the wire format by disabling
version checks and using an old hsmd with the altered daemons and
running the test suite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I'm about to update our utxo type, but Christian spotted that this is
part of the ABI for the hsm. So make that a private "hsm_utxo" type,
to insulate it from changes.
In particular, the HSM versions only contain the fields that the
hsm cares about, and the wire format is consistent (even though that
*did* include some of those fields, they are now dummies).
In the long term, this should be removed from the ABI: once we
no longer have "close_info" utxos, this information should already be
in the PSBT.
I tested this hadn't accidentally changed the wire format by disabling
version checks and using an old hsmd with the altered daemons and
running the test suite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not just the key index.
Also, remove FIXME: wallet_can_spend is no longer slow with lots of inputs!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're about to fix the feerate calculations in various places, and one
side effect is that we end up trying to add an empty anchor if none is
necessary (and failing, but we log a nasty message about it).
So don't do that, and fix the test which expected it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We actually need to mine another block, so that lightningd considers both
channels candidates for forgetting (and forgets the older one).
Reported-by: daywalker90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We say the offer amount arg can be either an `msat_or_any` or a `currency` string, but technically the order of evaluation is undefined, so it can check the other way, and it should not crash:
```
> offer = l3.rpc.offer(1000, 'test_pay_blindedpath_nodeaddr')
tests/test_pay.py:5692:
...
checker = <TypeChecker types={'array', 'bip340sig', 'boolean', 'currency', 'feerate', 'hash', 'hex', 'integer', 'msat', 'msat_or..._all', 'secret', 'short_channel_id', 'short_channel_id_dir', 'signature', 'string', 'txid', 'u16', 'u32', 'u64', 'u8'}>
instance = 1000
def is_currency(checker, instance):
"""currency including currency code"""
pattern = re.compile(r'^\d+(\.\d+)?[A-Z][A-Z][A-Z]$')
> if pattern.match(instance):
E TypeError: expected string or bytes-like object
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A log event LOG_TRACE submitted by a plugin was being logged as
**BROKEN** by lightningd before this commit.
Changelog-Fixed: plugins can now log events under the LOG_TRACE flag.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
This broke my build machine, because lightningd --version was malformed
(I had no tags somehow in that branch).
I dived into the code to figure out what was wrong, and I was horrified.
1. STOP. Never write this much code.
2. You just need a NodeVersion class. That's it. No others.
3. Don't throw away the entire first part if it starts with 'v'. Just remove the v.
4. Handle untagged versions cleanly.
5. Always fail on invalid strings in the constructor, NOT on the first time you
use it.
I have rewritten it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids the docker image builders breaking with:
ERROR [linux/amd64 builder 15/17] RUN poetry export -o requirements.txt --without-hashes
0.780 pyproject.toml changed significantly since poetry.lock was last generated. Run Resolving dependencies... to fix the lock file.
This occurred when the default installation version changed underneath us.
Changelog-None
Implements the LSPS0 service plugin for core lightning
Changelog-Added: lsps-plugin: lsps0 service support
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Adds an async safe JSON-RPC V2 client for a generic transport layer. The
transport layer we will use later on are BOLT8 lightning messages.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
fundwallet() actually mines a block, putting our count out. If we see
both blocks at once, we will say "52" blocks instead of "51":
```
2025-05-02T05:28:40.5315155Z have_forgotten = l2.daemon.is_in_log(
2025-05-02T05:28:40.5315650Z r"UNUSUAL {}-chan#1: Forgetting channel: It has been 51 blocks without the funding transaction ".format(l1.info['id'])
2025-05-02T05:28:40.5316105Z )
2025-05-02T05:28:40.5316263Z
2025-05-02T05:28:40.5316417Z if dopay:
2025-05-02T05:28:40.5316616Z assert not have_forgotten
2025-05-02T05:28:40.5317056Z assert set([c['peer_id'] for c in l2.rpc.listpeerchannels()["channels"]]) == set([l1.info['id'], l3.info['id']])
2025-05-02T05:28:40.5317477Z else:
2025-05-02T05:28:40.5317662Z > assert have_forgotten
2025-05-02T05:28:40.5317887Z E assert None
```
```
0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Forgetting channel: It has been 52 blocks without the funding transaction 0bb0579df6b1d983dda49dad47513afc71696c9d5bea3c8b955ba4b76bb053de getting deeply confirmed. We are fundee and can forget channel without loss of funds.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were supposed to put the sqlite db in a different directory, but
the test was wrong! So occasionally we would crash with:
```
Failed to commit DB transaction: Failed to commit a transaction: disk I/O error
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We try to do a JSON response if Python is not present, but it assumes sed. We should cleanly
exit on errors.
Before:
```
$ PATH=/tmp ./plugins/wss-proxy/wss-proxy
Something
./plugins/wss-proxy/wss-proxy: 12: sed: not found
{"jsonrpc":"2.0","id":,"result":{"disable":"No python3 binary found"}}
```
After:
```
$ PATH=/tmp ./plugins/wss-proxy/wss-proxy
something
./plugins/wss-proxy/wss-proxy: 12: sed: not found
```
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This finally happened on my local build machine, so I tracked it down using
py-spy, `apt-get install python3-dbg` and `py-local`.
Turns out the dev-memleak command was hanging, and the processes were stuck in
SIGSTOP. There are only two places we send that, and sure enough, this was
the test which was running at the time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
plugin_log inside after_send_scb_single was logging after sending peer storage
to each peer which could lead to spam in logs for big nodes, hence we should reduce
the log level to log_trace for it.
Changelog-Fixed: Suppress logs from chanbackup
This may be useful for their recovery, though they should see the spend onchain.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: We now reply to `channel_reestablish` even on long-closed channels.
This was always false. peer_start_channeld was called in various places
with the argument "NULL" instead of "false", which unfortunately compilers
didn't complain about :(
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Michael of Boltz told me this long ago, that when fees spiked some of their clients' opens got stuck. After two weeks their node forgot them, and when fees came back down the opens still failed. Unfortunately, I can't see an issue for this!
We can give some grace: we don't want to waste too many resources, but 100 channels is nothing.
The test needs to be adjusted to have *two* slow open channels, now.
Changelog-Changed: Protocol: we won't forget still-opening channels after 2016 blocks, unless there are more than 100.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to start loading them into memory for nicer responses if
people try to reestablish closed channels, but we don't care about ones
which were never actually opened. We could add a new state, but easier
to simply remove them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We are supposed to allocate of the ctx we're passed, not tmpctx.
Doesn't matter for now, because we don't use this result with anything which outlives tmpctx,
but we're going to:
```
==47574==ERROR: AddressSanitizer: heap-use-after-free on address 0x6040005a8f38 at pc 0x55d3c584d252 bp 0x7ffddfb1b090 sp 0x7ffddfb1b088
READ of size 8 at 0x6040005a8f38 thread T0
#0 0x55d3c584d251 in json_add_closed_channel /home/runner/work/lightning/lightning/lightningd/closed_channel.c:27:3
#1 0x55d3c584ca5a in json_listclosedchannels /home/runner/work/lightning/lightning/lightningd/closed_channel.c:118:3
#2 0x55d3c58c0cbe in command_exec /home/runner/work/lightning/lightning/lightningd/jsonrpc.c:808:8
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that documentation says invoice expiries can batch, but that's no
longer true, so delete it. Usually, we miss a number because the
change is too fast.
This adds the wait interface, but it doesn't actually fire until the next
commit, which wires it into the db code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `wait` now supports the `htlcs` (`listhtlcs`) subsystem.
We're not going to increment one at a time for bulk deletion of htlcs
when a channel closes. There could be millions of HTLCs!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It makes the schema simpler, and indeed, expressable by GRPC.
Changelog-Added: JSON-RPC: `wait` now has separate `invoices`, `forwards` and `sendpays` objects for each subsystem.
Changelog-Deprecated: JSON-RPC: `wait` reply `details` object: use subsytem specific object instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The last argument is a pointer, but we were handing `false`. Which, for
terrible historic reasons, gets treated as NULL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Requested-by: @whitslack
Closes: https://github.com/ElementsProject/lightning/issues/8233
Changelog-Added: JSON-RPC: `listpeerchannels` now has a `short_channel_id` parameter for just listing a specific channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
notleak() doesn't work for lightningd since the first span is created before
memleak (or anything else!) is initialized, so we have to mark it manually.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This doesn't happen very often, but can with autoclean.
However, we rarely traverse to the end, since we always expect to find
what we're looking for, and we fill from the front. So even a large
array (unless it's used) is fine.
Subtle: when we injected a parent, we used "active_spans" as the (arbitrary)
key. That can now change with reallocation, and so if that memory were reused
we could have a key clash. So we use "&active_spans" which doesn't change.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't ever actually close the remote span (we don't have its key,
after all), and we keep a pointer to the parent.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is much faster to give 64 bits of data, and we don't need
cryptographic randomness.
This brings us back to 413ns per trace.
Before:
real 0m5.819000-6.472000(6.2064+/-0.26)s
user 0m3.779000-4.101000(3.956+/-0.12)s
sys 0m2.040000-2.431000(2.2496+/-0.15)s
After:
real 0m3.981000-4.247000(4.1276+/-0.11)s
user 0m3.979000-4.245000(4.126+/-0.11)s
sys 0m0.000000-0.002000(0.001+/-0.00063)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightingd: trimmed overhead of tracing infrastructure.
There's an EBPF limit anyway, so stick with a 512-byte buffer.
This brings us back to 621ns per trace:
Before:
real 0m13.441000-14.592000(14.2686+/-0.43)s
user 0m11.265000-12.289000(11.9626+/-0.37)s
sys 0m2.175000-2.381000(2.3048+/-0.072)s
After:
real 0m5.819000-6.472000(6.2064+/-0.26)s
user 0m3.779000-4.101000(3.956+/-0.12)s
sys 0m2.040000-2.431000(2.2496+/-0.15)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Avoids allocations. Also assume that name and value parameters
outlive the trace span, so don't copy.
Before:
real 0m16.421000-18.407000(17.8128+/-0.72)s
user 0m14.242000-16.041000(15.5382+/-0.67)s
sys 0m2.179000-2.363000(2.273+/-0.061)s
After:
real 0m13.441000-14.592000(14.2686+/-0.43)s
user 0m11.265000-12.289000(11.9626+/-0.37)s
sys 0m2.175000-2.381000(2.3048+/-0.072)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. trace_span_start() is always called with a string literal, so
no copy needed (and we can use a macro to enforce this).
2. trace_span_tag() name and value are always longer-lived than
the span, so no need to copy these either.
Before:
real 0m18.524000-19.100000(18.7674+/-0.21)s
user 0m16.171000-16.833000(16.424+/-0.26)s
sys 0m2.259000-2.400000(2.337+/-0.059)s
After:
real 0m16.421000-18.407000(17.8128+/-0.72)s
user 0m14.242000-16.041000(15.5382+/-0.67)s
sys 0m2.179000-2.363000(2.273+/-0.061)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Testing parenting handling revealed several issues:
1. By calling "trace_span_start" when CLN_TRACEPARENT is set produces a bogus
entry, for which the span_id is overwritten so we never end it.
2. We don't need to close the remote parent when we close the first child: in
fact, this causes the remaining traces to be detached from the parent!
3. Suspension should return current to the parent, not to NULL.
Now the traces balance as we expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I added this debugging because the next test revealed a mismatch, so
I wanted to see where it was happening.
The comment in lightningd suggests it's possible, but I can't see any
code which suspends in the lightningd io_loop, so I cannot see how
this is triggered.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With an average runtime of 18.7674, this implies 1876ns
per trace, which is far in excess of the 370ns claimed in
doc/developers-guide/tracing-cln-performance.md.
We also add a tag in there, so we measure that!
Results on my laptop:
real 0m18.524000-19.100000(18.7674+/-0.21)s
user 0m16.171000-16.833000(16.424+/-0.26)s
sys 0m2.259000-2.400000(2.337+/-0.059)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Introduced the use of UPDATE FROM syntax in SQLite queries,
which is not supported in versions prior to 3.33.0.
This causes issues on systems with older SQLite versions,
as reported in issue #8231. Rewrite the query in
migrate_convert_old_channel_keyidx() to use a subquery
with IN clause instead of UPDATE FROM, ensuring compatibility with
older SQLite versions.
Changelog-Fixed: db: replace UPDATE FROM syntax for SQLite compat
Fixes 68f3649d6b
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
```
cargo build --quiet --example cln-plugin-startup
error: failed to run custom build command for `cln-grpc v0.4.0 (/home/runner/work/lightning/lightning/cln-grpc)`
Caused by:
process didn't exit successfully: `/home/runner/work/lightning/lightning/target/debug/build/cln-grpc-1c0900b8d6f448d4/build-script-build` (exit status: 101)
--- stdout
cargo:rerun-if-changed=proto/node.proto
cargo:rerun-if-changed=proto
--- stderr
thread 'main' panicked at cln-grpc/build.rs:7:10:
called `Result::unwrap()` on an `Err` value: Custom { kind: NotFound, error: "Could not find `protoc`. If `protoc` is installed, try setting the `PROTOC` environment variable to the path of the `protoc` binary. Try installing `protobuf-compiler` or `protobuf` using your package manager. It is also available at https://github.com/protocolbuffers/protobuf/releases For more information: https://docs.rs/prost-build/#sourcing-protoc" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
gmake: *** [cln-rpc/Makefile:15: target/debug/examples/cln-plugin-startup] Error 101
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
fatal: detected dubious ownership in repository at '/home/runner/work/lightning/lightning/external/libwally-core'
To add an exception for this directory, call:
git config --global --add safe.directory /home/runner/work/lightning/lightning/external/libwally-core
Reinitializing submodules src/secp256k1 ...
fatal: detected dubious ownership in repository at '/home/runner/work/lightning/lightning/external/libwally-core'
To add an exception for this directory, call:
git config --global --add safe.directory /home/runner/work/lightning/lightning/external/libwally-core
fatal: detected dubious ownership in repository at '/home/runner/work/lightning/lightning/external/libwally-core'
To add an exception for this directory, call:
git config --global --add safe.directory /home/runner/work/lightning/lightning/external/libwally-core
gmake: *** [external/Makefile:65: submodcheck] Error 128
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
| -- Found Python: /tmp/tmpyjjd1wyq/.venv/bin/python (found suitable version "3.11.11", minimum required is "3") found components: Interpreter Development.Module Development.SABIModule
| CMake Error at /tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/cmake/data/share/cmake-4.0/Modules/FindPackageHandleStandardArgs.cmake:227 (message):
| Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
| Call Stack (most recent call first):
| /tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/cmake/data/share/cmake-4.0/Modules/FindPackageHandleStandardArgs.cmake:591 (_FPHSA_FAILURE_MESSAGE)
| /tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/cmake/data/share/cmake-4.0/Modules/FindPkgConfig.cmake:110 (find_package_handle_standard_args)
| CMakeLists.txt:35 (find_package)
|
|
| -- Configuring incomplete, errors occurred!
| Traceback (most recent call last):
| File "/root/.local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
| main()
| File "/root/.local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
| json_out["return_val"] = hook(**hook_input["kwargs"])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/root/.local/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
| return _build_backend().build_wheel(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/hatchling/build.py", line 58, in build_wheel
| return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/hatchling/builders/plugin/interface.py", line 147, in build
| build_hook.initialize(version, build_data)
| File "/tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/scikit_build_core/hatch/plugin.py", line 125, in initialize
| self._initialize(build_data=build_data)
| File "/tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/scikit_build_core/hatch/plugin.py", line 228, in _initialize
| builder.configure(
| File "/tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/scikit_build_core/builder/builder.py", line 283, in configure
| self.config.configure(
| File "/tmp/tmpyjjd1wyq/.venv/lib/python3.11/site-packages/scikit_build_core/cmake.py", line 254, in configure
| raise FailedLiveProcessError(msg) from None
| scikit_build_core.errors.FailedLiveProcessError: CMake configuration failed
Note: This error originates from the build backend, and is likely not a problem with poetry but one of the following issues with coincurve (20.0.0)
- not supporting PEP 517 builds
- not specifying PEP 517 build requirements correctly
- the build requirements are incompatible with your operating system or Python version
- the build requirements are missing system dependencies (eg: compilers, libraries, headers).
You can verify this by running pip wheel --no-cache-dir --use-pep517 "coincurve (==20.0.0)".
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It didn't work, and it was downloading the Linux binary anyway!
```
2025-04-11 02:37:12 (1.61 MB/s) - ‘bitcoin-27.1-x86_64-linux-gnu.tar.gz’ saved [48920073/48920073]
tar: Error opening archive: Failed to open 'bitcoin-27.1-x86_64-linux-gnu.tar.bz2'
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
12 MiB to be downloaded.
[1/1] Fetching pkg-2.1.0.pkg: .......... done
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.21.3 to 2.1.0...
[1/1] Extracting pkg-2.1.0: .......... done
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'python38' have been found in the repositories
Error: The process '/usr/bin/bash' failed with exit code 1
```
Changelog-None: CI only
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Services that want to wrap the server implementation will need
access to the `NotificationStream` struct in order to pass
the stream through.
Changelog-Added: `cln-grpc` Exposed NotificationStream in the server module
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
@pytest.mark.openchannel('v1')
@pytest.mark.openchannel('v2')
def test_pay(node_factory):
> l1, l2 = node_factory.line_graph(2)
...
FAILED tests/test_pay.py::test_pay - OSError: AF_UNIX path too long
Changelog-None: symlink the socket to a tempfile which has a shorter path
Signed-off-by: Lakshya Singh <lakshay.singh1108@gmail.com>
get_channel_scid() returns the *first* channel, and we want all the channels.
So they may not actually be all visible in gossip.
```
@unittest.skipIf(TEST_NETWORK == 'liquid-regtest', "broken for some reason")
def test_hardmpp2(node_factory, bitcoind):
"""Credits to @daywalker90 for this test case."""
opts = {"disable-mpp": None, "fee-base": 0, "fee-per-satoshi": 10}
l1, l2, l3 = node_factory.get_nodes(3, opts=opts)
start_channels(
[
(l1, l2, 100_000),
(l1, l2, 200_000),
(l1, l2, 300_000),
(l1, l2, 400_000),
(l2, l3, 100_000),
(l2, l3, 200_000),
(l2, l3, 300_000),
(l2, l3, 600_000),
]
)
# FIXME: changing the last channel from 600k to 400k will fail the test due
# to l2 not accepting to forward any amount above 200k with error:
# CHANNEL_ERR_CHANNEL_CAPACITY_EXCEEDED, still investigating
inv = l3.rpc.invoice("800000sat", "inv", "description")
> l1.rpc.call("renepay", {"invstring": inv["bolt11"]})
tests/test_renepay.py:797:
...
elif "error" in resp:
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: renepay, payload: {'invstring': 'lnbcrt8m1pnl3zfksp54n6vhkrcj0nkn2uqhf232v62539048u2ree4g9ssytm47gasgctspp58twj5xf4h57s4h0rhy55p5q6nry36glummjexsf4lc446je7y27qdqjv3jhxcmjd9c8g6t0dcxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqzqqqqqqqqqqqqqqqzsqqcrzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqpcqqqqqqqqqqqqqqzsqqcrzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqgqqqqqqqqqqqqqqzsqqcrzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqpqqqqqqqqqqqqqqqzsqqc9qxpqysgqtgxcneu0u7xvetgh2kqmey86jssweneat5vx3fppuaphl9v9jweqa2ymczg9klau9jmsm6pm7m3cd28pggp7avuukjqxg63wy2vuvtcpggz4vt'}, error: {'code': 205, 'message': "minflow couldn't find a feasible flow: failed to find a feasible flow: find_admissible_path failed"}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The handshake targets were based on a false premise: that it is
impossible for any fuzzer to generate valid Act 1 or 2 packets. Niklas
Gogge proved this premise incorrect using AFL++ with the CMPLOG feature,
which enabled AFL++ to generate such valid packets.
We modify the targets to allow the scenario where the fuzzer finds these
valid packets and add the inputs AFL++ found to the corpus.
The cryptofuzz target was based on a false premise: that it is
impossible for any fuzzer to generate a valid ciphertext+MAC for the
decrypt function. Niklas Gogge proved this premise incorrect using AFL++
with the CMPLOG feature, which enabled AFL++ to generate such valid
messages.
We remove the assertions requiring decryption to fail and add the inputs
AFL++ found to the corpus.
BIP-141 specifies that a witness program must be between 2 and 40 bytes in
length. In our fallback address parsing, we were already checking the upper
bound, but missing the lower bound check. This commit adds validation to
ensure fallback address witness programs are at least 2 bytes long, bringing
our implementation in line with the spec and other implementations like
rust-lightning.
Changelog-Fixed: Enforced minimum witness program length of 2 bytes for
fallback addresses to comply with BIP-141 and prevent invalid decodings.
Also removes usage of pip due to this error that crops up on newer
versions of Ubuntu:
```
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.12/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
```
I've been recommending installing Poetry from the official
installer for the past several months and it always works.
Changelog-None
The workflow fails to trigger manually after recreating the release tag. It should support re-running even if the tag was previously deleted and recreated.
Changelog-None.
This happens with autoclean, which does a datastore request then frees
the parent command without waiting for a response (see clean_finished).
This leaks a trace, and causes a crash if the pointer is later reused.
My solution is to create a trace variant which declares the trace key
to be a tal ptr and then we can clean up in the destructor if this happens.
This fixes the issue for me.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: autoclean: fixed occasional crash when tracepoints compiled in.
This can happen with 24.11 and later. We scan back to exposed channel
opens, or that release.
The BROKEN log messages cause some tests to fail, so we fix those.
Fixes: https://github.com/ElementsProject/lightning/issues/8169
Changelog-Fixed: wallet: rescan for missing close outputs (can happen if peer doesn't support option_shutdown_anysegwit)
When we make mocks (which the next patch will do), these names cause a warning:
```
wallet/test/run-db.c:32:64: error: declaration of ‘bitcoind’ shadows a parameter [-Werror=shadow=compatible-local]
32 | void (*cb)(struct bitcoind *bitcoind UNNEEDED,
wallet/test/run-db.c:30:53: note: shadowed declaration is here
30 | struct bitcoind *bitcoind UNNEEDED,
wallet/test/run-db.c:33:51: error: declaration of ‘height’ shadows a parameter [-Werror=shadow=compatible-local]
33 | u32 height UNNEEDED,
wallet/test/run-db.c:31:40: note: shadowed declaration is here
31 | u32 height UNNEEDED,
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a bit more work to watch multiple addresses, but that's a small
price to pay for each channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We select the close key index at opening time, but the non-DF code didn't correctly register the
address as possibly used for P2WPKH for older nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: wallet: we could miss our own returned outputs on mutual closes if peer doesn't support option_shutdown_anysegwit (you will still need to rescan after update, if this happened to you!)
Reported-by: Grubles
1. Peer has to not support option_shutdown_anysegwit.
2. We have to restart between opening and closing the channel.
3. We won't see the to-us output, since it's p2wpkh not p2tr.
This bug was introduced in 24.11.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We pre-close incoming under some circumstances, so this does happen (it
didn't when this code was written). Don't walk all the HTLCs complaining
about them in this case, and don't freak out.
Changelog-Fixed: lightningd: incorrect spamming of log and potential crash on testnet case of duplicate HTLCs and slow closing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8176
It's not the *outgoing* HTLC which sets the deadline, it's the incoming.
Reported-by: @whitslack
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: Egregious anchor fee paid for unilateral close txs due to HTLC timeouts; it's not as urgent as our code made out!
chanbackup with many peers can do more than 128 concurrent rpc commands.
autoclean is the other plugin which can do many requests at once, so I
expect a similar issue there.
I tested this by rebuilding with `MAX_ACTIVE_SPANS` 1, which autoclean
tests triggered immediately.
The real fix is probably to use a hash table with a large initial size.
```
Mar 24 06:30:45 mlbb2 sh[28000]: chanbackup: common/trace.c:190: trace_span_slot: Assertion `s' failed.
Mar 24 06:30:45 mlbb2 sh[28000]: chanbackup: FATAL SIGNAL 6 (version v25.02)
Mar 24 06:30:45 mlbb2 sh[28000]: 0x5575232bac4f send_backtrace
Mar 24 06:30:45 mlbb2 sh[28000]: common/daemon.c:33
Mar 24 06:30:45 mlbb2 sh[28000]: 0x5575232baceb crashdump
Mar 24 06:30:45 mlbb2 sh[28000]: common/daemon.c:78
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958cd851f ???
Mar 24 06:30:45 mlbb2 sh[28000]: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958d2c9fc __pthread_kill_implementation
Mar 24 06:30:45 mlbb2 sh[28000]: ./nptl/pthread_kill.c:44
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958d2c9fc __pthread_kill_internal
Mar 24 06:30:45 mlbb2 sh[28000]: ./nptl/pthread_kill.c:78
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958d2c9fc __GI___pthread_kill
Mar 24 06:30:45 mlbb2 sh[28000]: ./nptl/pthread_kill.c:89
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958cd8475 __GI_raise
Mar 24 06:30:45 mlbb2 sh[28000]: ../sysdeps/posix/raise.c:26
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958cbe7f2 __GI_abort
Mar 24 06:30:45 mlbb2 sh[28000]: ./stdlib/abort.c:79
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958cbe71a __assert_fail_base
Mar 24 06:30:45 mlbb2 sh[28000]: ./assert/assert.c:94
Mar 24 06:30:45 mlbb2 sh[28000]: 0x7f2958ccfe95 __GI___assert_fail
Mar 24 06:30:45 mlbb2 sh[28000]: ./assert/assert.c:103
Mar 24 06:30:45 mlbb2 sh[28000]: 0x5575232ab7fa trace_span_slot
Mar 24 06:30:45 mlbb2 sh[28000]: common/trace.c:190
Mar 24 06:30:45 mlbb2 sh[28000]: 0x5575232abc9f trace_span_start
Mar 24 06:30:45 mlbb2 sh[28000]: common/trace.c:267
Mar 24 06:30:45 mlbb2 sh[28000]: 0x5575232a7c34 send_outreq
Mar 24 06:30:45 mlbb2 sh[28000]: plugins/libplugin.c:1112
```
Changelog-Fixed: autoclean/chanbackup: fixed tracepoint crash on large number of requests.
Fixes: https://github.com/ElementsProject/lightning/issues/8177
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We handed NULL as the logcb, resulting in a very uninformative crash:
```
2025-03-14T03:46:36.447Z INFO lightningd: Server started with public key 03d67f36c4f81789e2fe425028bacc96b199813eae426c517f589a45f1136c1fe5, alias Jubilee (color #dc42f4) and lightningd v25.02
topology: FATAL SIGNAL 11 (version v25.02)
0x560037f64aad send_backtrace
common/daemon.c:33
0x560037f64b49 crashdump
common/daemon.c:78
0x7f6c41ff351f ???
./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x0 ???
???:0
```
Changelog-Fixed: `topology` crash on invoice creation if a peer had a really high feerate.
Fixes: https://github.com/ElementsProject/lightning/issues/8156
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By limiting the commits that we look at to the changes since `master` we
can incrementally pull files under the coverage of these lints and checks.
Changelog-None: Not applicable, this is DX
We reconnect from l3->l2, but l2 is also trying to reconnect and
we can race:
```
# Now try when l3 uses scid for entry point of blinded path.
l3.stop()
l3.daemon.opts['dev-invoice-bpath-scid'] = None
l3.start()
> l3.rpc.connect(l2.info['id'], 'localhost', l2.port)
tests/test_pay.py:5667:
...
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: connect, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'host': 'localhost', 'port': 33557}, error: {'code': 402, 'message': 'disconnected during connection'}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By submitting them all at once, rather than serially, we should *definitely* hit the
ratelimit. We can also reduce the timeout (from 60 seconds) to speed the test up.
```
@pytest.mark.slow_test
def test_onionmessage_ratelimit(node_factory):
l1, l2 = node_factory.line_graph(2, fundchannel=False,
opts={'allow_warning': True})
offer = l2.rpc.call('offer', {'amount': '2msat',
'description': 'simple test'})
# Hopefully we can do this fast enough to reach ratelimit!
> with pytest.raises(RpcError, match="Timeout waiting for response"):
E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
tests/test_pay.py:5825: Failed
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We play with directory permissions, but sqlites needs that, and sometimes (due to a timer, perhaps?)
it gets really upset about it:
```
lightningd-1 2025-03-16T23:29:58.506Z INFO lightningd: Server started with public key 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518, alias JUNIORBEAM-f91eb11-modded (color #0266e4) and lightningd f91eb11-modded
lightningd-1 2025-03-16T23:29:58.617Z DEBUG lightningd: Adding block 101: 55af171ade598f8c4f9a494eaaffe6b9f5ea64c7b0f3b273694148adf7ad7385
lightningd-1 2025-03-16T23:29:58.752Z TRACE lightningd: Calling rpc_command hook of plugin cln-xpay
lightningd-1 2025-03-16T23:29:58.764Z TRACE lightningd: Plugin cln-xpay returned from rpc_command hook call
lightningd-1 2025-03-16T23:29:58.784Z TRACE lightningd: Calling rpc_command hook of plugin cln-xpay
lightningd-1 2025-03-16T23:29:58.840Z TRACE lightningd: Plugin cln-xpay returned from rpc_command hook call
lightningd-1 2025-03-16T23:29:58.854Z TRACE lightningd: Calling rpc_command hook of plugin cln-xpay
lightningd-1 2025-03-16T23:29:58.865Z DEBUG gossipd: REPLY WIRE_GOSSIPD_NEW_BLOCKHEIGHT_REPLY with 0 fds
lightningd-1 2025-03-16T23:29:58.901Z **BROKEN** lightningd: Error executing statement: db/exec.c:108: UPDATE vars SET intval = intval + 1 WHERE name = 'data_version' AND intval = ?: attempt to write a readonly database
lightningd-1 2025-03-16T23:29:58.916Z **BROKEN** lightningd: Error executing statement: db/exec.c:108: UPDATE vars SET intval = intval + 1 WHERE name = 'data_version' AND intval = ?: attempt to write a readonly database
lightningd-1 2025-03-16T23:29:58.941Z **BROKEN** lightningd: FATAL SIGNAL 6 (version f91eb11-modded)
{'github_repository': 'ElementsProject/lightning', 'github_sha': 'f91eb118388dc1455c67d16079168fd0267ba248', 'github_ref': 'refs/pull/8112/merge', 'github_ref_name': 'HEAD', 'github_run_id': 13888173501, 'github_head_ref': 'flake-memleak', 'github_run_number': 12573, 'github_base_ref': 'master', 'github_run_attempt': '1', 'testname': 'test_setconfig_access', 'start_time': 1742167762, 'end_time': 1742167800, 'outcome': 'fail'}
----------------------------- Captured stderr call -----------------------------
Error executing statement: db/exec.c:108: UPDATE vars SET intval = intval + 1 WHERE name = 'data_version' AND intval = ?: attempt to write a readonly database
lightningd: FATAL SIGNAL 6 (version f91eb11-modded)
Log dumped in /tmp/lightning-crash.log.20250316232958
Lost connection to the RPC socket.Lost connection to the RPC socket.
=========================== short test summary info ============================
FAILED tests/test_misc.py::test_setconfig_access - AssertionError: Regex pattern did not match.
Regex: 'Cannot write to config file /tmp/ltests-22a5dz1j/test_setconfig_access_1/lightning-1/regtest/config'
Input: "RPC call failed: method: check, payload: {'command_to_check': 'setconfig', 'config': 'min-capacity-sat', 'val': 1000000}, error: Connection to RPC server lost."
ERROR tests/test_misc.py::test_setconfig_access - ValueError:
Node errors:
- lightningd-1: had BROKEN messages
- lightningd-1: Node exited with return code -6
```
So we move the db file for sqlite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Enabling the cache makes signing significantly faster for segwit inputs,
particularly taproot which was designed with caching in mind.
Changelog-None
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Modified by Rusty:
- Simplify (we only need to suppress l1's txs, since it opens)
- Use synchronize_blockheight instead of log messages to ensure l2 has processed the block
- Use listpeerchannels instead of log messages for balance (should be more robust against changes)
- Open-code assertions for better debugging if they're wrong.
Since we included the spec for it, this is a good time to implement
it.
I also asked chatgpt to write some unit tests. I had to mangle them a
bit, but it probably saved me a few minutes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unfortunately a spec typo means the data fields are missing (PR pending),
so we still patch those in.
The message "your_peer_storage" got renamed to "peer_storage_retrieval",
and the option "want_peer_backup_storage" was removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `experimental-peer-storage` now only advertizes feature 43, not 41.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: RPC `listchannels` no longer includes private local channels (deprecated v23.08, disabled by default in v24.11).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: plugins which didn't accept string JSON RPC fields (deprecated v23.08, disabled by default in v24.11).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: allowing 0/1 instead of false/true for plugin options (deprecated v23.08, disabled by default in v24.11).
Changelog-Removed: --bind-addr and --addr on onion addresses and local sockets (deprecated v23.08, disabled by default in v24.11).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: connection/disconnection/block_added notification raw fields (deprecated v23.08, disabled by default in v24.11).
We haven't printed this since v24.08.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: listconfigs raw listing (deprecated v23.08, disabled by default in v24.11).
We now have to explicitly enable various deprecated commando commands, and now
when deprecations are disabled, we honour missing MPP option in bolt12 invoices.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we update the CLN_NEXT_VERSION, these will only be available with --i-promise-to-fix-broken-api-user.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In d18f564324 "pytest: stop using
deprecated commando_rune commands." we stopped using deprecated commands in these tests,
but we didn't remove the 'allow-deprecated-apis' flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
_in_ versions complain if used, _out_ don't (since we don't know if they are
going to use it). This was the wrong choice, and thus causes a BROKEN log
message when we update CLN_NEXT_VERSION.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current docker build failed with Error `1.082 pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock` to fix the lock file.`. Adding `poetry lock` command before `poetry export` will regenerate the lock file.
Changelog-None.
Really, any peer without a live channel is a bad prospect.
This requires us to wire the "enabled" flag through listincoming:
fortunately that's an internal, undocumented interface, so we don't
have a schema change.
Changelog-Fixed: Offline peers no longer selected for blinded paths..
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8127
This can happen if a descendent tx has no witness and we don't tell the main daemon
in time that we're not iterested.
Fixes: https://github.com/ElementsProject/lightning/issues/8133
Changelog-Fixed: lightningd: onchaind crash when seeing unrelated txs (usually when catching up with old closes)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In 6e4ff6a7d2 ("offers: add a blinded path
if we have no advertized address") we were overzealous, and set blinded
paths not just for offers and invoicerequests, but for invoices themselves.
This has revealed various interop issues (which is great, but not good
for our users!) so we should disable that. It also reduces the reliability
of payments in general.
Changelog-None: fixes previously overzealous addition
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Some enums, structs, functions in pyln-grpc-proto/cln-grpc/cln-rpc
have been slightly renamed so they follow grpc and rust's naming convention
We added twice, which caused spurious failures in real-world cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: https://github.com/hMsats
Fixes: https://github.com/ElementsProject/lightning/issues/8119
Changelog-Fixed: `xpay` would double the CLTV values in blinded paths, sometimes causing spurious failures.
Use larger CLTVs and we see the failure from l3, complaining the CLTV is outside
the amount in the payment_constraint field, like:
```
Failing HTLC because of an invalid payload (TLV 10 pos 104): cltv_expiry 609 > payment_constraint 376
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The old blindedpay fields would have an entry per hop, but that was changed
to a single per-path entry. The devtools hadn't caught up.
Similarly, it didn't like descriptionless offer fields in invreqs and invoices.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This corner case started triggering on my machine with latest Bitcoind.
This test sabotages the closing negotiation, and as a result l1
doesn't see l2's CLOSING_SIGNED. l2 is happy, however, and it is in
CLOSINGD_COMPLETE. When l1 reconnects, it gets an error, and this causes
it to drop the unilateral tx to chain.
This unilateral tx from l1 replaces or races the mutual close tx from
l2, causing a unilateral close, which breaks our test.
Though this is a corner case, it's much friendlier to allow the
closing negotiation again until we actually see the close onchain.
This fixes the tests here, too.
```
def test_closing_negotiation_reconnect(node_factory, bitcoind):
disconnects = ['-WIRE_CLOSING_SIGNED',
'+WIRE_CLOSING_SIGNED']
l1, l2 = node_factory.line_graph(2, opts=[{'disconnect': disconnects,
'may_reconnect': True},
{'may_reconnect': True}])
l1.pay(l2, 200000000)
assert bitcoind.rpc.getmempoolinfo()['size'] == 0
l1.rpc.close(l2.info['id'])
l1.daemon.wait_for_log(r'State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN')
l2.daemon.wait_for_log(r'State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN')
# Now verify that the closing tx is in the mempool.
bitcoind.generate_block(6, wait_for_mempool=1)
sync_blockheight(bitcoind, [l1, l2])
for n in [l1, l2]:
# Ensure we actually got a mutual close.
> n.daemon.wait_for_log(r'Resolved FUNDING_TRANSACTION/FUNDING_OUTPUT by MUTUAL_CLOSE')
tests/test_closing.py:275:
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: We now renegotiate an interrupted close, even if we don't need it, instead of sending an error.
This is a bit too much boilerplate for these, which mainly do the same
thing.
We add annotaitons to new_peer_fd so the compiler knows that it cannot
return NULL, otherwise with -O3 we get:
```
lightningd/peer_control.c: In function ‘peer_connected_hook_final’:
lightningd/peer_control.c:1388:28: error: ‘error’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1388 | take(towire_connectd_peer_send_msg(NULL, &channel->peer->id,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lightningd/peer_control.c:1313:19: note: ‘error’ was declared here
1313 | const u8 *error;
| ^~~~~
lightningd/peer_control.c: In function ‘peer_spoke’:
lightningd/peer_control.c:1999:28: error: ‘error’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
1999 | take(towire_connectd_peer_send_msg(NULL, &peer->id,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:311: lightningd/peer_control.o] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The vast majority of incoming channel updates seem to be cut due
to age, which results in noisy logs. Similarly, the chanbackup
logging verbosity might better match the equivalent actions in
channeld, which are at the debug level.
Fixes: #8058
Changelog-None: introduced in 25.02
If the nSequence in the tx it produces is not at least the value we
test in the script, the tx will always fail:
```
error code: -26\nerror message:\nmandatory-script-verify-flag-failed (Locktime requirement not satisfied)
```
If we have a lease, the nSequence is max(lease-time-remaining,
to-self-delay), so have onchaind tell lightningd the correct nSequence.
Fixes: https://github.com/ElementsProject/lightning/issues/7460
Reported-by: https://github.com/pabpas
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Correctly collect our own (delayed) funds if we have a unilateral close when we are still offering a lease.
current height + to_self_delay[LOCAL] is correct normally, but if we
have an outstanding lease it's longer. Not a big issue, because
lightningd will retry until its spendable, but wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
What we expect to happen:
1. l3, which unilaterally closed, waits 4032 blocks (minus those mined) before
trying to send get its "to_local" funds back.
2. This should then succeed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are other ways we can disable it, of course (e.g full path name)
but this is the most obvious.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is neater than appending to some random file: we only do that once
if there's no "include" line to include a ".setconfig" file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setconfig` now has a `transient` flag which means it won't rewrite your config file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The chanbackup plugin should update emergency.recover every time we
receive a new commitment secret.
Additionally, chanbackup should be able to serialize both the new
SCB format and the legacy format.
Key Changes:
- Added a commitment_revocation plugin hook to update emergency.recover whenever a new revocation secret is received.
- Implemented support for the new emergency.recover format while maintaining compatibility with the legacy format.
This function enables direct persistence of shachain data, ensuring all relevant information is saved upon retrieval.
Key Changes:
- Adds 'wallet_stub_shachain_init' function to store a retrieved shachain in the database.
- Saves initial metadata ('min_index' and 'num_valid') to the 'shachains' table.
- Iterates through known shachain secrets, adding each entry to 'shachain_known' with position, index, and hash.
These fields enhance 'stub_chan' to make channels which are capable to
create penalty transaction in case the peer broadcasts an old revoked state.
Key Changes:
- Added new params to stub_chan() to accomodate shachain, basepoints, opener, and remote_to_self_delay
- Check if any field is NULL inside scb_chan->tlvs and assign them appropriately
We define a new subtype 'modern_scb_chan' and a new tlvtype 'scb_tlvs' which includes
all the relevant information to create a penalty transaction when the peer tries to cheat.
Key Changes:
- Rename the old format to 'legacy_scb_chan' and define a new type 'modern_scb_chan'
- Include TLVs to 'modern_scb_chan'
- Create a new msgtype 'static_chan_backup_with_tlvs'
- Modify 'struct channel' to include 'struct modern_scb_chan'
- Add these two types to 'varsize_types' in generate.py
This change allows adding a length prefix to a serialized TLV. It will
be particularly useful for serializing all 'scb_chan' entries in
the 'emergency.recover' file.
Key Changes:
- Removed the need to loop in towire_tlv and fromwire_tlv, so the if conditions have been modified accordingly.
- During serialization, the length of the TLV is calculated before appending it, and it is stored in a temporary variable.
- For fromwire_tlv, only a simple length adjustment is required, and no loop is needed here either.
This change will allow subtypes in wiregen files to have tlvstreams.
Shifting tlv structs above subtypes in header_template is done to prevent
forward declaration.
Since generate-wire prepends 'tlv_' in tlvname, we
have to modify fromwire_subtype_field and towire_subtype_field in
impl_template to accommodate this.
Changelog-Added: This PR would turn our peers into watchtower and enable SCB to create penalty txn.
The parameter max_htlc_value_in_flight_msat stablished by peers on
channel opening (BOLT02) can now be retrived from the
gossmods_from_listpeerchannels API.
Adapted the corresponding callback functions in renepay and askrene to
take into account that value as a constraint to the value we can send
through a channel.
Changelog-Add: fetch max_htlc_value_in_flight_msat from gossmods_listpeerchannels API
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-Added: JSON-RPC: `listpeerchannels` new output fields `their_max_total_htlc_out_msat` and `our_max_total_htlc_out_msat` as the value of `max_htlc_value_in_flight` (as of BOLT02) set by the local and remote nodes on channel creation.
Changelog-Deprecated: JSON-RPC: `listpeerchannels` value `max_total_htlc_in_msat`: use `our_max_total_htlc_out_msat` instead to follow spec naming convention.
Otherwise, deprecating a field causes SELECT * to fail:
```
> l1.rpc.sql(f"SELECT * FROM peerchannels;")
...
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: sql, payload: ('SELECT * FROM peerchannels;',), error: {'code': -1, 'message': 'query failed with access to peerchannels.max_total_htlc_in_msat is prohibited (Deprecated column table peerchannels.max_total_htlc_in_msat)'}
```
So if they use a wildcard, allow access: though "SELECT *" is fraught,
"COUNT(*)" is perfectly legit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The `enableoffer` JSON schema is present, but it is not included in the `GENERATE_MARKDOWN` list within the Makefile. This resulted into missing `.7` and `.7.md` files, leading to missing manpage and the documentation portal page.
Changelog-None.
Eduardo points out that payment_failed kind of over-promises: it may
actually not fail the payment now (with slow mode).
It's more an indiciation that we're not trying any more payment parts,
so rename it to payment_give_up.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can create this scenario by having one path force close. We take
the opportunity to log this, even in non-slow-mode, since it's interesting
(not our bug, but someone just lost money!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was requested by Michael of Boltz; it's mainly useful if you plan to
try failed payments on a *different* node. In that case, there's a
theoretical possibility that slow parts of this payment could combine with
that from a different node and overpay.
We don't allow this from the same node, already.
Changelog-Added: xpay: `xpay-slow-mode` makes xpay wait for all parts of a payment to complete before returning success or failure.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Don't rely on the current attempt, make caller calculate total.
2. Save preimage inside attempt, for slow mode.
3. Hoist it higher in the file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When tests CI on occur the version number has the following shape "1a86e50-modded".
We will always assume this is the latest version to make the version
checks pass
The fixtures in `tests/fixtures.py` modifies the path to
`lightningd/lightningd`.
I've adapted `pyln-testing` to accept an executable in the constructor.
This approach is more transparant and allows us to use the executable
path to query the version in the constructor.
Christian reported that this flag doesn't work on restart.
Indeed, it made us attempt *transient* rather than *persistent*
connections, but we still told connectd to connect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: connectd: `dev-no-reconnect-private` is respected on restart.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: lightning-cli: `help` messages using new-lines is now printed properly, enhancing readability and consistency.
Changelog-Changed: Interpret \n as the line separator in plugins to enhance the readability of lightning-cli help.
Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
[ Modified to use \n not | --RR ]
The `doc` variable was being initialised and processed but not used anywhere.
Changelog-None
Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
Don't print unnecessary log entries every time we receive an rpc_command
hook event. Only log when `pay` is called.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Following 6e4ff6a, nodes now check for their public address and
generate a blinded path for invoices if the address is not advertized.
This breaks several of the tests where the blinded path is expected
to have the entrypoint be the node itself.
Changelog-None
When we merged blinded paths for nodes with no address
(6e4ff6a7d2), this test
broke. We need to prevent that, otherwise:
```
> assert ret['successful_parts'] == 2
E assert 1 == 2
tests/test_xpay.py:677: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It wasn't documented, and hopefully nobody was using it.
Changelog-Fixed: `decode` for bolt12 invoices "features" field renamed to "invoice_features" (as documentation said)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turns out we weren't wiring them through! And libplugin wasn't reading them anyway.
Changelog-Fixed: lightningd: tell plugins our bolt12 features (so our bolt12 invoices explicitly allow MPP).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is deeply annoying, and we may have to support this properly
(using a separate algorithm entirely) if other implementations don't
fix their crap.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: xpay: suppress multi-part payment if invoice doesn't allow it (please, fix your nodes!)
This is an inefficient hack. Can you tell I really didn't want to
implement this? MPP was finalized in 2018 FFS.
We do this by adding another "auto" layer, which removes all too-small
channels, and then makes our MPP node pile all the funds into the largest
channel it chooses.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, if the total amount we're sending is less than the minimum amount
the channel insists on, we can eliminate it.
This fixes the problem that we're really trying to send a de-minumus
amount (rather than the more obscure case where we divide the amount
and then it is below the minimum).
After trying several other approaches, this was by far the cleanest!
Reported-by: https://github.com/JssDWt
Fixes: https://github.com/ElementsProject/lightning/issues/8045
Changelog-Fixed: xpay: don't simply give up if our total amount is less than htlc_minimum_msat on some channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Suggested-by: Matt Corallo
Fixes: https://github.com/ElementsProject/lightning/issues/7806
Changelog-Changed: Offers: we will use a blinded path if we have no advertized address (so payers wouldn't be able to connect directly).
I should have done this immediately after last release :(.
This turns various things off by default, even if deprecated APIs are enabled.
We remove the test of the to-be-removed params.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Config: `rest-port`, `rest-protocol`, `rest-host` and `rest-certs` disabled by default (use clnrest-*, or `i-promise-to-fix-broken-api-user=rest-port.clnrest-prefix` etc and PLEASE REPORT if you need this!)
Changelog-Deprecated: Config: `max-locktime-blocks` disabled by default (use `i-promise-to-fix-broken-api-user=max-locktime-blocks` and PLEASE REPORT if you need this!)
These are about to start logging warnings, so use modern versions for tests
which aren't explicitly about testing obsolete ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happened in CI:
```
algrind error file: valgrind-errors.16800
==16800== Thread 4 tokio-runtime-w:
==16800== Conditional jump or move depends on uninitialised value(s)
==16800== at 0x2466BF: _ZN175_$LT$axum..middleware..from_fn..FromFn$LT$F$C$S$C$I$C$$LP$T1$C$T2$C$T3$RP$$GT$$u20$as$u20$tower_service..Service$LT$http..request..Request$LT$axum_core..body..Body$GT$$GT$$GT$4call28_$u7b$$u7b$closure$u7d$$u7d$17h4cc36de5dd56d9feE.llvm.119371497468325184 (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x57BFFA: <axum::middleware::from_fn::ResponseFuture as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x37312B: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x372D8D: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x372F59: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x373A88: <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x32100D: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x38E326: _ZN91_$LT$axum..routing..route..RouteFuture$LT$E$GT$$u20$as$u20$core..future..future..Future$GT$4poll17h3f11114fc5dd51f8E.llvm.194396656150191000 (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x372315: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x372C05: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x371A15: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800== by 0x373A48: <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll (in /home/runner/work/lightning/lightning/plugins/clnrest)
==16800==
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Searches the first unused groupid and uses that number for the current
payment attempt.
We previously used the highest value of used groupid + 1, which breaks
in a few corner cases.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Change functions json_pay and json_paystatus to json_renepay
and json_renepaystatus to match the conventional naming.
This is helpful for grep searches.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
On lookup, we update the htable if any new addresses have been added.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightningd: startup time vastly improved for large nodes with pending closes and many bitcoin addresses.
It was long obsoleted, and never appears in the DB, but we do still have
to handle old ones in the code.
We removed it from the enum in
f342630b92 (v24.02) after deprecating it
in 23.02.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
From Whitslack's "startup takes 15 minutes" bug report, we can see
that wallet_can_spend is extremely slow. We exacerbate this by setting a large
bip32_max_index:
```
91.29% 0.02% lightningd lightningd [.] wallet_can_spend
|
--91.27%--wallet_can_spend
|
|--47.81%--scriptpubkey_p2tr_derkey
| |
| |--42.80%--scriptpubkey_p2tr
...
|--42.16%--bip32_key_from_parent
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In a previous [commit](https://github.com/ElementsProject/lightning/pull/8065/commits), I mistakenly removed the `lightning-` replacement logic from the base name also. This commit restores that functionality to re-enable schema checks.
Changelog-None.
When paying with injectpaymentonion we need to manually decode the error
from the onion.
Changelog-None.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Always use a fake destination node, the self-payments are no longer a
corner case for the routing problem in this way. Also it is ok for
get_routes to return routes with zero length.
Changelog-None.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Use injectpaymentonion for payments with |routes|=0,
ie. self-payments and blinded paths where our node is
the first node in the blinded path.
Changelog-None.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Collect the shared secrets when making a payment request.
We would need this if we use injectpaymentonion instead of sendonion.
Changelog-None.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Refactor create_onion function, now we could use the same function to
build onions for sendonion and injectpaymentonion.
Changelog-None.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
sendonion RPC does not allow to set the total amount in lightningd's
wallet, therefore it mixing sendpay and sendonion payment parts would
not work. That means for the time being we cannot complete a payment
initialized with sendpay until we add a total_amount parameter to
sendonion.
Changelog-None.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Fix error handling since we moved from sendpay to sendonion rpc.
With sendonion once a route fails we don't get the scid and node_id that
failed along the route, so we have to deduce those from our own internal
data.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Use renesendpay to send the payment allowing to pay to BOLT12 invoices
from a higher level interface.
Changelog-Add: renepay: Add support for BOLT12 payments
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changing route.amount to route.amount_deliver
for clarity. This variable hold the value that
the route delivers to destination.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Add an rpc to renepay that is similar to sendpay that
handles BOLT11 and BOLT12 payments.
This is not the most elegant solution but it is a workaround
until we implement it into lightningd which has more development
friction.
Changelog-None.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
[Added version tag into documentation, to show it was added in 25.02 --RR]
Changelog-Added: Plugins: new nofitications `plugin_stopped` and `plugin_started`
Otherwise it appears to be a leak:
==612637== 11,264 bytes in 1 blocks are still reachable in loss record 1 of 1
==612637== at 0x484D953: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==612637== by 0x1301F2: trace_init (trace.c:153)
==612637== by 0x13065D: trace_span_start (trace.c:263)
==612637== by 0x173968: db_open_ (utils.c:367)
==612637== by 0x17AE43: create_test_wallet (run-wallet.c:1313)
==612637== by 0x17C726: test_shachain_crud (run-wallet.c:1548)
==612637== by 0x18300E: main (run-wallet.c:2329)
Changelog-None
In the time it takes connectd to flush the log message for
, gossipd can already have
the announcement sent.
lightningd-1 2025-02-11T15:26:06.745Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_in WIRE_ANNOUNCEMENT_SIGNATURES
lightningd-2 2025-02-11T15:26:06.887Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: peer_in WIRE_ANNOUNCEMENT_SIGNATURES
lightningd-1 2025-02-11T15:26:06.897Z TRACE gossipd: Received node_announcement for node 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518
lightningd-1 2025-02-11T15:26:06.915Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_out WIRE_ANNOUNCEMENT_SIGNATURES
Changelog-None
Reported by Grubles on ARM64:
```
VALGRIND=1 valgrind -q --error-exitcode=7 --track-origins=yes --leak-check=full --show-reachable=yes --errors-for-leak-kinds=all common/test/run-tal_arr_randomize > /dev/null
==151138== Source and destination overlap in memcpy(0x4d69f08, 0x4d69f08, 8)
==151138== at 0x48CB68C: __GI_memcpy (vg_replace_strmem.c:1147)
==151138== by 0x41B50B: tal_arr_randomize_ (pseudorand.c:84)
==151138== by 0x41BB07: main (run-tal_arr_randomize.c:166)
==151138==
make: *** [Makefile:750: unittest/common/test/run-tal_arr_randomize] Error 7
```
It is correct: you can't overlap src and dst in memcpy. It *probably* works in
this case, but it's undefined!
Fixes: https://github.com/ElementsProject/lightning/issues/7030
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
After analyzing various weird cases where we ended up with duplicate
gossip_store entries, it could be explained by us not fully processing
the gossip store.
It's not clear that my assumptions that we would always see our own writes
are true: technically this may require an fsync(). So we now add the
check, and do an fsync and try again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: gossipd: more sanity checks that we are correctly updating the gossip_store file.
We had at least one report of overwriting the gossip_store file at
offset 1. Make sure this doesn't happen.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's actually the only one that uses it. We also tweak the way
gossip_store handles failure: gossmap_manage now tells it when to
reset the corrupted store.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of making a copy.
To measure the performance impact, I timed
tests/test_askrene.py::test_real_biases on my laptop.
No checksum check: 194.52s
Copying for checksum check: 202.81s
Zero-copy checksum check: 194.40s
But these numbers proved noisy. Still, doesn't hurt.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We assume if it's incorrect, we simply need to wait. If this proves incorrect,
we will see a stream of BROKEN log messages.
To measure the performance impact, I timed
tests/test_askrene.py::test_real_biases on my laptop.
Before: 194.52s
After: 202.81s
So it's marginal.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
While this shouldn't happen, it does (pending other fixes), and we stop reading the
gossip store until next time. The result is partial gossip, demonstrated beautifully
by NicolasDorier's report:
```
lightning_gossipd: gossmap: redundant channel_announce for 864063x1306x1, offsets 1272259 and 1784859!"
```
Gossipd stalld there and don't make more progress. So gossipd itself
doesn't see the entire gossip_store.
Then things get really batshit:
```
2025-02-04T05:53:28.582Z DEBUG gossipd: Store compact time: 1429910 msec
```
This took 1429 seconds to process. Why?
Because it hasn't been processing the gossip store fully, gossipd kept adding "new" records to the end:
```
2025-02-04T05:53:28.583Z DEBUG gossipd: gossip_store: Read 62716143/1739952/5158256/0 cannounce/cupdate/nannounce/delete from store in 31634458462 bytes, now 31634458440 bytes (populated=true)
```
It has 31GB of gossip in there! No wonder it took so long...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8035
Changelog-Fixed: gossipd: corruption in the gossip_store could cause ever-longer startup times and no gossip updates.
Default goes to stderr for LOG_UNUSUAL and higher.
We have to whitelist more cases in map_catchup so we don't spam the logs
with perfectly-expected (but ignored) messages though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only use it in one place, and that was simply to share an fd between
gossipd writing and gossipd reading, which may be causing our zfs problem
anyway.
In fact, it fixes a race if we don't have HAVE_PWRITEV.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have a report of this happening under ZFS. We cannot do much if
this really is a problem where we can't read back what we write, but
this avoids the immediate crash.
Fixes: https://github.com/ElementsProject/lightning/issues/7971
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: gossmap: occasional crash (at least on ZFS) reading gossip_store.
In 4e7ba96729 (ightningd: don't kill onchaind
if we are forcing a disconnect.) actually was something which happened
in our generate examples script.
This updates that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
plugins/Makefile has target/${RUST_PROFILE}/cln-grpc depend on the
generated files via $(MSGGEN_GENALL), but cln-rpc/Makefile adds to
that variable, so needs to be included first.
Here's an example build error:
```
Combining schemas from /home/rusty/lightning-ltest/doc/schemas into /home/rusty/lightning-ltest/contrib/msggen/msggen/schema.json
Created /home/rusty/lightning-ltest/contrib/msggen/msggen/schema.json from 2 files
error: failed to run custom build command for `cln-grpc v0.3.0 (/home/rusty/lightning-ltest/cln-grpc)`
Caused by:
process didn't exit successfully: `/home/rusty/lightning-ltest/target/debug/build/cln-grpc-95489e3ba33c0ab3/build-script-build` (exit status: 101)
--- stdout
cargo:rerun-if-changed=proto/node.proto
cargo:rerun-if-changed=proto
--- stderr
thread 'main' panicked at cln-grpc/build.rs:7:10:
called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "protoc failed: node.proto:134:52: \"AskreneageResponse\" is not defined.\nnode.proto:135:23: \"GetroutesRequest\" is not defined.\nnode.proto:135:50: \"GetroutesResponse\" is not defined.\nnode.proto:136:32: \"AskrenedisablenodeRequest\" is not defined.\nnode.proto:136:68: \"AskrenedisablenodeResponse\" is not defined.\nnode.proto:137:34: \"AskreneinformchannelRequest\" is not defined.\nnode.proto:137:72: \"AskreneinformchannelResponse\" is not defined.\nnode.proto:138:34: \"AskrenecreatechannelRequest\" is not defined.\nnode.proto:138:72: \"AskrenecreatechannelResponse\" is not defined.\nnode.proto:139:34: \"AskreneupdatechannelRequest\" is not defined.\nnode.proto:139:72: \"AskreneupdatechannelResponse\" is not defined.\nnode.proto:140:32: \"AskrenebiaschannelRequest\" is not defined.\nnode.proto:140:68: \"AskrenebiaschannelResponse\" is not defined.\nnode.proto:141:37: \"AskrenelistreservationsRequest\" is not defined.\nnode.proto:141:78: \"AskrenelistreservationsResponse\" is not defined.\nnode.proto:142:32: \"InjectpaymentonionRequest\" is not defined.\nnode.proto:142:68: \"InjectpaymentonionResponse\" is not defined.\nnode.proto:143:18: \"XpayRequest\" is not defined.\nnode.proto:143:40: \"XpayResponse\" is not defined.\nnode.proto:145:33: \"StreamBlockAddedRequest\" is not defined.\nnode.proto:145:74: \"BlockAddedNotification\" is not defined.\nnode.proto:146:40: \"StreamChannelOpenFailedRequest\" is not defined.\nnode.proto:146:88: \"ChannelOpenFailedNotification\" is not defined.\nnode.proto:147:36: \"StreamChannelOpenedRequest\" is not defined.\nnode.proto:147:80: \"ChannelOpenedNotification\" is not defined.\nnode.proto:148:30: \"StreamConnectRequest\" is not defined.\nnode.proto:148:68: \"PeerConnectNotification\" is not defined.\nnode.proto:149:32: \"StreamCustomMsgRequest\" is not defined.\nnode.proto:149:72: \"CustomMsgNotification\" is not defined.\n" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
make: *** [plugins/Makefile:305: target/debug/cln-grpc] Error 101
make: *** Waiting for unfinished jobs....
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't update this when we extended the timeout to 120 seconds in
ee3133f198 ("lightningd: increase
startup time for plugins to 120 seconds.")
```
def test_failing_plugins(directory):
fail_plugins = [
os.path.join(os.getcwd(), 'contrib/plugins/fail/failtimeout.py'),
os.path.join(os.getcwd(), 'contrib/plugins/fail/doesnotexist.py'),
]
for p in fail_plugins:
> with pytest.raises(subprocess.CalledProcessError):
E Failed: DID NOT RAISE <class 'subprocess.CalledProcessError'>
tests/test_plugin.py:420: Failed
----------------------------- Captured stdout call -----------------------------
{'github_repository': 'ElementsProject/lightning', 'github_sha': '83dca18c5e9610bfaac766f957387b9a1ec48f50', 'github_ref': 'refs/pull/7887/merge', 'github_ref_name': 'HEAD', 'github_run_id': 13253210143, 'github_head_ref': 'guilt/bolt-updates-after-24.11', 'github_run_number': 12237, 'github_base_ref': 'master', 'github_run_attempt': '2', 'testname': 'test_failing_plugins', 'start_time': 1739239278, 'end_time': 1739239340, 'outcome': 'fail'}
=========================== short test summary info ============================
FAILED tests/test_plugin.py::test_failing_plugins - Failed: DID NOT RAISE <class 'subprocess.CalledProcessError'>
============= 1 failed, 80 passed, 2 skipped in 855.37s (0:14:15) ==============
```
We can actually delete it before counters are updated:
```
wait_for(lambda: len(l3.rpc.listinvoices()['invoices']) == 2)
> assert l3.rpc.autoclean_status()['autoclean']['expiredinvoices']['cleaned'] == 3
E assert 1 == 3
tests/test_plugin.py:3266: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We thought it was a good idea to terminate channel subds, but we included onchaind as well!
```
2025-01-29T21:31:46.053Z UNUSUAL 03…00-channeld-chan#255683: Adding HTLC 1737 too slow: killing connection
2025-01-29T21:31:46.053Z INFO 03…00-chan#255683: Peer transient failure in CHANNELD_NORMAL: Adding HTLC timed out: killed connection
2025-01-29T21:31:46.053Z DEBUG 03…00-channeld-chan#255683: Status closed, but not exited. Killing
2025-01-29T21:31:46.054Z DEBUG 03…00-chan#255683: Failing HTLC 1738 due to peer death
2025-01-29T21:31:46.058Z DEBUG 03…00-chan#255683: Failing HTLC 1737 due to peer death
2025-01-29T21:31:46.058Z DEBUG 03…00-chan#255673: Forcing disconnect due to One channel had an error
2025-01-29T21:31:46.059Z DEBUG 03…00-onchaind-chan#255673: Status closed, but not exited. Killing
2025-01-29T21:31:46.093Z DEBUG 03…00-connectd: disconnect_peer
2025-01-29T21:31:46.093Z DEBUG 03…00-lightningd: peer_disconnect_done
```
Reported-by: @whitslack
Fixes: https://github.com/ElementsProject/lightning/issues/8055
Changelog-Fixed: onchaind: don't die if we fail an unrelated channel with the same peer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Raspberry Pi, with bitcoind running and full gossip topology may actually hit
this, and we have a report in practice.
Note that the comment is wrong, so fix that too.
Fixes: https://github.com/ElementsProject/lightning/issues/7724
Reported-by: m-schmoock
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently, pyln tests fail if the `lightning-` prefix is removed from schema/*.json files. In this release, we will update pyln to remove its reliance on this prefix, and in the next release, we will remove the prefixes from the files as well.
Changelog-None.
Test flake where the balance for lightning-2 went negative
```
> assert account_balance(l2, channel_id) == 0
tests/test_closing.py:1314:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/utils.py:183: in account_balance
m_sum -= Millisatoshi(m['debit_msat'])
contrib/pyln-client/pyln/client/lightning.py:193: in __sub__
return Millisatoshi(int(self) - int(other))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = -10000msat, v = -10000
```
Led me to look into this test. lightning-2 should go negative since we
roll back the amounts it's received by going to a prior database state.
Rather than trying to do the right thing with obviously broken node
records, instead we just stop trying to account for them correctly
(impossible).
I also noticed that the anchor tests were failing the utxo output
matchup, which we should be asserting on it. The HTLC RBF that our
anchor code creates was causing an issue by creating another wallet
deposit utxo under the HTLC output. We now optionally add this utxo
in the case that anchors are turned on.
Changelog-None: Fix test flake
It's really hard to tell what on earth went wrong when a coin movement
check fails, since we dont' return good error info.
Here we replace almost every `assert` with a proper check + error with
message to help make debugging easier.
cc @rustyrussell
Changelog-None: improve failure messages
If a user tries to do a splice without signing their inputs we now provide them with a nice error message and cancel the RPC since that wouldn’t be productive for the user anyway.
We also add a helpful message if they do the opposite — try to sign a PSBT where they did not add any inputs.
Changelog-Changed: Update prevents users from trying to splice unsigned PSBTs — protecting against potential issues.
Sure, we have to convert to and from the db set-of-pairs, but that's far simpler
than dealing with the current structure now we want to add code to *remove* from
the blacklist.
Changelog-Changed: JSON-RPC: `blacklistrune` no longer supports of runes over id 100,000,000.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This migration was introduced in dccbccf8f2 (pre 23.08), so the only way they
would need this is if they migration straight from 23.05 to 25.02. And then
the solution is to migration to a prior one first, but I'll bet good money
we never, ever see this message:
Commando runes still present? Migration removed in v25.02: call Rusty!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When building from a release zipfile and there is no `git` present, the version number lacks a ‘v’.
This triggers db.c to fail it’s `is_released_version` check, which fails because the version not start with a ‘v’ — thereby the database upgrade is rejected.
Issue reported by TonyV on Discord:
“Hey guys, I just upgraded to the latest release and am now getting
```
Refusing to irreversibly upgrade db from version 219 to 261 in non-final version 24.11.1 (use --database-upgrade=true to override)
```
I have the db backed up... but am I good to make those changes without breaking my channels?”
Changelog-Changed: Fix for people upgrading using source release zip packages.
test_closing_different_fees fails:
```
2024-10-14T08:43:30.2733614Z
2024-10-14T08:43:30.2734133Z # Now wait for them all to hit normal state, do payments
2024-10-14T08:43:30.2735205Z > l1.daemon.wait_for_logs(['update for channel .* now ACTIVE'] * num_peers
2024-10-14T08:43:30.2736233Z + ['to CHANNELD_NORMAL'] * num_peers)
2024-10-14T08:43:30.2736725Z
2024-10-14T08:43:30.2736903Z tests/test_closing.py:230:
...
2024-10-14T08:43:30.2761325Z E TimeoutError: Unable to find "[re.compile('update for channel .* now ACTIVE')]" in logs.
```
For some reason one of the channel_update injections does *not* evoke this message
from gossipd...
Changelog-None: debug!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- Moved the `Usage` section further down in `createrune` and `commando-rune` for improved UX.
- Added a new example for creating a rune with `read-only` restrictions, extending it to allow only payments of `less than 100,000 sats per day` using the `pay` or `xpay` methods.
- Adjusted formatting by appending an extra space after the `dependentUpon` condition, fixing `[*start* [*end*]][*relist*]` to `[*start* [*end*]] [*relist*]`.
- Relocated `Examples` from the expandable section to a standard heading, as examples are now already placed at the end of the page.
Changelog-None.
After merging the Rust-based `clnrest` plugin into the master, all three reproducible build scripts failed with the following error:
```
error: package `socketioxide v0.15.1` cannot be built because it requires rustc 1.75.0 or newer, while the currently active rustc version is 1.73.0
Either upgrade to rustc 1.75.0 or newer, or use
cargo update -p socketioxide@0.15.1 --precise ver
where `ver` is the latest version of `socketioxide` supporting rustc 1.73.0
make: *** [plugins/Makefile:304: target/release/clnrest] Error 101
```
To resolve this, we can either downgrade `socketioxide` to `v0.11.1`, which is compatible with `Rust >=v1.67` OR Upgrade Rust to `v1.75`.
Since the latest Rust version is `1.84`, upgrading to `1.75` seems like a reasonable choice, as it is already 13 months old.
Changelog-None.
If you re-run a node several times, the log file fills with info from
previous runs. To avoid looking at old logs, only parse the most recent
run's logs when looking for the magic CLN rest startup/deactivated
strings
Changelog-Fixed: startup-regtest.sh now only inspects the most recent run's logs for the active status of the clnrest plugin
We used to not print what happened with an HTLC in the `pay`
plugin. This meant that to follow the HTLCs we'd have to map the `pay`
HTLCs to the `lightningd` HTLCs, and then trace that. BY having `pay`
print the outcome as it sees it, we can make that tracking much
simpler, even allowing for tooling to do it for us.
Changelog-None This is a log-only change
Adding the newly introduced RPCs, `editdescriptionbyoutpoint` and `editdescriptionbypaymentid`, to the `Makefile` for generating the corresponding `.md` files required for the documentation portal.
Changelog-None.
I'm not sure why this happens, and suspect it is caused by an issue elsewhere, so
add some verbose debugging, don't crash.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8017
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `xpay` will refuse to make a 0msat payment (0msat invoice, partial payment, or manually-set on amountless invoice).
Fixes: https://github.com/ElementsProject/lightning/issues/8016
We have CI runs which timeout (after 2 hours). It's not clear why,
but we can at least eliminate CLN lockups as the answer.
Since pytest disabled the --timeout option on test shutdown, we could be
seeing an issue on stopping taking a long time?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we get "lucky" then commit tx will have short sig, one less weight (1 in 256 chance):
```
@unittest.skipIf(TEST_NETWORK != 'regtest', 'elementsd anchors not supportd')
def test_onchain_slow_anchor(node_factory, bitcoind):
"""We still use anchors for non-critical closes"""
l1, l2 = node_factory.line_graph(2)
# Don't let l1 succeed in sending commit tx
def censoring_sendrawtx(r):
return {'id': r['id'], 'result': {}}
l1.daemon.rpcproxy.mock_rpc('sendrawtransaction', censoring_sendrawtx)
close_start_depth = bitcoind.rpc.getblockchaininfo()['blocks']
# Make l1 close unilaterally.
l1.rpc.disconnect(l2.info['id'], force=True)
l1.rpc.close(l2.info['id'], unilateraltimeout=1)
# We will have a super-low-prio anchor spend.
l1.daemon.wait_for_log(r"Low-priority anchorspend aiming for block {} \(feerate 253\)".format(close_start_depth + 2016))
# Restart with reduced block time.
l1.stop()
l1.daemon.opts['dev-low-prio-anchor-blocks'] = 20
l1.start()
l1.daemon.wait_for_log("Low-priority anchorspend aiming for block {}".format(close_start_depth + 20))
l1.daemon.wait_for_log("Anchorspend for local commit tx")
# Won't go under 12 blocks though.
# Make sure it sees all these blocks at once, to avoid test flakes!
l1.stop()
bitcoind.generate_block(7)
l1.start()
height = bitcoind.rpc.getblockchaininfo()['blocks']
l1.daemon.wait_for_log(r"Low-priority anchorspend aiming for block {} \(feerate 7458\)".format(height + 13))
> l1.daemon.wait_for_log(r"Anchorspend for local commit tx fee 12335sat \(w=714\), commit_tx fee 4545sat \(w=768\): package feerate 11390 perkw")
```
Here's the log we *did* get:
```
2025-01-25T08:46:40.9399213Z lightningd-1 2025-01-25T08:40:06.312Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Anchorspend for local commit tx fee 12328sat (w=714), commit_tx fee 4545sat (w=767): package feerate 11392 perkw
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Doesn't always die messily, it seems?
```
@unittest.skipIf(TEST_NETWORK != 'regtest', 'elementsd doesnt yet support PSBT features we need')
@pytest.mark.openchannel('v2')
def test_rbf_reconnect_tx_construct(node_factory, bitcoind, chainparams):
disconnects = ['=WIRE_TX_ADD_INPUT', # Initial funding succeeds
'-WIRE_TX_ADD_INPUT',
'+WIRE_TX_ADD_INPUT',
'-WIRE_TX_ADD_OUTPUT',
'+WIRE_TX_ADD_OUTPUT',
'-WIRE_TX_COMPLETE',
'+WIRE_TX_COMPLETE',
'-WIRE_COMMITMENT_SIGNED',
'+WIRE_COMMITMENT_SIGNED']
l1, l2 = node_factory.get_nodes(2,
opts=[{'disconnect': disconnects,
'may_reconnect': True,
'dev-no-reconnect': None},
{'may_reconnect': True,
'dev-no-reconnect': None,
'broken_log': 'dualopend daemon died before signed PSBT returned'}])
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
amount = 2**24
chan_amount = 100000
bitcoind.rpc.sendtoaddress(l1.rpc.newaddr()['bech32'], amount / 10**8 + 0.01)
bitcoind.generate_block(1)
# Wait for it to arrive.
wait_for(lambda: len(l1.rpc.listfunds()['outputs']) > 0)
res = l1.rpc.fundchannel(l2.info['id'], chan_amount)
chan_id = res['channel_id']
vins = bitcoind.rpc.decoderawtransaction(res['tx'])['vin']
assert(only_one(vins))
prev_utxos = ["{}:{}".format(vins[0]['txid'], vins[0]['vout'])]
# Check that we're waiting for lockin
l1.daemon.wait_for_log(' to DUALOPEND_AWAITING_LOCKIN')
# rbf the lease with a higher amount
rate = int(find_next_feerate(l1, l2)[:-5])
# We 4x the feerate to beat the min-relay fee
next_feerate = '{}perkw'.format(rate * 4)
# Initiate an RBF
startweight = 42 + 172 # base weight, funding output
initpsbt = l1.rpc.utxopsbt(chan_amount, next_feerate, startweight,
prev_utxos, reservedok=True,
excess_as_change=True)
# Run through TX_ADD wires
for d in disconnects[1:-4]:
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
with pytest.raises(RpcError):
l1.rpc.openchannel_bump(chan_id, chan_amount, initpsbt['psbt'])
wait_for(lambda: l1.rpc.getpeer(l2.info['id'])['connected'] is False)
# The first TX_COMPLETE breaks
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
bump = l1.rpc.openchannel_bump(chan_id, chan_amount, initpsbt['psbt'])
with pytest.raises(RpcError):
update = l1.rpc.openchannel_update(chan_id, bump['psbt'])
wait_for(lambda: l1.rpc.getpeer(l2.info['id'])['connected'] is False)
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
# l1 should remember, l2 has forgotten
# l2 should send tx-abort, to reset
l2.daemon.wait_for_log(r'tx-abort: Sent next_funding_txid .* doesn\'t match ours .*')
l1.daemon.wait_for_log(r'Cleaned up incomplete inflight')
# abort doesn't cause a disconnect
assert l1.rpc.getpeer(l2.info['id'])['connected']
# The next TX_COMPLETE break (both remember) + they break on the
# COMMITMENT_SIGNED during the reconnect
bump = l1.rpc.openchannel_bump(chan_id, chan_amount, initpsbt['psbt'])
with pytest.raises(RpcError):
update = l1.rpc.openchannel_update(chan_id, bump['psbt'])
wait_for(lambda: l1.rpc.getpeer(l2.info['id'])['connected'] is False)
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
l2.daemon.wait_for_logs([r'Got dualopend reestablish',
r'No commitment, not sending our sigs'])
l1.daemon.wait_for_logs([r'Got dualopend reestablish',
r'No commitment, not sending our sigs',
r'dev_disconnect: -WIRE_COMMITMENT_SIGNED',
'peer_disconnect_done'])
assert not l1.rpc.getpeer(l2.info['id'])['connected']
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
# COMMITMENT_SIGNED disconnects *during* the reconnect
# We can't bump because the last negotiation is in the wrong state
with pytest.raises(RpcError, match=r'Funding sigs for this channel not secured'):
l1.rpc.openchannel_bump(chan_id, chan_amount, initpsbt['psbt'])
# l2 reconnects, but doesn't have l1's commitment
> l2.daemon.wait_for_logs([r'Got dualopend reestablish',
r'No commitment, not sending our sigs',
# This is a BROKEN log, it's expected!
r'dualopend daemon died before signed PSBT returned'])
tests/test_opening.py:944:
...
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('dualopend daemon died before signed PSBT returned')]" in logs.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By having gossipwith filter out messages we don't want, we can get the counts of
expected messages correct, and not hit errors like this:
```
def test_gossip_throttle(node_factory, bitcoind, chainparams):
"""Make some gossip, test it gets throttled"""
l1, l2, l3, l4 = node_factory.line_graph(4, wait_for_announce=True,
opts=[{}, {}, {}, {'dev-throttle-gossip': None}])
# We expect: self-advertizement (3 messages for l1 and l4) plus
# 4 node announcements, 3 channel announcements and 6 channel updates.
# We also expect it to send a timestamp filter message.
# (We won't take long enough to get a ping!)
expected = 4 + 4 + 3 + 6 + 1
# l1 is unlimited
start_fast = time.time()
out1 = subprocess.run(['devtools/gossipwith',
'--all-gossip',
'--hex',
'--network={}'.format(TEST_NETWORK),
'--max-messages={}'.format(expected),
'{}@localhost:{}'.format(l1.info['id'], l1.port)],
check=True,
timeout=TIMEOUT, stdout=subprocess.PIPE).stdout.split()
time_fast = time.time() - start_fast
assert time_fast < 2
# Remove timestamp filter, since timestamp will change!
out1 = [m for m in out1 if not m.startswith(b'0109')]
# l4 is throttled
start_slow = time.time()
out2 = subprocess.run(['devtools/gossipwith',
'--all-gossip',
'--hex',
'--network={}'.format(TEST_NETWORK),
'--max-messages={}'.format(expected),
'{}@localhost:{}'.format(l4.info['id'], l4.port)],
check=True,
timeout=TIMEOUT, stdout=subprocess.PIPE).stdout.split()
time_slow = time.time() - start_slow
assert time_slow > 3
# Remove timestamp filter, since timestamp will change!
out2 = [m for m in out2 if not m.startswith(b'0109')]
# Contents should be identical (once uniquified, since each
# doubles-up on its own gossip)
> assert set(out1) == set(out2)
E AssertionError: assert {b'010054b1907bdf639c9060e0fa4bca02419c46f75a99f0908b87a2e09711d5d031ba76b8fd07acc8be1b2fac9e31efb808e5d362c32ef4665...
E Extra items in the left set:
E b'01010ad5be8b9ba029245c2ae2d667af7ead7c0129c479c7fd7145a9b65931e90222082e6e4ab37ef60ebd10f1493d73e8bf7a40c4ae5f7d87cc...8488830b60f7e744ed9235eb0b1ba93283b315c035180266e44a554e494f524245414d2d333930353033622d6d6f64646564000000000000000000'
E Extra items in the right set:
E b'01079f87eb580b9e5f11dc211e9fb66abb3699999044f8fe146801162393364286c6000000010000006c010101'
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So not every message type counts: this is useful when we want to get a specific number
of a specific type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This caused a "flake" in testing, because it's wrong:
```
_____________________________ test_gossip_pruning ______________________________
[gw2] linux -- Python 3.10.16 /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/bin/python
node_factory = <pyln.testing.utils.NodeFactory object at 0x7f0267530490>
bitcoind = <pyln.testing.utils.BitcoinD object at 0x7f0267532b30>
def test_gossip_pruning(node_factory, bitcoind):
""" Create channel and see it being updated in time before pruning
"""
l1, l2, l3 = node_factory.get_nodes(3, opts={'dev-fast-gossip-prune': None,
'allow_bad_gossip': True,
'autoconnect-seeker-peers': 0})
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
l2.rpc.connect(l3.info['id'], 'localhost', l3.port)
scid1, _ = l1.fundchannel(l2, 10**6)
scid2, _ = l2.fundchannel(l3, 10**6)
mine_funding_to_announce(bitcoind, [l1, l2, l3])
wait_for(lambda: l1.rpc.listchannels(source=l1.info['id'])['channels'] != [])
l1_initial_cupdate_timestamp = only_one(l1.rpc.listchannels(source=l1.info['id'])['channels'])['last_update']
# Get timestamps of initial updates, so we can ensure they change.
# Channels should be activated locally
> wait_for(lambda: [c['active'] for c in l1.rpc.listchannels()['channels']] == [True] * 4)
```
Here you can see it has pruned:
```
lightningd-1 2025-01-24T07:39:40.873Z DEBUG gossipd: Pruning channel 105x1x0 from network view (ages 1737704380 and 0)
...
lightningd-1 2025-01-24T07:39:50.941Z UNUSUAL lightningd: Bad gossip order: could not find channel 105x1x0 for peer's channel update
```
Changelog-Fixed: Protocol: we were overzealous in pruning channels if we hadn't seen one side's gossip update yet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can be a bit early in our assertion:
```
@unittest.skipIf(os.getenv('TEST_DB_PROVIDER', 'sqlite3') != 'sqlite3', "sqlite3-specific DB manip")
def test_reconnect_remote_sends_no_sigs(node_factory):
"""We re-announce, even when remote node doesn't send its announcement_signatures on reconnect.
"""
l1, l2 = node_factory.line_graph(2, wait_for_announce=True, opts={'may_reconnect': True,
'dev-no-reconnect': None})
# Wipe l2's gossip_store
l2.stop()
gs_path = os.path.join(l2.daemon.lightning_dir, TEST_NETWORK, 'gossip_store')
os.unlink(gs_path)
l2.start()
# l2 will now uses (REMOTE's) announcement_signatures it has stored
wait_for(lambda: l2.rpc.listchannels()['channels'] != [])
# Remove remote signatures from l1 so it asks for them (and delete gossip store)
l1.db_manip("UPDATE channels SET remote_ann_node_sig=NULL, remote_ann_bitcoin_sig=NULL")
gs_path = os.path.join(l1.daemon.lightning_dir, TEST_NETWORK, 'gossip_store')
os.unlink(gs_path)
l1.restart()
l1.connect(l2)
l1needle = l1.daemon.logsearch_start
l2needle = l2.daemon.logsearch_start
# l1 asks once, l2 replies once.
# Make sure we get all the msgs!
time.sleep(5)
l1.daemon.wait_for_log('peer_out WIRE_ANNOUNCEMENT_SIGNATURES')
l2.daemon.wait_for_log('peer_out WIRE_ANNOUNCEMENT_SIGNATURES')
l1msgs = [l.split()[4] for l in l1.daemon.logs[l1needle:] if 'WIRE_ANNOUNCEMENT_SIGNATURES' in l]
> assert l1msgs == ['peer_out', 'peer_in']
E AssertionError: assert ['peer_out'] == ['peer_out', 'peer_in']
E Right contains one more item: 'peer_in'
E Full diff:
E - ['peer_out', 'peer_in']
E + ['peer_out']
```
```
lightningd-2 2025-01-24T05:53:22.862Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_out WIRE_ANNOUNCEMENT_SIGNATURES
lightningd-1 2025-01-24T05:53:22.864Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_in WIRE_ANNOUNCEMENT_SIGNATURES
lightningd-1 2025-01-24T05:53:22.885Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: channel_gossip: received announcement sigs for 103x1x0 (we have 103x1x0)
{'github_repository': 'ElementsProject/lightning', 'github_sha': 'e9d36f2b8ecd45882753cbe062c355e40bc7109c', 'github_ref': 'refs/pull/8027/merge', 'github_ref_name': 'HEAD', 'github_run_id': 12943530601, 'github_head_ref':
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes they connect too fast, so we don't get a chance to ratelimit all of them:
```
def test_connect_ratelimit(node_factory, bitcoind):
"""l1 has 5 peers, restarts, make sure we limit"""
nodes = node_factory.get_nodes(6,
opts=[{'dev-limit-connections-inflight': None, 'may_reconnect': True}] + [{'may_reconnect': True}] * 5)
l1 = nodes[0]
nodes = nodes[1:]
addr = l1.rpc.newaddr()['bech32']
for n in nodes:
bitcoind.rpc.sendtoaddress(addr, (FUNDAMOUNT + 1000000) / 10**8)
bitcoind.generate_block(1, wait_for_mempool=len(nodes))
sync_blockheight(bitcoind, [l1])
for n in nodes:
l1.rpc.connect(n.info['id'], 'localhost', n.port)
l1.rpc.fundchannel(n.info['id'], FUNDAMOUNT)
# Make sure all channels are established and announced.
bitcoind.generate_block(6, wait_for_mempool=len(nodes))
wait_for(lambda: len(l1.rpc.listchannels()['channels']) == len(nodes) * 2)
assert not l1.daemon.is_in_log('Unblocking for')
l1.restart()
# The first will be ok, but others should block and be unblocked.
> l1.daemon.wait_for_logs((['Unblocking for ']
+ ['Too many connections, waiting'])
* (len(nodes) - 1))
tests/test_connection.py:4721:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyln.testing.utils.LightningD object at 0x7f6e288a3a60>
regexs = ['Unblocking for ', 'Too many connections, waiting', 'Unblocking for ', 'Too many connections, waiting', 'Unblocking for ', 'Too many connections, waiting', ...]
timeout = 180
def wait_for_logs(self, regexs, timeout=TIMEOUT):
"""Look for `regexs` in the logs.
The logs contain tailed stdout of the process. We look for each regex
in `regexs`, starting from `logsearch_start` which normally is the
position of the last found entry of a previous wait-for logs call.
The ordering inside `regexs` doesn't matter.
We fail if the timeout is exceeded or if the underlying process
exits before all the `regexs` were found.
If timeout is None, no time-out is applied.
"""
logging.debug("Waiting for {} in the logs".format(regexs))
exs = [re.compile(r) for r in regexs]
start_time = time.time()
while True:
if self.logsearch_start >= len(self.logs):
if not self.logs_catchup():
time.sleep(0.25)
if timeout is not None and time.time() > start_time + timeout:
print("Time-out: can't find {} in logs".format(exs))
for r in exs:
if self.is_in_log(r):
print("({} was previously in logs!)".format(r))
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('Unblocking for '), re.compile('Too many connections, waiting')]" in logs.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
____________________ ERROR at teardown of test_xpay_maxfee _____________________
...
# Format a nice list of everything that went wrong and raise an exception
request.node.has_errors = True
> raise ValueError(str(errors))
E ValueError:
E Node errors:
E - lightningd-1: Node exited with return code 1
E Global errors:
```
And:
```
@unittest.skipIf(TEST_NETWORK != 'regtest', 'too dusty on elements')
def test_xpay_maxfee(node_factory, bitcoind, chainparams):
"""Test which shows that we don't excees maxfee"""
outfile = tempfile.NamedTemporaryFile(prefix='gossip-store-')
subprocess.check_output(['devtools/gossmap-compress',
'decompress',
'--node-map=3301=022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59',
'tests/data/gossip-store-2024-09-22.compressed',
outfile.name]).decode('utf-8').splitlines()
AMOUNT = 100_000_000
# l2 will warn l1 about its invalid gossip: ignore.
# We throttle l1's gossip to avoid massive log spam.
> l1, l2 = node_factory.line_graph(2,
# This is in sats, so 1000x amount we send.
fundamount=AMOUNT,
opts=[{'gossip_store_file': outfile.name,
'subdaemon': 'channeld:../tests/plugins/channeld_fakenet',
'allow_warning': True,
'dev-throttle-gossip': None},
{'allow_bad_gossip': True}])
tests/test_xpay.py:509:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:1720: in line_graph
nodes = self.get_nodes(num_nodes, opts=opts)
contrib/pyln-testing/pyln/testing/utils.py:1602: in get_nodes
return [j.result() for j in jobs]
contrib/pyln-testing/pyln/testing/utils.py:1602: in <listcomp>
return [j.result() for j in jobs]
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/concurrent/futures/_base.py:458: in result
return self.__get_result()
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/concurrent/futures/_base.py:403: in __get_result
raise self._exception
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/concurrent/futures/thread.py:58: in run
result = self.fn(*self.args, **self.kwargs)
contrib/pyln-testing/pyln/testing/utils.py:1653: in get_node
node.start(wait_for_bitcoind_sync)
contrib/pyln-testing/pyln/testing/utils.py:1015: in start
self.daemon.start(stderr_redir=stderr_redir)
contrib/pyln-testing/pyln/testing/utils.py:671: in start
self.wait_for_log("Server started with public key")
contrib/pyln-testing/pyln/testing/utils.py:355: in wait_for_log
return self.wait_for_logs([regex], timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyln.testing.utils.LightningD object at 0x7f27ab586c20>
regexs = ['Server started with public key'], timeout = 180
def wait_for_logs(self, regexs, timeout=TIMEOUT):
"""Look for `regexs` in the logs.
The logs contain tailed stdout of the process. We look for each regex
in `regexs`, starting from `logsearch_start` which normally is the
position of the last found entry of a previous wait-for logs call.
The ordering inside `regexs` doesn't matter.
We fail if the timeout is exceeded or if the underlying process
exits before all the `regexs` were found.
If timeout is None, no time-out is applied.
"""
logging.debug("Waiting for {} in the logs".format(regexs))
exs = [re.compile(r) for r in regexs]
start_time = time.time()
while True:
if self.logsearch_start >= len(self.logs):
if not self.logs_catchup():
time.sleep(0.25)
if timeout is not None and time.time() > start_time + timeout:
print("Time-out: can't find {} in logs".format(exs))
for r in exs:
if self.is_in_log(r):
print("({} was previously in logs!)".format(r))
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('Server started with public key')]" in logs.
```
gossipd (and other plugins) simply take too long to digest the gossmap under valgrind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I'd been resisting this, but some tests really do want "hang up after
*receiving* this", and it's more reliable than getting the peer to
"hang up afer *sending* this" which seems not always work.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Pyln logger's configuration was unexpectedly being overwritten by the root logger during the autogenerated examples test, complicating error debugging.
We resolved this by introducing a pytest fixture to reapply the logger configuration before the tests executes.
Changelog-None.
Fixes: https://github.com/ElementsProject/lightning/issues/8023
Currently on I see link failures like the following:
```bash
./configure
<snip>
checking for libsodium with IETF chacha20 variants... yes
<snip>
ar libccan.a
ld ccan/ccan/cdump/tools/cdump-enumstr
ld: library 'sodium' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
The configure check passes under Clang here.
Changelog-Fixed: build: fix linking against libsodium on macOS.
The current check fails when building with newer GCC, i.e:
```bash
error: ‘crypto_secretstream_xchacha20poly1305_init_push’ reading 32 bytes from a region of size 3 [-Werror=stringop-overread]
12 | crypto_secretstream_xchacha20poly1305_init_push(&crypto_state, header,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
13 | data);
```
This inadvertently results in the release binaries for Ubuntu 22.04 &
24.04, not having a runtime libsodium dependency, but instead using the
bundled lib.
It's not clear to me this is actually enough to fix (all) the release builds,
as the build containers need to have `libsodium-dev`, not just `libsodium`
in them, and it's not clear to me which packages are actually present
looking at the repro build scripts.
Changelog-Fixed: build: libsodium configure check fixed to work with newer GCC.
The updated API requires typed htables to explicitly state whether they
allow duplicates: for most cases we don't, but we've had issues in the
past.
This is a big patch, but mainly mechanical.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I wanted to make sure we didn't have a bug in our htable routine,
so I wrote this test. We don't, but leave it in.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- Install `poetry-plugin-export` as a separate step.
- Remove `--no-update` option from `poetry lock` as it's now default behavior.
- Add `poetry lock` to the command chain after removing cln-rest and wss-proxy.
https://github.com/python-poetry/poetry/releases/tag/2.0.0
Undertaken to upgrade QEMU to 7.2. Also upgrades Python to 3.11
implicitly and migrates Python dependency management to virtual environments.
Changelog-Changed: Released Docker images are now based on Debian Bookworm
This doesn't do anything, because it's trying to create a symlink for a
verison that doesn't exist. The version installed via brew is 0.22.5.
In any case, on any recent macOS system, this should not be necessary.
Changelog-None.
Poetry will no longer include the `poetry-plugin-export` plugin by default, which is essential for exporting dependencies. So, we now need to install it explicitly.
Keep a proper cache of all possible ones. I think this may be the
timeout problem: according to the logs, channeld_fakenet stops responding
and thus HTLCs eventually time out.
```
```
2024-12-16T23:16:16.4874420Z lightningd-1 2024-12-16T22:45:14.068Z UNUSUAL 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: Adding HTLC 18446744073709551615 too slow: killing connection
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can get the reply_short_channel_ids_end in the messages when
we make a query:
```
2024-11-29T07:39:28.8550652Z time_fast = time.time() - start_fast
2024-11-29T07:39:28.8551067Z assert time_fast < 2
2024-11-29T07:39:28.8551487Z out3 = [m for m in out3 if not m.startswith(b'0109')]
2024-11-29T07:39:28.8552158Z > assert set(out1) == set(out3)
...
2024-11-29T07:39:28.8675516Z E Extra items in the right set:
2024-11-29T07:39:28.8675887Z E b'010606226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f01'
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This should help us fix the flakes. Dealing with them was a major
headache at the end of the last release (and they covered up real bugs!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Cut & paste from the forwarding code, where we don't let onions use the
unannounced scids. Obviously local commands can use them.
Reported-by: @michael1011
Changelog-Fixed: JSON-RPC: xpay now works through unannounced channels.
Note that the slight code reorder changes the JSON order, which is generally
undefined, but our doc checker is very strict!
Changelog-Changed: `xpay` now gives the same JSON success return as documented by `pay` when `xpay-handle-pay` is set.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7923
maxfeepercent is use by Zeus, so let's make that work.
maxfee is more precise, so it's the only xpay option (maxfee was added
to pay later).
[ Fix to ppm logic by Lagrang3, thanks! --RR ]
Fixes: https://github.com/ElementsProject/lightning/issues/7926
Changelog-Changed: JSON-RPC: With `xpay-handle-pay` set, xpay will now be used even if `pay` uses maxfeeprecent or exemptfee parameters (e.g. Zeus)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means that it gets shown in listsendpays: omitting this broke spark, apparently!
Changelog-Changed: `xpay` now populates more fields, so `listsendpays` and `listpays` show `destination` and `amount_msat` fields for xpay payments.
Fixes: https://github.com/ElementsProject/lightning/issues/7881
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If they give us the invstring, we can at least set who signed the invoice. Of course,
it might not be a real node_id (with blinded paths).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This appears in listsendpays / listpays, and is useful information (if we know!).
This doesn't fix old payments, but means that xpay can use this for new payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I used an unknown arg and it didn't complain. injectpaymentonion's schema predated
the sweep which fixed up use of "additionalProperties".
Indeed, we were missing some properties!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There was a wrong assumption that the number of bytes read
by `cli_read` would get us for each correctly read token
two extra CR characters. As a matter of fact one could read
enough characters to parse the first token, but the two
extra CR characters are not guaranteed.
```
==143570== Memcheck, a memory error detector
==143570== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==143570== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==143570== Command: /home/lagrange/BACKUP/l4-appdata/github/lagrang3/lightning/cli/lightning-cli --lightning-dir=/tmp/askrene_benchmark/lightning -k getroutes source=032ed0d87ba2bd68e3a386717cf2faaae4fa6d6da247986b1997113930e4f841d5 destination=03b2f16bf472dd03c55c2ce9910aab717321db4489cd87df5225adadb08031da4b amount_msat=100000sat final_cltv=6 layers=[] maxfee_msat=500sat
==143570==
==143570== Invalid read of size 1
==143570== at 0x484A430: memmove (vg_replace_strmem.c:1382)
==143570== by 0x10C3D2: main (lightning-cli.c:871)
==143570== Address 0x4a62f80 is 0 bytes after a block of size 1,040 alloc'd
==143570== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==143570== by 0x11402E: allocate (tal.c:256)
==143570== by 0x11471E: tal_alloc_ (tal.c:473)
==143570== by 0x1147EA: tal_alloc_arr_ (tal.c:517)
==143570== by 0x10C206: main (lightning-cli.c:816)
==143570==
==143570== Invalid read of size 1
==143570== at 0x484A43D: memmove (vg_replace_strmem.c:1382)
==143570== by 0x10C3D2: main (lightning-cli.c:871)
==143570== Address 0x4a62f81 is 1 bytes after a block of size 1,040 alloc'd
==143570== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==143570== by 0x11402E: allocate (tal.c:256)
==143570== by 0x11471E: tal_alloc_ (tal.c:473)
==143570== by 0x1147EA: tal_alloc_arr_ (tal.c:517)
==143570== by 0x10C206: main (lightning-cli.c:816)
==143570==
==143570== Invalid write of size 1
==143570== at 0x484A433: memmove (vg_replace_strmem.c:1382)
==143570== by 0x10C3D2: main (lightning-cli.c:871)
==143570== Address 0x4a62f80 is 0 bytes after a block of size 1,040 alloc'd
==143570== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==143570== by 0x11402E: allocate (tal.c:256)
==143570== by 0x11471E: tal_alloc_ (tal.c:473)
==143570== by 0x1147EA: tal_alloc_arr_ (tal.c:517)
==143570== by 0x10C206: main (lightning-cli.c:816)
```
Changelog-Fixed: lightning-cli: fix "malformed response" bug
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
We ratelimited DEBUG messages, but that can be annoying and cause us to miss things.
We demoted the worst offenders in the last release, to TRACE level.
Now, only log trace if it's wanted, and never suppress DEBUG.
Changelog-Changed: Logging: we no longer suppress DEBUG messages from subdaemons.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7917
header sys/errno.h gets re-directed to errno.h leading to warning and
then failure so instead directly referencing the header
Changelog-None: change header
Signed-off-by: Lakshya Singh <lakshay.singh1108@gmail.com>
We are checking if `txw->depth` is `-1` and then print it, when we
clearly want `depth` instead.
Changelog-Fixed logs: When printing depths some unsigned numbers could overflow
It's actually tested by fetchinvoice, but doing an explicit test in Python
allows for schema checking!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `injectonionmessage` API simplified and documented.
It's undocumented and only used in one place, so we can change it (it
was new in 24.08).
We really want to be able to just handle a raw onionmessage: this allows
oblivious sending of messages, but also, in combination with the coming
onionmessage_forward_fail notification, allows us to connect then
reinject.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Clarify that the ability to reproducibly build a piece of software has
nothing to do with being a maintainer.
Minor typo/grammar fixes.
Changelog-None.
See: https://github.com/ElementsProject/lightning/issues/7899
A node with 23 connections gets far too many debug messages.
Changelog-Fixed: `gossipd` now does logging at trace, not debug level.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- Removed branch testing as it is not required. Point releases will trigger the event due to `tag` push even when it is not from master branch.
- Added version in inputs for testing
- Added missing `inputs` tag
Changelog-None.
It's freaking people out when they see things like:
```
2024-11-11T05:26:41.281Z DEBUG ...53c-connectd: peer_out INVALID 22859
```
Fixes: https://github.com/ElementsProject/lightning/issues/7802
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: connectd: log unknown messages as "UNKNOWN" not "INVALID" to avoid freaking people out.
We were handing "maxfee" to every getroutes call, even if we had already
used some of the fees.
Reported-by: @daywalker90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: xpay is new this release.
In fact, there are several places where we try to decode old invoices,
and they should all work. The only place we should enforce expiration is
when we're going to pay.
This also revealed that xpay wasn't checking bolt11 expiries!
Reported-by: hMsats
Fixes: https://github.com/ElementsProject/lightning/issues/7869
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `decode` refused to decode expired bolt12 invoices.
```
error[E0277]: the trait bound `cln_rpc::model::requests::AskrenereserveRequest: From<pb::AskrenereserveRequest>` is not satisfied
--> cln-grpc/src/server.rs:3994:56
|
3994 | let req: requests::AskrenereserveRequest = req.into();
| ^^^^ the trait `From<pb::AskrenereserveRequest>` is not implemented for `cln_rpc::model::requests::AskrenereserveRequest`, which is required by `pb::AskrenereserveRequest: Into<_>`
|
= note: required for `pb::AskrenereserveRequest` to implement `Into<cln_rpc::model::requests::AskrenereserveRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::AskreneageRequest: From<pb::AskreneageRequest>` is not satisfied
--> cln-grpc/src/server.rs:4026:52
|
4026 | let req: requests::AskreneageRequest = req.into();
| ^^^^ the trait `From<pb::AskreneageRequest>` is not implemented for `cln_rpc::model::requests::AskreneageRequest`, which is required by `pb::AskreneageRequest: Into<_>`
|
= note: required for `pb::AskreneageRequest` to implement `Into<cln_rpc::model::requests::AskreneageRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::GetroutesRequest: From<pb::GetroutesRequest>` is not satisfied
--> cln-grpc/src/server.rs:4058:51
|
4058 | let req: requests::GetroutesRequest = req.into();
| ^^^^ the trait `From<pb::GetroutesRequest>` is not implemented for `cln_rpc::model::requests::GetroutesRequest`, which is required by `pb::GetroutesRequest: Into<_>`
|
= note: required for `pb::GetroutesRequest` to implement `Into<cln_rpc::model::requests::GetroutesRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::AskrenedisablenodeRequest: From<pb::AskrenedisablenodeRequest>` is not satisfied
--> cln-grpc/src/server.rs:4090:60
|
4090 | let req: requests::AskrenedisablenodeRequest = req.into();
| ^^^^ the trait `From<pb::AskrenedisablenodeRequest>` is not implemented for `cln_rpc::model::requests::AskrenedisablenodeRequest`, which is required by `pb::AskrenedisablenodeRequest: Into<_>`
|
= note: required for `pb::AskrenedisablenodeRequest` to implement `Into<cln_rpc::model::requests::AskrenedisablenodeRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::AskreneinformchannelRequest: From<pb::AskreneinformchannelRequest>` is not satisfied
--> cln-grpc/src/server.rs:4122:62
|
4122 | let req: requests::AskreneinformchannelRequest = req.into();
| ^^^^ the trait `From<pb::AskreneinformchannelRequest>` is not implemented for `cln_rpc::model::requests::AskreneinformchannelRequest`, which is required by `pb::AskreneinformchannelRequest: Into<_>`
|
= note: required for `pb::AskreneinformchannelRequest` to implement `Into<cln_rpc::model::requests::AskreneinformchannelRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::AskrenecreatechannelRequest: From<pb::AskrenecreatechannelRequest>` is not satisfied
--> cln-grpc/src/server.rs:4154:62
|
4154 | let req: requests::AskrenecreatechannelRequest = req.into();
| ^^^^ the trait `From<pb::AskrenecreatechannelRequest>` is not implemented for `cln_rpc::model::requests::AskrenecreatechannelRequest`, which is required by `pb::AskrenecreatechannelRequest: Into<_>`
|
= note: required for `pb::AskrenecreatechannelRequest` to implement `Into<cln_rpc::model::requests::AskrenecreatechannelRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::AskreneupdatechannelRequest: From<pb::AskreneupdatechannelRequest>` is not satisfied
--> cln-grpc/src/server.rs:4186:62
|
4186 | let req: requests::AskreneupdatechannelRequest = req.into();
| ^^^^ the trait `From<pb::AskreneupdatechannelRequest>` is not implemented for `cln_rpc::model::requests::AskreneupdatechannelRequest`, which is required by `pb::AskreneupdatechannelRequest: Into<_>`
|
= note: required for `pb::AskreneupdatechannelRequest` to implement `Into<cln_rpc::model::requests::AskreneupdatechannelRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::AskrenebiaschannelRequest: From<pb::AskrenebiaschannelRequest>` is not satisfied
--> cln-grpc/src/server.rs:4218:60
|
4218 | let req: requests::AskrenebiaschannelRequest = req.into();
| ^^^^ the trait `From<pb::AskrenebiaschannelRequest>` is not implemented for `cln_rpc::model::requests::AskrenebiaschannelRequest`, which is required by `pb::AskrenebiaschannelRequest: Into<_>`
|
= note: required for `pb::AskrenebiaschannelRequest` to implement `Into<cln_rpc::model::requests::AskrenebiaschannelRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::AskrenelistreservationsRequest: From<pb::AskrenelistreservationsRequest>` is not satisfied
--> cln-grpc/src/server.rs:4250:65
|
4250 | let req: requests::AskrenelistreservationsRequest = req.into();
| ^^^^ the trait `From<pb::AskrenelistreservationsRequest>` is not implemented for `cln_rpc::model::requests::AskrenelistreservationsRequest`, which is required by `pb::AskrenelistreservationsRequest: Into<_>`
|
= note: required for `pb::AskrenelistreservationsRequest` to implement `Into<cln_rpc::model::requests::AskrenelistreservationsRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::InjectpaymentonionRequest: From<pb::InjectpaymentonionRequest>` is not satisfied
--> cln-grpc/src/server.rs:4282:60
|
4282 | let req: requests::InjectpaymentonionRequest = req.into();
| ^^^^ the trait `From<pb::InjectpaymentonionRequest>` is not implemented for `cln_rpc::model::requests::InjectpaymentonionRequest`, which is required by `pb::InjectpaymentonionRequest: Into<_>`
|
= note: required for `pb::InjectpaymentonionRequest` to implement `Into<cln_rpc::model::requests::InjectpaymentonionRequest>`
error[E0277]: the trait bound `cln_rpc::model::requests::XpayRequest: From<pb::XpayRequest>` is not satisfied
--> cln-grpc/src/server.rs:4314:46
|
4314 | let req: requests::XpayRequest = req.into();
| ^^^^ the trait `From<pb::XpayRequest>` is not implemented for `cln_rpc::model::requests::XpayRequest`, which is required by `pb::XpayRequest: Into<_>`
|
= note: required for `pb::XpayRequest` to implement `Into<cln_rpc::model::requests::XpayRequest>`
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Testing autoconnect in --offline mode, the autoconnect never functions
if the seeker has not gotten out of the startup state. This commit
overloads the counter to start the autoconnect
attempt on the second round through the seeker_check.
Changelog-None
```
Program received signal SIGSEGV, Segmentation fault.
0x000000001014e9d8 in io_set_finish_ (conn=0x0, finish=0x0, arg=0x0) at ccan/ccan/io/io.c:137
137 conn->finish = finish;
(gdb) bt
incoming=true) at connectd/connectd.c:394
```
Fixes: #7871
Reported-by: grubles
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: broken in this release
A side-effect of having grpc start by default. Annoyingly, if it
can't bind it simply exits, with no message, so I had to guess what
was happening.
Reported-by: @daywalker90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- Run with environment variable `GENERATE_EXAMPLES`
- Update cln version in getinfo example on `make update-versions`
- Added two `dev` configs, dev-no-plugin-checksum and dev-no-version-checks, to match CI listconfigs
- Changed commando rpc example from `getinfo` to `newaddr` to avoid unneccessary file updates for future builds
- Stabilized `bkpr-editdescriptionbyoutpoint`, `listclosedchannels` and `listaddresses` examples
Large nodes were not always getting their own channel gossip out
reliably. The number of peers we spam our own channel gossip to
is limited to save large nodes on startup, but this should be
relaxed slightly to ensure propagation.
Changelog-Fixed: Own-channel gossip is broadcast to more peers on connect.
Using jsonrpc_request_sync, layers are loaded before we finish init,
so we never can be asked to create a layer before we've loaded it
(xpay creates a layer immediately on startup).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: persistent layers new this release.
Create lower-level versions of routines to create biases, layers,
constraints, etc and only save the ones called from the public APIs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: persistent layers were only added in this release
They're not always 34 (aka '"'). This is a side-effect of ids
changing from u64 to strings quite a while ago.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we now create a low-priority (2016 down to 12 blocks fee target) anchor for low-fee unilateral closes even if there's no urgency.
We're going to call this twice. But this patch also takes care
that a failed attempt to create an anchor doesn't alter other
variables!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. merge_deadlines can't really fail.
2. We don't care about incoming HTLCs with no preimage.
3. Debug print anchor points as soon as we calc them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Cleans up the API: we have two functions now, one which is explicitly for
"I'm failing this because I saw this tx onchain".
Now we can correctly report the tx which closed the channel (previously
we would always report our own tx(s)!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `close` now correctly reports the txid of the remote onchain unilateral tx if it races with a peer close.
Changelog-Fixed: Protocol: we no longer try to spend anchors if a commitment tx is already mined (reported by @niftynei).
Fixes: #7526
The seeker can send a full gossip query, which means the ping doesn't happen
(it needs 14-45 seconds of quiet!).
We disable the gossip_queries feature, so it doesn't ask.
```
def test_ping_timeout(node_factory):
# Disconnects after this, but doesn't know it.
l1_disconnects = ['xWIRE_PING']
l1, l2 = node_factory.get_nodes(2, opts=[{'dev-no-reconnect': None,
'disconnect': l1_disconnects},
{'dev-no-ping-timer': None}])
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
# This can take 10 seconds (dev-fast-gossip means timer fires every 5 seconds)
l1.daemon.wait_for_log('seeker: startup peer finished', timeout=15)
# Ping timers runs at 15-45 seconds, *but* only fires if also 60 seconds
# after previous traffic.
> l1.daemon.wait_for_log('dev_disconnect: xWIRE_PING', timeout=60 + 45 + 5)
tests/test_connection.py:4194:
...
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('dev_disconnect: xWIRE_PING')]" in logs.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We wait until a connection fails, or a subd is connected to the peer,
before letting another one through. This should prevent us from
overwhelming lightningd on large nodes, but unlike the previous back-off,
it's based on how fast lightningd is, not an arbitrary time.
We also let one through each second, in case we're connecting to many,
but not doing anything but gossip (e.g. 100 explicit connect
commands).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Reconnecting to peers at startup should be significantly faster (dependent on machine speed).
Rather than have lightningd call us repeatedly to try to connect, have
it tell us what peers are transient and aren't, and connectd will
automatically try to maintain that connection.
There's a new "downgrade_peer" message to tell it a peer is now
transient: to make it non-transient we simply tell connectd to
connect as a non-transient.
The first time, I missed that dual_open_control does its own state
transitions :(
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: `connectd` now handles maintaining/reconnecting to important peers, and we remember the last successful address we connected to.
This is more useful than the last address, which may be it connecting
to us. And use it when we restore it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we connected out, remember that address. We always remember the last
address, but that may be an incoming address. This is explicitly the last
outgoing address which worked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to use this to ask if there are any channels which make it
important to reconnect to the peer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In fact, only 951 of 17419 (5%) of node announcements are missing an address
(and gossipd doesn't know if we can connect to Tor addresses anyway) so
just check it *has* a node_announcement.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Let lightningd feed us hints to try first, but we can extract the
addresses from node_announcement messages ourselves.
(Lightningd used to ask gossipd on our behalf: this is far simpler!)
One side effect of this is that we don't hand back address hints given to us
by lightningd: it would use these again for reconnecting. This is breaks
test_sendpay_grouping, so we disable it temporarily.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It turns out that under some circumstances we end up clearing the
pointee of `current` but not the pointer. Thus when we select the next
slot we can end up reusing the same slot, making it its own parent.
We forcefull break these cycles by enforcing that `current` should
never be returned and be set as its own parent.
Changelog-None
Trace spans form a tree, but we don't actually check that the
structure doesn't break. Breakage can for example come if we use the
same key accidentally, making a new span its own ancestor.
We have the space in memory set aside anyway, so let's just copy the
`trace_id` into the span itself, rather than resolving the `root` at
time of emission.
This was a bit harder to identify: during an `io_loop` run we suspend
the current span before handing over to `io_loop`, and later when a callback
is called we resume the span again. Depending on how we return from
the `io_loop` instance that is used to drive the startup, we either
have resumed the last span, or we don't. Since we start a span before
`io_loop` and want it to be emitted afterwards, we need to take care
of the case where we returned from a callback that did not resume, and
therefore the current context is empty.
Making `trace_span_resume` idempotent means we can just resume it
manually.
Ideally we'd push the suspend / resume logic down into `io_loop`
itself, and then we'd have just one place. Maybe suspend and resume
callbacks that can be configured in `io_loop`?
After adding the DB query instrumentation we ran into a couple of
issues, with spans not being resumed correctly, and it was rather hard
to identify the problem. This adds debug statements so we can trace
the tracing (traception if you will).
Changelog-None
lightningd/test/run-find_my_abspath.c includes ../lightningd.c, which includes
header_versions_gen.h, a generated header file.
lightningd/Makefile correctly declares that lightningd/lightningd.o depends on
header_versions_gen.h, but lightningd/test/Makefile lacks any such declaration
regarding lightningd/test/run-find_my_abspath.c, which leads to build failure:
In file included from lightningd/test/run-find_my_abspath.c:5:
lightningd/test/../lightningd.c:64:10: fatal error: header_versions_gen.h: No such file or directory
64 | #include <header_versions_gen.h>
| ^~~~~~~~~~~~~~~~~~~~~~~
Declare the missing dependency in lightningd/test/Makefile so that Make will
ensure that header_versions_gen.h is generated before it attempts to build
lightningd/test/run-find_my_abspath.o.
Changelog-None
Due to Darwin-arm64 conditional setting of `CPPFLAGS`, the subsequent
`CPPFLAGS +=` is resolved earlier on ARM macOS which results in empty
paths being used.
Changelog-None
If the first one doesn't use the entire timeout, the second might need longer
(I used TIMEOUT=10 normally):
```
FAILED tests/test_gossip.py::test_gossip_pruning - TimeoutError: Unable to find "[re.compile('Pruning channel 103x1x0 from network view')]" in logs.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
pay does this, xpay does not. Which means if a block comes in (or you're behind),
you get gratuitous failures:
```
def test_xpay_simple(node_factory):
l1, l2, l3, l4 = node_factory.get_nodes(4, opts={'may_reconnect': True})
node_factory.join_nodes([l1, l2, l3], wait_for_announce=True)
node_factory.join_nodes([l3, l4], announce_channels=False)
# BOLT 11, direct peer
b11 = l2.rpc.invoice('10000msat', 'test_xpay_simple', 'test_xpay_simple bolt11')['bolt11']
> ret = l1.rpc.xpay(b11)
tests/test_xpay.py:148:
...
if not isinstance(resp, dict):
raise TypeError("Malformed response, response is not a dictionary %s." % resp)
elif "error" in resp:
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: xpay, payload: ('lnbcrt100n1pn5qu7csp53rp0mfwtfsyyy8gzsggepnxgslyalwvz3jkg9ptmqq452ln2nmgqpp58ak9nmfz9l93r0fpm266ewyjrhurhatrs05nda0r03p82cykp0vsdp9w3jhxazl0pcxz72lwd5k6urvv5sxymmvwscnzxqyjw5qcqp99qxpqysgqa798258yppu2tlfj8herr3zuz0zgux79zvtx6z57cmfzs2wdesmr4nvnkcmyssyu6k64ud54eg0v45c3mcw342jj6uy7tu202p6klrcp6ljc9w',), error: {'code': 203, 'message': "Destination said it doesn't know invoice: incorrect_or_unknown_payment_details"}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: xpay is new this release.
24.05 and before requires a "description" field. We should not have removed it here
until that was EOL!
Changelog-Fixed: pyln-client: plugins now compatible with CLN <= 24.05 (broken in 24.08)
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't want to to refresh the gossmap internally: this could invalidate the
gossmap held by the current callers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This does not validate a node announcement and address, but it
does select a node at random from the gossmap and asks lightningd
to attempt a connection to it.
Gossipd uses this to ask lightningd -> connectd to initiate
a connection to a new gossip peer. This can be used when
there are insufficient peers already connected to gossip with.
Changelog-Changed: Gossipd can now request connections to additional nodes for improved gossip sync
We can get more gossip_filter messages now. And we can also go over max-messages,
so increase that too.
```
del tally['query_short_channel_ids']
del tally['query_channel_range']
del tally['ping']
> assert tally == {'channel_announce': 1,
'channel_update': 3,
'node_announce': 1,
'gossip_filter': 1}
E AssertionError: assert {'channel_ann..._announce': 1} == {'channel_ann..._announce': 1}
E Omitting 2 identical items, use -vv to show
E Differing items:
E {'gossip_filter': 2} != {'gossip_filter': 1}
E {'channel_update': 2} != {'channel_update': 3}
E Full diff:
E {
E 'channel_announce': 1,...
E
E ...Full output truncated (10 lines hidden), use '-vv' to show
tests/test_gossip.py:2326: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We actually mine *300* blocks, not 200, and if timing is right l1
can have mined the txid before mine_txid_or_rbf() checks the mempool:
```
def test_onchaind_replay(node_factory, bitcoind):
disconnects = ['+WIRE_REVOKE_AND_ACK', 'permfail']
# Feerates identical so we don't get gratuitous commit to update them
l1, l2 = node_factory.line_graph(2, opts=[{'watchtime-blocks': 201, 'cltv-delta': 101,
'disconnect': disconnects,
'feerates': (7500, 7500, 7500, 7500)},
{'watchtime-blocks': 201, 'cltv-delta': 101}],
wait_for_announce=True)
inv = l2.rpc.invoice(10**8, 'onchaind_replay', 'desc')
rhash = inv['payment_hash']
routestep = {
'amount_msat': 10**8 - 1,
'id': l2.info['id'],
'delay': 101,
'channel': first_scid(l1, l2)
}
l1.rpc.sendpay([routestep], rhash, payment_secret=inv['payment_secret'])
l1.daemon.wait_for_log('sendrawtx exit 0')
bitcoind.generate_block(1, wait_for_mempool=1)
# Wait for nodes to notice the failure, this seach needle is after the
# DB commit so we're sure the tx entries in onchaindtxs have been added
l1.daemon.wait_for_log("Deleting channel .* due to the funding outpoint being spent")
l2.daemon.wait_for_log("Deleting channel .* due to the funding outpoint being spent")
# We should at least have the init tx now
assert len(l1.db_query("SELECT * FROM channeltxs;")) > 0
assert len(l2.db_query("SELECT * FROM channeltxs;")) > 0
# Generate some blocks so we restart the onchaind from DB (we rescan
# last_height - 100)
bitcoind.generate_block(100)
sync_blockheight(bitcoind, [l1, l2])
# l1 should still have a running onchaind
assert len(l1.db_query("SELECT * FROM channeltxs;")) > 0
l2.rpc.stop()
l1.restart()
# Can't wait for it, it's after the "Server started" wait in restart()
assert l1.daemon.is_in_log(r'Restarting onchaind \(ONCHAIN\): closed in block 109')
# l1 should still notice that the funding was spent and that we should react to it
_, txid, blocks = l1.wait_for_onchaind_tx('OUR_DELAYED_RETURN_TO_WALLET',
'OUR_UNILATERAL/DELAYED_OUTPUT_TO_US')
assert blocks == 200
bitcoind.generate_block(200)
# Could be RBF!
> l1.mine_txid_or_rbf(txid)
tests/test_closing.py:1864:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:1375: in mine_txid_or_rbf
wait_for(lambda: rbf_or_txid_broadcast(txids))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
success = <function LightningNode.mine_txid_or_rbf.<locals>.<lambda> at 0x7f9b129c4550>
timeout = 180
def wait_for(success, timeout=TIMEOUT):
start_time = time.time()
interval = 0.25
while not success():
time_left = start_time + timeout - time.time()
if time_left <= 0:
> raise ValueError("Timeout while waiting for {}".format(success))
E ValueError: Timeout while waiting for <function LightningNode.mine_txid_or_rbf.<locals>.<lambda> at 0x7f9b129c4550>
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The test program has a leak, so address sanitizer complains and makes it
"fail" the zlib detection test!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can fix the median calc by removing the (unused) reverse edges.
Also analyze the failure case in test_real_data: it's a real edge case, so
hardcode that one as "ok".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The ratio of the median of the fees and probability cost is overall not
a bad factor to combine these two features. This is what the
test_real_data shows.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
The fee_fallback test would fail after fixing the computation of the
median. Now by we can restore it by making the probability cost factor
1000x higher than the ratio of the median. This shows how hard it is to
combine fee and probability costs and why is the current approach so
fragile.
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Rusty: "We don't generally use NDEBUG in our code"
Instead use a compile time flag ASKRENE_UNITTEST to make checks on unit
tests that we don't normally need on release code.
Changelog-none
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
- use graph_max_num_arcs/nodes instead of tal_count in bound checks,
- don't use ccan/lqueue, use instead a minimalistic queue
implementation with an array,
- add missing const qualifiers to temporary tal allocators,
- check preconditions with assert,
- remove inline specifier for static functions,
Changelog-None
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
The calculation of the median values of probability and fee cost in the
linear approximation had a bug by counting on non-existing arcs.
Changelog-none: askrene: fix the median
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
We use an arc "array" in the graph structure, but not all arc indexes
correspond to real topological arcs. We must be careful when iterating
through all arcs, and check if they are enabled before making operations
on them.
Changelog-None: askrene: fix bug, not all arcs exists
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Add a new function to compute a MCF using a more general description of
the problem. I call it mcf_refinement because it can start with a
feasible flow (though this is not necessary) and adapt it to achieve
optimality.
Changelog-None: askrene: add a MCF refinement
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
It is just a copy-paste of "dijkstra" but the name
implies what it actually is. Not an implementation of minimum cost path
Dijkstra algorithm, but a helper data structure.
I keep the old "dijkstra.h/c" files for the moment to avoid breaking the
current code.
Changelog-EXPERIMENTAL: askrene: add priorityqueue
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
While we have (I hope!) fixed the underlying sync problem, this can still happen with
older gossip. So now we tell it the channel is dead, so it won't happen more than once.
Fixes: #7703
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This can help us backfill any missing gossip if our current
peers haven't been the most reliable.
Changelog-Changed: Gossipd requests a full sync from a random peer every hour.
Based on gossip sync data from random network peers, listening to only 5
peers will not reliably catch all gossip traffic. For now, add extra
redundancy.
Allows our peer to change their funding pub key during a splice.
Changelog-Changed: Support added for peers that wish to rotate their funding pubkey during a splice.
Set the remote funding pubkey on both lightningd and channeld when mutual splice lock is achieved.
This will be needed once rotating funding keys is enabled during splicing
Changelog-None.
As per eclair spec proposal.
1) A renaming to `funding_txid`
2) Adding of `batch_size` to indicate how many commitment_signed msgs are expected.
Changelog-None
In anticipation of adding support for rotating funding pubkeys during a splice, `channel_txs` is updated to support specifying these manually instead of using the channel’s funding pubkeys.
Changelog-None
Channeld stores its own cache of `inflight` and that needs to have a copy of `remote_funding` as well.
Since copying a secp256k1 pubkey isn’t documented and `copy_inflight` isn’t used anyway — we’re dropping `copy_inflight`.
Changelog-None
It is possible for prevtx to be larger than max packet size, so for shared outputs (currently only the funding tx) we add support for sending the `txid` only across the wire and filling in the prevtx locally.
Changelog-None
Enable storing the remote funding pubkey in DB if the channel peer decides to change it during splicing. It needs to be in DB incase of restarts mid-splice.
Changelog-None
Added and updated error messages when splicing to make it more clear to the user why a splice is failing.
Changelog-Changed: Improved error messaging for splice commands.
As we can see from the previous test, l3 tells us why it rejected the payment:
```
lightningd-3 2024-11-19T03:56:27.151Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Failing HTLC because of an invalid payload (TLV 10 pos 104): cltv_expiry 136 > payment_constraint 130
```
It turns out, we were not updating the block height in the plugin!
Without this, when we create a (non-dummy) blinded path we set a
too-low CLTV restriction, and it doesn't work after a few blocks!
Note we were actually triggering this error in the xpay tests!
Reported-by: Vincenzo Palazzo
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Offers: Receiving bolt12 payments where we have no public channels would fail a few blocks after startup.
Don't reply with update_fail_malformed_htlc, even though WIRE_INVALID_ONION_BLINDING
has BADONION set. Fail it with a normal error message.
This fixes a known FIXME.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: entry to blinded paths return more useful errors (e.g if it's the final node, you get a real error, otherwise you get invalid_onion_blinding).
Payer metadata is a field that controls the payer ID
provided during the fetchinvoice process.
There are use cases where this is highly useful, such as
proving that the payer has paid for the correct item.
Imagine visiting a merchant's website to pay for multiple offers, where
one of these offers is a default offer (with no description and no set amount).
In this scenario, the merchant could claim not to have received
payment for a specific item. Since the same offer may be used to
fetch invoices for different products, there needs to be a way to
identify which product the invoice corresponds to.
With this commit, it will be possible to inject payer metadata,
which helps solve the issue described above.
For example, possible payer metadata could be `to_hex(b"{payer_node_id}.{product_id}.{created_at}")`.
Changelog-Added: JSON-RPC: `fetchinvoice` allows setting invreq_metadata via `payer_metadata` parameter.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We actually pruned before we got all the channels. Extend the pruning time,
which unfortunately makes the test slower.
```
2024-11-18T02:13:11.7013278Z node_factory = <pyln.testing.utils.NodeFactory object at 0x7ff72969e820>
2024-11-18T02:13:11.7014386Z bitcoind = <pyln.testing.utils.BitcoinD object at 0x7ff72968fe20>
2024-11-18T02:13:11.7014996Z
2024-11-18T02:13:11.7015271Z def test_gossip_pruning(node_factory, bitcoind):
2024-11-18T02:13:11.7016222Z """ Create channel and see it being updated in time before pruning
2024-11-18T02:13:11.7017037Z """
2024-11-18T02:13:11.7017871Z l1, l2, l3 = node_factory.get_nodes(3, opts={'dev-fast-gossip-prune': None,
2024-11-18T02:13:11.7018971Z 'allow_bad_gossip': True})
2024-11-18T02:13:11.7019634Z
2024-11-18T02:13:11.7020236Z l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
2024-11-18T02:13:11.7021153Z l2.rpc.connect(l3.info['id'], 'localhost', l3.port)
2024-11-18T02:13:11.7021806Z
2024-11-18T02:13:11.7022226Z scid1, _ = l1.fundchannel(l2, 10**6)
2024-11-18T02:13:11.7022886Z scid2, _ = l2.fundchannel(l3, 10**6)
2024-11-18T02:13:11.7023458Z
2024-11-18T02:13:11.7023907Z mine_funding_to_announce(bitcoind, [l1, l2, l3])
2024-11-18T02:13:11.7025183Z l1_initial_cupdate_timestamp = only_one(l1.rpc.listchannels(source=l1.info['id'])['channels'])['last_update']
2024-11-18T02:13:11.7026179Z
2024-11-18T02:13:11.7027358Z # Get timestamps of initial updates, so we can ensure they change.
2024-11-18T02:13:11.7028171Z # Channels should be activated locally
2024-11-18T02:13:11.7029326Z > wait_for(lambda: [c['active'] for c in l1.rpc.listchannels()['channels']] == [True] * 4)
```
We can see in logs, it actually started pruning already:
```
2024-11-18T02:13:11.9622477Z lightningd-1 2024-11-18T01:52:03.570Z DEBUG gossipd: Pruning channel 105x1x0 from network view (ages 1731894723 and 0)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes l1 ratelimits before l2, and l2 receives the warning message, not l1:
```
> assert l1.daemon.is_in_log('WARNING: Ratelimited onion_message: exceeded one per 250msec')
E AssertionError: assert None
E + where None = <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7f13435f45b0>>('WARNING: Ratelimited onion_message: exceeded one per 250msec')
E + where <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7f13435f45b0>> = <pyln.testing.utils.LightningD object at 0x7f13435f45b0>.is_in_log
E + where <pyln.testing.utils.LightningD object at 0x7f13435f45b0> = <fixtures.LightningNode object at 0x7f13435cbb80>.daemon
...
lightningd-1 2024-11-19T00:45:43.721Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_in WIRE_ONION_MESSAGE
lightningd-1 2024-11-19T00:45:43.721Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_out WIRE_WARNING
lightningd-2 2024-11-19T00:45:43.722Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_out WIRE_ONION_MESSAGE
lightningd-2 2024-11-19T00:45:43.722Z DEBUG connectd: REPLY WIRE_CONNECTD_INJECT_ONIONMSG_REPLY with 0 fds
lightningd-2 2024-11-19T00:45:43.722Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_in WIRE_WARNING
lightningd-2 2024-11-19T00:45:43.722Z INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: Received WIRE_WARNING: WARNING: Ratelimited onion_message: exceeded one per 250msec
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I can't reproduce this, but CI did (with Elements):
```
[gw3] linux -- Python 3.8.18 /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.8/bin/python
node_factory = <pyln.testing.utils.NodeFactory object at 0x7fd0e20f57f0>
bitcoind = <pyln.testing.utils.ElementsD object at 0x7fd0e307dbe0>
executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x7fd0e307da30>
@pytest.mark.openchannel('v1')
@pytest.mark.openchannel('v2')
def test_lightningd_still_loading(node_factory, bitcoind, executor):
"""Test that we recognize we haven't got all blocks from bitcoind"""
mock_release = Event()
# This is slow enough that we're going to notice.
def mock_getblock(r):
conf_file = os.path.join(bitcoind.bitcoin_dir, 'bitcoin.conf')
brpc = RawProxy(btc_conf_file=conf_file)
if r['params'][0] == slow_blockid:
mock_release.wait(TIMEOUT)
return {
"result": brpc._call(r['method'], *r['params']),
"error": None,
"id": r['id']
}
# Start it, establish channel, get extra funds.
l1, l2, l3 = node_factory.get_nodes(3, opts=[{'may_reconnect': True,
'wait_for_bitcoind_sync': False},
{'may_reconnect': True,
'wait_for_bitcoind_sync': False},
{}])
node_factory.join_nodes([l1, l2])
# Balance l1<->l2 channel
l1.pay(l2, 10**9 // 2)
l1.stop()
# Now make sure l2 is behind.
bitcoind.generate_block(2)
# Make sure l2/l3 are synced
sync_blockheight(bitcoind, [l2, l3])
# Make it slow grabbing the final block.
slow_blockid = bitcoind.rpc.getblockhash(bitcoind.rpc.getblockcount())
l1.daemon.rpcproxy.mock_rpc('getblock', mock_getblock)
l1.start(wait_for_bitcoind_sync=False)
# It will warn about being out-of-sync.
assert 'warning_bitcoind_sync' not in l1.rpc.getinfo()
assert 'warning_lightningd_sync' in l1.rpc.getinfo()
# Make sure it's connected to l2 (otherwise we get TEMPORARY_CHANNEL_FAILURE)
wait_for(lambda: only_one(l1.rpc.listpeers(l2.info['id'])['peers'])['connected'])
# Payments will succced.
l1.pay(l2, 1000)
> assert l1.daemon.is_in_log(r"Sending HTLC while still syncing with bitcoin network \(104 vs 105\)")
E AssertionError: assert None
E + where None = <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0>>('Sending HTLC while still syncing with bitcoin network \\(104 vs 105\\)')
E + where <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0>> = <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0>.is_in_log
E + where <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0> = <fixtures.LightningNode object at 0x7fd0e20f59d0>.daemon
```
What was in logs was:
```
lightningd-1 2024-11-18T05:33:50.634Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Sending HTLC while still syncing with bitcoin network (103 vs 105)
```
Implying that l1 was an extra block behind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to wait for *l2* to see the channel in CHANNELD_NORMAL,
otherwise the array here is empty:
```
chan = only_one([c for c in l1.rpc.listpeerchannels(l2.info['id'])['channels'] if c['state'] == 'CHANNELD_NORMAL'])
amount = chan['funding']['local_funds_msat']
assert amount > Millisatoshi(str((1 << 24) - 1) + "sat")
# We should know we can spend that much!
spendable = chan['spendable_msat']
assert spendable > Millisatoshi(str((1 << 24) - 1) + "sat")
# So should peer.
> chan = only_one([c for c in l2.rpc.listpeerchannels(l1.info['id'])['channels'] if c['state'] == 'CHANNELD_NORMAL'])
tests/test_connection.py:3552:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
arr = []
def only_one(arr):
"""Many JSON RPC calls return an array; often we only expect a single entry
"""
> assert len(arr) == 1
E AssertionError
```
The old `long_description` was removed and deprecated a while ago
without adding a proper replacement for plugin developers.
The getmanifest JSON that was to be used for that only knows `name` and `usage`.
This PR adds an optional `description` parameter that will be filled
with the methods docstring `__doc__` (if set).
Example:
@p.method("example")
def some_method(...)
"""some description"""
...
Changelog-Add: optional description paramter to Plugin.Method
Commit 531845971c broke the build without
SQLite because this code:
new = tal_fmt(NULL, template,
IF_SQLITE3(sqlite3_libversion_number()));
preprocesses into:
new = tal_fmt(NULL, template,
);
which has a syntax error. Fix it by moving the comma into the macro
argument.
Fixes: 531845971c
Changelog-None
This is required for VLS which wants to know (and potentially decline) invoices
we're trying to pay.
As a nice side effect, our "check" command for xpay now does much more thorough
checking of arguments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As the first user of a persistent layer, this tripped tests which
assumed the datastore would be empty!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are automatically marked "important", in the sense that we won't startup
if they are not working, but this wasn't meant to disallow stopping them.
Changelog-Changed: JSON-RPC: built-in plugins can now be stopped using "plugin stop".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note: won't work with grpc (or probably other tools), since the output
is different. But good for testing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: option `xpay-handle-pay` can be used to call xpay when pay is used in many cases (but output is different from pay!)
Because we initalized plugin->io_rpc_conn *after* calling plugin->init,
send_outreq would do a (harmless, in our case) wakeup on an uninitialized address:
```
==1164079== Conditional jump or move depends on uninitialised value(s)
==1164079== at 0x1628FC: backend_wake (poll.c:227)
==1164079== by 0x160B98: io_wake (io.c:384)
==1164079== by 0x1160A8: ld_rpc_send (libplugin.c:255)
==1164079== by 0x1187E0: send_outreq (libplugin.c:1099)
==1164079== by 0x115041: init (xpay.c:1620)
```
Solution is simple: set plugin->io_rpc_conn to NULL, and don't wake it in this case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
fail->msg can be NULL for local failures (the error message itself is more informative
in this case). Use the generic "something went wrong" message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Both for HTLC txs and the to-self outputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Wallet: Taproot addresses are used for unilateral-close change addresses.
It only works on BOLT11, and has long been replaced by the more
generic "decode".
Removing it will stop the confusion!
(Note: documentation claims it was introduced in 23.08, but that was
wrong, as it's been in CLN since the beginning).
[ Fixup from: niftynei <niftynei@gmail.com> ]
Fixes: https://github.com/ElementsProject/lightning/issues/6419
Changelog-Deprecated: JSON-RPC: `decodepay`: use `decode`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fails when l3 doesn't know address for l1, to connect to it:
```
2024-11-16T04:45:42.2243366Z lightningd-3 2024-11-16T04:35:10.582Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_in WIRE_ONION_MESSAGE
2024-11-16T04:45:42.2244342Z lightningd-3 2024-11-16T04:35:10.582Z DEBUG lightningd: Got onionmsg reply_path
2024-11-16T04:45:42.2245398Z lightningd-3 2024-11-16T04:35:10.582Z DEBUG plugin-offers: Note: disallowing deprecated onion_message_recv.blinding
2024-11-16T04:45:42.2246408Z lightningd-3 2024-11-16T04:35:10.586Z UNUSUAL plugin-offers: No incoming channel for 5msat, so no blinded path
2024-11-16T04:45:42.2247289Z lightningd-3 2024-11-16T04:35:10.605Z DEBUG hsmd: Client: Received message 25 from client
2024-11-16T04:45:42.2248372Z lightningd-3 2024-11-16T04:35:10.606Z DEBUG plugin-offers: connecting directly to 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518
2024-11-16T04:45:42.2249451Z lightningd-3 2024-11-16T04:35:10.606Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds
2024-11-16T04:45:42.2250743Z lightningd-3 2024-11-16T04:35:10.607Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: Failed connected out: Unable to connect, no address known for peer
```
This is because the test which was supposed to wait for addresses is
wrong: it passes when l3 knows nothing! (`all([])` == `True`)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Couldn't figure out why hsmtool.proc.wait(WAIT_TIMEOUT) returns 1?
hsmtool doesn't ever seem to exit status 1!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Looking at the logs (and comparing a successful run), it seems the connect happens before
the connect_stream is ready, so we miss it:
```
________________________ test_grpc_connect_notification ________________________
[gw7] linux -- Python 3.8.18 /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.8/bin/python
node_factory = <pyln.testing.utils.NodeFactory object at 0x7fb08bb969d0>
def test_grpc_connect_notification(node_factory):
l1, l2 = node_factory.get_nodes(2)
# Test the connect notification
connect_stream = l1.grpc.SubscribeConnect(clnpb.StreamConnectRequest())
l2.connect(l1)
> for connect_event in connect_stream:
tests/test_cln_rs.py:425:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.8/lib/python3.8/site-packages/grpc/_channel.py:543: in __next__
return self._next()
../../../.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.8/lib/python3.8/site-packages/grpc/_channel.py:960: in _next
_common.wait(self._state.condition.wait, _response_ready)
../../../.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.8/lib/python3.8/site-packages/grpc/_common.py:156: in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
../../../.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.8/lib/python3.8/site-packages/grpc/_common.py:116: in _wait_once
wait_fn(timeout=timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x7fb089730f00>, 0)>
timeout = 0.1
def wait(self, timeout=None):
"""Wait until notified or until a timeout occurs.
If the calling thread has not acquired the lock when this method is
called, a RuntimeError is raised.
This method releases the underlying lock, and then blocks until it is
awakened by a notify() or notify_all() call for the same condition
variable in another thread, or until the optional timeout occurs. Once
awakened or timed out, it re-acquires the lock and returns.
When the timeout argument is present and not None, it should be a
floating point number specifying a timeout for the operation in seconds
(or fractions thereof).
When the underlying lock is an RLock, it is not released using its
release() method, since this may not actually unlock the lock when it
was acquired multiple times recursively. Instead, an internal interface
of the RLock class is used, which really unlocks it even when it has
been recursively acquired several times. Another internal interface is
then used to restore the recursion level when the lock is reacquired.
"""
if not self._is_owned():
raise RuntimeError("cannot wait on un-acquired lock")
waiter = _allocate_lock()
waiter.acquire()
self._waiters.append(waiter)
saved_state = self._release_save()
gotit = False
try: # restore state no matter what (e.g., KeyboardInterrupt)
if timeout is None:
waiter.acquire()
gotit = True
else:
if timeout > 0:
> gotit = waiter.acquire(True, timeout)
E Failed: Timeout >1200.0s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since wait_for_onchaind_tx doesn't actually wait for the call to bitcoind to
return, we have a race in checking if the txid is in the mempool. Fix this
by making wait_for_onchaind_tx actually wait for the response (except for delayed txs!).
```
2024-11-15T07:15:22.0836959Z def test_htlc_in_timeout(node_factory, bitcoind, executor):
2024-11-15T07:15:22.0837722Z """Test that we drop onchain if the peer doesn't accept fulfilled HTLC"""
2024-11-15T07:15:22.0838208Z
2024-11-15T07:15:22.0838585Z # HTLC 1->2, 1 fails after 2 has sent committed the fulfill
2024-11-15T07:15:22.0839137Z disconnects = ['-WIRE_REVOKE_AND_ACK*2']
2024-11-15T07:15:22.0839741Z # Feerates identical so we don't get gratuitous commit to update them
2024-11-15T07:15:22.0840304Z l1 = node_factory.get_node(disconnect=disconnects,
2024-11-15T07:15:22.0840839Z options={'dev-no-reconnect': None},
2024-11-15T07:15:22.0841285Z feerates=(7500, 7500, 7500, 7500))
2024-11-15T07:15:22.0841673Z l2 = node_factory.get_node()
2024-11-15T07:15:22.0842278Z # Give it some sats for anchor spend!
2024-11-15T07:15:22.0842679Z l2.fundwallet(25000, mine_block=False)
2024-11-15T07:15:22.0843013Z
2024-11-15T07:15:22.0843342Z l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
2024-11-15T07:15:22.0843753Z chanid, _ = l1.fundchannel(l2, 10**6)
2024-11-15T07:15:22.0844058Z
2024-11-15T07:15:22.0844291Z sync_blockheight(bitcoind, [l1, l2])
2024-11-15T07:15:22.0844606Z
2024-11-15T07:15:22.0844958Z amt = 200000000
2024-11-15T07:15:22.0845713Z inv = l2.rpc.invoice(amt, 'test_htlc_in_timeout', 'desc')['bolt11']
2024-11-15T07:15:22.0846612Z assert only_one(l2.rpc.listinvoices('test_htlc_in_timeout')['invoices'])['status'] == 'unpaid'
2024-11-15T07:15:22.0847141Z
2024-11-15T07:15:22.0847430Z executor.submit(l1.dev_pay, inv, dev_use_shadow=False)
2024-11-15T07:15:22.0847805Z
2024-11-15T07:15:22.0848041Z # l1 will disconnect and not reconnect.
2024-11-15T07:15:22.0848660Z l1.daemon.wait_for_log('dev_disconnect: -WIRE_REVOKE_AND_ACK')
2024-11-15T07:15:22.0850393Z
2024-11-15T07:15:22.0851297Z # Deadline HTLC expiry minus 1/2 cltv-expiry delta (rounded up) (== cltv - 3). cltv is 5+1.
2024-11-15T07:15:22.0852146Z # shadow route can add extra blocks!
2024-11-15T07:15:22.0852622Z status = only_one(l1.rpc.call('paystatus')['pay'])
2024-11-15T07:15:22.0853044Z if 'shadow' in status:
2024-11-15T07:15:22.0853861Z shadowlen = 6 * status['shadow'].count('Added 6 cltv delay for shadow')
2024-11-15T07:15:22.0854325Z else:
2024-11-15T07:15:22.0854547Z shadowlen = 0
2024-11-15T07:15:22.0854845Z bitcoind.generate_block(2 + shadowlen)
2024-11-15T07:15:22.0855292Z assert not l2.daemon.is_in_log('hit deadline')
2024-11-15T07:15:22.0855669Z bitcoind.generate_block(1)
2024-11-15T07:15:22.0855950Z
2024-11-15T07:15:22.0856406Z l2.daemon.wait_for_log('Fulfilled HTLC 0 SENT_REMOVE_COMMIT cltv .* hit deadline')
2024-11-15T07:15:22.0856997Z l2.daemon.wait_for_log('sendrawtx exit 0')
2024-11-15T07:15:22.0857360Z l2.bitcoin.generate_block(1)
2024-11-15T07:15:22.0857741Z l2.daemon.wait_for_log(' to ONCHAIN')
2024-11-15T07:15:22.0858137Z l1.daemon.wait_for_log(' to ONCHAIN')
2024-11-15T07:15:22.0858644Z
2024-11-15T07:15:22.0859068Z # L2 will collect HTLC (iff no shadow route)
2024-11-15T07:15:22.0859741Z _, txid, blocks = l2.wait_for_onchaind_tx('OUR_HTLC_SUCCESS_TX',
2024-11-15T07:15:22.0860287Z 'OUR_UNILATERAL/THEIR_HTLC')
2024-11-15T07:15:22.0860662Z assert blocks == 0
2024-11-15T07:15:22.0860908Z
2024-11-15T07:15:22.0861262Z # If we try to reuse the same output as we used for the anchor spend, then
2024-11-15T07:15:22.0861951Z # bitcoind can reject it. In that case we'll try again after we get change
2024-11-15T07:15:22.0862433Z # from anchor spend.
2024-11-15T07:15:22.0862768Z if txid not in bitcoind.rpc.getrawmempool():
2024-11-15T07:15:22.0863354Z bitcoind.generate_block(1)
2024-11-15T07:15:22.0863735Z > bitcoind.generate_block(1, wait_for_mempool=1)
2024-11-15T07:15:22.0864019Z
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Because it wasn't in ALL_OBJS. Copy the Makefile pattern!
```
Submodule 'src/secp256k1' (https://github.com/ElementsProject/secp256k1-zkp.git) registered for path 'external/libwally-core/src/secp256k1'
Cloning into '/home/runner/work/lightning/lightning/external/libwally-core/src/secp256k1'...
cc tests/plugins/channeld_fakenet.c
In file included from ./bitcoin/script.h:4,
from tests/plugins/channeld_fakenet.c:14:
./bitcoin/signature.h:6:10: fatal error: secp256k1.h: No such file or directory
6 | #include <secp256k1.h>
| ^~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:301: tests/plugins/channeld_fakenet.o] Error 1
make: *** Waiting for unfinished jobs....
Submodule path 'external/libwally-core/src/secp256k1': checked out
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's a known issue with hsm passwords and valgrind:
```
write_all(master_fd, (password + '\n').encode("utf-8"))
> l1.daemon.wait_for_log("Server started with public key")
tests/test_plugin.py:4526:
...
if self.is_in_log(r):
print("({} was previously in logs!)".format(r))
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('Server started with public key')]" in logs.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Based on the patch by bstin <barry.github@capsmx.com>, which added a separate command,
this simply extends "generatehsm" to allow more options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: hsmtool: generatehsm can run non-interactive, taking options on the cmdline.
The `send_outreq` function is a good place to suspend and resume
traces, since these are usually the places where we hand off control
back to the `io_loop`. This assumes that we do not continue doing
heavy liftin after we have queued an `outreq` call, but that is most
likely the case anyway. This frees us from having to track suspensions
whenever we call the RPC from a plugin.
Christian noted that if we don't do this we could flood onchaind with messages:
particularly in Greenlight where the HSM (remote) may delay indefinitely, so
onchaind doesn't process messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means it always tells us explicitly whether to keep watching or not,
and we know it's processed it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This may help the cases we see where gossipd doesn't realize channels
are closed (because of shutdown before it processed the closing).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `gossipd` will no longer miss some channel closes on restart.
And we hook in the replay watch code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `onchaind` can miss conclusion of final txs in some cases, will now replay independently.
We start by telling onchaind about the funding spend, and anything
which spends it, and it tells us the txids it *doesn't* want to watch
any more. We're going to use a separate set of watches for the replay
case: this implements that code.
Once we're caught up, we convert any remaining watches to normal ones
to follow future blocks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And use it for `exposesecret-passphrase`. This is probably overly
cautious, but it makes me feel a little better that we won't leak it
to someone with read-only access.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Being able to back up the hsm_secret is critical, but you cannot do
this through a UI, because of course we do not allow such access.
People have lost funds because they didn't back up.
This allows access to the hsm_secret if you use a password set in the
config file. (If it's not set, the command does not work). This is a
compromise, of course.
Changelog-Added: `exposesecret` command for encouraging hsm_secret backups.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. It's called listpays not listpay.
2. "index" does NOT have a default value (it must be specified if limit or start are used)
3. Note that limit and start have effects on accuracy, since we combine records.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't consume *all* the CPU.
Changelog-Fixed: Plugins: `autoclean` is now gentler on the node when doing giant cleans.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
On a large node, especially with postgres, this causes every other command
to take 30 seconds plus. The first, obvious, step is to reduce how many
commands we will do at once.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Given an {outpoint}, sets the description on the matching outpoint (if exists).
Note that if no outpoint exists in bookkeeper, will return an empty list
Changleog-Added: PLUGINS: bookkeeper has a new RPC `bkrp-editdescriptionbyoutpoint` which will set/update a description for an outpoint creation event.
This takes an {payment_id} and {description}.
It looks for all chain + channel events that match
that {payment_id} and updates the description for those events.
We return all the updated events. If no events are updated, an empty
list is returned.
Changelog-Added: PLUGINS: bookkeeper has a new RPC `bkpr-editdescriptionbypaymentid` which will update the description for any event with matching payment_id
The fee maximum is used to issue a warning to the user their feerate is high in case they accidentally make a large donation to miners.
During python testing the feerates are high on purpose so we raise the warning level to at least the penality feerate.
The command called “splice” can take a json payload or a ‘splice script’, process it into a list of ‘actions’ and then execute those actions.
These actions include or will include everything you would want to do with a splice:
* Splice into a channel
* Splice out of a channel
* Fund from wallet
* Deposit to wallet
* Send funds to bitcoin address
Changelog-Added: A new magic “splice” command is added that can take a ‘splice script’ or json payload and perform any complex splice across multiple channels merging the result into a single transaction. Some features are disabled and will be added in time.
When set this flag tells addpsbtoutput to add the intiator serial_id to the added output.
Changelog-Changed: addpsbtoutput now allows serial_id to be set while adding which is needed for splicing and dual.
This is the sister command of addpsbtoutput.
Adds inputs equal to or greater than the amount requests, reservers them, and reports important information back out to the user.
Changelog-Added: New low-level RPC command addpsbtinput to fund PSBTs directly and help with complex splices & dual-opens.
The ability to stfu channels in bulk is required to do complex multi channel operations. When stfu’ing in this manner, the available funds at the moment of stfu is returned to the user.
In order to cancel the stfu we also add a bulk tx_abort command.
Changelog-Added: `stfu_channels` and `abort_channels` are added for bulk multi-channel splice commands. These allow the user to pause (and resume) multiple channels in place.
On certain well timed restarts we lose their siganture from memory and don’t receive it from them. In these cases we can extract it from the PSBT directly.
This is needed to all multi-channel splices. When channeld can return the signatures to the user (based on signing order precedent), it now does from splice_update.
Additionally, we move sending of the initial psbt from splice_init down to splice_update. This is also necessary for correct psbt diff detection during multi-channel splices.
Changelog-Changed: splice_update can in some cases now return the remotely partiall signed psbt to the user, if so `signtures_secured` will be true.
`splice_signed` now searchs the PSBT for channel ids
Changelog-Changed: `splice_signed` parameters are switched in order to make `channel_id` an optional parameter, enabling multi-splice-signatures.
A routine for getting a signature back out of an input’s list of pending signatures via pubkey search.
This is needed for certain kinds of restarts as we lose our peer’s signature from memory but a copy is kept in the PSBT.
New compiler for splice scripts that parses splice scripts, validates them, converts them to json and back again.
Changelog-Added: Splice script parser — takes a custom splice query language to bundle multiple complex splices into a single task in a simple way.
This does not mean it won't change, just that it will be backwards compatible.
Changelog-Added: Plugins: `askrene` which provides `getroutes` and a complete API for adding information in layers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This lets you place annotated biases on channels, to influence routing.
Uses include avoiding TOR nodes, slow channels or other local preferences.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: askrene is new anyway.
On `dev-memleak`, if someone is using rpc_command_hook, we'll call
it when the hook returns. But it will see these contexts as a leak.
So attach them to tmpctx (which is excluded from leak detection).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without knowing what method was called, we can't have useful general logging
methods, so go through the pain of adding "const char *method" everywhere,
and add:
1. ignore_and_complete - we're done when jsonrpc returned
2. log_broken_and_complete - we're done, but emit BROKEN log.
3. plugin_broken_cb - if this happens, fail the plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we used to allow cmd to be NULL, we had to hand the plugin
everywhere. We no longer do.
1. Various jsonrpc_ functions no longer need the plugin arg.
2. send_outreq no longer needs a plugin arg.
3. The init function takes a command, not a plugin.
4. Remove command_deprecated_in_nocmd_ok.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were dereferencing the first character of the id, (always '"') which meant
everything was id 34.
Before:
plugin-pay: cmd 34 partid 5
After:
cmd pytest:pay#62/cln:pay#105 partid 0
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `pay`: debug logging now uses correct JSON ids.
This means we replace p->cmd with an auxillary command after we've
finished, so we have a valid command to use.
It also means we weave `struct command_result` returns back through
all the callers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This does not code changes, but makes the next changes easier.
We short-cut the "we are a child" case and de-indent the main
cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is cleaner: everything can now be associated with a command
context.
You're supposed to eventually dispose of it using timer_complete().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes we want to clean up *after* a command has completed, but
we're moving to a model where all libplugin operations require a
`struct command`. This adds `aux_command` to create an
independent-lifetime command with the same id.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I started getting "WIRE_TEMPORARY_CHANNEL_FAILURE: Too many HTLCs" after
two hundred xpay attempts.
This was nice (it found some bugs in injectpaymentonion's handling of
local errors, and in xpay's reporting), but shouldn't happen.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Start with a random capacity (linear prob), and remember in-progess
payments so we can simulate them using capacity properly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note the impedence mismatch between sendpay and getroutes: we have to shift
amounts and delays by 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our gossmap_store uncompresser generates nodeids with well-known
privkeys, so we can decrypt and respond to HTLCs sent to such nodes.
By replacing channeld with a fake, we can connect a node to another
node, but then once the channel is established, allow payments to be
sent into the generated network, and respond appropriately.
This minimal version handles MPP timeouts, but doesn't insert any
delays or runtime capacity for channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: Testing only
secp256k1_ctx is used by pubkey_from_node_id. Don't try to pick and
choose where to initialize secp256k1_ctx, just always do it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you change branches and have a generated .md file, index.rst
will pick it up. Use the Makefile variable, not the contents of
the filesystem!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This remove an unnecessary check for existing description field if the
description_hash is provided in the invoice. The bolt11_decode function
already checks the description against the hash if both are provided.
Changelog-Fix: renepay: allow to pay BOLT11 invoices with description_hash, the description field is made optional
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Now pay learns, it sometimes learns not to try again:
```
> assert(len(l1.rpc.listpays()['pays']) == 2)
E AssertionError: assert 1 == 2
E + where 1 = len([{'amount_sent_msat': 0, 'bolt11': 'lnbcrt1pnjj7mysp5tfx8n6nyx7ehszgqn7gqm2r6n079p22u2yddtg797ka3pa9557tspp5f89z6genjqrl3knymvav9ajwcxrm5w7arxux06rrhjux88derjyqdq8v3jhxccxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqsqqqqqqqqpqqqqqzsqqc9qxpqysgqcuyr7qlyctf9w96fqg4wetqt7t5v938dagmv0r777n902utjufujzjxl3289r97yngft966zly3ehxfp469dh3lq0hkv6r684snvunqppuyvsl', 'created_at': 1730771812, 'destination': '035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d', ...}])
tests/test_pay.py:5147: AssertionError
```
We fix this by creating a fresh channel, so it will try.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can see peer_in before the state changes:
```
l3.daemon.wait_for_log("peer_in WIRE_ERROR")
> assert only_one(l3.rpc.listpeerchannels(l2.info['id'])['channels'])['state'] == 'AWAITING_UNILATERAL'
E AssertionError: assert 'CHANNELD_NORMAL' == 'AWAITING_UNILATERAL'
E - AWAITING_UNILATERAL
E + CHANNELD_NORMAL
```
From the logs, there is 0.2 seconds between them:
```
lightningd-3 2024-11-05T01:58:41.695Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_in WIRE_ERROR
lightningd-3 2024-11-05T01:58:41.726Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: billboard perm: Received ERROR channel cecf36a62a09d4f1bdb42aa61d0770964bf3b245b8943a3e5b86dafc572f63d1: Forcibly closed by `close` command timeout
lightningd-3 2024-11-05T01:58:41.745Z UNUSUAL 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Peer permanent failure in CHANNELD_NORMAL: channeld: received ERROR channel cecf36a62a09d4f1bdb42aa61d0770964bf3b245b8943a3e5b86dafc572f63d1: Forcibly closed by `close` command timeout (reason=protocol)
lightningd-3 2024-11-05T01:58:41.890Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: We have 3 anchor points to use
lightningd-3 2024-11-05T01:58:41.897Z DEBUG lightningd: Broadcasting txid ff2f44b37d96a81fcaa2a4b11746a06be70f3f800fbce941e61a47abb61f70c0
lightningd-3 2024-11-05T01:58:41.901Z DEBUG lightningd: sendrawtransaction: 02000000000101cecf36a62a09d4f1bdb42aa61d0770964bf3b245b8943a3e5b86dafc572f63d1000000000096b64c80044a01000000000000220020525df7a97bd0506c9ec41ee4e5f095e6e5316db01846a0a687404628017e88494a010000000000002200206db2ec9041ba3ccb6309dcad26015f32637e6869be09d3c3d3a1cb1439296f0b400d030000000000220020c2468acf761754e2533fcb13235326d1f1173b697b132048d6409be7818ed9a96a1f0c0000000000220020b1b561b95c1bccd50fb21bd417a95cabbc3efc351b1353063fe5e9f185d21c8a0400473044022036ab981a2642527c2019a4d15847f6b2fb1d0b92f1a2faff463ab109ed70c4d002205eeb0fcd610a9dba477ee7d66f02d0384080383fbb003d4ee423bdae0970b0430147304402202f7de3481ce00478e539cd9d6bac95ec3c80ec906c46ad2561118dca321b1458022048efd1aa04ff5fafbb99d14f7c9671072b035cbd2c031547793eb0cb6a62302d0147522102d595ae92b3544c3250fb772f214ad8d4c51425033740a5bcc357190add6d7e7a2102d6063d022691b2490ab454dee73a57c6ff5d308352b461ece69f3c284f2c241252ae6e14d920
lightningd-3 2024-11-05T01:58:41.907Z INFO 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The original complaint which caused my investigation was the 100% CPU
consumption of connectd, which we traced to the queue to gossipd.
However, the issue is not really connectd's overproduction, but
gossipd's underconsumption, probably caused by its own queueing issues
with the trace messages to lightningd, which the prior patch fixed.
Nonetheless, gossipd *can* get busy, and if we were to ask multiple
nodes for full gossip, we could see a few hundred thousand messages
come it at once. Hence I'm increasing the warning limit to 250,000
messages.
This commit is also where we attach the Changelog message, even
though it's really "common/msg_queue: use membuf for greater efficiency."
and "gossipd: fix excessive msg_queue length from status_trace()" which
solved the problem.
Here's the backtrace from a previous debug patch:
```
lightning_connectd: msg_queue length excessive (version v24.08.1-17-ga780ad4-modded)
0x5580534051f0 send_backtrace
common/daemon.c:33
0x55805340bd5b do_enqueue
common/msg_queue.c:66
0x55805340bde5 msg_enqueue
common/msg_queue.c:82
0x5580534057ce daemon_conn_send
common/daemon_conn.c:161
0x5580533fe3ff handle_gossip_in
connectd/multiplex.c:624
0x5580533ff23b handle_message_locally
connectd/multiplex.c:763
0x5580533ff2d6 read_body_from_peer_done
connectd/multiplex.c:1112
```
Reported-by: https://github.com/JssDWt
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `connectd` and `gossipd` message queues are much more efficient.
When this (very spammy) "handle_recv_gossip" message was changed
from debug to trace, the suppression code wasn't updated: we suppress
overly active debug messages, but not trace messages.
This is the backtrace from an earlier version of the "too large queue"
patch:
```
lightning_gossipd: msg_queue length excessive (version v24.08.1-17-ga780ad4-modded)
0x557e521e833f send_backtrace
common/daemon.c:33
0x557e521eefb9 do_enqueue
common/msg_queue.c:66
0x557e521ef043 msg_enqueue
common/msg_queue.c:82
0x557e521e891d daemon_conn_send
common/daemon_conn.c:161
0x557e521f14f0 status_send
common/status.c:90
0x557e521f1804 status_vfmt
common/status.c:169
0x557e521f1433 status_fmt
common/status.c:180
0x557e521de7c6 handle_recv_gossip
gossipd/gossipd.c:206
0x557e521de9f5 connectd_req
gossipd/gossipd.c:307
0x557e521e862d handle_read
common/daemon_conn.c:35
```
Without this, we have hardly any enforcement. This is why the schema
mistake fixed in the previous patches weren't spotted immediately.
The hard work was done by:
```
$ for f in lightning-*.json; do grep -v '^ "additionalProperties": false,' $f | bagto $f; done
$ for f in lightning-*.json; do sed 's/"properties": {/"additionalProperties": false, "properties": {/' $f | bagto $f; done
$ make fmt-schemas
```
Then checking where 'additionalProperties: true' had been turned to
false (we deliberately use it in some places where there are if
statements in the schema, or occasionally where there can be arbitrary
fields).
[Including doc/rpc-schema-draft.json update by Shahana]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows the next patch (which makes the schemas stricter) to not
break our tests.
We add some missing fields (including dev fields, but they're empty and hidden),
and add a few minor clarifications and a spelling fix. Most of these are new
schemas for this release, so no mention in Changelog.
Here is the difference in the man pages:
--- doc/lightning-askrene-inform-channel.7.md.old 2024-10-29 17:33:07.714521584 +1030
+++ doc/lightning-askrene-inform-channel.7.md 2024-10-29 17:42:37.434280109 +1030
@@ -24,6 +24,8 @@
- **short\_channel\_id\_dir** (short\_channel\_id\_dir): The short channel id and direction
+- **layer** (string): The name of the layer to apply this change to.
+- **timestamp** (u64): The UNIX timestamp when this constraint was created.
- **maximum\_msat** (msat, optional): The maximum value which this channel could pass.
--- doc/lightning-askrene-listlayers.7.md.old 2024-10-29 17:33:07.716521571 +1030
+++ doc/lightning-askrene-listlayers.7.md 2024-10-29 17:42:37.424280316 +1030
@@ -29,13 +29,16 @@
- **channel\_updates** (array of objects):
+ - **short\_channel\_id\_dir** (short\_channel\_id\_dir): The short channel id and direction this update applies to.
+ - **enabled** (boolean, optional): True if this can be used, false otherwise.
- **htlc\_minimum\_msat** (msat, optional): The minimum value allowed in this direction.
- **htlc\_maximum\_msat** (msat, optional): The maximum value allowed in this direction.
- **fee\_base\_msat** (msat, optional): The base fee to apply to use the channel in this direction.
- **fee\_proportional\_millionths** (u32, optional): The proportional fee (in parts per million) to apply to use the channel in this direction.
- - **delay** (u16, optional): The CLTV delay required for this direction.
+ - **cltv\_expiry\_delta** (u16, optional): The CLTV delay required for this direction.
- **constraints** (array of objects):
- **short\_channel\_id\_dir** (short\_channel\_id\_dir): The short channel id and direction
+ - **timestamp** (u64, optional): The UNIX timestamp when this constraint was created.
- **maximum\_msat** (msat, optional): The maximum value which this channel could pass.
--- doc/lightning-askrene-listreservations.7.md.old 2024-10-29 17:33:07.719521550 +1030
+++ doc/lightning-askrene-listreservations.7.md 2024-10-29 17:42:37.428280233 +1030
@@ -16,7 +16,7 @@
-On success, an object containing **layers** is returned. It is an array of objects, where each object contains:
+On success, an object containing **reservations** is returned. It is an array of objects, where each object contains:
--- doc/lightning-autoclean-status.7.md.old 2024-10-29 17:33:07.732521462 +1030
+++ doc/lightning-autoclean-status.7.md 2024-10-29 17:42:37.441279965 +1030
@@ -9,7 +9,7 @@
-The **autoclean-status** RPC command tells you about the status of the autclean plugin, optionally for only one subsystem.
+The **autoclean-status** RPC command tells you about the status of the autoclean plugin, optionally for only one subsystem.
--- doc/lightning-renepay.7.md.old 2024-10-29 17:33:07.927520140 +1030
+++ doc/lightning-renepay.7.md 2024-10-29 17:42:37.996268504 +1030
@@ -58,6 +58,9 @@
- **status** (string) (one of "complete", "pending", "failed"): Status of payment.
+- **bolt11** (string, optional): The bolt11 invoice paid. *(added v23.08)*
+- **bolt12** (string, optional): The bolt12 invoice paid. *(added v23.08)*
+- **groupid** (u64, optional): The groupid used for these payment parts (as can be seen in listsendpays) *(added v23.08)*
- **destination** (pubkey, optional): The final destination of the payment.
--- doc/lightning-sendonion.7.md.old 2024-10-29 17:33:07.937520073 +1030
+++ doc/lightning-sendonion.7.md 2024-10-29 17:42:37.957269309 +1030
@@ -22,7 +22,7 @@
-- **first\_hop** (object): Instructs Core Lightning which peer to send the onion to. It is a JSON dictionary that corresponds to the first element of the route array returned by *getroute*.:
+- **first\_hop** (object): Instructs Core Lightning which peer to send the onion to. It is a JSON dictionary that corresponds to the first element of the route array returned by *getroute* (so fields not mentioned here are ignored).:
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `renepay` return fields documented in schema (`bolt11`, `bolt12` and `groupid`)
This is like `sendonion` but unwraps the onion as the first hop,
avoiding nasty special cases for blinded paths which start with this
node, and also self-pay.
Tests split into multiple ones after Christian's review.
Changelog-Added: JSON-RPC: `injectpaymentonion` for initiating an HTLC like a peer would do.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Don't assume we have an outgoing HTLC at this level.
Note that previously we didn't save the failed onion unless it was
unparsable: we keep that both for space savings and because our
`waitsendpay` logic assumes that when it fetches from the db if
there's a failonion it was unparsable!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This message is supposed to include the msat amount received. But this is
obviously per-HTLC, and we hacked it to use the value for the first one.
And we add logging whenever we fail an HTLC set, since we removed logging
by not calling failmsg_incorrect_or_unknown() (which, now, no longer needs
to log).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Removes the `COMPAT_V070` functionality for `listfowards`.
Changelog-Changed: The `listforwards` command will now return a value
of 0 for `received_time` for very old forward attempts.
Changelog-Fixed: The documentation version was calculated as `pre-v24.08` for point releases like v24.08.1` also because `CLN_NEXT_VERSION` has not been included in the point release branches. Updating the script to build documentation on new tags and change the version to `pre-cln-next-version` for non-tagged commits.
Changelog-Added: JSON-RPC: `decode` now used modern BOLT 4 language for blinded paths, `first_path_key`.
Changelog-Deprecated: JSON-RPC: `decode` `blinding` in blinded path: use `first_path_key`.
Changelog-Added: Plugins: `onion_message_recv` and `onion_message_recv_secret` hooks now used modern BOLT 4 language for blinded paths, `first_path_key`.
Changelog-Deprecated: JSON-RPC: `onion_message_recv` and `onion_message_recv_secret` hooks `blinding` in blinded path: use `first_path_key`.
No code changes, just catching up with the BOLT changes which rework our
blinded path terminology (for the better!).
Another patch will sweep the rest of our internal names, this tries only to
make things compile and fix up the BOLT quotes.
1. Inside payload: current_blinding_point -> current_path_key
2. Inside update_add_htlc TLV: blinding_point -> blinded_path
3. Inside blinded_path: blinding -> first_path_key
4. Inside onion_message: blinding -> path_key.
5. Inside encrypted_data_tlv: next_blinding_override -> next_path_key_override
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is obsolete (since modern onions) and so removed from spec.
We should not set it, and don't need to handle it specially.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we should support it by default.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `option_quiesce` enabled by default.
Changelog-Deprecated: Config: --experimental-quiesce: it's now the default.
We build with this: it changes the blinded_path field to sciddir_or_pubkey.
But it wasn't committed, so if someone rebuilt the wire files they'd be wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we know the total reservations on each hop, we can more easily
determine probabilities than using flowset_probability() which has to
replicate this collision detection.
We leave both in place for now, to check. The results are not
identical, due to slightly different calculation methods.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were trying to get the max capacity of a flow to see if we could add some
more sats, and hit an assertion:
tests/test_askrene.py:707:
```
DEBUG plugin-cln-askrene: notify msg info: Flow reduced to deliver 88070161msat not 90008000msat, because 107x1x0/1 has remaining capacity 88071042msat
DEBUG plugin-cln-askrene: notify msg info: Flow reduced to deliver 284138158msat not 284787000msat, because 108x1x0/1 has remaining capacity 284141000msat
**BROKEN** plugin-cln-askrene: Flow delivers 129565000msat but max only 56506138msat
INFO plugin-cln-askrene: Killing plugin: exited during normal operation
```
We need to *unreserve* our flow before asking for max capacity. We were
also missing a few less important cases where we altered flows without altering
the reservation, so fix those too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that increasing mu a little bit sometimes made a big difference,
because by completely ignoring fees we were choosing the worst of two channels
in some cases.
Start at 1% fees; this saves a lot on initial fees in this test!
Here's the new stats on mu levels:
96 mu=1
90 mu=10
41 mu=20
30 mu=30
24 mu=40
19 mu=50
22 mu=60
8 mu=70
95 mu=80
19 mu=90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `askrene` is now better at finding low-fee paths.
While the `k=8` value worked for the current main network tests with the
amounts in those tests, it wasn't robust across a wider range of values
(as demonstrated when other test changes broke tests!).
Time to do this properly: calculate the ratio at the time we combine them,
using median values.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Even after the previous fix, we still occasionally increase fees when my increases.
This is due to the difference between MCF's linear fees, and actual fees, and
is unavoidable, but add a check if it somehow happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed this in the logs:
plugin-cln-askrene: notify msg unusual: The flows had a fee of 151950msat, greater than max of 53697msat, retrying with mu of 10%...
plugin-cln-askrene: notify msg unusual: The flows had a fee of 220126msat, greater than max of 53697msat, retrying with mu of 20%...
We would expect increasing mu to *reduce* the fee!
Turns out that our linear fee is a bad terrible approximation, because I
was using base_fee_penalty of 10.0.
|
| / __ <- real fee, with base: fee = base + propfee * amount.
| / __/
| _//
| __/
| __/_/
|/ _/
| _/ <- linearized fee: fee = linear * amount
|/
+-----------------------------------
These cross over where linear = propfee + base / amount. Assume we split the
payment into 10 parts, this implies that the base_fee_penalty should be 10 / amount
(this gives a slight penalty to the normal case, but that's ok).
This gives better results, too: we get down to 650099 sats in fees, vs 801613
before.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During "test_real_data", then only successes with reduced fees were 92 on "mu=10", and only
1 on "mu=30": the rest went to mu=100 and failed.
I tried numerous approaches, and in the end, opted for the simplest:
The typical range of probability costs looks likes:
min = 0, max = 924196240, mean = 10509.4, stddev = 1.9e+06
The typical range of linear fee costs looks like:
min = 0, max = 101000000, mean = 81894.6, stddev = 2.6e+06
This implies a k factor of 8 makes the two comparable.
This makes the two numbers comparable, and thus makes "mu" much more
effective. Here are the number of different mu values we succeeded at:
87 mu=0
90 mu=10
42 mu=20
24 mu=30
17 mu=40
19 mu=50
19 mu=60
11 mu=70
95 mu=80
19 mu=90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current prob_cost_factor setting does not seem to make mu very
effective, in fact, it gives strange results:
plugin-cln-askrene: notify msg unusual: The flows had a fee of 151950msat, greater than max of 53697msat, retrying with mu of 10%...
plugin-cln-askrene: notify msg unusual: The flows had a fee of 220126msat, greater than max of 53697msat, retrying with mu of 20%...
We would expect increasing mu to *reduce* the fee!
As a first step, simplify (it can't be infinite, and the -1 are weird).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We ask it again, but reduce fees by 1msat from the previous answer.
This is really nasty, as it frequently exercises the case where we
only go over fee when we do the refinement step.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I tested with a really large gossmap (hacked to be 4GB), and when we
keep retrying to minimize cost (calling minflow 11 times), and we
don't free tmpctx.
Due to an issue with how gossmap estimates the index sizes, we ended
up running out of memory. This fixes it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Flow cycles can occur if we have arc zero arc costs.
The previous path construction from the flow in the network assumed the
absence of such cycles and would enter an infinite loop if it hit one.
With his patch wee add cycle detection and removal during the path
construction phase.
Reported-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-EXPERIMENTAL: `askrene` infinite loop fixed
This happens in the coming "real network" test!
We add fees and hit htlc_max, but don't have another flow to add to.
Rather than MCF again, we split the flow into two.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The fp16_t values are approximations (overestimate for htlc_max,
underestimate for htlc_min), so in the refinement step we should use
the exact values.
This also fixes a logic bug: flow_remaining_capacity returned the
total capacity, not the additional capacity!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `askrene` now honors exact htlc_maximum_msat limits.
In particular, this lets you find the exact htlc_maximum_msat/htlc_minimum_msat
values.
This means we actually create real channel_updates for local mods, which
requires a second "local" scratch region.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `getroutes` now applies `auto.sourcefree` layer in the order specified, so doesn't alter channels changed in later layers.
Rather than adding to the gossmap modifications directly, populate
the layer and have the normal layer application logic do it.
This is consistent when we query layers in the next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And unify logging for better debugging.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `askrene` now has better logging, gives notifications of progress.
The "init_cupdate" message is for gossipd to tell lightningd about our *own* latest
channel_update messages, not the remote ones! The "remote_channel_update" message
is for messages from the peer.
This appeared as an occasional BROKEN message in CI:
```
**BROKEN** 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-chan#4: gossipd gave us channel_update for channel in gossip_state CGOSSIP_NEED_PEER_SIGS
```
Where we had sent (and not received) announcement_signatures, and restarted: the peer had meanwhile
sent us their channel_announcement and channel_update.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: we could get confused on restart and not re-transmit our own channel_updates.
tmpctx may not get cleaned immediately, so the timeout (a child of
the struct early_peer at this point) can still outlast the conn.
Do the clearer thing, and explicitly free the timeout.
Changelog-Fixed: connectd: crash on erroneous timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Whenever we build without `make clean` first, this file gets
overwritten. It asks the user for input because it is not run with
`-f`. This causes the build to hang until the user inputs `y`.
Here is an example build:
```
$ make -j7 && sudo make install
CC: cc -DCLN_NEXT_VERSION="v24.08" -DPKGLIBEXECDIR="/usr/local/libexec/c-lightning" -DBINDIR="/usr/local/bin" -DPLUGINDIR="/usr/local/libexec/c-lightning/plugins" -DCCAN_TAL_NEVER_RETURN_NULL=1 -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -Werror -Wno-maybe-uninitialized -Wshadow=local -std=gnu11 -g -fstack-protector-strong -Og -I ccan -I external/libwally-core/include/ -I external/libwally-core/src/secp256k1/include/ -I external/jsmn/ -I external/libbacktrace/ -I external/gheap/ -I external/build-x86_64-linux-gnu/libbacktrace-build -I external/libsodium/src/libsodium/include -I external/libsodium/src/libsodium/include/sodium -I external/build-x86_64-linux-gnu/libsodium-build/src/libsodium/include -I . -I/usr/local/include -I/usr/include/postgresql -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS -DCOMPAT_V052=1 -DCOMPAT_V060=1 -DCOMPAT_V061=1 -DCOMPAT_V062=1 -DCOMPAT_V070=1 -DCOMPAT_V072=1 -DCOMPAT_V073=1 -DCOMPAT_V080=1 -DCOMPAT_V081=1 -DCOMPAT_V082=1 -DCOMPAT_V090=1 -DCOMPAT_V0100=1 -DCOMPAT_V0121=1 -c -o
LD: cc -Og config.vars -Lexternal/build-x86_64-linux-gnu -lwallycore -lsecp256k1 -ljsmn -lbacktrace -lsodium -L/usr/local/include -lm -lsqlite3 -L/usr/lib/x86_64-linux-gnu -lpq -o
mv: replace 'tools/headerversions', overriding mode 0755 (rwxr-xr-x)? cc lightningd/test/run-check_node_announcement.c
cc lightningd/test/run-find_my_abspath.c
cc lightningd/test/run-invoice-select-inchan.c
cc lightningd/test/run-jsonrpc.c
cc lightningd/test/run-log_filter.c
cc lightningd/test/run-log-pruning.c
cc lightningd/test/run-shuffle_fds.c
ld lightningd/test/run-find_my_abspath
ld lightningd/test/run-log-pruning
ld lightningd/test/run-check_node_announcement
ld lightningd/test/run-log_filter
ld lightningd/test/run-jsonrpc
ld lightningd/test/run-shuffle_fds
ld lightningd/test/run-invoice-select-inchan
^Cmake: *** [tools/Makefile:16: tools/headerversions] Interrupt
```
One workaround is to just know that you need to enter `y` here.
But the best solution is probably to update the Makefile like so.
Changelog-None
Multiple places in the pay lifecycle depend on mods to be set. By
setting the mods directly after the first listpeerchannels call,
subsequent calls to listpeerchannels are avoided.
Changelog-Fixed: pay-plugin: only call listpeerchannels once during a
payment lifecycle.
Since we don't compact the gossmap on the fly (FIXME!) we can
easily surpass 4GB in the gossmap, and 32 bit offsets are not
sufficient.
I'm a bit surprised we don't crash immediately, but we've definitely
seen issues.
Changelog-Fixed: gossipd: crash errors with large gossip_store (>4MB) growth on longer-running nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For nodes with many channels this is a tremendous improvement in pay
performance. PR #7611 improves payment performance from 15 seconds to
13.5 seconds on one of our nodes. This commit improves payment
performance from 13.5 seconds to 5.7 seconds.
Changelog-Fixed: Improved pathfinding speed for large nodes.
This minimizes the need to convert back and forth from and to sat
values, and it also removes a new instance of sats in the public
interface (`channel_hints`).
Suggested-By: Rusty Russell <@rustyrussell>
We were ignoring more reliable information because of the
stricter-is-better logic. This meant that we were also ignoring local
channel information, despite knowing better.
By simplifying the logic to pick the one with the newer timestamp we
ensure that later observations always trump earlier ones. There is a
bit of interplay between the relaxation of the constraints updating
the timestamp, and comparing to real observation timestamps, but that
should not impact us for local channels.
It was failing because the channel_hint from one attempt would prevent
us from retrying. By changing the amounts so that the channel_hints do
not concern them (value smaller than estimate) we can make things work
as before again.
A failing payment would doom all subsequent ones. Now we step down the
amount a single satoshi so any prior channel_hints do not doom the
payment outright.
Changelog-None
We were using `channel_hint` to temporarily tweak the graph inside of
a payment. However, these ad-hoc `channel_hints` are stickier than
their predecessors, in that they outlive the payment attempt itself,
and interfere with later ones.
Changelog-Changed: pay: Discarding an overly long or expensive route does not blocklist channels anymore.
If we have a large channel, fail to send through a small amount, and
then add a `channel_hint`, then it can happen that the call to
`channel_hint_set_update` is already late enough that it refills the
1msat we removed from the attempted amount, thus making the edge we
just excluded eligible again, which can lead into an infinite loop.
We slow down the updating of the channel_hints to once every
hysteresis timeout. This allows us to set tight constraints, while not
incurring in the accidental relaxation issue.
We need to know the overall channel capacity, i.e., the amount_msat
that the channel was funded with, in order to relax the channel_hint
to refill over time.
When cln is run without deprecated apis, the `listchannels` call would
still query `listpeerchannels`, even though it wouldn't use the result.
This adds unnecessary time to the call. Especially in case the node has
many channels. This commit skips the listpeerchannels call in case the
outcome won't be used.
Changelog-Fixed: Improve listchannels performance
1. describe_disabled should point out if node itself is disabled.
2. Hoist constraint check for neater if branching.
3. Use amount_msat_max/min for greater clarity.
4. Simply disable channels, don't zero htlc_min/max when node disabled.
I also fixed the diagnostic of htlc_max correctly, which removes a FIXME.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The code is a bit too complex for gcc to track it:
```
In file included from ccan/ccan/tal/str/str.h:7,
from plugins/askrene/askrene.c:11:
plugins/askrene/askrene.c: In function ‘do_getroutes’:
ccan/ccan/tal/tal.h:324:23: error: ‘routes’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
324 | #define tal_count(p) (tal_bytelen(p) / sizeof(*p))
| ^~~~~~~~~~~
plugins/askrene/askrene.c:476:24: note: ‘routes’ was declared here
476 | struct route **routes;
| ^~~~~~
plugins/askrene/askrene.c:475:29: error: ‘amounts’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
475 | struct amount_msat *amounts;
| ^~~~~~~
plugins/askrene/askrene.c:488:69: error: ‘probability’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
488 | json_add_u64(response, "probability_ppm", (u64)(probability * 1000000));
| ~~~~~~~~~~~~~^~~~~~~~~~
cc plugins/askrene/dijkstra.c
cc1: all warnings being treated as errors
```
On my local machine, it also warns in param_dev_channel, so I fixed that too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This turns out to be critical for users: also stops them from
bothering us when their node is offline or has insufficient capacity!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Lagrang3 points out it's less useful (when we time them out), and probably
a premature optimization anyway.
Suggested-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows for explicit partial updates to channels (e.g. just change
fees, or just disable) without haveing to set the other fields.
This generalizes askrene-disable-channel, which is removed.
We also take the chance to use the proper BOLT 7 terms in the API:
- htlc_minimum_msat
- htlc_maximum_msat
- cltv_expiry_delta
- fee_base_msat
- fee_proportional_millionths
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was weird not to have a capacity associated with localmods channels, and
fixing it has some very nice side effects.
Now the gossmap_chan_get_capacity() call never fails (we prevented reading
of channels from gossmap in the partially-written case already), so we
make it return the capacity. We do this in msat, because that's what
all the callers want.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is actually what we want in several places: to only override one or
two fields in a channel_update.
We add a gossmap_local_setchan() with a similar API to the old
gossmap_local_updatechan(), for the case where we want to set every
field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Lagrang3 points out that if we hit a maximum, we should take into account
the reserve. This is true, but it's hard for the caller to do, so change
the API to be slightly higher level.
Tell "inform" what happened, and it adjust the constraints appropriately.
This makes the least assumptions possible (a reserve does *not* mean that
the capacity was actually used at that time).
We also add a mode to say "this succeeded": for now this does nothing,
but it could reduce both min/max capacities, and add capacity in the
other direction. This is useful for future payments, but not as useful
for the current one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I got confused, as we had a struct containing two arrays. Simply expose the
reserve_hop struct and use arrays directly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's generally better to be explicit with these things: currently typos
would be ignored. But it's also much easier to clean up entire layers
as we use them for temporary (per-payment) effects.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We allow adding them, but crash when we remove the localmods. Yet
this could theoretically happen if a channel we modified was removed
from the gossmap, anyway.
Reported-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I like the clarity, but this is a hot path. Fortunately these arrays
have very well defined lengths.
Before: 5.81 seconds
After: 1.06 seconds
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only ever visit each node once, so we can just use an array. This
avoids calling tal() all the time, which is *especially* slow when we're
memory tracking.
I had an old canned gossmap which I benchmarked for these (and in
particular one node was unreachable, and that was slow):
Before: 17.27 seconds
After: 5.80 seconds
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows tools to validate that it is accessing the correct hsm_secret for this node!
This is extremely important for backups: if they are using VLS, they need to back *that*
up instead, for example.
Changelog-Added: `hsmtool`: `getnodeid` command derives the node id from the hsm_secret, to verify it's the correct secret.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We bump the version to minor version 0.2 to allow ourselves to bump on
patch level when we need to fix a bug or update a dependency.
Changelog-Changed: Plugins: `cln-grpc` Upgrade tonic version and
introduce new versioning scheme.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Tonic had some breaking changes since 0.8. We want to be closer to newer
versions for downstream consumers to be able to benefit from changes in tonic.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
The `anchors/even` channel type was missing from the grpc bindings.
It is now the default channel type, so `fundchannel` over grpc fails
with:
```
Failed to deserialize response : unknown variant `anchors/even`,
expected one of `static_remotekey/even`, `anchor_outputs/even`,
`anchors_zero_fee_htlc_tx/even`, `scid_alias/even`, `zeroconf/even`
```
Changelog-Changed: Channel type `anchors/even` was added
to the grpc bindings.
This name is clearer than the old one.
And since the struct contains a string, it's more natural for the
struct to be the tal parent of the string so it's a real object. This means
we need an array of pointers, so each struct can be its own tal object.
wallet_state_change_get is hoisted higher in the code and made static.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And instead of loading them in listpeerchannels, use them. This means
listpeerchannels no longer touches the db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSONRPC: `listpeerchannels` (and thus, pay) sped up on very large nodes.
Indirection via a string and an enum is just adding confusion here.
And move the `struct channel_stats` into channel.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids a db lookup on every iteration of listpeerchannels, which
can be slow on large nodes (Postgres, I assume).
We can now simply add the fields we want to channel load, and remove
wallet_channel_stats_load entirely.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
askrene.c was getting quite long, and this is self-contained.
The only code change is a convenience accessor for the per-htlc-cost
hash table.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
"spendable" is for a single HTLC: if we own the channel, this amount
decreases with every HTLC, as we have to pay fees. We have access to this since
we call listpeerchannels anyway, so we can calculate the additional costs and
use it in the refining phase.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't actually hit the htlc_max cases, since the flow code already
constrains us to that.
And handling htlc_min is better done in the caller, where diagnostics
are better (basically, we should eliminate them, and if that means no
route, give a clear error message).
And the refinement step can handle any extra millisats from rounding.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the root cause of the problem worked around in 50949b7b9c
"askrene: hack in some padding so we don't overflow capacities."
When adding fees to flows, we didn't recheck the boundary conditions: in
renepay this is done by routebuilder.
Fortunately, we can use our "reservations" infrastructure to temporarily
use capacity as we process flows, so we handle the cases where they are
not independent correclty.
My assumption is that the resulting errors are small, so we divide
them between the remaining flows based on highest-to-least
probability.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Simply calculate it when we need it, which means we don't have to keep it
up-to-date as we tweak the flow.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than making the callers do this, make the invoice decoder perform
the various sanity checks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a workaround for channels added by "auto.local", but instead we
should make it work properly.
I didn't do this before because we can't manipulate the localmods while
they're applied, but it's simple to do it in two stages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not quite the same, as it doesn't have the "auto.local" layer, but it exhibits
the same problem if we revert the fix for test_live_spendable.
And it's much faster!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I used `amount_msat_eq(x, AMOUNT_MSAT(0))` because I forgot this
function existed. I probably missed it because the name is surprising,
so add "is" in there to make it clear it's a boolean function.
You'll note almost all the places which did use it are Eduardo's and
Lisa's code, so maybe it's just me.
Fix up a few places which I could use it, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If I put in X, how much can I get out after fees are subtracted?
This was inspired by Eduardo's channel_maximum_forward in renepay, which
is basically the same thing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was incorrect once we stopped removing old payments on failure,
which was the reason we had to remove the HTLCs.
It also removed by partid, which is wrong since it should have done
old_payment->partid and old_payment->groupid!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- Removes CI value error for Broken logs
- Fixes CI errors due to deprecated listconfigs 'important-plugins'
- Removed listchannels deprecated local test
Changelog-None.
Thanks to Michael Schmook for the excellent bug report.
Fixes: https://github.com/ElementsProject/lightning/issues/7671
Changelog-Fixed: lightningd: no longer crash if a plugin dies during lightningd startup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In commit 60e1532dd8 (released in crate 0.1.8), which switched the
logging framework to tracing-subscriber, the default log level filter
was (accidentally) set to ERROR and above, instead of INFO and above.
Change this back to INFO as it was before. It can still be overridden
with CLN_PLUGIN_LOG.
Follows the example in https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html#method.from_envCloses#7658.
Changelog-Fixed: cln-plugin: Change default log level filter back to INFO
It's an output field (which we don't complain about), not an input field!
Fixes: https://github.com/ElementsProject/lightning/issues/7652
Changelog-Fixed: Logging: removed bogus "**BROKEN** plugin-topology: DEPRECATED API USED: listchannels.include_private" message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For BOLT12 invoices, the "description" is field missing in the invoice
responses. Update the schemas accordingly:
- `doc/schemas/lightning-waitanyinvoice.json`
- `doc/schemas/lightning-waitinvoice.json`
Also commit the generated msggen, cln-grpc, cln-rpc and pyln-grpc-proto files.
Changelog-Fixed: schemas: Make description in `Wait(any)invoiceResponse` optional to handle BOLT12
Our low-level ccan/io IO routines return three values:
-1: error.
0: call me again, I'm not finished.
1: I'm done, go onto the next thing.
In the last release, we tweaked the sematics of "-1": we now opportunistically
call a routine which returns 0 once more, in case there's more data. We use errno to
distinguish between "EAGAIN" which means there wasn't any data, and real errors.
However, if the underlying read() returns 0 (which it does when the peer has closed
the other end) the value of errno is UNDEFINED. If it happens to be EAGAIN, we will
call it again, rather than closing. This causes us to spin: in particular people reported
hsmd consuming 100% of CPU.
The ccan/io read code handled this by setting errno to 0 in this case, but our own
wire low-level routines *did not*.
Fixes: https://github.com/ElementsProject/lightning/issues/7655
Changelog-Fixed: Fixed intermittant bug where hsmd (particularly, but also lightningd) could use 100% CPU.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we pull and old channel from the database, it might not.
Fixes: https://github.com/ElementsProject/lightning/issues/7645
Changelog-Fixes: lightningd: crash when starting channeld for older channel with no local alias.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Lightningd should log if a builtin plugin fails to start instead of
silently skip over it.
Changelog-Add: log message if builtin plugin fails to start.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
send_tlvs is NULL if no special features are supported, and peer
sets `next_to_send` anyway:
```
0x5ed1c6719538 peer_reconnect
channeld/channeld.c:5205
0x5ed1c6719dab init_channel
channeld/channeld.c:5959
0x5ed1c6719f04 main
channeld/channeld.c:6005
```
Backport: v24.08
Fixes: https://github.com/ElementsProject/lightning/issues/7486
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- Updated `release-checklist` with new GitHub actions for Docker and PyPI publishing.
- Updated `repro` with information about the `cl-repro.sh` script.
Changelog-None.
Running `git describe` in local setup correctly identifies the tag available on the tag. But `docker buildx` via actions does not identify it without `--tags` param.
Reference Issue: https://github.com/ElementsProject/lightning/issues/7626
Changelog-Fixed: Docker image created via github actions correctly reads the tag available on the HEAD.
Python was installing `pyln-client` directly from the server for testplugpass plugin. This commit is updating the requirements.txt file to install pyln-client with absolute local path.
Commit a1fdeee76b "Makefile: clean up install path handling."
broke the ability to configure with one path and then run in a
different path. Turns out people actually do this! So, we have
to use relative paths, compared to our existing binary.
And we can't use path_rel, because that requires that the path
exist (thanks @Lagrang3!).
Fixes: https://github.com/ElementsProject/lightning/issues/7595
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Of course, we still will, since spendable is for a single HTLC, but
this also shows why we should treat *minimum* as the incorrect answer
if they cross, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7563
It seems we didn't handle it correctly: we need to cap the first
segment as well as the others, as far as I can tell.
Also, it can be less than the maximum capacity.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Conversion is lossy, and we don't want to spend more than the channel,
so it's conservative to round down here.
This doesn't actually help our test though!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Based on great test case from https://github.com/daywalker90
```
E AssertionError: assert {'107x2x0/1': 'Path total 285720859 > spendable 285718000', '108x1x0/1': 'Path total 384721849 > spendable 384718000'} == {}
E Left contains 2 more items:
E {'107x2x0/1': 'Path total 285720859 > spendable 285718000',
E '108x1x0/1': 'Path total 384721849 > spendable 384718000'}
E Full diff:
E {
E - ,
E + '107x2x0/1': 'Path total 285720859 > spendable 285718000',
E + '108x1x0/1': 'Path total 384721849 > spendable 384718000',
E }
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- **Poetry Issue:** The Docker build for `linux/arm/v7` failed in recent RC releases on the Poetry installation step in the `builder-python` stage. This issue occurred because the `builder-python` stage builds on target's arch but poetry was unable to install on arm/v7 without rust >= v1.56.1.
- **Solution:** Instead of installing poetry on the `builder-python` stage, we leveraged the existing multi-arch `builder` stage, which already had Poetry. Now, we export the dependencies from `pyproject.toml` to `requirements.txt` within the `builder` stage and then copy `requirements.txt` to the `builder-python` stage for pip installation.
- **Cryptography installation Issue:** python installations for `pyln-proto` started failing due to Cryptography upgrade from v41 to v42 (#7475). It is because now Cryptography needs cargo/rust also.
- **Solution:** Installing cargo in `builder-python` stage also.
- **Configure Prefix Issue:** Previously, we used `RUN ./configure --prefix=/tmp/lightning_install --enable-static` in the `builder` image and then copied `/tmp/lightning_install` from the `builder` stage to `/usr/local` in the `final` stage. However, this approach is now causing errors due to missing binaries/plugins at their default locations.
- **Solution:** We are now configuring the installation to use the default location (`/usr/local`). To prevent the local image size from increasing by up to 87MB, instead of copying the entire `/usr/local/` directory, we are explicitly copying only the core lightning binaries.
Changelog-Fixed: Fixes failing Docker build for `arm32` arch.
- Temporarily adding `rc` tag trigger for testing `Build and push multi-platform docker images` action flow before the final release.
- Added some variable inputs for testing like repo, platforms, etc.
- Added more logs for future debugging.
You need to know it to make an onion, and in theory if we decided to
fuzz it could be different for different paths.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Channel filter must apply to the modified gossmap+localmods,
otherwise we disable local channels with htlcmax=0.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Add a little bit of uncertainty to the local channels to avoid
consuming precisely all spendable_msat on our side, which leads to
temporary channel failure if the spendable_msat changes during the
course of the payment.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Refresh gossmap once before we read the routehints.
Make sure that if channels in the routehints are public,
we retrieve their capacities from gossip.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
- add more checks
- add more error messages
- compute probabilities without fees during MCF
- compute probabilities with fees during get_routes
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
In theory we should not have htlc_total<=known_max.
But for some strange race condition we do sometimes.
Until we find a solution to ensure the correct state
of the uncertainty network we remove the assertion.
Thanks to signed arithmetic and MIN guards, the rest
of the code in linearize_channel can handle the
weird cases with known_max<htlc_total.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
The probability for a success forward of x amount on a channel is now
computed as:
P_success(x) = b * P_rene(x)
where P_rene(x) is the probability distribution proposed by Rene
Pickhardt with a uniform distribution on the known liquidity of the
channel and 'b' is a new parameter we add in this PR, by the name of
"base probability". The "base probability" represents the probability
for a channel in the network choosen at random to be able to forward at
least 1msat, ie. of being alive, non-depleted and with at least one HTLC
slot. We don't know the value of 'b', but for the moment we assume that
it is 0.98 and use that as default.
As a consequence the probability cost becomes non-linear and non-convex
because of the additional constant term:
Cost(x) = - log P_success(x)
= - log b - - log P_rene(x)
= - log b + Cost_rene(x)
We currently don't handle well base fees and neither this additional
"base probability" but we can as a first approximation linearize the
cost function in this way:
Cost_linear(x) = (- log b)*x + Cost_rene_linear(x)
Changelog-Added: renepay: Add a dev parameter to renepay that represents
a constant probability of availability for all channels in the network.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Commit 901342b50d ("pyln-testing: require
explicit pattern for BROKEN messages.") changed to a manual scan of
logs rather than using is_in_log, so it needs to manually refresh,
otherwise we miss final log messages.
This causes us to often miss memleak messages, which are printed
only in the exit path!
Reported-by: Lagrang3
Fixes: https://github.com/ElementsProject/lightning/issues/7565
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Mainly, it just causes complaints if we're running a lightningd which is installed
(in which case, my_path is not stolen, unlike running locally installed).
Reported-by: daywalker90
Fixes: https://github.com/ElementsProject/lightning/issues/7569
Changelog-None: regression this release
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our hash tables allow duplicate keys, and we use that in a few places.
However, the get() function only returns the first, so it's not a good
idea with such hash tables.
Another patch fixes this at a deeper level (using different hash table
types depending on whether this table can have duplicates), but this
is the minimal fix for existing code.
This may be the cause behind us occasionally missing onchain events:
Fixes: https://github.com/ElementsProject/lightning/issues/7460
Fixes: https://github.com/ElementsProject/lightning/issues/7377
Fixes: https://github.com/ElementsProject/lightning/issues/7118
Fixes: https://github.com/ElementsProject/lightning/issues/6951
This fixes them in future: fixing them now will require something else.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightningd: occasionally we could miss transaction outputs (not telling gossipd, or even onchaind)
msggen cannot handle the complex type in renepay-exclude,
therefore I added a rule override for it, just like pay-exclude.
```
msggen cln-grpc/proto/node.proto
Traceback (most recent call last):
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/__main__.py", line 131, in <module>
main()
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/__main__.py", line 115, in main
run()
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/__main__.py", line 72, in run
service = load_jsonrpc_service(
^^^^^^^^^^^^^^^^^^^^^
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/utils/utils.py", line 241, in load_jsonrpc_service
methods = [load_jsonrpc_method(name) for name in grpc_method_names]
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/utils/utils.py", line 209, in load_jsonrpc_method
request = CompositeField.from_js(schema["methods"][rpc_name]['request'], path=name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/model.py", line 297, in from_js
field = ArrayField.from_js(fpath, ftype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/model.py", line 464, in from_js
itemtype = UnionField.from_js(child_js, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rusty/devel/cvs/lightning/contrib/msggen/msggen/model.py", line 393, in from_js
itemtype = PrimitiveField(
^^^^^^^^^^^^^^^
TypeError: PrimitiveField.__init__() missing 2 required positional arguments: 'added' and 'deprecated'
```
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-EXPERIMENTAL: renepay: add cli option "exclude" to manually disable channels and nodes.
We add a channel filtering paymod that disables channels that have very
low max_htlc. It can be expanded to consider other properties as well,
for instance high base fee, low capacity or high latency.
Changelog-Added: renepay: prune the network by disabling channels we
don't like, eg. very low max_htlc.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Expecting to have more than just a bunch of disabled channels if we
prune the lightning network heavily I am changing the internal data
structure of disabledmap from a simple array to a hashtable.
Have a finer control over disabled channels by targeting
short_channel_id_dir instead of short_channel_id,
ie. we can disable a single direction of a channel.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
If the rpc plugin is run while an older version of reckless is found on
PATH, it produces an error:
2024-08-01T18:32:00.849Z DEBUG plugin-recklessrpc: reckless-stderr:usage: reckless [-h] [-d RECKLESS_DIR] [-l LIGHTNING] [-c CONF] [-r] [--network NETWORK] [-v]
{install,uninstall,search,enable,disable,source,help} ...
reckless: error: unrecognized arguments: --json
Catch this and don't try to parse the output as json.
This allows generic subcommands to be passed to reckless-rpc along with
the target to search/install/etc.. These commands are unvalidated so
far and may crash the reckless process.
Changelog-Added: reckless-rpc plugin: issue commands to reckless over rpc.
This is actually totally fair, and LND, which does not support it without an option, rejects.
Thanks to Vincenzo and roasbeef for debugging this for me!
Fixes: https://github.com/ElementsProject/lightning/issues/7221
Changelog-Fixed: Protocol: we can now open unannounced channels with LND nodes again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The documentation and the implementation of inform-channel have
different orders for the arguments minimum_msat and maximum_msat.
We change the order in the documentation to match that of the
implementation: minimum_msat comes before maximum_msat.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Often I find this in the logs:
DEBUG plugin-cln-renepay: gossmap ignored 94692259263600 channel updates
Apparently gossmap_refresh does not initialize the input variable num_channel_updates_rejected.
Fix this by explicitely initializing it before calling gossmap_refresh.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
routefail object allocation was linked to route,
we had a crash of the plugin with the following error:
0x561f424fc07a send_backtrace
common/daemon.c:33
0x561f424fc102 crashdump
common/daemon.c:75
0x7f5b0e7dc04f ???
???:0
0x7f5b0e82ae2c ???
???:0
0x7f5b0e7dbfb1 ???
???:0
0x7f5b0e7c6471 ???
???:0
0x561f4252581f call_error
ccan/ccan/tal/tal.c:95
0x561f425258c8 check_bounds
ccan/ccan/tal/tal.c:169
0x561f425258f9 to_tal_hdr
ccan/ccan/tal/tal.c:179
0x561f42526283 tal_free
ccan/ccan/tal/tal.c:525
0x561f424e5379 routefail_end
plugins/renepay/routefail.c:52
0x561f424e557b handle_failure
plugins/renepay/routefail.c:431
apparently there was a race condition for which the route was first
freed before we arrived to routefail_end where we manually free
routefail. I don't see how this could have happened, but anyways
this subtle bug can be avoided by linking the routefail to the payment.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-Fixed: renepay: finalized routes have to be processed and
determine the payment status even after the payment goes into the
background (no current command active). Not doing so leads to finalized
routes getting stuck in the payment internal data structure and their
associated HTLCs in the uncertainty network don't get released.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
We rather have a payment hash table to look up payments based on the
payment hash than having a list.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Use success_data_from_listsendpays to check if there are "complete"
sendpays instead of imposing the presence of "complete" sendpays as a
precondition for success_data_from_listsendpays.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Simply move the "computed routes" array from the payment to the
routetracker. It makes sense to put all temporary stages of routing into
a single data structure: the routetracker.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
There were some dummy rpc calls to waitblockheight in the payment workflow
to allow the function stack to clear. But it is better if some steps in
the payment are executed "atomically" to avoid strange race conditions.
For example: all steps between getting pending sendpays, computing new
routes and sending those routes.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
It feels unsafe to rely on the internal state of the plugin's database
to tell how many pending sendpays there are for the current payment.
The safest way is to assume lightningd knows and thus use listsendpay
before computing routes.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
- remove payment pointer from routetracker, fetch payment if necessary
from payment_hash;
- "have results" condition as a function call to routetracker.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
The time of decay "TIMER_FORGET_SEC" was set to 1 hour,
which is very low, it would make the plugin try depleted channels after
just a couple of seconds for very small amounts.
I set it to 1 week ~ 6e5 seconds.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
When you have *lots* of events in your bkpr database looking up a
specific event via calling bkpr-listaccountevents and using jq or
grep to filter gets very slow (and wasteful of CPU and disk resources).
This commit adds the paremeter payment_id to the call to filter for a
specific payment id via a where clause in the request to the database of bkpr.
Changelog-Added: Plugins: Add payment_id parameter to bkpr-listaccountevents to filter events.
When a peer connects, we always send all our own gossip (even if they
had set the timestamp filters to filter it out). But we weren't
forcing it out to them when it changed, so this logic only applied to
unstable or frequently-restarting nodes.
So now, we tell all the peers whenever we tell gossipd about our new
gossip.
Fixes: #7276
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now send current peers our changed gossip (even if they set timestamp_filter otherwise), not just on reconnect.
We didn't actually check that we *send* the refreshed gossip, just
that we print the message saying we're going to.
So check that everyone received updated gossip when this happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes, for various reasons, a user disables an offer
and then wants to re-enable it. This should be allowed because,
from the CLN point of view, it is just an internal state.
If a user has constraints on the description of the invoice
because they are using services that link some sort of user ID
to an offer, it is important for the user to be able to re-enable the
offer, not create a new one. Creating a new offer would
require a different description.
Link: https://github.com/ElementsProject/lightning/issues/7360
Co-Developed-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
I will commit doc/schemas/lightning-*.json files separately to keep this commit easier to review.
- test-autogenerate-rpc-examples updates all example request & responses in doc/schemas/lightning-*.json files.
- Updated tools/fromschema.py to accommodate the sql JSON example requirement where it does not accept -o in the query but shell does (for queries containing the = sign).
Changelog-None.
Bitcoind's web server has a default of 4 threads, with queue depth 16 and it fails rather than queue beyond that.
Increasing thread count to 20 for running >10 lightning test nodes simultaneously.
1) We can't simply cast away const to manipulate a string, the compiler can assume
we don't. The type must be made non-const.
2) cisspace() is nicer to use than isspace() (no cast required!)
3) Simply place a NUL terminator instead of using memmove to set it.
4) Use cast_const to add const to char **, where necessary.
5) Add Changelog line, for CHANGELOG.md
Changelog-Fixed: Config: whitespace at the end of (most) options is now ignored, not complained about.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Did a sweep to find any others, give this from sanitizer:
```
2024-08-09T18:06:45.1729472Z plugins/bkpr/recorder.c:2057:23: runtime error: load of value 190, which is not a valid value for type 'bool'
2024-08-09T18:06:45.1729877Z SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior plugins/bkpr/recorder.c:2057:23 in
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Valgrind found this. I think two PRs were added in parallel, which is why we
only found it after merge:
```
2024-08-09T01:57:23.7375752Z ==34263== Uninitialised byte(s) found during client check request
2024-08-09T01:57:23.7376275Z ==34263== at 0x172405: memcheck_ (mem.h:247)
2024-08-09T01:57:23.7376661Z ==34263== by 0x172585: db_bind_int (bindings.c:49)
2024-08-09T01:57:23.7377086Z ==34263== by 0x126233: log_chain_event (recorder.c:2057)
2024-08-09T01:57:23.7377544Z ==34263== by 0x11BF8B: json_utxo_deposit (bookkeeper.c:1735)
2024-08-09T01:57:23.7378207Z ==34263== by 0x12BED3: ld_command_handle (libplugin.c:1847)
2024-08-09T01:57:23.7378674Z ==34263== by 0x12C649: ld_read_json_one (libplugin.c:1998)
2024-08-09T01:57:23.7379114Z ==34263== by 0x12C780: ld_read_json (libplugin.c:2018)
2024-08-09T01:57:23.7379534Z ==34263== by 0x2990CB: next_plan (io.c:60)
2024-08-09T01:57:23.7379881Z ==34263== by 0x299D21: do_plan (io.c:422)
2024-08-09T01:57:23.7380230Z ==34263== by 0x299D88: io_ready (io.c:439)
2024-08-09T01:57:23.7380585Z ==34263== by 0x29C1BC: io_loop (poll.c:455)
2024-08-09T01:57:23.7380980Z ==34263== by 0x12D439: plugin_main (libplugin.c:2230)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When renepay starts, one of the first operations it does is to check for
pending sendpays for the same invoice. Those are added up to an internal
database to keep track of. Also the amount they deliver to the
destination is computed so that the current payment rpc call would try
to complete the payment for whatever amount remains to pay.
For an incomplete payment after calling the RPC renepay I found this in
the logs:
2024-05-22T19:44:19.853Z DEBUG plugin-cln-renepay: There are pending sendpays to this invoice. groupid = 6 delivering = 0msat, last_partid = 10
Where delivering should be the sum of the amounts delivered by pending routes.
The `pay` plugin, as well as other plugins making use of the tree-pay
executor, will now emit their observations as they see them. The
notifications are sent on the `channel_hint_updated` topic, and any
subscriber will get them.
We also added a `timestamp` to the `struct channel_hint`, as these
observations now outlive the `pay` call, and have to be attenuated /
relaxed as they age, until we can eliminate them completely (when the
restriction is equal to the structural information gathered from the
gossip).
Changelog-Added: pay: Payments now emit `channel_hint_updated` notification to share inferred balances and observations across multiple payments.
It might be nice to let the bookkeeper keep track of external accounts
as well as the internal onchain wallet? To this end, we add some new
custom notifications, which the bookkeeper will ingest and add to its
ledger.
Suggested-By: @chrisguida
Changelog-Added: PLUGINS: `bookkeeper` now listens for two custom events: `utxo_deposit` and `utxo_spend`. This allows for 3rd party plugins to send onchain coin events to the `bookkeeper`. See the new plugins/bkpr/README.md for details on how these work!
We used to always target `now() + 300`, which ends up never really
confirming, as the fee estimate bumps into the min-relay-fee
limit. With this commit we set an absolute target of 2 weeks, and a
linear fee rampup, until we are at T-2h, at which point we just stick
with the estimate, and try with this increased feerate to try and get
the sweep confirmed.
This ought to make RBF transactions much more efficient for closing
channels.
Changelog-Fixed onchaind: The sweep deadline for to_us outputs would be reset on each restart of the subdaemon. Now the deadline is absolute in terms of the close height.
We do some fancy accounting for channel closures; since we're tagging
splice txs as closes we need to mark them as splices so we can treat them
as any other 'normal' on chain event.
We weren't properly notifying that a channel output has been spent in
the case of it being spent in a splice. This fixes the notification side
of the equation, however there's still some issues remaining for the
bookkeeper side (to come).
Changelog-Fixed: We now send a `coin_movement` notification for splice confirmations of channel funding outpoint spends.
Changelog-Added: pay: The pay plugin now checks whether we have enough spendable capacity before computing a route, returning a clear error message if we don't
This allows us to directly returnan error code based on where we
decided to abort, rather than attemtping to infer it from the parts.
Changelog-Added: pay: The pay plugin now returns better error codes
Also redirect config creation prompts to stderr in order to not interfere
with json output on stdout.
Changelog-Added: reckless provides json output with option flag -j/--json
This follows the same structure that enables python virtual environments:
reckless/
<plugin_name>/
<symlink to compiled bin>
source/
<clone of original source plugin dir>/
Changelog-Added: Reckless: added the ability to install rust plugins.
We had some weird code to try to do relative paths. Instead, use absolute
ones and keep life simple. Also rename "daemon_dir" to the clearer
"subdaemon_dir" as that's what it's used for.
We now need special magic to do "installcheck", which is a bit awkward,
but at least it's contained.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This populates information on both topology (i.e. unannounced channels) and capacity for the local node using `listpeerchannels`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This marks all channels around the source node as free (no delay, no fee). This is normally what we want, if we are calculating a path for ourselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In general, we should be using tmpctx unless there's a specific reason not to.
It's clear, and simplifies the code somewhat.
If tmpctx is not cleaned often enough, we can look at a per-MCF context, but this
seems like premature optimization.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We let the caller choose mu, and iterate if necessary: it can also
check its limits for fees, etc. Rationalize it to 0-100 inclusive for
human consumption.
This means we don't loop internally, and in fact there's only one
failure mode: we cannot find enough capacity.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't know anything about most channels, so we create an array of
fp16_t containing them. We zero out ones where we do know something,
and use the previous code as the slow path.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we never have to look up a local channel when asked the capacity.
We mark these dummy constraints with an MAX timestamp.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We apply all the gossmods for the layers they specified, and create a
naive routine to give the capacity of a channel given those layers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They tell us what paths they're using, so we can adjust capacity estimates
accordingly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'reserve-fixup.patch':
fixup! askrene: reservation implementation.
These are the repositories of all information.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'layers-fixup.patch':
fixup! askrene: add layers infrastructure.
This avoids globals (and means memleak traverses the variables!): we
only change over the test plugin though, to avoid unnecessary churn.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies the callers significantly: all channel_announcements now
have an amount, so gossmap_chan_get_capacity() only fails on a local
modification.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we actually check the other fields too, as per BOLT!
Reported-by: https://github.com/hMsatsFixes: #7513
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This helps code using generate_gossip_store() too, since it can map its identifiers
to the nodeids which were used.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Category, description and long description from `json_command` and `plugin_command` have been removed in favour of getting them from json schema. Reference PR: Add categories in RPC documentation #7485
Deprecating them in pyln-client as well. They will remove completely in future releases.
Changelog-Deprecated: pyln-client: category, description and long descriptions for RPC commands are deprecated now.
This is a hack, but we've had nodes missing gossip. LDK does this,
so until we get a better workaround, use this one.
Changelog-Changed: Protocol: we now always ask the first peer for all its gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
> warnings.warn(f"Error reporting testrun: {e}: {e.read()}")
E AttributeError: 'URLError' object has no attribute 'read'
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In CI, this would sometimes fail: we would timeout waiting for the
fetchinvoice reply. Never happened locally, so was annoying to debug.
What happened was simple: we called injectonionmessage then when it
returned, put the "sent" object in the linked list so we could recognize
any reply onion messages.
However, we were getting that reply before the plugin processed the response
to injectonionmessage. This is possible because there are two fds for
plugins: one for it to receive notifications and hooks (like onion messages)
and one for normal RPC usage (like commands to inject onion messages).
The fix is simple: put in the list *before* calling JSON RPC.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We ask each channeld to report its leaks, and keep a refcount of how
many are outstanding. When the channeld replies, or dies, we decrement
the count in a destructor.
But if the last channeld we're waiting for dies, we can call the
destructor in an unexpected place, and thus run leak detection when
we were partway through some operation, which gives a false positive.
Here's a backtrace showing it:
```
0x5f93405897e3 send_backtrace
common/daemon.c:33
0x5f93405381cf finish_report
lightningd/memdump.c:139
0x5f93405382a0 leak_detect_req_done
lightningd/memdump.c:172
0x5f9340705c41 notify
ccan/ccan/tal/tal.c:243
0x5f9340705ca5 del_tree
ccan/ccan/tal/tal.c:437
0x5f9340705ce8 del_tree
ccan/ccan/tal/tal.c:447
0x5f93407061f3 tal_free
ccan/ccan/tal/tal.c:532
0x5f9340563f5a subd_release_channel
lightningd/subd.c:924
0x5f934050fb91 channel_set_owner
lightningd/channel.c:31
0x5f9340511b84 channel_err
lightningd/channel.c:1081
0x5f93405121a3 channel_fail_transient
lightningd/channel.c:1095
0x5f934054e547 peer_channels_cleanup
lightningd/peer_control.c:187
0x5f9340550411 peer_connected
lightningd/peer_control.c:1689
0x5f9340525101 connectd_msg
lightningd/connect_control.c:629
```
Instead, do the lightningd detection up-front, where it's in a
clean environment.
Reported-by: Shahana Farooqui
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a final sweep to match the current BOLT12 text:
1563d13999d342680140c693de0b9d65aa522372 ("More bolt12 test vectors.")
Only two code changes, to change the order of checks to match the bolt,
and to give a warning on decode if a path is empty.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: `invoicerequest` will set a blinded path if we're an unannounced node.
Changelog-EXPERIMENTAL: offers: `sendinvoice` will use a blinded path in an invoice_request, if specified.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's an internal undocumented interface, which makes this change less painful.
We *do* check that the invreq_metadata maps to the given invreq_payer_id, which would
is required for us to sign it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's an internal difference, so doesn't actually break compatibility
(it would if we tried to prove we owned an old invoicerequest, but we
don't have infrastructure for that anyway).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They can do it now: before it would have been awkward to look up previous
payments to match it up for recurring offers (which need to use the same
key, hence the same invreq_metadata).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is going to allow us to move it out from lightingd into plugin,
easily. It's legal because the combination of offer id and label must
be unique (with recurrence, we use the same metadata anyay, since they
want to correlate with prior payments anyway).
We already broke recurrence in this PR, so we don't need another note to say
we're changing the key derivation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current interface, if given a tweak, uses a *different secret key*
and tweaks it. This was an early experiment: we will switch to using
a secret tweak for invoice_requests like we do for invoice path ids.
To make sure there's no funny business, *hsmd* hashes to form the
tweak (i.e. no zero tweaks!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For now we only use a fake id for requesting invoices (as a payer_key), but we
will eventually use this generically, and we want plugins to be able to map them
too, so use the same scheme as path_id: a generated secret using the makesecret
API.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
invoice_path_id is actually a generic path_id thing, so rename it.
We're going to use the same scheme for path secrets and the tweak to
node_id when we create a fake pubkey for invoice_requests, so a new
header is appropriate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Missing offer_description iff offer_amount also missing.
2. Missing offer_issuer_id iff offer_paths is present.
3. Short channel id on introduction point.
4. Experimental range.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The latest spec allows this to be omitted iff there is a blinded path
and it would be made up anyway.
In that case, the key they will use to sign the invoice will be the final
blinded key in the path we use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The `invreq_recurrence_counter` clashes with the coming addition of
invreq_paths, so we might as well move them all to the new experimental
ranges.
Changelog-EXPERIMENTAL: Recurring offers had incompatible changes, won't work against older versions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The signer may not be present at this time. If we want to keep the
check to protect against bit flips we should move it into `onchaind`
where it doesn't matter as much that the signer may be slow to
respond.
If we need to iterate forward to find a timestamp (only happens if we have gossip older than
2 hours), we didn't exit the loop, as it didn't actually move the offset.
Fixes: https://github.com/ElementsProject/lightning/issues/7462
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The latest draft allows these experimental ranges, which involves more
changes than I expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: handle experimental ranges in offers/invoice_requests/invoices.
Turns out we set this to False instead of None, so new field get "deprecated": False.
When we actually deprecate one, we get the following error:
ValueError: Field Decode.offer_node_id changed `deprecated` annotation: v24.08 vs False
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Things are often equivalent but different types:
1. u8 arrays in libwally.
2. sha256
3. Secrets derived via sha256
4. txids
Rather than open-coding a BUILD_ASSERT & memcpy, create a macro to do it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was only ever expermental, so I don't feel bad about just removing it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: `sendonionmessage` (was experimental only, use `injectonionmessage`)
Changelog-Added: Protocol: pay can now pay to bolt12 invoices if entry to blinded hop is specified as a short_channel_id (rather than node id).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This also lets us test offers and invoices work if we are the start of
the blinded path.
Changelog-Added: offers: automatically add a blinded path from a peer if we have no public channels, so unannounced nodes can have offers too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use this for invoices published by unannounced nodes: want
something very similar for offers, so generalize and expose it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Ideally, we would be able to submit a payment exactly as an incoming
HTLC would, but our forwarding and decoding logic is currently very
much tied to the HTLC. It would be wonderful to detach that and
have an "injectonion" interface which was unwrapped like any other
(see "injectonionmessage") which would handle self-pay without any
special paths.
Since I'm not prepared to rewrite that all now, instead we use an
interface to decrypt the first hop if it's us, and use the remainder
of the blinded path.
Changelog-Fixed: plugins: pay can now pay a bolt12 invoice even if we, ourselves, are the head of the blinded path within it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For bolt12, we have blinded paths so we route to the head of the blinded
path, which may not be the same as the final payment destination.
This matters mainly for detecting self-pay.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We check out the master bolts branch, and that recently changed test vectors
causing our CI to change. We should test them against our current BOLTVERSION,
which is in .tmp.lightningrfc/
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We generate a reply path by simply reversing the outgoing path:
A->B->C gives reply path B->A
A->B gives reply path A
But if we are not a public node, we can't use ourselves as the first
entry of the reply path: this happens if we directly connect to the
head of a blinded path (as we now support).
In this case, give the entire path as a blinded path. We could do
this all the time, but there are some cases where nodes don't like
sending replies where the node itself is the head of the blinded
path (like CLN v24.05 or before!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only need a connection with a peer, not an actual channel. So
add all peers to the local gossmap.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we can specify a blinded path in an offer, we have to check they
used it (or didn't use it, if we didn't have one!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We iterate through the blinded paths until we can use one, and because we use
the modern code, we properly join paths if we need to route more than one hop
to reach the start of the blinded path.
Changelog-EXPERIMENTAL: fixed: fetchinvoice tries all blinded paths until one is usable, and handles case where we have to route more than one hop to reach the entry point.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
send_onion_reply() does that for us now, so we don't need to do it up-front.
Simplifies the code quite a bit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And make it use inject_onionmessage. This means we have to be more careful
in createinvoice_done where we had a payload field allocated off tmpctx.
The new code correctly handles the case where we find a path (not just
a peer!) to the start of a blinded path, and need to join the paths.
It worked before if we had to connect directly, just not in the case
where we actually found a usable route of more than 1 hop.
Changelog-EXPERIMENTAL: fixed: onionmessage replies now work even if we need to route to the start of the blinded reply path.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will obsolete the existing calls to RPC "sendonionmessage", but
we transition by introducing it separately. It's designed to work with
the common/onion_message routines and "injectonionmessage".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is complicated, and I needed to write it down. All the current routines
are spread through the code, and I wanted it all in one place.
This implementation also support *joining* two paths together, which we
previously didn't.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We already parse some fields, so hand them directly rather than
having fetchinvoice behave as if it's a raw hook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means only a single gossmap, and they already share the fetchinvoice-noconnect option
and autoconnect code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: the `fetchinvoice` plugin has been combined into the `offers` plugin.
The *schemas* were added in v24.05, but the actual fields are much older. Unfortunately,
fixing this required a manual edit of the .msggen.json file, as msggen won't let the
added version change (for good reason).
Versions when these notifications were originally added:
connect: v0.6.3
channel_opened: v0.7.2.1:
channel_state_changed: v0.9.1
channel_open_failed: v0.9.3
block_added: v22.11
custommsg: v24.02
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Making sure that we are able to fetch the invoice
from an offer without description
E pyln.client.lightning.RpcError: RPC call failed: method:
offer, payload: {'amount': '2msat'}, error: {'code': -32602, 'message':
'bolt12: Offer does not contain a description: invalid token
\'"lno1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcgqyppvggz953rvg9rtxj8lalh43z8epwydjfrmffn3y3p5qz5cywpu09rr4vs"\''}
Link: https://github.com/ElementsProject/lightning/issues/7405
Link: https://github.com/ElementsProject/lightning/issues/7404
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Added: Protocol: onion messages are now supported by default.
Changelog-Deprecated: Config: the --experimental-onion-messages option is ignored (on by default).
However fast we can handle them, it's antisocial to allow others to
make us spam the rest of the network.
Changelog-Protocol: onion messages: we limit incoming to 4 per second, allowing a little burst.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is now permitted in the offers PR, so we should support it. But
we can't just look up in the gossmap, since the "short_channel_id"
could be an alias. So we get lightningd to tell us all scid->peer
mappings, and look up in that.
Changelog-Added: Protocol: onion messages can now be forwarded by short_channel_id.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we set them (i.e. at lockin), when we fire up channeld (for
aliases, which we create at channel init, but aren't really useful
until we have finished channel opening), and at startup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unlike "sendonionmessage" which instructs us to send to a peer, this
process it locally (presumably, it contains the next hop). This is
useful because it allows us to process an onion message which starts
with us (a legal case for a blinded path supplied by someone else!).
It also opens the door to bolt12 self-pay.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Allows for caller to log, but more importantly, when we add a command to
inject onion messages, allows for us to capture the error.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For some reason CI now hits a race where it tries to analyze a still-being-generated file:
```
tools/headerversions.c:52:30: error: syntax error [syntaxError]
new = tal_fmt(NULL, template,
^
make: *** [Makefile:552: check-cppcheck] Error 123
```
Restrict it to real source files instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A bit tricky, since we get more than one message at a time. However,
this just means we go over quota for a bit, and will get caught when
those are sent (we do this for a single message already, so it's not
that much worse).
Note: this not only limits sending, but it limits the actuall query
processing, which is nice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This basically means moving the code from gossipd to connectd to handle
these queries.
This will get connectd have finer control over ratelimiting them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was removed from the spec on Apr 25, 2022. We stopped ever sending them
in 0.12.0 (2022-08-23). Now we remove receive support.
Changelog-Protocol: Removed support for zlib-compressed short-channel-ids in query responses.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is more efficient in a few ways:
1. It's trivial to get to the end of the gossip_store, we don't have
to iterate.
2. It tends to be mmaped so we don't have to call pread().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently stream gossip as fast as we can, even if they start at
timestamp 0. Instead, use a simple token bucket filter and only let
them have 1MB per second (500 bytes per second for testing).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: connectd: we now throttle outgoing gossip at 1MB/second per peer.
We never call `listconfigs` in our tests with `experimental-offers` enabled,
so we didn't notice that the schema is wrong: it does not expect the
"plugin" field in 'fetchinvoice-noconnect'.
The next patch folds the fetchinvoice plugin into the offers plugin,
which is enabled even if `experimental-offers` isn't (for `decode`),
so we notice it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Document and enforce the --experimental-anchors deprecation, which was somehow missed in v24.02
Changelog-Deprecated: Config: the --experimental-anchors option is ignored (on by default since v24.02).
Only sphinx internally uses the hmac field: it's actually a general descriptor
of onion contents, which we can use elsewhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't use the payment_secret in bolt12, but in onion_decode() we
do put the path_secret (if of correct length) into payment_secret. Not
realizing this confused me, so document that, and make sure we insist
on it being present for bolt12.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was when we handled pre-TLV onions where the first byte was 0. We haven't
done that for a while: you can tell, because process_onionpacket doesn't use
the parameter at all!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This can't happen because we go the self-pay path in this case, but
once we fix that for bolt12, this can be reached.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were getting the following message in test_feerate_stress:
```
2024-07-08T02:15:45.5663941Z lightningd-2 2024-07-08T02:13:45.696Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: Peer did not close, forcing close
```
I can reproduce it locally if I run the test enough, and finally found
the issue by printing the status of the fd when we time it out (using
routines from connectd.c).
The peer fd alternates between reading and writing. When we go to
discard it, we wake the write queue, so write_to_peer() get called.
It won't shutdown the socket if there are still subds attached, and
will wait again for a read.
The last subd exit has to also wake the write queue if we're draining,
so it can do the io_sock_shutdown. Otherwise, we hit the timeout,
causing the message above.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Theoretical only, but we could leak an fd if we closed a conn before
the fd was sent. This doesn't happen in our current codebase because
we only hand fds to connectd, which only closes at shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only hand fds to connectd for now, so this doesn't happen (we don't
destroy that queue except on shutdown). But it's still a potential issue.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If the peer sends you their signatures before we return from the *first*
openchannel_update then the state would still be in
MULTIFUNDCHANNEL_STARTED (not UPDATED) and we'd incorrectly switch the
state to MULTIFUNDCHANNEL_SIGNED, which would plunge us headfirst into
`check_sigs_ready`.
However the `mfc->txid` hadn't been set yet, so we'd crash when trying
to confirm that we've got sigs for the correct txid. Oops.
To fix this, we simply invert the check such that the *only* state that
will move into SIGNED is SECURED
Interestingly, this patch and the last take my total test time on my
build machine down by about 10%. From 403.93s (0:06:43) to 366.64s (0:06:06)
though there were some unrelated errors.
Changelog-Changed: connectd: I/O optimizations to significantly speed up larger nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A fairly simple change: ccan/io will now call the underlying I/O
routines repeatedly until they indicate they are unfinished, *or* fail
with EAGAIN. This should make a significant difference to large
nodes, which currently spend far too much time calling poll() to
discover a single fd is still writable (mainly, for streaming gossip).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: connectd: now should use far less CPU on large nodes.
common/version.o depends on common/version_gen.h explicitly already,
and header_versions_gen.h is only used by lightningd, so we can make
that dependency explicit.
This means when version changes (i.e. different git commit) we only
relink, not recompile.
Before:
```
real 0m6.578s
user 0m14.705s
sys 0m13.621s
```
After:
```
real 0m2.098s
user 0m6.763s
sys 0m4.844s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This slows compilation somewhat when we make small changes (e.g. making the tree dirty).
Instead, simply mark them as "pre-XXX" or "XXX".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Their dependencies seem correct, so we shouldn't have to build them every time.
Before:
```shell
$ time make -s
real 0m5.183s
user 0m5.134s
sys 0m11.715s
```
After:
```shell
$ time make -s
real 0m0.784s
user 0m0.775s
sys 0m1.159s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The file `cln-rpc/src/notifications.rs` is generated by `msggen`.
The `Makefile` has tooling to update the file when needed
- `make check-gen-updated` should error if file isn't properly updated
- `make gen` updates the file
This comit fixes both behaviors mentioned above.
As suggested by @cdecker, I compared `poetry show --tree` results from before/after `cd plugins/clnrest/ && poetry export --output requirements.txt && pip install -r requirements.txt` to check for any significant difference. There is no difference in the tree.
But poetry export removed coincurve & other dependencies from clnrest's requirements.txt.
Changelog-None.
clnrest installation instruction on the markdown tries to install `flask_restx` not `flask-restx`.
Reference: clnrest plugin complains of flask_restx missing (#7383)
Changelog-None.
Documents which are deleted from lightning schema were still listed on readme server.
This script was adding or updating the list but delete action was missing.
Changelog-None.
Shahana decided this was the optimal UX path, though I insisted that we still
report the actual problem directly when in dev mode, as a compromise.
Suggested-by: https://github.com/Amperstrand
Changelog-Changed: JSON-RPC: Do not return the contents of invalid parameters in error messages, refer to logs (use 'check' to get full error messages)
Fixes: https://github.com/ElementsProject/lightning/issues/7338
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It doesn't actually log, just gets the `struct logger`, so this name is better.
We're also going to implement command_log() as an actual logging
function in next commit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
2024-06-24T05:09:32.5233603Z assert l1.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234187Z > assert l2.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234917Z E AssertionError: assert 'FUNDING_SPEND_SEEN' == 'ONCHAIN'
2024-06-24T05:09:32.5235464Z E - ONCHAIN
2024-06-24T05:09:32.5235773Z E + FUNDING_SPEND_SEEN
2024-06-24T05:09:32.5236096Z
2024-06-24T05:09:32.5236242Z tests/test_misc.py:2907: AssertionError
```
It's possible that l2 hasn't seen the onchain tx yet. What we really want to do is
wait for it, then test that l1 really can spend the output it has recovered.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
2024-06-24T05:14:14.9939033Z > l1.rpc.call('dev-feerate', [l2.info['id'], rate - 5])
2024-06-24T05:14:14.9939354Z
2024-06-24T05:14:14.9939466Z tests/test_connection.py:3439:
...
2024-06-24T05:14:14.9967617Z > raise RpcError(method, payload, resp['error'])
2024-06-24T05:14:14.9968833Z E pyln.client.lightning.RpcError: RPC call failed: method: dev-feerate, payload: ['022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 2290], error: {'code': -1, 'message': 'Peer bad state'}
```
The disconnect can actually take a while: wait for both sides to
believe they're reconnected. Also wait a little to make sure the
feerate change doesn't create a bad signature.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
lightningd-3 penalizes lightningd-2 but then it can see the preimage for the HTLC which
has already been timed out:
```
2024-06-24T02:41:29.4633900Z lightningd-3 2024-06-24T02:33:54.073Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-onchaind-chan#1: HTLC already resolved by THEIR_HTLC_TIMEOUT_TO_THEM when we found preimage
```
This is fair: the test deliberately takes l3 offline for long enough
that the HTLC can get timed out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Found by very slow CI: the two io_breaks() can combine into one, so we
block waiting for the second initialization which never happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
checkchain_timer is run by chaintopology, so why have the pointer in
bitcoind?
And since we free the timers, we don't need them to self-disable by
checking the stopped flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Handling half in main() and half here was a mess. And the name
"max_blockheight" was poor: it was the max in the db, or UINT32_MAX,
but then we changed it depending on what height we wanted to start at.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current code is confusing: there are polling loops, but we wait for
them to run once.
Be explicit: make the calls once, then start the loops in begin_topology.
This removes various chain_topology struct members which only exist for
startup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Example usage has been either merged with examples or usage keys. This heading was not required separately.
This PR edits & rearranges Examples, Example Notifications and Example Usage data.
Changelog-None.
- Added missing Example Notifications Heading
- Renamed `example_json_notifications` to `example_notifications`
- Moved `example_notifications` to the end of the page
- Changed ALL `doc/schemas/lightning-*.json` file's `json_example` to `examples`
- Change the heading from example to examples
- Bring shell command before the json command
- Move Example to the end of the page
- Remove horizontal line from Example
I was trying to debug a node with several multiplexed channels, and
was finding it a bit difficult to determine which channel index
matches which result of `listpeerchannels` as well as figuring out
what their status was. This just prints the status in string format
when loading the channel from the DB.
Changelog-Changed: wallet: The channel status is printed when loading it from the DB
[ Fix not to include 'value' and 'default' (if None) in getmanifest response --RR ]
[ Fix to support [] operator for existing plugins (including our test ones!) --RR ]
I was looking into using the `threading.Condition` but since we're
already rather heavily using callbacks, this allows us to stay
single-threaded, and not having to completely hook the `setconfig`
function.
Changelog-Added: pyln-client: Added a notification mechanism for config changes
We didn't actually *change* the value you'd see, when we got a setconfig call!
Changelog-Added: pyln-client: implement setconfig hook for plugins so you can see changes in `dynamic` options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can filter down to only the list* fields we need. In the case of a
node with 1M forwards, this reduces listforwards time from 5 seconds
to 4 seconds. It will also reduce memory consumption.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are really three subsystems: invoices, forwards and sendpays,
each of which has two variants we care about (successes and failures).
If we split the code that way, we can extract the core differences in
each of these cases and share most of the logic.
It's a bit awkward to iterate over each "subsystem" in the JSON
parameter sense, so we have some iteration code to do that where we
need to.
The result is going to be much easier to paginate!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're about to make this a more complex struct, so introducing a
new_clean_info() function unifies the code paths.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
From test_penalty_htlc_tx_timeout[False] flake (with keys replaced by l2 and l5, for clarity)
```
lightningd-3 2024-06-04T04:34:00.942Z UNUSUAL l2-chan#1: Peer permanent failure in CHANNELD_NORMAL: Funding transaction spent
lightningd-3 2024-06-04T04:34:01.570Z UNUSUAL l5-chan#3: Peer permanent failure in CHANNELD_NORMAL: Funding transaction spent
lightningd-3 2024-06-04T04:34:01.655Z UNUSUAL l5-chan#3: Abandoning unresolved onchain HTLC at block 132 (expired at 125) to avoid peer closing incoming HTLC at block 131
lightningd-3 2024-06-04T04:34:02.802Z **BROKEN** l5-chan#3: FUNDS LOSS of 50000000msat: peer took funds onchain before we could time out the HTLC, but we abandoned incoming HTLC to save the incoming channel
```
So, we were already closing l2, no reason to abandon it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2923 lines of output? Even with warnings-only, it's 59 lines, so only
enable that with `make V=1`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
v4.0 no longer works for me (see below, and widely reported elsewhere).
v5.0 doesn't understand f strings, and creates a flood of complaints.
v6.0 requires python >= 3.8.1, so we need to update that.
v7.0 is the latest, but why push it.
```
make check-python-flake8
Traceback (most recent call last):
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/bin/flake8", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/cli.py", line 22, in main
app.run(argv)
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 375, in run
self._run(argv)
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 363, in _run
self.initialize(argv)
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 343, in initialize
self.find_plugins(config_finder)
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 157, in find_plugins
self.check_plugins = plugin_manager.Checkers(local_plugins.extension)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/plugins/manager.py", line 363, in __init__
self.manager = PluginManager(
^^^^^^^^^^^^^^
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/plugins/manager.py", line 243, in __init__
self._load_entrypoint_plugins()
File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/plugins/manager.py", line 261, in _load_entrypoint_plugins
eps = importlib_metadata.entry_points().get(self.namespace, ())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'EntryPoints' object has no attribute 'get'
make: *** [Makefile:535: check-python-flake8] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This PR is adding github action for auto Docker releases. This will streamline Docker image builds & releases for enhanced efficiency, reliability, and consistency in managing Docker-based deployments.
The build & release event will be triggered:
- On any new tag creation without rc suffix.
- Manually for testing purposes.
Changelog-None.
This update will add lightning-cli examples and make examples heading collapsible.
- The JSON EXAMPLES heading is collapsible now
- Examples in lightning-cli format also
- manpage examples have better formatting
Changelog-None.
We could get the current key from the reestablish message even if we'd
lost our db, but there are very few of these channels left: we upgraded to use them
in the 2019-01-09 release.
We will eventually remove support altogether, but this is a nice removal of
some ugly code for something which "never happens".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Everyone understands gossip_queries now, but peers leave it unset to indicate
they have nothing useful to say.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have assumed this for a long time, so nothing changes.
Confusingly, this BOLT commit also cleaned up one reamining `option_anchors_zero_fee_htlc_tx`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a difficult transition for us: this string appears in channel
types. We make the transition now in the understanding that it will
be more difficult in future.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `listpeers` `features` array string "option_anchors_zero_fee_htlc_tx": use "option_anchors" (spec renamed it).
Changelog-Added: JSON-RPC: `listpeers` `features` array string uses "option_anchors" for feature 22/23, following renaming in BOLT 9.
Changelog-Changed: JSON-RPC: `listclosedchannels`, `listpeerchannels`, `openchannel_update`, `openchannel_init`, `fundchannel`, `fundchannel_start` and `multifundchannel`: `channel_type` array `names` now contains "anchors" instead of "anchors_zero_fee_htlc_tx".
Changelog-Changed: lightningd: `--list-features-only` now lists "option_anchors" instead of "option_anchors_zero_fee_htlc_tx".
These were removed from the spec.
We still support existing ones, though we were the only implementation
which ever did, and only in experimental mode, so we should be able to
upgrade them and avoid a forced close, with a bit of engineering...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We still support *existing* channels. Just not new ones (before they could,
in theory, explicitly ask for one).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Just use substrings not regexes, since we have more complex characters now LaTeX
is entering the spec!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we do have to set the network correctly though,
and also we can get query messages from lightningd which we have to filter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since `poetry=1.8.0` there is a non-package mode. This can be used to
create `pyproject.toml` files that are only used to track dependencies.
Performing `poetry install` in the project-root with a recent version of `poetry`
is currently generating a warning which will become an error in the
future.
Setting `package-mode=false` makes this error go away.
A side-effect is that the `name`, `version` and `description` in the
`pyproject.toml`-file will be ignored. Keeping the parameters in the
`pyproject.toml` file ensures it will still work for developers who are
using `poetry<1.8.0`.
We documented them as deprecated in v23.08 but unintentionally didn't deprecate them in the code. Thus we are starting their actual deprecation cycle from v24.08 release.
Updating removal version for commands `commando-rune`, `commando-listrunes`, and `commando-blacklist`
Changelog-None.
We do this with typesafe_cb and it's so useful I'm not going to remove it. But clang 18 complains:
```
ccan/ccan/tal/tal.c:246:6: runtime error: call to function destroy_conn_close_fd through pointer to incorrect function type 'void (*)(void *)'
/home/rusty/devel/cvs/lightning/ccan/ccan/io/poll.c:251: note: destroy_conn_close_fd defined here
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Somehow, this documentation got lost during the Great Rewrite, so
restore that too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `offer` removed `@` prefix support from `recurrence_base` (use `recurrence_start_any_period` set to `false`)
Changelog-Removed: Plugins: `estimatefees` returning feerates by name (e.g. "opening"); deprecated in v23.05.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Issue: Docker images for v24.05 were built on clean lightning directory and the published image should be running on v24.05 but it is running as v24.05-modded.
Root cause: Dockerfile builder was running different versions of grpcio-tools and protobuf. It resulted in auto generated update of contrib/pyln-grpc-proto/pyln/grpc/node_pb2.py and contrib/pyln-grpc-proto/pyln/grpc/primitive_pb2.py files.
Solution: Run `poetry install` before make in the dockerfile's builder stage to ensure that the grpcio-tools and protobuf versions are the same.
References:
https://github.com/ElementsProject/lightning/issues/7370#issuecomment-2152952820https://github.com/ElementsProject/lightning/pull/7376#issuecomment-2161102381
Changelog-None.
1. Plugin things should all be prefixed by `Plugins:` then the name of the plugin altered.
2. New config options should always be named in CHANGELOG.md!
3. Unify groups of changes into a single line, in this case, GRPC.
4. Command, plugin and option names are in backticks.
5. Offers changes are still under EXPERIMENTAL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This PR will help in publishing CLN reproducible binaries for Ubuntu v24.04 (noble).
Please note that I adjusted Dockerfiles for focal and jammy also to keep the base image creation script same for all three images. The step update was required because `noble` only runs with ubuntu:noble setup.
Changelog-None.
We do this by literally creating the modern-style TLV, and pretending we found it in the onion.
This isolates us from messing with any callers, who don't even know.
Co-programmed-with: Alex Myers <alex@endothermic.dev>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7347
Changelog-Fixed: Protocol: forward legacy non-TLV onions which we removed in 22.11 and spec itself in Feb 2022. Still sent by LND nodes who haven't seen our node_announcement.
This fails, because l2 can't decode the onion:
```
lightningd-2 2024-05-28T21:43:35.137Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Rejecting their htlc 0 since onion is unprocessable WIRE_INVALID_ONION_HMAC ss=4202c24ea44d9029a2ea3abb24cded51da93164f8bb5cddce9cc824af9945435
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to fire up channeld to send this, but:
1. That's silly, we have all the information to make it ourselves.
2. We didn't do it if there was an error on the channel, which as of 24.02
there always is!
3. When it did work, running channeld *stops* onchaind, indefinitely slowing recovery.
Fixes: https://github.com/Blockstream/greenlight/issues/433
Changelog-Fixed: Protocol: we once again send CHANNEL_REESTABLISH responses on closing channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fedora image building by build-release.sh currently throws error ` git: command not found`.
Fixed it by adding git in the Dockerfile and loading the image in the script.
Changelog-None.
This seems to be happening to some people, so don't panic. Unfortunately we don't have
a good error callback here, so msg to stderr.
Fixes: https://github.com/ElementsProject/lightning/issues/7249
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only write these in two places: one where we get a message from lightningd about
our own channel, and one where we get a reply from lightningd about a txout check.
The former case we explicitly check that we don't already have it in gossmap, so
add checks to the latter case, and give verbose detail if it's found.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
LDK will pick the *upper* limit (see: https://github.com/lightningdevkit/rust-lightning/issues/3014)!
It should not do this, but since you can set a manual range for mutual close, it's probably better to disable this option for close, as it's even more dangerous than documented.
Changelog-Changed: config/JSON: --ignore-fee-limits / setchannel ignorefeelimits no longer applies to mutual close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7242
Thought it's intuitive, it's not documented to, and LDK's current behavior makes it quite dangerous to do so.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is violate what the docs of getchainfo is telling us that the
last_block_height is a u64.
```
2024-05-17T09:53:23.464Z **BROKEN** plugin-offers: Got error reply to getchaininfo: '{\"error\":{\"code\":-1,\"data\":null,\"message\":\"invalid type: string \\\"0\\\", expected u64\"},\"id\":\"init/offers:getchaininfo#1\",\"jsonrpc\":\"2.0\"}\n\n'
2024-05-17T09:53:23.496Z INFO plugin-offers: Killing plugin: exited before replying to init
2024-05-17T09:53:23.496Z **BROKEN** plugin-offers: Plugin marked as important, shutting down lightningd!
2024-05-17T09:53:23.496Z DEBUG lightningd: io_break: lightningd_exit
2024-05-17T09:53:23.496Z **BROKEN** plugin-topology: Reading JSON input: Connection reset by peer
2024-05-17T09:53:23.504Z INFO plugin-topology: Killing plugin: exited before replying to init
2024-05-17T09:53:23.504Z **BROKEN** plugin-topology: Plugin marked as important, shutting down lightningd!
2024-05-17T09:53:23.504Z DEBUG lightningd: io_break: lightningd_exit
2024-05-17T09:53:23.504Z DEBUG plugin-bookkeeper: Setting up database at sqlite3://accounts.sqlite3
2024-05-17T09:53:23.504Z DEBUG connectd: REPLY WIRE_CONNECTD_START_SHUTDOWN_REPLY with 0 fds
2024-05-17T09:53:23.504Z DEBUG lightningd: io_break: connectd_start_shutdown_reply
```
Fixing the following crash
```
Got error reply to getchaininfo: '{"error":{"code":-1,"data":null,"message":"invalid type: string \"0\", expected u64"},"id":"init/offers:getchaininfo#1","jsonrpc":"2.0"}
'lightningd: lightningd already running? Error locking PID file: Resource temporarily unavailable
```
Fixes: 847208f5d8
Changelog-None: offer: fix type to the getchaininfo rpc
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This is on me, and the last height is optional, and not required,
because sometimes you do not want wait the sync of the blockchain
but just get the information of the current status.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Issue: rpc documentation on the readme server is not being updated.
It is an extension of PR #7326 to fix runtime error from github actions.
Changelog-None.
Issue: rpc documentation on the readme server is not being updated.
Updating the path from `doc/**.md` to `doc/schemas/lightning-*.json` for triggering the action because all `doc/*.7.md` files are fully generated by the `tools/fromschema.py` now.
Added manual trigger event for readme's documentation update.
Changelog-None.
Add a test that checks `get_route` and `uncertainty_update` behaviour's
handling of missing channel capacities.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
`bech32_charset_rev` is only 128 bytes in size but the `c < 0 || c > 128` check allows for `c` to be equal to 128 which would be out-of-bounds. Fix this off-by-one bug by changing the check to `c >= 128`.
```
plugins/renepay/mods.c: In function 'payment_continue':
plugins/renepay/mods.c:63:7: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
63 | (u64)payment_virtual_program[payment->exec_state++];
| ^
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now that our rpc documenation is generated from schema, we need to build
the docs before trying to update the rdme contents. The goal is to fix
the following issue with the rdme-rpc-sync workflow:
Run python .github/scripts/sync-rpc-cmds.py
lightning-addgossip lightning-addgossip.7.md
Traceback (most recent call last):
File .github/scripts/sync-rpc-cmds.py, line 92, in <module>
main()
File .github/scripts/sync-rpc-cmds.py, line 82, in main
with open(doc/ + file) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'doc/lightning-addgossip.7.md'
Currently make a plugin that do reportings of logs on
a services like graphana is not possible. So this commit
include the possibility to write a plugin that do the report
of this analisys.
Changelog-Added: core: notify plugins when a log line is emitted.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
When running `make check-python` we'll expclicitly include
`pyln-grpc-proto` in the `PYTHONPATH`.
This ensures we always run the tests using the version installed in
`./contrib/pyln-grpc-proto`
In rust enum are expected to be CamelCase.
However, we are generating enum's using CAPITAL_SNAKE_CASE.
This generates a warning and breaks CI.
See `cln_rpc::notifications::ConnectAddressType::LOCAL_SOCKET`
The schema in the docs for the `ConnectNotification` was faulty.
I've already fix this in https://github.com/ElementsProject/lightning/pull/7085
The new schema is
```
{
"connect" : {
"address" : {
"address" : "127.0.0.1",
"port" : 38012,
"type" : "ipv4"
},
"direction" : "in",
"id" : "022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59"
}
}
```
The `address` in the `connect` field will be encoded in protobuf as
`ConnectAddress`. However, this collides with the `ConnectAddress` that
is defined in the `connect` rpc-method.
This commit
- Updates the schema in `doc/schemas/notification/connect.json`
- Changes `msggen` to include an override to `PeerConnect` for any
notification typename that starts with `Connect`
Both have an `address` field which is a composite type. This results in
naming collisions
schema's: Updated schema for connect-notification
schema for connect notification + overrides
Override ConnectAddress to `PeerConnectAddress` in protobuf
for notifications
I'm working to expose a stream of notifications over grpc.
This requries me to define structs that can be used to request
a stream of notifications.
These structs are all empty.
**Problem Description**
In previous commits I introduced some new fields to `msggen`.
One example is `CustomMsgResponse` in `cln-grpc/src/notification.rs`.
```rust
pub struct CustomMsgResponse {
#[serde(skip_serializing_if = "Option::is_none")]
pub peer_id: Option<PublicKey>,
#[serde(skip_serializing_if = "Option::is_none")]
pub payload: Option<String>,
}
```
The `peer_id` and `payload` are required parameters.
However, the generated code is still marking them as `Optional`.
This is a choice made by `msggen`. It does this because `payload` and
`peer_id` are recently added fields. By marking the field as optional
the language bindings would also work when used on an older version of
Core-Lightning.
In this scenario. Marking them as optional is overkill.
The `CustomMsgStruct` and `payload` field are created in the same
version of CoreLightning.
This commit solves this behavior.
In the next commit I'll change the behavior of `OptionalPatch`.
The changes require me to have access to the `parent` of a field.
Splitting it up in a separate commit makes it easier to review.
You can run `msggen` against this version and the previous `version`.
I've tested it. It returns exactly the same output.
In Core Lightning notifications are JSON-messages. This commit
introduces structs that can be used to parse the notification
messages.
Using `msggen` all required tructs are automatically generated
This means we can see the values in listconfigs, even if we haven't set
them yet.
In particular, we now see the following:
* autoclean-cycle.value_int=3600
* bitcoin-rpcclienttimeout.value_int=60
* bitcoin-retry-timeout.value_int=60
* funder-max-their-funding.value_str=4294967295sat
* funder-per-channel-min.value_str=10000sat
* funder-reserve-tank.value_str=0sat
* funder-fund-probability.value_int=100
Changelog-Changed: plugins: libplugin now shows plugin option default values (where they're non-trivial)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. sql's dev-sqlfilename should be registered as a dev option.
2. bcli's timeout is an integer, not a string.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The spec has moved a bit here: the `outgoing_cltv_value` in the final onion
is basically the blockheight now (plus the 1 block delta we give ourselves).
Also, we were doubling ours, since p->min_final_cltv_expiry was already set
to p->blindedpay->cltv_expiry_delta above.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this, it was broken because our peer will no longer forward
via scids for private channels. We could use the scid alias, but the
node id is right at hand.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This demonstrates a number of issues reported by Carla: no surprise
since there was no test!
(We create a one-hop blinded path when we only have private channels).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently, anything which doesn't have a live channel is considered transient.
We free this first under stress, and also if they're still connecting.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our fetchinvoice always creates a reply path which terminates at their peer,
so we need a dev overrride for that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You can disable this with `fetchinvoice-noconnect`.
Changelog-EXPERIMENTAL: We will now reply to invoice_request messages even if reply path requires us to make an outgoing connection (LDK does this)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't find one searching from our random spot in the peer table,
we're supposed to wrap, not crash!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously, we only had a single python plugin `clnrest` and the instructions were embedded under other heading.
Adding a new heading for python plugins and their installation instructions will make it easier for users to locate them.
Changelog-None.
Changelog-Changed: hsmd: the hsmd now supports HSM_VERSION 6
This is actually optional, everything would be ok leaving native hsmd
support at HSM_VERSION 5 instead.
Changelog-None: channeld internal
This factoring makes it much clearer which callers only need the pubkey and
which only need the old_secret.
VLS has merged a workaround which prevents crashing when fetching a
per-commitment-point beyond the allowed range (the secret is just not
returned in this case.
https://gitlab.com/lightning-signer/validating-lightning-signer/-/merge_requests/643
In HSM_VERSION 6 the semantic is cleaned up; get_per_commitment_point
never returns a secret and safely be called on any commitment number.
The feerate security margin is a multiplicative factor applied to the
feerate of some transactions in order to guarantee that the
transaction remains publishable and has a sufficient chance of being
confirmed, that we can base some of our decisions on that.
The multiplicative factor is >=1 and was so far a constant 2. This
might have been sensible in the low-fee environment, where the fees
are expected to oscillate, and almost guaranteeing that we will
eventually have rising feerates but in high-fee environments that is
no longer the case, and the 100% margin that the multiplicator 2
brings is excessive. We therefore opt to start out with 100%, then
linearly interpolate up to a given maxfeerate (which does not have to
be a real feerate ever reached, it just indicates the feerate after
which we apply the constant 10% margin.
Fixes#6974Closes#6976
[Fixed up all the other changes required, including spendable calcualtion
comments and unit test and pytest tests --RR]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: we now understand blinded paths which use a short-channel-id(+direction) as entry point.
We don't actually support it yet, but this threads through the type change,
puts it in "decode" etc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And deprecate the --max-locktime-blocks which allows them to set it.
Hilariously, the spec misspells CLTV as CTLV at one point, so we work around it for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. onion-message
2. blinded-payments
3. route-blinding
4. channel-type
5. warnings.
Now they'll be checked correctly, and if the spec changes, we'll know
to reexamine this code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We expect:
UNUSUAL.*WARNING: we have 1 channels but can file descriptors limited to 65536
We get:
lightningd: WARNING: we have 1 channels but can file descriptors limited to 32768!
This is strange, since the first line is from Python's hard limit. Presumably something is restricting the fd limit of children
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They're cheap. The 2x channels heuristic is nice, but does assume
they restart every so often. If someone hits 64k connections I would
like to know anyway!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1024 is a common limit, and people are starting to hit that many channels, so we should increase it: twice the number of channels seems reasonable, though we only do this at restart time.
Changelog-Changed: lightningd: we now try to increase the number of file descriptors, if it's less than twice the number of channels at startup (and log if we cannot!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use a crude heuristic: if we were trying to contact them, it's a
"deliberate" connection, and should be preserved.
Changelog-Changed: connectd: prioritize peers with channels (and log!) if we run low on file descriptors.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I thought I was going to want to have a convenient way of counting
these, but it turns out unnecessary. Still, this is slightly more
efficient and simple, so I am including it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This can happen if we're totally out of fds, but previously we gave
no log message indicating this!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The `chainlag` is defined as the positive difference between the
height of the last block processed by the node and the best height
known by the bitcoin backend. The chainlag is positive when we are
still catching up with the blockchain, and `0` otherwise.
The `chainlag` is used as an additional offset to the CLTV values when
sending payments, allowing payments to be sent even before the chain
sync completes.
Listpeerchannels would update the local channel information setting the
liquidity in the outgoing channel to known_min=known_max=capacity,
when in fact it should be known_min=known_max=spendable.
The route-builder checks the liquidity bounds of each route one at a
time. Every route that satisfy the contraints is recorded in the
uncertainty network and produces an HTLC burden on the channels it uses,
so that the following routes cannot count on the same liquidity twice.
Routes contain only routing information and the payment they're linked
to can be obtained through the payment_hash. We remove the dependency of
route building routines from the payment itself. In order to make
plain payment information available we define a payment_info structure.
- use switch case over all possible WIRE_* errors,
- remove the virtual machine for routefail, use a simple two step
solution: 1. update the gossip and 2. handle error cases
Using enum renepay_errorcode simplifies the low level API of chan_extra and flow.
We can extract information about the nature of a function call failure
from its return value.
Routefail consist mainly of an API `routefail_start` that handles the
failure of a forwarding request (encoded in a route). Internally there
is a routefail datastructure that goes through a series of execution
steps, eg. updating the gossmap, updating the uncertainty network, etc.
Routebuilder constitute the module that builds the routes we try in the
payment. The public API is simply `get_routes`, it replaces the similar
interface for pay_flows.
Add a new implementation of the payment state machine.
This is based in the `pay` plugin concept of payment modifiers,
but here we take it to the next level.
The payment goes through a virtual machine that includes calling
functions and evaluating conditions.
This is fixing a bug that @chrisguida reported,
when we notify a plugin we also notify a plugin
that it is in the middle of initialization.
So imagine the following use case:
- 1. Plugin A sends get manifest
- 2. cln makes a check and generates a warning
- 3. Plugin A is subscribed to the warning or * notification
- 4. Core Lightning gets upset because it receives a warning message but it was waiting for a init.
I think it is still a bug in how Core Lightning handles the init, but also I think we should communicate with the plugin only if it is initialized.
```
lightningd-1 2024-04-23T23:20:34.154Z DEBUG plugin-clnrest: Notification: {'warning': {'level': 'warn', 'time': '1713914433.818230531', 'timestamp': '2024-04-23T23:20:33.818Z', 'source': 'plugin-bookkeeper', 'log': \"topic 'utxo_deposit' is not a known notification topic\"}}
lightningd-1 2024-04-23T23:20:34.154Z DEBUG hsmd: new_client: 0
lightningd-1 2024-04-23T23:20:34.154Z **BROKEN** plugin-test_libplugin: Did not receive 'init' yet, but got 'warning' instead
lightningd-1 2024-04-23T23:20:34.155Z INFO plugin-test_libplugin: Killing plugin: exited before we sent init
lightningd-1 2024-04-23T23:20:34.155Z **BROKEN** plugin-test_libplugin: Plugin marked as important, shutting down lightningd!
lightningd-1 2024-04-23T23:20:34.155Z DEBUG lightningd: io_break: lightningd_exit
lightningd-1 2024-04-23T23:20:34.155Z DEBUG lightningd: io_loop: connectd_init
Time-out: can't find [re.compile('Server started with public key')] in logs
{'github_repository': 'ElementsProject/lightning', 'github_sha': '014c1eb383b0a65394cf166b3aa0174cf2077896', 'github_ref': 'refs/pull/7258/merge', 'github_ref_name': 'HEAD', 'github_run_id': 8808124001, 'github_head_ref': 'cguida/onchain_notif', 'github_run_number': 9395, 'github_base_ref': 'master', 'github_run_attempt': '1', 'testname': 'test_self_disable', 'start_time': 1713914432, 'end_time': 1713914612, 'outcome': 'fail'}
----------------------------- Captured stderr call -----------------------------
Did not receive 'init' yet, but got 'warning' insteadlightningd: lightningd/connect_control.c:767: connectd_init: Assertion `ret == ld->connectd' failed.
lightningd: FATAL SIGNAL 6 (version 014c1eb-modded)
0x555b1dc2d6f3 send_backtrace
common/daemon.c:33
0x555b1dc2d8de crashdump
common/daemon.c:75
0x7fda4584251f ???
???:0
0x7fda458969fc ???
???:0
0x7fda45842475 ???
???:0
0x7fda458287f2 ???
???:0
0x7fda4582871a ???
???:0
0x7fda45839e95 ???
???:0
0x555b1db98f8e connectd_init
lightningd/connect_control.c:767
0x555b1dbb0307 main
lightningd/lightningd.c:1249
0x7fda45829d8f ???
???:0
0x7fda45829e3f ???
???:0
0x555b1db6fd64 ???
???:0
0xffffffffffffffff ???
???:0
```
Reported-by: @chrisguida
Co-Developed-by: @chrisguida
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
These are very noisy at the debug level:
DEBUG lightningd: Calling rpc_command hook of plugin clboss
DEBUG lightningd: Plugin clboss returned from rpc_command hook call
As seen in https://github.com/ZmnSCPxj/clboss/issues/194
Temporary patch to avoid issue #6973
[ Add much more logging! --RR ]
Changelog-Fixed: lightningd: avoid crash on signing failure when trying to spend anchor outputs.
This didn't trigger the bug, but worth explicitly testing: we spend a
to-remote output from a previous unilateral close, to spend an anchor
on a current unilateral close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`jq` is not available in the default repositories of the Ubuntu Focal image. To resolve this, adding the official jq package repository and then installing it.
Changelog-None.
We wire through --dev options into the hsmd, and test preapprove accept and deby
with both old and new protocols.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Apparently VLS actually does something when we preapprove: if caller is just
checking we want to tell it not to do that!
I put in a flag so we can test both old and new APIs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This should make VLS's life easier: they can ignore dev flags they
don't understand, but we will know their capabilites after init and so
know what they didn't understand (if required).
The only flag for now is a flag to force failure for "preapprove" calls.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we can check that the setconfig would actually parse. We do this
by handing NULL to the calback to indicate it's a test, which may not
work in general but works for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `check` `setconfig` now checks that the new config setting would be valid.
We need to pass through setconfig in check mode, then we need to have libplugin
add (and use!) a `check_only` parameter to all option setting.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` `setconfig` on plugin options can now check the config value would be accepted.
We don't thoroughly handle `check setconfig`: it would be good to
allow this to do further checking!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: Can now opt in to handle `check` command on their commands, for more thorough checking.
If this is set, instead of mangling the request in-place, we will hand
it through raw. Plugins will use this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to extend it to plugins, and we want it to be allowed to be async for more power,
so rather than not completing the cmd if we're checking, do it in command_check_done()
and call it.
This is cleaner than the special case we had before, and allows check to us all the
normal jsonrpc mechanisms, especially async requests (which we'll need if we want to
hand check requests to plugins!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
gcc-12 (Ubuntu 12.3.0-9ubuntu2) 12.3.0 with -O3 gives:
```
In file included from plugins/renepay/test/../mcf.c:5,
from plugins/renepay/test/run-arc.c:13:
ccan/ccan/tal/str/str.h: In function ‘minflow’:
ccan/ccan/tal/str/str.h:43:9: error: ‘errmsg’ may be used uninitialized [-Werror=maybe-uninitialized]
43 | tal_fmt_(ctx, TAL_LABEL(char, "[]"), __VA_ARGS__)
| ^~~~~~~~
plugins/renepay/test/../mcf.c:1565:15: note: ‘errmsg’ was declared here
1565 | char *errmsg;
| ^~~~~~
cc1: all warnings being treated as errors
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this, everything came out as level INFO.
Changelog-Fixed: pyln-client: Fix Plugin.notify_message() not to ignore `level` parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This test reproduces a bug in the `cln-plugin`-crate.
Core Lightning supports a wildcard `*` that plugins can use to
subscribe to all notifications.
However, `cln-plugin` does not support this case.
It allows the developer to subscribe `*`.
But the plug-in crashes when the first notification is received
Creates an example package that subscribes to all notifications and logs
them. This is useful for testing the behavior of subscribing to "*".
I've also edited the Makefile to ensure that `make` builds the example
and that `make clean` removes the example
Uncommited channels are missing several fields which would normally be
populated by an active channel. This simply skips them for the purposes
of finding a route. The particular culprit was:
{
"peer_id": "038cd9f3679d5b39bb2105978467918d549572de472f07dd729e37c7a6377d41d5",
"peer_connected": true,
"state": "OPENINGD",
"owner": "lightning_openingd",
"opener": "local",
"to_us_msat": 8317559000,
"total_msat": 8317559000,
"features": [
"option_static_remotekey",
"option_anchors_zero_fee_htlc_tx"
]
}
Fixes#7197 - SEGV in direct_pay_listpeerchannels when field private missing
Changelog-Fixed: Fixed crash in pay plugin caused by parsing uncommitted dual open channels
- Updated config params and plugin
- Updated documentation
Changelog-Added: Added a new configuration for clnrest plugin to change the default Swagger UI path from `/` to custom url.
- Updated config params and plugin
- Updated documentation
Changelog-Added: Added a new configuration for clnrest plugin to change the default Swagger UI path from `/` to custom url.
Changelog-Changed: Plugins: bcli: Add a path that tries to fetch blocks
from a peer if we can not find them locally.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
`getblockfrompeer` was introduced in v23.0.0, we want to skip this path
if the version of bitcoind used is below that.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
In the case that we have peers left in the peers array and that we could
not find the block yet, we ask the next peer for the desired block.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
We introduce the peers array that contains the known set of connected
peers for future reference in case that we couldn't find the block we
are looking for.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
We move the `geblock` call into a separate function. This allows us to
call if from various places in the future.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
This allows us to intercept getblock on a non 0 exit when the block was
not found. We fill this with something meaningull in future commits.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Not singular (though we used to have a listinvoice, removed in v0.6.1).
Reported-by: "Plant" via email
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than `allow_broken_log`, we have `broken_log` which is a regex
indicating what log lines are expected. This tightens our tests
significantly, as it will catch *unexpected* brokenness.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
an edge case like MultifundchannelChannel_idsChannel_type
was previously converted to MultifundchannelChannelIdschannelType
instead of the correct MultifundchannelChannelIdsChannelType
Apparently NixOS didn't have Python (sometimes!) so let's not assume.
By reusing the JSON "parsing" code from cowsay, we can self-disable
to handle this case.
Reported-by: Shahana Farooqui <shahana.farooqui@gmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `clnrest` now correctly self-disables if Python not present at all.
Changelog-Changed: Documentation: Merged `example_json_request` and `example_json_response` in a single `json_examples` array to maintain the request and its corresponding response together.
Tests whether we're able to route around a low-fee channel in the graph. We
should be able to find the more expensive route if the cheaper route is
depleted.
Changelog-Fixed: Plugins: pay now correctly estimates channel capacity
after payment failure, fixing some retry cases.
The `estimated_capacity` was properly substracted from the channel
hint, but adding the amount back did not work. The amount was added back
and then immediately substracted again. This caused the
`estimated_capacity` to ever decrease. This commit makes sure re-adding
the amount to the `estimated_capacity` works as expected.
is_local, ie. "is this side of the channel ours?" is not needed since we
can determine that predicate by evaluating
`scidd->dir == node_id_idx(self, peer)`
In the Min. Cost Flow solver we put a constraint on arcs capacities,
such that the total flow that can be allocated through a channel does
not exceed htlc_max. This solves two previous problem of the htlc_max
constraint at the MCF level.
Add spendable/receivable and is_local fields to the callback function
used in gossmods_from_listpeerchannels. This allows to do more fine
grained use of the listpeerchannels call.
The first use case is renepay, for which we need to ignore the htlc_max
of the local channels only.
Covered up by notleak() :(
Changelog-Fixed: lightingd: slow memory leak when using plugin hooks fixed (introduced in v23.11)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `fundchannel_start` now disallows a non-zero `mindepth` parameter if you ask for a zeroconf `channel_type`.
I was trying to debug test_zeroconf_open and getting very confused.
The reason: l0 is lightning-1, l1 is lightning-2, etc! And there are
two other tests where an l0 has been added at the front: fix them all
to avoid future confusion!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows for faster startup for Greenlight. We still require full sync
before *incoming* htlcs, and onchain operations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now *never* consider asking them anything, even if they are
capable (e.g. enabling full gossip stream). This aligns with
the latest spec proposal, where lack of `gossip_queries` means
"we don't have anything useful to say".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
LDK doesn't set this feature if they don't have any useful gossip (mobile nodes)
and it was agreed at the spec meeting that we should repurpose this feature
to mean "I don't have any useful gossip".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `createrune` new restriction `pinv` to examine bolt11/bolt12 invoice fields (e.g. amount of invoice).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugins: no longer allow missing `id` field in commando requests (deprecated v23.02, EOL v24.02)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These were deprecated in v22.08 (invoice hook) and v0.8 (htlc_accepted hook), and
marked EOL in v23.02.
*PLEASE* complain if this breaks things for you: it's kind of a test canary of
the deprecation system!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugins: `invoice_payment` and `htlc_accepted` hook `failure_code` response (derepcated v22.08 and v0.8, EOL v23.02)
Indeed, we now need to adjust our tests to use --i-promise-to-fix-broken-api-user
for the specific APIs past end-of-life.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were duplicating the command name (e.g. "autocleaninvoice.autocleaninvoice"), and also not
handling listconfigs if they specified the (currently unused!) i-promise-to-fix-broken-api-user
option, which we are going to use next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We tried to put *everything* into the "all" output, which didn't work
if there were other outputs!
Fixes: https://github.com/ElementsProject/lightning/issues/6664
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `multifundchannel` with `all` as an amount works as expected.
This line is a cosmetic off-by-one error. There are 7 languages defined, but because counting starts at 0, the display is "Select [0-7]" when in fact only 0-6 are valid choices.
It's a u64, we should pass by copy. This is a big sweeping change,
but mainly mechanical (change one, compile, fix breakage, repeat).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the convention everywhere else: allocation ctx comes first, any
other context comes second.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This has the benefit of being shorter, as well as more reliable (you
will get a link error if we can't print it, not a runtime one!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have various functions to convert to a string, rename them all so we can
count on fmt_X being the formatter for struct X, and make them all return
`char *`.
Sometimes they existed but were private, sometimes they had a
different name. Most take a pointer, but simple types pass by copy:
short_channel_id, amount_msat and amount_sat.
The following public functions changed:
1. psbt_to_b64 -> fmt_wally_psbt.
2. pubkey_to_hexstr -> fmt_pubkey.
3. short_channel_id_to_str -> fmt_short_channel_id (scid by copy now!)
4. fmt_signature -> fmt_secp256k1_ecdsa_signature
5. fmt_amount_sat/fmt_amount_msat pass copy not pointer, return non-const char *.
6. node_id_to_hexstr -> fmt_node_id
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
lightningd doesn't provide a formatter for `struct channel` (channeld does!), so
it will simply print "**BROKEN** UNKNOWN TYPE channel" for this case.
Fortunately, the logging is to the channel, so we know which one it's
talking about anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: runes: named parameters (e.g. `pnameamountmsat`) no longer need to remove underscores (i.e. `pnameamount_msat` now works as expected).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than speaking 'rune' we should speak english in error messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @ShahanaFarooqui
This reveals an inadequacy in our rune error reporting:
we complain a missing parameter is "not an integer field" instead
of "not present":
```
# Rune requires amount_msat < 10,000!
> with pytest.raises(RpcError, match='Not permitted: pnameamountmsat is not present') as exc_info:
E AssertionError: Regex pattern did not match.
E Regex: 'Not permitted: pnameamountmsat is not present'
E Input: "RPC call failed: method: checkrune, payload: {'nodeid': '0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518', 'rune': 'b3hXuEM7Pqzk-C7HUw83xzvHOV7fmuGaWjdo-wHdfg89MCZtZXRob2Q9cGF5JnBuYW1lYW1vdW50bXNhdDwxMDAwMA==', 'method': 'pay', 'params': {'bolt11': 'lnbcrt123n1pj7flqdsp5ndqgxpwk2hf50gzm0d4ssgjnd90cwkrc8udh7lfr5x583jms7yqspp5kn5stlnkv70celgw4vmdva9m7a57drd2403vnx4whq2p5nawkh3sdq5v3jhxcmjd9c8g6t0dccsxqyjw5qcqp99qxpqysgqhrgp7wp640gyujxk0mz4l6e6dxmqp7fz8pnnpnnqjfxg2scvuzfpwlxrj332u72p5g709eqr8rwaueruce84h0qmh6kc5c2zxgg9q4qps4cu8k'}}, error: {'code': 1502, 'message': 'Not permitted: pnameamountmsat is not an integer field'}"
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We do this for DEBUG_SUBD already, but I wanted to debug the main lightningd.
(We rename --debugger to the more accurate --dev-debug-self)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
a.k.a. "Pay with a friend!".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `pay` has a new parameter `partial_msat` to only pay part of an invoice (someone else presumably will pay the rest at the same time!)
Suggested-by: Calle
This PR will generate doc/lightning-*.md file for rpc commands. It will get the information from consolidated doc/schemas/lightning-*.json file and use it to generate markdown file for RPC commands.
Changelog-Changed: Documentation: great documentation rewrite, all reference pages now generated from the fully-tested JSON schemas and include examples.
- Updated doc/Makefile for generating schemas/lightning-sql.json
- Read the lightning-sql template from schemas/lightning-sql-template.json
- Generate sql tables data and merge it with json object read from above template
- Save final merged object in schemas/lightning-sql.json
- Updated doc/Makefile to auto generate lightning-sql.7.md and lightning-sql.7
- Deleted schemas/lightning-sql.json and adding it into .gitignore
- Added lightning-sql specific titles in the fromschema.py script
- Fixed test_sql by changing the sequence for listpeerchannels `reestablished` field
- Ignoring lightning-sql.json for msggen schema.json because it is auto generated by sql plugin and lightning-sql-template.json.
- Update `make doc-all` script
- `fromschema.py` script
- Updated to generate whole doc/lightning-*.md via doc/schemas/lightning-*.json file
- Updated to print request params
- Added `oneOfMany` condition for request params
- Added `pairedWith` condition for request params
- Added `dependentUpon` condition for request params
- Updated for pre and post_return_value for response
- Hiding `hidden` fields
- `descriptions` are array now
- Unified gaps between titles
- Added default key-value pair
- script: msggen, sql-schema, listconfig and pyln-testing script updates
This does not add created markdowns for cleaner review. We will add the markdown files in a separate commit.
Merge information from `*.request.json` & `*.schema.json`. Also consolidate remaining details from `*.md` files and create a single file in schemas folder.
This commit will remove parameter descriptions from RPC markdown but we will fix it in next commits by reading these descriptions directly from json.
- Removing parameter description text
- Adding/removing newlines for cleaner formatting
- Adding ERRORS title wherever needed
- Updating titles for consistency
- Adding resources links
Added descriptions for rpc command parameters
This also performs the following fixes:
1. delforward parameters are compulsory (required).
2. disableinvoicerequest request added `added` field.
3. invoice request order fixed (label then description, not vice-versa!).
4. listpeers log levels are a proper enum
5. description parameter documented for sendonion requests.
6. deprecatred amount_msat removed from sendpay request.
7. sendpay request partid type fixed to u64 (was u16!)
8. sendpay request localinvreqid type tightened to hash (was hex)
9. sendpay request payment_metadata and description fields documented.
10. sendpsbt request reserve type fixed to u32 (was boolean)
11. utxopsbt request satoshi type fixed to msat_or_all (was msat)
12. withdraw request parameter satoshi is compulsory (required)
13. fundchannel_start request amount is sat, not msat_or_all.
14. openchannel_init request amount is sat, not msat
15. openchannel_init close_to is a string, not hex.
16: invoice labels can be strings OR numbers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We actually use this in multiple places for requests. The key
difference between this and msat is what we do when presented with a
raw number! I prefer insisting on explicit suffixes, for this reason,
but at least this will document the field correctly!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In general, results should always be an object, so we can add new fields
later (e.g. warnings!). But we violated this for "stop", and when "recover"
used the same infrastructure, it started doing the same thing.
I'm assuming nobody cares, so we don't need to do a deprecation cycle.
Changelog-Changed: JSON-RPC: `stop` and `recover` now return a JSON object (not a raw string!) like every other command does.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The pattern of making tal-allocated copies of wally data to pass around
was made redundant after these calls were added by the use of
tal_wally_start/tal_wally_end to parent wally allocations. We can thus
just pass the data directly and avoid the allocations.
Removes redundant allocations when checking tx filters and computing fees.
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Avoids a hash160 for every key we derive to test. Callers that need the
key re-derive it without the skip flag, so there are no side effects
from this optimization.
Changelog-Changed: core: Processing blocks should now be faster
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Standardizes the is_xxx script function all take a script length, and changes
their first-level callers to pass it. This has several knock on benefits:
- We remove the repeated tal_count/tal_bytelen calls on the script, in
particular the redundant calls that result when we must check for multiple
types of script - which is almost all cases.
- We remove the dependency on the memory being tal-allocated (It is, in
all cases, but theres no reason we need to require that).
- We remove all cases where we create a copy of the script just to id it.
- We remove all allocations for non-interesting scripts while iterating block
txs in process_getfilteredblock_step1().
- We remove all allocations *including for potentially interesting scripts* in
topo_add_utxos().
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
The is_xxx script functions already perform these checks for us without
allocating. They currently expect their memory to be tal-allocated,
which is why a copy is made. However:
- The memory already *is* tal-allocated since wally is configured to do so.
- The tal-dependency is artifical and is removed in a future commit in
this PR.
This reverts commit c329756723.
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
In some weird situations it may happen that some channel along the route
could have htlcmax=htlcmin, so that the supremum of htlcmin and the
infimum of htlcmax are the same number. In that case there is only one
allowed amount that can go through that route.
Without this patch renepay would not handle correctly this cornercase.
Set up a simple line of channel pairs, where one should be preferred
over the other for our various reasons. Make sure this works, both
using a low-level call to Dijkstra and at a higher level as the pay
plugin does.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that run-route-infloop chose some worse-looking paths after
routing was fixed, eg the second node:
Before:
Destination node, success, probability, hops, fees, cltv, scid...
02b3aa1e4ed31be83cca4bd367b2c01e39502cb25e282a9b4520ad376a1ba0a01a,1,0.991856,2,1004,40,2572260x39x0/1,2131897x45x0/0
After:
Destination node, success, probability, hops, fees, cltv, scid...
02b3aa1e4ed31be83cca4bd367b2c01e39502cb25e282a9b4520ad376a1ba0a01a,1,0.954540,3,1046,46,2570715x21x0/1,2346882x26x14/1,2131897x45x0/0
This is because although the final costs don't reflect it, routing was taking
into account local channels, and 2572260x39x0/1 has a base fee of 2970.
There's an easy fix: when we the pay plugin creates localmods for our
gossip graph, add all local channels with delay and fees equal to 0.
We do the same thing in our unit test. This improves things across
the board:
Linear success probability (when found): min-max(mean +/- stddev)
Before: 0.487040-0.999543(0.952548+/-0.075)
After: 0.486985-0.999750(0.975978+/-0.053)
Hops:
Before: 1-5(2.98374+/-0.77)
After: 1-5(2.09593+/-0.63)
Fees:
Before: 0-50848(922.457+/-2.7e+03)
After: 0-50041(861.621+/-2.7e+03)
Delay (blocks):
Before: 0-196(65.8081+/-60)
After: 0-190(60.3285+/-60)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `pay` route algorithm doesn't bias against our own "expensive" channels any more.
The "path_score" callback was supposed to evaluate the *entire path*,
but that was counter-intuitive and opened the door to a cost function
bug which caused this path cost to be less than the closer path.
In particular, the capacity bias code didn't understand this at all.
1. Rename the function to `channel_score` and remove the "distance"
parameter (always "1" since you're supposed to be evaluating a
single hop).
2. Rename "cost" to the more specific "fee": "score" is our
actual cost function result (we avoid the word "cost" as it
may get confused with satoshi amounts).
3. For capacity biassing, we do want to know the amount, but
explicitly hand that as a separate parameter "total".
4. Fix a minor bug where total handed to scoring function previously
included channel fee (this is wrong: fee is paid before sending into
channel).
5. Remove the now-unused total_delay member from the dijkstra
struct.
Here are the results of our test now (routing 4194303 msat, which
didn't crash the old code, so we could compare). In both cases
we could find routes to 615 nodes:
Linear success probability (when found): min-max(mean +/- stddev)
Before: 0.484764-0.999750(0.9781+/-0.049)
After: 0.487040-0.999543(0.952548+/-0.075)
Hops:
Before: 1-5(2.13821+/-0.66)
After: 1-5(2.98374+/-0.77)
Fees:
Before: 0-50041(2173.75+/-5.3e+03)
After: 0-50848(922.457+/-2.7e+03)
Delay (blocks):
Before: 0-294(83.1642+/-68)
After: 0-196(65.8081+/-60)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7092
Changelog-Fixed: Plugins: `pay` would occasionally crash on routing.
Changelog-Fixed: Plugins: `pay` route algorithm fixed and refined to balance fees and capacity far better.
It gave 0. A lot.
Firstly, rmsat was often very small, because delays are often small. Much
smaller than the actual fee.
We really just want to offset the bias and multiply it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We attempted to introduce a "capacity bias" so we would penalize
channels where we were using a large amount of their total capacity.
Unfortunately, this was both naive, and applied wrongly:
-log((capmsat + 1 - amtmsat) / (capmsat + 1));
As an integer gives 0 up to about 65% capacity. We then applied this
by multiplying:
(cmsat * rmsat * bias) / (cmsat + rmsat + bias + 1);
Giving a total score of 0 in these cases! If we're using the
arithmetic mean we really need to add 1 to bias. We might as well
use a double the whole way through, for a slightly more fine-grained
approach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The amount is set not to crash by default, but run
"common/test/run-route-infloop 8388607" and you'll see a crash.
Sorry about the 7MB blob, but this testing was quite revealing and
I consider it worth adding.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Wrote a test program which passed num_channel_updates_rejected as NULL
(which we don't usually do), and valgrind complained:
```
==1048302== Conditional jump or move depends on uninitialised value(s)
==1048302== at 0x118B90: update_channel (gossmap.c:550)
==1048302== by 0x119EEE: map_catchup (gossmap.c:663)
==1048302== by 0x11A299: load_gossip_store (gossmap.c:726)
==1048302== by 0x11A352: gossmap_load (gossmap.c:1052)
==1048302== by 0x125362: main (run-route-infloop.c:90)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So far we would call `preapproveinvoice` once for each payment split,
i.e., at least once per HTLC, and potentially more often. There is no
point in doing so repeatedly, and especially in remote signer setup
this is actually counterproductive due to the additional roundtrips.
Changelog-Changed pay: Improved performance by removing repeated `preapproveinvoice` calls
We have installation instructions that tell the user to use `poetry`
and then we ourselves think we're clever and install only a known
subset? It was only a matter of time until we broke this.
Changelog-None
Thanks to amazing debugging assistance from grubles, we figured out
that indeed, my memory was correct: write and mmap are not consistent
on all platforms. The easiest fix is to disable mmap on OpenBSD for now:
the better fix is to do in-place updates using the mmap, and only rely
on write() for append (which always causes a remap anyway before it's accessed).
Fixes: https://github.com/ElementsProject/lightning/issues/7109
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This workflow has never had a particularly good signal-to-noise ratio,
and disabling it saves us a couple of GH Actions cycles, with only
minor reduction in test reach. We should rather rethink this and use
the installation instructions to write Dockerfiles for each described
architecture, and then have the CI test-build those Dockerfiles. That
would also cover the docs during testing, rather than having yet another
place where the instructions can bitrot away.
Changelog-None No user-visible change.
Increasing the min version of the hsmd due that we
added new code that required the hsmd to sign an announcements.
One of the solution is to increase the min version in this way
a signer like VLS fails directly during the init phase.
Link: https://github.com/ElementsProject/lightning/issues/7074
Changelog-None: hsmd: increase the min version
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Due to the API ratelimit, this allows cloning a github repo and searching
the result rather than searching via the REST API. If a source has
already been cloned, it is fetched and the default branch checked out.
Fixes a failure reported by @farscapian
Changelog-Fixed: Reckless no longer fails on github API ratelimit.
Github API calls are now ratelimited to 60 per hour. Searching through
the subdirectory contents of lightningd/plugins will hit this limit after
two searches or installs. The simple answer is to look for the directory
rather than verifying a valid entrypoint in a suitably-named directory
prior to cloning the repository.
Also corrects a bug that was flagging submodules as files while
populating directory contents.
We were extracting the output script for all outputs, and discarding
them immediately again if they were not P2WSH outputs which are the
ones of interest to us. This patch move the extraction until after we
have determined it is useful, and so we should save a couple thousand
`tal()` and `tal_free()` calls.
Changelog-Changed: lightningd: Speed up blocksync by not parsing unused parts of the transactions
Processing blocks is rather slow at the moment, but one thing we can
do, is to prevent copying all output scripts, when really all we are
interested in are the couple of outputs that are P2WSH.
This builds the foundation of that by adding a method to introspect
the script without having to clone it first, saving us some
allocations, and deallocations.
Changelog-Changed: core: Processing blocks should now be faster
You otherwise get thousands of these zero count Splice resume checks
spamming your logs while most of the time it is only informing you
of "nothing is happening".
Reduces log noise.
Signed-off-by: Warren Togami <wtogami@gmail.com>
If we delete it the first time a channel before it is closed, we will
crash when we try to move it again:
```
$ lightning_gossipd: gossip_store: get delete entry offset 47411992/51608943 (version v23.11-378-gac2a386-modded)
0x1002544b send_backtrace
common/daemon.c:33
0x1003415f status_failed
common/status.c:221
0x10016ef3 gossip_store_get_with_hdr
gossipd/gossip_store.c:466
0x10016faf check_msg_type
gossipd/gossip_store.c:491
0x1001722b gossip_store_set_flag
gossipd/gossip_store.c:509
0x1001752b gossip_store_del
gossipd/gossip_store.c:561
0x10017f5b remove_channel
gossipd/gossmap_manage.c:299
0x100181cf kill_spent_channel
gossipd/gossmap_manage.c:1144
0x1001a7df gossmap_manage_new_block
gossipd/gossmap_manage.c:1183
0x10014673 new_blockheight
gossipd/gossipd.c:483
0x10014d73 recv_req
gossipd/gossipd.c:594
```
Reported-by: @microsatosi on Discord
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Watchtowers changed the code so that we *always* have a channel->shutdown_scriptpubkey[LOCAL]
(see new_channel()). The previous code had several problems:
1. It tested this for NULL, unnecessarily.
2. It allowed overriding if it was a default, *even* if we were already using it.
3. If the peer opened without option_shutdown_anysegwit, but upgraded before we closed,
we would not recognize the default.
4. It set the final scriptpubkey (and other things!) even if the command failed.
Changelog-Fixed: JSON-RPC: `close` with `destination` works even if prior `destination` was rejected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Whenever there is a payment failure that requires gossip update, for
example changing the fee rates of remote channels, we call addgossip.
For renepay to consider this changes in the coming payment attempts, it
must update gossmap.
- previous pending sendpays must add up so that the plugin tries to pay
the rest of the amount,
- avoid groupid, partid collisions,
- add shadow fees if the option is set and the payment amount - total
delivering = 0
- add a test,
- also fix a buggy shadow routing test
If we accept a channel_update in state "NEED_SIGS" we should not set
refresh timer: we're simply holding it for the moment we get to that state
(which will happen as we mine the block).
```
0x7fd1cce39205 __assert_fail
./assert/assert.c:101
0x55c103cc6ee9 check_channel_gossip
lightningd/channel_gossip.c:128
0x55c103cc8a13 channel_gossip_update_from_gossipd
lightningd/channel_gossip.c:821
0x55c103cd752d handle_init_cupdate
lightningd/gossip_control.c:138
0x55c103cd79a3 gossip_msg
lightningd/gossip_control.c:190
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This code was trying to check that the address type is not one of the ADDR_TYPE_TOR*
types, but the is_toraddr() function checks a domain name! The cast should have been
a clue that this was wrong!
Anyway, wireaddr_to_addrinfo() aborts on these cases already, so the asserts here are
superfluous.
Found in unrelated CI run:
```
Valgrind error file: valgrind-errors.20610
==20610== Conditional jump or move depends on uninitialised value(s)
==20610== at 0x484ED28: strlen (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==20610== by 0x138FA3: is_toraddr (wireaddr.c:344)
==20610== by 0x11499B: conn_init (connectd.c:729)
==20610== by 0x28FD73: next_plan (io.c:59)
==20610== by 0x28FF94: io_new_conn_ (io.c:116)
==20610== by 0x11531B: try_connect_one_addr (connectd.c:927)
==20610== by 0x1182A8: try_connect_peer (connectd.c:1781)
==20610== by 0x11834E: connect_to_peer (connectd.c:1797)
==20610== by 0x119241: recv_req (connectd.c:2074)
==20610== by 0x12836F: handle_read (daemon_conn.c:35)
==20610== by 0x28FD73: next_plan (io.c:59)
==20610== by 0x2909A8: do_plan (io.c:407)
==20610==
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was triggered by the recover plugin tests (not yet merged!) and causes a crash
because we don't have signatures yet. It can only happen if we lost our database,
but at least don't crash!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We would sometimes propose fees which we couldn't afford, and thus the
peer would get upset: if we didn't recover it could cause force closes.
This was because we didn't take into account that pending htlcs will
remove our total available funds.
Changelog-Fixes: Protocol: Don't upset peers by sending `update_fee` with fees we cannot afford in the case where HTLCs are large.
This happens if:
1. The peer sets a timestamp filter to non-zero, and
2. We have a channel_announcement without a channel_update.
The timestamp is 0 as a placeholder as part of the recent gossip rework
(we used to hold these channel_announcement in memory, which was complex).
But this means we won't send it in this case, and if we later send the
channel_update, CI will complain about 'Bad gossip order'.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Normally, channels are marked dying, the 12 blocks later, removed.
But for local channels, we can access any spliced channel already, so
we remove them immediately from our local gossip. This left a hole in
our logic, if that channel was the last one keeping a
node_announcement alive.
Solution is to unify with the "moved node_announcement" path.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We make sure a node_announcement is preceeded by at least one channel_announcement,
but dying ones don't count (as they are not broadcast!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We accept node_announcements on dying channels, but make sure we
set the dying flag it channels are alll dying.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can update dying channels, though it seems weird! We accept gossip about them,
we just don't propagate it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids us gossiping about nodes which don't have live channels.
Interstingly, we previously tested that we *did* gossip such node
announcements, and now we fix that test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It prints a message to stderr, but actually it's fine with this version:
```
dump-gossipstore: UNKNOWN GOSSIP minor VERSION 14 (expected 12)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If the peer isn’t required to send signatures first but does while we are awaiting the next user RPC action — we should be caching the message and using it later.
Before we would leave the message cached in the socket itself, but tx_abort semantics require us to check the socket more often.
Add checking for and sending tx_abort to channeld.
When receiving it we first ACK it back, send a request to restart to lightningd, and then shutdown channeld.
We also must update the splice tests that relied on reconnect checks for splice warnings (as some are now tx_aborts instead).
In the old version requesting the config-value of an option
was a little bit tricky.
Let's say we want to have a plugin which uses a default
port of 1234.
```rust
let value = plugin.option("plugin-port");
match value {
Some(Value::Integer(_)) => {},
Some(Value::String(_)) => {}, // Can never happen
Some(Value::Boolean(_)) => {}, // Can never happen
None => {}, // Can never happen
}
```
Many users of the `cln_plugin` crate are overly cautious
and handle all these error scenario's. Which is completely unneeded.
Core Lightning will complain if you put a `String` where an `Integer` is
expected and will never send the value to the plug-in.
This change makes the API much more ergonomical and actually motivates
some of the changes in previous commits.
```
const MY_OPTION : ConfigOption<i64> = ConfigOption::new_i64_with_default(
"plugin-port',
1235,
"Description");
let value : Result<i64> = plugin.option(MY_OPTION);
```
The result will provide a proper error-message.
It is also safe to `unwrap` the result because it will
only be triggered if the user neglected to provide the
option to the `Builder`.
This is the first part of two commits that attempts to simplify
the API that is used to retrieve configuration values.
Previously, we encouraged the following pattern to build a plugin.
```rust
let configured_plugin =
Builder::new(
tokio::io::stdin(),
tokio::io::stdout())
.option(ConfigOption::new_i64_with_default("some-option", 0, "Description"))
.configure();
let value = configured_plugion.option("some-option");
match value {
Some(Integer(i)) => {}, // Config provided
None => {}, // No config set
_ => {}, // This should never happened
}
```
This commit helps to move to the following pattern
```rust
const SOME_OPTION : ConfigOption<i64> = ConfigOption::new_i64_with_default(
"some-option",
"description");
async fn main() -> Result<()> {
let plugin = Builder::new(tokio::io::stdin(), tokio::io::stdoout())
.option(SOME_OPTION)
.configure()
.await?;
let value : i64 = plugin.option(SOME_OPTION)?;
}
```
Breaking changes here.
This improves the semantics of `ConfigOption` and `options::Value`
drastically.
We've been using `options::Value` for 2 purposes
1. To specify the type and default value of an option
2. Coummunicate how a user configured an option
We fall here in the pit-fall of being poor at both purposes.
I've edited the code to ensure
- `options::Value` -> To read or specify the value of an option
- `option::ValueType` -> To specify the type of an option
**Configure an option**
Let's create an string-typed option create an option named `"required-string-opt"` the
developer has to use the following code.
```rust
let option = ConfigOption::new("required-string", Value::OptString, "description");
```
The semantics of `OptString` might falsely suggest that it is optional to specify the config.
However, we use `OptString` to say we are not providing a default-value.
After this commit we can use instead
```rust
let option = ConfigOption::new_str_no_default("required-string", "description");
```
For reading a configured value the `option::Value` is somewhat
cumbersome. The old version of had 6 different types of value
```rust
let value = plugin.option("required-string");
match value {
String(s) => {}, // User has configured string value or default
Integer(i) => {}, // User has configured int value or default
Boolean(b) => {}, // User has configured bool value or default
OptString => {}, // User has not configured value and no default
OptInteger = {}, // User has not configured value and no default
OptBOolean => {}, // User has not configured value and no default
}
```
This has been changed to
```rust
let value = plugin.option("required-string");
match value {
Some(String(s)) => {},
Some(Integer(i)) => {},
Some(Boolean(b)) => {},
None => {},
}
```
There are quite some things we want to generate from the schema
definitions, both inside and outside of CLN itself. By bundling the
schemas into the library we can make use of the tooling without
running in the CLN source tree. This is in part used to generate some
of the interfaces in Greenlight.
Changelog-None
Commit dac8964093 set aliases for channels at
creation time, but neglected to convert channels in the database. Do that now!
Fixes: #7039
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we try to reuse the same UTXO for the HTLC as the anchor, it will clash.
One block later we can spend the anchor change, all good.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We still want to test non-anchor channels, as we still support them, but
we've made it non-experimental. To test non-anchor channels, we
use dev-force-features: -23.
Changelog-Added: Protocol: `option_anchors_zero_fee_htlc_tx` enabled, no longer experimental.
Changelog-Changed: Config: `experimental-anchors` now does nothing (it's enabled by default).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'fixup!_options__make_anchors_enabled_by_default,_ignore_experimental-anchors.patch':
fixup! options: make anchors enabled by default, ignore experimental-anchors.
This simplifies our tests which will want to turn off anchors,
even though they won't be set for elements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to look for either other outputs which are sufficient for
reserve, *or* be able to create sufficient change to meet the
emergency reserve. Allow the sum of both to meet the requirements:
otherwise test_funder_contribution_limits can flake.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We tried to open a channel with feerate 0 in this case! Rework so it's
clear that we have two different feerates here.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The `amount` field is intended to be either a native unit (`msat`,
`sat` or `btc`) or it can also be a non-native unit that needs to be
converted into native when creating the invoice. As such limiting the
`amount` to only be native is very restrictive.
This uses pip to install to a venv when the dependencies are specified
with requirements.txt and poetry when it's present on the system and the
plugin has a pyproject.toml.
The directory structure will be:
reckless/
source/ (original plugin code here)
plugin_entrypoint (original entrypoint)
plugin_name (symlink or wrapper to activate venv)
.metadata (installation information)
.venv/ (python virtual environment)
The wrapper matches the naming of the original plugin entrypoint. The
shebang is modified to use the virtual environment's python binary,
then the original plugin is imported as a module and executed as though
it was run directly.
Changelog-Changed: reckless installs python plugins in virtual environments
The metadata includes an original retrieval source, timestamp, and commit
hash. This will allow a future update command to more easily evaluate the
status of the existing installation.
This creates a separate staging directory from the one used to clone
the source. A symlink is then added to the plugin's entrypoint which is
now in the source directory.
Christian points out that this makes the type harder, and it's just awkward.
Changelog-EXPERIMENTAL: JSON-RPC: Deprecated `offer` parameter `recurrence_base` with `@` prefix: use `recurrence_start_any_period`.
Changelog-EXPERIMENTAL: JSON-RPC: Added `offer` parameter `recurrence_start_any_period`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For better performance it is recommended to use the modern OpenSSL
EVP_MD_fetch API to load digest algorithms (i.e. explicit fetching),
instead of the older implicit fetching API.
As a side effect, using this API seems to avoid memory leaks with some
versions of OpenSSL.
We can get bad gossip if a node processes a gossip message after we've closed:
```
_________________________________________ ERROR at teardown of test_closing_specified_destination _________________________________________
...
> raise ValueError(str(errors))
E ValueError:
E Node errors:
E - lightningd-1: had warning messages
E - lightningd-4: had bad gossip messages
E Global errors:
...
lightningd-1 2024-02-03T00:29:02.299Z INFO 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 105x1x0
lightningd-1 2024-02-03T00:29:02.300Z DEBUG 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199-connectd: peer_in WIRE_WARNING
lightningd-1 2024-02-03T00:29:02.300Z INFO 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 103x1x0
lightningd-1 2024-02-03T00:29:02.339Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: peer_in WIRE_WARNING
lightningd-1 2024-02-03T00:29:02.339Z INFO 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 103x1x0
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of "new" and "load", we don't really need to "load" anything,
so do everything in gossip_store_new.
Have it do the compaction/rewrite, and collect the dying records
gossmap offsets are to the beginning of the message, whereas
the gossip_store uses the header offset. Convert the internals
of gossip_store to use gossmap-style uniformly, even where it's
a little less convenient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't use the dying flag, and we can manually append the
addendum rather than having gossip_store_add present a
bizarre interface.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The gossip_store_load is now basically a noop, since gossmap
does that.
gossipd removes a pile of routines dealing with messages,
in favor of just handing them to gossmap_manage.
The stub gossmap_manage constructor is removed entirely.
We simplified behaviour around channel_announcements with
no channel update: we now add them to the store, and go
back to fix the timestamp later. This changes a test,
which explicitly tests for the old behaviour.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a bit less optimal than before, where we had an ordered map of
channels and could easily serve "channels between scids 800000x and
900000x". We now iterate all of them.
The rest is fairly mechanical.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We add a temporary stub gossmap_manage constructor, which simply opens
the gossmap and doesn't do anything else.
Then seeker uses this, rather than routing.c, to probe.
We optimize our "get random node announcements" a bit by traversing a
random set of nodes directly, and seeing if we have no
node_announcement, then querying their first channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we've got more than 10000 pending channel_announcements,
complain and stop processing any more.
If this becomes a problem, we can limit individual peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's an unnecessary round-trip, and can cause us to complain in CI, in
the case where the channel has been closed by the time we ask.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
At initialization, gossipd is supposed to send all the local
channel_updates and any node_announcement it knows, so lightningd
doesn't generate fresh ones unnecessarily.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a fair amount of code, but much is taken from
the old routing.c, with the difference that this uses
common/gossmap instead of our own structures.
The interfaces are fairly clear:
1. gossmap_manage_new - allocator
2. gossmap_manage_channel_announcement
- handle new channel announcement msg
- if too early, keeps it in early map
- queues it, asks lightingd about UTXO.
3. gossmap_manage_handle_get_txout_reply
- handle response from lightningd for above.
4. gossmap_manage_channel_update
- handle channel_update message
- may have to wait on pending channel_announcement
5. gossmap_manage_node_announcement
- handle node_announcement msg
- may have to wait on pending channel_announcement
6. gossmap_manage_new_block
- see if early announces can now be processed.
7. gossmap_manage_channel_spent
- lightningd tells us UTXO is spent
- may prepare channel for closing in 12 blocks.
8. gossmap_manage_channel_dying
- gossip_store load tells us channel was spent earlier.
- like gossmap_manage_channel_spent, but maybe < 12.
9. gossmap_manage_get_gossmap
- gossmap accessor: seeker and queries will need this.
10. gossmap_manage_new_peer
- a new peer has connected, give them all our gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We weakened this progressively over time, and gossip v1.5 makes spam
impossible by protocol, so we can wait until then.
Removing this code simplifies things a great deal!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: we no longer ratelimit gossip messages by channel, making our code far simpler.
We never enabled it, because we seemed to be eliminating valid
channels. We discard zombie-marked records on loading.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Most nodes don't really care about exact timestamps on gossip filters,
so just keep a flag on whether we have anything in the gossip_store,
and use that to determine whether we ask peers for everything.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is easier for future callers, which don't have a convenient peer
structure: in particular, asynchronous processing of gossip for peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was an obscure dev command, as it never worked reliably.
It would be much easier to re-implement once this is done.
This turned out to reveal a tiny leak on
tests/test_gossip.py::test_gossip_store_load_amount_truncated where we
didn't immedately free chan_ann if it was dangling.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, allow callers to see unknown records we ignore (and let
them fail as a result), and get called if we can't pack a
channel_update into our internal format.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
gossip_store_del - takes a gossmap-style offset-of-msg not offset-of-hdr.
gossip_store_flag: set an arbitrary flag on a gossip_store hdr.
gossip_store_get_timestamp/gossip_store_set_timestamp: access gossip_store hdr.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The only way you'll see private channel_updates is if you put them
there yourself with localmods.
I also renamed the confusing gossmap_chan_capacity to gossmap_chan_has_capacity.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
update_count is just the count of the records for a tx. To calculate onchain fees
for an account we must sum all credits vs debits. We don't need to GROUP BY update_count
nor ORDER BY update_count since it is just a running index of updates to this tx.
Remove grouping by update_count which resulted in a crash due to bad arithmetic
caused by fee calculation returned rows not being consolidated.
Remove xfail.
If $BITCOIN_BIN is empty and bitcoin-cli or bitcoind are not found
startup_regtest will shutdown and display a corresponding error message.
Similarly if lightning-cli and lightningd are not found.
```
2024-02-02T10:30:38.6479143Z __________________ ERROR at teardown of test_multifunding_one __________________
...
2024-02-02T10:30:38.6491895Z > raise ValueError(str(errors))
2024-02-02T10:30:38.6492208Z E ValueError:
2024-02-02T10:30:38.6492465Z E Node errors:
2024-02-02T10:30:38.6492833Z E - lightningd-2: had bad gossip messages
2024-02-02T10:30:38.6493535Z E Global errors:
...
2024-02-02T10:30:38.7458545Z lightningd-2 2024-02-02T10:25:48.410Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: Ignoring future channel_announcment for 103x2x1 (current block 102)
2024-02-02T10:30:38.7460140Z lightningd-2 2024-02-02T10:25:48.411Z UNUSUAL lightningd: Bad gossip order: could not find channel 103x2x1 for peer's channel update
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Tacking the fuzz testing onto the PR testing not a good idea. It
causes completely unrelated tests to fail, and doesn't guarantee it
will find newly introduced ones. Running daily is likely better.
Changelog-None No user visible change
By default, UBSan reports runtime errors but does not stop execution.
We already abort in debug builds, and this commit makes us also abort in
regular builds when UBSan is enabled. Arguably, this is what users
expect when they enable UBSan, so it is a good default.
I know I've missed some UBSan bugs in the past because of this issue,
and dergoegge mentioned that this also happened to him.
Changelog-None: fixes hole in ([#7010]), see ([#7026])
Explicit hsmd_revoke_commitment_tx instead of get_per_commitment_point
needed when revokes are retried. The hsmd_revoke_commitment_tx message
is idempotent.
Changelog-Added: added a withdraw all to the end of test_onchain_their_unilateral_out to ensure that the unilateral close info is correct with anchors. Tests https://github.com/Blockstream/greenlight/issues/348
This caused a flake in test_gossip_lease_rates, since the peer would ignore
the node_announcement due to duplicate timestamps!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We mine blocks too fast, and l3 discard the channel_announcment as too far in the future:
```
lightningd-3 2023-12-14T06:40:04.744Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: Ignoring future channel_announcment for 103x1x1 (current block 102)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This isuseful to find completely dead channels which are worthy of
closing.
Changelog-Added: JSON-RPC: `listpeerchannels` field `last_stable_connection` showing when we last held an established channel for a minute or more.
Changelog-Added: JSON-RPC: `listclosedchannels` field `last_stable_connection` showing when we last held an established channel for a minute or more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a nice reflection of channel stability: in particular, worse case
ping time is 45 seconds, so we should have had some traffic.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is useful to see if we've really made progress, or just bounced
off the peer (e.g. it doesn't know the channel, or we have fallen
behind somehow).
Changelog-Added: JSON-RPC: `listpeerchannels` new field `reestablished` set once we've exchanged `channel_reestablish` messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, we were sending `announcement_signatures` before
`channel_reestablish`; we allow this because LND used to do it, but
it's not spec compliant.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It then waits 10 more seconds (for plugins to call setlease, especially)
before it will update a node_announcement.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than take the first two from peers with committed channels, use
the most common address given by at least 2 peers, and accept the majority
from non-committed peers if there are no committed peers.
This works well with the node_announcement rework, which waits until
everyone has a chance to connect before creating the node_announcement.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now lightningd just doesn't tell us about private channels, doesn't
expect us to generate channel gossip, and tells us about public channels
via the addgossip call, we don't need any of this in gossipd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This commit is a bit messy, but it tries to do the minimal switchover.
Some tests change, so those are included here.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Don't return false on db errors (we always fail on those), but return
false if they don't exist.
Also, add routine to clear them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before this it was channeld doing it, which was tied to a particular
channel. Create an API for lightningd to sign for any channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Mining 13 blocks to close the l1->l2 channel also causes the l2->l3
channel to confirm, and I was reliably getting the
channel_announcement to l3 in time to confuse this wait_for test. So
query the channel we want directly to make it more robust.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
After this, the invoice (correctly!) gives the zeroconf channel as a routehint,
so this test fails. Simple workaround: make invoice before creating zeroconf
channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was triggering on private updates after gossip changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1diff --git a/tests/test_gossip.py b/tests/test_gossip.py
index 677ec4825..285290b71 100644
When we rewrite, we actually use the remote channel_announcement signatures from the db, and notice that l1 here is actually l2:
```
0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: gossipd rejected our channel announcement: WARNING: Bad node_signature_1 304402204ad9f0ca9b5f1f8eec61290b6008bf2c15f011e35543d25c35c5b89d2e1a183402201aaca5883714dbe11d6ae00a146fac7302352e1b9e987eee5ce98e8eb6a82909 hash e713698e14d3d8f973210e8d93c96d1a7c117c58340ccb195785a672fd0b5ef1 on channel_announcement 01004ad9f0ca9b5f1f8eec61290b6008bf2c15f011e35543d25c35c5b89d2e1a18341aaca5883714dbe11d6ae00a146fac7302352e1b9e987eee5ce98e8eb6a829095c05509264fcf72e19180f0285cf0ca0d786f280a75d79e46e5627be20c9dd9a4e9105aa61d7faf3a375454db94376b9b9df29cf078130ed827836b0849f119014ec750ab163baef701e7889f430ee37ea7e38857aa2b2745440600999c85f2903abefde5210dd06ae12f24b8801416800db84f5c73d35594af594f9001800025dee15519d29eb81e9d1ac761deebdad91ba1d08a05c027c16ea5fc08f3a9b663f8433209392395f0107f6b4d266c46b0946bd098f88c0693817fb560e0526a8000006226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006900000100000266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c035180266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c0351802324266de8403b3ab157a09f1f784d587af61831c998c151bcc21bb74c2b2314b0206540493a98599967a61c45172df49681e398a6cb0890a506e74f76c2d2be242
```
Explicitly remove those messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- Documented `mindepth` and `reserve` in `fundchannel_start`. These
parameters are supported but had not been documented (yet).
- Removed `mindepth` from return-type. I'm unable to find any scenario
in which a value is returned.
- Documented the 312 error code
We had a complete copy of the `db_provider` and associated classes in
teh `tests/` directory, causing them to drift apart. Consolidating on
one version makes this more maintainable.
Changelog-None
For example, lnprototest got the error 'You gave bad parameters: Did not support channel_type ' which doesn't make it clear that it's rejecting the empty channel type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Let's tell the caller what channel_type they got!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel`, `multifundchannel`, `fundchannel_start` and `openchannel_init`: new field `channel_type`.
As suggested in https://github.com/lightning/bolts/pull/1092.
We still support channels opened without it, but you can no longer open new ones without it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `option_gossip_queries` is now required (advertized by all but 16 nodes)
We're about to make static_remotekey compulsory, but we still want to
do tests for pre-existing channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And add a request schema for multifundchannel.
Changelog-Added: JSON-RPC: `fundchannel` and `multifundchannel` now take an optional `channel_type` parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And add request schemas for openchannel_init and fundchannel_start.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel_start` and `openchannel_init` now take an optional `channel_type` parameter.
We use an array of bit numbers. We could use an array of names, but the JSON typing is then harder.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As suggested in https://github.com/lightning/bolts/pull/1092.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `option_gossip_queries` is now required (advertized by all but 11 nodes)
As suggested in https://github.com/lightning/bolts/pull/1092.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `option_data_loss_protect` is now required (advertized by all but 11 nodes)
It fails because this is an array of enum values, which is a
combination we hadn't seen before. Replacing this with a manually
managed enum, since it is likely we'll want to reuse it in the future.
Min. Cost Flow does not take into account fees when computing a flow
with liquidity constraints.
This is a work-around solution that reduces the amount on every route to
respect the liquidity bound. The deficity in the delivered amount is
solved by running MCF once again.
Changes:
1. the function `flow_complete` allocates amounts to send over the set of routes
computed by the MCF algorithm, but it does not allocate more than liquidity
bound of the route. For this reason `minflow` returns a set of routes that
satisfy the liquidity bounds but it is not guaranteed that the total payment
reaches the destination therefore there could a deficit in the delivery:
`deficit = amount_to_deliver - delivering`.
2. in the function `add_payflows` after `minflow` returns a set of routes we
call `flows_fit_amount` that tries to a allocate the `deficit` in the routes
that the MCF have computed.
3. if the resulting flows pass all payment constraints then we update
`amount_to_deliver = amount_to_deliver - delivering`, and the loop
repeats as long as `amount_to_deliver` is not zero.
In other words, the excess amount, beyond the liquidity bound,
in the routes is removed and then we try to allocate it
into known routes, otherwise we do a whole MCF again just for the
remaining amount.
Fixes issue #6599
We stopped doing empty journal logs, so we no longer need to switch
our log severity based on whether or not an account exists.
Should make bookkeeper less chatty and remove noisy logs
Changelog-None
We were putting out a lot of empty journal entries. Let's
stop doing that.
Now the wallet balance stays uninitialized until/unless you have
funds in it.
Fixes#5672
And we don't need to handle 0.9 lightningd which didn't include
allow-deprecated-apis in getmanifest call.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unfortunately, this is awkward: we just copy through most requests,
so we can't easily add a "deprecation" field to each one. So we do
a notification if the next command has a different deprecation status
than the global one, but that requires opt-in from the plugin.
We didn't previously document the subscriptions array, so do that now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `deprecated_oneshot` notifiction subscription to change deprecated status for a single command.
I did some CHANGELOG and git digging to see when these were deprecated, and
some were very old (v0.8.2!). But since they didn't warn users loudly, I
chose to do so this release only.
I renamed ld's `deprecated_apis` to `deprecated_ok` to make sure I
caught them all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Type-checking in Python is so loose you could already do this, but this updates the
mypy type annotations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We still accept boolean: the plugin may not want to commit to a deprecation schedule.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: rpcmethods and options can set `deprecated` to a pair of version strings, not just a boolean.
This command allows more fine-grained testing, without having to change the config of the
lightning node.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `deprecations` to enable/disable deprecated APIs from this caller.
In this case the cmd is `sql` but the field we're talking about is from
a different command, so we need a new libplugin API.
Note: there are still no deprecations in any tables used by `sql`, so this
is a bit moot for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows the user to specify the feature *by name*, and hopefully
complain to the developer to fix their code, knowing it will be removed entirely
in the next release!
Changelog-Added: config: `i-promise-to-fix-broken-api-user` allows for a one-release re-enablement of long-deprecated features.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we allow deprecation to be set per-connection, we need to
generalize this approach. Instead of filtering out deprecated
fields at schema loading, we need to load them, then refuse
to serve deprecated fields on a per-request basis.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`struct column` has a dynamically allocated member, which is neater if
it is a tal object itself (we allocated the member off the array, instead).
We're about to add two new members, so clean this up first.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Updating this every release was just busywork, and it turns out we don't actually
care: if something is marked deprecated we want to make it optional, whenever
it is for, and the only real test is if it was added before our lowest-supported
version we can consider it non-optional.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Each feature has a name, and says when deprecation begins and ends.
There's an API coming to allow you to re-enable on a per-feature basis
even if it's ended (as long as it's not been removed from the code ofc!).
Default end is 6 months after deprecation, i.e. we complain about it
at that point, if we can detect its use.
e.g, a standard deprecation in v24.05:
v24.02: allowed
v24.02 with mods: allowed
master after v24.02: allowed unless deprecated APIs disabled.
v24.05: allowed unless deprecated APIs disabled.
v24.08: allowed unless deprecated APIs disabled.
v24.11: allowed unless deprecated APIs disabled, but logs at BROKEN level.
v25.02: allowed only if --i-promise-to-fix-broken-api-user=FEATURE.
v25.05: code is actually removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Config: `disable-ip-discovery` (deprecated in v23.02): use `announce-addr-discovered`
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Experimental: Plugins: `funder` option "lease-fee-base-msat" removed (deprecated in v0.11, use "lease-fee-base-sat")
Seriously, it's taproot time, let's get rid of p2sh wrapped segwit.
Changelog-Removed: wallet: removal of p2sh-segwit addresses; newaddr won't issue them, we won't watch them for new funds (deprecated in *23.02*)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The `struct notification` lost type-safety, but avoided a redundant
string. The string is better, I think.
Since all notifications now contain an object of same name (some have
deprecated fields outside that), we can add helpers to do that, too.
Also, add some const (easy to do now we're typesafe!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Adds a "canonical" error handling in the module mcf.c,
adhering to the same convention we have adopted in flow.c.
So that when a function fails it signals the caller with an invalid
return value, or bool false, and it communicates an error message.
Changelog-Added: hsmd: Added hsmd_forget_channel to enable explicit channel deletion. ([#6987])
Motivation: Previously, a signer prematurely forgetting a channel led
to failures in unresolved channel requests. This update introduces
hsmd_forget_channel, allowing nodes to explicitly notify signers when
a channel is irrevocably resolved and can be safely forgotten. This
ensures synchronized channel cleanup between nodes and signers.
This change maintains backward and forward compatibility. Nodes
explicitly check whether a signer has `WIRE_HSMD_FORGET_CHANNEL`
capability before sending the message. Nodes without
`WIRE_HSMD_FORGET_CHANNEL` capability won't send this message. Signers
capable of handling this message but not receiving it will continue to
use conservative pruning methods.
Fixes#6987
This is good info to have, that I often forget and have to spend time
looking for. It'd be nice to have it documented in the project somewhere
so people new the project (or new to maintaining maybe?) know where to
find it.
This switches the logging implementation from using the `log`-facade
to using the `tracing-subscriber` instead. This allows us to also tap
into the tracing instrumentation if desired, which was not possible
with `log`.
Changelog-Changed cln-plugin: The logging adapter now uses tracing-subscriber allowing the `tracing` ecosystem to be used. No format changes.
The `cln-plugin` can be used to create a plugin that registers
additional rpc-methods. However, it doesn't allow to specify the `usage`
of the command.
This change makes it possible to specify the `usage`. It should not
contain any breaking changes.
I've opted to add a new method called `rpcmethod_from_builder` to the
`Builder` struct. This approach allows us to add new parameters in the
future.
Removes the tal_count checking overhead when iterating constant arrays.
Separated from the previous commit to make review easier.
Changelog-None
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
- Avoid overhead from tal checks when iterating block txs
- Skip pegin txs as well as coinbase txs while iterating
- Early-exit if the txout cannot possibly be p2wsh
- Don't re-calculate the txid when we already have it
- Don't allocate a script for non-policy asset outputs
- Don't copy txids for non-interesting UTXOs
Note the below -Changed line covers the previous wally and PSBT commits
which also provide general block processing speedups.
Changelog-Changed: core: Processing blocks should now be faster
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
There are many contexts in which it doesn't make sense to create a PSBT
wrapping the tx. There are also tx features that are not supported in
wallys PSBT implementation yet (such as pegins), which fail if a PSBT
wrapper is created, even though such features are unlikely to ever be
used in txs that cln will create/manipulate.
Allow these cases to be explicit that they only want the tx. This avoids
hitting such errors, is clearer on the caller side, and is more efficient.
Changelog-None
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Input cloning has not been exposed yet; I'll add that to wally in a
future release.
Changelog-None
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Wally release builds are significantly faster than debug builds. Plus we
pass down our build flags to libsecp, which means release builds have
been disabling the asm optimisations for both libraries.
Changelog-Changed: Enable optimizations for libwally/libsecp256k1-zkp
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
As of wally v1.0.0, Elements/Liquid support is enabled and part of the
library ABI by default.
Changelog-None
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Also removes incorrect/redundant configure flags when building.
Changelog-Changed: Update libwally to 1.0.0
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Rename the offending functions from wally_foo to cln_wally_foo.
For the sake of a minimal diff, only calls which conflict with wally
v1.0.0 have been changed. However it is bad form to use the wally_
function namespace; the remaining such calls should also be renamed.
Changelog-None
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
The method linkify just added links for an entry to a list and produced
duplicates if we got more than one changelog line in a PR. We now only
produce a link once per PR.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
I've broken the error-handling for `call` in a previous commit.
The key problem is that it failed to parse Json-RPC error's
that were returned by the server. It always returned a parse-
error instead.
To fix it I've adapted `call_raw_request`.
In the previous implementation local errors (e.g: Failing to find the
socket-file) where returned in a Result::Err. However, when the
rpc-server returned an error the data was encoded in the Result::Ok.
In this commit every error is returned as a `Result::Err` and
various methods and other calls have been edited to match this behavior.
I've also added additonal testing
One limitation of the `call_typed`-method was that it could not be used
with types defined outside of this crate. (See dependency on
`IntoRequest`)
This is useful for types that are not (yet) defined in this crate. A
possible case is an rpc-method that is defined in a plug-in that is
external to core-lightning or any method which isn't yet a part of the
`msggen`-script.
I've implemented a `TypedRequest` trait to make it work.
PS: This change is breaking. Users of `call_typed` must import
`cln_rpc::models::TypedRequest` instead of
`cln_rpc::models::IntoRequest`
The `cln::ClnRpc` plugin has a `call` and a `call`-typed method
which worked only on structs that are mentioned in
`src::primitives::Request`.
The consequence is that any rpc-method that is not (yet) defined in this
crate could not be used.
I've adapted the `ClnRpc`-method and create a low-level binding named
`call_raw`. All changes in this commit should be backward compatible.
I noticed this while working on a nix devShell to work on CLN. We are
blanked overriding the `pkg-config` search path, which can cause some
trouble. Specifically `nix` uses content addressable locations, and
macOS arm64 and x84_64 use separate `pkg-config` search paths, and by
overwriting it we can cause a mix of different architectures failing
the compilation
Changelog-Fixed: configure: We now respect the `PKG_CONFIG_PATH` environment variable
We used to use the zip archive, which comes with some baggage,
especially for some of the submodule-based dependencies. Using `git
clone` ensures that we have a clean snapshot, based on the latest
commit, and we can skip some of the wildcard operations on zip files.
Changelog-None
The integration with opentelemetry was sub-optimal: it was generating
jaeger-style traces, with short traceIds and we were considering the
entire lifetime as a single trace. This PR changes that to a trace for
startup and then a trace for any event that doesn't already have a
parent.
We also allow using the `CLN_TRACEPARENT` envvar to attach the startup
to a remote / external trace, potentially by whatever started the main
process. This is useful to see the startup trace in the wider context
of whatever tooling is built around it.
Changelog-Added: tracing: It is now possible to inject a parent for the startup trace by setting the `CLN_TRACEPARENT` envvar
As reported by @wtogami, LND nodes are using a default
min_final_cltv_expiry_delta of 9, which makes them unable to pay invoices
using the modern spec default of 18. Forcing inclusion of the c field
allows interoperability until broader support of the 18 block default.
Fixes: #6956
Changelog-Fixed: Default bolt11 invoices are payable by LND nodes.
Now _msat fields are all integers (last conversion 23.08) we can simply
leave them alone, rather than trying to convert them.
And for turning Millisatoshi into JSON, we simply globally replace the
default encoding function to try ".to_json()" on items, which allows
anything to be marshalled.
The global replacement was interfering with other uses of JSON, such
as the clnrest plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: pyln-client: no longer autoconverts _msat field to Millisatoshi class (leaves as ints).
Since the class interacts with int very well now, we don't have to
make explicit accesses, so it's easy to write code which works with
Millisatoshi or int.
Also, don't access rpc.decoder in one test, it's unnecessary.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using `pyln-testing` with `python3.11` results in the following warning.
```
.../venv/lib/python3.11/site-packages/cheroot/__init__.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
```
I've updated the `cheroot`-dependency.
However, the changes will not take effect immediately because we are
facing an internal conflict.
This conflict comes from the following requirements
- `//contrib/pyln-testing/pyproject.toml` requires `cheroot`. We can pick
the requried version
- `./external/lnprototest/pyproject.toml` requires `pyln="^0.12"` requires
`cheroot="^8"
It appears we have to do a multi-stage upgrade here.
**Step 1**
This commit configures `./contrib/pyln-testing/pyproject.toml`
to require `cheroot=">=8 <=10`. This allows users of `pyln-testing`
to start using a new release of `cheroot` immediately.
However, we still require `cheroot="^8"` because of `lnprototests`.
Even after running `poetry install` in the project root the warning
will remain.
**Step 2**
Not a part of this commit.
Publish a new release of `pyln-testing` on PyPI.
Once this release is finished we can update
`./external/lnprototests/pyproject.toml` to use the new
version of `pyln-testing` and the warning will disappear.
Create a notification that is triggered when a `costummsg` is received.
Changelog-Added: Plugins: notification custommsg for receiving an unknown protocol message
On July 18th, @jgriffiths wrote:
> You need to set this to NULL after freeing it, otherwise if line 72 returns you have a dangling pointer and potential later use-after-free here. Alternately use wally_psbt_set_input_final_witness(NULL) which will free any existing witness and set the value to NULL.
Reported-By: @jgriffiths
In the original [$6173] pr `bind` was a typo for `bind-addr`.
`bind` never existed, so this commit updates uses of `bind` to `bind-addr`.
This links lightningd-config.5.md to configuration.md since `bind-addr` has options of interest such as ipv4/ipv6.
Turns out that if we publish from PRs we may end up with versioning
collisions, causing errors. This publishes only from `master` where we
have a linearizable history, and should therefore get unique versions.
Changelog-None
When the value of a `dynamic` option is updated core Lightning
will make a JSON-RPC request to the plugin.
The documentation falsely mentioned the request has a method named
`setvalue. It appears it is actually called `setconfig`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: `listchannels` no longer uses local knowledge to set `active` to false if disconnected.
This alters a few remaining tests, as well.
[ Inclused even more test fixes from Alex Myers <alex@endothermic.dev>! ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `listchannels` listing private channels: use listpeerchannels
This breaks our tests a bit, which assumed we can always see ourselves
even if we don't have a proper channel.
Usually we would deprecate this first, but it's unlikely to break
anyone since it's a bit obscure that this worked at all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `listnodes` no longer shows private (peer) nodes: use listpeers
We temporarily use a second gossmap so we can just switch private info off
for listincoming and not listchannels.
Note that listchannels now uses the local alias (if no scid), so we have
to change that in the routehint caller.
Since we now *always* use a channel alias in hints if one exists, a
test broke, so fix that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
l2 gave us a routehint, but it should have seen l1 as a dead-end. It
didn't due to the presence of a channel alias, which was a bug. We're
about to fix this, which breaks the test.
Add a dummy node off l1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We will get localmods from gossmods_from_listpeerchannels in the next commit,
so we need to save routehints to add to that later.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have multiple different routines here, and we want to wean them off private
gossip one at a time. This converts `getroute`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is more thorough than the minimal one required for getroute(), including the feerates
and cltv deltas.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is redundant if it's a public channel, but vital if it's not. Publishing unconditionally makes
it easier for gossmap: we create a local modification all the time, even if redundant (and we can
have the actual capacity ceiling accurate in this case, since we know it for local channels).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listpeerchannels` now shows gossip update contents (even if channel unannounced).
Not just when it's a private channel. This is useful for listpeerchannels in the next patch.
Most of this is renaming.
It also means that source can be NULL, so move it out of the struct and put it in the message,
where it logically belongs, and make it an optional field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
and stash in the database.
Rusty: I added the bad gossip message so we would see unknown updates in CI, and made sure we don't send our own generated updates to lightningd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CI revealed one:
```
cc plugins/libplugin-pay.c
plugins/libplugin-pay.c: In function ‘payment_getroute’:
plugins/libplugin-pay.c:888:17: error: ‘errstr’ may be used uninitialized [-Werror=maybe-uninitialized]
888 | payment_fail(p, "%s", errstr);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
plugins/libplugin-pay.c:851:21: note: ‘errstr’ was declared here
851 | const char *errstr;
| ^~~~~~
cc1: all warnings being treated as errors
```
My local compiler gave another:
```
channeld/channeld.c: In function ‘resume_splice_negotiation’:
channeld/channeld.c:3734:23: error: ‘final_tx’ may be used uninitialized [-Werror=maybe-uninitialized]
3734 | msg = towire_channeld_splice_confirmed_signed(tmpctx, final_tx,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3735 | chan_output_index);
| ~~~~~~~~~~~~~~~~~~
channeld/channeld.c:3461:28: note: ‘final_tx’ was declared here
3461 | struct bitcoin_tx *final_tx;
| ^~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:298: channeld/channeld.o] Error 1
```
So fix both.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The data-format changed in 2019.
The field `amount_msat` was renamed to `msat`
(See 2907e430d5)
The data-format changed again in november 2023.
The content of the field `amount_msat` was changed.
It was a string `"1000msat"` and became a number `1000`.
(See 0b23133ab2)
The current documentation still mentions the data-format before
2907e430d5.
I've updated the docs to reflect the latest changes.
I've also incluced a short mention of 0b23133ab2 to help help
developers create plugins that work both pre 23.11 and post 23.11.
Turns out we were sending feerate updates to daemons that do not
understand it. Don't do that!
Closes#6932
Changelog-Fixed: channeld: We could crash `closingd` by sending it a `channeld` message
This allows use of clnrest + websocket in the regtest environment.
If clnrest starts up, a rune is generated and provided along with
the connection info.
Also uses the snap installed bitcoind path if appropriate.
Format of the start_ln output is:
Commands:
l1-cli, l1-log,
l2-cli, l2-log,
bt-cli, stop_ln, fund_nodes
Node Info:
l1 rest: https://127.0.0.1:3110 rune: "lgkWSp0PQK-pkbElLpjcNVQnX7yfEjouJaJHjLuW8w89OA=="
l2 rest: https://127.0.0.1:3111 rune: "sUqGqkevGYG2r1e_JUiz8Me00GhtTv5-IuGk4o9Beyc9OQ=="
Changelog-None
We were (dumbly?) using the `onchain_hrp` for 'chain_mvts' and the
`lightning_hrp` for 'channel_mvts'.
This works fine everywhere *except* for on a signet, where we use
different prefixes.
Since the lightning-hrp set is more diversified (testnet btc
+ signet btc use the same HRP 'onchain'), let's use that.
Should have zero impact on anything other than nodes running on signet.
To preserve your current accounts database without needing to delete,
restart, execute the following: (note preferrably when your node isn't
running).
```
UPDATE chain_events SET currency = 'tbs' WHERE currency = 'tb';
```
Fixes#6534
Changelog-Fixed: `bkpr-listbalances` would crash for nodes on signet with payments in channels, because onchain events were using a different currency than inchannel events.
Now that 'developer' isn't a compile time flag, let's always use it
when using the startup regtest nodes.
Should always make gossip + polling bitcoind fast
We were using similar variables for different things (directories vs binary).
Update this to separate them out, while also adding ability to use a different directory
for lightning nodes
Changelog-Changed: startup_regtest.sh PATH_TO_LIGHTNING + PATH_TO_BITCOIN are no more. Use LIGHTNING_BIN and BITCOIN_DIR
We are tracking the flakiness of tests [here][1], so no need to suffer
through fail-looping CI while trying to fix a PR or get it merged.
Changelog-None
[1]: http://35.239.136.52:3170/run
We used to run the publication on each PR, which could cause conflicts
due to the naming convention being based on the commit history length,
and multiple PRs potentially hitting the same name.
Adding a `workflow_dispatch` trigger allows us to still test the
workflow in PRs but only do so manually (when we're watching anyway).
Changelog-None
This caused a segmentation fault when calling `getcodexsecret` without
id.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Changelog-Fixed: Hsmtool: Fix segmentation fault when calling `getcodexsecret` without id.
This fixes a crash on startup of core-lightning where gevent could not
be imported. This happens before sys is imported and throws us into the
except clause which calls sys.
By importing it explicitly in the except clause we are not dependend of
the order of imports in the try bracket.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Changelog-Fixes: Plugin: clnrest crashed on startup when gevent was
missing.
Reported-by: Shahana Farooqui
Changelog-Fixed: JSON-RPC: Plugin notification `msat` fields in `invoice_payment` and `invoice_created` hooks now a number, not a string with "msat" suffix.
Changelog-Fixed: JSON-RPC: Plugin hook `payment` `msat` field is now a number, not a string with "msat" suffix.
Adds tests for when the connection fails during
1) splice tx_signature
2) splice commitment_signed
Fleshed out the reestablish flow for these two cases and implemented the fixes to make these reestablish flows work.
Part of this work required changing commit process for splices: Now we send a single commit_part for the splice where previously we sent all commits, and accordingly, we no longer revoke in response.
Changelog-Fixed: Implemented splicing restart logic for tx_signature and commitment_signed. Splice commitments are reworked in a manner incompatible with the last version.
Since these worked in v23.08, we can't just rename them. So if they are
used and unclaimed, we should rename them internally (if they're claimed,
it's probably clightning-rest, and we should *NOT* touch them!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `clnrest` parameters `rest-port`, `rest-protocol`, `rest-host` and `rest-certs`: prefix `cln` to them
Add a version bump for clnrest and the main poetry.lock to the makefile
and update the release checklist to include the new target.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
This will allow users to use clnrest with c-lightning-REST without conflicts.
It was required for applications to have enough time for migrating from c-lightning-REST to clnrest.
Changelog-Changed:
config option `rest-certs` changed to `clnrest-certs`
config option `rest-protocol` changed to `clnrest-protocol`
config option `rest-host` changed to `clnrest-host`
config option `rest-port` changed to `clnrest-port`
config option `rest-cors-origins` changed to `clnrest-cors-origins`
config option `rest-csp` changed to `clnrest-csp`
We have a global JSON encoder hack, which means that any field ending in msat gets special treatment (so we can safely talk to lightningd, even if a field expects satoshi amounts, we are explicit). However, requests uses the JSON parser and DOES NOT want this conversion when sending it out as an HTTP response!
The simplest local fix we could find was Shahana's suggestion to iterate and covert away from Millisatoshi(): the reverse of what our JSON encoder does.
Fixes: https://github.com/ElementsProject/lightning/issues/6848
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It already throws an exception or error, or decodes the JSON response:
we can simply pass it through.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- cln-grpc certificate reuse
- new certificate generation
- `GET` and `POST` requests
- websocket server
- config options and HTTP headers
Link to #6436.
We need to upgrade the rust version of the ubuntu docker images as
time-core v0.1.2 needs a version newer than 1.67.0
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
The fedora image installs a setuptools version that can not be upgraded
via pip. A fix is to remove the files manually and reinstall it via pip.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Code using `CI_SERVER` was added recently to track something. It
attempts to short-cut if `CI_SERVER` is not set, but on gitlab it
is set to `yes`. https://docs.gitlab.com/ee/ci/variables/
Instead use `CI_SERVER_URL`, maybe that is what is intended?
If we disconnect, we lose the open_attempt record. Which is fine, but we
should prevent the user from starting another RBF if the last one isn't
done yet!
Originally the accepter waited for the peer to send us their commitment
sigs before we send ours; this changes things so that the accepter
sends their commitment sigs ASAP.
This test fails: when cln is not the channel initiator, it waits for the other node to send commit_sig before sending its own commit_sig. There is no reason to do that, both nodes should send commit_sig immediately after exchanging tx_complete? Otherwise it's a missed opportunity to finalize the channel creation on reconnection, because in that case cln hasn't saved the channel and fails it on reconnection.
Reported-By: @t-bast
When we got our peer's sigs, if we were the remote, we would re-notify
the plugin, which in turn would re-send the tx-sigs to use.
In the case of CLN, we'd then
- break, because we'd re-forward the sigs to the `openchannel` plugin,
which was then in the wrong state (MULTIFUNDCHANNEL_SIGNED)
spenderp: plugins/spender/openchannel.c:598: json_peer_sigs: Assertion `dest->state == MULTIFUNDCHANNEL_SECURED' failed.
spenderp: FATAL SIGNAL 6 (version 5880d59-modded)
In the case of eclair, they'd just see our 2nd TX_SIGS message and
@t-bast would complain:
> This test works, with one minor issue: on reconnection, cln sends its tx_signatures twice (duplicate?).
This commit does two things:
- has the openchannel / spender plugin log a broken instead of
crashing when the state is not what we're expecting
- stops us from calling the `funder` plugin if this is a
replay/second receipt of commit-sigs.
Here we conform to the specification, which requires that we handle
next-funding-id in a specific way.
Note that we were already sending it, but now we actually correctly
handle its presence.
Changelog-Changed: Spec: dual-funding now follows the next-funding-id rules.
In the case where you're echoing back a tx-abort, just let it through.
Not doing this causes problems in the case where your node has forgotten
about an in-progress open.
This fixes the following problem:
- you send a tx-abort (even tho you have marked tx-sigs as received)
- peer echos it back (we echo back tx-aborts always)
- you throw an error because you're already in a tx-abort unallowed
state
In this commit, we allow for echos to come thru no matter our current state and
this fixes things/makes them work as expected.
If you get the right series of disconnects, it's possible for your peer
to send you a tx-sigs even though the current state of the channel open
is that you've seen the funding open on chain (your channel_ready[LOCAL]
= true)
In this case, if we haven't marked that we've seen the tx sigs yet,
we go ahead and mark them as seen and just ignore this tx-sigs msg.
If we get a commitment-signed message from a peer, outside of a normal
flow, process it!
We're about to send these during reconnect, so we need to be able to
handle them!
We're going to need to reuse this for reconnect; make the method
standalone in that it can figure out what to send to HSMD independent of
where it's located in the setup call flow.
We need to keep track of if we've gotten the last negotiation's
commitment sigs, for reconnect logic (helps us know what messages to
send in the reconnect case)
If an openchannel_update fails (due to disconnect etc) it's possible
that it could 'resolve' itself later due to the auto reconnect logic
If you call an openchannel_update and we've already got an inflight
record saved, go ahead and return the info from the inflight (including
info about whether or not the commitments are secured.)
This makes openchannel_update a bit more 'robust'/idempotent, in that
you can make repeat calls to it after the channel is inflight and get
the info you need back to continue (call openchannel_signed)
Changelog-Changed: RPC: `openchannel_update` will now echo back a result if there's a matching inflight record for this open.
Since we can now get a COMMITMENT_SIGNED message due to a reconnect,
in addition to the 'inline' open process, it's possible that we might
have cleaned up / lost the open_attempt object.
This is fine, we have (almost) all the data we need to round this off
successfully/send out a notice.
Note that the only exception is the `close_to` data is lost/forgotten in
the case of a restart; this is largely fine.
If the peer's disconnected but the caller sends us valid sigs for the
channel open, we should go ahead and store them to disk before we reject
the call based on the fact that the peer is disconnected.
This way if the peer reconnects later, the channel open will succeed
Changelog-Changed: RPC: `openchannel_signed` will now remember the details of a signed PSBT even if the peer is disconnected.
When we reconnect, if we get a note from the peer that they dont know
about a pending inflight, we need to be able to clean it up so we can
restart/re-negotiate a new RBF etc.
This adds a cleanup method to remove any inflights for a channel without
a last_tx (commitment tx)
We don't actually use this internal to this method? Weird.
Anyway, if we don't want/need it allow the caller to signal that by
passing in NULL, if desired.
Here, we split up what was "commit_received" into two phases:
- commit-ready, where we're about to send our commitment tx to
peer
- commit-received, when we've gotten the commitment tx from our
peer
This lets us do the right thing (as far as the spec is concerned) with
returning the correct 'next_funding_txid' on reconnect (later commits).
From the spec:
Once peers are ready to exchange commitment signatures, they must remember
the details of the funding transaction to allow resuming the signatures
exchange if a disconnection happens.
Basically this means we add channels to the database before we've gotten
commitments for them; it's nice that there's now a state for commitments
recevied but we now save the channel prior to that.
This commit makes it possible to track the pre-commit-rcvd but not quite
open-init state.
"Patrick, I'm sorry I doubted you."
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Config: `large-channels` is now the default, wumbology for all.
Adding a fee offset as the channel opener reduces the likelihood of a
disconnect by the peer do to slight variation in feerate calculation
between nodes.
Changelog-Fixed: Some peer disconnects due to update_fee disagreements are avoided.
We want to use this for boosting txs: either attaching fees to
zero-fee HTLCs, or making anchor transactions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- certificate generation
- config options validation
- log level from 'error' to 'info'
- sending method as None instead ""
- added `listclnrest-notifications` for websocket server rune method
Changelog-Fixed: websocket server notifications are available with
restriction of `readonly` runes
startup_regtest now checks if wallet directory exists before creating
the default wallet, then it will check if wallet is loaded before
attempting to load it. This prevents unnecessary errors during the
execution of this bash script.
[ Squashed fixup: `regtest: restore double quotes on PATH_TO_BITCOIN` -- RR ]
If we ever re-enabled a channel too fast, if we considered it spam it
wouldn't propagate. For the moment, consider our own updates to never
be spam.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This should only have an effect if someone queries, but in practice, peers actually
see the disabling of channels.
This is a workaround until we rework the code so gossipd doesn't generate these at all
any more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This functionality already exists in the Python framework; this feature
enables it for Rust plugins as well.
Changelog-Added: cln-plugin: Implement send_custom_notification to allow sending custom notifications to other plugins.
While running the integration testing in VLS we noted that there is a problem with the old_secret during that revoke and ack.
The built-in signer of core lightning return always the secret, but with a signer with more strict policies this can not be true.
In fact, the VLS return the old_secret only if it is the last tx candidate. So we should keep track of the old_secret during the recursion.
The current core is unsage because we can only revoke transaction M-1 once we have transaction M signed by the counterparty. If there is a splice candidate that is unsigned yet, and we revoke commitment transaction M-1 for it, and it gets into the blockchain, and the peer goes away, we can't force close anymore.
lightningd-1 2023-10-25T15:24:25.151Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: (null):0 ((null)) 0xffffffffffffffff
lightningd-1 2023-10-25T15:24:25.151Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: FATAL SIGNAL (version v23.08-64-gbffe599)
lightningd-1 2023-10-25T15:24:25.151Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: common/daemon.c:38 (send_backtrace) 0x55c4a2ebd97b
lightningd-1 2023-10-25T15:24:25.151Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: common/status.c:221 (status_failed) 0x55c4a2ecf0ae
lightningd-1 2023-10-25T15:24:25.151Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: common/subdaemon.c:18 (status_backtrace_exit) 0x55c4a2ecf42b
lightningd-1 2023-10-25T15:24:25.151Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: common/daemon.c:78 (crashdump) 0x55c4a2ebdb16
lightningd-1 2023-10-25T15:24:25.151Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x7f084e44251f
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317 ((null)) 0x7f084e5a09cd
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: wire/towire.c:17 (towire) 0x55c4a2ed610e
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: bitcoin/privkey.c:33 (towire_secret) 0x55c4a2eea03b
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: wire/peer_wiregen.c:2737 (towire_revoke_and_ack) 0x55c4a2ee1d3d
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:1850 (make_revocation_msg_from_secret) 0x55c4a2e9d80e
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:1931 (send_revocation) 0x55c4a2e9de7a
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:2249 (handle_peer_commit_sig) 0x55c4a2e9f175
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:2880 (interactive_send_commitments) 0x55c4a2ea0c2e
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:3937 (splice_initiator_user_finalized) 0x55c4a2ea40ce
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:4011 (splice_initiator_user_update) 0x55c4a2ea44d4
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:5756 (req_in) 0x55c4a2ea8d9d
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: channeld/channeld.c:6151 (main) 0x55c4a2eaa3f8
lightningd-1 2023-10-25T15:24:25.152Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x7f084e429d8f
lightningd-1 2023-10-25T15:24:25.153Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: ../csu/libc-start.c:392 (__libc_start_main_impl) 0x7f084e429e3f
lightningd-1 2023-10-25T15:24:25.153Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: (null):0 ((null)) 0x55c4a2e98734
lightningd-1 2023-10-25T15:24:25.153Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: backtrace: (null):0 ((null)) 0xffffffffffffffff
lightningd-1 2023-10-25T15:24:25.153Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: STATUS_FAIL_INTERNAL_ERROR: FATAL SIGNAL
Reported-by: devrandom
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This table doesn't have `id`, except as the implicit one in Sqlite3,
so we need to add it for postgres.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
peer_htlcs has become a bit of a dumping ground: move listforwards
etc to its own file.
Also move `struct channel_info` from peer_htlcs.h to channel.h where
it more logically belongs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Looking through logs I was surprise to see:
```
lightningd-1 2023-10-26T03:42:36.824Z INFO lightningd: Sending 200000000msat over 1 hops to deliver 200000000msat
```
On a re-payment where we simply returned from sendpay immediately! Move that log to later.
We used to have "unsaved" payments: now we don't we can use
our normal "iterator" pattern rather than returning arrays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It used to be used for both `sendpay` and `waitsendpay` but now it's
only for the latter, so the name is confusing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't write to db immediately, but waited until it the actual HTLC got
added (or failed). That way we didn't have a separate transaction to
write the payment into the db, but the complexity is not worth it: it
makes the next refactors harder, since we can't use the normal
iterator patterns like we do with the rest of the db (as we have to add
the unstored ones).
We might as well also make sendpay return immediately: we used to return
once the HTLC had been confirmed sent, since we entered it in the db
at that point, but we can keep it simple now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We generalize the invoice routine, though it's (known) buggy, so we have
to copy it. We rename the invoice routine to a more specific name though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Adding an index means:
1. Add the new subsystem, and new updated_index field to the db, and
create xxx_index_deleted/created/updated APIs.
2. Hook up these functions to the points they need to be called.
3. Add index, start and limit fields to the list command.
4. Add created_index and updated_index into the list command.
This does #1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The log messages were changed but the test fields weren’t updated, resulting in some test flakiness. Being more explicit with the log message we’re looking for should help.
Changelog-None
This means refactoring out some of the generic anchor info, from the
per-commitment-tx info (we can have at least two, perhaps more with
splicing!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's going to want to remember these, in case it encounters peers'
commitment tx and needs to boost it with CPFP on the anchor.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
channel_txs was a thin wrapper around channel_splice_txs, but that's
just confusing. Rename channel_splice_txs to channel_txs, and just
call it everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We try to use anchors to CPFP our own commitment, but what if they
get there first? We also need to use anchors on the commitment
txs they broadcast.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Under some circumstances we may want to not log to `lightningd`
directly, but rather configure the logging ourselves. This is useful
for example if we want to use `tracing` and `tracing-subscriber` to
add custom handling, or add opentelemetry span tracing.
Changelog-Changed: cln-plugin: Suppress internal logging handler via `with_logging(false)`
Tihis commit is implementing a 2-phase commit between
the signer the node and the peer.
The main reason for this is that everybody must agree on the lock,
otherwise one of them will want N signatures (on the splice candidates),
and another will produce only 1 signature.
check_outpoint is the "prepare" for the signer, and lock_outpoint is the
"commit". if check_outpoint returns true, lock_outpoint must not fail.
Link: https://github.com/ElementsProject/lightning/issues/6722
Suggested-by: @devrandom
Co-Developed-by: Ken Sedgwick <ken@bonsai.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
In general, a validating signer may be under a different operational
environment than the node, and therefore may have a different
source of on-chain data. The signer may therefore temporarily disagree
on whether a funding or splice transaction is locked (buried).
We would like to ensure agreement between the signer and the
node on how to progress a channel's state.
The following message are added to provide a solution:
- `check_outpoint(outpoint) -> bool` - check if the signer agrees that a funding candidate outpoint is buried
- `lock_outpoint(outpoint)` - change the funding/splice state to locked
Link: https://github.com/ElementsProject/lightning/issues/6722
Suggested-by: @devrandom
Co-Developed-by: Ken Sedgwick <ken@bonsai.com>
Changelog-Added: hsmd protocol: Added hsmd_check_outpoint and hsmd_lock_outpoint
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Added: JSON-RPC: `recover` command to force (unused) lightningd node to restart with `--recover` flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes `check` much more thorough, and useful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` now does much more checking on every command (not just basic parameter types).
Put an assertion inside db.c, and run every command we do (in testing) through
a `check` variant.
I inserted a deliberate bug (made addpsbtoutput call wallet_get_newindex()
before returning when running `check`, and indeed, backtrace as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We often want to do more parameter checks after param(), so allow a
new param_check(), with the proviso that the caller needs to also return
command_check_done() after other checks if command_check_only(cmd) is true.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a complaint that you can't CPFP a mutual close, which you
should be able to do.
Fixes: #6692
Changelog-Fixed: wallet: close change outputs show up immediately in `listfunds` so you can CPFP.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
datastoreusage returns the total_bytes that are stored under a given
{Key} or from root. {Key} is the entry point from which we begin to
traverse the datastore.
Changelog-Added: JSON-RPC: `datastoreusage`: returns the total bytes that are stored under a given key.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Under certain conditions, when splicing a new channel quickly enough, an old channel announcement would emit *after* `mutual_splice_lock` and *before* announcement signature exchange.
Since the original channeld wouldn’t start the announcement timer until signatures were exchagned, this wasn’t an issue before.
Now splicing enables us to go from having announcement sigs to losing them, so we have to be prepared for this case.
Changelog-None
```
FAILED tests/test_connection.py::test_remote_addr_port - TimeoutError: Unable to find "[re.compile('Update our node_announcement for discovered address')]" in logs.
```
Because it can happen before the "Already have funding locked in" message:
```
lightningd-2 2023-10-24T22:07:02.018Z DEBUG gossipd: Update our node_announcement for discovered address: 127.0.0.1:1234
lightningd-2 2023-10-24T22:07:02.019Z DEBUG lightningd: Plugin chanbackup returned from peer_connected hook call
lightningd-2 2023-10-24T22:07:02.019Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-chan#2: Peer has reconnected, state CHANNELD_NORMAL: connecting subd
lightningd-2 2023-10-24T22:07:02.036Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-channeld-chan#2: pid 63142, msgfd 67
lightningd-2 2023-10-24T22:07:02.037Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-chan#2: Already have funding locked in (and ready to announce)
```
Also, wait_for_log() asserts itself, no need to assert on result.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
Already in transaction from lightningd/plugin.c:727
```
There are two callers, and one didn't disable transactions, so do it in plugin_exclusive_loop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we've asserted that channeld would tell lightningd the same thing it
would do anyway, we can simply have channeld say "enable=True|False" and
lightningd fill in the other fields.
This means there's a pile of things channeld doesn't need to know any more!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
channeld used to talk directly to gossipd, so it made sense for it to
tell gossipd directly when it wanted it to make a new channel_update.
When that changed with v0.11, we simply directed the message via
lightningd.
But much of the information is actually told to channeld by lightningd!
So I applied this assertion and ran the test suite, before the next patch makes it redundant.
We got one assertion: test_setchannel_zero deliberately drives the
advertized htlc_max over the real htlc max in test_setchannel_zero for
testing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`[False]` is considered True in Python, so this flake fix didn't work.
```
>>> if [False]:
... print('x')
...
x
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Wait to be sure l1->l2 is ready. And use the same pattern for l2->l3.
```
def test_excluded_adjacent_routehint(node_factory, bitcoind):
"""Test case where we try have a routehint which leads to an adjacent
node, but the result exceeds our maxfee; we crashed trying to find
what part of the path was most expensive in that case
"""
l1, l2, l3 = node_factory.line_graph(3)
# We'll be forced to use routehint, since we don't know about l3.
wait_for(lambda: len(l3.rpc.listchannels(source=l2.info['id'])['channels']) == 1)
inv = l3.rpc.invoice(10**3, "lbl", "desc", exposeprivatechannels=l2.get_channel_scid(l3))
# This will make it reject the routehint.
err = r'Fee exceeds our fee budget: 1msat > 0msat, discarding route'
with pytest.raises(RpcError, match=err):
> l1.rpc.pay(bolt11=inv['bolt11'], maxfeepercent=0, exemptfee=0)
tests/test_pay.py:3420:
...
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt10n1pjntczasp59x0weqkg4u9amd364yaeyw6rmgnmf9qtra6epylcntvt65yalpzspp5x8wgtmjhq33qruk6mmhutyr7w74xxjhct7v9tppel0t9p4rtautsdq8v3jhxccxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqgqqqqqqqqpqqqqqzsqqc9qxpqysgq4euy2qyzl2nufpxv6ahf0s5zry0h5dgrpa5adwu4swrdvjw7qe48cj8kp5fl7k20ex0x3dnk6e8xk5jp82snrdcr6he7eyqd0wrmvlgqwe5nma', 'maxfeepercent': 0, 'exemptfee': 0}, error: {'code': 210, 'message': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 1000msat}]}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make sure plugin has got message to connectd before sending!
```
def test_even_sendcustommsg(node_factory):
l1, l2 = node_factory.get_nodes(2, opts={'log-level': 'io',
'allow_warning': True})
l1.connect(l2)
# Even-numbered message
msg = hex(43690)[2:] + ('ff' * 30) + 'bb'
# l2 will hang up when it gets this.
l1.rpc.sendcustommsg(l2.info['id'], msg)
l2.daemon.wait_for_log(r'\[IN\] {}'.format(msg))
l1.daemon.wait_for_log('Invalid unknown even msg')
wait_for(lambda: l1.rpc.listpeers(l2.info['id'])['peers'] == [])
# Now with a plugin which allows it
l1.connect(l2)
l2.rpc.plugin_start(os.path.join(os.getcwd(), "tests/plugins/allow_even_msgs.py"))
l1.rpc.sendcustommsg(l2.info['id'], msg)
l2.daemon.wait_for_log(r'\[IN\] {}'.format(msg))
> l2.daemon.wait_for_log(r'allow_even_msgs.*Got message 43690')
tests/test_misc.py:3623:
...
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('allow_even_msgs.*Got message 43690')]" in logs.
contrib/pyln-testing/pyln/testing/utils.py:327: TimeoutError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have to work quite hard to do this, since we don't want to call
finish if the broadcast has been freed in the meantime.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously, every broadcast was attached to a channel, but we can
make it explicit, so when the context is freed, the re-broadcast stops
(if rebroadcast is set).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If the context is freed, the callback isn't called. This doesn't matter
yet, since our callbacks tend to be such that the callback itself is
required to free things, but it's clearer this way and allows more
flexible usage in following patches.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We remove it from the pending_requests strmap before calling it,
so it doesn't get called again by destroy_plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We should really unify the cases of a local request, vs a forwarded
request, but for now, don't steal the request onto the plugin, and
if we return from the plugin and the request is gone, don't get upset.
This uncovered a case where we weren't inside a transaction, in
test_hook_crash, so fix that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now the internal code will generate a "PLUGIN_TERMINATED" response
when the plugin dies, we can handle it in plugin_hook.
But we can also simplify it by turning the snapshot of hooks into
a simple array: this means we are robust against any combination of plugins
exiting at any time.
Note: this reveals an issue with test_rpc_command_hook where we run
the request hook again (unexpectedly), so we disable that for the next
patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had special code to fail a forwarded request, but not for an
internally-generated request. Instead, we should pretend the (dead)
plugin responded with a PLUGIN_TERMINATED error, and handle the
request through the normal paths.
This breaks the case where a plugin crashes (or stops itself) in a
hook, so we handle that next.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was always a bit weird they weren't, and it seems a premature
optimization to make the callbacks to this themselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's per-plugin, so why is there a single map for all plugins? It
works because we always make unique ids, but it's weird.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When using DEBUG_SUBD with pytest:
```
lightningd: Unknown decode for --dev-debugger=<subprocess>
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During the changeset calculation after the `openchannel2_sign`
hook.
So this commit patch the problem with the following change:
- Addressed an issue where `psbt_get_changeset` was modifying the original PSBT unnecessarily.
- This modification led to problems with a different hsmd, as referenced in [Issue #6672](https://github.com/ElementsProject/lightning/issues/6672).
- Noted a potential optimization where only a subpart of the PSBT
needs to be cloned, as the mutation is specific to inputs.
Link: https://github.com/ElementsProject/lightning/issues/6672
Reported-by: @devrandom
Suggested-by: Ken Sedgwick <ken@bonsai.com>
Co-Developed-by: Ken Sedgwick <ken@bonsai.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Added: Plugins: plugins can now specify (unknown) even messages we should accept from peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an alternative version to [1] that
update only the necessary information around our codebase
to upgrade the bitstring package.
This upgrade is necessary for the user of our package that
will find conflics in their codebase about packages version.
[1] https://github.com/ElementsProject/lightning/pull/6585
Co-Developed-by: @dni
Chanegelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Some methods (`withdraw`) require their parameter to be in satoshis
rather than millisats. In order for us not to have to come up with yet
another triple of sat, sat_or_all, and sat_or_any, we just bolt it
onto the conversion.
If we get a WIRE_TX_ABORT then another message, we send the other message to the same
subd (even though the tx abort causes it to shutdown). This means we effectively
lose the next message, and timeout (see below from CI, reproduced locally).
So, have connectd ignore the subd after it forwards the WIRE_TX_ABORT. The next
message will, correctly, cause a fresh subdaemon to be spawned.
```
@unittest.skipIf(TEST_NETWORK != 'regtest', 'elementsd doesnt yet support PSBT features we need')
@pytest.mark.openchannel('v2')
def test_v2_rbf_multi(node_factory, bitcoind, chainparams):
l1, l2 = node_factory.get_nodes(2,
opts={'may_reconnect': True,
'dev-no-reconnect': None,
'allow_warning': True})
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
amount = 2**24
chan_amount = 100000
bitcoind.rpc.sendtoaddress(l1.rpc.newaddr()['bech32'], amount / 10**8 + 0.01)
bitcoind.generate_block(1)
# Wait for it to arrive.
wait_for(lambda: len(l1.rpc.listfunds()['outputs']) > 0)
res = l1.rpc.fundchannel(l2.info['id'], chan_amount)
chan_id = res['channel_id']
vins = bitcoind.rpc.decoderawtransaction(res['tx'])['vin']
assert(only_one(vins))
prev_utxos = ["{}:{}".format(vins[0]['txid'], vins[0]['vout'])]
# Check that we're waiting for lockin
l1.daemon.wait_for_log(' to DUALOPEND_AWAITING_LOCKIN')
# Attempt to do abort, should fail since we've
# already gotten an inflight
with pytest.raises(RpcError):
l1.rpc.openchannel_abort(chan_id)
rate = int(find_next_feerate(l1, l2)[:-5])
# We 4x the feerate to beat the min-relay fee
next_feerate = '{}perkw'.format(rate * 4)
# Initiate an RBF
startweight = 42 + 172 # base weight, funding output
initpsbt = l1.rpc.utxopsbt(chan_amount, next_feerate, startweight,
prev_utxos, reservedok=True,
min_witness_weight=110,
excess_as_change=True)
# Do the bump
bump = l1.rpc.openchannel_bump(chan_id, chan_amount,
initpsbt['psbt'],
funding_feerate=next_feerate)
# Abort this open attempt! We will re-try
aborted = l1.rpc.openchannel_abort(chan_id)
assert not aborted['channel_canceled']
# We no longer disconnect on aborts, because magic!
assert only_one(l1.rpc.listpeers()['peers'])['connected']
# Do the bump, again, same feerate
> bump = l1.rpc.openchannel_bump(chan_id, chan_amount,
initpsbt['psbt'],
funding_feerate=next_feerate)
tests/test_opening.py:668:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-client/pyln/client/lightning.py:1206: in openchannel_bump
return self.call("openchannel_bump", payload)
contrib/pyln-testing/pyln/testing/utils.py:718: in call
res = LightningRpc.call(self, method, payload, cmdprefix, filter)
contrib/pyln-client/pyln/client/lightning.py:398: in call
resp, buf = self._readobj(sock, buf)
contrib/pyln-client/pyln/client/lightning.py:315: in _readobj
b = sock.recv(max(1024, len(buff)))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyln.client.lightning.UnixSocket object at 0x7f34675aae80>
length = 1024
def recv(self, length: int) -> bytes:
if self.sock is None:
raise socket.error("not connected")
> return self.sock.recv(length)
E Failed: Timeout >1200.0s
```
Make sure l1 can see l2's channel and it's in the gossip_store.
```
def test_excluded_adjacent_routehint(node_factory, bitcoind):
"""Test case where we try have a routehint which leads to an adjacent
node, but the result exceeds our maxfee; we crashed trying to find
what part of the path was most expensive in that case
"""
l1, l2, l3 = node_factory.line_graph(3)
# We'll be forced to use routehint, since we don't know about l3.
wait_for(lambda: len(l3.rpc.listchannels(source=l2.info['id'])['channels']) == 1)
inv = l3.rpc.invoice(10**3, "lbl", "desc", exposeprivatechannels=l2.get_channel_scid(l3))
# This will make it reject the routehint.
err = r'Fee exceeds our fee budget: 1msat > 0msat, discarding route'
with pytest.raises(RpcError, match=err):
> l1.rpc.pay(bolt11=inv['bolt11'], maxfeepercent=0, exemptfee=0)
tests/test_pay.py:3420:
...
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt10n1pjjm8hesp5kp4nfgrj2ev6dz6xuqgxg29hz7263ltlafylhw7nglhtjxeqpn7spp5w92tjq8a354psfhdzmeuytfc6eye4f5egl7tj7s0f5ftz0k4pmcqdq8v3jhxccxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqgqqqqqqqqpqqqqqzsqqc9qxpqysgqetjwr6ql24jrz02qhj7pdw3kqynw6j3sgj2h32ufeyzasjyp2j6yc5durewjjpjy5yqtfgdxmdj52n7jk0ylzk5wudk4ffmjyyw6jmsqkvjex9', 'maxfeepercent': 0, 'exemptfee': 0}, error: {'code': 210, 'message': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 1000msat}]}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The assertion may not make sense if we change the
meaning of `push_value` to be relative, especially since
negative values appear as large positive.
Suggested-by: Ken Sedgwick <ken@bonsai.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
The signer needs to know when the splice operation starts and the
splice parameters for each splice transaction candidate.
The channel establishment v2 (dual funding) code path already
notifies the signer via the hsmd API hsmd_ready_channel calls
However, the splicing code path does not.
Link: https://github.com/ElementsProject/lightning/issues/6723
Suggested-by: @devrandom
Co-Developed-by: @devrandom
Co-Developed-by: Ken Sedgwick <ken@bonsai.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Originally VLS used hsmd_ready_channel as an early call during channel
setup, but later the BOLT-2 spec changed the name of funding_locked to channel_ready.
This is very confusing because the hsmd_ready_channel is not directly
related to the new channel_ready.
This commit is renaming the hsmd_ready_channel to hsmd_setup_channel.
Link: https://github.com/ElementsProject/lightning/issues/6717
Suggested-by: Ken Sedgwick
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
While rebasing the PR for per rune restriction, I unintentionally merged the
`{SQL("ALTER TABLE runes ADD last_used_nsec BIGINT DEFAULT NULL"), NULL}` database
alteration command ahead of {NULL, migrate_runes_idfix} (commit eacf0b502c (diff-1abcdf1b9d822b30079d6450b790274bdfb7c7fa04baa43ad2d9bd449865d4c9R978)).
`migrate_runes_idfix` was the 234th change (deployed with version 23.08.1)
and adding the `last_used_nsec` column should have been the next
(235th, added in current release) change. Due to this incorrect ordering,
nodes updating from version 23.08.1 to the master branch will not add the
`last_used_nsec` column as they should, and instead execute `migrate_runes_idfix`
again, leading to the error in issue #6770.
After the reordering, db_get_runes method also has to be fixed for only
selecting rune NOT last_used_nsec. Because this column was added after
`migrate_runes_idfix` calls it. I am tempted to change the method name from
`db_get_runes` to `db_migrate_runes` for more clarity on its functionality though.
Changelog-None.
We do it here, but it's not necessary, and we also deprive them of the
chance to do so (since we kill them).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was intermittant before: I added a sleep(1) in the code before
sending the error (temporarily) to make it always triggers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously, we would forward the message to a subd, but now we have
the case where the subd is gone, but we're still connected. If the
peer anything but a reestablish in that state, we drop the connection.
Instead, an error should always make us fail the channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We truncate the file on stop(), but don't re-created it on start().
We didn't notice it before, but the net
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We generalize the current df-only "aborted" flag (and invert it) to a
"disconnected" flag in the peer status message.
We convert it back to the aborted flag for now inside subd.c, but that's
next.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
On Mac most tests report BROKEN because sodium creating an untracked fd pointing to /dev/random. dev_report_fd’s finds it at tear down and reports a BROKEN message.
We allow a single “char special” fd without reporting it as broken improving QOL for Mac developers.
While we’re here we added the fd mode to the log to help with future rogue fd issues.
ChangeLog-None
Rather than crashing the entire node on invalid pubkey, check the
validity of the pubkey in decode_n, and return an error if invalid.
Detected by libFuzzer:
==265599== ERROR: libFuzzer: deadly signal
#7 abort
#8 bolt11_decode common/bolt11.c:999:4
Invalid recovery IDs cause
secp256k1_ecdsa_recoverable_signature_parse_compact to abort, which
crashes the entire node. We should return an error instead.
Detected by libFuzzer:
[libsecp256k1] illegal argument: recid >= 0 && recid <= 3
Remove the assertion so that an error is returned for invalid bech32.
An error is preferable to crashing the entire node if there's an extra
"lightning:" prefix:
$ lightning-cli pay "lightning:lightning:"
Node log:
pay: common/bolt11.c:718: bolt11_decode_nosig: Assertion `!has_lightning_prefix(str)' failed.
pay: FATAL SIGNAL 6
...
INFO plugin-pay: Killing plugin: exited during normal operation
**BROKEN** plugin-pay: Plugin marked as important, shutting down lightningd
If both databits and *data_len are 0, pull_uint return uninitialized
stack memory in *val.
Detected by valgrind and UBSan.
valgrind:
==173904== Use of uninitialised value of size 8
==173904== __sanitizer_cov_trace_cmp8
==173904== decode_c (bolt11.c:292)
==173904== bolt11_decode_nosig (bolt11.c:877)
UBSan:
common/bolt11.c:79:29: runtime error: shift exponent 64 is too large for 64-bit type 'uint64_t' (aka 'unsigned long')
Corpus input e6f7b9744a7d79b2aa4f7c477707bdd3483f40fa triggers the UBSan
report, but we didn't previously realize this because UBSan has been
disabled in the CI run. We rename the input to indicate its usefulness
as a permanent regression test.
Otherwise, if pull_all fails, we attempt to create a script from NULL,
causing a UBSan report:
bitcoin/script.c:29:28: runtime error: null pointer passed as argument 2, which is declared to never be null
Corpus input bf703c2c20c0818af70a8c4caad6e6fd8cfd1ac6 triggers the UBSan
report, but we didn't previously realize this because UBSan has been
disabled in the CI run. We rename the input to indicate its usefulness
as a permanent regression test.
Move the "no lease, return" to the top, to avoid testing twice. Also,
we won't spam now for most channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes the pay helper function being able to route
a payment using the optional `route` paramter that defaults to `False`.
I added this, as some plugins maintained their own version of `pay` that
needed routed payment helper.
Changelog-None
During our development, we modified the way
we report backtraces.
On a minimal configuration in OpenBSD, it seems that we
no longer compile from commit a9f26b7d07 because
our conditional code is buggy.
With the following compiler
vultr# cc -v
OpenBSD clang version 13.0.0
Target: amd64-unknown-openbsd7.3
Thread model: posix
InstalledDir: /usr/bin
We have the following error
cc common/channel_id.c
cc common/daemon.c
common/daemon.c:218:2: error: implicit declaration of function 'add_steal_notifiers' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
add_steal_notifiers(NULL);
^
1 error generated.
gmake: *** [Makefile:298: common/daemon.o] Error 1
Reported-by: @grubles
Fixes a9f26b7d07
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
As a node matures and is no longer new, it can take some time
to determine which version of `cln` it's running.
To address this, we now display the version earlier. This ensures
that even in the event of a crash, we're aware of the running version.
(cln-meta-project-py3.11) ➜ lightning git:(macros/log-version) ✗ ./lightningd/lightningd
2023-10-12T19:21:00.899Z INFO lightningd: v23.08.1-209-gae94be4-modded
2023-10-12T19:21:00.994Z INFO lightningd: Creating configuration directory /home/vincent/.lightning/bitcoin
2023-10-12T19:21:01.235Z INFO lightningd: Creating database
2023-10-12T19:21:01.279Z UNUSUAL hsmd: HSM: created new hsm_secret file
Could not connect to bitcoind using bitcoin-cli. Is bitcoind running?
Make sure you have bitcoind running and that bitcoin-cli is able to connect to bitcoind.
You can verify that your Bitcoin Core installation is ready for use by running:
$ bitcoin-cli echo 'hello world'
2023-10-12T19:21:01.349Z **BROKEN** plugin-bcli: \nCould not connect to bitcoind using bitcoin-cli. Is bitcoind running?\n\nMake sure you have bitcoind running and that bitcoin-cli is able to connect to bitcoind.\n\nYou can verify that your Bitcoin Core installation is ready for use by running:\n\n $ bitcoin-cli echo 'hello world'\n
2023-10-12T19:21:01.349Z INFO plugin-bcli: Killing plugin: exited before we sent init
The Bitcoin backend died.
Link: https://github.com/ElementsProject/lightning/issues/6374
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We update `test_grpc_no_auto_start` test to check that we do not
generate certificates when the cln-grpc plugin is not started.
This fails currently, so next commit fix it up.
Add test `test_rune_method_not_equal_and_method_empty` that reproduces
issue #6725.
This fails currently, so next commit fix it up.
Error:
```
> with pytest.raises(RpcError, match='Not permitted: method not present'):
E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
tests/test_runes.py:605: Failed
```
Without `developer` option, after compiling CLN and sourcing
`contrib/startup_regtest.sh`, if we try to start 2 nodes on regtest
running the command `start_ln` we get the following error:
```
lightningd: Config file /tmp/l1-regtest/config line 6: dev-fast-gossip: requires --developer
...
lightningd: Config file /tmp/l2-regtest/config line 6: dev-fast-gossip: requires --developer
```
Changelog-None
The tests will wait until it's locally enabled, but it might not have
the update in the gossip store. So have renepay enhance its local
view even if it already knows about the channel (this is correct
anyway, it just isn't very important usually).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This breaks Dijkstra, which is presumably why it was actually
disabled. Remove the code altoghether, instead.
Changelog-Fixed: JSON-RPC: `getroute` now documents that it ignores `fuzzpercent`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In this case, we were allocating off NULL, which meant a leak:
```
MEMLEAK: 0x565086722e98
label=channeld/channeld.c:3433:struct inflight
backtrace:
ccan/ccan/tal/tal.c:477 (tal_alloc_)
channeld/channeld.c:3433 (inflights_new)
channeld/channeld.c:3573 (splice_accepter)
channeld/channeld.c:4145 (peer_in)
channeld/channeld.c:6051 (main)
parents:
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also added splice_out tests that use the new PSBT command.
ChangeLog-Added: New `addpsbtoutput` command for creating a PSBT that can receive funds to the on-chain wallet.
We show where the pointer was allocated, but it can be confusing if it
was later stolen onto another context. Save and report those too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes it easier to use outside simple subds, and now lightningd can
simply dump to log rather than returning JSON.
JSON formatting was a lot of work, and we only did it for lightningd, not for
subdaemons. Easier to use the logs in all cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't apply the inflight to the channel struct before asserting, so
we can break test_rbf_non_last_mined:
```
lightningd: lightningd/dual_open_control.c:981: dualopend_tell_depth: Assertion `bitcoin_txid_eq(&channel->funding.txid, txid)' failed.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6696
Changelog-Fixed: rune: use runes table `id` instead `runes_uniqueid` from `vars` because it returns incorrect unique id if rune/s migrated from datastore.
Removed `bin-` while copying Fedora sums from release captain's file. Sums are saved as `clightning-$VERSION-Fedora-28-amd64.tar.gz` not as `clightning-$VERSION-bin-Fedora-28-amd64.tar.gz`
Changelog-None.
Now we're not always using the same functions to watch during
dual-funding opening, we need to make sure we're watching the close
(in particular, df close before the opening is confirmed).
So, keep a pointer, and if it's not set in drop_to_chain, set it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used the original channel funding output number. I'm not sure if this
was true in the previous code, or a regression I introduced, but it
caused occasonal failures in test_splice_gossip!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use the *same* callback for the funding tx, as well as for inflight dual-funding txs, as well as inflight splice txs. This is deeply confusing!
Instead, use explicit cbs for splicing and df. Once they're locked in, use the normal callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We never do this, but we're about to (we always watch before
we broadcast a tx).
We use a `depth` member to avoid calling the callback multiple times
for the same event, but we initialize it to 0. This means if we
register a watch, and the first thing that happens is that it
reorganizes out, we *don't* make the callback.
Use an impossible value at initialization, instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The latter is used when we're put in the db, the former is the uncommitted state.
Currently dbid == 0 is used in addition to the state, which is unwieldy.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Experimental: JSON-RPC: added new dual-funding state `DUALOPEND_OPEN_COMMITTED`
This is the variant of DUALOPEND_OPEN_INIT which you see once
the channel is in the db: we'll be adding it next, but to reduce
clutter the docs are added as a separate commit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We usually hand times by copy, not by pointer (and if we did, they should
be const!). I noticed this particularly for the state changed code, but
it goes down to to json_add_timeiso, so I fixed that too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We should use capability tests for states (can you add htlcs?) rather than vague
descriptions (are you closing?).
And as much as possible, use switch () statements to force us to think
about all the cases, especially when we add new states!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently it's half done in funding_depth_cb, and half in
channeld_tell_depth. It's very confusing as a result,
with splicing, dual-funding and zeroconf.
This does introduce a behaviour change: if a channel is NORMAL and
it gets reorganized, we force close (unless we were the one who funded
it, or it's zeroconf anyway). This is safer than continuing to use
the channel in this case!
Some tests are changed to zeroconf to make them work, but v2 doesn't
support zeroconf, so that's removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a workaround, the real fix is to use a different
callback for inflight splice attempts, which comes later.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not just if htlc addition is too slow, make this the default. dual-open's txabort
is excluded, however.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is actually a real issue (l1 doesn't see the warning before l2
drops the connection), but it's unrelated to this PR, and will require
another one to fix.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Even after migrating from pip install to poetry install for clnrest,
we still need to keep `plugins/clnrest/requirements.txt` for Fedora-28-amd64
build. `Dockerfile.builder.fedora` does not have poetry and will need
requirements.txt to install libraries with pip.
It is also required for non-developers who want to build cln from a tagged or master version.
This commit is just bumping the Cargo.lock file to
fix the following build issue with the more
recent rust version.
cargo build --quiet --example cln-plugin-startup
error[E0635]: unknown feature `proc_macro_span_shrink`
--> /home/vincent/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.59/src/lib.rs:92:30
|
92 | feature(proc_macro_span, proc_macro_span_shrink)
| ^^^^^^^^^^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0635`.
error: could not compile `proc-macro2` (lib) due to previous error
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
In its default mode, `xargs` interprets any whitespace as a delimiter,
and it interprets single and double quotes and backslashes specially.
This can play havoc when fed by tools that don't escape these special
characters in their output.
Fortunately, `xargs` (at least the GNU version of it) has a `-0` option
that changes the behavior so that only NUL characters act as delimiters,
and no other characters have any special meaning. Use this option
consistently to avoid any nasty surprises.
Note that `git-ls-files` has a `-z` option that causes it to terminate
filenames on output with a NUL character instead of a newline, and
`grep` has a `-z` option that causes it to operate on NUL-terminated
records rather than on lines. Use these options together with
`xargs -0`.
Note: This commit corrects an oversight introduced in
9d94687b0d.
Changelog-None
We were accidentally testing against `bitcoind` rather than
`elementsd` which meant we believed everything was find, when in
reality some tests broke. By only installing the required daemon we
ensure that such a mixup causes a CI failure.
Use xargs rm -f which doesn't care if it's already deleted.
```
find contrib/pyln-grpc-proto/pyln/ -type f -name "*.py.bak" -delete
find contrib/pyln-grpc-proto/pyln/ -type f -name "*.py.bak" -delete
find: cannot delete ‘contrib/pyln-grpc-proto/pyln/grpc/node_pb2_grpc.py.bak’: No such file or directory
make: *** [Makefile:390: contrib/pyln-grpc-proto/pyln/grpc/node_pb2.py] Error 1
make: *** Waiting for unfinished jobs....
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you previously configured with `--enable-developer` we turn that into `--enable-debugbuild`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: build: `--enable-developer` arg to configure (and DEVELOPER variables): use `./configure --enable-debugbuild` and `developer` setting at runtime.
And require --developer to use them.
Also refuse redirection to deprecated APIs if deprecated APIs are disabled!
Changelog-Removed: `dev-sendcustommsg` (use `sendcustommsg`, which was added in v0.10.1)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We still refuse to run dev commands if lightningd sends it to us
despite us not being in developer mode, but that's mainly paranoia.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Mainly removing the PSBT re-marshalling which hasn't had any issues in
recent libwally, and making dev_no_grind into the clearer
dev_no_signature_grind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also requires us to expose memleak when !DEVELOPER, however we only
ever used the memleak tracking when the LIGHTNINGD_DEV_MEMLEAK
environment variable was set, so keep that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently it just defaults to the DEVELOPER compile option, but we'll
move over to this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `--developer` enables developer options and changes default to be "disable deprecated APIs".
We check for list_empty, so it's always actually set.
```
lightningd/peer_control.c: In function ‘drop_to_chain’:
lightningd/peer_control.c:353:17: error: ‘tx’ may be used uninitialized [-Werror=maybe-uninitialized]
353 | resolve_close_command(ld, channel, cooperative, tx);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lightningd/peer_control.c:341:36: note: ‘tx’ was declared here
341 | struct bitcoin_tx *tx;
| ^~
cc1: all warnings being treated as errors
make: *** [Makefile:298: lightningd/peer_control.o] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Added a test for splicing out that exposed some behavior and code glitches that are addressed in this commit.
Added test for splice gossip.
Also added documentation for how to do a splice out.
ChangeLog-Fixed: Added docs, testing, and some fixes related to splicing out, insufficent balance handling, and restarting during a splice.
json_add_timeabs only printed in milliseconds and json_add_time outputs a string which is weird
Changelog-Changed: JSON-RPC time fields now have full nanosecond precision (i.e. 9 decimals not 3): `listfowards` `received_time` `resolved_time` `listpays`/`listsendpays` `created_at`.
Some functions have vanished in master, and it's confusing to
see this change when we run `make update-mocks` later in the
series.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we changed the default, it used to be required to set it. That was a while ago, though, so we can make it optional again.
Changelog-Changed: Protocol: `invoice` no longer explicitly encodes `c` if it's the default (18)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't need to patch it in anymore, now it's merged. However, we do
move the message itself from onion_wire.csv to peer_wire.csv (we
should get more sophisticated with our parsing, but this works for
now!).
The resulting peer_wire.csv is identical, the onion_wire.csv file is
slightly reordered.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Explicitly allow all-zero in the onion_hash: we didn't do anything except log if it was unexpected anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were allowed to, but the spec removed that. So we handle warnings
differently from errors now.
This also means the LND "internal error" workaround is done in
lightningd (we still disconnect, but we don't want to close channel).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we no longer disconnect every time we receive a warning message.
These patches are no longer necessary now blinded payments were
merged into the spec.
However, the spec did rename "blinding_point" inside `payload` to
"current_blinding_point" so we temporarily add a patch to change it back.
You can see wire/onion_wire.csv is reordered, but unchanged.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They didn't actually apply properly!
I regenerated the dual-funding CSV from the latest commit
(091397fc0798c4b2bdb36dbaa9f5b4b1d4a463e1) and made it a single patch.
I tried doing the same for splicing, but the implementation has drifted
far from the spec, so I simply opted for a patch which didn't change anything.
You can see the resulting "wire/peer_wire.csv" is the same, except fields
are now in a less-random order!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We do not, in fact, require Python to build, so we should still work
after `make distclean`.
Fixes: #6536
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't want a rule to generate config.vars, since user should run configure manually
with their desired options. However, we do want to be able to *refresh* it, because
otherwise, if we need a new configuration var for our Makefile, it won't get refreshed:
```
$ make
CC: cc -DBINTOPKGLIBEXECDIR="../libexec/c-lightning" -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -Werror -Wshadow=local -std=gnu11 -g -fstack-protector-strong -I ccan -I external/libwally-core/include/ -I external/libwally-core/src/secp256k1/include/ -I external/jsmn/ -I external/libbacktrace/ -I external/gheap/ -I external/build-x86_64-linux-gnu/libbacktrace-build -I external/libsodium/src/libsodium/include -I external/libsodium/src/libsodium/include/sodium -I external/build-x86_64-linux-gnu/libsodium-build/src/libsodium/include -I . -I/usr/local/include -I/usr/include/postgresql -DCCAN_TAKE_DEBUG=1 -DCCAN_TAL_DEBUG=1 -DCCAN_JSON_OUT_DEBUG=1 -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS -DCOMPAT_V052=1 -DCOMPAT_V060=1 -DCOMPAT_V061=1 -DCOMPAT_V062=1 -DCOMPAT_V070=1 -DCOMPAT_V072=1 -DCOMPAT_V073=1 -DCOMPAT_V080=1 -DCOMPAT_V081=1 -DCOMPAT_V082=1 -DCOMPAT_V090=1 -DCOMPAT_V0100=1 -DCOMPAT_V0121=1 -DBUILD_ELEMENTS=1 -c -o
PYTHONPATH=contrib/msggen contrib/msggen/msggen/__main__.py
LD: cc config.vars -Lexternal/build-x86_64-linux-gnu -lwallycore -lsecp256k1 -ljsmn -lbacktrace -lsodium -L/usr/local/include -lm -lsqlite3 -lz -L/usr/lib/x86_64-linux-gnu -lpq -o
/bin/sh: 1: contrib/msggen/msggen/__main__.py: Permission denied
make: *** [Makefile:371: cln-grpc/proto/node.proto] Error 126
make: *** Waiting for unfinished jobs....
```
Here, PYTHON is the new var (here, unset). If we were building a binary, we'd depend on config.h,
and the Makefile says how to refresh that. But the Makefile includes config.vars (causing an
implicit dependency) so we really should have a rule to make that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can keep a single array of 'already considered' utxos, with the same
result as Tony's patch prior.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If a node has an onchain balance with at least one uneconomical UTXO, the fundchannel RPC call will lock up the node and will eventually crash it with OOM issues if the economical UTXO(s) do not add up to the fundchannel amount. This is because the while loop never exits because it keeps pulling in the same uneconomical UTXOs forever.
Changelog-Fixed: wallet: fundchannel no longer loops forever if the wallet contains insufficient funds, but an uneconomical UTXO.
Tony Giorgio <tonygiorgio@protonmail.com> says:
Reproduce:
1. Add 1 600 sat UTXO to a fresh node
2. Verify the fundchannel command fails with a low fee rate:
```
./lightning-cli fundchannel 0366abc8eb4da61e31a8d2c4520d31cabdf58cc5250f855657397f3dd62493938a 100000 1000
{
"code": 301,
"message": "Could not afford 100000sat using all 1 available UTXOs: 99522sat short"
}
```
3. Now do the command again, but with a higher fee rate, making the 600 sat UTXO uneconomical:
```
./lightning-cli fundchannel 0366abc8eb4da61e31a8d2c4520d31cabdf58cc5250f855657397f3dd62493938a 100000 10000
```
4. Observe the RPC call and the logs. The RPC call will never return, and the logs will stop after this:
```
2023-04-16T10:58:45.839Z DEBUG plugin-spenderp: mfc 34: multiconnect done.
2023-04-16T10:58:45.839Z DEBUG plugin-spenderp: mfc 34: 'parsefeerate' done
2023-04-16T10:58:45.839Z DEBUG plugin-spenderp: mfc 34: fundpsbt.
```
5. Keep CLN running long enough and you'll eventually run OOM.
The former seemed to replace the wrong line with the copied checksum.
We now add read it from the first line and add it on top of our sums.
This expression also seems a fair bit easier to understand now.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Removing the Fedora fake as it seems to be not necessary. We later copy
the checksum from the release captains checksums anyway.
Also adding the sum check as it gives more details about which file did
not match if so.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
This ensures that we have the release captains checksum file where we
expect it to be and gives a little hint where to get it if needed.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
This seems to be a cut & paste bug (mine, AFAICT!) from the command code:
```
rune = rune_derive_start(cmd, master_rune,
tal_fmt(tmpctx, "%"PRIu64,
rune_counter ? *rune_counter : 0));
```
In that case, rune_counter was a pointer, which could be NULL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It always is for runes we create, but in theory you can take our secret key
and make our own runes with your own tools.
(We correctly refuse runes without uniqueids if they're *not* ours
anyway: uniqueid is only used for our own runes).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was a misunderstanding: nodeid is useful for commando, where it's the
peer's nodeid, and Noise-XK guarantees that we know who that is. It's
not useful for clnrest, so don't require it (it was our node id, which
is redundant).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
nodeid is only useful when we know the peer we're talking to (e.g. commando).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No-schema-diff-check: We're simply making optional, not deprecating!
Usage line isn't correct, as fields are not optional, and return
needs fleshing out for error codes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were using `number` instead of `u32` which caused it to be
serialized to string with a decimal, which then would be rejected by
lightningd.
Changelog-Fixed: proto: Fixed a wrong number type being used in routes
OpenAPI readme always includes `content-type: application/json` header, even when body parameters are empty.
But the server expects data if the content-type has been sent.
This results in a "Server Error" response for non-param requests from readme doc.
This only affects readme requests as it is designed to send the header by default.
Changelog-None
1. When we add a shadow amount, we were using the wrong channel for
the fee calculation.
2. Similarly, when calculating the delay amount.
The result is that we can get WIRE_INCORRECT_CLTV_EXPIRY repeatedly
from nodes.
Reported-by: https://github.com/SjorsFixes: #6620
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changlog-Experimental: Fixed: `renepay` handles ctlv correctly when it varies along a path.
"id" is a magic name, so it was being populated by sqlite3
automatically, starting at 0. Fortunately, we only fetched by id in
one place: to indicate the `stored` flag when asked about an explicit
rune in `showrunes`.
Reported-by: @ShahanaFarooqui
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `showrunes` on a specific rune would always say `stored`: false.
```
Flow 391: amount=23528000msat prob=0.000 fees=1023msat delay=140 path=-2471854x37x4/1(min=max=23528783msat)->-2414928x98x0/0->
Flow 391: Failure of 23529023msat for 2471854x37x4/1 capacity [23528783msat,23528783msat] -> [23528783msat,23528783msat]
```
We added fees and went over capacity! This screams of a deeper logic
bug, but renepay is experimental and it's release day so hack around
it for now...
Reported-by: https://github.com/daywalker90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Eduardo is on holiday right now, but he pinged me asking for this. It
makes some sense, and using half the *failed value* covers the case where
it's less than half what we expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It now looks like (for test_hardmpp):
```
# we have computed a set of 1 flows with probability 0.328, fees 0msat and delay 23
# Flow 1: amount=1800000000msat prob=0.328 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0->-103x3x0/1->
# Flow 1: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
# Flow 1: Failure of 1800000000msat for 103x5x0/0 capacity [0msat,3000000000msat] -> [0msat,1799999999msat]
# we have computed a set of 2 flows with probability 0.115, fees 0msat and delay 23
# Flow 2: amount=500000000msat prob=0.475 fees=0msat delay=12 path=-103x6x0/0(min=max=4294967295msat)->-103x1x0/1->-103x4x0/1->
# Flow 3: amount=1300000000msat prob=0.242 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1799999999msat)->-103x3x0/1->
# Flow 3: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
# Flow 3: Failure of 1300000000msat for 103x5x0/0 capacity [0msat,1799999999msat] -> [0msat,1299999999msat]
# we have computed a set of 2 flows with probability 0.084, fees 0msat and delay 23
# Flow 4: amount=260000000msat prob=0.467 fees=0msat delay=12 path=-103x6x0/0(500000000msat in 1 htlcs,min=max=4294967295msat)->-103x1x0/1(500000000msat in 1 htlcs)->-103x4x0/1(500000000msat in 1 htlcs)->
# Flow 5: amount=1040000000msat prob=0.179 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1299999999msat)->-103x3x0/1->
# Flow 5: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
# Flow 5: Failure of 1040000000msat for 103x5x0/0 capacity [0msat,1299999999msat] -> [0msat,1039999999msat]
# we have computed a set of 2 flows with probability 0.052, fees 0msat and delay 23
# Flow 6: amount=120000000msat prob=0.494 fees=0msat delay=12 path=-103x6x0/0(760000000msat in 2 htlcs,min=max=4294967295msat)->-103x1x0/1(760000000msat in 2 htlcs)->-103x4x0/1(760000000msat in 2 htlcs)->
# Flow 7: amount=920000000msat prob=0.105 fees=0msat delay=12 path=-103x2x0/1(min=max=4294967295msat)->-103x5x0/0(max=1039999999msat)->-103x3x0/1->
# Flow 7: Success
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I am doing to add more more debugging, but sent here is 0.
Document that clearly, and put a real value in sent.
Also: since we already sub 1 msat from x, amount_msat_less_eq should
be amount_msat_less (it may be equal to our min, in theory).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The code to workaround the intermittant error didn't work,
and we finally hit it again:
```
# If reject happens fast enough, connect fails with "disconnected
# during connection"
try:
l3.connect(l1)
except RpcError as err:
> assert "disconnected during connection" in err.error
E assert 'disconnected during connection' in {'code': 402, 'message': 'disconnected during connection'}
E + where {'code': 402, 'message': 'disconnected during connection'} = RpcError("RPC call failed: method: connect, payload: {'id': '0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518', 'host': '127.0.0.1', 'port': 41865}, error: {'code': 402, 'message': 'disconnected during connection'}").error
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During tests we can see that the subdaemon can be restarted unnecessarily if we're slow enough; we don't need to do so if it's still running.
Reported-by: Matt Morehouse <mattmorehouse@gmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
l3.rpc.setconfig('autoclean-cycle', 10)
# First it expires.
> wait_for(lambda: only_one(l3.rpc.listinvoices('inv1')['invoices'])['status'] == 'expired')
```
If we're slow enough, the invoice is cleaned before we see it expire!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
Aug 18 13:45:13 lightningd: 0x7fa921f8ffcf ???
Aug 18 13:45:13 lightningd: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
Aug 18 13:45:13 lightningd: 0x55b3bb54e6d3 pay_flow_finished_adding_gossip
Aug 18 13:45:13 lightningd: plugins/renepay/pay_flow.c:675
Aug 18 13:45:13 lightningd: 0x55b3bb54af25 addgossip_done
Aug 18 13:45:13 lightningd: plugins/renepay/pay.c:171
```
The assert we fail is almost certainly due to the flow being freed:
```
struct pf_result *pay_flow_finished_adding_gossip(struct pay_flow *pf)
{
assert(pf->state == PAY_FLOW_FAILED_GOSSIP_PENDING);
```
Reported-by: https://github.com/daywalker90Fixes: #6567
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We determine whether they are allowed or not based on the hook return
value of `mindepth`. To do so we need to pass that value down to
`openingd` and verify that the `channel_type` and our permissions
match up.
There's a fascinating bug report which suggests this happens on local channels,
implying spendable_msat is wrong?
See-also: #6567
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. announce-addr-discovered-port takes a port option.
2. accept-htlc-tlv-types was deprecated in favor of multiple accept-htlc-tlv-type.
3. Document clnrest.py options.
4. Don't list --version twice in lightningd --help (initial_config_opts calls
opt_register_version() already).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It didn't handle the case where an undocumented option was
was a flag (i.e. didn't end in =), so rework it to be
a simple list and use grep.
Add some more options we don't document, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our formats changed, so this didn't work any more!
Add a sanity check that in future if we get no options from a command,
we complain.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Doesn't happen on x86, but struct gossmap_chan defines:
```
u32 private: 1;
u32 plus_scid_off: 31;
```
And complains when we initialize plus_scid_off and access it later:
```
VALGRIND=1 valgrind -q --error-exitcode=7 --track-origins=yes --leak-check=full --show-reachable=yes --errors-for-leak-kinds=all plugins/renepay/test/run-mcf > /dev/null
==186886== Conditional jump or move depends on uninitialised value(s)
==186886== at 0x10076388: chan_iter (gossmap.c:1098)
==186886== by 0x100797F3: gossmap_next_chan (gossmap.c:1112)
==186886== by 0x1008C5AF: main (run-mcf.c:309)
==186886== Uninitialised value was created by a heap allocation
==186886== at 0x40F0A44: malloc (vg_replace_malloc.c:431)
==186886== by 0x10072BAF: allocate (tal.c:256)
==186886== by 0x100737A7: tal_alloc_ (tal.c:463)
==186886== by 0x100738DF: tal_alloc_arr_ (tal.c:506)
==186886== by 0x10079507: load_gossip_store (gossmap.c:690)
==186886== by 0x10079667: gossmap_load (gossmap.c:978)
==186886== by 0x1008C4AF: main (run-mcf.c:295)
```
Reported-by: @grubles
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6557
As side-effect, getroute(0) is special too.
Reported-by: MiddleW4y in Discord
Fixes: #6577
Changelog-Fixed: `pay` will still use an invoice routehint if path to it doesn't take 1-msat payments.
Reported-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: we clean up properly if a plugin fails to start, and we don't kill all processes if it's from `plugin startdir`.
Don’t send the funding spend to onchaind if we detect it in inflights (aka. a splice). While we already prevented onchaind_funding_spent from being called directly, the call to wallet_channeltxs_add meant onchaind_funding_spent would be called *anyway* on restart. This is now fixed.
Additionally there was a potential for a race problem depending on the firing order of the channel depth and and funding spent events.
Instead of requiring these events fire in a specific order, we make a special “memory only” inflight object to prevent the race regardless of firing order.
Changelog-Fixed: Splice: bugfix for restart related race condition interacting with adversarial close detection.
This should provide the default help message and exit, but was
resulting in a segmentation fault from freeing pointers passed to
the default config.
Changelog-Fixed: lightning-cli properly returns help without argument
We have a report that LND said our (unannounced) channel was disabled, so we didn't
use it for routehints. We're better off ignoring that in this case (if the peer is
actually not connected, the routehint code will check that and ignore anyway).
Fixes: #6555
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: pay: use channels in routehints even if peer says they're "disabled" (LND compat)
This is actually a valid complaint (though this is a sanity check for
things we make ourselves, still!).
```
In file included from common/test/run-blindedpath_onion.c:9:
common/test/../sphinx.c: In function ‘sphinx_add_hop_has_length’:
common/test/../sphinx.c:117:12: error: ‘prepended_len’ may be used uninitialized [-Werror=maybe-uninitialized]
117 | if (lenlen + prepended_len != tal_bytelen(payload))
| ^
common/test/../sphinx.c:109:27: note: ‘prepended_len’ was declared here
109 | bigsize_t lenlen, prepended_len;
| ^~~~~~~~~~~~~
cc1: all warnings being treated as errors
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Compiler can't tell that we always set have_state[PAY_FLOW_FAILED_FINAL]
when we set this:
```
plugins/renepay/payment.c: In function ‘payment_reconsider’:
plugins/renepay/payment.c:287:25: error: ‘final_error’ may be used uninitialized [-Werror=maybe-uninitialized]
287 | payment_fail(payment, final_error, "%s", final_msg);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
plugins/renepay/payment.c:194:30: note: ‘final_error’ was declared here
194 | enum jsonrpc_errcode final_error, ecode;
| ^~~~~~~~~~~
plugins/renepay/payment.c:287:25: error: ‘final_msg’ may be used uninitialized [-Werror=maybe-uninitialized]
287 | payment_fail(payment, final_error, "%s", final_msg);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
plugins/renepay/payment.c:195:21: note: ‘final_msg’ was declared here
195 | const char *final_msg;
| ^~~~~~~~~
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Indeed, we can fall through this if it's not a valid enum value.
gcc-12 (Ubuntu 12.2.0-17ubuntu1) 12.2.0
```
In file included from plugins/commando.c:10:
ccan/ccan/tal/str/str.h: In function ‘rune_altern_to_english’:
ccan/ccan/tal/str/str.h:43:9: error: ‘cond_str’ may be used uninitialized [-Werror=maybe-uninitialized]
43 | tal_fmt_(ctx, TAL_LABEL(char, "[]"), __VA_ARGS__)
| ^~~~~~~~
plugins/commando.c:97:21: note: ‘cond_str’ was declared here
97 | const char *cond_str;
| ^~~~~~~~
cc1: all warnings being treated as errors
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
In function ‘peer_reconnect’,
inlined from ‘init_channel’ at channeld/channeld.c:5890:3,
inlined from ‘main’ at channeld/channeld.c:5951:2:
channeld/channeld.c:5028:21: error: ‘next_matches_inflight’ may be used uninitialized [-Werror=maybe-uninitialized]
5027 | if (remote_next_funding && !next_matches_current
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5028 | && !next_matches_inflight) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~
channeld/channeld.c: In function ‘main’:
channeld/channeld.c:4595:36: note: ‘next_matches_inflight’ was declared here
4595 | bool next_matches_current, next_matches_inflight;
| ^~~~~~~~~~~~~~~~~~~~~
channeld/channeld.c:5042:57: error: ‘inflight’ may be used uninitialized [-Werror=maybe-uninitialized]
5042 | &inflight->outpoint.txid),
| ^
channeld/channeld.c:4594:26: note: ‘inflight’ was declared here
4594 | struct inflight *inflight;
| ^~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:300: channeld/channeld.o] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We cannot carry pointers into the gossmap across localmod addition
or removal.
We didn't notice because the map->chan_arr is not normally resized,
but if we change gossmap.c line 689 to only allocate 1 to start, we see this:
```
VALGRIND=1 valgrind -q --error-exitcode=7 --track-origins=yes --leak-check=full --show-reachable=yes --errors-for-leak-kinds=all plugins/renepay/test/run-mcf > /dev/null
==2349744== Invalid read of size 4
==2349744== at 0x1788C2: gossmap_chan_scid (gossmap.c:558)
==2349744== by 0x1872A2: get_chan_extra_half_by_chan (flow.c:346)
==2349744== by 0x187797: remove_completed_flow (flow.c:488)
==2349744== by 0x187927: remove_completed_flow_set (flow.c:518)
==2349744== by 0x18DF4D: main (run-mcf.c:393)
==2349744== Address 0x4b80f38 is 88 bytes inside a block of size 136 free'd
==2349744== at 0x4848C63: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2349744== by 0x173D71: tal_resize_ (tal.c:744)
==2349744== by 0x177E36: next_free_chan (gossmap.c:336)
==2349744== by 0x177ED3: new_channel (gossmap.c:351)
==2349744== by 0x178441: add_channel (gossmap.c:458)
==2349744== by 0x1798D4: gossmap_apply_localmods (gossmap.c:904)
==2349744== by 0x18DEDB: main (run-mcf.c:388)
==2349744== Block was alloc'd at
==2349744== at 0x4848C63: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2349744== by 0x173D71: tal_resize_ (tal.c:744)
==2349744== by 0x177E36: next_free_chan (gossmap.c:336)
==2349744== by 0x177ED3: new_channel (gossmap.c:351)
==2349744== by 0x178441: add_channel (gossmap.c:458)
==2349744== by 0x178B6D: map_catchup (gossmap.c:635)
==2349744== by 0x178F45: load_gossip_store (gossmap.c:697)
==2349744== by 0x179D71: gossmap_load (gossmap.c:978)
==2349744== by 0x18D22F: main (run-mcf.c:295)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's not there if it's a local error:
```
{
"code": 202,
"message": "Parsing '{message:%,data:{erring_index:%,failcode:%,raw_message:': object does not have member raw_message"
}
```
Reported-by: https://github.com/daywalker90Fixes: #6553
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In spec commit 498f104fd399488c77f449d05cb21c0b604636a2 (August 2021),
Bastien Teinturier removed the requirement that the mutual close fee be
less than or equal the final commitment tx.
We adopted that change in v0.10.2, but we made sure to never offer a fee
under the final commitment tx's fee, so we didn't break older nodes.
However, the closing tx can actually be larger than the final commitment tx!
The final commit tx has a 22-byte P2WKH output and a 34-byte P2WSH output;
the closing can have two 34-byte outputs, making it 4*8 = 32 Sipa heavier.
Previously this would only happen if both sides asked for P2WSH outputs,
but now it happens with P2TR, which we now do.
The result is that we create a tx which is below the finally commitment
tx fee, and may be below minrelayfee (as it was in regtest).
So it's time to remove that backwards-compatibility hack.
Changelog-Fixed: Protocol: We may propose mutual close transaction which has a slightly higher fee than the final commitment tx (depending on the outputs, e.g. two taproot outputs).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6545
Unifies the pay_flow resolve functions, and moves remove_htlc_payflow
and commit_htlc_payflow to the top.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to make sure that on every path, we terminate the flow. The simplest
way to do this is encourage the pattern "return pay_flow_xxx(flow)".
Indeed, this caught a few places I missed!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The main function here is payment_reconsider:
* Each payment has a list of pay_flow.
* This is populated in try_paying(), calling add_payflows & sendpay_new_flows.
* When we get a notification, we resolve a pay_flow using one of the pay_flow_failedxxx
or pay_flow_succeeded functions.
* They call payment_reconsider() which cleans up finished flows decides what to do:
often calling try_paying again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's not required, but it should be there so we might as well use it
(though we sometimes don't put one in, esp if it's a private channel).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Avoids a gratuitous "ctx" field, and the simplified declaration
is now understood by `make update-mocks`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Treat it just like "PAY_TRY_OTHER_ROUTE", except it is from the final node:
this means we correctly process that it "succeeded".
Add a test: this crashes sometimes, but it's cleaned up soon...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As recommended by your TODO, a bit simpler: we also make the hash function
return a ptr rather than the (now rather large) struct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Use json_scan(), and use the new pay_flow_from_notification() routine.
Also, the tal_dup_or_null can be tal_dup, since &preimage is never NULL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are a few fields in `struct renepay` which are genuinely
transient, but it makes the code much harder to follow than simply
having a single structure.
More cleanups will follow, but this is the minimal set.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The general pattern for xxx_new is that it should populate all
fields, for encapsulation and so you never can have a half-formed
object.
This means a fair bit of work for now, but it pays off in the next
patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You cannot refresh the gossmap with localmods applied, nor apply localmods
when others have applied localmods in the same process.
There are optimizations we could do, but for now always apply/unapply before
querying gossmap.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
dump_our_gossip() is mainly useful for propagating our gossip when we
are poorly connected, not when we have many peers. @whitslack
reported excessive memory use queueing messages on a large node, so we
limit it beyond the first 5 peers, to 5 channels each.
This assumes we have ~ the same number of peers as channels, which
is probably reasonable.
In the long term, we should move this to connectd, which is properly
equipped to trickle out these messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6540
This was recommended by @t-bast: if the final spec commits to something
compatible, we can simply advertize and accept both features, but if it
does change in incompatible ways we won't cause problems for nodes
who implement the official spec.
(I split this, so first, we remove the OPT_SPLICE entirely, to make
sure we caught them all. --RR)
Suggested-by: @t-bast
Changelog-None
We've stomped errno, so if exec fails we don't get a reliable result:
```
2023-08-07T17:58:45.713Z **BROKEN** plugin-bcli: bitcoin-cli exec failed: Bad file descriptor
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
EXPERIMENTAL_SPLICING=1 turns it on for *all* tests, to make sure we don't
accidentally break those. But we can (and should!) run the splice test
under every possible CI scenario.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The nomenclature confusion mean that we were ANDING a capability
with a message number (29) which always returned non-zero. We really
do need a new capability which we can hand to channeld to make these
splice txs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I obviously like the word "capabilities" since I reused it to refer
to the HSM's overall features :(
Suggested-by: @ksedgwic
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
See: https://github.com/bitcoindevkit/bdk/issues/1047#issuecomment-1660645669
In general, futures produced by most libraries in the ecosystem of Rust, and bounds placed
on users of famous runtimes like tokio and its spawn method all lack Sync requirements.
Because of this, anyone who creates a callback using any sort of library that returns a
non-Sync future (which most libraries fit this description) inside of it will get some
cryptic error messages (async error messages still leave a lot to be desired).
Removing these Sync requirements will make the library more useful.
Apparently MacOS doesn't always have fdatasync, so use fsync. Even more importantly
check whether it succeeds!
Fixes: #6516
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed this while debugging an issue with ACINQ, that we got upset,
but didn't trigger a reconnect cycle.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: We now close connection with a peer if adding an HTLC times out (which may be a TCP connectivity issue).
This make ACINQ seize up, and not send revoke_and_ack. Eventually,
this can cause a bad signature error, should payments go in both
directions, which is a separate bug, but this is the trigger.
See: #6500
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use this file as a proxy for breaking changes in the signer
protocol. It may not catch all the breaking changes, but it's a
good first approximation.
In this case, the user's default was info, but they specifically asked for debug
from one plugin. Since there were no per-file filters, it set filtering to the
default level, info, and rejected it. Since it's been explicitly filtered in,
we need to pass it at this point.
Reported-by: @wtogami
Fixes: #6503
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Steps to Reproduce: Run `sudo make install` then again `sudo make install` without uninstalling plugins in between.
First command will install the plugin at `/usr/local/libexec/c-lightning/plugins/clnrest` as expected but the
second command will install it inside the first folder like `/usr/local/libexec/c-lightning/plugins/clnrest/clnrest`.
Fix: Check and delete if the folder already exists in `install-program`.
Changelog-None
Fixes 32-bit builds:
```
In file included from plugins/renepay/pay.c:5:
./plugins/renepay/pay_flow.h: In function 'fmt_payflow_key':
./plugins/renepay/pay_flow.h:54:17: error: format '%ld' expects argument of type 'long int', but argument 4 has type 'u64' {aka 'long long unsigned int'} [-Werror=format=]
54 | "key: groupid=%ld, partid=%ld, payment_hash=%s",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
55 | k->groupid,k->partid,
| ~~~~~~~~~~
| |
| u64 {aka long long unsigned int}
```
etc
This was changed by mistake in 23fafe98e3: if
it's null we turn it into 0 (which is what the default call does, but it
does log BROKEN about it!):
```
2023-08-03T14:10:49.001Z **BROKEN** lightningd: Accessing a null column total_msat/15 in query SELECT id, status, destination, msatoshi, payment_hash, timestamp, payment_preimage, path_secrets, route_nodes, route_channels, msatoshi_sent, description, bolt11, paydescription, failonionreply, total_msat, partid, local_invreq_id, groupid, completed_at FROM payments ORDER BY id;
```
Fixes: #6501
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since it's only for transitory splicing info, the new name makes sense.
```
cc channeld/channeld.c
In file included from channeld/channeld.c:23:
./channeld/splice.h:37:8: error: redefinition of 'splice'
struct splice {
^
/usr/include/sys/socket.h:140:8: note: previous definition is here
struct splice {
^
```
Reported-by: @grubles
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6486
This was strongly recommended by Russell O'Connor: the "ms" implies that
it's a BIP-32 master secret, and this is CLN specific.
If we changed the hrp to "cln" it would be better, but apparently that
means we no longer fit in a "standard billfold metal wallet" (and
our code assumes a 2-byte prefix anyway).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's expressed in bits, but really it's clearer as a quantity, given
how it's used.
Suggested-by: @Lagrang3
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
ccan/ccan/base64/base64.c:34:10: error: result of comparison of constant 255 with expression of type 'int8_t' (aka 'signed char') is always false [-Werror,-Wtautological-constant-out-of-range-compare]
if (ret == (char)0xff) {
~~~ ^ ~~~~~~~~~~
ccan/ccan/base64/base64.c:44:57: error: result of comparison of constant 255 with expression of type 'const signed char' is always true [-Werror,-Wtautological-constant-out-of-range-compare]
return (maps->decode_map[(const unsigned char)b64char] != (char)0xff);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~
```
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It really has to be 0, since it's the complete secret. And we didn't handle
it well, (`a` would be treated as 0, for example!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Thread the signed tx through so close's JSON return contains that,
rather than the unsigned channel->last_tx.
We have to split the "get cmd_id" from "resolve the close commands" though;
and of course, as before, we don't actually print the txids of multiple
transactions even though we may have multi in flight due to splice!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `close` returns a `tx` field with witness data populated (i.e. signed).
Fixes: #6440
Update the lightningd <-> channeld interface with lots of new commands to needed to facilitate spicing.
Implement the channeld splicing protocol leveraging the interactivetx protocol.
Implement lightningd’s channel_control to support channeld in its splicing efforts.
Changelog-Added: Added the features to enable splicing & resizing of active channels.
New daemon process means we don’t have to deal with gossip, so that gets removed along with error cleanup and a refactoring of how we calculating PDBT diffs.
Update gossip routiens and various other hecks on the channel state to consider AWAITING_SPLICE to be routable and treated similar to CHANNELD_NORMAL.
Small updates to psbt interface
Changelog-None
Using the -n flag on echo is non-standard for zsh. printf ‘%s’ accomplishes the equivilent thing.
Changelog-Added: Small fix for Mac OS building
# Conflicts:
# plugins/Makefile
It's confusing: we can (and should) load this before other operations, though we don't actually need to yet. But more importantly, don't put it under the "outpointfilters_init" trace span.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you actually ran your node with the botched "last_invoice_created_index" typo migration
(fortunately, not release, just master) you can get a db with both the real "last_invoices_created_index" and the bad "last_invoice_created_index" entries.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- adopt "const <type> *"convention
- remove use_shadow option for some pyln tests
- show prob. information of flows into paynotes
- show prob. of success of entire payment flow in paynotes
- minflow: We were not releasing the memory of flow arrays when replacing
them with a new canditate.
- use memleak_scan_obj in memleak_check
- replace u64 with size_t
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
The global is an *internal* hack because dijkstra_item_mover doesn't
take a context arg! It should be used with care.
Easy, since all the accessors exist: we just hand in the struct dijkstra.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- remove internal gheap checks
- add check for arc_t.chanidx overflow
- remove outdated comments
- check the delta flow bounds before augmenting along a path
- get_flow_paths uses a dynamic tal array instead of a list.
- fix a unit test that depended on the order of returned flows
- fix bug: lightnind doesn't like if I reuse the partid of a failed
flow, therefore use a higher partid than any of the previous attempts.
- plugin_err instead of LOG_BROKEN if sendpay fails and we cannot get a
an error code.
- fix wrong comments.
- remove the background timer.
- This is a bugfix. Previous to this the MCF network was built using the
knowledge of the min and max liquidity but it didn't take into account
pending HTLCs.
- Also remove the min_prob_success option but hardcode a 90% value.
Removing some options that are not relevant to the user, they're kept
for developer mode only:
- base_fee_penalty
- min_prob_success
- prob_cost_factor
- remove heap.h, not used
Signed-off-by: Lagrang3 <eduardo.quintana@pm.me>
Firstly, I wanted the results easier to use:
1. Make them always lower case, even if the string was UPPER.
2. Decode the payload for them.
3. Don't give the user any fields they don't need, and make
the field sizes explicit.
Secondly, I wanted to avoid the pattern of "check in one place, assume
in another", in favour of "check on use".
So, I changed the code to lower the string if it needs to at the start,
and then changed the pull functions so we always use them to get data:
this way we should fail clearly and gracefully if we don't have enough data.
I made all the checks explicit, where we assign the fields.
I also addressed the FIXME: I think the array is *often* one shorter,
but not always, so I trim the last byte at the end if needed.
[ Aditya modified the tests to work ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Nothing major here:
1. size_t for lengths.
2. pass engine to checksum_verify, as caller wants ->len (avoid repeating 13/15 magic numbers).
3. Use x.member instesad of (&x)->member.
4. Return memcmp result directly instead of if.
5. Spacing removal, `;;` removal.
6. codexl is a bool `true`/`false` not 0/1 (it's the same, but clearer)
7. Make sanity_check assign *fail directly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Clean restart of daemon after a tx-abort is a nice way to work around
the 'persistent' disconnect that we t-bast noticed.
Changelog-Fixed: `dualopend`: Fix behavior for tx-aborts. No longer hangs, appropriately continues re-init of RBF requests without reconnction msg exchange.
Bug Report:
- initiate a channel open eclair -> cln
- wait for the transaction to be published
- eclair initiates rbf, and cancels it by sending tx_abort before exchanging commit_sig
- at that point everything looks good, cln echoes the tx_abort and stays connected
- eclair initiates another RBF attempt and sends tx_init_rbf: for some unknown reason,
cln answers with channel_reestablish (??) followed by an error saying
"Bad reestablish message: WIRE_TX_INIT_RBF"
Diagnosis:
CLN is doing a reconnect after a tx-abort is sent.
Extra Test:
Realized that if we abort, we won't correctly advanced to NORMAL if
blocks are mined while we're in hanging state. CLN should advance
after block containing channel open is mined.
Reported-By: @t-bast
This way unreserving the PSBT will work as intended, and we don't have
to keep track of how many times we've called reserved for any one input.
Technically we're supposed to not reserve inputs at *all* while doing
opens, this moves us slightly closer to that.
Also as update_own_node_announcement is called nearly continuously
under normal operation by maybe_send_own_node_announce, the timer should
not be freed continuously - better to only free before actually
refreshing.
When an outdated own node announcement is present, it fails the
nannounce_different test and also fails to kick off the forced regen
timer.
Changelog-Fixed: Node announcements are refreshed more reliably.
This will at least *help* the case where these were not populated, causing us
to send errors without channel_updated appended.
It's not perfect: we can still send such errors if the gossip store is
corrupted, and we still send them for private channels, but it should
help.
(The much better fix is far more invasive, so slips to next release!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The alias may not be set for non-alias channels after they
confirm. The other branch is safe because we only consider active
channels.
Changelog-None
Fixes#6450
Make sure we've completely processed htlc, so we will definitely consider it an old spend. If we're too fast, l2 might consider it a legitimate unilateral close:
```
# Make sure both sides got revoke_and_ack for final.
l1.daemon.wait_for_log('peer_in WIRE_REVOKE_AND_ACK')
l2.daemon.wait_for_log('peer_in WIRE_REVOKE_AND_ACK')
# Now we really mess things up!
bitcoind.rpc.sendrawtransaction(tx)
bitcoind.generate_block(1)
l2.daemon.wait_for_log(' to ONCHAIN')
# FIXME: l1 should try to stumble along!
# l2 should spend all of the outputs (except to-us).
# Could happen in any order, depending on commitment tx.
needle = l2.daemon.logsearch_start
((_, txid1, blocks1), (_, txid2, blocks2)) = \
> l2.wait_for_onchaind_txs(('OUR_PENALTY_TX',
'THEIR_REVOKED_UNILATERAL/DELAYED_CHEAT_OUTPUT_TO_THEM'),
('OUR_PENALTY_TX',
'THEIR_REVOKED_UNILATERAL/OUR_HTLC'))
tests/test_closing.py:687:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:1264: in wait_for_onchaind_txs
r = self.daemon.wait_for_log('Telling lightningd about {} to resolve {}'
contrib/pyln-testing/pyln/testing/utils.py:346: in wait_for_log
return self.wait_for_logs([regex], timeout)
```
You can see l2 here:
```
lightningd-2 2023-07-27T03:34:24.533Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-onchaind-chan#1: Their unilateral tx, old commit point
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The fields were missing because they weren't annotated with a type and
a description. Adding those fixes them.
Changelog-Fixed: msggen: `listpays` now includes the missing `amount_msat` and `amount_sent_msat` fields
No-schema-diff-check: fields were always there, just undocumented!
Abstracts search and directory traversal. Adds support for installing
from a local git repository, a local directory, or a web hosted git repo
without relying on an api.
Changelog-Changed: Reckless can now install directly from local sources.
This cause of cascading failure was pointed out by @t-bast: if fees spike and
you don't timeout an outgoing onchain HTLC, you should nonetheless fail the incoming htlc
because otherwise the incoming peer will close on you.
Of course, there's a risk of losing funds, but this only happens if you weren't going to get the HTLC spend in time anyway. And it would also catch any other reason that the downstream onchain goes wrong, containing the damage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @t-bast
Changelog-Fixed: Protocol: We will close incoming HTLCs early if the outgoing HTLC is stuck onchain long enough, to avoid cascating failure.
The test actually triggers this:
1. We don't get our commitment tx mined at all (we block it).
2. By the time the peer does, the HTLC is expired.
3. We have the preimage but we don't even try, since it's expired.
We should at least *try* to collect the HTLC in this case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Caught by leak detection, we just re-assigned this when we retried: sure,
it's temporary, but it's technically a leak.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I shut down bitcoind during a test, and bcli leak reports flooded in.
They're all temporary, but this fixes them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This update incorporates the proposed version of lnprototest from
the patch [1], which includes the following fixes:
- Corrects the `ExpectError` event and updates BOLT 7 to expect a
warning instead of an error.
- Implements a new test for when the runner sends a bad signature
within the announcement_signatures message.
[1] https://github.com/rustyrussell/lnprototest/pull/100
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This commit addresses an issue to enhance the resilience of core
lightning when receiving node announcements.
According to BOLT 7 (The announcement_signatures Message),
if the node_signature OR the bitcoin_signature is NOT correct,
it is recommended to either send a warning and close the connection or send an error and fail the channel.
In this commit, we take a strict approach. If any error is detected, we
send an error and fail the open channel operation.
This is because the announcement_signatures operation is optional,
and we assume that it must be correct.
lnprototest at commit dea47c29b5541dbfe7fe53cc2598330e897fa4f4 report
the following error now.
```
2023-07-06T21:03:20.930Z DEBUG hsmd: Shutting down
ERROR root:helpers.py:170 Traceback (most recent call last):
File "/home/vincent/Github/lightning/external/lnprototest/tests/helpers.py", line 167, in run_runner
runner.run(test)
File "/home/vincent/Github/lightning/external/lnprototest/lnprototest/runner.py", line 99, in run
all_done = sequence.action(self)
^^^^^^^^^^^^^^^^^^^^^
File "/home/vincent/Github/lightning/external/lnprototest/lnprototest/structure.py", line 55, in action
all_done &= e.action(runner)
^^^^^^^^^^^^^^^^
File "/home/vincent/Github/lightning/external/lnprototest/lnprototest/event.py", line 365, in action
raise EventError(self, "{}: message was {}".format(err, msg.to_str()))
lnprototest.errors.EventError: `Expected msgtype-warning, got msgtype-error: message was error channel_id=a37362839b13f61cfe82d35bd397b1264c389b245847cfb6111b38892546dc77 data=4661696c656420746f20766572696679206e6f64655f7369676e61747572652e` on event [{"event": "ExpectMsg", "file": "test_bolt2-01-close_channel.py", "pos": "157"},]
============================================================================================================================================================== short test summary info ===============================================================================================================================================================
FAILED tests/test_bolt2-01-close_channel.py::test_close_channel_shutdown_msg_normal_case_receiver_side - AssertionError: `Expected msgtype-shutdown, got msgtype-error: message was error channel_id=a37362839b13f61cfe82d35bd397b1264c389b245847cfb6111b38892546dc77 data=4661696c656420746f20766572696679206e6f64655f7369676e61747572652e` on event [{"event": "ExpectMsg", "file": "test_bolt2-01-close_channel.py", "pos": "75"},]
FAILED tests/test_bolt2-01-close_channel.py::test_close_channel_shutdown_msg_wrong_script_pubkey_receiver_side - AssertionError: `Expected msgtype-warning, got msgtype-error: message was error channel_id=a37362839b13f61cfe82d35bd397b1264c389b245847cfb6111b38892546dc77 data=4661696c656420746f20766572696679206e6f64655f7369676e61747572652e` on event [{"event": "ExpectMsg", "file": "test_bolt2-01-close_channel.py", "pos": "157"},]
```
Changelog-Fixes: channeld: Verify the signature sent in announcement_signatures by the counterparty
Reported-by: lnprototest
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This is the simplest solution, not the best, but there's significant risk in try to remove the "we have a path" assumption in the code pay code.
Includes removing a `tal_steal` which was incorrect: the buffer has the same lifetime as the plugin, so if we steal it then things get messy when we free the struct payment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `pay` will now pay your own invoices if you try.
Previously, the "payment" and "invoice" paths were completely separate, but this now calls both. It bypasses htlc_sets (and thus, cannot do MPP), and bypasses the hook too: the former is tied closely to HTLCs, and the hook is also very htlc-centric.
Includes finishing unfinished sentence in sendpay man page, as a bonus.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sendpay` now allows self-payment of invoices, by specifying an empty route.
Clean these up: they were debug logs, but we want to pass this information
back for self-payments.
Also fixes "Attept" typo which altered tests!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For non-v0 witness programs we weren't stripping the data push byte
before writing into the fallback address.
According to BIP14, all witness scripts will be data pushes (up to 40-bytes)
so trimming the datapush byte should be kosher.
From BIP141:
A scriptPubKey (or redeemScript as defined in BIP16/P2SH) that
consists of a 1-byte push opcode (for 0 to 16) followed by a
data push between 2 and 40 bytes gets a new special meaning.
The value of the first push is called the "version byte". The
following byte vector pushed is called the "witness program".
Changelog-Fixed: Adding a >0 version witness program to a fallback address now is *just* the witness program, as per bolt11 spec
Pointed out by @ShahanaFarooqui, we leave a single unused entry in the datastore,
so we should clean that up too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If they have invalid runes, we bail, but if they have runes which used
a different master secret (old commando.py allowed you to override
secret), we just complain and delete them.
Note that this requires more mocks in wallet/test/run-db.c...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The wallet_datastore_first() SELECT statement only iterates from the
given key (if any), relying on the caller to notice when the key no
longer applies. (e.g. startkey = ["foo", "bar"] will return key
["foo", "bar"] then ["foo", "bar", "child" ], then ["foo", "baz"]).
The only caller (listdatastore) would notice the keychange and stop
looping, but reallly wallet_datastore_next() should do this. When I
tried to use it for migrations, I got very confused!
Also, several places want a simple "wallet_datastore_get()" function,
so provide that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During migrations, wallet doesn't exist yet, so we use raw db. Split
functions into lower-level ones and make public API a simple wrapper.
Unfortunately, this means db_datastore_next needs to proceed db_datastore_first
since they're now static (and first calls next), plus, fix some weird indents,
so diff is bigger than expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to access this in db migrations, which happen very early, but
runes_init needs the db, creating a circular dependency which must be
split.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means (temporarily) that blacklisting won't work (fix later), and
means that old-style (commando.py) master-secret-override doesn't work.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugins: `commando` no longer allows datastore ['commando', 'secret'] to override master secret (re-issue runes if you were using that!).
In preparation for going async:
1. Split try_command's tail into a new function called execute_command() after
the rune checks have succeeded.
2. Put all the info execute_command() needs into struct cond_info, to make it
a simple callback style.
So we create new_cond_info() which dynamically allocates `struct cond_info`
and sets the destructor.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We would create a `struct commando` to marshal our incoming messages,
then try_command would create a *new* one. We can simply reuse, but
when I did I noticed a trick: the new one was not in the `incomings`
array, so didn't work towards the ratelimit. So we need to remove it
from `incomings` in `try_command`, but at least it's now explicit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to activate on the first rune creation, but we're no longer in charge
of runes, so we can't make that call.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And do them on the first run (where we check parameters), instead
of every time. Might as well do them in non-developer mode too,
since they're simply programmer correctness.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The uid isn't enough: it could be someone else's rune. This is tested
in the command rune list tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make this a bit less difficult to see what's going on; use common
variables for the paths etc.
Also make the final `rm` step recursive, same as the above find step
which patches the files.
Files are in `contrib/pyln-grpc-proto/pyln/grpc`, but the upper level
find command acts at a level above.
At cold start, if your node is behind the blocktip and you've sent your
peer a blockheight counter from the future, we shouldn't confuse ourselves
with our rollback/replay.
Should fix flakes in CI that were spotting BROKEN blockheight updates.
Logs below from a previuos CI fail (edited for relative clarity)
The one that sasy "{ SENT_ADD_ACK_REVOCATION:111 }, our current 108` is
the tell; the last line is the node finally catching up to the tip.
In the test we get into this state by stopping and restarting the node.
```
2023-07-22T11:24:28.2754533Z lightningd-1 2023-07-22T11:19:34.188Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#2: Already have funding locked in
2023-07-22T11:24:28.2755486Z lightningd-1 2023-07-22T11:19:34.188Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#2: attempting update blockheight a5b23dff5177badd6df725c
efeb83ceccbfc52dc64a16b38894a41f0ad8fa181
2023-07-22T11:24:28.2755778Z lightningd-1 2023-07-22T11:19:34.188Z DEBUG lightningd: update_blockheight: height = 108
2023-07-22T11:24:28.2766210Z lightningd-1 2023-07-22T11:19:34.210Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: init LOCAL: remote_per_commit = 029563e7c898
5d8b95bdfe19e47e494bb8ec8d53ff4edb93f156be57667bfee8c9, old_remote_per_commit = 02bf3117c149d324361f0b418db8984b1e29af70c773eb2865a41ff7f583c7c9ed next_idx_local = 3 next_idx_remote = 3 revocations_recei
ved = 2 feerates { SENT_ADD_ACK_REVOCATION:3750 } range 253-150000 blockheights { SENT_ADD_ACK_REVOCATION:111 }, our current 108
2023-07-22T11:24:28.2768866Z lightningd-1 2023-07-22T11:19:34.211Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: peer_out WIRE_CHANNEL_REESTABLISH
2023-07-22T11:24:28.2769416Z lightningd-1 2023-07-22T11:19:34.211Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: billboard: Sent reestablish, waiting for the
irs
2023-07-22T11:24:28.2771115Z lightningd-1 2023-07-22T11:19:34.212Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: peer_in WIRE_CHANNEL_REESTABLISH
2023-07-22T11:24:28.2774150Z lightningd-1 2023-07-22T11:19:34.212Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: Got reestablish commit=3 revoke=2
2023-07-22T11:24:28.2776056Z lightningd-1 2023-07-22T11:19:34.212Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: next_revocation_number = 2
2023-07-22T11:24:28.2805639Z lightningd-1 2023-07-22T11:19:34.239Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 108 less than last 111
2023-07-22T11:24:28.2823960Z lightningd-1 2023-07-22T11:19:34.240Z DEBUG lightningd: Adding block 109: 5f67b6e110eb3c3457bea4fcf0d04ce9be90efeee5df8e083ed4266074ca911f
2023-07-22T11:24:28.2833154Z lightningd-1 2023-07-22T11:19:34.251Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 108 less than last 111
2023-07-22T11:24:28.2833630Z lightningd-1 2023-07-22T11:19:34.252Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: Trying commit
2023-07-22T11:24:28.2834165Z lightningd-1 2023-07-22T11:19:34.252Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: current blockheight 108 less than last 111
2023-07-22T11:24:28.2835070Z lightningd-1 2023-07-22T11:19:34.252Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: Can't send commit: nothing to send, feechange not wanted ({ SENT_ADD_ACK_REVOCATION:3750 }) blockheight not wanted ({ SENT_ADD_ACK_REVOCATION:111 })
2023-07-22T11:24:28.2835516Z lightningd-1 2023-07-22T11:19:34.350Z DEBUG lightningd: Adding block 110: 5f43f3ac9d808e3a309720d1b0727a00d5a3d3ddca71d97401e233637e87639c
2023-07-22T11:24:28.2835962Z lightningd-1 2023-07-22T11:19:34.476Z DEBUG lightningd: Adding block 111: 55b0d1e0a08ff6233e186e6735cb1cbec33e2b0a6e7d08f2622e8c1db30b54b9
```
This looked like a test flake, but was real:
```
l1.daemon.wait_for_log("closing soon due to the funding outpoint being spent")
# We won't gossip the dead channel any more (but we still propagate node_announcement). But connectd is not explicitly synced, so wait for "a bit".
time.sleep(1)
> assert len(get_gossip(l1)) == 2
E assert 4 == 2
```
We can see that two channel_updates come in *after* we mark it dying:
```
gossipd: channel 103x1x0 closing soon due to the funding outpoint being spent
gossipd: REPLY WIRE_GOSSIPD_NEW_BLOCKHEIGHT_REPLY with 0 fds
022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Received channel_update for channel 103x1x0/0 now DISABLED
022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Received channel_update for channel 103x1x0/1 now DISABLED
```
We should keep marking channel_updates the same way.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We get intermittant reports of subd->conn being leaked, but I could never find it.
That's because it's actually subd which is not referenced any more: subd->conn
gets reported because it's subd's tal_parent (and, except for the reference in
subd, not referenced either).
The real issue is that the channel->owner is reassigned to the new subdaemon,
and the old one is still exiting. During that time, we can see a "leak".
```
- Node /tmp/ltests-hkr089bp/test_sql_1/lightning-3/ has memory leaks: [
{
"backtrace": [
"ccan/ccan/tal/tal.c:477 (tal_alloc_)",
"ccan/ccan/io/io.c:91 (io_new_conn_)",
"lightningd/subd.c:774 (new_subd)",
"lightningd/subd.c:828 (new_channel_subd_)",
"lightningd/dual_open_control.c:3662 (peer_restart_dualopend)",
"lightningd/peer_control.c:1161 (connect_activate_subd)",
"lightningd/peer_control.c:1273 (peer_connected_hook_final)",
"lightningd/plugin_hook.c:213 (plugin_hook_callback)",
"lightningd/plugin.c:591 (plugin_response_handle)",
"lightningd/plugin.c:702 (plugin_read_json_one)",
"lightningd/plugin.c:747 (plugin_read_json)",
"ccan/ccan/io/io.c:59 (next_plan)",
"ccan/ccan/io/io.c:407 (do_plan)",
"ccan/ccan/io/io.c:417 (io_ready)",
"ccan/ccan/io/poll.c:453 (io_loop)",
"lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)",
"lightningd/lightningd.c:1249 (main)"
],
"label": "ccan/ccan/io/io.c:91:struct io_conn",
"parents": [
"lightningd/lightningd.c:107:struct lightningd"
],
"value": "0x556c63c859f8"
}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
channeld WARNING: Bad update_fail_malformed_htlc failure code 4103
```
Warren Togami reports this happening with Bitrefill on every reconnect,
so it's clearly something LND does.
(4103 is TEMPORARY_CHANNEL_FAILURE, which does not belong in update_fail_malformed_htlc).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We allow update_fail_malformed_htlc with invalid error codes (LND?)
GH was complaining about Node v12 being used in v2.2.4. We used to
hold back from this upgrade because v3 was breaking quotas, but we
should try again.
The channel selection did not consider the amounts that we are trying
to transfer, which in a multiplexed channel world could end up always
selecting a channel that is too small for the payment. We also log
which channel was selected based on the selector that is passed in,
allowing us to better follow the decisions.
Changelog-Fixed: pay: `sendonion` and `sendpay` will now consider amounts involved when using picking one channel for a peer
Debugging a number of payments showed that we sometimes waste a number
of attempts routing through a channel via its alias, rather than its
scid. This is because while we annotate the scid when it has been set,
we do not do so for the alias. The alias then is picked for routing
despite not having enough capacity, failing the attempt locally.
It can also happen that we alternate between scid and alias, doubling
the number of failed attempts before we can make progress.
This patch sets the hint for the alias to a capacity of 0 and disables
it as if the peer were offline. This means when available we'll always
use the scid, which is also far easier to read in the logs.
Changelog-Fixed: pay: We now track spendable amounts when routing on both the local alias as well as the short channel ID
If you miss a wait event, you can catch up by doing listinvoices and
getting the max of these fields. It's also a good debugging clue.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we have defined ordering, we can add a start param.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listinvoices` has `index` and `start` parameters for listing control.
This will initially be for listinvoices, but can be expanded to other
list commands.
It's documented, but it makes promises which currently don't exist:
* listinvoice does not support `index` or `start` yet.
* It doesn't actually fire when invoices change yet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `wait`: new generic command to wait for events.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setchannel` adds a new `ignorefeelimits` parameter to allow peer to set arbitrary commitment transaction fees on a per-channel basis.
Recent cppcheck doesn't like our code; until we fix that, make it easy
to run every other source check.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We tried to fix this flake before, but now it actually happened again it shows that
b5845afd43 wasn't correct.
```
# If this happens fast enough, connect fails with "disconnected
# during connection"
try:
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
except RpcError as err:
> assert "disconnected during connection" in err.error
E assert 'disconnected during connection' in {'code': 402, 'message': 'disconnected during connection'}
E + where {'code': 402, 'message': 'disconnected during connection'} = RpcError("RPC call failed: method: connect, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'host': 'localhost', 'port': 41849}, error: {'code': 402, 'message': 'disconnected during connection'}").error
tests/test_misc.py:2728: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
l1 shuts down too fast, channeld doesn't get to tell gossipd about the channel,
and l2 sends (private) update_channel and we complain:
```
lightningd-2 2023-07-20T03:42:37.744Z DEBUG gossipd: received private channel announcement from channeld for 103x1x0
lightningd-2 2023-07-20T03:42:37.791Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-hsmd: Got WIRE_HSMD_CUPDATE_SIG_REQ
lightningd-2 2023-07-20T03:42:37.796Z DEBUG hsmd: Client: Received message 3 from client
lightningd-1 2023-07-20T03:42:37.857Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x0/0
lightningd-1 2023-07-20T03:42:37.864Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x0/0
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Writing to the gossip_store file is not explicitly synchronized, so it
seems that connectd has not caught up with the dying flags we've set.
Simply wait for a second. Hacky, but should work.
```
def test_gossip_not_dying(node_factory, bitcoind):
l1 = node_factory.get_node()
l2, l3 = node_factory.line_graph(2, wait_for_announce=True)
l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
# Wait until it sees all the updates, node announcments.
wait_for(lambda: len([n for n in l1.rpc.listnodes()['nodes'] if 'alias' in n])
+ len(l1.rpc.listchannels()['channels']) == 4)
def get_gossip(node):
out = subprocess.run(['devtools/gossipwith',
'--initial-sync',
'--timeout-after=2',
'{}@localhost:{}'.format(node.info['id'], node.port)],
check=True,
timeout=TIMEOUT, stdout=subprocess.PIPE).stdout
msgs = []
while len(out):
l, t = struct.unpack('>HH', out[0:4])
msg = out[2:2 + l]
out = out[2 + l:]
# Ignore pings, timestamp_filter
if t == 265 or t == 18:
continue
# channel_announcement node_announcement or channel_update
assert t == 256 or t == 257 or t == 258
msgs.append(msg)
return msgs
assert len(get_gossip(l1)) == 5
# Close l2->l3, mine block.
l2.rpc.close(l3.info['id'])
bitcoind.generate_block(1, wait_for_mempool=1)
l1.daemon.wait_for_log("closing soon due to the funding outpoint being spent")
# We won't gossip the dead channel any more (but we still propagate node_announcement)
> assert len(get_gossip(l1)) == 2
E assert 4 == 2
E + where 4 = len([b'\x01\x01L\xc2\xbe\x08\xbb\xa8~\x8f\x80R\x9e`J\x1cS\x18|\x12\n\xe5_6\xb0\xa6S\x9fU\xae\x19\x9c\x1fXB\xab\x81N\x13\xdc\x8e}\xb9\xb0\xb6\xe6\x14h\xd4:\x90\xce\xc3\xad\x9ezR`~\xba@\xc9\x91e\x89\xab\x00\x07\x88\xa0\x00\n\x02i\xa2d\xb8\xa9`\x02-"6 \xa3Y\xa4\x7f\xf7\xf7\xacD|\x85\xc4l\x92=\xa53\x89"\x1a\x00T\xc1\x1c\x1e<\xa3\x1dY\x02-"SILENTARTIST-27fc801-modded\x00\x00\x00\x00\x00\x00\x00', b'\x01\x01M\x00\x86\x8e4\xc8\x90p\n\x98\xf7\xce4\x1e\xd9\xd6-6\xfb(\xf0\xe4\xb7\x90\x7f\x89\xb9\xfa\x00\x82\x1b\xeb\x1fY\x93\x1e\xe0c\xb2\x0e<\xe6\x06x\xb7\xe54};\xfbd\xa0\x01S\xcf\xe8{\xf8\x8f/\xa7\xc0\xe2h\x00\x07\x88\xa0\x00\n\x02i\xa2d\xb8\xa9`\x03]+\x11\x92\xdf\xba\x13N\x10\xe5@\x87]6n\xbc\x8b\xc3S\xd5\xaavk\x80\xc0\x90\xb3\x9c:]\x88]\x03]+HOPPINGFIRE-27fc801-modded\x00\x00\x00\x00\x00\x00\x00\x00', b'\x01\x02~\xe0\x13\xb4\x84Gz\xcf(\xd4w\xa7\x9bZ\x1a\xe82\xd1\xe1\x1bLm\xc8\n\xcd\xd4\xfb\x88\xf8\xc6\xdbt\\v\x89~\xd1.e\xc8\xa8o\x9c`\xd5\xa8\x97\x11l\xf2g\xcb\xa8\xcf\r\x869\xd3\xb5\xd5\x9a\xa0my\x9f\x87\xebX\x0b\x9e_\x11\xdc!\x1e\x9f\xb6j\xbb6\x99\x99\x90D\xf8\xfe\x14h\x01\x16#\x936B\x86\xc6\x00\x00g\x00\x00\x01\x00\x00d\xb8\xa9d\x01\x02\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\n\x00\x00\x00\x00;\x023\x80', b"\x01\x0284\xf1a\x86z\x8e\xf2\xe5'\xf7\xfe1\x8d\x96R\x0c\xe7\x1fj#\xaf\xbd/\xba\x10e\xd1\xccQ-\xcf/>\xa5g\xc6\xd8\x9cO \xe7~\xb3\xda\xe0\\vg\xfb\x02&T\x93\xa0\xd4\x95\x8e\xd5L\x12\x9a\xf7\xe6\x9f\x87\xebX\x0b\x9e_\x11\xdc!\x1e\x9f\xb6j\xbb6\x99\x99\x90D\xf8\xfe\x14h\x01\x16#\x936B\x86\xc6\x00\x00g\x00\x00\x01\x00\x00d\xb8\xa9d\x01\x03\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\n\x00\x00\x00\x00;\x023\x80"])
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rune functionality moved into core from commando plugin.
Changelog-Added: JSON-RPC: `checkrune`: check rune validity for authorization; `createrune` to create/modify rune; `listrunes` to list existing runes; `blacklistrune` to revoke permission of rune
Changelog-Deprecated: JSON-RPC: `commando-rune`, `commando-listrunes` and `commando-blacklist`.
No-schema-diff-check
This extracts the core checking functionality for a rune, so they can
easily be used more widely than just commando.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We also document how the grpc test mode works, and why it currently
lacks coverage.
Changelog-Changed: pyln-testing: The grpc dependencies are now optional.
The presplitter modifier would split a payment before trying the first
attempt based on some common sizes. Its goal was to have smaller parts
in flight over different paths, in order to make it more difficult for
a forwarding node to learn payment amount. However it was causing some
issues for direct payments, and estimates on spendable amounts which
considers only the first HTLC being added, but presplitter would
always cause multiple HTLCs to be kicked off, causing the estimate to
be off.
Removing the presplitter fixes this, making draining channels easier,
and worse success rates, due to more HTLCs in flight directly
impacting the changes of getting stuck.
Changelog-Removed: pay: `pay` no longer splits based on common size, as it was causing issues in various scenarios.
These tests make assumptions about the presplitter behavior which
we'll remove in the next commit. We remove them here so we don't cause
temporary breaks in the git history.
While one side was not produced by us, we have a vested interest in propagating it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: When we send our own gossip when a peer connects, also send any incoming channel_updates.
@endothermicdev and I found this while investigating a "nobody sees my node_announcement" bug report.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6410
Reported-by: benjaminchodroff on discord
Changelog-Fixed: Protocol: When we send our own gossip when a peer connects, send our node_announcement too (regression in v23.05)
Alex and I were reading it and I got confused: it's really a simpler loop
than it seems, with all those redundant `continue` statements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6368
Changelog-Fixed: Protocol: we no longer gossip about recently-closed channels (Eclair gets upset with this).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't actually delete them for 12 blocks, but we can't avoid
propagating them. We don't mark node_announcements, which is a bit
weird, but avoids us tracking logic to un-dying them if a channel is
opened.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this, only per-peer daemons were filtered correctly. For generic
daemons, we need to filter with the actual nodeid they use (if any).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: config: `log-level` filters now apply correctly to messages from `connectd`.
Rather than initializating the "print_level" field on first use, we can
do it in logging_options_parsed(), now we have a linked list of them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`struct log` becomes `struct logger`, and the member which points to the
`struct log_book` becomes `->log_book` not `->lr`.
Also, we don't need to keep the log_book in struct plugin, since it has
access to ld's log_book.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently our CI is not able to complete the compilation because
there is the following compilation error introduced in `0bcff1e76d`
```
cc wallet/db.c
wallet/db.c: In function 'migrate_normalize_invstr':
wallet/db.c:1734:3: error: too many arguments to function 'db_bind_text'
1734 | db_bind_text(update_stmt, 0, invstr);
| ^~~~~~~~~~~~
In file included from wallet/db.c:10:
./db/bindings.h:25:6: note: declared here
25 | void db_bind_text(struct db_stmt *stmt, const char *val);
| ^~~~~~~~~~~~
wallet/db.c:1735:3: error: too many arguments to function 'db_bind_u64'
1735 | db_bind_u64(update_stmt, 1, id);
| ^~~~~~~~~~~
In file included from wallet/db.c:10:
./db/bindings.h:23:6: note: declared here
23 | void db_bind_u64(struct db_stmt *stmt, u64 val);
| ^~~~~~~~~~~
wallet/db.c:1758:3: error: too many arguments to function 'db_bind_text'
1758 | db_bind_text(update_stmt, 0, invstr);
| ^~~~~~~~~~~~
In file included from wallet/db.c:10:
./db/bindings.h:25:6: note: declared here
25 | void db_bind_text(struct db_stmt *stmt, const char *val);
| ^~~~~~~~~~~~
wallet/db.c:1759:3: error: too many arguments to function 'db_bind_u64'
1759 | db_bind_u64(update_stmt, 1, id);
| ^~~~~~~~~~~
In file included from wallet/db.c:10:
./db/bindings.h:23:6: note: declared here
23 | void db_bind_u64(struct db_stmt *stmt, u64 val);
| ^~~~~~~~~~~
make: *** [Makefile:299: wallet/db.o] Error 1
make: *** Waiting for unfinished jobs....
rm external/build-x86_64-linux-gnu/libwally-core-build/src/secp256k1/libsecp256k1.la
```
Fixes: 0bcff1e76d
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This is almost always true already; fix up the few non-standard ones.
This is enforced with an assert, and I ran the entire test suite to
double-check.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can expose the dbid, rather than pretending we have some "struct
invoice" which is actually just the dbid. And don't have a pile of
"wallet_" wrappers for redirection.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
2023-07-14T05:32:54.3688763Z # Set ECONOMICAL/6 feerate, for unilateral_close and htlc_resolution
2023-07-14T05:32:54.3689123Z l1.set_feerates((15000, 11000, 0, 0), True)
2023-07-14T05:32:54.3689484Z feerates = l1.rpc.feerates('perkw')
2023-07-14T05:32:54.3689919Z > assert feerates['perkw']['unilateral_close'] == 11000
2023-07-14T05:32:54.3690226Z E assert 15000 == 11000
2023-07-14T05:32:54.3690391Z
2023-07-14T05:32:54.3690514Z tests/test_misc.py:1572: AssertionError
```
The rough checks in set_feerates don't actually ensure that we've digested
the changes, so copy the check from elsewhere that makes sure
feerates['estimates'] has indeed been updated.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We assume that RBFs will happen in order (txid1, txid2) but that doesn't always happen.
```
for depth in range(2, 10):
bitcoind.generate_block(1)
# l2 should RBF, twice even, one for the l1 main output,
# one for the l1 HTLC output.
# Don't assume a specific order!
start = l2.daemon.logsearch_start
> txid1 = get_rbf_txid(l2, txid1)
tests/test_closing.py:1671:
...
print("({} was previously in logs!)".format(r))
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('RBF onchain .*1fe38fe22852baaedccc3a9fd9d897e46bae5b7ca31daf23e0aa456fb235475e')]" in logs.
contrib/pyln-testing/pyln/testing/utils.py:328: TimeoutError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We hand "estimatefees.feerate_floor" as method, for example, and then
crash instead of reporting the plugin which gave us the bad answer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This commit make a db migration to canonicalize all the
invoice string stored inside the database.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Previously, our code checked for the presence of the `lightning:`
prefix while decoding a bolt11 string. Although this prefix is valid
and accepted by the core lightning pay command, it was causing issues
with how we managed invoices. Specifically, we were skipping the prefix
when creating a copy of the invoice string and storing the raw invoice
(including the prefix) in the database, which caused inconsistencies
in the user experience.
To address this issue, we need to strip the `lightning:` prefix before
calling each core lightning command. In addition, we should
modify the invstring inside the db with the canonical one.
This commit fixes the issue by stripping the `lightning:` prefix
from the `listsendpays` function, which will improve the
user experience and ensure consistency in our invoice management (see
next commit).
Reported-by: @johngribbin
Link: ElementsProject#6207
Fixes: debbdc0
Changelog-Fixes: trim the `lightning:` prefix from invoice everywhere.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
As per cd3c99e722
we should send the next_funding_txid if we've sent our commitment sigs,
but we haven't received the peer's tx_signatures.
Note that we send here, but don't verify that it's arrived.
We fixed the others. There are no fields, but this keeps it consistent.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `shutdown` notification contains `shutdown` object (notification consistency)
Requested-by: Shahana Farooqui @Shahana
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: plugins can subscribe to all notifications using "*".
The lnprototest occasionally fails on our CI due to the
timeout duration we set while waiting for cln to start up.
This timeout is insufficient for machines like the ones
used in the GitHub CI environment.
To address this issue, this commit introduces the use of
environment variables to increase the lnprototest timeout,
ensuring sufficient time for cln to initialize.
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This commit addresses a limitation in our CLI argument
checking for the hsmtool. With this update, we introduce
support for the Signet network.
In addition, it introduce a code semplification by directly passing the
BIP32 version instead of using network testnet flag. This change
improves code readability and minimaze code changes
to support other networks.
Link: https://github.com/ElementsProject/lightning/issues/6371
Reported-by: @grubles
Changelog-Fixes: hsmtool: Add support for Signet network
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
When modifying the schema, regenerating the file can be challenging.
This change sets the autogenerate file as the default target, m
aking the process more convenient and simplifying our workflow.
Fixes https://github.com/ElementsProject/lightning/issues/6365
Report-by: Dusty Daemon
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This is just housekeeping that allows up
to do not spam the logs of people with not
useful information.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
People running master notice that calling listconfigs fails, because
we don't handle objects called xxx_msat correctly (see
d348554ff4). This makes it painful
to test, until we release a pyln-client version.
Fortunately, the three changes in master are all fully backwards compatible,
so we can simply cut a release now and upload to pipy.org.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This silences a compilation warning about accessing a deprecated
field. This was triggering on each build and we'll notice when they go
away, so why upset users with it?
Turns out we resubmit two txs (the commitment tx, and the anchor spend), but only wait
for one of them: if we mine a block before the anchor spend, it doesn't go in:
```
@unittest.skipIf(TEST_NETWORK != 'regtest', 'elementsd anchors unsupported')
@pytest.mark.developer("needs dev_disconnect")
def test_closing_anchorspend_htlc_tx_rbf(node_factory, bitcoind):
...
l1.daemon.wait_for_log('Peer permanent failure in CHANNELD_NORMAL: Offered HTLC 0 SENT_ADD_ACK_REVOCATION cltv 116 hit deadline')
l1.daemon.wait_for_log('Creating anchor spend for CPFP')
wait_for(lambda: len(bitcoind.rpc.getrawmempool()) == 2)
# But we don't mine it! And fees go up again!
l1.set_feerates((3000, 3000, 3000, 3000))
bitcoind.generate_block(1, needfeerate=5000)
l1.daemon.wait_for_log('RBF anchor spend')
l1.daemon.wait_for_log('sendrawtx exit 0')
# And now we'll get it in (there's some rounding, so feerate a bit lower!)
bitcoind.generate_block(1, needfeerate=2990)
> wait_for(lambda: 'ONCHAIN:Tracking our own unilateral close' in only_one(l1.rpc.listpeerchannels()['channels'])['status'])
```
If we mine too fast, gossip can reach a node which considers it too far in the guture. Break it up.
```
@pytest.mark.developer("Too slow without --dev-fast-gossip")
def test_routing_gossip(node_factory, bitcoind):
nodes = node_factory.get_nodes(5)
for i in range(len(nodes) - 1):
src, dst = nodes[i], nodes[i + 1]
src.rpc.connect(dst.info['id'], 'localhost', dst.port)
src.openchannel(dst, CHANNEL_SIZE, confirm=False, wait_for_announce=False)
# openchannel calls fundwallet which mines a block; so first channel
# is 4 deep, last is unconfirmed.
# Allow announce messages.
mine_funding_to_announce(bitcoind, nodes, num_blocks=6, wait_for_mempool=1)
# Deep check that all channels are in there
comb = []
for i in range(len(nodes) - 1):
comb.append((nodes[i].info['id'], nodes[i + 1].info['id']))
comb.append((nodes[i + 1].info['id'], nodes[i].info['id']))
def check_gossip(n):
seen = []
channels = n.rpc.listchannels()['channels']
for c in channels:
seen.append((c['source'], c['destination']))
missing = set(comb) - set(seen)
logging.debug("Node {id} is missing channels {chans}".format(
id=n.info['id'],
chans=missing)
)
return len(missing) == 0
for n in nodes:
> wait_for(lambda: check_gossip(n))
tests/test_gossip.py:721:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
success = <function test_routing_gossip.<locals>.<lambda> at 0x7f3200534ef0>
timeout = 180
def wait_for(success, timeout=TIMEOUT):
start_time = time.time()
interval = 0.25
while not success():
time_left = start_time + timeout - time.time()
if time_left <= 0:
> raise ValueError("Timeout while waiting for {}".format(success))
E ValueError: Timeout while waiting for <function test_routing_gossip.<locals>.<lambda> at 0x7f3200534ef0>
As it did recently:
```
/usr/bin/ld: bitcoin/pubkey.o: in function `pubkey_to_hash160':
/home/rusty/devel/cvs/lightning/bitcoin/pubkey.c:101: undefined reference to `ccan_ripemd160'
/usr/bin/ld: bitcoin/script.o: in function `hash160':
/home/rusty/devel/cvs/lightning/bitcoin/script.c:22: undefined reference to `ccan_ripemd160'
/usr/bin/ld: bitcoin/script.o: in function `bitcoin_wscript_htlc_offer':
/home/rusty/devel/cvs/lightning/bitcoin/script.c:662: undefined reference to `ccan_ripemd160'
/usr/bin/ld: bitcoin/script.o: in function `bitcoin_wscript_htlc_receive':
/home/rusty/devel/cvs/lightning/bitcoin/script.c:780: undefined reference to `ccan_ripemd160'
collect2: error: ld returned 1 exit status
make: *** [Makefile:660: devtools/create-gossipstore] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`deprecated_apis` is now inside `ld`.
```
lightningd/notification.c: In function ‘connect_notification_serialize’:
lightningd/notification.c:60:13: error: ‘deprecated_apis’ undeclared (first use in this function)
60 | if (deprecated_apis)
| ^~~~~~~~~~~~~~~
lightningd/notification.c:60:13: note: each undeclared identifier is reported only once for each function it appears in
lightningd/notification.c: In function ‘disconnect_notification_serialize’:
lightningd/notification.c:97:13: error: ‘deprecated_apis’ undeclared (first use in this function)
97 | if (deprecated_apis)
| ^~~~~~~~~~~~~~~
lightningd/notification.c: In function ‘block_added_notification_serialize’:
lightningd/notification.c:612:13: error: ‘deprecated_apis’ undeclared (first use in this function)
612 | if (deprecated_apis) {
| ^~~~~~~~~~~~~~~
make: *** [Makefile:299: lightningd/notification.o] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
ignore the min fee as specified by the user setting.
We explicitly allow the user to ignore the fee limits, although this comes with inherent risks.
By enabling this option, users are explicitly
I was aware of the potential dangers.
There are situations, such as the one described in [1], where it
becomes necessary to bypass the fee limits to resolve issues like a stuck channel.
BTW experimental-anchors should fix this.
[1] https://github.com/ElementsProject/lightning/issues/6362
Reported-by: @pabpas
Fixes: 64b1ddd761
Link: https://github.com/ElementsProject/lightning/issues/6362
Changelog-Fixes: do not ignore the ignore-fee-limit option during
update_fee
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Added: JSON-RPC: `connect` and `disconnect` notifications now wrap `id` field in a `connect`/`disconnect` object (consistency with other notifications)
This commit is adding the instructions for compiling cln on FreeBSD,
because it looks that we not longer compile with the FreeBSD package
manager, and I have no idea who the mantainer of this package is.
Link: https://github.com/ElementsProject/lightning/issues/6301
Reported-by: @bektar
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Having the grpc bindings in the pyln-testing package was always a bit
strange, however it came with additional issues. Due to the way that
protos are handled by protobuf, any name clash, independently of where
we import the protos, would cause an import error. This usually
happens in diamond-pattern dependencies, and so pull out the generated
files into their own package that everyone else can rely on.
Changelog-None
Updated build release for:
- force-version and force-mtime checks
- zipfile is not optional anymore, it will be created before all other action
- CLN's amd64, arm32v7 and arm64v8 docker setup
- Verify release feature
Updating bionic, focal and jammy cl-repro builds:
- Rust version from 1.62 to 1.65
- Removed git clone, only mount option
- Removed sha256sums generation
Greg Sanders helped debug this:
```
# Payment should succeed.
> l1.bitcoin.generate_block(1, wait_for_mempool=txid1)
tests/test_closing.py:2145:
...
> raise ValueError("Timeout while waiting for {}".format(success))
E ValueError: Timeout while waiting for <function BitcoinD.generate_block.<locals>.<lambda> at 0x7f7cd7271560>
```
The lgos show the HTLC tx doesn't go through because it double-spent an input but didn't spend enough:
```
2023-07-06T03:05:54.3424456Z lightningd-2 2023-07-06T02:57:37.490Z DEBUG plugin-bcli: sendrawtx exit 26 (bitcoin-cli -regtest -datadir=/tmp/ltests-yihsd7f4/test_onchain_middleman_simple_1/lightning-2/ -rpcport=39033 -rpcuser=... -stdinrpcpass sendrawtransaction 0200000000010220f632933db174def3b4bd879ad892e28f4422ee6e844bda27c4e5bdbc754fda030000000001000000975e48aeaced953f7c2b85f20e85b5c241258cb9dbd2ba13de3038daba6316330100000000fdffffff02a1860100000000002200202df545ea882889846c52fc5e111ac07ce07e0c09418ac15743a6f6284c2a4fa739391e000000000016001461ed06c0632af284ec9e192e97758fc04692c8290500473044022064446978d7f15d923237d44d7701e4a09a2d03ebb0a7e2c42e22c67435ad2fcc02201c51002fb72920978c79872e427acd90a13423e841b4198717c1771e7355ba4683473044022059ed9e2c536a71fac3cd63c7349c64a4445f0f936270295518a8aa03607d1e9d022064d318669a7602f585c84fe80aa95487816920c2a8ac26836601fbb369068ff38320478171dbde0e1c243e5c1ae23bcce446ab361197ca10d57c1d8f030cc9ff52158c76a914f5fb7361abefe39af0c3ab31d5be71096deeb4198763ac6721034bbdd4e5a933a3a83f6c3d22714cf23c452cb0c5ac8e429eea14992262787e687c8201208763a9147a3d3592e3d93a525beed57acce3eb70ba1b514288527c21039a62150fd6808d6c68aabd1e9d144c93e84a8f54f4e0f9ec5b3c37eee0a051c752ae6775017eb175ac6851b275680247304402200fde6a943a2f7f0287af2f5c5872326a4b8ad7020a098c4e05c28d2c2a0f2018022053a1263f982b98bb8dfcbb3a66641fe9f298e7ab432a78d21a2e79190d74753f012102397b0449a60d9d35634401bceaf3beb6118fc229b8552bd7122f735808b28aee00000000) error code: -26\nerror message:\ninsufficient fee, rejecting replacement 76f438f176d8f9beabb286f53c81aa7dcb4948d12f034f51753f4dd9071d6a74; new feerate 0.00029576 BTC/kvB <= old feerate 0.00054659 BTC/kvB
```
This is because sometimes we reuse the same UTXO for the anchor push spend as we do for the HTLC. That would be fine, except that we can have bitcoind mine the commitment tx and not the anchor push, and then we fail to replace it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With the exception of fuzzing, make all builds in the `compile` job,
and simply download them in the other steps.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to have to use environment variables to pass through to
github/scripts/build.sh, but now we run ./configure directly it's
clearer to use explicit flags (though some matrixes still use env vars
for simplicity).
We also don't need to set COMPAT, as it's the default (MacOS tests
that we build without it, but otherwise we assume it's on).
And we make `gather` actually depend on all the other steps!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This controls debug flags for the build, rather than --developer,
which is going away.
I thought about making this flag control the RUST_PROFILE too, but
it seems that we want that set to "release" for CI, whereas for the
C code we want --enable-debugbuild.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We usually have access to `ld`, so avoid the global.
The only place generic code needs it is for the json command struct,
and that already has accessors: add one for libplugin and lightningd
to tell it if deprecated apis are OK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We keep several peer pointers, but we just add a hook to NULL them
manually when a peer dies, rather than using voodoo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use a "softref" which is a magic pointer which gets NULL'ed when
the object is freed. But it's heavy, and a bit tricky to use, and we
only use it in gossipd.
Instead, keep the nodeid, and do a lookup (now that's fast) if we want
to credit the sender for valid gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We will access the freed connection to gossipd. This is weird to track
down when the *actual* issue is that gossipd died!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids the mess where we override db_fatal for teqsts, and keeps it
generic.
Also allows us to get rid of one #if DEVELOPER, and an ugly global for
bookkeeper.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When the `--daemon` flag is used, `lightningd` requires that `--log-file`
is used as well. By consequence, the `lightningd.service` didn't work
out-of-the-box for me.
This changes also sends the logs to `journald`.
The new approach is consistent with the `bitcoind.service` file in the bitcoin-core repository.
I prefer this approach because it comes with automatic log-rotation.
If we're opening a channel with a peer which support anchors (and
we do), we tell fundpsbt/utxopsbt to enforce the emergency reserve;
this matters, as it doesn't know about the channel yet, and thus
won't (if it's our first anchor channel).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `fundchannel` and `multifundchannel` will refuse to spend funds below `min-emergency-msat` if we have any anchor channels (or are opening one).
This is needed when we know we're *opening* an anchor channel, to
override the "do we already have an anchor channel open?" logic.
Also, document the nonwrapped arg added in v23.02.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundpsbt` and `utxopsbt` new parameter `opening_anchor_channel` so lightningd knowns it needs emergency reserve for anchors.
This is the simple version which always tries to keep some sats if we
have an anchor channel. Turns out that we need something more
sophisticated for multifundchannel, so that's next.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` will refuse to spend funds below `min-emergency-msat` if we have any anchor channels (and `all` will be reduced appropriately).
Changelog-Changed: JSON-RPC: `fundpsbt` and `utxopsbt` will refuse to spend funds below `min-emergency-msat` if we have any anchor channels.
For anchors, we need some sats sitting around in case we need to CPFP
a close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `min-emergency-msat` setting for (currently experimental!) anchor channels, to keep funds in reserve for forced closes.
This was added to fundpsbt/utxopsbt in v0.10, but the spender plugin
didn't take advantage of it, instead calculating its own change amount
and output.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was added to fundpsbt/utxopsbt in v0.10, but the txprepare plugin
didn't take advantage of it, instead calculating its own change amount
and output.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you did call fundpsbt with amount 'all' and `excess_as_change`
true, you would get everything going to the change output. That's
obviously not the intention, and we'd like to use this to add change
outputs even for "all" when have keep emergency reserves.
And change the finish_psbt() API to take an explicit change amount:
at the moment it's either all or nothing, but that will change with
emergency-sat reserves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were marking our inputs very late, which means any early failure
would not know to unreserve them.
This becomes particularly bad when we start enforcing emergency reserves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use parameterization here. The old `anchor_expected()` was for
non-zero-fee anchors, and have bitrotted so there are some other
changes as well.
Unfortunately, all the anchor accounting seems to be broken, but I
cannot understand these tests at all. I had to simply disable them
for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We disabled experimental support for opening non-zero-fee anchor
channels (though old nodes may still have such channels if they turned
that on!).
So we simply call this `experimental-anchors`, since this is the variant
which we expect to be used widely.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: protocol: added support for zero-fee-htlc anchors (`option_anchors_zero_fee_htlc_tx`), using `--experimental-anchors`.
In most cases, it's the same as option_anchor_outputs, but for
fees it's different. This transformation is the simplest:
pass it as a pair, and test it explicitly.
In future we could rationalize some paths, but this was nice
and mechanical.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since HTLC txs when using anchors are
SIGHASH_SINGLE|SIGHASH_ANYONECANPAY, we can attach other inputs to
give it a higher feerate. But we need the HSMd to actually sign the
combo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to know both, because in theory we could negotiate a
non-anchor channel even if they support it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to know both, because in theory we could negotiate a
non-anchor channel even if they support it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we can CPFP, we don't have to track the feerate as closely. But
it still needs to get in the mempool, so we use 10 sat/byte, or the
100 block estimate if that is higher.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `feerates` has new fields `unilateral_anchor_close` to show the feerate used for anchor channels (currently experimental), and `unilateral_close_nonanchor_satoshis`.
Changelog-Changed: JSON-RPC: `feerates` `unilateral_close_satoshis` now assumes anchor channels if enabled (currently experimental).
mfc->feerate_str is *never* NULL, since we set it in getfeerate; this is
confusing, as many places check for NULL.
Indeed, the logic in perform_fundpsbt() was *wrong* in this case: it used
`normal` (if it was NULL, which it never was) instead of `opening` to fundpsbt.
And the correct thing is for multifundchannel to not use a string here at
all, but to use the exact feerate it is counting on (even the same
string may have different values now if a block has come in).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't actually use it anywhere, but we actually want to now for
CPFP. So give it more parameters and make it return bool so it can
be set without necessarily suppressing rexmit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turns out it's a single sig, identical to the already-handled
case where we spend a to_remote output.
We also close a temporary memleak: stack was unused, but
tallocated off the psbt, so it lives as long as the PSBT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It depends on whether we negotiated anchors: just document that this
field doesn't apply for anchors (it becomes zero by the end of this
patch series, which is weird).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were setting the wrong input number: don't assume it's the
same as the UTXO number, but simply the last-appended input.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It'll wrap, probably be in the past, and infinite loop. This was caused by an invoice
with expiry set at 2076. This wrap caused us to think the expiry has already
passed, and keep looping!
Reported-by: @telelvis
Fixes: #6339
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightnind: don't infinite loop on 32 bit platforms if only invoices are expiring after 2038.
And add contrib/pyln-testing/pyln/testing/grpc2py.py since we didn't previously know
how to build it!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
See the section headed "Rules with Grouped Targets" on the Texinfo page
`(make)Multiple Targets`.
Without this fix, Make does not know that these recipes unconditionally
make *all* of their named targets regardless of which target triggers
their execution, and Make will blissfully execute multiple instances of
any such recipe in parallel, not only wasting CPU cycles but potentially
producing incorrect results if the recipe is not atomic in its effects
on the file system. With this fix, Make understands that it need only
execute such a recipe once to make all of its targets.
In pursuit of the above, move and combine two redundant msggen recipes
into the top-level Makefile, and populate its grouped targets from the
subordinate Makefiles.
Changelog-None
Add a fuzz test for BOLT 8 message encryption and decryption. The fuzz
test is based on the unit test at common/test/run-cryptomsg.c and uses a
static initial state with fuzzer-generated messages to encrypt or
decrypt.
This header will be used by multiple fuzz targets to fuzz Acts 1, 2, and
3 of the BOLT 8 handshake.
We could make this header into a full library, but considering its
narrow use let's try not to over-engineer things.
It seems to reliably be getting a SIGTERM after 17-18 minutes:
```
[gw1] [ 91%] PASSED tests/test_plugin.py::test_forward_event_notification
Error: Process completed with exit code 143.
```
Perhaps this is out of mem? So, try -n2. We also make the configure
line explicit, rather than relying on environment vars (we should probably
do this for the other cases, too)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we didn't hash the descriptions properly (see previous commit), we
cannot immediately deprecate omitting the descriptions (since you'd
have to omit them for backwards compat!).
And move the "must have description or hash" test into bolt11.c core.
Changelog-Deprecated: `pay` has *undeprecated* paying a description-hash invoice without providing the description.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we need to push off requring this for another full deprecation cycle!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `pay` and `decodepay` with description now correctly handle JSON escapes (e.g " inside description)
This is almost certainly because the HTLCs are not fully settled, so wait for that:
```
2023-06-20T11:37:56.2332158Z assert apys_2[0]['our_start_balance_msat'] == Millisatoshi(0)
2023-06-20T11:37:56.2332443Z > assert apys_1[0]['routed_out_msat'] == apys_2[0]['routed_in_msat']
2023-06-20T11:37:56.2332571Z E assert 1892216msat == 2810170msat
2023-06-20T11:37:56.2332580Z
2023-06-20T11:37:56.2332717Z tests/test_pay.py:81: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we reconnect before the channel is completely closed, we might get
a "reconnected" message, so mine a block after and make sure it's
processed.
```
2023-06-20T11:37:56.1302058Z if errors.has_errors():
2023-06-20T11:37:56.1302648Z # Format a nice list of everything that went wrong and raise an exception
2023-06-20T11:37:56.1303781Z request.node.has_errors = True
2023-06-20T11:37:56.1304091Z > raise ValueError(str(errors))
2023-06-20T11:37:56.1304370Z E ValueError:
2023-06-20T11:37:56.1304624Z E Node errors:
2023-06-20T11:37:56.1305042Z E - lightningd-2: had unexpected reconnections
2023-06-20T11:37:56.1305340Z E Global errors:
```
...
```
2023-06-20T11:37:56.1960525Z lightningd-2 2023-06-20T11:21:28.638Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: Peer has reconnected, state CLOSINGD_SIGEXCHANGE: connecting subd
```
I added a plugin arg and was surprised that compile didn't break.
This is because typesafe_cb et al are conditional casts: if the type
isn't as expected it has no effect, but we're passing plugin_option() through
varargs, so everything is accepted!
Add a noop inline to check type, and fix up the two cases where we
used `const char *` instead of `char *`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If it's a plugin opt, we'll need a callback, so reshuffle logic. Also
add infra to map option name to plugin and option.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Do it slightly intelligently, so if we had set previously using setconfig
we don't keep appending new ones, but replace it in-place.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently only implemented for min-capacity-sat.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: new command `setconfig` allows a limited number of configuration settings to be changed without restart.
Previously, if these failed we always exited; once we have dymamic
configs this would be a (tiny) memory leak, so use tmpctx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To test, we do min-capacity-sat which is simple. We also update the
listconfigs man page which contained some obsolete information.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We get "disconnected during connection" if we haven't finished processing
the connection when the peer sends error and hangs up.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We re-enable sendrawtransaction then mine a block to kick off RBF, but there's
a race where it can get a tx in that block, and then we timeout waiting for
both txs to get into the next block.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This isn't the cause of the test_gossip_ratelimit flake I saw (since
the final gossip msg clearly was received), but it's still good to fix
since it means we might not send the final messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were seeing hangs in disconnect in
tests/test_connection.py::test_feerate_stress, and looking at the logs
it's because we're not actually telling connectd to disconnect.
These days, we have a connect counter, so connectd knows to ignore it
if we simply haven't read the message about it already disconnecting.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
build.sh did this, but we didn't carry it across to the new test
matrix, leading to spurious CI failures:
```
lightningd-1 2023-06-06T16:15:34.931Z DEBUG plugin-bookkeeper: Setting up database at ***localhost:39061/accounts_0_7ccmg745
lightningd-1 2023-06-06T16:15:34.931Z INFO plugin-bookkeeper: Creating database
lightningd-1 2023-06-06T16:15:34.931Z **BROKEN** plugin-bookkeeper: Error vacuuming db: VACUUM command failed: ERROR: deadlock detected\nDETAIL: Process 77248 waits for AccessShareLock on relation 1260 of database 0; blocked by process 77414.\nProcess 77414 waits for RowExclusiveLock on relation 1214 of database 0; blocked by process 77248.\nHINT: See server log for query details.\n
lightningd-1 2023-06-06T16:15:34.931Z INFO plugin-bookkeeper: Killing plugin: exited before replying to init
lightningd-1 2023-06-06T16:15:34.931Z **BROKEN** plugin-bookkeeper: Plugin marked as important, shutting down lightningd!
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Clang's coverage instrumentation [1] is the best I've seen, with
precision down to the expressions within a line of code. Add an option
to use this instrumentation for better coverage reports.
[1] https://clang.llvm.org/docs/SourceBasedCodeCoverage.html
This fixes the compile issue that we are having on
alpine.
```
cc wallet/wallet.c
cc wallet/walletrpc.c
cc wallet/reservation.c
cc wallet/db_sqlite3_sqlgen.c
cc wallet/db_postgres_sqlgen.c
cc common/addr.c
cc common/bolt11.c
cc common/bolt11_json.c
cc common/bolt12.c
cc common/configdir.c
cc common/configvar.c
cc common/scb_wiregen.c
common/configvar.c: In function 'configvar_remove':
common/configvar.c:118:9: error: unknown type name 'ssize_t'; did you mean 'size_t'?
118 | ssize_t prev = -1;
| ^~~~~~~
| size_t
make: *** [Makefile:292: common/configvar.o] Error 1
make: *** Waiting for unfinished jobs....
```
Link: https://github.com/ElementsProject/lightning/issues/6321
Reported-by: @gruve-p
Fixes: 36200a6593
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
I noted in some PR that we are failing during the diff of the autogenerated files.
So this will generate the last version of them!
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
When core lightning is asking the information about
the blockchain with `getchaininfo` command lightningd
know already the information about the min and max block height.
the problem is when we have a smarter Bitcoin backend that is able
to switch between different clients in some cases is helpful
give the information about current known height by lightningd and
pass it down to the plugin.
In this way, the plugin knows what is the correct known height from lightnind, and can
try to fix some problems if any exit.
This is particularly useful when you are syncing a new backend from scratch
like https://github.com/cloudhead/nakamoto and we avoid returning the
lower height from the known, and avoid the crash of core lightning.
With this information, the plugin can start to sync the chain and return
the answer back only when the chain is in sync with the current status of
lightningd.
Another reason to add this field and not wait the correct block in core
lightning itself is because Bitcoin Core is extremely slow to sync up,
so the question here is, how long should we wait? The time depends
on various factors.
With this approach of informing the plugin about the height, in some cases,
you can start the syncing but move the execution to another backend until
the previous one is ready.
The problem I want to solve is that I don't want to be left in the dark when
we run `getchaininfo`, and I want to have the opportunity to wait for
the blockchain sync or decide to dispatch the request elsewhere.
Changelog-Added: Pass the current known block height down to the getchaininfo call.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
option_scid_alias inside a channel_type allows for more private
channels: in particular, it tells the peer that it MUST NOT allow
routing via the real short channel id, and MUST use the alias.
It only makes sense (and is only permitted!) on unannounced channels.
Unfortunately, we didn't set this bit in the channel_type in v12.0
when it was introduced, instead relying on the presence of the feature
bit with the peer. This was fixed in 23.05, but:
1. Prior to 23.05 we didn't allow it to be set at all, and
2. LND has a limited set of features they allow, and this isn't allowed without
option_anchors_zero_fee_htlc_tx.
We could simply drop this channel_type until we merge anchors, *but*
that has nasty privacy implications (you can probe the real channel id).
So, if we don't negotiate anchors (we don't!), we don't set this
channel_type bit even if we want it, and *intuit* it, based on:
1. Is this a non-anchor channel_type?
2. Did we both send channel_type?
3. Is this an unannounced channel?
4. Did both peers announce support for scid aliases?
In addition, while looking at the previous backwards-compat code, I
realized that v23.05 violated the spec and send accept_channel with
OPT_SCID_ALIAS if it intuited it, even if it wasn't offered. Stop
doing this, but allow our peers to.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Fix incompatibility with LND which prevented us opening private channels
Fixes: #6208
Due to a security report by github, we should increase
our cryptography lib version.
This may impact potential another version that is stuck
with a cryptography version.
Link: https://github.com/ElementsProject/lightning/issues/6164
Reported-by: @dni
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Detected by ASan in test_hsmtool_generatehsm:
==58698==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 120 byte(s) in 1 object(s) allocated from:
#0 0x4e6247 in malloc
#1 0x7f078452d672 in getdelim
SUMMARY: AddressSanitizer: 120 byte(s) leaked in 1 allocation(s).
defaults were deprecated in 0df97547dd, but that was a bit
harsh as several plugins do that (summary, for example). So allow false, but warn
that we ignore anything else.
Reported-by: @microsatoshi on Discord.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: ...actually, `default` `false` still accepted on `flag` type parameters.
Previously any attempt would result in "is not an integer field"; we
now recognize valid JSON integers as integer fields.
Changelog-Fixed: Plugins: `commando` runes can now compare integer parameters using '<' and '>' as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This PR:
- adds all the guides (in markdown format) that is published at https://docs.corelightning.org/docs
- adds a github workflow to sync any future changes made to files inside the guides folder
- does not include API reference (json-rpc commands). Those will be handled in a separate PR since they're used as manpages and will require a different github workflow
Note that the guides do not exactly map to their related files in doc/, since we reorganized the overall documentation structure on readme for better readability and developer experience. For example, doc/FUZZING.md and doc/HACKING.md#Testing are merged into testing.md in the new docs. As on the creation date of this PR, content from each of the legacy documents has been synced with the new docs. Until this PR gets merged, I will continue to push any updates made to the legacy documents into the new docs.
If this looks reasonable, I will add a separate PR to clean up the legacy documents from doc/ (or mark them deprecated) to avoid redundant upkeep and maintenance.
Changelog-None
This is a simplification for our build system
that allows for managing and cleaning the external
build artefacts in an easy way.
We have a hard time with the suffix `-modded` inside the
version with some architecture when building a tagged version.
Link: https://discord.com/channels/899980449231814676/899989729183940629/1110957992158965770
Reported-by: @ctrlbreak-
Suggested-by: Rusty Russell <rusty@rustcorp.com.au>
Co-developed-by: @jsarenik
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Memory leak detected by ASan:
==880002==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
#0 0x5039e7 in malloc (lightningd/lightningd+0x5039e7)
#1 0x7f2e8c203884 in __alloc_dir (/lib64/libc.so.6+0xd2884)
Fixes nullability errors detected by UBSan:
wire/fromwire.c:173:46: runtime error: null pointer passed as argument 1, which is declared to never be null
external/libwally-core/src/secp256k1/include/secp256k1.h:432:3: note: nonnull attribute specified here
#0 0x65214a in fromwire_secp256k1_ecdsa_signature wire/fromwire.c:173:6
#1 0x659500 in printwire_secp256k1_ecdsa_signature devtools/print_wire.c:331:1
#2 0x646ba2 in printwire_channel_update wire/peer_printgen.c:1900:7
#3 0x637182 in printpeer_wire_message wire/peer_printgen.c:128:11
#4 0x65a097 in main devtools/decodemsg.c:85:10
Otherwise we later copy the uninitialized memory to descendants,
triggering undefined behavior:
plugins/libplugin-pay.c:2882:34: runtime error: load of value 190, which is not a valid value for type 'bool'
It is possible for db_column_bytes() to return 0 and for
db_column_blob() to return NULL even when db_column_is_null() returns
false. We need to short circuit in this case.
Detected by UBSan:
db/bindings.c:479:12: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:44:28: note: nonnull attribute specified here
#0 0x95f117 in db_col_arr_ db/bindings.c:479:2
#1 0x95ef85 in db_col_channel_type db/bindings.c:459:32
#2 0x852c03 in wallet_stmt2channel wallet/wallet.c:1483:9
#3 0x81f396 in wallet_channels_load_active wallet/wallet.c:1749:23
#4 0x81f03d in wallet_init_channels wallet/wallet.c:1765:9
#5 0x72f1f9 in load_channels_from_wallet lightningd/peer_control.c:2257:7
#6 0x672856 in main lightningd/lightningd.c:1121:25
The function is tiny and was only used in one location. And that one
location was leaking memory.
Detected by ASan:
==2637667==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 7 byte(s) in 1 object(s) allocated from:
#0 0x4cd758 in __interceptor_strdup
#1 0x64c70c in json_stream_log_suppress_for_cmd lightning/lightningd/jsonrpc.c:597:31
#2 0x68a630 in json_getlog lightning/lightningd/log.c:974:2
...
SUMMARY: AddressSanitizer: 7 byte(s) leaked in 1 allocation(s).
Detected by UBSan:
$ UBSAN_OPTIONS=print_stacktrace=1 ./wallet/test/run-psbt_fixup
bitcoin/psbt.c:733:2: runtime error: applying zero offset to null pointer
#0 0x53c829 in psbt_from_bytes lightning/bitcoin/psbt.c:733:2
#1 0x5adcb0 in main lightning/wallet/test/run-psbt_fixup.c:174:10
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior bitcoin/psbt.c:733:2
We now know the base reference, and we've rebased, so we can do a
simple diff. Also, this means we can use a magic commit message
`No-schema-diff-check` to suppress false positives.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This reflects what we actually do when we apply the commit, and also
means we can easily iterate the commits.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `listconfigs` direct fields, use `configs` sub-object and `set`, `value_bool`, `value_str`, `value_int`, or `value_msat` fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This integrates them with configvars properly: they almost "just work"
in listconfigs now, and we don't put them in a special sub-object
under their plugin.
Unfortunately, this means `listconfigs` now has a loose schema: any
plugin can add something to it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: reloaded plugins get passed any vars from configuration files.
Changelog-Deprecated: Config: boolean plugin options set to `1` or `0` (use `true` and `false` like non-plugin options).
We use multi-specifiable options elsewhere, this is just another.
Otherwise you can't add, you can only set them all.
Changelog-Added: Config: `accept-htlc-tlv-type` (replaces awkward-to-use `accept-htlc-tlv-types`)
Changelog-Deprecated: Config: `accept-htlc-tlv-types` (use `accept-htlc-tlv-type` multiple times)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `default` no longer accepted on `flag` type parameters (it was silently ignored, so just don't set it).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
listconfigs is convenient, but it doesn't handle multi-options well: it
outputs an object with duplicate fields in this case (e.g. log-file), nor
is it extensible to show more than raw values.
However, listconfigs doesn't do what other list commands do (use a
sub-object "configs") so we can put the new values under that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listconfigs` now has `configs` subobject with more information about each config option.
It literally contained \" to avoid it being interpreted as a literal;
now we have OPT_SHOWINT we no longer need this hack.
It's obscure, so I'm not bothering with a deprecation cycle.
Changelog-Fixed: JSON-RPC: `listconfigs` `rpc-file-mode` no longer has gratuitous quotes (e.g. "0600" not "\"0600\"").
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have hacky code to show some listconfigs values as literals; instead
explicitly encode the types.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Clearly, listconfigs shouldn't list these.
Also, hoist the opt_hidden check since it's independent of whether
there's an arg or not.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's an obscure command, but this we won't see old plugin commands in
listconfigs (once it uses configvars).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Developers, rejoice (we already have --testnet, --signet and --mainnet!).
Changelog-Added: Config: `--regtest` option as alias for `--network=regtest`
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we wire in the code which gathers configvars and parses from there;
lightningd keeps the array of configuration variables for future use.
Note that lightning-cli also needs to read the config, but it has its
own options (including short ones!) and doesn't want to use this
configvar mechanism, so we have a different API for that now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a nod towards moving to runtime-developer-mode, and also
quite clear; we currently have all these options prefixed with dev,
but this flags makes handling explicit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will be used for listconfig, which knows these should be presented
as arrays, not single values.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are gathered from the config files and the commandline, but the
process is rather complex! We want to remember where the options came
from in future (for a `setconfig` command), and also generalize
and simplify handling.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that this actually changes listconfigs output for three msat
fields, which were not changed with the great msat merge. Since
listconfigs isn't actually used by grpc, and the values are always a
little vague, I simply changed this.
Changelog-Fixed: JSON-RPC: `listconfigs` `htlc-minimum-msat`, `htlc-maximum-msat` and `max-dust-htlc-exposure-msat` fields are now numbers, not strings.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds:
1. ability to search for an option by name.
2. allowance to set our own bits when registering options.
3. show callbacks which can say "don't show", and variable length.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We leave the code in contrib/pyln-client/pyln/client/lightning.py to handle
msat null fields for now, though, for a bit more compatibility.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently it fails if a field is missing, but sometimes that's OK. So
allow a fieldname ending in `?` to mean "skip over if it's missing".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported in #6270, there was an attempt to delete gossip overrunning
the end of the gossip_store. This logs the gossip type that was attempted to be deleted and avoids an immediate crash (tombstones would be fine to
skip over at least.)
Changelog-None
The `cln-grpc` crate really has a dual purpose: server and
client. Having the server feature be included by default means that we
are pulling in `cln-rpc` which is a Unix only crate, because of the
use of UDS to talk to CLN. We want to use `cln-grpc` on clients too,
and those might not be Unix variants, hence they'd fail when compiling
`cln-rpc`. This PR guards the Unix-related parts behind the `server`
feature flag.
These are only likely to confuse users, by silently changing behavior.
Changelog-Deprecated: Config: bind-addr=xxx.onion and addr=xxx.onion, use announce=xxx.onion (which was always equivalent).
Changelog-Deprecated: Config: addr=/socketpath, use listen=/socketpath (which was always equivalent).
This obsoletes the use of --announce-addr-dns which I know Michael
didn't really like either.
Changelog-Deprecated: Config: `announce-addr-dns`; use `--bind-addr=dns:ADDR` for finer control.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a major cleanup to how we parse addresses.
1. parse_wireaddr now supports the "dns:" prefix to support dns records (type 5).
2. We are less reliant on separate_address_and_port() which gets confused by
that colon.
3. We explicitly test every possible address type we can get back from
parsing, and handle them appropriately.
We update the documentation to use the clearer HOSTNAME vs DNS prefixes now
we also have `dns:` as a prefix.
Changelog-Added: Config: `bind` can now take `dns:` prefix to advertize DNS records.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I never really liked this hack: websockets are useful, advertizing
them not so much.
Note that we never actually documented that we would advertize these!
Changelog-EXPERIMENTAL: Protocol: Removed support for advertizing websocket addresses in gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Make it the standard "return the error" pattern.
2. Rather than flags to indicate what types are allowed, have the callers
check the return explicitly.
3. Document the APIs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This contained cut & paste code, and it wasn't clear to me that
the first loop included DNS entries with IPv6 entries.
Instead, allow the iterator to take multiple types, and use
a switch statement so compile will break as new types are added.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an internal type: it has no API guarantees (indeed, I'm about
to change it, which is how I discovered scb was using it).
Fortunately for every case we care about, it is actually a wireaddr
(in theory the peer can connect locally using a local socket, but this
is mostly for testing and is a very strange setup, and so simply don't
do scb for those).
In this case, the wire encoding is a single byte followed by the
wireaddr, so open-code that in scb_wire.csv for compatibility.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I need to update gprcio-tools, but it needs protobuf v4. Christian
added this restriction in a99509db36 to
"Update protobuf dependency to silence dependabot", but perhaps it's
time to actually update?
```
$ poetry update
Updating dependencies
Resolving dependencies... (1.0s)
Because pyln-testing (23.05rc2) @ file:///home/rusty/devel/cvs/lightning/contrib/pyln-testing depends on protobuf (>=3.20.3,<4)
and grpcio-tools (1.54.0) depends on protobuf (>=4.21.6,<5.0dev), pyln-testing (23.05rc2) @ file:///home/rusty/devel/cvs/lightning/contrib/pyln-testing is incompatible with grpcio-tools (1.54.0).
So, because cln-meta-project depends on both grpcio-tools (1.54.0) and pyln-testing (23.05rc2) @ file:///home/rusty/devel/cvs/lightning/contrib/pyln-testing, version solving failed.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We removed the (experimental-only!) annotation output in 611795beee
but we still loaded them from the db. Turns out that we were putting bogus
annotations into the db, and accessing out of range when loading them.
Consider the following db entry in transaction_annotations:
```
CREATE TABLE transaction_annotations ( txid BLOB, idx INTEGER, location INTEGER, type INTEGER, channel INTEGER REFERENCES channels(id), UNIQUE(txid, idx));
...
INSERT INTO transaction_annotations VALUES(X'19706f9af2875508a06c7db1754ef7ecb3da745ead005992e626441e4e83465f',18,1,129,53699);
```
Here is the corresponding entry in txs:
```
INSERT INTO transactions VALUES(X'19706f9af2875508a06c7db1754ef7ecb3da745ead005992e626441e4e83465f',710327,966,X'02000000000101f2add69112a1d557317826120e1f4ea3bc1cbe4674d720325695b26ecfe8355d120000000000000000013634000000000000160014dca21f104359bbb81e88ed7da985549f2cd0cbc30347304402201cdc854b76c4c7523e3ca09f38a81539d3b2f7fbd9a0de6ae10b7ceaa65ed9d402205a1770058cd1ef081c77c2fe957c07a334cb3a11bc0cc502834a29c59424fe010120589da1f809d955c7af150bf53123c27ffc0a741489b5291f6be811189863ec838576a9142f188d0d973c4ad1865a619d3748340b30746e088763ac672103b7bbcd592197ba6501e7176aabd3f908d94b126ae82ab1e7a4c58f5a789782e57c820120876475527c21029bcf62114eb36758fcb1aead7e67b6f707d32f34e67816894d5211ac9f2d6ce752ae67a9144483a115219ba65c63a3844be8445f739703bea988ac686800000000',NULL,NULL);
```
The annotation refers to output 18 of the tx, but it only has one output!
However, decoding the tx shows that it spent output 18 of a previous tx, so
that's probably where the `18` came from.
Remove this logic: we can remove the remaining (clearly broken!) annotation
adding code in another cleanup commit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CI hit this issue where it would get a tx_abort in fundchannel. This
happens when the dualopend we use to query the feerates has not exited
yet (it waits for the tx_abort reply), and we mistakenly reuse it.
With multi-channel support, this is wrong: just run another one and it
all Just Works.
This means we need to rework our dual_open_control.c logic, since it
would previously create an unsaved channel then not clean up if
something went wrong.
Most people will never try to negotiate opening multiple channels to
the same peer at the same time (vs. having an established channel and
opening a new one), so this case is a bit weird.
```
rates = l1.rpc.dev_queryrates(l2.info['id'], amount, amount)
# l1 leases a channel from l2
l1.rpc.fundchannel(l2.info['id'], amount, request_amt=amount,
feerate='{}perkw'.format(feerate),
> compact_lease=rates['compact_lease'])
tests/test_opening.py:1611:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-client/pyln/client/lightning.py:833: in fundchannel
return self.call("fundchannel", payload)
contrib/pyln-testing/pyln/testing/utils.py:721: in call
res = LightningRpc.call(self, method, payload, cmdprefix, filter)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyln.testing.utils.PrettyPrintingLightningRpc object at 0x7f6cbcd97950>
method = 'fundchannel'
payload = {'amount': 500000, 'announce': True, 'compact_lease': '029a00640064000000644c4b40', 'feerate': '2000perkw', ...}
cmdprefix = None, filter = None
def call(self, method, payload=None, cmdprefix=None, filter=None):
"""Generic call API: you can set cmdprefix here, or set self.cmdprefix
...
if not isinstance(resp, dict):
raise ValueError("Malformed response, response is not a dictionary %s." % resp)
elif "error" in resp:
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: fundchannel, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'amount': 500000, 'feerate': '2000perkw', 'announce': True, 'request_amt': 500000, 'compact_lease': '029a00640064000000644c4b40'}, error: {'code': -1, 'message': 'Abort requested', 'data': {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'method': 'openchannel_init'}}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Failure under CI:
```
> bitcoind.generate_block(1000)
tests/test_closing.py:853:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:496: in generate_block
return self.rpc.generatetoaddress(numblocks, to_addr)
contrib/pyln-testing/pyln/testing/utils.py:374: in f
res = proxy._call(name, *args)
../../../.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.7/lib/python3.7/site-packages/bitcoin/rpc.py:246: in _call
response = self._get_response()
../../../.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.7/lib/python3.7/site-packages/bitcoin/rpc.py:276: in _get_response
http_response = self.__conn.getresponse()
/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/http/client.py:1373: in getresponse
response.begin()
/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/http/client.py:319: in begin
version, status, reason = self._read_status()
/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/http/client.py:280: in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <socket.SocketIO object at 0x7fa21aa5d710>
b = <memory at 0x7fa21b771390>
def readinto(self, b):
"""Read up to len(b) bytes into the writable buffer *b* and return
the number of bytes read. If the socket is non-blocking and no bytes
are available, None is returned.
If *b* is non-empty, a 0 return value indicates that the connection
was shutdown at the other end.
"""
self._checkClosed()
self._checkReadable()
if self._timeout_occurred:
raise OSError("cannot read from timed out object")
while True:
try:
> return self._sock.recv_into(b)
E socket.timeout: timed out
/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/socket.py:589: timeout
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
07413c20b9 et al reworked how onchaind
does broadcasts, meaning tests needed to be updated to the new helpers
rather than searching logs themselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were passing a max_output_len that was too small, so every call to
bech32_encode was failing. Now we set max_output_len to the full size of
bech32_str.
s/max_input_len/max_output_len
This maximum length applies to the output parameter, not the data
parameter. Thus it is more intuitive to name it max_output_len.
Changelog-EXPERIMENTAL: Build: all experimental features are now runtime-enabled; no more ./configure --enable-experimental-features
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This currently means anchors tests are disabled, awaiting the
PR which implements zero-fee-htlc anchors to reenable them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And no longer insist on opt_quiesce.
Changelog-EXPERIMENTAL: Config: `--experimental-upgrade-protocol` enables simple channel upgrades.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since this was merged, `make extract-peer-csv` was broken!
But the field names changed:
1. `tlv_update_add_tlvs` -> `tlv_update_add_htlc_tlvs`
2. `blinding` -> `blinding_point`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The modern style is to assert that all messages have tlvs, but many
are currently empty. In particular,
c4c5a8e5fb30b1b99fa5bb0aba7d0b6b4c831ee5 added "update_add_htlc_tlvs"
without adding an explicit field of that type to "update_add_htlc".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are no feature-dependent fields left in the spec. (I've also
made a PR for the spec to remove the tooling for it there).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to check that the key is valid for two reasons:
1) towire_ext_key() aborts if the key is invalid
2) fromwire_ext_key() doesn't check the parsed key for validity
Since bip32_key_get_fingerprint() fails if the key is invalid, we can
call it first to guarantee the key is valid before serializing.
By using fuzzer instrumentation for dependencies, we get more coverage
signal during fuzzing. This is useful when the fuzzer must figure out
how to take certain branches in a dependency.
In our case, the fuzz-bip32 target was failing to create a data buffer
that successfully passed fromwire_ext_key() parsing because the fuzzer
couldn't see what was happening inside libwally-core.
Without this, a stuck test (such as before the previous commit, where a plugin crashed
when running a command) simply gets timed out by the full CI timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we release too fast, the plugin crashes:
```
thread 'tokio-runtime-worker' panicked at 'called Result::unwrap() on an Err value: SendError(())', plugins/examples/cln-plugin-reentrant.rs:31:27
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
```
This happens with CI under VALGRIND!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a regression that we introduced in this release
due to some dirty parts of our codebase.
For historical reasons (I think), we were using a `json_add_sat_only`
procedure defined in `peer_control.c`. So when @rustyrussell removed the _msat,
we thought that all the fields were reflecting the new behavior, but
we were wrong.
This PR fixes this bug and also removes the additional function
from `peer_control.c`. This way, we can be sure that there is no other part
of our codebase that uses this method (except for other `json_add` methods).
Link: https://github.com/ElementsProject/lightning/issues/6244
Reported-by: @hMsats
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We were using per-type overrides which caused some asymmetries, where
conversions could end up dropping fields as we went along. Essentially
each conversion would need to override a superset of the previous one,
which then caused issues when attempting to close the loop. By
overriding on the model level we ensure that all representations are
equivalent and convertible into one another, at the expense of
overriding a bit more aggressively, which should be fine anyway.
We use overrides that omit fields in some cases, which makes the
conversion lossy. This also means that until we complete the mapping
we can't reconvert back.
gcc 13 add an extra check for the enum in the definition
of a method. In our case the code was failing with the
following error, and the compiler is right, our definition
is different from the implementation.
```
$ make
CC: cc -DBINTOPKGLIBEXECDIR="../libexec/c-lightning" -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -Werror -Wno-maybe-uninitialized -Wshadow=local -std=gnu11 -g -fstack-protector-strong -Og -I ccan -I external/libwally-core/include/ -I external/libwally-core/src/secp256k1/include/ -I external/jsmn/ -I external/libbacktrace/ -I external/gheap/ -I external/x86_64-redhat-linux/libbacktrace-build -I external/libsodium/src/libsodium/include -I external/libsodium/src/libsodium/include/sodium -I external/x86_64-redhat-linux/libsodium-build/src/libsodium/include -I . -I/usr/local/include -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS -DCOMPAT_V052=1 -DCOMPAT_V060=1 -DCOMPAT_V061=1 -DCOMPAT_V062=1 -DCOMPAT_V070=1 -DCOMPAT_V072=1 -DCOMPAT_V073=1 -DCOMPAT_V080=1 -DCOMPAT_V081=1 -DCOMPAT_V082=1 -DCOMPAT_V090=1 -DCOMPAT_V0100=1 -DCOMPAT_V0121=1 -DBUILD_ELEMENTS=1 -c -o
LD: cc -Og config.vars -Lexternal/x86_64-redhat-linux -lwallycore -lsecp256k1 -ljsmn -lbacktrace -lsodium -L/usr/local/include -lm -lgmp -lsqlite3 -lz -o
cc plugins/spender/multifundchannel.c
plugins/spender/multifundchannel.c:71:6: error: conflicting types for ‘fail_destination_msg’ due to enum/integer mismatch; have ‘void(struct multifundchannel_destination *, enum jsonrpc_errcode, const char *)’ [-Werror=enum-int-mismatch]
71 | void fail_destination_msg(struct multifundchannel_destination *dest,
| ^~~~~~~~~~~~~~~~~~~~
In file included from plugins/spender/multifundchannel.c:13:
./plugins/spender/multifundchannel.h:263:6: note: previous declaration of ‘fail_destination_msg’ with type ‘void(struct multifundchannel_destination *, int, const char *)’
263 | void fail_destination_msg(struct multifundchannel_destination *dest,
| ^~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:307: plugins/spender/multifundchannel.o] Error 1
```
The gcc 13 is not released yet, but fedora beta is out for public testing,
so it is useful fix this error in this release candidate cycle.
Changelog-Fixed: Build: Compilation with upcoming gcc 13
Reported-by: @grubles
Link: https://github.com/ElementsProject/lightning/issues/6175
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
After the first iteration of the loop, we call memmem with a buflen that
points past the end of buf.
In practice we probably never read the uninitialized memory since we
guarantee the buffer ends with "\r\n", and since most/all libc
implementations probably read the haystack sequentially. But maybe
there's some libc with a crazy optimization out there. It's good to use
an accurate buflen just in case.
Discovered this while running some unit tests with MSan.
Avoids failing the test with the pip warning:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
reported by: @ksedgwic
Changelog-None
Fixes a bug in installer registration where executable is evaluated
before entrypoints and other details are added.
***RECKLESS STDERR***
Traceback (most recent call last):
File lightning/tools/reckless, line 382, in <module>
INSTALLERS['nodejs'].add_entrypoint('{name}')
KeyError: 'nodejs'
Reported by @ksedgwic
Changelog-None
When enabling or disabling a plugin, the entrypoint is inferred
from the user provided name. A canonical name should be used, which
the installer entrypoint formats help to determine (this generally strips
the file extension if one is provided.)
Also adds a timeout when testing a plugin. Previously the behavior
of pyln-client was relied upon to exit if not communicating with
lightningd, however, this behavior is not universal.
Changlelog-Changed: reckless now installs node.js plugins
Also removes support for pip editable install using pyproject.toml
`pip install -e .` This was a fallback method when a requirements
file was not present, but was hacky and often failed anyway.
reckless: remove installation via pyproject.toml
This method relied on pip install in editable mode (hacky) and often
failed to complete anyhow. We should instead encourage a requirements
file to be created/used for user installation.
This was pointed out by Daywalker [1]: we are synchronously processing
events from `lightningd` which means that if processing one of the
hooks or requests was slow or delayed, we would not get notifications,
hooks, or RPC requests, massively impacting the flexbility.
This highlights the issue with a failing test (it times out), and in
the next commit we will isolate event processing into their own task,
so to free the event loop from having to wait for an eventual
response.
[1] https://community.corelightning.org/c/developers/hold-invoice-plugin#comment_wrapper_16754493
The push bit was convenient for connectd to send our own gossip
to peers upon connecting by naively traversing the gossip_store
and sending anything flagged `push`. This function is now
performed by gossipd leaving no use for the push bit.
Changelog-Changed: `gossipd`: gossip_store PUSH bit is no longer set.
This implements the proposal to simply use timestamp as "all", "none"
or "stream". There's also a rough spec draft which I will post soon.
This *also* removes the last place where we would sometimes sweep the
entire gossip_store looking for their given timestamps.
We could also get rid of the actual timestamp filtering logic in
gossip_store_next if we want to, as it's now basically unused.
Changelog-Changed: Protocol: Simplify gossip_timestamp_filter handling to "all", "none" or "recent" instead of exact timestamp.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This removes the sweep logic as soon as they connect. This should save
connectd a significant number of CPU cycles and make @whitslack finally
stop hitting me.
Changelog-Changed: `connectd` no longer sweeps gossip_store file when peer connects, saving CPU for large nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was previously the role of connectd, but it's actually more
efficient for us to do it: connectd has to sweep through the entire
gossip_store, but we have datastructures for this already.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were in fact feeding l1 its own gossip, which it doesn't ratelimit (this was
a bit fuzzy before, but definitely is the case now!).
So make this node actually l3, so we test what we expected to test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This test makes l2 save db, make a payment, then rollback.
*Sometimes* under CI (but not here) we don't shutdown fast enough
after the payment, so the moves.json from the coin_movements.py
records it. Sure enough, the result is the node ends up with
a -10000msat balance.
Validated by putting a time.sleep(5) between:
```
l2.rpc.pay(inv['bolt11'])
import time
time.sleep(5)
# stop both nodes, roll back l2's database
```
The answer, of course, is to save and rollback *both* the db and
moves.json file.
Here's the error:
```
def test_penalty_htlc_tx_timeout(node_factory, bitcoind, chainparams):
...
assert account_balance(l3, channel_id) == 0
> assert account_balance(l2, channel_id) == 0
tests/test_closing.py:1527:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/utils.py:184: in account_balance
m_sum -= Millisatoshi(m['debit_msat'])
contrib/pyln-client/pyln/client/lightning.py:197: in __sub__
return Millisatoshi(int(self) - int(other))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = -10000msat, v = -10000
...
if self.millisatoshis < 0:
> raise ValueError("Millisatoshi must be >= 0")
E ValueError: Millisatoshi must be >= 0
contrib/pyln-client/pyln/client/lightning.py:82: ValueError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to wait until we're sure bcli has handed results to lightningd:
```
> assert feerates['perkw']['mutual_close'] == 5000
E assert 6250 == 5000
tests/test_misc.py:1617: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This finally happened on my test box; tests simply stopped. Turns out
we were spinning here, with waitpid returning -1.
Since this is during shutdown, that also explains why pytest under CI
would hang, since timeouts don't apply during test teardown!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are several benefits of doing this:
- prevent fuzz target bit rot
- more test coverage in CI
- greater visibility of fuzz tests, encouraging contributions to the
seed corpus and tests themselves
Turns out this was accidentally changed for v0.10.1 in
d8e68893f5 where we made fee levels less
aggressive.
Oops. I guess we can fix the docs. And we now have "2blocks" if you
want it really fast!
Closes: #6129
Changelog-Fixed: JSON-RPC: `feerates` document correctly that urgent means 6 blocks (not 2), and give better feerate examples.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
mempool.info gives sat/vB, which is 1000 too low for us!
See-also: #6161 (complains setting feerate to 5 doesn't work)
See-also: #6129
Suggested-by: @lightingorb
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You still need to actually make a rune when lightningd starts, as
commando (for safety) won't work unless you actually generate a rune
(that it knows of!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: hsmtool: `makerune` command to make a master rune for a node.
1. Rename get_hsm_secret to get_unencrypted_hsm_secret.
2. Create a common helper for fetching full file contents.
3. Create new routine to decrypt if necessary: get_hsm_secret().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were debugging a number of issues related to the forwarding logic,
when using public scids on private channels, and we noticed that we
are very verbose everywhere, except where it counts, i.e., what
decisions are being taken. So we add a couple of debug logs, and a
final info one that tells the operator which resolution was chosen in
the end.
This was always the intent, but now we have to reconstruct from the
disparate fields.
This means `option_anchor_outputs` is now redundant.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: we will upfront reject channel_open which asks for a zeroconf channel unless we are going to do a zerconf channel.
I tested this indeed breaks if we don't accept it, then implemented
the code to accept it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: protocol: We now correctly accept the `option_scid_alias` bit in `open_channel` `channel_type`.
Changelog-Deprecated: protocol: Not setting `option_scid_alias` in `option_channel` `channel_type` for unannounced channels.
1. anchor_to_remote_redeem => bitcoin_wscript_to_remote_anchored,
which matches other witness script producing functions and makes
it clear that it's a to_remote variant.
2. is_anchor_witness_script => is_to_remote_anchored_witness_script
makes it clear that it's about a to_remote output (as altered
when anchors are enabled) not an anchor output!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We accept that we will fail to listen if we bind both IPv6 and IPv4 to
the same socket on a dual-stack machine (e.g. normal Linux), but we weren't
closing the fd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Most of this is piping the flag through so we know it's a websocket!
Reported-by: @ShahanaFarooqui
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we've set everything up, the replacement code is quite simple.
Some tests now have to deal with RBF though, and our rbf tests need work
since they look for the old onchaind messages.
In particular, when we can't afford the fee we want, we back off to
the next blockcount estimate, rather than spending all on fees
(necessarily). So test_penalty_rbf_burn no longer applies.
Changelog-Changed: Protocol: spending unilateral close transactions now use dynamic fees based on deadlines (and RBF), instead of fixed fees.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We also do it on every block, but since bitcoind can't always be counted
to rebroadcast for us, we might as well be aggressive!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We would only set it the first time, which was OK for how we were using it
before. Now we want to also set it for rebroadcast.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Splitting into onchaind_tx() into onchaind_tx_unsigned() and
sign_and_get_witness() makes it easier to reuse for RBF.
Adding more information in onchain_signing_info is required too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `feerates`: added `floor` field for current minimum feerate bitcoind will accept
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #4473
Changelog-Deprecated: Plugins: `estimatefees` returning feerates by name (e.g. "opening"); use `fee_floor` and `feerates`.
Changelog-Fixed: Plugins: `bcli` now tells us the minimal possible feerate, such as with mempool congestion, rather than assuming 1 sat/vbyte.
Changelog-Added: Plugins: `estimatefees` can return explicit `fee_floor` and `feerates` by block number.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And consolidate descriptions into lightning-feerates().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` now allow "minimum" and NN"blocks" as `feerate` (`feerange` for `close`).
Drop try_get_feerate() in favor of explicit feerate_for_deadline() and
smoothed_feerate_for_deadline().
This shows us everywhere we deal with old-style feerates by names.
`delayed_to_us` and `htlc_resolution` will be moving to dynamic fees,
so deprecate those.
Note that "penalty" is still used for generating penalty txs for
watchtowers, and "unilateral_close" still used until we get zero-fee
anchors.
Changelog-Added: JSON-RPC: `feerates` `estimates` array shows fee estimates by blockcount from underlying plugin (usually *bcli*).
Changelog-Changed: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) value *slow* is now 100 block-estimate, not half of 100-block estimate.
Changelog-Deprecated: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) expressed as, "delayed_to_us", "htlc_resolution", "max_acceptable" or "min_acceptable". Use explicit block counts or *slow*/*normal*/*urgent*/*minimum*.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than have specific-purpose levels, have an array of
[blockcount, feerate], and rebuild the specific-purpose levels
for now on top.
We also keep a *separate* smoothed feerate, so you can ask for that
explicitly.
Since all the plugins used the same formula to derive the different
named fee levels, we apply the reverse to return to the underlying
estimates: updating the interface comes next.
This is ugly for now, but various specific-purpose levels will be
going away, as we shift to deadline-driven fees.
This temporarily breaks the floor calculation, so that test is
disabled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have the FEERATE_FLOOR constant if you don't care, but usually you want
to use the current bitcoind lower limit, so call get_feerate_floor()
(which is currently the same, but coming!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we're messing with feerates, it's good to test this directly upfront.
Also, fix documentation!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turns out the two bcli replacements I checked (`sauron` and
`trustedcoin`) don't even implement this, and the multiplier makes
more sense in lightningd, especially as we move to bcli just providing
raw feerate estimates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Adding a new field with `added` fails:
```
AssertionError: Field Feerates.perkb.estimates[] does not have an `added` annotation
```
Looks like this assertion is wrong: we should get an added from the field itself or
from the .msggen.json file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These corpora were generated with default libFuzzer flags with 30+ hours
of CPU time, and then minimized with:
./fuzz-TARGET -merge=1 -shuffle=0 -prefer_small=1 -use_value_profile=1 corpora/fuzz-TARGET UNMINIMIZED_CORPUS
This should have been added earlier as @cdecker suggested, but is needed
to enable CI testing.
Changelog-Changed: Reckless - added support for networks beyond bitocoin and regtest
In particular:
- Bolt 4: add route blinding construction
- Bolt 4: add blinded payments
And this means it's not experimental, so we can turn it on
by default!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: blinded payments are now supported by default (not just with `--experimental-onion-messages`)
"Allow nodes to overshoot final htlc amount and expiry (#1032)"
Note that this also renamed `min_final_cltv_expiry` to the more-correct
`min_final_cltv_expiry_delta`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
"Allow nodes to overshoot the MPP `total_msat` when paying (#1031)"
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Allow slight overpaying, even with MPP, as spec now recommends.
"BOLT 4: Remove legacy format, make var_onion_optin compulsory."
This also renamed the redundant "tlv_payload" to "payload", so we
replace "tlv_tlv_payload" with "tlv_payload" everyhere!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Inside our integration testing we get another timeout,
so this commit adds a timeout to the waitpay command to avoid waiting forever.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We do this for HTLCs which will timeout to them: we watch them in case we
want to fulfill them as a preimage comes in, but once they reach depth we
can forget about them.
We change the message, which causes some more test churn.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using single tuples in Python is ugly, so:
1. Rename wait_for_onchaind_tx to wait_for_onchaind_txs.
2. Make it take tuples explicitly.
3. Make wait_for_onchaind_tx a simpler wrapper/unwrapper.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is when they closed the channel, we can simply make our own tx to
expire the HTLC. (The other case is where we closed the channel, and
we have a special htlc_timeout tx which we have their signature for).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This breaks tests/test_closing.py::test_onchain_all_dust's accouting
checks.
That test doesn't really test what it claims to test; sure, onchaind
*says* it's going to ignore the output due to high fees, but the tx
still gets mined.
I cannot figure out what the test is supposed to look like, so I
simply disabled the accounting checks :(
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We'll want this, as lightningd will want to produce htlc txs based on
what it's told from onchaind, so we need a lower-level accessor.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can no longer grab the tx in one line as we did with
wait_for_onchaind_broadcast, we need to track the broadcast from
lightningd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we do both our own internal handling and handing it to
lightningd, we add to `proposed_resolution` to handle the lightningd
case.
Note, in particular, that we fix the blockheight calculation: it's out
by one, in that if we see a tx and our CSV lock is 5, we only need to
wait 4 more blocks, not 5. This will matter as we start using it, and
convert the tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We add code for the case of spending a (timelocked) to-us output of an
HTLC output, so lightningd can do it (rather than onchaind doing all
the work itself).
onchaind still needs to know whether we bothered to create the tx
(fees might have caused it to evaporate, so it should consider it
immediately resolved rather than waiting for it), and what the
witnesses were, and which parts of the witnesses were signatures (as
these parts might change, with RBF or in future, combining other txs).
The inputs (known to onchaind) and the witnesses (told by lightningd)
uniquely identify the spend for the purposes of onchaind. In
particular, they definitely distinguish HTLC-timeout and HTLC-success
cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We previously used WIRE_HSMD_SIGN_DELAYED_PAYMENT_TO_US,
WIRE_HSMD_SIGN_REMOTE_HTLC_TO_US, WIRE_HSMD_SIGN_PENALTY_TO_US and
WIRE_HSMD_SIGN_LOCAL_HTLC_TX which allow onchaind to sign txs,
but only for its specific channel.
We now want lightningd to sign these, but it's not bound to a specific
channel. So let's add variants that don't require that.
We are also now explicit about *what input* to sign. It's always zero
for now, but future combinations may change that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Importantly, the code in jsonrpc.c which actually does the io_break:
```
/* Once the stop_conn conn is drained, we can shut down. */
if (jcon->ld->stop_conn == conn && jcon->ld->state == LD_STATE_RUNNING) {
/* Return us to toplevel lightningd.c */
log_debug(jcon->ld->log, "io_break: %s", __func__);
io_break(jcon->ld);
```
By having the state not set until later, we avoid running this. Of course,
we need to avoid calling the main loop when we get there, if we've already
been told to shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In CI we see crashes in this case:
```
lightningd: lightningd/connect_control.c:734: void connectd_activate(struct lightningd *): Assertion `ret == ld->connectd' failed.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a visitor that ensures every new field has at least an `added`
field, and that we don't change the `added` or `deprecated` annotation
after the fact.
Changelog-Changed: msggen: The generated interfaces `cln-rpc` anc `cln-grpc` can now work with a range of versions rather than having to match the CLN version
This patch annotates the fields with a new `optional` attribute which
determines whether the field should be considered an inferred optional
due to being added or deprecated.
The patching system allows us to enrich the raw schema with some
additional information. In this specific case we want to backfill the
`added` and `deprecated` fields for the multiversion support.
I couldn't figure out why my new SQL query was returning 0 rows,
and it was because we were ignoring errors.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CI seems to block; Christian suggests the throttler may be to blame somehow?
Since trying to fix it made it worse, let's just remove it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Firstly, amount should not be `static`, so use a separate line to
declare those (fee is static, as it's cached across calls).
Secondly, new_tracked_output doesn't take(), it copies.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only do this in one place now, but we're going to add another.
Also, make queued messages const.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fun story. We're changing onchaind to hand txs to us, and we will
construct them and do the broadcast for it. lightningd tells onchaind
the witness it used (with flags to indicate which fields were
signatures so should be ignored) so onchaind can recognize the tx
when/if it is mined.
And when onchaind was waiting for a CLTV delay, it wouldn't tell
lightningd yet, but wait until the parent was sufficiently deep
But this caused bugs!
In particular, on replay, onchaind would see transactions which it
hasn't sent yet. This was not a problem before, as onchaind had
created the tx, even if it hadn't told lightningd to broadcast it, so
recognized the variant when it came in. When we're relying on
lightningd to tell us what the tx will look like, this doesn't work
any more.
The cause of this is that we fire off txowatches ("this output was
spent!") while we process blocks, and only fire off txwatches ("this
tx increased depth") once all the current blocks are processed. Often
this didn't matter, since we replay messages to onchaind from the
database, *but* we trim the last few blocks on restart (or, if there's
a small reorg while we're stopped), and we can hit this misordering.
Changing our topology code to only ever process one block at a time
would be a solution, but slows down catchup (and tests, where we often
mine a run of blocks).
So, this seems like a premature optimization, but it's really
required! And in future, lightningd can use this knowledge of pending
transactions to combine them in more clever ways.
Note that if a tx is valid at block N, we broadcast it once we see
block N-1, to get it in the mempool for block N.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was once only called on failure, now it's always called (if set).
It was called different things in different places, so unify it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We always call channel_fail_transient() on all channels when a peer
connects, to clean up any previous connections. However, when
we startup, this channel doesn't have an owner yet, resulting in
a fairly weird INFO level message.
Reported-by: Michael Schmook @mschmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `lightningd`: don't log gratuitous "Peer transient failure" message on first connection after restart.
There are cases (difficult to reproduce with a test) where
a payment will fail one time and succeed later.
As far I understand in this case the groupid field of the payment
is the same, and the only thing that change is the status, so
our logic inside the delpay is ambiguous where it is not
possible to delete a payment as described in https://github.com/ElementsProject/lightning/issues/6114
A sequence of commands that explain the problem is
```
$ lc -k listpays payment_hash=H
{
"pays": [
{
"bolt11": "I",
"destination": "redacted",
"payment_hash": "H",
"status": "complete",
"created_at": redacted,
"completed_at": redacted,
"preimage": "P",
"amount_msat": "redacted",
"amount_sent_msat": "redacted"
}
]
}
$ lc delpay H complete
{
"code": 211,
"message": "Payment with hash H has failed status but it should be complete"
}
```
In this case, the delpay is not able to delete a payment because the
listpays is returning only the succeeded one, so by running the
listsendpays we may see the following result where our delpay logic
will be stuck because it works to ensure that all the payments stored
in the database has the status specified by the user
```
➜ VincentSSD clightning --testnet listsendpays -k payment_hash=7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4
{
"payments": [
{
"id": 322,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 1,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 1664,
"created_at": 1679510203,
"completed_at": 1679510205,
"status": "failed",
"bolt11": "lntb1pjpkj4xsp52trda39rfpe7qtqahx8jjplhnj3tatxy8rh6sc6afgvmdz7n0llspp50lr5hmdm0re0xvcp2hv3nf2wwvx0r8q3h3e7jmqz0awdfg6w206qdp0w3jhxarfdenjqargv5sxgetvwpshjgrzw4njqun9wphhyaqxqyjw5qcqp2rzjqtp28uqy77te96ylt7ek703h4ayldljsf8rnlztgf3p8mg7pd0qzwf8a3yqqpdqqqyqqqqt2qqqqqqgqqc9qxpqysgqgeya2lguaj6sflc4hx2d89jvah8mw9uax4j77d8rzkut3rkm0554x37fc7gy92ws9l76yprdva2lalrs7fqjp9lcx40zuty8gca0g5spme3dup"
},
{
"id": 323,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 2,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 3663,
"created_at": 1679510205,
"completed_at": 1679510207,
"status": "failed"
},
{
"id": 324,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 3,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 3663,
"created_at": 1679510207,
"completed_at": 1679510209,
"status": "failed"
},
{
"id": 325,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 4,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 4663,
"created_at": 1679510209,
"completed_at": 1679510221,
"status": "complete",
"payment_preimage": "43f746f2d28d4902489cbde9b3b8f3d04db5db7e973f8a55b7229ce774bf33a7"
}
]
}
```
This commit solves the problem by forcing the delete query in the
database to specify status too, and work around this kind of
ambiguous case.
Fixes: f52ff07558 (lightningd: allow delpay to delete a specific payment.)
Reported-by: Antoine Poinsot <darosior@protonmail.com>
Link: https://github.com/ElementsProject/lightning/issues/6114
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Co-Developed-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: delpay be more pedantic about delete logic by allowing
delete payments by status directly on the database.
- moves offset into GossipHeader hdr which is passed to all constuctors
- reads .flags as u16 instead of extracting it from the .length, see 0274d88ba
- adds zombie and ratelimit flag to GossipHeader
- bytes_read start at 0 instead of 1 which is more correct,
the one byte is then corrected for when setting the offset of new header.
- bytes_read is increased in pull_bytes as this is the only place where
something is read
- use new style for various format-strings
It's defined to be nonull:
```
channeld/channeld.c:2381:2: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/stdlib.h:856:3: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior channeld/channeld.c:2381:2 in
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With the warning that we were trying to put "inf" into a u64, we can
see that this calculation was wrong to use integers!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For example, if we use -fsanitize=undefined, we can't do unaligned
integer access, but since we didn't test with the sanitizer flags, we
didn't know this, and set `HAVE_UNALIGNED_ACCESS=1`.
Also, add -fno-sanitize-recover= in developer mode, so we actually
fail binaries if something is detected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Most importantly, configurator used to use bitshifts on signed
integers which -fsanitize=undefined caught.
But also, tal played fast and loose with typing and aliases, which was
a signficant amount of rework.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
pubkey_from_hexstr() was failing, which we didn't notice because we
weren't checking the return value. The problem was that we were passing
it a strlen that was half the actual length.
Relevant error:
[libsecp256k1] illegal argument: !secp256k1_fe_is_zero(&ge->x)
==417723== ERROR: libFuzzer: deadly signal
#7 0x7f5deaacc7fb in abort
#8 0x51b0b0 in secp256k1_default_illegal_callback_fn secp256k1.c
#9 0x51bd8e in secp256k1_ec_pubkey_serialize
#10 0x4e235b in pubkey_to_der bitcoin/pubkey.c:29:7
#11 0x4e2941 in pubkey_cmp bitcoin/pubkey.c:89:2
#12 0x4e333d in bitcoin_redeem_2of2 bitcoin/script.c:144:6
#13 0x4f1396 in run tests/fuzz/fuzz-close_tx.c:78:19
We have 3 personas:
- Users
- Developers
- Maintainers
The first one basically cover the installation documentation. The
latter two are sorted into the "Developer" category, and the reference
category serves as a quick lookup for facts on anything CLN related.
Changelog-Added: JSON-RPC: `listclosedchannels` to show old, dead channels we previously had with peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't handle the case of an array inside a subobject. But that
happens when we add the next commit!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't cover three common patterns:
1. Optional integers (db_col_u64 has different form from structs)
2. Optional strings.
3. Optional array fields.
But it does neaten and reduce the scope for cut&paste errors in the
common "if not-NULL, tal and assign".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We actually have an assertion that there are no channels remaining when
we delete peers, so this is confusing!
Actually removing the constraint is db-specific and deeply non-trivial.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
While we are cleaning up the list forwards with the autoclean plugin we are
not taking into count the forward's payments with the status set to
`local_failed`. In this case, the forwards have no resolved
time because it was not resolved by us due to some local error.
So, this commit is fixing the auto clean plugin by allowing to delete
of the forwards with status set to local_failed by taking into count
the received_time, with the assumption that the received_time, in this case,
is equal to the resolved time (?)
Reported-by: @denis2342
Link: https://github.com/ElementsProject/lightning/issues/6058
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Fixed: plugin: autoclean: considerer the forwards with status
set to `local_failed`.
Also, fix the case where we didn't use --network with EXPOSE_TCP,
as reported by @theborakompanioni:
```
I get Wrong network! Our Bitcoin backend is running on 'regtest', but we expect 'main'. with LIGHTNINGD_NETWORK := regtest when param --network is not provided.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
PSBTv2 support is quite low in the ecosystem, so having a call to convert
log messages and the like should be useful since they'll often be in v2.
Changelog-Added: Added setpsbtversion RPC to aid debugging and compatibility
Libwally update breaks compatibility, so
we do this in one large step.
Changelog-Changed: JSON-RPC: elements network PSET now only supports PSETv2.
Changelog-Added: JSON-RPC: PSBTv2 supported for fundchannel_complete, openchannel_update, reserveinputs, sendpsbt, signpsbt, withdraw and unreserveinputs parameter psbt, openchannel_init and openchannel_bump parameter initialpsbt, openchannel_signed parameter signed_psbt and utxopsbt parameter utxopsbt
The issue is that common_setup() wasn't called by the fuzz target,
leaving secp256k1_ctx as NULL.
UBSan error:
$ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1" \
./fuzz-channel_id crash-1575b41ef09e62e4c09c165e6dc037a110b113f2
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1153355603
INFO: Loaded 1 modules (25915 inline 8-bit counters): 25915 [0x563bae7ac3a8, 0x563bae7b28e3),
INFO: Loaded 1 PC tables (25915 PCs): 25915 [0x563bae7b28e8,0x563bae817c98),
./fuzz-channel_id: Running 1 inputs 1 time(s) each.
Running: crash-1575b41ef09e62e4c09c165e6dc037a110b113f2
bitcoin/pubkey.c:22:33: runtime error: null pointer passed as argument 1, which is declared to never be null
external/libwally-core/src/secp256k1/include/secp256k1.h:373:3: note: nonnull attribute specified here
#0 0x563bae41e3db in pubkey_from_der bitcoin/pubkey.c:19:7
#1 0x563bae4205e0 in fromwire_pubkey bitcoin/pubkey.c:111:7
#2 0x563bae46437c in run tests/fuzz/fuzz-channel_id.c:42:3
#3 0x563bae2f6016 in LLVMFuzzerTestOneInput tests/fuzz/libfuzz.c:23:2
#4 0x563bae20a450 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)
#5 0x563bae1f4c3f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long)
#6 0x563bae1fa6e6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))
#7 0x563bae223052 in main (tests/fuzz/fuzz-channel_id+0x181052) (BuildId: f7f56e14ffc06df54ab732d79ea922e773de1f25)
#8 0x7fa7fa113082 in __libc_start_main
#9 0x563bae1efbdd in _start
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior bitcoin/pubkey.c:22:33 in
`struct lightningd` is not completely initialized, so we added a
"migration_context" which only had some of the fields. But we ended
up handing in `struct lightningd` anyway, so I don't think this
complexity is worthwhile.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
At the moment only lightingd needs it, and this avoids missing any
places where we do bip32 derivation.
This uses a hsm capability to mean we're backwards compatible with older
hsmds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: we now always double-check bitcoin addresses are correct (no memory errors!) before issuing them.
It's needed as the db and wallet is being set up (db migrations), so
it's simpler this way to always use ld->bip32_base for the next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Importantly, adds the version number at the *front* to help future
parsing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'fix-hsm-check-pubkey.patch':
fixup! hsmd: capability addition: ability to check pubkeys.
We were handing 3 to hsmd (and Ken added that in 7b2c5617c1,
so I guess he's OK with that being the minimum supported version!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We would sleep for 10msec (default) and try again, spamming the logs
every second. But we're waiting for revoke_and_ack, and that handler
already sets off the timer, so there's no need to spin at all!
Fixes: #6077
Changelog-Fixed: `channeld`: no longer spin and spam logs when waiting for revoke_and_ack.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This changes connectd to use `status_fail()` on TOR problems during statup
instead of `err()`. Using `err()` did not write to the logfile.
To find out TOR problems during startup, the user needed to stop the system
daemon and call `lightningd` manually in console to see the error.
`status_fail()` logs and exits, but also prints a whole stacktrace,
which is a bit too much imho on config errors. But currently there is
no `status_SOMETHING` method that logs, prints and exists on an error
without stacktrace.
Changelog-None
While the user trying to fetch an invoice by specifing the quantity we do
not work as expected.
Running the command
```
lightning-cli fetchinvoice -k offer='lno1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcgqffqszsk2p6hycmgv9ek2grpyphxjcm9ypmkjer8v46pyzmhd9jxwet5wvhxxmmdzsqs593pq0ylsvakdua5h976f4g3eautgjt3udvtyga47eaw7339sjrhpwpwz' quantity=2
```
and we answer back with
```json
{
"code": -32602,
"message": "quantity parameter required"
}
```
This is caused because we forget to bind the `quanity` field from the
RPC into the `invrequest`.
Reported-by: @aaronbarnardsound
Link: https://github.com/ElementsProject/lightning/issues/6089
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-EXPERIMENTAL: fetchinvoice: fix: do not ignore the `quantity` field
into the invreq field.
When plugins receive a "shutdown" notification, then can call this
method which will shutdown `cln_plugin`.
Then they can await `plugin.join()` and do any remaining cleanup
there.
This helps avoid a pain-point where plugin authors need to handle
2 separate plugin shutdown mechanisms https://github.com/ElementsProject/lightning/issues/6040
It seems that bitcoind frequently dies on this test. I assume running
the multiple nodes under valgrind with the extra 214 blocks is too
memory-hungry?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: require the `"jsonrpc": "2.0"` property (requests without this deprecated in v0.10.2).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: `checkmessage` now always returns an error when the pubkey is not specified and it is unknown in the network graph (deprecated v0.12.0)
This reintroduce lnprototest after 2 releases,
there was a lot of breaking around it and this
will patch them (most of them)!
However, there are some issue related to channel opening and closing
that need some additional love and are disabled for now, but I think it
is good to introduce lnprototest now again in the CI, to be able to
stress the fix for now and see if there are other problem around.
I will take care of it!
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
It's still deprecated: we need the description since
1. This information is useful for any validation we want to do, such as
the HSM, or runes.
2. We want this information in listpays so we can tell what we actually paid.
3. In general, we should never sign commitments to things we don't have!
I expect to have this information about payments *whatever the frontend* is,
which is why we deprecated (and then removed) this unintended use. The spec
is pretty clear on this:
BOLT #11:
```
A reader:
...
- MUST check that the SHA2 256-bit hash in the `h` field exactly matches the hashed
description.
```
However, neither BTCPayServer nor lnbits updated despite the long deprecation
period, so revert 2afe7a1856.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Don't refer to obsolete send_invoice flag.
2. Don't refer to obsolete quantity_min field.
3. Don't refer to unsigned vs signed offers: they're all unsigned.
4. Add references to invoicerequest(7).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As reported on Discord, these are undocumented. And thus, um, hard to find!
Reported-by: Aaron Barnard
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Seems like LND is hanging up on receiving these messages, even though
they're odd :(
So, when a peer connects, check if it supplies or wants peer backup
(even if it doesn't support both, it shouldn't hang up, and I didn't
want to separate the two paths).
And when we go to send our own, updated backup, check features before
sending.
Fixes: #6065
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `experimental-peer-storage` caused LND to hang up on us, so only send to peers which support it.
e778ebb9af ("wallet: only log broken if we
have duplicate scids in channels.") downgraded the fatal() to a broken
log message, but the user reports it still won't start up.
Perhaps they're hitting the fatal() outside the loop? (And we're
not getting that output).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
ccan/io stores the context pointer for io_new_conn, but we were using
`daemon->listeners` which we reallocate, so it can use a stale pointer.
```
0x3e1700 call_error
ccan/ccan/tal/tal.c:93
0x3e1700 check_bounds
ccan/ccan/tal/tal.c:165
0x3e1700 to_tal_hdr
ccan/ccan/tal/tal.c:174
0x3e1211 to_tal_hdr_or_null
ccan/ccan/tal/tal.c:186
0x3e1211 tal_alloc_
ccan/ccan/tal/tal.c:426
0x3db8f4 io_new_conn_
ccan/ccan/io/io.c:91
0x3dd2e1 accept_conn
ccan/ccan/io/poll.c:277
0x3dd2e1 io_loop
ccan/ccan/io/poll.c:444
0x3419fa main
connectd/connectd.c:2081
```
Fixes: #6060
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This reverts us to the v22.11 behaviour, pending a revisit for the
next release.
Changelog-Changed: gossipd: revert zombification change, keep all gossip for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this patch, we only ever loaded the "nodes" table once, then
didn't see updates.
How this ever got past CI is a mystery; perhaps valgrind was so slow that
the updated node_announcement hit the gossmap before we even asked sql
on l3 about the nodes table?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `sql` nodes table now gets refreshed when gossip changes.
Loading the gossip_store would not create a pending node announcement
when the node already had a zombie channel. This would cause the node
announcement to attempt to be loaded, but fail because it had no
broadcastable channels. Accepting a pending node announcement as when
normally loading from the channel corrects this.
`node_has_public_channels` taking into account zombie channels enables
this behavior.
Separately, node_announcements were still being flagged as zombies
in the gossip store despite that feature being removed.
Changelog-None
Without inheriting zombie status, gossipd would allow regular channel updates
into the store until the pruning cycle hits (and the channel is properly
flagged) which is 3.5 days. Applying zombie status when reading channel
updates from the store prevents this.
Changelog-None
remove_chan_from_node already corrects the ordering if a node_announcement
is left ahead of the next oldest channel_announcement, but zombifying should
do that check (and reorder if necessary) too.
Changelog-None
Since it is an optional field in the `listconfigs` output we can't use
the `rpc_scan` mechanism (doesn't handle optionals yet). We'll use
that list of accepted types later to avoid stripping them.
Commit a418615b7f ("rpc: adds
num_channels to listpeers") broke the sql tests. Turns out, no
openchannel v2 tests are run in CI!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This commit will disable the peerstorage plugins
when the feature is not enabled.
I found this issue with lnprototest, and I guess
we did not find it with normal run because
other the unknown messages are ingored?
Changelog-Fixed: Disable the protocol messages when peerstorage is disabled.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
It works for the trivial case, where groupid and partid are the same,
but silently deletes nothing in the other cases (or worse, deletes the
wrong entry!).
See: #5835
Changelog-Fixed: `delpay`: actually delete the specified payment (mainly found by `autoclean`).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we had two things to clean, we fired off two requests (eg.
listforwards and listinvoices) and both marked the timer as finished,
triggering an assert.
We already have a refcount for outstanding requests to avoid this
for e.g. outstanding del commands, so use it here too!
```
Jan 19 19:20:00 lightningd[748044]: autoclean: plugins/libplugin.c:445: timer_complete: Assertion `p->in_timer > 0' failed.
Jan 19 19:20:00 lightningd[748044]: autoclean: FATAL SIGNAL 6 (version v22.11.1)
Jan 19 19:20:00 lightningd[748044]: 0x562c388136e4 send_backtrace
Jan 19 19:20:00 lightningd[748044]: common/daemon.c:33
Jan 19 19:20:00 lightningd[748044]: 0x562c3881376c crashdump
Jan 19 19:20:00 lightningd[748044]: common/daemon.c:46
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0898d5f ???
Jan 19 19:20:00 lightningd[748044]: ./signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0898ce1 __GI_raise
Jan 19 19:20:00 lightningd[748044]: ../sysdeps/unix/sysv/linux/raise.c:51
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0882536 __GI_abort
Jan 19 19:20:00 lightningd[748044]: ./stdlib/abort.c:79
Jan 19 19:20:00 lightningd[748044]: 0x7f26d088240e __assert_fail_base
Jan 19 19:20:00 lightningd[748044]: ./assert/assert.c:92
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0891661 __GI___assert_fail
Jan 19 19:20:00 lightningd[748044]: ./assert/assert.c:101
Jan 19 19:20:00 lightningd[748044]: 0x562c3880355d timer_complete
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:445
Jan 19 19:20:00 lightningd[748044]: 0x562c38800b54 clean_finished
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:122
Jan 19 19:20:00 lightningd[748044]: 0x562c388010ed clean_finished_one
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:132
Jan 19 19:20:00 lightningd[748044]: 0x562c388011b6 del_done
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:149
Jan 19 19:20:00 lightningd[748044]: 0x562c388058b5 handle_rpc_reply
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:768
Jan 19 19:20:00 lightningd[748044]: 0x562c38805a39 rpc_read_response_one
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:944
Jan 19 19:20:00 lightningd[748044]: 0x562c38805ad7 rpc_conn_read_response
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:968
Jan 19 19:20:00 lightningd[748044]: 0x562c38876b60 next_plan
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/io.c:59
Jan 19 19:20:00 lightningd[748044]: 0x562c38876fe7 do_plan
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/io.c:407
Jan 19 19:20:00 lightningd[748044]: 0x562c38877080 io_ready
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/io.c:417
Jan 19 19:20:00 lightningd[748044]: 0x562c3887823c io_loop
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/poll.c:453
Jan 19 19:20:00 lightningd[748044]: 0x562c38805d11 plugin_main
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:1801
Jan 19 19:20:00 lightningd[748044]: 0x562c38801c7a main
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:613
```
Fixes: #5912
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Missed a DEFAULT in the db clause.
Feb 15 16:02:12 citrine lightningd[902093]: Accessing a null column lease_satoshi/15 in query SELECT funding_tx_id, funding_tx_outnum, funding_feerate, funding_satoshi, our_funding_satoshi, funding_psbt, last_tx, last_sig, funding_tx_remote_sigs_received, lease_expiry, lease_commit_sig, lease_chan_max_msat, lease_chan_max_ppt, lease_blockheight_start, lease_fee, lease_satoshi FROM channel_funding_inflights WHERE channel_id = ? ORDER BY funding_feerate
Fixes#6016
Closing channels would previously require moving the node announcements
in the gossip store on occasion. They incorrectly lost their spam flag
during this process (would no longer be squelched.)
Changelog-None
A zombie channel is not considered broadcastable, so if all channels
are zombies (i.e. is_node_zombie() is true), then
node_has_broadcastable_channels() is false.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies things (we'll get node_announcement if they ever
rebroadcast), since we clearly have an issue with node_announcement for
zombie nodes.
Changes:
1. Remove now-unused gossip_store_mark_nannounce_zombie and resurrect_nannouncements.
2. Don't consider zombie channels to count when deciding whether to move node_announcement
(node_announcement must be preceded by at least one broadcastable channel_announcement).
3. Treat incoming node_announcement where we have all-zombie channels the same as if
we had no channels.
4. Remove node_announcement whenever we have no announcable channels (not just zombies).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This could always happen if we armed the timer when we did have public
channels, and by the time we did our node_announcement we no longer
did, but it gets triggered in our tests when we remove (our own!)
zombied node_announcement in the next patch.
This will fix a crash that I caused on armv7
and by looking inside the coredump with gdb
(by adding an assert on n that must be
different from null) I get the following stacktrace
```
(gdb) bt
\#0 0x00000000 in ?? ()
\#1 0x0043a038 in send_backtrace (why=0xbe9e3600 "FATAL SIGNAL 11") at common/daemon.c:36
\#2 0x0043a0ec in crashdump (sig=11) at common/daemon.c:46
\#3 <signal handler called>
\#4 0x00406d04 in node_announcement (map=0x938ecc, nann_off=495146) at common/gossmap.c:586
\#5 0x00406fec in map_catchup (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:643
\#6 0x004073a4 in load_gossip_store (map=0x938ecc, num_rejected=0xbe9e3a40) at common/gossmap.c:697
\#7 0x00408244 in gossmap_load (ctx=0x0, filename=0x4e16b8 "gossip_store", num_channel_updates_rejected=0xbe9e3a40) at common/gossmap.c:976
\#8 0x0041a548 in init (p=0x93831c, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., config=0x939cdc) at plugins/topology.c:622
\#9 0x0041e5d0 in handle_init (cmd=0x938934, buf=0x9399d4 "\n\n{\"jsonrpc\":\"2.0\",\"id\":\"cln:init#25\",\"method\":\"init\",\"params\":{\"options\":{},\"configuration\":{\"lightning-dir\":\"/home/vincent/.lightning/testnet\",\"rpc-file\":\"lightning-rpc\",\"startup\":true,\"network\":\"te"..., params=0x939c8c)
at plugins/libplugin.c:1208
\#10 0x0041fc04 in ld_command_handle (plugin=0x93831c, toks=0x939bec) at plugins/libplugin.c:1572
\#11 0x00420050 in ld_read_json_one (plugin=0x93831c) at plugins/libplugin.c:1667
\#12 0x004201bc in ld_read_json (conn=0x9391c4, plugin=0x93831c) at plugins/libplugin.c:1687
\#13 0x004cb82c in next_plan (conn=0x9391c4, plan=0x9391d8) at ccan/ccan/io/io.c:59
\#14 0x004cc67c in do_plan (conn=0x9391c4, plan=0x9391d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
\#15 0x004cc6dc in io_ready (conn=0x9391c4, pollflags=1) at ccan/ccan/io/io.c:417
\#16 0x004cf8cc in io_loop (timers=0x9383c4, expired=0xbe9e3ce4) at ccan/ccan/io/poll.c:453
\#17 0x00420af4 in plugin_main (argv=0xbe9e3eb4, init=0x41a46c <init>, restartability=PLUGIN_STATIC, init_rpc=true, features=0x0, commands=0x6167e8 <commands>, num_commands=4, notif_subs=0x0, num_notif_subs=0, hook_subs=0x0, num_hook_subs=0, notif_topics=0x0, num_notif_topics=0) at plugins/libplugin.c:1891
\#18 0x0041a6f8 in main (argc=1, argv=0xbe9e3eb4) at plugins/topology.c:679
```
I do not know if this is a solution because I do not know
when I can parse a node announcement for a node that
it is not longer in the gossip map.
So, I hope this is just usefult for @rustyrussell
Changelog-Fixed: fixes `FATAL SIGNAL 11` on gossmap node announcement parsing.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
When doing some plugin related work, I discovered that the datastore API
has two issues:
- Error messages on startup of plugins init method when the datastore is
still completely empty: "Parsing '{datastore:[0:': token has no index 0: []"
- Data is escaped but not unwrapped again when sending and getting from
the API.
[ Removed xfail, it now passes! --RR ]
Closes: #5990
We were feeding in the raw JSON, which escapes \". Then we were
escaping *again* to return it.
Reported-by: @m-schmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `datastore` handles escapes in `string` parameter correctly.
This fixes the following compilation error
and allow rebuilding again on 32-bit platform.
```
lightningd/dual_open_control.c: In function 'validate_input_unspent':
lightningd/dual_open_control.c:2627:43: error: format '%llu' expects argument of type 'long long unsigned int', but argument 4 has type 'size_t' {aka 'unsigned int'} [-Werror=format=]
2627 | err = tal_fmt(pv, "PSBT input at index %"PRIu64
| ^~~~~~~~~~~~~~~~~~~~~~~
2628 | " missing serial id", i);
| ~
| |
| size_t {aka unsigned int}
ccan/ccan/tal/str/str.h:43:46: note: in definition of macro 'tal_fmt'
43 | tal_fmt_(ctx, TAL_LABEL(char, "[]"), __VA_ARGS__)
| ^~~~~~~~~~~
```
PS: apparently I'm the only remaining people that ran cln on an old raspberry pi 2?
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
People are upgrading to 22.11.1 not, and in some configurations like the one
mentioned in the issue, we should
put some info information in the log when we are not able to upgrade.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
These were deprecated in v0.12.0, hence scheduled for removal next version anyway
(use local_fund_msat and remote_funds_msat).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since it's not spec-final yet (hell, it's not even properly specified
yet!) we need to put it behind an experimental flag.
Unfortunately, we don't have support for doing this in a plugin; a
plugin must present features before parsing options. So we need to do
it in core.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When you return an allocated pointer, you should always hand in the
context you want it allocated from. This is more explicit, because it may
really matter to the caller!
This also folds some simple operations, and avoids doing too much
variable assignment in the declarations themselves: some coding styles
prohibit such initializers, but that's a bit exteme.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And we should always represent them as is, not as optional: it's
possible in future we could *require* "WANT_PEER_BACKUP_STORAGE".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is needed for the next patch, which does this from the peer_connected hook!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `sendcustommsg` can now be called by a plugin from within the `peer_connected` hook.
We keep the node_id, not a pointer to the peer.
This also means that it might have reconnected while we were in the hook, so make
sure we ignore the result if it's in state PEER_CONNECTED.
And remove the `tal_steal(peer, hook_payload)` which doesn't do anything: the
plugin_hook call steals hook_payload anyway!
Fixes: #5944
- Renamed zerod_channel_ids to temporary_channel_id
- Renamed witness_stack->witnesses
- Renamed witness_element->witness_elements
- open_channel2 now includes second commitment point
- accept_channel2 now includes second commitment point
Current commit on rfc branch 64f7f360b9f3c2664d078e2129cfe83098fc4617
Changelog-EXPERIMENTAL: Protocol: dual-funding spec changed in incompatible ways, won't work with old versions (but maybe soon with Eclair!!)
It's not likely but possible that the node's settings will shift btw a
start and an RBF; we persist the setting to the database so we don't
lose it.
Right now holding onto it forever is kind of extra but maybe we'll
reuse the setting for splices? idk.
Should this be a channel type??
`openchannel_init` takes a psbt, which we pipe over to dualopend
process.
If the peer requests that they'll only accept confirmed inputs, we need
to go validate those before we continue.
This wires up the harness for this (validation check yet tc)
technically we don't need this info after the channel opens, but for any
subsequent RBF (and maybe splice?) we need to remember what the
open/accept peer signaled
not amazing, since we'll probably call openchannel_update multiple
times per open, but this is the simplest way to confirm that we're
not sending unconfirmed outputs to peer.
This will save a lot of RPC ping/pong when plugins still need to iterate
both, `listpeers` and `listpeerchannels`. When `num_channels` is 0 they
can skip additional calls.
Changelog-Added: RPC `listpeers` output now has `num_channels`.
Though there's already a `createinvoice` command, there are usecases where a
user may want to sign an invoice that they don't yet have the preimage to. For
example, they may have an htlc_accepted plugin that pays to obtain the preimage
from someone else and returns a `{ "result": "resolve", ... }`.
This RPC command addresses this usecase without overly complicating the
semantics of the existing `createinvoice` command.
Changelog-Added: JSON-RPC: `signinvoice` new command to sign BOLT11
invoices.
The existing description is incorrect. `createinvoice` doesn't actually
work when supplied with a custom-encoded bolt11 invoice without the
final 520 signature bits appended. If a users tries to do so, some of
their tagged fields will be incorrectly truncated.
`createinvoice` actually expects that the signatures are there, and it
simply ignores them.
See common/bolt11.c's bolt11_decode_nosig:
/* BOLT #11:
*
* The data part of a Lightning invoice consists of multiple sections:
*
* 1. `timestamp`: seconds-since-1970 (35 bits, big-endian)
* 1. zero or more tagged parts
* 1. `signature`: Bitcoin-style signature of above (520 bits)
*/
if (!pull_uint(&hu5, &data, &data_len, &b11->timestamp, 35))
return decode_fail(b11, fail, "Can't get 35-bit timestamp");
> while (data_len > 520 / 5) {
const char *problem = NULL;
u64 type, data_length;
This allows to accept safely long paths as options
and does not truncate them
as https://github.com/ElementsProject/lightning/issues/5576
described
Changelog-Fixed: cli: accepts long paths as options
Suggested-by: @rustyrussell <rusty@rustcorp.com.au>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
In v0.11 (71f736678f) we changed lightningd to wait for gossipd to
acknowledge blocks before updating blockheight: this resolved a problem
which lnprototest had where it wanted to know when we'd fully digested
a block.
However, it broke the syncing case: until then we don't even tell
gossipd, so this stayed at zero. We should use the current blockheight
for that corner case!
Fixes: #5894
Changelog-Fixed: JSON-RPC: `getinfo` `blockheight` no longer sits on 0 while we sync with bitcoind the first time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Added Alex since he's Release Captain this time.
Changelog-Added: SECURITY.md: Where to send sensitive bug reports, and dev GPG fingerprints.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We might be disconnected, but the subd isn't dead yet:
```
> assert out[0] == '# peer is offline, will negotiate once they reconnect (5 seconds before unilateral close).'
E AssertionError: assert '# Timed out, forcing close.' == ('# peer is offline, will negotiate once they reconnect (5 seconds before '\n 'unilateral close).')
E - # peer is offline, will negotiate once they reconnect (5 seconds before unilateral close).
E + # Timed out, forcing close.
tests/test_closing.py:164: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
make the number of blocks mined father away from the cltv timeout
from borked/flakey test run:
lightningd-3 2023-01-26T21:45:19.261Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: billboard perm: Received error channel 27a4a4dd880e86
1e390517de3e786a237c5ad1f00faab277382664e76b5c3870: Fulfilled HTLC 0 SENT_REMOVE_COMMIT cltv 116 hit deadline
Wait until we get a tx-abort back to terminate the process.
Nota Bene: this can cause RPC calls to hang if the peer never
responds back with tx-abort.
Note that we also have to re-route how open-abort + negotiation_failed
handle failures, as open_abort no longer closes the process
automagically.
When a channel open fails, we use tx-abort instead of warning/error.
This means that the peer won't disconnect! And instead when a new
message arrives, we'll need to rebuild the dualopend subd (if missing).
Makes opens a bit easer to retry (no reconnect needed), as well as keeps
the connection alive for other channels we may have with that peer.
Changelog-Changed: Experimental-Dual-Fund: open failures don't disconnect, but instead fail the opening process
Dropping the connection is bad behavior on an openchannel failure,
especially given that there might be other channels currently connected.
We should maintain the connection but close out the dualopend
daemon for that attempt.
This test partially documents the behaivor, but fails
Changelog-None
```
----------------------------- Captured stderr call -----------------------------
Sending onchaind an invalid message 03ed00000000000000004e52a9129a66619d6809b1024eb9a0159f173a988f3a5d0bdd2447b4fcc24cef
lightningd: FATAL SIGNAL 6 (version 3c57147-modded)
```
The channel state can also be `FUNDING_SPEND_SEEN` if onchaind is still
starting up.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
v2 opens require you to use native segwit inputs
Changelog-Added: JSONRPC: `upgradewallet` command, sweeps all p2sh-wrapped outputs to a native segwit output
We need to be able to only use non-wrapped inputs for v2/interactive tx
protocol.
Changelog-Added: JSONRPC: `fundpsbt` option `nonwrapped` filters out p2sh wrapped inputs
The `tmpctx` is free'd before the error is read out/sent over the wire;
there's a call that will copy the array before sending it, let's use
that instead and take() the object?
------------------------------- Valgrind errors --------------------------------
Valgrind error file: valgrind-errors.2181501
==2181501== Syscall param write(buf) points to unaddressable byte(s)
==2181501== at 0x49E4077: write (write.c:26)
==2181501== by 0x1C79A3: do_write (io.c:189)
==2181501== by 0x1C80AB: do_plan (io.c:394)
==2181501== by 0x1C81BA: io_ready (io.c:423)
==2181501== by 0x1CA45B: io_loop (poll.c:453)
==2181501== by 0x118593: main (connectd.c:2053)
==2181501== Address 0x4afb158 is 40 bytes inside a block of size 140 free'd
==2181501== at 0x483F0C3: free (vg_replace_malloc.c:872)
==2181501== by 0x1D103C: del_tree (tal.c:421)
==2181501== by 0x1D130A: tal_free (tal.c:486)
==2181501== by 0x1364B8: clean_tmpctx (utils.c:172)
==2181501== by 0x1266DD: daemon_poll (daemon.c:87)
==2181501== by 0x1CA334: io_loop (poll.c:420)
==2181501== by 0x118593: main (connectd.c:2053)
==2181501== Block was alloc'd at
==2181501== at 0x483C855: malloc (vg_replace_malloc.c:381)
==2181501== by 0x1D0AC5: allocate (tal.c:250)
==2181501== by 0x1D1086: tal_alloc_ (tal.c:428)
==2181501== by 0x1D124F: tal_alloc_arr_ (tal.c:471)
==2181501== by 0x126204: cryptomsg_encrypt_msg (cryptomsg.c:161)
==2181501== by 0x11335F: peer_connected (connectd.c:318)
==2181501== by 0x118A8A: peer_init_received (peer_exchange_initmsg.c:135)
==2181501== by 0x1C751E: next_plan (io.c:59)
==2181501== by 0x1C8126: do_plan (io.c:407)
==2181501== by 0x1C8168: io_ready (io.c:417)
==2181501== by 0x1CA45B: io_loop (poll.c:453)
==2181501== by 0x118593: main (connectd.c:2053)
==2181501==
{
<insert_a_suppression_name_here>
Memcheck:Param
write(buf)
fun:write
fun:do_write
fun:do_plan
fun:io_ready
fun:io_loop
fun:main
}
--------------------------------------------------------------------------------
This adds the `cln_parse_rpcversion` helper that is already used in
various plugins to pyln-client, so it does not need to be copied
around anymore.
Changelog-None
The htlc_budget only exists iff the hint is a 'local' one; we were
failing to write to the htlc_budget field for non-local cases.
To avoid this, we make `local` into a struct that contains the fields
that pertain to local-only payments (in this case, `htlc_budget`).
Valgrind error file: valgrind-errors.1813487
==1813487== Conditional jump or move depends on uninitialised value(s)
==1813487== at 0x4A9C958: __vfprintf_internal (vfprintf-internal.c:1687)
==1813487== by 0x4AB0F99: __vsnprintf_internal (vsnprintf.c:114)
==1813487== by 0x1D2EF9: do_vfmt (str.c:66)
==1813487== by 0x1D3006: tal_vfmt_ (str.c:92)
==1813487== by 0x11A60A: paymod_log (libplugin-pay.c:167)
==1813487== by 0x11B749: payment_chanhints_apply_route (libplugin-pay.c:534)
==1813487== by 0x11EB36: payment_compute_onion_payloads (libplugin-pay.c:1707)
==1813487== by 0x12000F: payment_continue (libplugin-pay.c:2135)
==1813487== by 0x1245B9: adaptive_splitter_cb (libplugin-pay.c:3800)
==1813487== by 0x11FFB6: payment_continue (libplugin-pay.c:2123)
==1813487== by 0x1206BC: retry_step_cb (libplugin-pay.c:2301)
==1813487== by 0x11FFB6: payment_continue (libplugin-pay.c:2123)
==1813487==
{
<insert_a_suppression_name_here>
Memcheck:Cond
fun:__vfprintf_internal
fun:__vsnprintf_internal
fun:do_vfmt
fun:tal_vfmt_
fun:paymod_log
fun:payment_chanhints_apply_route
fun:payment_compute_onion_payloads
fun:payment_continue
fun:adaptive_splitter_cb
fun:payment_continue
fun:retry_step_cb
[sesh] 0:[tmux]*Z
Suggested-By: @nothingmuch
In various circumstances we can start a reconnection while one is
already going on. These can stockpile if the node really is unreachable.
Reported-by: @whitslack
Fixes: #5654
Changelog-Fixed: lightningd: we no longer stack multiple reconnection attempts if connections fail.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Allow 'any' as an option to zeroconf-selective.py plugin so we can use
it in line_graph where we don't know ids yet.
2. Use modern helpers like line_graph and remove debugging statement.
3. Don't use listchannels(): it's true that it shows local channels as well,
but that's a quirk I'd like to remove.
4. Make flake8 happy.
5. Rename to be more specific now it's a more narrow test.
I manually tested that the test still failed with the fixes removed, too,
so it is still the same test!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we only specify the node_id, we get the "first" channel.
Closes: #5803
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `pay` uses the correct local channel for payments when there are multiple available (not just always the first!)
We fixed most of them. Now hone in to the case which fails: `pay`
when it needs to use the direct zero-conf channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Modifications from issue #5803 to work here:
1. import json
2. Add xfail
3. Increase channel sizes by 10x so we can open them
4. Fix plugin path
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can actually catch l2 with HTLCs still closing and mine blocks,
then it force closes due to HTLC timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that our subtables were not being cleaned, despite being "ON
DELETE CASCADE". This is because foreign keys were not enabled, but
then I got foreign key errors: rowid cannot be a foreign key anyway!
So create a real "rowid" column. We want "ON DELETE CASCADE" for
nodes and channels (and other tables in future) where we update
partially.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When doing the updates on the plugin repo, I discovered that
this helper function got broken by the `listpeerchannels` upgrade.
Its rarely used in the main repo, just at the end of the pyln-testing
'pay' helper.
If unfixed, this bug may result in test flakes when using the `pay`
helper, because its not correctly waiting for all HTLCs to be resolved
before returning.
Changelog-None
The leak-detector can't find unconnected_htlcs_in on the stack and
incorrectly flags this as a leak. However, it is appropriately tal
allocated and freed.
Changelog-None
Valgrind correctly reports it as uninitialized for this log message, and
the only way this can happen is channel_hints_update() when we receive a
temporary_channel_failure. Put a dummy value here in this case.
```
Valgrind error file: valgrind-errors.23404
==23404== Conditional jump or move depends on uninitialised value(s)
==23404== at 0x49E4B56: __vfprintf_internal (vfprintf-internal.c:1516)
==23404== by 0x49F6519: __vsnprintf_internal (vsnprintf.c:114)
==23404== by 0x1EBCEB: do_vfmt (str.c:66)
==23404== by 0x1EBDF8: tal_vfmt_ (str.c:92)
==23404== by 0x11A336: paymod_log (libplugin-pay.c:167)
==23404== by 0x11B4B2: payment_chanhints_apply_route (libplugin-pay.c:534)
==23404== by 0x11E999: payment_compute_onion_payloads (libplugin-pay.c:1707)
==23404== by 0x11FF4C: payment_continue (libplugin-pay.c:2135)
==23404== by 0x1245C0: adaptive_splitter_cb (libplugin-pay.c:3800)
==23404== by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404== by 0x1205FE: retry_step_cb (libplugin-pay.c:2301)
==23404== by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404==
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There were cases where address it's empty, and this cases are not right if the
field is considered optional.
This makes it required and add the field also when `--offline` is set.
Changelog-Changed: JSON-RPC: `getinfo` `address` array is always present (though may be empty)
Avoids the following when postgres returns no query result:
==63458== Conditional jump or move depends on uninitialised value(s)
==63458== at 0x226A1F: db_postgres_step (db_postgres.c:156)
==63458== by 0x22535B: db_step (utils.c:155)
==63458== by 0x1E089A: db_data_version_get (exec.c:49)
==63458== by 0x194F6F: db_setup (db.c:1029)
==63458== by 0x199A2F: wallet_new (wallet.c:101)
==63458== by 0x154B70: main (lightningd.c:1035)
Changelog-None
We need to check if the key parameter is an empty array in
`listdatastore` as we do assume an array of at least length 1 in
`wallet.c:5306`.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Changelog-None
The sections on SQLite Litestream, sqlite3 .dump and VACUUM INTO commands
were to be removed six months after 0.10.3, due to issues observed in #4857.
We now recommend using --wallet=sqlite3://${main}:${backup} instead.
Also, put the "added" lines in the request schemas for new commands:
this doesn't do anything (yet?) but it keeps `make schema-added-check` happy.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This *would* be a 1-line change (add it to Makefile) except that we
previously assumed a "list" prefix on commands.
These use the default refreshing, but they could be done better using
the time-range parameters.
Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, we generate the schema part from the plugin itself.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
We now add tables to the strmap as we allocate them, since we don't
want to call "finish_td" when we're merely invoked for the
documentation, and don't need a database.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For now, we ignore every deprecated field, but put in the logic so
that future deprecations will work as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`"deprecated": true` is obsolete; we don't document them anyway.
Where it would have otherwise changed the GRPC wrappers, I actually put the
version number in.
We allow "listchannels" to have "satoshis" since we have some tests
that run in deprecated-api mode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This requires us to rename "index" fields, rename fields if we have a
sub-object, and create sub-tables if we have an array, and handle the
fact that some listX commands don't contain array X (listsendpays
contains "payments").
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than two arrays "columns" (for names) and "fieldtypes" (for
types), use a struct. This makes additions easier for successive
patches.
Also pull process_json_obj() out of the loop in list_done().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only handle top-level objects with an array of objects:
make sure it is one before we call the routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's actually two separate u16 fields, so actually treat it as
such!
Cleans up zombie handling code a bit too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a core concept in the spec which isn't directly exposed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
We didn't actually populate them properly, and the real annotations
are on inputs and outputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `listtransactions` `channel` and `type` field removed at top level.
We only ever use this table for output and input transactions: indeed, my node
doesn't have any annotation types 0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We removed the command for v22.11.
Also, we removed the `refund_for` offer parameter, so remove its description
from the manpage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Though BOLT 7 says a channel may be pruned when one side becomes inactive
and fails to refresh their channel_update, in practice, the
channel_announcement can be difficult to recover if deleted entirely.
Here the channel_announcement is tagged as zombie such that gossip_store
consumers may safely ignore it, but it may be retained should the channel
come back online in the future. Node_announcements and channel_updates may
also be retained in such a fashion until the channel is ready to be
resurrected.
Changelog-Fixed: Pruned channels are more reliably restored.
Fixes d9fed06b90:
```
common/bolt11.c:868:31: error: format specifies type 'size_t' (aka 'unsigned long') but the argument has type 'u64' (aka 'unsigned long long') [-Werror,-Wformat]
bech32_charset[type], field_len);
^~~~~~~~~
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our pay code handles this correctly, but decode was still using an old model
where there was a payinfo per hop, not per path.
Reported-by: @t-bast
See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-offers" BOLTVERSION=guilt/offers
```
In this case, only trivial mods.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-onion-message" BOLTVERSION=guilt/offers
```
Mainly textual, though I neatened the extra fields check for TLVs with
blinding, and implemented the "no other fields" requirement for
non-final onion message hops.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-route-blinding" BOLTVERSION=guilt/offers
```
Other than textual changes, this does:
1. Ensures we put total_amount_msat in onion final hop (reported by @t-bast).
2. Require that they put total_amount_msat in onion final hop.
3. Return `invalid_onion_blinding` exactly as defined by the spec (i.e. less
aggressive when we're the final hop) (also reported by @t-bast, but I already
knew).
See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (total_amount_sat required, Eclair compat)
You can use rs->nextcase, but we don't always keep that around, so
keep a flag in onion_payload.
We'll use this in the "do we need to return a blinded error code"
patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was reported by @valentinewallace: Dave would only use padding to
make all his own encrypted_recipient_data equal-length. We did it
across the entire path, which includes the hop added by Alice, which
Dave wouldn't know about.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rename is necessary to roundtrip, otherwise the rust name is used.
This also remove the rename if they are not necessary.
Note that:
```
#[serde(rename="foo", skip_serializing_if=="bar")]
pub field: bool,
```
is equivalent to:
```
#[serde(rename="foo")]
#[serde(skip_serializing_if=="bar")]
pub field: bool,
```
and for simplicity of construction the latter is used
And make pull_bits return a uniform error message, since that's what
callers want, rather than asserting success.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: pay: don't assert() on malformed BOLT11 strings.
Also, we don't need to pass the total length to the field parsers,
just the length for this field (confusingly, this was called
"data_length").
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will give the user an option to set a custom port when using
discovered IPs for node_announcents. Without this, only the selected
networks default port can used.
Changelog-Added: Adds --announce-addr-discovered-port config option to set custom port for IP discovery.
This switch was not doing anything useful anymore.
We deprecate it anyways to notify the user about the new switch.
Changelog-Deprecated: The old --disable-ip-discovery config switch
This adds the option to explicitly enable ip-discovery, which maybe
helpful for example when a user wants TOR announced along with
discovered IPs to improve connectivity and have TOR just as a fallback.
Changelog-Added: Adds config switch 'announce-addr-discovered': on/off/auto
This fixes the CI errors when doing `make check-source`
steps 'schema-added-check' and 'schema-removed-check'.
These errors prevented CI from performing these steps correctly:
```
fatal: ambiguous argument 'main': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
```
I changed it so that CI does a `git fetch origin` at first and do
the `git diff` against 'origin/master' (which then exist).
Also fixed a bug in the script that was missing $$master in the same line.
Also I added that the script shows the actual diff before failing,
so the user quickly sees whats wrong.
After connecting 100,000 peers with one channel each (not all at
once!), we see various places where we exhibit O(N^2) behaviour.
Fix these by keeping a map of id->peer instead of a simple
linked-list.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The channel->htlcs map was exhibiting unbounded growth, as elements were
never removed from it. This was causing lightning_channeld processes to
consume ever-increasing amounts of memory, and iterating over the map
was causing ever-increasing CPU utilization. There were FIXME comments
suggesting that the intention was to remove HTLCs from the map upon
their deaths. This commit implements that intention.
Changelog-Fixed: channeld no longer retains dead HTLCs in memory.
Rework the logic of the version check used in the
database migration, and make sure
that it is full functional to avoid confusion
at release time.
Changelog-Fixed: database: Correctly identity official release versions for database upgrade.
Reported-by: @urza
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
```
E Global errors:
E - Node /tmp/ltests-adkwu44c/test_logging_1/lightning-2/ has memory leaks: [
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E "lightningd/peer_control.c:2203 (load_channels_from_wallet)",
E "lightningd/lightningd.c:1105 (main)"
E ],
E "label": "lightningd/peer_control.c:2203:struct htlc_in_map",
E "parents": [
E "lightningd/lightningd.c:107:struct lightningd"
E ],
E "value": "0x55d920a345e8"
E }
E ]
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
lightningd/jsonrpc.c: In function ‘destroy_json_command’:
lightningd/jsonrpc.c:1180:63: error: the comparison will always evaluate as ‘false’ for the address of ‘canary’ will never be NULL [-Werror=address]
lightningd/jsonrpc.c:108:53: note: ‘canary’ declared here
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can require a status to be successful in PRs, but they are
referenced by name, and so we'd have to add each matrix job as
required. That's rather cumbersome, so have this artificial gather
step at the end which signals success on the entire run.
We often have a number of changes in flight, and we amend PRs before the previous run completes. This allows us to cancel that prior run, preserving our precious runners.
They are run in parallel with the integration tests, not on the
critical path, but can be a bit lengthy since they build a log of C
code, and run in valgrind.
The tester CI job uses absolute paths to ensure it is testing the
correct binaries. That clashes with the transfer between builder and
tester job using the `install` target because that switches things
around. This commit introduces a new target that just collects
artifacts in place, and tars them. Then we can use `tar` to unpack
them on the tester jobs again.
We want to compile with one set of dependencies, and run the tests
with another. This also helps us cut down on the times we compile CLN
itself, by sharing them among stages, and simplifies the logic of each
stage to have one specific goal.
The lnprototests are often blocking PRs, due to them failing, but we
can't restart them when valgrind runs are still ongoing, since they'd
also restart. Splitting allows us to rerun them selectively and waste
less time.
Ideally we'd just fix them, but I am by no means knowledgeable enough
to fix them now.
The close call can fail, since we already unilaterally closed since we mined blocks
too fast:
```
2023-01-14T01:00:10.2502199Z E pyln.client.lightning.RpcError: RPC call failed: method: close, payload: ['107x1x1', None, 'bcrt1qeyyk6sl5pr49ycpqyckvmttus5ttj25pd0zpvg'], error: {'code': -32602, 'message': "Short channel ID not active: '107x1x1'"}
...
2023-01-14T01:00:10.5288050Z lightningd-4 2023-01-14T00:59:59.650Z UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Peer permanent failure in CHANNELD_NORMAL: Fulfilled HTLC 0 SENT_REMOVE_COMMIT cltv 113 hit deadline
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to create some p2sh-segwit addresses just to mix things up. This
streamlines back to just bech32.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We removed a warning about the channel_update being malformed since
the warning could cause lnd to disconnect (seems they treat
channel-unrelated warnings as fatal?). This was caused by lnd not
enforcing the `htlc_maximum`, thus the parsing would fail. We can
re-add the warning once our assumption that `htlc_maximum` being set
is valid.
Changelog-Fixed: gossip: We no longer send warning that lnd would not understand if we get outdated gossip
Some are best copied into the schema, but some are already
out-of-date, so cleanest to remove them and rely on the generated (and
thus, checked!) fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With the next change (which, as a side-effect, speeds up listpeers),
we seem to hit a race in this test. The bookkeeper doesn't get to
process the final payment before the node is shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Don't parse the listpeers.channels output ourselves: with two extra fields
we can simply reuse json_to_listpeers_channels().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of returning a peers -> channels heirarchy, return (as callers
want!) a flat array of channels.
This is actually most of the transition work to make them work with
listpeerchannels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: new command `listpeerchannels` now contains information on direct channels with our peers.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We're soon going to call json_add_unsaved_channel and
json_add_uncommitted_channel from a new place, where we want the peer
state directly included.
Based on patch by @vincenzopalazzo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's no guarantee as to iteration order for accounts/channels, but
this test was relying on them.
Adding account attribution and comparing by account_ids fixes
Fixes: #5869
Reported-By: @rustyrussell
TODO: It would be great to similarly annotate new/deprecated commands and
their parameters.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: doc: we now annotate what versions JSON field additions and deprecations happenened.
This means we will document deprecations and additions, rather than just
pretending they've always been that way!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. When we receive a commando command from a remote using the `filter`
field, use it.
2. Add a `filter` parameter to `commando` to send it: this is usually
more efficient than using filtering locally.
Of course, older remote nodes will ignore the filter, but that's
harmless.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `commando` now supports `filter` as a parameter (for send and receive).
This was reported a while ago: now do it properly.
Fixes: #5637
Changelog-Fixed: Plugins: `commando` now responds to remote JSON calls with the correct JSON `id` field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We change the libplugin API so commando can provide its own ID base.
This id chaining enables much nicer diagnostics!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't do this yet, so we add deprecated to those test (until next
patch!).
Changelog-Deprecated: plugins: `commando` JSON commands without an `id` (see doc/lightningd-rpc.7.md for how to construct a good id field).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The top of the file indicates the following errors:
#define NO_ERROR 0
#define ERROR_FROM_LIGHTNINGD 1
#define ERROR_TALKING_TO_LIGHTNINGD 2
#define ERROR_USAGE 3
But we didn't use the right one for opt_parse failure, and didn't use the
correct constants everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids any confusion between primitive and string ids, and in
particular stops an issue with commando once it starts chaining ids,
that weird ids can be double-escaped and commando will not recognize
the response, leaving the client hanging. It's the client's fault for
using a weird id, but it's still rude (and triggered by our tests!).
It also makes substituting the id in passthrough simpler, FTW.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When called with `"id": 1` we replied with `"id": "1"`. lightningd doesn't
actually care, but it's weird.
Copy the entire token: this way we don't have to special case anything.
Also, remove the doubled test in json_add_jsonstr.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Issue #5363 documented an earlier bug in mkfunding. These tests validate that fix as well as checking that basic usage produces the proper response and exit status.
Changelog-None
When a channel is not yet confirmed onchain and only has an alias,
`getroute` will return the alias under the "channel" key. However, if
you supply that to `sendpay`, it will not be able to find the channel
and will fail since it calls `find_channel_for_htlc_add` and that only
looks for channels by their scid and doesn't consider their alias.
This patch makes `find_channel_for_htlc_add` also consider channel
aliases when looking for channels.
Changelog-Fixed: JSON-RPC: `sendpay` now can send to a short-channel-id alias for the first hop.
local_connected allocates a struct node_map off of the struct
listchannels_opts but fails to ever release the internal table, so those
table allocations hang around forever. Add node_map_clear as a
destructor for the struct node_map to ensure that the internal table is
freed when the enclosing struct is freed.
Changelog-Fixed: topology: Fixed memleak in `listchannels`
Signed-off-by: Matt Whitlock <c-lightning@mattwhitlock.name>
We were stressing the servers if node cannot be found. Only do lookup
on manual connect commands.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: lightningd: Only use DNS server address lookup on manual `connect` commands, not normal reconnection attempts.
I discovered this while reworking the CI workflow, and it seems like
the HTLC queries do not order, while the tests assume a specific
order. This matches sqlite3 which without an explicit ORDER clause
will use insertion order, while postgres does not keep things in
insertion order, thus breaking the assumption. Ordering by `id`
re-establishes that implicit assumption
Changelog-Changed: postgres: Ordering of HTLCs in `listhtlcs` are now ordered by time of creation
This often helps the msat purge and pyln msat replacement mischmasch issues.
I changed it in a way that the `AttributeError: 'int' object has no attribute 'millisatoshis'`
Error will still be raised whever a `Millisatoshi` is compared to
something else then a `Millisatoshi` or `int`.
Changelog-None
There are several cases where you want to access to the configuration,
and given the nature of the rust API, there is no way to access to
the `configuration` field at any point after the configuration logic.
Suggested-by: Sergi Delgado Segura <@sr-gi>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
```
error: failed to run custom build command for `cln-grpc v0.1.2 (/home/rusty/devel/cvs/lightning/cln-grpc)`
Caused by:
process didn't exit successfully: `/home/rusty/devel/cvs/lightning/target/debug/build/cln-grpc-933c4bf5006f522c/build-script-build` (exit status: 101)
--- stdout
cargo:rerun-if-changed=proto/node.proto
cargo:rerun-if-changed=proto
--- stderr
thread 'main' panicked at 'Could not find `protoc` installation and this build crate cannot proceed without
this knowledge. If `protoc` is installed and this crate had trouble finding
it, you can set the `PROTOC` environment variable with the specific path to your
installed `protoc` binary.If you're on debian, try `apt-get install protobuf-compiler` or download it from https://github.com/protocolbuffers/protobuf/releases
For more information: https://docs.rs/prost-build/#sourcing-protoc
', /home/rusty/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.11.4/src/lib.rs:1296:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
make: *** [plugins/Makefile:211: target/debug/examples/cln-plugin-startup] Error 101
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to install it only when Rust was configured, but for some
reason all builds now seem to be Rust builds. That's ok, so just
provide the necessary dependency.
We will likely re-add that dependency later as a feature (to
automatically provide an RPC instance if compiled with that feature)
but for now that dependency is unused.
Changelog-None Not user visible
This commit updates outdated dependencies and hangs all
bitcoin-related dependencies off of the `bitcoin` crate, using its
re-exports. This means that as long as the bitcoin crate matches, all
of its dependents will also match.
Changelog-None
Changelog-Removed: JSON-RPC: `fundpsbt`/`utxopsbt` `reserve` must be a number, not bool (deprecated v0.11.0)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Deprecated v0.11.0.
Changelog-Removed: JSON-RPC: `pay` for a bolt11 which uses a `description_hash`, without setting `description` (deprecated v0.11.0).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The only time underscores aren't special in Markdown is when they appear
in preformatted text. We have gotten away with not escaping underscores
where an asterisk-enclosed span or the paragraph ends before the next
underscore appears, but this is fragile and bad practice. Conversely,
there are many places where we have not escaped underscores but needed
to.
Escape all underscores that do not appear in preformatted blocks or
preformatted spans and are not themselves delineating emphasized spans.
The changes in this commit are exactly the result of executing the
following Bash code:
```bash
e=':x;' # begin loop
e+='s/^' # anchor match at beginning of line
e+='(' # begin capturing subexpression
e+='(' # begin list of alternatives
e+='[^`_\\]|' # any mundane character, or
e+='`([^`\\]|\\.)*`|' # backtick-enclosed span, or
e+='\b_|_\b|' # underscore at boundary, or
e+='\\.' # backslash-escaped character
e+=')*' # any number of the preceding alternatives
e+=')' # end capturing subexpression
e+='\B_\B/\1\\_/;' # escape non-formatting underscore
e+='tx' # repeat loop if we escaped an underscore
escape_underscores=(
sed
# use extended regular expressions
-E
# skip over indented blocks (following an empty line)
-e '/^$/{:i;n;/^( {4,}|\t)/bi}'
# skip over preformatted blocks
-e '/^\s*```/,/^\s*```/{p;d}'
# skip over generated sections
-e '/GENERATE-FROM-SCHEMA-START/,/GENERATE-FROM-SCHEMA-END/{p;d}'
# escape underscores
-e "${e}"
)
"${escape_underscores[@]}" -i doc/*.[0-9].md
```
Changelog-None
This broke BTCPayServer, so revert. I originally (accidentally!)
implemented this such that it broadcast both DNS and IP entries, but
Michael reported earlier that they still don't propagage well, so
simply suppress them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5795
Changelog-Changeed: Config: `announce-addr-dns` needs to be set to *true* to put DNS names into node announcements, otherwise they are suppressed.
Changelog-Deprecated: Config: `announce-addr-dns` (currently defaults to `false`). This will default to `true` once enough of the network has upgraded to understand DNS entries.
The hostname part of a DNS FQDN can allow for additional characters
other than specified in `man 7 hostname`.
This extends is_dnsaddr and the test issue #5657.
Also fixes a typo in a comment.
Changelog-Fixed: wireaddr: #5657 allow '_' underscore in hostname part of DNS FQDN
This was reported, but the channel was closed. So however we ended
up with a duplicate, we're no *worse* off than we were before migration?
Fixes: #5760
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's obviously the incorrect type, while our CI didn't catch it, Nicholas did:
```
plugins/fetchinvoice.c:1362:30: error: conversion from 'long long unsigned int' to 'size_t' {aka 'unsigned int'} changes value from '18446744073709551615' to '4294967295' [-Werror=overflow]
1362 | || tlv_span(wire, 1001, UINT64_MAX, NULL) != 0) {
```
Reported-by: @NicholasDorier
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Build: arm32 compiler error in fetchinvoice, due to bad types on 32-bit platforms.
Fixes: #5776
This reverts commit 43b037ab0b.
Nicholas Dorier says BTC Payserver still wants this for another year
or so.
Changelog-Added: JSON-RPC: reverts requirement for "jsonrpc" "2.0" inside requests (still deprecated though, just for a while longer!)
The switch to logging enabled verbose output regardless of the option
flag. Here the functionality is restored.
Changelog-Fixed: reckless verbosity properly applied.
The read_json() call expects len_read to be the amount of *new*
data read. If we call this back without resetting, it will parse
this much random junk in the buffer.
Fixes: #5766
Changelog-Fixed: Plugin: `autoclean` could misperform or get killed due to lightningd's invalid handling of JSON batching.
We only incremented this if it was the wrong state.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `autoclean-once` response `uncleaned` count is now correct.
> 25.09 FREEZE July 28TH: Non-bugfix PRs not ready by this date will wait for 25.12.
>
> RC1 is scheduled on _August 11th_
>
> The final release is scheduled for September 1st.
## Checklist
Before submitting the PR, ensure the following tasks are completed. If an item is not applicable to your PR, please mark it as checked:
- [ ] The changelog has been updated in the relevant commit(s) according to the [guidelines](https://docs.corelightning.org/docs/coding-style-guidelines#changelog-entries-in-commit-messages).
- [ ] Tests have been added or modified to reflect the changes.
- [ ] Documentation has been reviewed and updated as needed.
- [ ] Related issues have been listed and linked, including any that this PR closes.
This implementation has been in production use on the Bitcoin mainnet since early 2018, with the launch of the [Blockstream Store][blockstream-store-blog].
We recommend getting started by experimenting on `testnet` (or `regtest`), but the implementation is considered stable and can be safely used on mainnet.
We recommend getting started by experimenting on `testnet` (`testnet4` or `regtest`), but the implementation is considered stable and can be safely used on mainnet.
## Reach Out to Us
Any help testing the implementation, reporting bugs, or helping with outstanding issues is very welcome.
Don't hesitate to reach out to us on IRC at [#lightning-dev @ libera.chat][irc1], [#c-lightning @ libera.chat][irc2], or on the implementation-specific mailing list [c-lightning@lists.ozlabs.org][ml1], or on the Lightning Network-wide mailing list [lightning-dev@lists.linuxfoundation.org][ml2], or on Discord [core-lightning][discord], or on Telegram [Core Lightning][telegram].
Don't hesitate to reach out to us on [Build-on-L2][bol2], or on the implementation-specific [mailing list][ml1], or on [CLN Discord][discord], or on [CLN Telegram][telegram], or on IRC at [dev][irc1]/[gen][irc2] channel.
## Getting Started
Core Lightning only works on Linux and macOS, and requires a locally (or remotely) running `bitcoind` (version 0.16 or above) that is fully caught up with the network you're running on, and relays transactions (ie with `blocksonly=0`).
Core Lightning only works on Linux and macOS, and requires a locally (or remotely) running `bitcoind` (version 25.0 or above) that is fully caught up with the network you're running on, and relays transactions (ie with `blocksonly=0`).
Pruning (`prune=n` option in `bitcoin.conf`) is partially supported, see [here](#pruning) for more details.
### Installation
There are 4 supported installation options:
There are 3 supported installation options:
- Installation from the [Ubuntu PPA][ppa].
- Installation of a pre-compiled binary from the [release page][releases] on GitHub.
- Using one of the [provided docker images][dockerhub] on the Docker Hub.
- Compiling the source code yourself as described in the [installation documentation](doc/INSTALL.md).
For the impatient here's the gist of it for Ubuntu:
- Compiling the source code yourself as described in the [installation documentation](doc/getting-started/getting-started/installation.md).
### Starting `lightningd`
@ -66,13 +60,6 @@ of the `startup_regtest.sh` file for details on how to use it.
. contrib/startup_regtest.sh
```
Note that your local nodeset will be much faster/more responsive if
you've configured your node to expose the developer options, e.g.
```bash
./configure --enable-developer
```
#### Mainnet Option
To test with real bitcoin, you will need to have a local `bitcoind` node running:
@ -91,7 +78,7 @@ You can start `lightningd` with the following command:
lightningd --network=bitcoin --log-level=debug
```
This creates a `.lightning/` subdirectory in your home directory: see `man -l doc/lightningd.8` (or https://lightning.readthedocs.io/) for more runtime options.
This creates a `.lightning/` subdirectory in your home directory: see `man -l doc/lightningd.8` (or https://docs.corelightning.org/docs) for more runtime options.
### Using The JSON-RPC Interface
@ -117,17 +104,12 @@ Once you've started for the first time, there's a script called
the lightning network.
There are also numerous plugins available for Core Lightning which add
capabilities: in particular there's a collection at:
https://github.com/lightningd/plugins
Including [helpme][helpme-github] which guides you through setting up
your first channels and customizing your node.
capabilities: in particular there's a collection at: https://github.com/lightningd/plugins
For a less reckless experience, you can encrypt the HD wallet seed:
see [HD wallet encryption](#hd-wallet-encryption).
You can also chat to other users at [#c-lightning @ libera.chat][irc2];
You can also chat to other users at Discord [core-lightning][discord];
we are always happy to help you get started!
@ -143,11 +125,11 @@ lightning-cli newaddr
`lightningd` will register the funds once the transaction is confirmed.
You may need to generate a p2sh-segwit address if the faucet does not support bech32:
Alternatively you can generate a taproot address should your source of funds support it:
```bash
# Return a p2sh-segwit address
lightning-cli newaddr p2sh-segwit
# Return a taproot address
lightning-cli newaddr p2tr
```
Confirm `lightningd` got funds by:
@ -202,6 +184,8 @@ Command line options will always override the values in the configuration file.
To use a configuration file, create a file named `config` within your top-level lightning directory or network subdirectory
(eg. `~/.lightning/config` or `~/.lightning/bitcoin/config`). See `man -l doc/lightningd-config.5`.
A sample configuration file is available at `contrib/config-example`.
## Further information
### Pruning
@ -221,25 +205,28 @@ If you encrypt your `hsm_secret`, you will have to pass the `--encrypted-hsm` st
### Developers
Developers wishing to contribute should start with the developer guide [here](doc/HACKING.md).
You should also configure with `--enable-developer` to get additional checks and options.
Developers wishing to contribute should start with the developer guide [here](doc/contribute-to-core-lightning/coding-style-guidelines.md).
We have a 3 month release cycle, and the last two versions are supported.
## Reporting a Vulnerability
To report security vulnerabilities, please send an email to one of the following addresses:
- `rusty@rustcorp.com.au`
- `security@blockstream.com`
Note: These email addresses are exclusively for vulnerability reporting.
For all other inquiries/communication, please refer to the [Reach Out to Us](https://github.com/ElementsProject/lightning?tab=readme-ov-file#reach-out-to-us) section in our README.
## Signatures For Releases
The following keys may be used to communicate sensitive information to
developers, and to validate signatures on releases:
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
