moved akka conf to eclair-node application.conf

use a local queue for outgoing rpc requests. this should be a better solution than
inceasing the number of concurrent requests.
see https://doc.akka.io/docs/akka-http/current/scala/http/client-side/host-level.html#using-the-host-level-api-with-a-queue
for more information.

.travis.yml

@@ -2,7 +2,7 @@ sudo: required
 dist: trusty
 language: scala
 scala:
- - 2.11.8
+ - 2.11.11
 env:
  - export LD_LIBRARY_PATH=/usr/local/lib
 script:

BUILD.md

@@ -10,7 +10,12 @@ To build the project, simply run:
 ```shell
 $ mvn package
 ```
-or
+To skip the tests, run:
 ```shell
 $ mvn package -DskipTests
 ```
+To generate the windows installer along with the build, run the following command:
+```shell
+$ mvn package -DskipTests -Pinstaller
+```
+The generated installer will be located in `eclair-node-gui/target/jfx/installer`

README.md

@@ -27,7 +27,7 @@ Please see the latest [release note](https://github.com/ACINQ/eclair/releases) f
 
 ## Installation
 
-:warning: **Those are valid for the most up-to-date, unreleased, version of eclair. Here are the [instructions for Eclair 0.2-alpha3](https://github.com/ACINQ/eclair/blob/v0.2-alpha3/README.md#installation)**.
+:warning: **Those are valid for the most up-to-date, unreleased, version of eclair. Here are the [instructions for Eclair 0.2-alpha5](https://github.com/ACINQ/eclair/blob/v0.2-alpha5/README.md#installation)**.
 
 ### Configuring Bitcoin Core
 
@@ -46,6 +46,8 @@ zmqpubrawtx=tcp://127.0.0.1:29000
 
 ### Installing Eclair
 
+The released binaries can be downloaded [here](https://github.com/ACINQ/eclair/releases).
+
 #### Windows
 
 Just use the windows installer, it should create a shortcut on your desktop.
@@ -54,52 +56,79 @@ Just use the windows installer, it should create a shortcut on your desktop.
 
 You need to first install java, more precisely a [JRE 1.8](http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html).
 
- :warning: If you are using the OpenJDK JRE, you will need to build OpenJFX yourself, or run the application in `--headless` mode.
+ :warning: If you are using the OpenJDK JRE, you will need to build OpenJFX yourself, or run the application in headless mode (see below).
 
-Then download the latest fat jar and run:
+Then download the latest fat jar and depending on whether or not you want a GUI run the following command:
+* with GUI:
 ```shell
-java -jar eclair-node_xxxxxx-fat.jar
+java -jar eclair-node-gui-<version>-<commit_id>.jar
+```
+* without GUI:
+```shell
+java -jar eclair-node-<version>-<commit_id>.jar
 ```
 
 ### Configuring Eclair
 
-#### Command-line parameters
-
-option         | description                     | default value
----------------|---------------------------------|--------------
---datadir      | Path to the data directory      | ~/.eclair
---headless     | Run Eclair without the GUI      |
---help, -h     | Display usage text              |
-
-
-:warning: Using separate `datadir` is mandatory if you want to run **several instances of eclair** on the same machine. You will also have to change ports in the configuration (see below).
-
 #### Configuration file
 
-To change your node configuration, edit the file `eclair.conf` in `datadir`.
+Eclair reads its configuration file, and write its logs, to a `datadir` directory, located in `~/.eclair` by default.
 
-option                       | description               | default value
+To change your node's configuration, create a file named `eclair.conf` in `datadir`. Here's an example configuration file:
+
+```
+eclair.server.port=9735
+eclair.node-alias=eclair
+eclair.node-color=49daaa
+```
+
+Here are some of the most common options:
+
+name                         | description               | default value
 -----------------------------|---------------------------|--------------
- eclair.server.port          | TCP port                  | 9735
- eclair.api.port             | HTTP port                 | 8080
+ eclair.server.port          | Lightning TCP port        | 9735
+ eclair.api.port             | API HTTP port             | 8080
  eclair.bitcoind.rpcuser     | Bitcoin Core RPC user     | foo
  eclair.bitcoind.rpcpassword | Bitcoin Core RPC password | bar
  eclair.bitcoind.zmq         | Bitcoin Core ZMQ address  | tcp://127.0.0.1:29000
 
-&rarr; see [`application.conf`](eclair-node/src/main/resources/application.conf) for full reference.
+Quotes are not required unless the value contains special characters. Full syntax guide [here](https://github.com/lightbend/config/blob/master/HOCON.md).
+
+&rarr; see [`reference.conf`](eclair-core/src/main/resources/reference.conf) for full reference. There are many more options!
+
+#### Java Environment Variables
+
+Some advanced parameters can be changed with java environment variables. Most users won't need this and can skip this section.
+
+:warning: Using separate `datadir` is mandatory if you want to run **several instances of eclair** on the same machine. You will also have to change ports in eclair.conf (see above).
+
+name                  | description                                | default value
+----------------------|--------------------------------------------|--------------
+eclair.datadir        | Path to the data directory                 | ~/.eclair
+eclair.headless       | Run eclair without a GUI                   | 
+eclair.printToConsole | Log to stdout (in addition to eclair.log)  |
+
+For example, to specify a different data directory you would run the following command:
+```shell
+java -Declair.datadir=/tmp/node1 -jar eclair-node-gui-<version>-<commit_id>.jar
+```
 
 ## JSON-RPC API
 
  method       |  params                                       | description
  -------------|-----------------------------------------------|-----------------------------------------------------------
-  connect     | host, port, nodeId                            | connect to another lightning node through a secure connection
-  open        | host, port, nodeId, fundingSatoshis, pushMsat | opens a channel with another lightning node
+  getinfo     |                                               | return basic node information (id, chain hash, current block height) 
+  connect     | nodeId, host, port                            | connect to another lightning node through a secure connection
+  open        | nodeId, host, port, fundingSatoshis, pushMsat | opens a channel with another lightning node
   peers       |                                               | list existing local peers
   channels    |                                               | list existing local channels
   channel     | channelId                                     | retrieve detailed information about a given channel
-  network     |                                               | list all nodes that have been announced
-  receive     | amountMsat                                    | generate a payment request for a given amount
+  allnodes    |                                               | list all known nodes
+  allchannels |                                               | list all known channels
+  receive     | amountMsat, description                       | generate a payment request for a given amount
   send        | amountMsat, paymentHash, nodeId               | send a payment to a lightning node
+  send        | paymentRequest                                | send a payment to a lightning node using a BOLT11 payment request
+  send        | paymentRequest, amountMsat                    | send a payment to a lightning node using a BOLT11 payment request and a custom amount
   close       | channelId                                     | close a channel
   close       | channelId, scriptPubKey (optional)            | close a channel and send the funds to the given scriptPubKey
   help        |                                               | display available methods

TESTING.md

@@ -1,120 +0,0 @@
-# Testing eclair and lightningd
-
-## Configure bitcoind to run in regtest mode
-Important: you need a segwit version of bitcoin core for this test (see https://github.com/sipa/bitcoin/tree/segwit-master).
-Make sure that bitcoin-cli is on the path and edit ~/.bitcoin/bitcoin.conf and add:
-```shell
-server=1
-regtest=1
-rpcuser=***
-rpcpassword=***
-```
-
-To check that segwit is enabled run:
-```shell
-bitcoin-cli getblockchaininfo
-```
-and check bip9_softforks:
-
-```
-...
-"bip9_softforks": {
-    "csv": {
-      "status": "active",
-      "startTime": 0,
-      "timeout": 999999999999
-    },
-    "witness": {
-      "status": "active",
-      "startTime": 0,
-      "timeout": 999999999999
-    }
-  }
-```
-
-## Start bitcoind
-Mine enough blocks to activate segwit blocks:
-```shell
-bitcoin-cli generate 500
-```
-##
-Start lightningd (here we’ll use port 46000)
-```shell
-lightningd --port 46000
-```
-##
-Start eclair:
-```shell
-mvn exec:java -Dexec.mainClass=fr.acinq.eclair.Boot
-```
-## Tell eclair to connect to lightningd
-
-```shell
-curl -X POST -H "Content-Type: application/json" -d '{
-     "method": "connect",
-     "params" : [ "localhost", 46000, 3000000 ]
- }' http://localhost:8080
-```
-Since eclair is funder, it will create and publish the funding tx
-
-Mine a few blocks to confirm the funding tx:
-```shell
-bitcoin-cli generate 10
-```
-eclair and lightningd are now both in NORMAL state.
-You can check this by running:
-```shell
-lightning-cli getpeers
-```
-or
-```shell
-curl -X POST -H "Content-Type: application/json" -d '{
-     "method": "list",
-     "params" : [ ]
- }' http://localhost:8080
- ```
-
-
-## Tell eclair to send a htlc
-We’ll use the following values for R and H:
-```
-R = 0102030405060708010203040506070801020304050607080102030405060708
-H = 8cf3e5f40cf025a984d8e00b307bbab2b520c91b2bde6fa86958f8f4e7d8a609
-```
-
-You’ll need a unix timestamp that is not too far into the future. Now + 100000 is fine:
-```shell
-curl -X POST -H "Content-Type: application/json" -d "{
-    \"method\": \"addhtlc\",
-    \"params\" : [ 70000000, \"8cf3e5f40cf025a984d8e00b307bbab2b520c91b2bde6fa86958f8f4e7d8a609\", $((`date +%s` + 100000)), \"021acf75c92318d3723098294d2a6a4b08d9abba2ebb5f2df2b4a8e9153e96a5f4\"  ]
-}" http://localhost:8080
-```
-
-## Tell eclair to commit its changes
-```shell
-curl -X POST -H "Content-Type: application/json" -d "{
-    \"method\": \"sign\",
-    \"params\" : [ \"d3f056a084e266ad06ea1ca28a1e080ca07c6b61fac7ce116e48a5c31d688eee\" ]
-}" http://localhost:8080
-```
-## Tell lightningd to fulfill the HTLC:
-```shell
-./lightning-cli fulfillhtlc 03befb4f8ad1d87d4c41acbb316791fe157f305caf2123c848f448975aaf85c1bb 0102030405060708010203040506070801020304050607080102030405060708
-```
-Check balances on both eclair and lightningd
-
-## Close the channel
-```shell
-./lightning-cli close 03befb4f8ad1d87d4c41acbb316791fe157f305caf2123c848f448975aaf85c1bb
-```
-Mine a few blocks to bury the closing tx
-```shell
-bitcoin-cli generate 10
-```
-The channel is now in CLOSED state
-
-
-
-
-
-

eclair-core/eclair-cli

@@ -2,25 +2,41 @@
 
 [ -z "$1" ] && (
 echo "usage: "
-echo "eclair-cli list"
-echo "eclair-cli sign channel-id"
-echo "eclair-cli fulfill channel-id htlc-id htlc-preimage"
+echo "  eclair-cli help"
 ) && exit 1
 
-case $1 in 
-"list")	
+URL="http://localhost:8080"
+CURL_OPTS="-sS -X POST -H \"Content-Type: application/json\""
 
-	curl -X POST -d '{ "method": "list", "params" : [] }' "http://localhost:8080"
-	;;
-"sign")
-	curl -X POST -d '{ "method": "sign", "params" : ["'${2?"missing channel id"}'"] }' "http://localhost:8080"
-	;;	
-"add")
-	curl -X POST -d '{ "method": "addhtlc", "params" : ['${2?"missing amount"}', "'${3?"missing payment hash"}'", '${4?"missing expiry"}', "'${5?"missing node id"}'"] }' "http://localhost:8080"
-	;;	
-"fulfill")
-	curl -X POST -d '{ "method": "fulfillhtlc", "params" : ["'${2?"missing channel id"}'", '${3?"missing htlc id"}', "'${4?"missing htlc preimage"}'"] }' "http://localhost:8080"
-	;;	
+case $1 in
+"help")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"help\", \"params\" : [] }' $URL" | jq -r ".result[]"
+        ;;
+"getinfo")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"getinfo\", \"params\" : [] }' $URL" | jq ".result"
+        ;;
+"channels")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"channels\", \"params\" : [] }' $URL" | jq ".result[]"
+        ;;
+"channel")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"channel\", \"params\" : [\"${2?"missing channel id"}\"] }' $URL" | jq ".result | { nodeid, channelId, state, balanceMsat: .data.commitments.localCommit.spec.toLocalMsat, capacitySat: .data.commitments.commitInput.txOut.amount.amount }"
+        ;;
+"open")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"open\", \"params\" : [\"${2?"missing node id"}\", \"${3?"missing ip"}\", ${4?"missing port"}, ${5?"missing amount (sat)"}, ${6?"missing push amount (msat)"}] }' $URL" | jq -r "if .error == null then .result else .error.message end"
+        ;;
+"close")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"close\", \"params\" : [\"${2?"missing channel id"}\"] }' $URL"
+        ;;
+"receive")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"receive\", \"params\" : [${2?"missing amount"}, \"something\"] }' $URL" | jq -r "if .error == null then .result else .error.message end"
+        ;;
+"send")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"send\", \"params\" : [\"${2?"missing request"}\"] }' $URL" | jq -r "if .error == null then .result else .error.message end"
+        ;;
+"network")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"network\", \"params\" : [] }' $URL" | jq ".result"
+        ;;
+"peers")
+        eval curl "$CURL_OPTS -d '{ \"method\": \"peers\", \"params\" : [] }' $URL" | jq ".result"
+        ;;
 esac
-echo
-

eclair-core/pom.xml

@@ -80,6 +80,20 @@
                 <bitcoind.sha1>3ab7e537bd00bf35e6a78fca108d0d886f8289c1</bitcoind.sha1>
             </properties>
         </profile>
+        <profile>
+            <id>Mac</id>
+            <activation>
+                <os>
+                    <family>mac</family>
+                </os>
+            </activation>
+            <properties>
+                <bitcoind.url>https://bitcoin.org/bin/bitcoin-core-0.14.0/bitcoin-0.14.0-osx64.tar.gz
+                </bitcoind.url>
+                <bitcoind.md5>1521e1d0901169004b9c1c9b552868b7</bitcoind.md5>
+                <bitcoind.sha1>7216298f77162618f322fdf499f1f1b67a0048b7</bitcoind.sha1>
+            </properties>
+        </profile>
         <profile>
             <id>Windows</id>
             <activation>
@@ -140,6 +154,11 @@
             <artifactId>jeromq</artifactId>
             <version>0.4.0</version>
         </dependency>
+        <dependency>
+            <groupId>fr.acinq</groupId>
+            <artifactId>bitcoinj-core</artifactId>
+            <version>${bitcoinj.version}</version>
+        </dependency>
         <!-- SERIALIZATION -->
         <dependency>
             <groupId>org.scodec</groupId>
@@ -153,6 +172,11 @@
             <version>1.3.1</version>
         </dependency>
         <!-- OTHER -->
+        <dependency>
+            <groupId>org.xerial</groupId>
+            <artifactId>sqlite-jdbc</artifactId>
+            <version>3.20.0</version>
+        </dependency>
         <dependency>
             <groupId>org.jgrapht</groupId>
             <artifactId>jgrapht-core</artifactId>
@@ -182,12 +206,6 @@
             <version>${akka.version}</version>
             <scope>test</scope>
         </dependency>
-        <dependency>
-            <groupId>com.google.guava</groupId>
-            <artifactId>guava</artifactId>
-            <version>18.0</version>
-            <scope>test</scope>
-        </dependency>
         <dependency>
             <groupId>ch.qos.logback</groupId>
             <artifactId>logback-classic</artifactId>

eclair-core/src/main/resources/electrum/servers_regtest.json

@@ -0,0 +1,10 @@
+{
+  "127.0.0.1": {
+    "t": "51001",
+    "s": "51002"
+  },
+  "10.0.2.2": {
+    "t": "51001",
+    "s": "51002"
+  }
+}

eclair-core/src/main/resources/electrum/servers_testnet.json

@@ -0,0 +1,14 @@
+{
+  "testnetnode.arihanc.com": {
+    "t": "51001",
+    "s": "51002"
+  },
+  "testnet.hsmiths.com": {
+    "t": "53011",
+    "s": "53012"
+  },
+  "electrum.akinbo.org": {
+    "t": "51001",
+    "s": "51002"
+  }
+}

eclair-core/src/main/resources/reference.conf

@@ -1,13 +1,20 @@
 eclair {
+
+  chain = "test" // "regtest" for regtest, "test" for testnet. Livenet is not supported.
+
   server {
-    public-ip = "127.0.0.1"
+    public-ips = [] // external ips, will be announced on the network
     binding-ip = "0.0.0.0"
     port = 9735
   }
+
   api {
     binding-ip = "127.0.0.1"
     port = 8080
   }
+
+  watcher-type = "bitcoind" // other *experimental* values include "bitcoinj" or "electrum"
+
   bitcoind {
     host = "localhost"
     rpcport = 18332
@@ -16,13 +23,33 @@ eclair {
     zmq = "tcp://127.0.0.1:29000"
   }
 
+  bitcoinj {
+    static-peers = [
+      #{ // currently used in integration tests to override default port
+      #  host = "localhost"
+      #  port = 28333
+      #}
+    ]
+  }
+
+  default-feerates { // those are in satoshis per byte
+    delay-blocks {
+      1 = 210
+      2 = 180
+      6 = 150
+      12 = 110
+      36 = 50
+      72 = 20
+    }
+  }
+
   node-alias = "eclair"
   node-color = "49daaa"
   global-features = ""
-  local-features = "03" // channels_public and initial_routing_sync
-
+  local-features = "08" // initial_routing_sync
+  channel-flags = 1 // announce channels
   dust-limit-satoshis = 542
-  default-feerate-perkw = 10000 # corresponds to bitcoind's default value of feerate-perkB=20000 for a standard commit tx
+  default-feerate-per-kb = 20000  // default bitcoin core value
 
   max-htlc-value-in-flight-msat = 100000000000 // 1 BTC ~= unlimited
   htlc-minimum-msat = 1000000
@@ -46,6 +73,7 @@ eclair {
   // than this ratio.
   update-fee_min-diff-ratio = 0.1
 
+  channel-exclude-duration = 60 seconds // when a temporary channel failure is returned, we exclude the channel from our payment routes for this duration
   router-broadcast-interval = 10 seconds // this should be 60 seconds on mainnet
   router-validate-interval = 2 seconds // this should be high enough to have a decent level of parallelism
 
@@ -53,21 +81,4 @@ eclair {
   auto-reconnect = true
 
   payment-handler = "local"
-}
-akka {
-  loggers = ["akka.event.slf4j.Slf4jLogger"]
-  loglevel = "DEBUG"
-
-  actor {
-    debug {
-      # enable DEBUG logging of all LoggingFSMs for events, transitions and timers
-      fsm = on
-    }
-  }
-
-  http {
-    host-connection-pool {
-      max-open-requests = 64
-    }
-  }
 }

eclair-core/src/main/scala/fr/acinq/eclair/DBCompatChecker.scala

@@ -0,0 +1,21 @@
+package fr.acinq.eclair
+
+import grizzled.slf4j.Logging
+
+import scala.util.{Failure, Success, Try}
+
+object DBCompatChecker extends Logging {
+
+  /**
+    * Tests if the DB files are compatible with the current version of eclair; throws an exception if incompatible.
+    *
+    * @param nodeParams
+    */
+  def checkDBCompatibility(nodeParams: NodeParams): Unit =
+    Try(nodeParams.networkDb.listChannels() ++ nodeParams.networkDb.listNodes() ++ nodeParams.peersDb.listPeers() ++ nodeParams.channelsDb.listChannels()) match {
+      case Success(_) => {}
+      case Failure(_) => throw IncompatibleDBException
+    }
+}
+
+case object IncompatibleDBException extends RuntimeException("DB files are not compatible with this version of eclair.")

eclair-core/src/main/scala/fr/acinq/eclair/FSMDiagnosticActorLogging.scala

@@ -0,0 +1,26 @@
+package fr.acinq.eclair
+
+import akka.actor.{Actor, FSM}
+import akka.event.{DiagnosticLoggingAdapter, LoggingAdapter}
+
+/**
+  * A version of akka.actor.DiagnosticActorLogging compatible with an FSM
+  * See https://groups.google.com/forum/#!topic/akka-user/0CxR8CImr4Q
+  */
+trait FSMDiagnosticActorLogging[S, D] extends FSM[S, D] {
+
+  import akka.event.Logging._
+
+  val diagLog: DiagnosticLoggingAdapter = akka.event.Logging(this)
+
+  def mdc(currentMessage: Any): MDC = emptyMDC
+
+  override def log: LoggingAdapter = diagLog
+
+  override def aroundReceive(receive: Actor.Receive, msg: Any): Unit = try {
+    diagLog.mdc(mdc(msg))
+    super.aroundReceive(receive, msg)
+  } finally {
+    diagLog.clearMDC()
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/Features.scala

@@ -2,7 +2,6 @@ package fr.acinq.eclair
 
 
 import java.util.BitSet
-import java.util.function.IntPredicate
 
 import fr.acinq.bitcoin.BinaryData
 
@@ -11,23 +10,39 @@ import fr.acinq.bitcoin.BinaryData
   * Created by PM on 13/02/2017.
   */
 object Features {
+  // reserved but not used as per lightningnetwork/lightning-rfc/pull/178
+  val INITIAL_ROUTING_SYNC_BIT_MANDATORY = 2
+  val INITIAL_ROUTING_SYNC_BIT_OPTIONAL = 3
 
-  val CHANNELS_PUBLIC_BIT = 0
-  val INITIAL_ROUTING_SYNC_BIT = 2
+  /**
+    *
+    * @param features feature bits
+    * @return true if an initial dump of the routing table is requested
+    */
+  def initialRoutingSync(features: BitSet): Boolean = features.get(INITIAL_ROUTING_SYNC_BIT_OPTIONAL)
 
-  // NB: BitSet operates on little endian, hence the reverse
+  /**
+    *
+    * @param features feature bits
+    * @return true if an initial dump of the routing table is requested
+    */
+  def initialRoutingSync(features: BinaryData): Boolean = initialRoutingSync(BitSet.valueOf(features.reverse.toArray))
 
-  def isSet(features: BinaryData, bitIndex: Int): Boolean = BitSet.valueOf(features.reverse.toArray).get(bitIndex)
+  /**
+    * Check that the features that we understand are correctly specified, and that there are no mandatory features that
+    * we don't understand (even bits)
+    */
+  def areSupported(bitset: BitSet): Boolean = {
+    // for now there is no mandatory feature bit, so we don't support features with any even bit set
+    for (i <- 0 until bitset.length() by 2) {
+      if (bitset.get(i)) return false
+    }
+    return true
+  }
 
   /**
     * A feature set is supported if all even bits are supported.
     * We just ignore unknown odd bits.
     */
-  def areSupported(features: BinaryData): Boolean = {
-    val bitset = BitSet.valueOf(features.reverse.toArray)
-    bitset.stream().noneMatch(new IntPredicate {
-      override def test(value: Int) = value % 2 == 0 && value > INITIAL_ROUTING_SYNC_BIT
-    })
-  }
-
+  def areSupported(features: BinaryData): Boolean = areSupported(BitSet.valueOf(features.reverse.toArray))
 }

eclair-core/src/main/scala/fr/acinq/eclair/Globals.scala

@@ -1,7 +1,8 @@
 package fr.acinq.eclair
 
-import java.util.concurrent.atomic.AtomicLong
+import java.util.concurrent.atomic.{AtomicLong, AtomicReference}
 
+import fr.acinq.eclair.blockchain.fee.{FeeratesPerByte, FeeratesPerKw}
 
 /**
   * Created by PM on 25/01/2016.
@@ -11,16 +12,21 @@ object Globals {
   /**
     * This counter holds the current blockchain height.
     * It is mainly used to calculate htlc expiries.
-    * The value is updated by the [[fr.acinq.eclair.blockchain.PeerWatcher]] and read by all actors, hence it needs to be thread-safe.
+    * The value is read by all actors, hence it needs to be thread-safe.
     */
   val blockCount = new AtomicLong(0)
 
   /**
-    * This counter holds the current feeratePerKw.
-    * It is used to maintain an up-to-date fee in commitment tx so that they get confirmed fast enough.
-    * The value is updated by the [[fr.acinq.eclair.blockchain.PeerWatcher]] and read by all actors, hence it needs to be thread-safe.
+    * This holds the current feerates, in satoshi-per-bytes.
+    * The value is read by all actors, hence it needs to be thread-safe.
     */
-  val feeratePerKw = new AtomicLong(0)
+  val feeratesPerByte = new AtomicReference[FeeratesPerByte](null)
+
+  /**
+    * This holds the current feerates, in satoshi-per-kw.
+    * The value is read by all actors, hence it needs to be thread-safe.
+    */
+  val feeratesPerKw = new AtomicReference[FeeratesPerKw](null)
 }
 
 

eclair-core/src/main/scala/fr/acinq/eclair/NodeParams.scala

@@ -2,18 +2,19 @@ package fr.acinq.eclair
 
 import java.io.File
 import java.net.InetSocketAddress
-import java.nio.file.{Files, Paths}
+import java.nio.file.Files
+import java.sql.DriverManager
 import java.util.concurrent.TimeUnit
 
 import com.typesafe.config.{Config, ConfigFactory}
-import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
+import fr.acinq.bitcoin.Crypto.PrivateKey
 import fr.acinq.bitcoin.DeterministicWallet.ExtendedPrivateKey
-import fr.acinq.bitcoin.{BinaryData, DeterministicWallet}
-import fr.acinq.eclair.channel.Data
-import fr.acinq.eclair.db.{Dbs, SimpleFileDb, SimpleTypedDb}
-import fr.acinq.eclair.io.PeerRecord
-import fr.acinq.eclair.wire.LightningMessage
+import fr.acinq.bitcoin.{BinaryData, Block, DeterministicWallet}
+import fr.acinq.eclair.NodeParams.WatcherType
+import fr.acinq.eclair.db._
+import fr.acinq.eclair.db.sqlite.{SqliteChannelsDb, SqliteNetworkDb, SqlitePeersDb, SqlitePreimagesDb}
 
+import scala.collection.JavaConversions._
 import scala.concurrent.duration.FiniteDuration
 
 /**
@@ -23,11 +24,11 @@ case class NodeParams(extendedPrivateKey: ExtendedPrivateKey,
                       privateKey: PrivateKey,
                       alias: String,
                       color: (Byte, Byte, Byte),
-                      address: InetSocketAddress,
+                      publicAddresses: List[InetSocketAddress],
                       globalFeatures: BinaryData,
                       localFeatures: BinaryData,
                       dustLimitSatoshis: Long,
-                      maxHtlcValueInFlightMsat: Long,
+                      maxHtlcValueInFlightMsat: UInt64,
                       maxAcceptedHtlcs: Int,
                       expiryDeltaBlocks: Int,
                       htlcMinimumMsat: Int,
@@ -38,60 +39,92 @@ case class NodeParams(extendedPrivateKey: ExtendedPrivateKey,
                       feeProportionalMillionth: Int,
                       reserveToFundingRatio: Double,
                       maxReserveToFundingRatio: Double,
-                      channelsDb: SimpleTypedDb[BinaryData, Data],
-                      peersDb: SimpleTypedDb[PublicKey, PeerRecord],
-                      announcementsDb: SimpleTypedDb[String, LightningMessage],
+                      channelsDb: ChannelsDb,
+                      peersDb: PeersDb,
+                      networkDb: NetworkDb,
+                      preimagesDb: PreimagesDb,
                       routerBroadcastInterval: FiniteDuration,
                       routerValidateInterval: FiniteDuration,
                       pingInterval: FiniteDuration,
                       maxFeerateMismatch: Double,
                       updateFeeMinDiffRatio: Double,
                       autoReconnect: Boolean,
-                      chainHash: BinaryData)
+                      chainHash: BinaryData,
+                      channelFlags: Byte,
+                      channelExcludeDuration: FiniteDuration,
+                      watcherType: WatcherType)
 
 object NodeParams {
 
+  sealed trait WatcherType
+
+  object BITCOIND extends WatcherType
+
+  object BITCOINJ extends WatcherType
+
+  object ELECTRUM extends WatcherType
+
   /**
     * Order of precedence for the configuration parameters:
     * 1) Java environment variables (-D...)
     * 2) Configuration file eclair.conf
-    * 3) default values in reference.conf
+    * 3) Optionally provided config
+    * 4) Default values in reference.conf
     */
-  def loadConfiguration(datadir: File) =
+  def loadConfiguration(datadir: File, overrideDefaults: Config = ConfigFactory.empty()) =
     ConfigFactory.parseProperties(System.getProperties)
       .withFallback(ConfigFactory.parseFile(new File(datadir, "eclair.conf")))
+      .withFallback(overrideDefaults)
       .withFallback(ConfigFactory.load()).getConfig("eclair")
 
-  def makeNodeParams(datadir: File, config: Config, chainHash: BinaryData): NodeParams = {
+  def makeNodeParams(datadir: File, config: Config): NodeParams = {
 
-    val seedPath = Paths.get(datadir.getAbsolutePath, "seed.dat")
-    val seed: BinaryData = Files.exists(seedPath) match {
-      case true => Files.readAllBytes(seedPath)
+    datadir.mkdirs()
+
+    val seedPath = new File(datadir, "seed.dat")
+    val seed: BinaryData = seedPath.exists() match {
+      case true => Files.readAllBytes(seedPath.toPath)
       case false =>
         val seed = randomKey.toBin
-        Files.write(seedPath, seed)
+        Files.write(seedPath.toPath, seed)
         seed
     }
     val master = DeterministicWallet.generate(seed)
     val extendedPrivateKey = DeterministicWallet.derivePrivateKey(master, DeterministicWallet.hardened(46) :: DeterministicWallet.hardened(0) :: Nil)
 
-    val dbPath = Paths.get(datadir.getAbsolutePath, "db")
-    val db = new SimpleFileDb(dbPath.toFile)
+    val chain = config.getString("chain")
+    val chainHash = chain match {
+      case "test" => Block.TestnetGenesisBlock.hash
+      case "regtest" => Block.RegtestGenesisBlock.hash
+      case _ => throw new RuntimeException("only regtest and testnet are supported for now")
+    }
+
+    val sqlite = DriverManager.getConnection(s"jdbc:sqlite:${new File(datadir, "eclair.sqlite")}")
+    val channelsDb = new SqliteChannelsDb(sqlite)
+    val peersDb = new SqlitePeersDb(sqlite)
+    val networkDb = new SqliteNetworkDb(sqlite)
+    val preimagesDb = new SqlitePreimagesDb(sqlite)
 
     val color = BinaryData(config.getString("node-color"))
     require(color.size == 3, "color should be a 3-bytes hex buffer")
 
+    val watcherType = config.getString("watcher-type") match {
+      case "bitcoinj" => BITCOINJ
+      case "electrum" => ELECTRUM
+      case _ => BITCOIND
+    }
+
     NodeParams(
       extendedPrivateKey = extendedPrivateKey,
       privateKey = extendedPrivateKey.privateKey,
       alias = config.getString("node-alias").take(32),
       color = (color.data(0), color.data(1), color.data(2)),
-      address = new InetSocketAddress(config.getString("server.public-ip"), config.getInt("server.port")),
-      globalFeatures = BinaryData(""),
-      localFeatures = BinaryData("05"), // channels_public and initial_routing_sync
-      dustLimitSatoshis = 542,
-      maxHtlcValueInFlightMsat = Long.MaxValue,
-      maxAcceptedHtlcs = 100,
+      publicAddresses = config.getStringList("server.public-ips").toList.map(ip => new InetSocketAddress(ip, config.getInt("server.port"))),
+      globalFeatures = BinaryData(config.getString("global-features")),
+      localFeatures = BinaryData(config.getString("local-features")),
+      dustLimitSatoshis = config.getLong("dust-limit-satoshis"),
+      maxHtlcValueInFlightMsat = UInt64(config.getLong("max-htlc-value-in-flight-msat")),
+      maxAcceptedHtlcs = config.getInt("max-accepted-htlcs"),
       expiryDeltaBlocks = config.getInt("expiry-delta-blocks"),
       htlcMinimumMsat = config.getInt("htlc-minimum-msat"),
       delayBlocks = config.getInt("delay-blocks"),
@@ -99,17 +132,21 @@ object NodeParams {
       smartfeeNBlocks = 3,
       feeBaseMsat = config.getInt("fee-base-msat"),
       feeProportionalMillionth = config.getInt("fee-proportional-millionth"),
-      reserveToFundingRatio = 0.01, // recommended by BOLT #2
-      maxReserveToFundingRatio = 0.05, // channel reserve can't be more than 5% of the funding amount (recommended: 1%)
-      channelsDb = Dbs.makeChannelDb(db),
-      peersDb = Dbs.makePeerDb(db),
-      announcementsDb = Dbs.makeAnnouncementDb(db),
+      reserveToFundingRatio = config.getDouble("reserve-to-funding-ratio"),
+      maxReserveToFundingRatio = config.getDouble("max-reserve-to-funding-ratio"),
+      channelsDb = channelsDb,
+      peersDb = peersDb,
+      networkDb = networkDb,
+      preimagesDb = preimagesDb,
       routerBroadcastInterval = FiniteDuration(config.getDuration("router-broadcast-interval").getSeconds, TimeUnit.SECONDS),
       routerValidateInterval = FiniteDuration(config.getDuration("router-validate-interval").getSeconds, TimeUnit.SECONDS),
       pingInterval = FiniteDuration(config.getDuration("ping-interval").getSeconds, TimeUnit.SECONDS),
       maxFeerateMismatch = config.getDouble("max-feerate-mismatch"),
       updateFeeMinDiffRatio = config.getDouble("update-fee_min-diff-ratio"),
       autoReconnect = config.getBoolean("auto-reconnect"),
-      chainHash = chainHash)
+      chainHash = chainHash,
+      channelFlags = config.getInt("channel-flags").toByte,
+      channelExcludeDuration = FiniteDuration(config.getDuration("channel-exclude-duration").getSeconds, TimeUnit.SECONDS),
+      watcherType = watcherType)
   }
 }

eclair-core/src/main/scala/fr/acinq/eclair/PortChecker.scala

@@ -0,0 +1,26 @@
+package fr.acinq.eclair
+
+import java.net.{InetAddress, ServerSocket}
+
+import scala.util.{Failure, Success, Try}
+
+object PortChecker {
+
+  /**
+    * Tests if a port is open
+    * See https://stackoverflow.com/questions/434718/sockets-discover-port-availability-using-java#435579
+    *
+    * @return
+    */
+  def checkAvailable(host: String, port: Int): Unit = {
+    Try(new ServerSocket(port, 50, InetAddress.getByName(host))) match {
+      case Success(socket) =>
+        Try(socket.close())
+      case Failure(_) =>
+        throw TCPBindException(port)
+    }
+  }
+
+}
+
+case class TCPBindException(port: Int) extends RuntimeException

eclair-core/src/main/scala/fr/acinq/eclair/Setup.scala

@@ -5,134 +5,210 @@ import java.net.InetSocketAddress
 
 import akka.actor.{ActorRef, ActorSystem, Props, SupervisorStrategy}
 import akka.http.scaladsl.Http
-import akka.stream.ActorMaterializer
+import akka.pattern.after
+import akka.stream.{ActorMaterializer, BindFailedException}
 import akka.util.Timeout
-import fr.acinq.bitcoin.{Base58Check, OP_CHECKSIG, OP_DUP, OP_EQUALVERIFY, OP_HASH160, OP_PUSHDATA, Script}
-import fr.acinq.eclair.api.Service
-import fr.acinq.eclair.blockchain.rpc.BitcoinJsonRPCClient
-import fr.acinq.eclair.blockchain.zmq.ZMQActor
-import fr.acinq.eclair.blockchain.{ExtendedBitcoinClient, PeerWatcher}
+import com.typesafe.config.{Config, ConfigFactory}
+import fr.acinq.bitcoin.{BinaryData, Block}
+import fr.acinq.eclair.NodeParams.{BITCOIND, BITCOINJ, ELECTRUM}
+import fr.acinq.eclair.api.{GetInfoResponse, Service}
+import fr.acinq.eclair.blockchain.bitcoind.rpc.{BitcoinJsonRPCClient, ExtendedBitcoinClient}
+import fr.acinq.eclair.blockchain.bitcoind.zmq.ZMQActor
+import fr.acinq.eclair.blockchain.bitcoind.{BitcoinCoreWallet, ZmqWatcher}
+import fr.acinq.eclair.blockchain.bitcoinj.{BitcoinjKit, BitcoinjWallet, BitcoinjWatcher}
+import fr.acinq.eclair.blockchain.electrum.{ElectrumClient, ElectrumEclairWallet, ElectrumWallet, ElectrumWatcher}
+import fr.acinq.eclair.blockchain.fee.{ConstantFeeProvider, _}
+import fr.acinq.eclair.blockchain.{EclairWallet, _}
 import fr.acinq.eclair.channel.Register
 import fr.acinq.eclair.io.{Server, Switchboard}
 import fr.acinq.eclair.payment._
 import fr.acinq.eclair.router._
-import fr.acinq.eclair.wire.{ChannelAnnouncement, ChannelUpdate, NodeAnnouncement}
 import grizzled.slf4j.Logging
-import org.json4s.JsonAST.JString
 
-import scala.compat.Platform
+import scala.collection.JavaConversions._
 import scala.concurrent.duration._
-import scala.concurrent.{Await, ExecutionContext, Promise}
-import scala.util.Try
+import scala.concurrent.{Await, ExecutionContext, Future, Promise}
 
 /**
   * Created by PM on 25/01/2016.
   */
-class Setup(datadir: String, actorSystemName: String = "default") extends Logging {
+class Setup(datadir: File, overrideDefaults: Config = ConfigFactory.empty(), actorSystem: ActorSystem = ActorSystem()) extends Logging {
 
   logger.info(s"hello!")
   logger.info(s"version=${getClass.getPackage.getImplementationVersion} commit=${getClass.getPackage.getSpecificationVersion}")
-  val config = NodeParams.loadConfiguration(new File(datadir))
+
+  val config = NodeParams.loadConfiguration(datadir, overrideDefaults)
+  val nodeParams = NodeParams.makeNodeParams(datadir, config)
+  val chain = config.getString("chain")
+
+  // early checks
+  DBCompatChecker.checkDBCompatibility(nodeParams)
+  PortChecker.checkAvailable(config.getString("server.binding-ip"), config.getInt("server.port"))
+
+  logger.info(s"nodeid=${nodeParams.privateKey.publicKey.toBin} alias=${nodeParams.alias}")
+  logger.info(s"using chain=$chain chainHash=${nodeParams.chainHash}")
 
   logger.info(s"initializing secure random generator")
   // this will force the secure random instance to initialize itself right now, making sure it doesn't hang later (see comment in package.scala)
   secureRandom.nextInt()
 
-  implicit lazy val system = ActorSystem(actorSystemName)
+  implicit val system = actorSystem
   implicit val materializer = ActorMaterializer()
   implicit val timeout = Timeout(30 seconds)
-
-  val bitcoinClient = new ExtendedBitcoinClient(new BitcoinJsonRPCClient(
-    user = config.getString("bitcoind.rpcuser"),
-    password = config.getString("bitcoind.rpcpassword"),
-    host = config.getString("bitcoind.host"),
-    port = config.getInt("bitcoind.rpcport")))
-
   implicit val formats = org.json4s.DefaultFormats
   implicit val ec = ExecutionContext.Implicits.global
 
-  val future = for {
-    json <- bitcoinClient.client.invoke("getblockchaininfo")
-    chain = (json \ "chain").extract[String]
-    blockCount = (json \ "blocks").extract[Long]
-    progress = (json \ "verificationprogress").extract[Double]
-    chainHash <- bitcoinClient.client.invoke("getblockhash", 0).map(_.extract[String])
-  } yield (chain, blockCount, progress, chainHash)
-  val (chain, blockCount, progress, chainHash) = Try(Await.result(future, 10 seconds)).recover { case _ => throw BitcoinRPCConnectionException }.get
-  logger.info(s"using chain=$chain chainHash=$chainHash")
-  chain match {
-    case "test" | "regtest" => ()
-    case _ => throw new RuntimeException("only regtest and testnet are supported for now")
-  }
-  val nodeParams = NodeParams.makeNodeParams(new File(datadir), config, chainHash)
-  logger.info(s"nodeid=${nodeParams.privateKey.publicKey.toBin} alias=${nodeParams.alias}")
-  assert(progress > 0.99, "bitcoind should be synchronized")
-
-  Globals.blockCount.set(blockCount)
-  val defaultFeeratePerKw = config.getLong("default-feerate-perkw")
-  val feeratePerKw = if (chain == "regtest") defaultFeeratePerKw else {
-    val feeratePerKB = Await.result(bitcoinClient.estimateSmartFee(nodeParams.smartfeeNBlocks), 10 seconds)
-    if (feeratePerKB < 0) defaultFeeratePerKw else feerateKB2Kw(feeratePerKB)
-  }
-
-  logger.info(s"initial feeratePerKw=$feeratePerKw")
-  Globals.feeratePerKw.set(feeratePerKw)
-  val bitcoinVersion = Await.result(bitcoinClient.client.invoke("getinfo").map(json => (json \ "version").extract[String]), 10 seconds)
-  // we use it as final payment address, so that funds are moved to the bitcoind wallet upon channel termination
-  val JString(finalAddress) = Await.result(bitcoinClient.client.invoke("getnewaddress"), 10 seconds)
-  logger.info(s"finaladdress=$finalAddress")
-  // TODO: we should use p2wpkh instead of p2pkh as soon as bitcoind supports it
-  //val finalScriptPubKey = OP_0 :: OP_PUSHDATA(Base58Check.decode(finalAddress)._2) :: Nil
-  val finalScriptPubKey = Script.write(OP_DUP :: OP_HASH160 :: OP_PUSHDATA(Base58Check.decode(finalAddress)._2) :: OP_EQUALVERIFY :: OP_CHECKSIG :: Nil)
-
-  val zmqConnected = Promise[Boolean]()
-  val zmq = system.actorOf(SimpleSupervisor.props(Props(new ZMQActor(config.getString("bitcoind.zmq"), Some(zmqConnected))), "zmq", SupervisorStrategy.Restart))
-  val watcher = system.actorOf(SimpleSupervisor.props(PeerWatcher.props(nodeParams, bitcoinClient), "watcher", SupervisorStrategy.Resume))
-  val paymentHandler = system.actorOf(SimpleSupervisor.props(config.getString("payment-handler") match {
-    case "local" => LocalPaymentHandler.props(nodeParams)
-    case "noop" => Props[NoopPaymentHandler]
-  }, "payment-handler", SupervisorStrategy.Resume))
-  val register = system.actorOf(SimpleSupervisor.props(Props(new Register), "register", SupervisorStrategy.Resume))
-  val relayer = system.actorOf(SimpleSupervisor.props(Relayer.props(nodeParams.privateKey, paymentHandler), "relayer", SupervisorStrategy.Resume))
-  val router = system.actorOf(SimpleSupervisor.props(Router.props(nodeParams, watcher), "router", SupervisorStrategy.Resume))
-  val switchboard = system.actorOf(SimpleSupervisor.props(Switchboard.props(nodeParams, watcher, router, relayer, finalScriptPubKey), "switchboard", SupervisorStrategy.Resume))
-  val paymentInitiator = system.actorOf(SimpleSupervisor.props(PaymentInitiator.props(nodeParams.privateKey.publicKey, router, register), "payment-initiator", SupervisorStrategy.Restart))
-  val tcpBound = Promise[Unit]()
-  val server = system.actorOf(SimpleSupervisor.props(Server.props(nodeParams, switchboard, new InetSocketAddress(config.getString("server.binding-ip"), config.getInt("server.port")), Some(tcpBound)), "server", SupervisorStrategy.Restart))
-
-  val _setup = this
-  val api = new Service {
-    override val switchboard: ActorRef = _setup.switchboard
-    override val router: ActorRef = _setup.router
-    override val register: ActorRef = _setup.register
-    override val paymentHandler: ActorRef = _setup.paymentHandler
-    override val paymentInitiator: ActorRef = _setup.paymentInitiator
-    override val system: ActorSystem = _setup.system
-  }
-  val httpBound = Http().bindAndHandle(api.route, config.getString("api.binding-ip"), config.getInt("api.port"))
-
-  Try(Await.result(zmqConnected.future, 5 seconds)).recover { case _ => throw BitcoinZMQConnectionTimeoutException }.get
-  Try(Await.result(tcpBound.future, 5 seconds)).recover { case _ => throw new TCPBindException(config.getInt("server.port")) }.get
-  Try(Await.result(httpBound, 5 seconds)).recover { case _ => throw new TCPBindException(config.getInt("api.port")) }.get
-
-  val tasks = new Thread(new Runnable() {
-    override def run(): Unit = {
-      nodeParams.peersDb.values.foreach(rec => switchboard ! rec)
-      nodeParams.channelsDb.values.foreach(rec => switchboard ! rec)
-      nodeParams.announcementsDb.values.collect { case ann: ChannelAnnouncement => router ! ann }
-      nodeParams.announcementsDb.values.collect { case ann: NodeAnnouncement => router ! ann }
-      nodeParams.announcementsDb.values.collect { case ann: ChannelUpdate => router ! ann }
-      if (nodeParams.channelsDb.values.size > 0) {
-        val nodeAnn = Announcements.makeNodeAnnouncement(nodeParams.privateKey, nodeParams.alias, nodeParams.color, nodeParams.address :: Nil, Platform.currentTime / 1000)
-        router ! nodeAnn
+  val bitcoin = nodeParams.watcherType match {
+    case BITCOIND =>
+      val bitcoinClient = new ExtendedBitcoinClient(new BitcoinJsonRPCClient(
+        user = config.getString("bitcoind.rpcuser"),
+        password = config.getString("bitcoind.rpcpassword"),
+        host = config.getString("bitcoind.host"),
+        port = config.getInt("bitcoind.rpcport")))
+      val future = for {
+        json <- bitcoinClient.rpcClient.invoke("getblockchaininfo").recover { case _ => throw BitcoinRPCConnectionException }
+        progress = (json \ "verificationprogress").extract[Double]
+        chainHash <- bitcoinClient.rpcClient.invoke("getblockhash", 0).map(_.extract[String]).map(BinaryData(_)).map(x => BinaryData(x.reverse))
+        bitcoinVersion <- bitcoinClient.rpcClient.invoke("getnetworkinfo").map(json => (json \ "version")).map(_.extract[String])
+      } yield (progress, chainHash, bitcoinVersion)
+      // blocking sanity checks
+      val (progress, chainHash, bitcoinVersion) = Await.result(future, 10 seconds)
+      assert(chainHash == nodeParams.chainHash, s"chainHash mismatch (conf=${nodeParams.chainHash} != bitcoind=$chainHash)")
+      assert(progress > 0.99, "bitcoind should be synchronized")
+      // TODO: add a check on bitcoin version?
+      Bitcoind(bitcoinClient)
+    case BITCOINJ =>
+      logger.warn("EXPERIMENTAL BITCOINJ MODE ENABLED!!!")
+      val staticPeers = config.getConfigList("bitcoinj.static-peers").map(c => new InetSocketAddress(c.getString("host"), c.getInt("port"))).toList
+      logger.info(s"using staticPeers=$staticPeers")
+      val bitcoinjKit = new BitcoinjKit(chain, datadir, staticPeers)
+      bitcoinjKit.startAsync()
+      Await.ready(bitcoinjKit.initialized, 10 seconds)
+      Bitcoinj(bitcoinjKit)
+    case ELECTRUM =>
+      logger.warn("EXPERIMENTAL ELECTRUM MODE ENABLED!!!")
+      val addressesFile = chain match {
+        case "test" => "/electrum/servers_testnet.json"
+        case "regtest" => "/electrum/servers_regtest.json"
       }
-    }
-  })
+      val stream = classOf[Setup].getResourceAsStream(addressesFile)
+      val addresses = ElectrumClient.readServerAddresses(stream)
+      val electrumClient = system.actorOf(SimpleSupervisor.props(Props(new ElectrumClient(addresses)), "electrum-client", SupervisorStrategy.Resume))
+      Electrum(electrumClient)
+  }
+
+  def bootstrap: Future[Kit] = {
+    val zmqConnected = Promise[Boolean]()
+    val tcpBound = Promise[Unit]()
+
+    val defaultFeerates = FeeratesPerByte(block_1 = config.getLong("default-feerates.delay-blocks.1"), blocks_2 = config.getLong("default-feerates.delay-blocks.2"), blocks_6 = config.getLong("default-feerates.delay-blocks.6"), blocks_12 = config.getLong("default-feerates.delay-blocks.12"), blocks_36 = config.getLong("default-feerates.delay-blocks.36"), blocks_72 = config.getLong("default-feerates.delay-blocks.72"))
+    Globals.feeratesPerByte.set(defaultFeerates)
+    Globals.feeratesPerKw.set(FeeratesPerKw(defaultFeerates))
+    logger.info(s"initial feeratesPerByte=${Globals.feeratesPerByte.get()}")
+    val feeProvider = (chain, bitcoin) match {
+      case ("regtest", _) => new ConstantFeeProvider(defaultFeerates)
+      case (_, Bitcoind(client)) => new FallbackFeeProvider(new EarnDotComFeeProvider() :: new BitcoinCoreFeeProvider(client.rpcClient, defaultFeerates) :: new ConstantFeeProvider(defaultFeerates) :: Nil) // order matters!
+      case _ => new FallbackFeeProvider(new EarnDotComFeeProvider() :: new ConstantFeeProvider(defaultFeerates) :: Nil) // order matters!
+    }
+    system.scheduler.schedule(0 seconds, 10 minutes)(feeProvider.getFeerates.map {
+      case feerates: FeeratesPerByte =>
+        Globals.feeratesPerByte.set(feerates)
+        Globals.feeratesPerKw.set(FeeratesPerKw(defaultFeerates))
+        system.eventStream.publish(CurrentFeerates(Globals.feeratesPerKw.get))
+        logger.info(s"current feeratesPerByte=${Globals.feeratesPerByte.get()}")
+    })
+
+    val watcher = bitcoin match {
+      case Bitcoind(bitcoinClient) =>
+        system.actorOf(SimpleSupervisor.props(Props(new ZMQActor(config.getString("bitcoind.zmq"), Some(zmqConnected))), "zmq", SupervisorStrategy.Restart))
+        system.actorOf(SimpleSupervisor.props(ZmqWatcher.props(bitcoinClient), "watcher", SupervisorStrategy.Resume))
+      case Bitcoinj(bitcoinj) =>
+        zmqConnected.success(true)
+        system.actorOf(SimpleSupervisor.props(BitcoinjWatcher.props(bitcoinj), "watcher", SupervisorStrategy.Resume))
+      case Electrum(electrumClient) =>
+        zmqConnected.success(true)
+        system.actorOf(SimpleSupervisor.props(Props(new ElectrumWatcher(electrumClient)), "watcher", SupervisorStrategy.Resume))
+    }
+
+    val wallet = bitcoin match {
+      case Bitcoind(bitcoinClient) => new BitcoinCoreWallet(bitcoinClient.rpcClient, watcher)
+      case Bitcoinj(bitcoinj) => new BitcoinjWallet(bitcoinj.initialized.map(_ => bitcoinj.wallet()))
+      case Electrum(electrumClient) =>
+        val electrumSeedPath = new File(datadir, "electrum_seed.dat")
+        val electrumWallet = system.actorOf(ElectrumWallet.props(electrumSeedPath, electrumClient, ElectrumWallet.WalletParameters(Block.RegtestGenesisBlock.hash, allowSpendUnconfirmed = true)), "electrum-wallet")
+        new ElectrumEclairWallet(electrumWallet)
+    }
+    wallet.getFinalAddress.map {
+      case address => logger.info(s"initial wallet address=$address")
+    }
+
+    val paymentHandler = system.actorOf(SimpleSupervisor.props(config.getString("payment-handler") match {
+      case "local" => LocalPaymentHandler.props(nodeParams)
+      case "noop" => Props[NoopPaymentHandler]
+    }, "payment-handler", SupervisorStrategy.Resume))
+    val register = system.actorOf(SimpleSupervisor.props(Props(new Register), "register", SupervisorStrategy.Resume))
+    val relayer = system.actorOf(SimpleSupervisor.props(Relayer.props(nodeParams, register, paymentHandler), "relayer", SupervisorStrategy.Resume))
+    val router = system.actorOf(SimpleSupervisor.props(Router.props(nodeParams, watcher), "router", SupervisorStrategy.Resume))
+    val switchboard = system.actorOf(SimpleSupervisor.props(Switchboard.props(nodeParams, watcher, router, relayer, wallet), "switchboard", SupervisorStrategy.Resume))
+    val paymentInitiator = system.actorOf(SimpleSupervisor.props(PaymentInitiator.props(nodeParams.privateKey.publicKey, router, register), "payment-initiator", SupervisorStrategy.Restart))
+    val server = system.actorOf(SimpleSupervisor.props(Server.props(nodeParams, switchboard, new InetSocketAddress(config.getString("server.binding-ip"), config.getInt("server.port")), Some(tcpBound)), "server", SupervisorStrategy.Restart))
+
+    val kit = Kit(
+      nodeParams = nodeParams,
+      system = system,
+      watcher = watcher,
+      paymentHandler = paymentHandler,
+      register = register,
+      relayer = relayer,
+      router = router,
+      switchboard = switchboard,
+      paymentInitiator = paymentInitiator,
+      server = server,
+      wallet = wallet)
+
+    val api = new Service {
+
+      override def getInfoResponse: Future[GetInfoResponse] = Future.successful(GetInfoResponse(nodeId = nodeParams.privateKey.publicKey, alias = nodeParams.alias, port = config.getInt("server.port"), chainHash = nodeParams.chainHash, blockHeight = Globals.blockCount.intValue()))
+
+      override def appKit = kit
+    }
+    val httpBound = Http().bindAndHandle(api.route, config.getString("api.binding-ip"), config.getInt("api.port")).recover {
+      case _: BindFailedException => throw TCPBindException(config.getInt("api.port"))
+    }
+
+    val zmqTimeout = after(5 seconds, using = system.scheduler)(Future.failed(BitcoinZMQConnectionTimeoutException))
+    val tcpTimeout = after(5 seconds, using = system.scheduler)(Future.failed(TCPBindException(config.getInt("server.port"))))
+    val httpTimeout = after(5 seconds, using = system.scheduler)(Future.failed(TCPBindException(config.getInt("api.port"))))
+
+    for {
+      _ <- Future.firstCompletedOf(zmqConnected.future :: zmqTimeout :: Nil)
+      _ <- Future.firstCompletedOf(tcpBound.future :: tcpTimeout :: Nil)
+      _ <- Future.firstCompletedOf(httpBound :: httpTimeout :: Nil)
+    } yield kit
+
+  }
 
-  def boostrap: Unit = tasks.start()
 }
 
-case class TCPBindException(port: Int) extends RuntimeException
+
+// @formatter:off
+sealed trait Bitcoin
+case class Bitcoind(extendedBitcoinClient: ExtendedBitcoinClient) extends Bitcoin
+case class Bitcoinj(bitcoinjKit: BitcoinjKit) extends Bitcoin
+case class Electrum(electrumClient: ActorRef) extends Bitcoin
+// @formatter:on
+
+case class Kit(nodeParams: NodeParams,
+               system: ActorSystem,
+               watcher: ActorRef,
+               paymentHandler: ActorRef,
+               register: ActorRef,
+               relayer: ActorRef,
+               router: ActorRef,
+               switchboard: ActorRef,
+               paymentInitiator: ActorRef,
+               server: ActorRef,
+               wallet: EclairWallet)
 
 case object BitcoinZMQConnectionTimeoutException extends RuntimeException("could not connect to bitcoind using zeromq")
 

eclair-core/src/main/scala/fr/acinq/eclair/UInt64.scala

@@ -0,0 +1,35 @@
+package fr.acinq.eclair
+
+import java.math.BigInteger
+
+import fr.acinq.bitcoin.BinaryData
+
+case class UInt64(underlying: BigInt) extends Ordered[UInt64] {
+
+  require(underlying >= 0, s"uint64 must be positive (actual=$underlying)")
+  require(underlying <= UInt64.MaxValueBigInt, s"uint64 must be < 2^64 -1 (actual=$underlying)")
+
+  override def compare(o: UInt64): Int = underlying.compare(o.underlying)
+
+  override def toString: String = underlying.toString
+}
+
+
+object UInt64 {
+
+  private val MaxValueBigInt = BigInt(new BigInteger("ffffffffffffffff", 16))
+
+  val MaxValue = UInt64(MaxValueBigInt)
+
+  def apply(bin: BinaryData) = new UInt64(new BigInteger(1, bin))
+
+  def apply(value: Long) = new UInt64(BigInt(value))
+
+  object Conversions {
+
+    implicit def intToUint64(l: Int) = UInt64(l)
+
+    implicit def longToUint64(l: Long) = UInt64(l)
+  }
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/api/JsonSerializers.scala

@@ -1,7 +1,7 @@
 package fr.acinq.eclair.api
 
-import fr.acinq.bitcoin.{BinaryData, Script, ScriptElt, Transaction}
 import fr.acinq.bitcoin.Crypto.{Point, PrivateKey, PublicKey, Scalar}
+import fr.acinq.bitcoin.{BinaryData, Transaction}
 import fr.acinq.eclair.channel.State
 import fr.acinq.eclair.crypto.ShaChain
 import fr.acinq.eclair.transactions.Transactions.TransactionWithInputInfo

eclair-core/src/main/scala/fr/acinq/eclair/api/Service.scala

@@ -2,7 +2,7 @@ package fr.acinq.eclair.api
 
 import java.net.InetSocketAddress
 
-import akka.actor.{ActorRef, ActorSystem}
+import akka.actor.ActorRef
 import akka.http.scaladsl.model.HttpMethods._
 import akka.http.scaladsl.model.StatusCodes
 import akka.http.scaladsl.model.headers.CacheDirectives.{`max-age`, `no-store`, public}
@@ -15,10 +15,11 @@ import de.heikoseeberger.akkahttpjson4s.Json4sSupport
 import de.heikoseeberger.akkahttpjson4s.Json4sSupport.ShouldWritePretty
 import fr.acinq.bitcoin.Crypto.PublicKey
 import fr.acinq.bitcoin.{BinaryData, MilliSatoshi, Satoshi}
+import fr.acinq.eclair.Kit
 import fr.acinq.eclair.channel._
 import fr.acinq.eclair.io.Switchboard.{NewChannel, NewConnection}
 import fr.acinq.eclair.payment.{PaymentRequest, PaymentResult, ReceivePayment, SendPayment}
-import fr.acinq.eclair.wire.NodeAnnouncement
+import fr.acinq.eclair.wire.{ChannelAnnouncement, NodeAnnouncement}
 import grizzled.slf4j.Logging
 import org.json4s.JsonAST.{JInt, JString}
 import org.json4s.{JValue, jackson}
@@ -36,6 +37,8 @@ case class JsonRPCBody(jsonrpc: String = "1.0", id: String = "scala-client", met
 case class Error(code: Int, message: String)
 case class JsonRPCRes(result: AnyRef, error: Option[Error], id: String)
 case class Status(node_id: String)
+case class GetInfoResponse(nodeId: PublicKey, alias: String, port: Int, chainHash: BinaryData, blockHeight: Int)
+case class ChannelInfo(shortChannelId: String, nodeId1: PublicKey , nodeId2: PublicKey)
 // @formatter:on
 
 trait Service extends Logging {
@@ -49,17 +52,9 @@ trait Service extends Logging {
 
   import Json4sSupport.{marshaller, unmarshaller}
 
-  def switchboard: ActorRef
+  def appKit: Kit
 
-  def router: ActorRef
-
-  def register: ActorRef
-
-  def paymentInitiator: ActorRef
-
-  def paymentHandler: ActorRef
-
-  def system: ActorSystem
+  def getInfoResponse: Future[GetInfoResponse]
 
   val customHeaders = `Access-Control-Allow-Origin`(*) ::
     `Access-Control-Allow-Headers`("Content-Type, Authorization") ::
@@ -69,7 +64,7 @@ trait Service extends Logging {
 
   def getChannel(channelId: String): Future[ActorRef] =
     for {
-      channels <- (register ? 'channels).mapTo[Map[BinaryData, ActorRef]]
+      channels <- (appKit.register ? 'channels).mapTo[Map[BinaryData, ActorRef]]
     } yield channels.get(BinaryData(channelId)).getOrElse(throw new RuntimeException("unknown channel"))
 
   val route =
@@ -78,37 +73,64 @@ trait Service extends Logging {
         post {
           entity(as[JsonRPCBody]) {
             req =>
+              val kit = appKit
+              import kit._
               val f_res: Future[AnyRef] = req match {
-                case JsonRPCBody(_, _, "connect", JString(host) :: JInt(port) :: JString(nodeId) :: Nil) =>
+                case JsonRPCBody(_, _, "getinfo", _) => getInfoResponse
+                case JsonRPCBody(_, _, "connect", JString(nodeId) :: JString(host) :: JInt(port) :: Nil) =>
                   (switchboard ? NewConnection(PublicKey(nodeId), new InetSocketAddress(host, port.toInt), None)).mapTo[String]
-                case JsonRPCBody(_, _, "open", JString(host) :: JInt(port) :: JString(nodeId) :: JInt(fundingSatoshi) :: JInt(pushMsat) :: Nil) =>
-                  (switchboard ? NewConnection(PublicKey(nodeId), new InetSocketAddress(host, port.toInt), Some(NewChannel(Satoshi(fundingSatoshi.toLong), MilliSatoshi(pushMsat.toLong))))).mapTo[String]
+                case JsonRPCBody(_, _, "open", JString(nodeId) :: JString(host) :: JInt(port) :: JInt(fundingSatoshi) :: JInt(pushMsat) :: options) =>
+                  val channelFlags = options match {
+                    case JInt(value) :: Nil => Some(value.toByte)
+                    case _ => None // TODO: too lax?
+                  }
+                  (switchboard ? NewConnection(PublicKey(nodeId), new InetSocketAddress(host, port.toInt), Some(NewChannel(Satoshi(fundingSatoshi.toLong), MilliSatoshi(pushMsat.toLong), channelFlags)))).mapTo[String]
                 case JsonRPCBody(_, _, "peers", _) =>
                   (switchboard ? 'peers).mapTo[Map[PublicKey, ActorRef]].map(_.map(_._1.toBin))
                 case JsonRPCBody(_, _, "channels", _) =>
                   (register ? 'channels).mapTo[Map[Long, ActorRef]].map(_.keys)
                 case JsonRPCBody(_, _, "channel", JString(channelId) :: Nil) =>
                   getChannel(channelId).flatMap(_ ? CMD_GETINFO).mapTo[RES_GETINFO]
-                case JsonRPCBody(_, _, "network", _) =>
+                case JsonRPCBody(_, _, "allnodes", _) =>
                   (router ? 'nodes).mapTo[Iterable[NodeAnnouncement]].map(_.map(_.nodeId))
-                case JsonRPCBody(_,_, "receive", JInt(amountMsat) :: Nil) =>
-                  (paymentHandler ? ReceivePayment(new MilliSatoshi(amountMsat.toLong))).mapTo[PaymentRequest].map(PaymentRequest.write(_))
+                case JsonRPCBody(_, _, "allchannels", _) =>
+                  (router ? 'channels).mapTo[Iterable[ChannelAnnouncement]].map(_.map(c => ChannelInfo(c.shortChannelId.toHexString, c.nodeId1, c.nodeId2)))
+                case JsonRPCBody(_, _, "receive", JInt(amountMsat) :: JString(description) :: Nil) =>
+                  (paymentHandler ? ReceivePayment(MilliSatoshi(amountMsat.toLong), description)).mapTo[PaymentRequest].map(PaymentRequest.write)
                 case JsonRPCBody(_, _, "send", JInt(amountMsat) :: JString(paymentHash) :: JString(nodeId) :: Nil) =>
                   (paymentInitiator ? SendPayment(amountMsat.toLong, paymentHash, PublicKey(nodeId))).mapTo[PaymentResult]
+                case JsonRPCBody(_, _, "send", JString(paymentRequest) :: rest) =>
+                  for {
+                    req <- Future(PaymentRequest.read(paymentRequest))
+                    amount = (req.amount, rest) match {
+                      case (Some(_), JInt(amt) :: Nil) => amt.toLong // overriding payment request amount with the one provided
+                      case (Some(amt), _) => amt.amount
+                      case (None, JInt(amt) :: Nil) => amt.toLong // amount wasn't specified in request, using custom one
+                      case (None, _) => throw new RuntimeException("you need to manually specify an amount for this payment request")
+                    }
+                    sendPayment = req.minFinalCltvExpiry match {
+                      case None => SendPayment(amount, req.paymentHash, req.nodeId)
+                      case Some(value) => SendPayment(amount, req.paymentHash, req.nodeId, value)
+                    }
+                    res <- (paymentInitiator ? sendPayment).mapTo[PaymentResult]
+                  } yield res
                 case JsonRPCBody(_, _, "close", JString(channelId) :: JString(scriptPubKey) :: Nil) =>
                   getChannel(channelId).flatMap(_ ? CMD_CLOSE(scriptPubKey = Some(scriptPubKey))).mapTo[String]
                 case JsonRPCBody(_, _, "close", JString(channelId) :: Nil) =>
                   getChannel(channelId).flatMap(_ ? CMD_CLOSE(scriptPubKey = None)).mapTo[String]
                 case JsonRPCBody(_, _, "help", _) =>
                   Future.successful(List(
-                    "connect (host, port, nodeId): connect to another lightning node through a secure connection",
-                    "open (host, port, nodeId, fundingSatoshi, pushMsat): open a channel with another lightning node",
+                    "connect (nodeId, host, port): connect to another lightning node through a secure connection",
+                    "open (nodeId, host, port, fundingSatoshi, pushMsat, channelFlags = 0x01): open a channel with another lightning node",
                     "peers: list existing local peers",
                     "channels: list existing local channels",
                     "channel (channelId): retrieve detailed information about a given channel",
-                    "network: list all the nodes announced in network",
-                    "receive (amountMsat): generate a payment request for a given amount",
+                    "allnodes: list all known nodes",
+                    "allchannels: list all known channels",
+                    "receive (amountMsat, description): generate a payment request for a given amount",
                     "send (amountMsat, paymentHash, nodeId): send a payment to a lightning node",
+                    "send (paymentRequest): send a payment to a lightning node using a BOLT11 payment request",
+                    "send (paymentRequest, amountMsat): send a payment to a lightning node using a BOLT11 payment request and a custom amount",
                     "close (channelId): close a channel",
                     "close (channelId, scriptPubKey): close a channel and send the funds to the given scriptPubKey",
                     "help: display this message"))

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/BlockchainEvents.scala

@@ -1,6 +1,7 @@
 package fr.acinq.eclair.blockchain
 
 import fr.acinq.bitcoin.{Block, Transaction}
+import fr.acinq.eclair.blockchain.fee.FeeratesPerKw
 
 /**
   * Created by PM on 24/08/2016.
@@ -14,6 +15,4 @@ case class NewTransaction(tx: Transaction) extends BlockchainEvent
 
 case class CurrentBlockCount(blockCount: Long) extends BlockchainEvent
 
-case class CurrentFeerate(feeratePerKw: Long) extends BlockchainEvent
-
-case class MempoolTransaction(tx: Transaction)
+case class CurrentFeerates(feeratesPerKw: FeeratesPerKw) extends BlockchainEvent

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/EclairWallet.scala

@@ -0,0 +1,41 @@
+package fr.acinq.eclair.blockchain
+
+import fr.acinq.bitcoin.{BinaryData, Satoshi, Transaction}
+
+import scala.concurrent.Future
+
+/**
+  * Created by PM on 06/07/2017.
+  */
+trait EclairWallet {
+
+  def getBalance: Future[Satoshi]
+
+  def getFinalAddress: Future[String]
+
+  def makeFundingTx(pubkeyScript: BinaryData, amount: Satoshi, feeRatePerKw: Long): Future[MakeFundingTxResponse]
+
+  /**
+    * Committing *must* include publishing the transaction on the network.
+    *
+    * We need to be very careful here, we don't want to consider a commit 'failed' if we are not absolutely sure that the
+    * funding tx won't end up on the blockchain: if that happens and we have cancelled the channel, then we would lose our
+    * funds!
+    *
+    * @param tx
+    * @return true if success
+    *         false IF AND ONLY IF *HAS NOT BEEN PUBLISHED* otherwise funds are at risk!!!
+    */
+  def commit(tx: Transaction): Future[Boolean]
+
+  /**
+    * Cancels this transaction: this probably translates to "release locks on utxos".
+    *
+    * @param tx
+    * @return
+    */
+  def rollback(tx: Transaction): Future[Boolean]
+
+}
+
+final case class MakeFundingTxResponse(fundingTx: Transaction, fundingTxOutputIndex: Int)

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/Types.scala

@@ -1,43 +0,0 @@
-package fr.acinq.eclair.blockchain
-
-import akka.actor.ActorRef
-import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
-import fr.acinq.bitcoin.{BinaryData, Satoshi, Transaction}
-import fr.acinq.eclair.channel.BitcoinEvent
-import fr.acinq.eclair.wire.ChannelAnnouncement
-
-/**
-  * Created by PM on 19/01/2016.
-  */
-
-// @formatter:off
-
-sealed trait Watch {
-  def channel: ActorRef
-  def event: BitcoinEvent
-}
-final case class WatchConfirmed(channel: ActorRef, txId: BinaryData, minDepth: Long, event: BitcoinEvent) extends Watch
-final case class WatchSpent(channel: ActorRef, txId: BinaryData, outputIndex: Int, event: BitcoinEvent) extends Watch
-final case class WatchSpentBasic(channel: ActorRef, txId: BinaryData, outputIndex: Int, event: BitcoinEvent) extends Watch // we use this when we don't care about the spending tx, and we also assume txid already exists
-// TODO: notify me if confirmation number gets below minDepth?
-final case class WatchLost(channel: ActorRef, txId: BinaryData, minDepth: Long, event: BitcoinEvent) extends Watch
-
-trait WatchEvent {
-  def event: BitcoinEvent
-}
-final case class WatchEventConfirmed(event: BitcoinEvent, blockHeight: Int, txIndex: Int) extends WatchEvent
-final case class WatchEventSpent(event: BitcoinEvent, tx: Transaction) extends WatchEvent
-final case class WatchEventSpentBasic(event: BitcoinEvent) extends WatchEvent
-final case class WatchEventLost(event: BitcoinEvent) extends WatchEvent
-
-/**
-  * Publish the provided tx as soon as possible depending on locktime and csv
-  */
-final case class PublishAsap(tx: Transaction)
-final case class MakeFundingTx(localCommitPub: PublicKey, remoteCommitPub: PublicKey, amount: Satoshi, feeRatePerKw: Long)
-final case class MakeFundingTxResponse(parentTx: Transaction, fundingTx: Transaction, fundingTxOutputIndex: Int, priv: PrivateKey)
-final case class ParallelGetRequest(ann: Seq[ChannelAnnouncement])
-final case class IndividualResult(c: ChannelAnnouncement, tx: Option[Transaction], unspent: Boolean)
-final case class ParallelGetResponse(r: Seq[IndividualResult])
-
-// @formatter:on

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/WatcherTypes.scala

@@ -0,0 +1,67 @@
+package fr.acinq.eclair.blockchain
+
+import akka.actor.ActorRef
+import fr.acinq.bitcoin.Crypto.PublicKey
+import fr.acinq.bitcoin.{BinaryData, Script, ScriptWitness, Transaction}
+import fr.acinq.eclair.channel.BitcoinEvent
+import fr.acinq.eclair.wire.ChannelAnnouncement
+
+import scala.util.{Failure, Success, Try}
+
+/**
+  * Created by PM on 19/01/2016.
+  */
+
+// @formatter:off
+
+sealed trait Watch {
+  def channel: ActorRef
+  def event: BitcoinEvent
+}
+// we need a public key script to use bitcoinj or electrum apis
+final case class WatchConfirmed(channel: ActorRef, txId: BinaryData, publicKeyScript: BinaryData, minDepth: Long, event: BitcoinEvent) extends Watch
+object WatchConfirmed {
+  // if we have the entire transaction, we can get the redeemScript from the witness, and re-compute the publicKeyScript
+  // we support both p2pkh and p2wpkh scripts
+  def apply(channel: ActorRef, tx: Transaction, minDepth: Long, event: BitcoinEvent): WatchConfirmed = WatchConfirmed(channel, tx.txid, extractPublicKeyScript(tx.txIn.head.witness), minDepth, event)
+
+  def extractPublicKeyScript(witness: ScriptWitness): BinaryData = Try(PublicKey(witness.stack.last)) match {
+    case Success(pubKey) =>
+      // if last element of the witness is a public key, then this is a p2wpkh
+      Script.write(Script.pay2wpkh(pubKey))
+    case Failure(_) =>
+      // otherwise this is a p2wsh
+      witness.stack.last
+  }
+}
+
+final case class WatchSpent(channel: ActorRef, txId: BinaryData, outputIndex: Int, publicKeyScript: BinaryData, event: BitcoinEvent) extends Watch
+object WatchSpent {
+  // if we have the entire transaction, we can get the publicKeyScript from the relevant output
+  def apply(channel: ActorRef, tx: Transaction, outputIndex: Int, event: BitcoinEvent): WatchSpent = WatchSpent(channel, tx.txid, outputIndex, tx.txOut(outputIndex).publicKeyScript, event)
+}
+final case class WatchSpentBasic(channel: ActorRef, txId: BinaryData, outputIndex: Int, publicKeyScript: BinaryData, event: BitcoinEvent) extends Watch // we use this when we don't care about the spending tx, and we also assume txid already exists
+object WatchSpentBasic {
+  // if we have the entire transaction, we can get the publicKeyScript from the relevant output
+  def apply(channel: ActorRef, tx: Transaction, outputIndex: Int, event: BitcoinEvent): WatchSpentBasic = WatchSpentBasic(channel, tx.txid, outputIndex, tx.txOut(outputIndex).publicKeyScript, event)
+}
+// TODO: notify me if confirmation number gets below minDepth?
+final case class WatchLost(channel: ActorRef, txId: BinaryData, minDepth: Long, event: BitcoinEvent) extends Watch
+
+trait WatchEvent {
+  def event: BitcoinEvent
+}
+final case class WatchEventConfirmed(event: BitcoinEvent, blockHeight: Int, txIndex: Int) extends WatchEvent
+final case class WatchEventSpent(event: BitcoinEvent, tx: Transaction) extends WatchEvent
+final case class WatchEventSpentBasic(event: BitcoinEvent) extends WatchEvent
+final case class WatchEventLost(event: BitcoinEvent) extends WatchEvent
+
+/**
+  * Publish the provided tx as soon as possible depending on locktime and csv
+  */
+final case class PublishAsap(tx: Transaction)
+final case class ParallelGetRequest(ann: Seq[ChannelAnnouncement])
+final case class IndividualResult(c: ChannelAnnouncement, tx: Option[Transaction], unspent: Boolean)
+final case class ParallelGetResponse(r: Seq[IndividualResult])
+
+// @formatter:on

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoind/BitcoinCoreWallet.scala

@@ -0,0 +1,214 @@
+package fr.acinq.eclair.blockchain.bitcoind
+
+import akka.actor.{Actor, ActorRef, ActorSystem, Props}
+import fr.acinq.bitcoin.Crypto.PrivateKey
+import fr.acinq.bitcoin.{Base58Check, BinaryData, OP_PUSHDATA, OutPoint, SIGHASH_ALL, Satoshi, Script, ScriptFlags, ScriptWitness, SigVersion, Transaction, TxIn, TxOut}
+import fr.acinq.eclair.blockchain._
+import fr.acinq.eclair.blockchain.bitcoind.rpc.{BitcoinJsonRPCClient, JsonRPCError}
+import fr.acinq.eclair.channel.{BITCOIN_OUTPUT_SPENT, BITCOIN_TX_CONFIRMED}
+import fr.acinq.eclair.transactions.Transactions
+import grizzled.slf4j.Logging
+import org.json4s.JsonAST.{JBool, JDouble, JInt, JString}
+
+import scala.concurrent.duration._
+import scala.concurrent.{ExecutionContext, Future, Promise}
+
+/**
+  * Due to bitcoin-core wallet not fully supporting segwit txes yet, our current scheme is:
+  * utxos <- parent-tx <- funding-tx
+  *
+  * With:
+  *   - utxos may be non-segwit
+  *   - parent-tx pays to a p2wpkh segwit output
+  *   - funding-tx is a segwit tx
+  *
+  * Created by PM on 06/07/2017.
+  */
+class BitcoinCoreWallet(rpcClient: BitcoinJsonRPCClient, watcher: ActorRef)(implicit system: ActorSystem, ec: ExecutionContext) extends EclairWallet with Logging {
+
+  override def getBalance: Future[Satoshi] = ???
+
+  override def getFinalAddress: Future[String] = rpcClient.invoke("getnewaddress").map(json => {
+    val JString(address) = json
+    address
+  })
+
+  case class FundTransactionResponse(tx: Transaction, changepos: Int, fee: Double)
+
+  case class SignTransactionResponse(tx: Transaction, complete: Boolean)
+
+  case class MakeFundingTxResponseWithParent(parentTx: Transaction, fundingTx: Transaction, fundingTxOutputIndex: Int, priv: PrivateKey)
+
+  def fundTransaction(hex: String, lockUnspents: Boolean): Future[FundTransactionResponse] = {
+    rpcClient.invoke("fundrawtransaction", hex, BitcoinCoreWallet.Options(lockUnspents)).map(json => {
+      val JString(hex) = json \ "hex"
+      val JInt(changepos) = json \ "changepos"
+      val JDouble(fee) = json \ "fee"
+      FundTransactionResponse(Transaction.read(hex), changepos.intValue(), fee)
+    })
+  }
+
+  def fundTransaction(tx: Transaction, lockUnspents: Boolean): Future[FundTransactionResponse] =
+    fundTransaction(Transaction.write(tx).toString(), lockUnspents)
+
+  def signTransaction(hex: String): Future[SignTransactionResponse] =
+    rpcClient.invoke("signrawtransaction", hex).map(json => {
+      val JString(hex) = json \ "hex"
+      val JBool(complete) = json \ "complete"
+      SignTransactionResponse(Transaction.read(hex), complete)
+    })
+
+  def signTransaction(tx: Transaction): Future[SignTransactionResponse] =
+    signTransaction(Transaction.write(tx).toString())
+
+  def getTransaction(txid: BinaryData): Future[Transaction] = {
+    rpcClient.invoke("getrawtransaction", txid.toString()).map(json => {
+      val JString(hex) = json
+      Transaction.read(hex)
+    })
+  }
+
+  def publishTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[String] =
+    publishTransaction(Transaction.write(tx).toString())
+
+  def publishTransaction(hex: String)(implicit ec: ExecutionContext): Future[String] =
+    rpcClient.invoke("sendrawtransaction", hex) collect {
+      case JString(txid) => txid
+    }
+
+  /**
+    *
+    * @param fundingTxResponse a funding tx response
+    * @return an updated funding tx response that is properly sign
+    */
+  def sign(fundingTxResponse: MakeFundingTxResponseWithParent): MakeFundingTxResponseWithParent = {
+    // find the output that we are spending from
+    val utxo = fundingTxResponse.parentTx.txOut(fundingTxResponse.fundingTx.txIn(0).outPoint.index.toInt)
+
+    val pub = fundingTxResponse.priv.publicKey
+    val pubKeyScript = Script.pay2pkh(pub)
+    val sig = Transaction.signInput(fundingTxResponse.fundingTx, 0, pubKeyScript, SIGHASH_ALL, utxo.amount, SigVersion.SIGVERSION_WITNESS_V0, fundingTxResponse.priv)
+    val witness = ScriptWitness(Seq(sig, pub.toBin))
+    val fundingTx1 = fundingTxResponse.fundingTx.updateSigScript(0, OP_PUSHDATA(Script.write(Script.pay2wpkh(pub))) :: Nil).updateWitness(0, witness)
+
+    Transaction.correctlySpends(fundingTx1, fundingTxResponse.parentTx :: Nil, ScriptFlags.STANDARD_SCRIPT_VERIFY_FLAGS)
+    fundingTxResponse.copy(fundingTx = fundingTx1)
+  }
+
+  /**
+    *
+    * @param fundingTxResponse funding transaction response, which includes a funding tx, its parent, and the private key
+    *                          that we need to re-sign the funding
+    * @param newParentTx       new parent tx
+    * @return an updated funding transaction response where the funding tx now spends from newParentTx
+    */
+  def replaceParent(fundingTxResponse: MakeFundingTxResponseWithParent, newParentTx: Transaction): MakeFundingTxResponseWithParent = {
+    // find the output that we are spending from
+    val utxo = newParentTx.txOut(fundingTxResponse.fundingTx.txIn(0).outPoint.index.toInt)
+
+    // check that it matches what we expect, which is a P2WPKH output to our public key
+    require(utxo.publicKeyScript == Script.write(Script.pay2sh(Script.pay2wpkh(fundingTxResponse.priv.publicKey))))
+
+    // update our tx input we the hash of the new parent
+    val input = fundingTxResponse.fundingTx.txIn(0)
+    val input1 = input.copy(outPoint = input.outPoint.copy(hash = newParentTx.hash))
+    val unsignedFundingTx = fundingTxResponse.fundingTx.copy(txIn = Seq(input1))
+
+    // and re-sign it
+    sign(MakeFundingTxResponseWithParent(newParentTx, unsignedFundingTx, fundingTxResponse.fundingTxOutputIndex, fundingTxResponse.priv))
+  }
+
+  def makeParentAndFundingTx(pubkeyScript: BinaryData, amount: Satoshi, feeRatePerKw: Long): Future[MakeFundingTxResponseWithParent] =
+    for {
+      // ask for a new address and the corresponding private key
+      JString(address) <- rpcClient.invoke("getnewaddress")
+      JString(wif) <- rpcClient.invoke("dumpprivkey", address)
+      JString(segwitAddress) <- rpcClient.invoke("addwitnessaddress", address)
+      (prefix, raw) = Base58Check.decode(wif)
+      priv = PrivateKey(raw, compressed = true)
+      pub = priv.publicKey
+      // create a tx that sends money to a P2SH(WPKH) output that matches our private key
+      parentFee = Satoshi(250 * 2 * 2 * feeRatePerKw / 1024)
+      partialParentTx = Transaction(
+        version = 2,
+        txIn = Nil,
+        txOut = TxOut(amount + parentFee, Script.pay2sh(Script.pay2wpkh(pub))) :: Nil,
+        lockTime = 0L)
+      FundTransactionResponse(unsignedParentTx, _, _) <- fundTransaction(partialParentTx, lockUnspents = true)
+      // this is the first tx that we will publish, a standard tx which send money to our p2wpkh address
+      SignTransactionResponse(parentTx, true) <- signTransaction(unsignedParentTx)
+      // now we create the funding tx
+      partialFundingTx = Transaction(
+        version = 2,
+        txIn = Seq.empty[TxIn],
+        txOut = TxOut(amount, pubkeyScript) :: Nil,
+        lockTime = 0)
+      // and update it to spend from our segwit tx
+      pos = Transactions.findPubKeyScriptIndex(parentTx, Script.pay2sh(Script.pay2wpkh(pub)))
+      unsignedFundingTx = partialFundingTx.copy(txIn = TxIn(OutPoint(parentTx, pos), sequence = TxIn.SEQUENCE_FINAL, signatureScript = Nil) :: Nil)
+    } yield sign(MakeFundingTxResponseWithParent(parentTx, unsignedFundingTx, 0, priv))
+
+  /**
+    * This is a workaround for malleability
+    *
+    * @param pubkeyScript
+    * @param amount
+    * @param feeRatePerKw
+    * @return
+    */
+  override def makeFundingTx(pubkeyScript: BinaryData, amount: Satoshi, feeRatePerKw: Long): Future[MakeFundingTxResponse] = {
+    val promise = Promise[MakeFundingTxResponse]()
+    (for {
+      fundingTxResponse@MakeFundingTxResponseWithParent(parentTx, _, _, _) <- makeParentAndFundingTx(pubkeyScript, amount, feeRatePerKw)
+      input0 = parentTx.txIn.head
+      parentOfParentTx <- getTransaction(input0.outPoint.txid)
+      _ = logger.debug(s"built parentTxid=${parentTx.txid}, initializing temporary actor")
+      tempActor = system.actorOf(Props(new Actor {
+        override def receive: Receive = {
+          case WatchEventSpent(BITCOIN_OUTPUT_SPENT, spendingTx) =>
+            if (parentTx.txid != spendingTx.txid) {
+              // an input of our parent tx was spent by a tx that we're not aware of (i.e. a malleated version of our parent tx)
+              // set a new watch; if it is confirmed, we'll use it as the new parent for our funding tx
+              logger.warn(s"parent tx has been malleated: originalParentTxid=${parentTx.txid} malleated=${spendingTx.txid}")
+            }
+            watcher ! WatchConfirmed(self, spendingTx.txid, spendingTx.txOut(0).publicKeyScript, minDepth = 1, BITCOIN_TX_CONFIRMED(spendingTx))
+
+          case WatchEventConfirmed(BITCOIN_TX_CONFIRMED(tx), _, _) =>
+            // a potential parent for our funding tx has been confirmed, let's update our funding tx
+            val finalFundingTx = replaceParent(fundingTxResponse, tx)
+            promise.success(MakeFundingTxResponse(finalFundingTx.fundingTx, finalFundingTx.fundingTxOutputIndex))
+        }
+      }))
+      // we watch the first input of the parent tx, so that we can detect when it is spent by a malleated avatar
+      _ = watcher ! WatchSpent(tempActor, input0.outPoint.txid, input0.outPoint.index.toInt, parentOfParentTx.txOut(input0.outPoint.index.toInt).publicKeyScript, BITCOIN_OUTPUT_SPENT)
+      // and we publish the parent tx
+      _ = logger.info(s"publishing parent tx: txid=${parentTx.txid} tx=${Transaction.write(parentTx)}")
+      // we use a small delay so that we are sure Publish doesn't race with WatchSpent (which is ok but generates unnecessary warnings)
+      _ = system.scheduler.scheduleOnce(100 milliseconds, watcher, PublishAsap(parentTx))
+    } yield {}) onFailure {
+      case t: Throwable => promise.failure(t)
+    }
+    promise.future
+  }
+
+  override def commit(tx: Transaction): Future[Boolean] = publishTransaction(tx)
+    .map(_ => true) // if bitcoind says OK, then we consider the tx succesfully published
+    .recoverWith { case JsonRPCError(_) => getTransaction(tx.txid).map(_ => true).recover { case _ => false } } // if we get a parseable error from bitcoind AND the tx is NOT in the mempool/blockchain, then we consider that the tx was not published
+    .recover { case _ => true } // in all other cases we consider that the tx has been published
+
+
+  /**
+    * We currently only put a lock on the parent tx inputs, and we publish the parent tx immediately so there is nothing
+    * to do here.
+    *
+    * @param tx
+    * @return
+    */
+  override def rollback(tx: Transaction): Future[Boolean] = Future.successful(true)
+}
+
+object BitcoinCoreWallet {
+
+  case class Options(lockUnspents: Boolean)
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoind/ZmqWatcher.scala

@@ -1,13 +1,15 @@
-package fr.acinq.eclair.blockchain
+package fr.acinq.eclair.blockchain.bitcoind
 
 import java.util.concurrent.Executors
 
 import akka.actor.{Actor, ActorLogging, Cancellable, Props, Terminated}
 import akka.pattern.pipe
 import fr.acinq.bitcoin._
+import fr.acinq.eclair.Globals
+import fr.acinq.eclair.blockchain._
+import fr.acinq.eclair.blockchain.bitcoind.rpc.ExtendedBitcoinClient
 import fr.acinq.eclair.channel.BITCOIN_PARENT_TX_CONFIRMED
 import fr.acinq.eclair.transactions.Scripts
-import fr.acinq.eclair.{Globals, NodeParams, feerateKB2Kw}
 
 import scala.collection.SortedMap
 import scala.concurrent.duration._
@@ -20,10 +22,15 @@ import scala.util.Try
   * - also uses bitcoin-core rpc api, most notably for tx confirmation count and blockcount (because reorgs)
   * Created by PM on 21/02/2016.
   */
-class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implicit ec: ExecutionContext = ExecutionContext.global) extends Actor with ActorLogging {
+class ZmqWatcher(client: ExtendedBitcoinClient)(implicit ec: ExecutionContext = ExecutionContext.global) extends Actor with ActorLogging {
+
+  import ZmqWatcher.TickNewBlock
 
   context.system.eventStream.subscribe(self, classOf[BlockchainEvent])
 
+  // this is to initialize block count
+  self ! TickNewBlock
+
   case class TriggerEvent(w: Watch, e: WatchEvent)
 
   def receive: Receive = watching(Set(), SortedMap(), None)
@@ -33,39 +40,39 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
     case NewTransaction(tx) =>
       //log.debug(s"analyzing txid=${tx.txid} tx=${Transaction.write(tx)}")
       watches.collect {
-        case w@WatchSpentBasic(_, txid, outputIndex, event) if tx.txIn.exists(i => i.outPoint.txid == txid && i.outPoint.index == outputIndex) =>
+        case w@WatchSpentBasic(_, txid, outputIndex, _, event) if tx.txIn.exists(i => i.outPoint.txid == txid && i.outPoint.index == outputIndex) =>
           self ! TriggerEvent(w, WatchEventSpentBasic(event))
-        case w@WatchSpent(_, txid, outputIndex, event) if tx.txIn.exists(i => i.outPoint.txid == txid && i.outPoint.index == outputIndex) =>
+        case w@WatchSpent(_, txid, outputIndex, _, event) if tx.txIn.exists(i => i.outPoint.txid == txid && i.outPoint.index == outputIndex) =>
           self ! TriggerEvent(w, WatchEventSpent(event, tx))
       }
 
     case NewBlock(block) =>
       // using a Try because in tests we generate fake blocks
       log.debug(s"received blockid=${Try(block.blockId).getOrElse(BinaryData(""))}")
-      nextTick.map(_.cancel()) // this may fail or succeed, worse case scenario we will have two 'ticks in a row (no big deal)
+      nextTick.map(_.cancel()) // this may fail or succeed, worse case scenario we will have two ticks in a row (no big deal)
       log.debug(s"scheduling a new task to check on tx confirmations")
       // we do this to avoid herd effects in testing when generating a lots of blocks in a row
-      val task = context.system.scheduler.scheduleOnce(2 seconds, self, 'tick)
+      val task = context.system.scheduler.scheduleOnce(2 seconds, self, TickNewBlock)
       context become watching(watches, block2tx, Some(task))
 
-    case 'tick =>
+    case TickNewBlock =>
       client.getBlockCount.map {
         case count =>
           log.debug(s"setting blockCount=$count")
           Globals.blockCount.set(count)
           context.system.eventStream.publish(CurrentBlockCount(count))
       }
-      client.estimateSmartFee(nodeParams.smartfeeNBlocks).map {
+      /*client.estimateSmartFee(nodeParams.smartfeeNBlocks).map {
         case feeratePerKB if feeratePerKB > 0 =>
           val feeratePerKw = feerateKB2Kw(feeratePerKB)
           log.debug(s"setting feeratePerKB=$feeratePerKB -> feeratePerKw=$feeratePerKw")
           Globals.feeratePerKw.set(feeratePerKw)
           context.system.eventStream.publish(CurrentFeerate(feeratePerKw))
         case _ => () // bitcoind cannot estimate feerate
-      }
+      }*/
       // TODO: beware of the herd effect
       watches.collect {
-        case w@WatchConfirmed(_, txId, minDepth, event) =>
+        case w@WatchConfirmed(_, txId, _, minDepth, event) =>
           log.debug(s"checking confirmations of txid=$txId")
           client.getTxConfirmations(txId.toString).map {
             case Some(confirmations) if confirmations >= minDepth =>
@@ -99,10 +106,11 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
         require(tx.txIn.size == 1, s"watcher only supports tx with 1 input, this tx has ${tx.txIn.size} inputs")
         val parentTxid = tx.txIn(0).outPoint.txid
         log.info(s"txid=${tx.txid} has a relative timeout of $csvTimeout blocks, watching parenttxid=$parentTxid tx=${Transaction.write(tx)}")
-        self ! WatchConfirmed(self, parentTxid, minDepth = 1, BITCOIN_PARENT_TX_CONFIRMED(tx))
+        val parentPublicKey = fr.acinq.bitcoin.Script.write(fr.acinq.bitcoin.Script.pay2wsh(tx.txIn.head.witness.stack.last))
+        self ! WatchConfirmed(self, parentTxid, parentPublicKey, minDepth = 1, BITCOIN_PARENT_TX_CONFIRMED(tx))
       } else if (cltvTimeout > blockCount) {
         log.info(s"delaying publication of txid=${tx.txid} until block=$cltvTimeout (curblock=$blockCount)")
-        val block2tx1 = block2tx.updated(cltvTimeout, tx +: block2tx.getOrElse(cltvTimeout, Seq.empty[Transaction]))
+        val block2tx1 = block2tx.updated(cltvTimeout, block2tx.getOrElse(cltvTimeout, Seq.empty[Transaction]) :+ tx)
         context.become(watching(watches, block2tx1, None))
       } else publish(tx)
 
@@ -113,13 +121,10 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
       val absTimeout = blockHeight + csvTimeout
       if (absTimeout > blockCount) {
         log.info(s"delaying publication of txid=${tx.txid} until block=$absTimeout (curblock=$blockCount)")
-        val block2tx1 = block2tx.updated(absTimeout, tx +: block2tx.getOrElse(absTimeout, Seq.empty[Transaction]))
+        val block2tx1 = block2tx.updated(absTimeout, block2tx.getOrElse(absTimeout, Seq.empty[Transaction]) :+ tx)
         context.become(watching(watches, block2tx1, None))
       } else publish(tx)
 
-    case MakeFundingTx(ourCommitPub, theirCommitPub, amount, feeRatePerKw) =>
-      client.makeFundingTx(ourCommitPub, theirCommitPub, amount, feeRatePerKw).pipeTo(sender)
-
     case ParallelGetRequest(ann) => client.getParallel(ann).pipeTo(sender)
 
     case Terminated(channel) =>
@@ -133,7 +138,7 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
 
   def addWatch(w: Watch, watches: Set[Watch], block2tx: SortedMap[Long, Seq[Transaction]]) = {
     w match {
-      case WatchSpentBasic(_, txid, outputIndex, _) =>
+      case WatchSpentBasic(_, txid, outputIndex, _, _) =>
         // not: we assume parent tx was published, we just need to make sure this particular output has not been spent
         client.isTransactionOuputSpendable(txid.toString(), outputIndex, true).collect {
           case false =>
@@ -141,7 +146,7 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
             self ! TriggerEvent(w, WatchEventSpentBasic(w.event))
         }
 
-      case w@WatchSpent(_, txid, outputIndex, _) =>
+      case w@WatchSpent(_, txid, outputIndex, _, _) =>
         // first let's see if the parent tx was published or not
         client.getTxConfirmations(txid.toString()).collect {
           case Some(_) =>
@@ -156,6 +161,8 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
                       case txs =>
                         log.warning(s"found ${txs.size} txs since blockhash=$blockhash")
                         txs.foreach(tx => self ! NewTransaction(tx))
+                    } onFailure {
+                      case t: Throwable => log.error(t, "")
                     }
                 }
                 client.getMempool().map {
@@ -166,7 +173,7 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
             }
         }
 
-      case w: WatchConfirmed => self ! 'tick
+      case w: WatchConfirmed => self ! TickNewBlock
 
       case w => log.warning(s"ignoring $w (not implemented)")
     }
@@ -186,14 +193,17 @@ class PeerWatcher(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implici
         import akka.pattern.after
 
         import scala.concurrent.duration._
-        after(3 seconds, context.system.scheduler)(Future.successful(Unit)).map(x => publish(tx, isRetry = true))
+        after(3 seconds, context.system.scheduler)(Future.successful({})).map(x => publish(tx, isRetry = true))
       case t: Throwable => log.error(s"cannot publish tx: reason=${t.getMessage} txid=${tx.txid} tx=${BinaryData(Transaction.write(tx))}")
     }
   }
+
 }
 
-object PeerWatcher {
+object ZmqWatcher {
 
-  def props(nodeParams: NodeParams, client: ExtendedBitcoinClient)(implicit ec: ExecutionContext = ExecutionContext.global) = Props(new PeerWatcher(nodeParams, client)(ec))
+  def props(client: ExtendedBitcoinClient)(implicit ec: ExecutionContext = ExecutionContext.global) = Props(new ZmqWatcher(client)(ec))
+
+  case object TickNewBlock
 
 }

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoind/rpc/BitcoinJsonRPCClient.scala

@@ -1,4 +1,4 @@
-package fr.acinq.eclair.blockchain.rpc
+package fr.acinq.eclair.blockchain.bitcoind.rpc
 
 import java.io.IOException
 
@@ -8,12 +8,15 @@ import akka.http.scaladsl.marshalling.Marshal
 import akka.http.scaladsl.model._
 import akka.http.scaladsl.model.headers.{Authorization, BasicHttpCredentials}
 import akka.http.scaladsl.unmarshalling.Unmarshal
-import akka.stream.ActorMaterializer
+import akka.stream.{ActorMaterializer, OverflowStrategy, QueueOfferResult}
+import akka.stream.scaladsl.{Keep, Sink, Source}
 import de.heikoseeberger.akkahttpjson4s.Json4sSupport._
 import org.json4s.JsonAST.JValue
 import org.json4s.{DefaultFormats, jackson}
 
-import scala.concurrent.{ExecutionContext, Future}
+import scala.concurrent.{ExecutionContext, Future, Promise}
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.util.{Failure, Success}
 
 // @formatter:off
 case class JsonRPCRequest(jsonrpc: String = "1.0", id: String = "scala-client", method: String, params: Seq[Any])
@@ -26,16 +29,35 @@ class BitcoinJsonRPCClient(user: String, password: String, host: String = "127.0
 
   val scheme = if (ssl) "https" else "http"
   val uri = Uri(s"$scheme://$host:$port")
-
-  implicit val materializer = ActorMaterializer()
-  val httpClient = Http(system)
   implicit val serialization = jackson.Serialization
   implicit val formats = DefaultFormats
 
+  implicit val materializer = ActorMaterializer()
+  val httpClientFlow = Http().cachedHostConnectionPool[Promise[HttpResponse]](host, port)
+
+  val queueSize = 512
+  val queue = Source.queue[(HttpRequest, Promise[HttpResponse])](queueSize, OverflowStrategy.dropNew)
+      .via(httpClientFlow)
+      .toMat(Sink.foreach({
+        case ((Success(resp), p)) => p.success(resp)
+        case ((Failure(e), p))    => p.failure(e)
+      }))(Keep.left)
+      .run()
+
+  def queueRequest(request: HttpRequest): Future[HttpResponse] = {
+    val responsePromise = Promise[HttpResponse]()
+    queue.offer(request -> responsePromise).flatMap {
+      case QueueOfferResult.Enqueued    => responsePromise.future
+      case QueueOfferResult.Dropped     => Future.failed(new RuntimeException("Queue overflowed. Try again later."))
+      case QueueOfferResult.Failure(ex) => Future.failed(ex)
+      case QueueOfferResult.QueueClosed => Future.failed(new RuntimeException("Queue was closed (pool shut down) while running the request. Try again later."))
+    }
+  }
+
   def invoke(method: String, params: Any*)(implicit ec: ExecutionContext): Future[JValue] =
     for {
       entity <- Marshal(JsonRPCRequest(method = method, params = params)).to[RequestEntity]
-      httpRes <- httpClient.singleRequest(HttpRequest(uri = uri, method = HttpMethods.POST).addHeader(Authorization(BasicHttpCredentials(user, password))).withEntity(entity))
+      httpRes <- queueRequest(HttpRequest(uri = "/", method = HttpMethods.POST).addHeader(Authorization(BasicHttpCredentials(user, password))).withEntity(entity))
       jsonRpcRes <- Unmarshal(httpRes).to[JsonRPCResponse].map {
         case JsonRPCResponse(_, Some(error), _) => throw JsonRPCError(error)
         case o => o
@@ -47,7 +69,7 @@ class BitcoinJsonRPCClient(user: String, password: String, host: String = "127.0
   def invoke(request: Seq[(String, Seq[Any])])(implicit ec: ExecutionContext): Future[Seq[JValue]] =
     for {
       entity <- Marshal(request.map(r => JsonRPCRequest(method = r._1, params = r._2))).to[RequestEntity]
-      httpRes <- httpClient.singleRequest(HttpRequest(uri = uri, method = HttpMethods.POST).addHeader(Authorization(BasicHttpCredentials(user, password))).withEntity(entity))
+      httpRes <- queueRequest(HttpRequest(uri = "/", method = HttpMethods.POST).addHeader(Authorization(BasicHttpCredentials(user, password))).withEntity(entity))
       jsonRpcRes <- Unmarshal(httpRes).to[Seq[JsonRPCResponse]].map {
         //case JsonRPCResponse(_, Some(error), _) => throw JsonRPCError(error)
         case o => o

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoind/rpc/ExtendedBitcoinClient.scala

@@ -1,13 +1,9 @@
-package fr.acinq.eclair.blockchain
+package fr.acinq.eclair.blockchain.bitcoind.rpc
 
-import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
 import fr.acinq.bitcoin._
-import fr.acinq.eclair.blockchain.rpc.{BitcoinJsonRPCClient, JsonRPCError}
-import fr.acinq.eclair.channel.Helpers
+import fr.acinq.eclair.blockchain.{IndividualResult, ParallelGetResponse}
 import fr.acinq.eclair.fromShortId
-import fr.acinq.eclair.transactions.Transactions
 import fr.acinq.eclair.wire.ChannelAnnouncement
-import org.bouncycastle.util.encoders.Hex
 import org.json4s.JsonAST._
 
 import scala.concurrent.{ExecutionContext, Future}
@@ -16,28 +12,26 @@ import scala.util.Try
 /**
   * Created by PM on 26/04/2016.
   */
-class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
-
-  import ExtendedBitcoinClient._
+class ExtendedBitcoinClient(val rpcClient: BitcoinJsonRPCClient) {
 
   implicit val formats = org.json4s.DefaultFormats
 
   // TODO: this will probably not be needed once segwit is merged into core
   val protocolVersion = Protocol.PROTOCOL_VERSION
 
-  def tx2Hex(tx: Transaction): String = Hex.toHexString(Transaction.write(tx, protocolVersion))
+  def tx2Hex(tx: Transaction): String = toHexString(Transaction.write(tx, protocolVersion))
 
   def hex2tx(hex: String): Transaction = Transaction.read(hex, protocolVersion)
 
   def getTxConfirmations(txId: String)(implicit ec: ExecutionContext): Future[Option[Int]] =
-    client.invoke("getrawtransaction", txId, 1) // we choose verbose output to get the number of confirmations
+    rpcClient.invoke("getrawtransaction", txId, 1) // we choose verbose output to get the number of confirmations
       .map(json => Some((json \ "confirmations").extractOrElse[Int](0)))
       .recover {
         case t: JsonRPCError if t.error.code == -5 => None
       }
 
   def getTxBlockHash(txId: String)(implicit ec: ExecutionContext): Future[Option[String]] =
-    client.invoke("getrawtransaction", txId, 1) // we choose verbose output to get the number of confirmations
+    rpcClient.invoke("getrawtransaction", txId, 1) // we choose verbose output to get the number of confirmations
       .map(json => (json \ "blockhash").extractOpt[String])
       .recover {
         case t: JsonRPCError if t.error.code == -5 => None
@@ -45,7 +39,7 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
 
   def getBlockHashesSinceBlockHash(blockHash: String, previous: Seq[String] = Nil)(implicit ec: ExecutionContext): Future[Seq[String]] =
     for {
-      nextblockhash_opt <- client.invoke("getblock", blockHash).map(json => ((json \ "nextblockhash").extractOpt[String]))
+      nextblockhash_opt <- rpcClient.invoke("getblock", blockHash).map(json => ((json \ "nextblockhash").extractOpt[String]))
       res <- nextblockhash_opt match {
         case Some(nextBlockHash) => getBlockHashesSinceBlockHash(nextBlockHash, previous :+ nextBlockHash)
         case None => Future.successful(previous)
@@ -54,7 +48,7 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
 
   def getTxsSinceBlockHash(blockHash: String, previous: Seq[Transaction] = Nil)(implicit ec: ExecutionContext): Future[Seq[Transaction]] =
     for {
-      (nextblockhash_opt, txids) <- client.invoke("getblock", blockHash).map(json => ((json \ "nextblockhash").extractOpt[String], (json \ "tx").extract[List[String]]))
+      (nextblockhash_opt, txids) <- rpcClient.invoke("getblock", blockHash).map(json => ((json \ "nextblockhash").extractOpt[String], (json \ "tx").extract[List[String]]))
       next <- Future.sequence(txids.map(getTransaction(_)))
       res <- nextblockhash_opt match {
         case Some(nextBlockHash) => getTxsSinceBlockHash(nextBlockHash, previous ++ next)
@@ -64,7 +58,7 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
 
   def getMempool()(implicit ec: ExecutionContext): Future[Seq[Transaction]] =
     for {
-      txids <- client.invoke("getrawmempool").map(json => json.extract[List[String]])
+      txids <- rpcClient.invoke("getrawmempool").map(json => json.extract[List[String]])
       txs <- Future.sequence(txids.map(getTransaction(_)))
     } yield txs
 
@@ -79,7 +73,7 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
     * @return a Future[txid] where txid (a String) is the is of the tx that sends the bitcoins
     */
   def sendFromAccount(account: String, destination: String, amount: Double)(implicit ec: ExecutionContext): Future[String] =
-    client.invoke("sendfrom", account, destination, amount) collect {
+    rpcClient.invoke("sendfrom", account, destination, amount) collect {
       case JString(txid) => txid
     }
 
@@ -89,7 +83,7 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
     * @return
     */
   def getRawTransaction(txId: String)(implicit ec: ExecutionContext): Future[String] =
-    client.invoke("getrawtransaction", txId) collect {
+    rpcClient.invoke("getrawtransaction", txId) collect {
       case JString(raw) => raw
     }
 
@@ -98,8 +92,8 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
 
   def getTransaction(height: Int, index: Int)(implicit ec: ExecutionContext): Future[Transaction] =
     for {
-      hash <- client.invoke("getblockhash", height).map(json => json.extract[String])
-      json <- client.invoke("getblock", hash)
+      hash <- rpcClient.invoke("getblockhash", height).map(json => json.extract[String])
+      json <- rpcClient.invoke("getblock", hash)
       JArray(txs) = json \ "tx"
       txid = txs(index).extract[String]
       tx <- getTransaction(txid)
@@ -107,7 +101,7 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
 
   def isTransactionOuputSpendable(txId: String, ouputIndex: Int, includeMempool: Boolean)(implicit ec: ExecutionContext): Future[Boolean] =
     for {
-      json <- client.invoke("gettxout", txId, ouputIndex, includeMempool)
+      json <- rpcClient.invoke("gettxout", txId, ouputIndex, includeMempool)
     } yield json != JNull
 
 
@@ -121,7 +115,7 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
   def getTransactionShortId(txId: String)(implicit ec: ExecutionContext): Future[(Int, Int)] = {
     val future = for {
       Some(blockHash) <- getTxBlockHash(txId)
-      json <- client.invoke("getblock", blockHash)
+      json <- rpcClient.invoke("getblock", blockHash)
       JInt(height) = json \ "height"
       JString(hash) = json \ "hash"
       JArray(txs) = json \ "tx"
@@ -131,60 +125,14 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
     future
   }
 
-  def fundTransaction(hex: String)(implicit ec: ExecutionContext): Future[FundTransactionResponse] = {
-    client.invoke("fundrawtransaction", hex /*hex.take(4) + "0000" + hex.drop(4)*/).map(json => {
-      val JString(hex) = json \ "hex"
-      val JInt(changepos) = json \ "changepos"
-      val JDouble(fee) = json \ "fee"
-      FundTransactionResponse(Transaction.read(hex), changepos.intValue(), fee)
-    })
-  }
-
-  def fundTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[FundTransactionResponse] =
-    fundTransaction(tx2Hex(tx))
-
-  def signTransaction(hex: String)(implicit ec: ExecutionContext): Future[SignTransactionResponse] =
-    client.invoke("signrawtransaction", hex).map(json => {
-      val JString(hex) = json \ "hex"
-      val JBool(complete) = json \ "complete"
-      SignTransactionResponse(Transaction.read(hex), complete)
-    })
-
-  def signTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[SignTransactionResponse] =
-    signTransaction(tx2Hex(tx))
-
   def publishTransaction(hex: String)(implicit ec: ExecutionContext): Future[String] =
-    client.invoke("sendrawtransaction", hex) collect {
+    rpcClient.invoke("sendrawtransaction", hex) collect {
       case JString(txid) => txid
     }
 
   def publishTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[String] =
     publishTransaction(tx2Hex(tx))
 
-
-  def makeFundingTx(localFundingPubkey: PublicKey, remoteFundingPubkey: PublicKey, amount: Satoshi, feeRatePerKw: Long)(implicit ec: ExecutionContext): Future[MakeFundingTxResponse] =
-    for {
-    // ask for a new address and the corresponding private key
-      JString(address) <- client.invoke("getnewaddress")
-      JString(wif) <- client.invoke("dumpprivkey", address)
-      JString(segwitAddress) <- client.invoke("addwitnessaddress", address)
-      (prefix, raw) = Base58Check.decode(wif)
-      priv = PrivateKey(raw, compressed = true)
-      pub = priv.publicKey
-      // create a tx that sends money to a P2SH(WPKH) output that matches our private key
-      parentFee = Satoshi(250 * 2 * 2 * feeRatePerKw / 1024)
-      partialParentTx = Transaction(version = 2, txIn = Nil, txOut = TxOut(amount + parentFee, Script.pay2sh(Script.pay2wpkh(pub))) :: Nil, lockTime = 0L)
-      FundTransactionResponse(unsignedParentTx, _, _) <- fundTransaction(partialParentTx)
-      // this is the first tx that we will publish, a standard tx which send money to our p2wpkh address
-      SignTransactionResponse(parentTx, true) <- signTransaction(unsignedParentTx)
-      // now we create the funding tx
-      (partialFundingTx, _) = Transactions.makePartialFundingTx(amount, localFundingPubkey, remoteFundingPubkey)
-      // and update it to spend from our segwit tx
-      pos = Transactions.findPubKeyScriptIndex(parentTx, Script.pay2sh(Script.pay2wpkh(pub)))
-      unsignedFundingTx = partialFundingTx.copy(txIn = TxIn(OutPoint(parentTx, pos), sequence = TxIn.SEQUENCE_FINAL, signatureScript = Nil) :: Nil)
-    } yield Helpers.Funding.sign(MakeFundingTxResponse(parentTx, unsignedFundingTx, 0, priv))
-
-
   /**
     * We need this to compute absolute timeouts expressed in number of blocks (where getBlockCount would be equivalent
     * to time.now())
@@ -193,26 +141,10 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
     * @return the current number of blocks in the active chain
     */
   def getBlockCount(implicit ec: ExecutionContext): Future[Long] =
-    client.invoke("getblockcount") collect {
+    rpcClient.invoke("getblockcount") collect {
       case JInt(count) => count.toLong
     }
 
-  /**
-    * We need this to keep commitment tx fees in sync with the state of the network
-    *
-    * @param nBlocks number of blocks until tx is confirmed
-    * @param ec
-    * @return the current
-    */
-  def estimateSmartFee(nBlocks: Int)(implicit ec: ExecutionContext): Future[Long] =
-    client.invoke("estimatesmartfee", nBlocks).map(json => {
-      json \ "feerate" match {
-        case JDouble(feerate) => Btc(feerate).toLong
-        case JInt(feerate) if feerate.toLong < 0 => feerate.toLong
-        case JInt(feerate) => Btc(feerate.toLong).toLong
-      }
-    })
-
   def getParallel(awaiting: Seq[ChannelAnnouncement]): Future[ParallelGetResponse] = {
     case class TxCoordinate(blockHeight: Int, txIndex: Int, outputIndex: Int)
 
@@ -226,8 +158,8 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
     implicit val formats = org.json4s.DefaultFormats
 
     for {
-      blockHashes: Seq[String] <- client.invoke(coordinates.map(coord => ("getblockhash", coord._1.blockHeight :: Nil))).map(_.map(_.extractOrElse[String]("00" * 32)))
-      txids: Seq[String] <- client.invoke(blockHashes.map(h => ("getblock", h :: Nil)))
+      blockHashes: Seq[String] <- rpcClient.invoke(coordinates.map(coord => ("getblockhash", coord._1.blockHeight :: Nil))).map(_.map(_.extractOrElse[String]("00" * 32)))
+      txids: Seq[String] <- rpcClient.invoke(blockHashes.map(h => ("getblock", h :: Nil)))
         .map(_.zipWithIndex)
         .map(_.map {
           case (json, idx) => Try {
@@ -235,24 +167,16 @@ class ExtendedBitcoinClient(val client: BitcoinJsonRPCClient) {
             txs(coordinates(idx)._1.txIndex).extract[String]
           } getOrElse ("00" * 32)
         })
-      txs <- client.invoke(txids.map(txid => ("getrawtransaction", txid :: Nil))).map(_.map {
+      txs <- rpcClient.invoke(txids.map(txid => ("getrawtransaction", txid :: Nil))).map(_.map {
         case JString(raw) => Some(Transaction.read(raw))
         case _ => None
       })
-      unspent <- client.invoke(txids.zipWithIndex.map(txid => ("gettxout", txid._1 :: coordinates(txid._2)._1.outputIndex :: true :: Nil))).map(_.map(_ != JNull))
+      unspent <- rpcClient.invoke(txids.zipWithIndex.map(txid => ("gettxout", txid._1 :: coordinates(txid._2)._1.outputIndex :: true :: Nil))).map(_.map(_ != JNull))
     } yield ParallelGetResponse(awaiting.zip(txs.zip(unspent)).map(x => IndividualResult(x._1, x._2._1, x._2._2)))
 
   }
 }
 
-object ExtendedBitcoinClient {
-
-  case class FundTransactionResponse(tx: Transaction, changepos: Int, fee: Double)
-
-  case class SignTransactionResponse(tx: Transaction, complete: Boolean)
-
-}
-
 
 /*object Test extends App {
 

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoind/zmq/ZMQActor.scala

@@ -1,4 +1,4 @@
-package fr.acinq.eclair.blockchain.zmq
+package fr.acinq.eclair.blockchain.bitcoind.zmq
 
 import akka.actor.{Actor, ActorLogging}
 import fr.acinq.bitcoin.{Block, Transaction}
@@ -15,6 +15,8 @@ import scala.util.Try
   */
 class ZMQActor(address: String, connected: Option[Promise[Boolean]] = None) extends Actor with ActorLogging {
 
+  import ZMQActor._
+
   val ctx = new ZContext
 
   val subscriber = ctx.createSocket(ZMQ.SUB)
@@ -75,3 +77,13 @@ class ZMQActor(address: String, connected: Option[Promise[Boolean]] = None) exte
   }
 
 }
+
+object ZMQActor {
+
+  // @formatter:off
+  sealed trait ZMQEvent
+  case object ZMQConnected extends ZMQEvent
+  case object ZMQDisconnected extends ZMQEvent
+  // @formatter:on
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoinj/BitcoinjKit.scala

@@ -0,0 +1,152 @@
+package fr.acinq.eclair.blockchain.bitcoinj
+
+import java.io.File
+import java.net.InetSocketAddress
+
+import akka.actor.ActorSystem
+import com.google.common.util.concurrent.{FutureCallback, Futures}
+import fr.acinq.bitcoin.Transaction
+import fr.acinq.eclair.Globals
+import fr.acinq.eclair.blockchain.CurrentBlockCount
+import fr.acinq.eclair.blockchain.bitcoinj.BitcoinjKit._
+import grizzled.slf4j.Logging
+import org.bitcoinj.core.TransactionConfidence.ConfidenceType
+import org.bitcoinj.core.listeners._
+import org.bitcoinj.core.{Block, Context, FilteredBlock, NetworkParameters, Peer, PeerAddress, StoredBlock, VersionMessage, Transaction => BitcoinjTransaction}
+import org.bitcoinj.kits.WalletAppKit
+import org.bitcoinj.params.{RegTestParams, TestNet3Params}
+import org.bitcoinj.utils.Threading
+import org.bitcoinj.wallet.Wallet
+
+import scala.collection.JavaConversions._
+import scala.concurrent.Promise
+import scala.util.Try
+
+/**
+  * Created by PM on 09/07/2017.
+  */
+class BitcoinjKit(chain: String, datadir: File, staticPeers: List[InetSocketAddress] = Nil)(implicit system: ActorSystem) extends WalletAppKit(chain2Params(chain), datadir, "bitcoinj", true) with Logging {
+
+  if (staticPeers.size > 0) {
+    logger.info(s"using staticPeers=${staticPeers.mkString(",")}")
+    setPeerNodes(staticPeers.map(addr => new PeerAddress(params, addr)).head)
+  }
+
+  // tells us when the peerGroup/chain/wallet are accessible
+  private val initializedPromise = Promise[Boolean]()
+  val initialized = initializedPromise.future
+
+  // tells us as soon as we know the current block height
+  private val atCurrentHeightPromise = Promise[Boolean]()
+  val atCurrentHeight = atCurrentHeightPromise.future
+
+  // tells us when we are at current block height
+  //  private val syncedPromise = Promise[Boolean]()
+  //  val synced = syncedPromise.future
+
+  private def updateBlockCount(blockCount: Int) = {
+    // when synchronizing we don't want to advertise previous blocks
+    if (Globals.blockCount.get() < blockCount) {
+      logger.debug(s"current blockchain height=$blockCount")
+      system.eventStream.publish(CurrentBlockCount(blockCount))
+      Globals.blockCount.set(blockCount)
+    }
+  }
+
+  override def onSetupCompleted(): Unit = {
+
+    logger.info(s"peerGroup.getMinBroadcastConnections==${peerGroup().getMinBroadcastConnections}")
+    logger.info(s"peerGroup.getMinBroadcastConnections==${peerGroup().getMinBroadcastConnections}")
+
+    peerGroup().setMinRequiredProtocolVersion(70015) // bitcoin core 0.13
+    wallet().watchMode = true
+
+    //    setDownloadListener(new DownloadProgressTracker {
+    //      override def doneDownload(): Unit = {
+    //        super.doneDownload()
+    //        // may be called multiple times
+    //        syncedPromise.trySuccess(true)
+    //      }
+    //    })
+
+    // we set the blockcount to the previous stored block height
+    updateBlockCount(chain().getBestChainHeight)
+
+    // as soon as we are connected the peers will tell us their current height and we will advertise it immediately
+    peerGroup().addConnectedEventListener(new PeerConnectedEventListener {
+      override def onPeerConnected(peer: Peer, peerCount: Int): Unit = {
+        if ((peer.getPeerVersionMessage.localServices & VersionMessage.NODE_WITNESS) == 0) {
+          peer.close()
+        } else {
+          Context.propagate(wallet.getContext)
+          // we wait for at least 3 peers before relying on the information they are giving, but we trust localhost
+          if (peer.getAddress.getAddr.isLoopbackAddress || peerCount > 3) {
+            updateBlockCount(peerGroup().getMostCommonChainHeight)
+            // may be called multiple times
+            atCurrentHeightPromise.trySuccess(true)
+          }
+        }
+      }
+    })
+
+    peerGroup.addBlocksDownloadedEventListener(new BlocksDownloadedEventListener {
+      override def onBlocksDownloaded(peer: Peer, block: Block, filteredBlock: FilteredBlock, blocksLeft: Int): Unit = {
+        Context.propagate(wallet.getContext)
+        logger.debug(s"received block=${block.getHashAsString} (size=${block.bitcoinSerialize().size} txs=${Try(block.getTransactions.size).getOrElse(-1)}) filteredBlock=${Try(filteredBlock.getHash.toString).getOrElse("N/A")} (size=${Try(block.bitcoinSerialize().size).getOrElse(-1)} txs=${Try(filteredBlock.getTransactionCount).getOrElse(-1)})")
+        Try {
+          if (filteredBlock.getAssociatedTransactions.size() > 0) {
+            logger.info(s"retrieving full block ${block.getHashAsString}")
+            Futures.addCallback(peer.getBlock(block.getHash), new FutureCallback[Block] {
+              override def onFailure(throwable: Throwable) = logger.error(s"could not retrieve full block=${block.getHashAsString}")
+
+              override def onSuccess(fullBlock: Block) = {
+                Try {
+                  Context.propagate(wallet.getContext)
+                  fullBlock.getTransactions.foreach {
+                    case tx =>
+                      logger.debug(s"received tx=${tx.getHashAsString} witness=${Transaction.read(tx.bitcoinSerialize()).txIn(0).witness.stack.size} from fullBlock=${fullBlock.getHash} confidence=${tx.getConfidence}")
+                      val depthInBlocks = tx.getConfidence.getConfidenceType match {
+                        case ConfidenceType.DEAD => -1
+                        case _ => tx.getConfidence.getDepthInBlocks
+                      }
+                      system.eventStream.publish(NewConfidenceLevel(Transaction.read(tx.bitcoinSerialize()), 0, depthInBlocks))
+                  }
+                }
+              }
+            }, Threading.USER_THREAD)
+          }
+        }
+      }
+    })
+
+    chain().addNewBestBlockListener(new NewBestBlockListener {
+      override def notifyNewBestBlock(storedBlock: StoredBlock): Unit =
+        updateBlockCount(storedBlock.getHeight)
+    })
+
+    wallet().addTransactionConfidenceEventListener(new TransactionConfidenceEventListener {
+      override def onTransactionConfidenceChanged(wallet: Wallet, bitcoinjTx: BitcoinjTransaction): Unit = {
+        Context.propagate(wallet.getContext)
+        val tx = Transaction.read(bitcoinjTx.bitcoinSerialize())
+        logger.info(s"tx confidence changed for txid=${tx.txid} confidence=${bitcoinjTx.getConfidence} witness=${bitcoinjTx.getWitness(0)}")
+        val (blockHeight, confirmations) = bitcoinjTx.getConfidence.getConfidenceType match {
+          case ConfidenceType.DEAD => (-1, -1)
+          case ConfidenceType.BUILDING => (bitcoinjTx.getConfidence.getAppearedAtChainHeight, bitcoinjTx.getConfidence.getDepthInBlocks)
+          case _ => (-1, bitcoinjTx.getConfidence.getDepthInBlocks)
+        }
+        system.eventStream.publish(NewConfidenceLevel(tx, blockHeight, confirmations))
+      }
+    })
+
+    initializedPromise.success(true)
+  }
+
+}
+
+object BitcoinjKit {
+
+  def chain2Params(chain: String): NetworkParameters = chain match {
+    case "regtest" => RegTestParams.get()
+    case "test" => TestNet3Params.get()
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoinj/BitcoinjWallet.scala

@@ -0,0 +1,68 @@
+package fr.acinq.eclair.blockchain.bitcoinj
+
+import fr.acinq.bitcoin.{BinaryData, Satoshi, Transaction}
+import fr.acinq.eclair.blockchain.{EclairWallet, MakeFundingTxResponse}
+import grizzled.slf4j.Logging
+import org.bitcoinj.core.{Coin, Context, Transaction => BitcoinjTransaction}
+import org.bitcoinj.script.Script
+import org.bitcoinj.wallet.{SendRequest, Wallet}
+
+import scala.collection.JavaConversions._
+import scala.concurrent.{ExecutionContext, Future}
+
+/**
+  * Created by PM on 08/07/2017.
+  */
+class BitcoinjWallet(val fWallet: Future[Wallet])(implicit ec: ExecutionContext) extends EclairWallet with Logging {
+
+  fWallet.map(wallet => wallet.allowSpendingUnconfirmedTransactions())
+
+  override def getBalance: Future[Satoshi] = for {
+    wallet <- fWallet
+  } yield {
+    Context.propagate(wallet.getContext)
+    Satoshi(wallet.getBalance.longValue())
+  }
+
+  override def getFinalAddress: Future[String] = for {
+    wallet <- fWallet
+  } yield {
+    Context.propagate(wallet.getContext)
+    wallet.currentReceiveAddress().toBase58
+  }
+
+  override def makeFundingTx(pubkeyScript: BinaryData, amount: Satoshi, feeRatePerKw: Long): Future[MakeFundingTxResponse] = for {
+    wallet <- fWallet
+  } yield {
+    logger.info(s"building funding tx")
+    Context.propagate(wallet.getContext)
+    val script = new Script(pubkeyScript)
+    val tx = new BitcoinjTransaction(wallet.getParams)
+    tx.addOutput(Coin.valueOf(amount.amount), script)
+    val req = SendRequest.forTx(tx)
+    wallet.completeTx(req)
+    val txOutputIndex = tx.getOutputs.find(_.getScriptPubKey.equals(script)).get.getIndex
+    MakeFundingTxResponse(Transaction.read(tx.bitcoinSerialize()), txOutputIndex)
+  }
+
+  override def commit(tx: Transaction): Future[Boolean] = {
+    // we make sure that we haven't double spent our own tx (eg by opening 2 channels at the same time)
+    val serializedTx = Transaction.write(tx)
+    logger.info(s"committing tx: txid=${tx.txid} tx=$serializedTx")
+    for {
+      wallet <- fWallet
+      _ = Context.propagate(wallet.getContext)
+      bitcoinjTx = new org.bitcoinj.core.Transaction(wallet.getParams(), serializedTx)
+      canCommit = wallet.maybeCommitTx(bitcoinjTx)
+      _ = logger.info(s"commit txid=${tx.txid} result=$canCommit")
+    } yield canCommit
+  }
+
+  /**
+    * There are no locks on bitcoinj, this is a no-op
+    *
+    * @param tx
+    * @return
+    */
+  override def rollback(tx: Transaction) = Future.successful(true)
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/bitcoinj/BitcoinjWatcher.scala

@@ -0,0 +1,193 @@
+package fr.acinq.eclair.blockchain.bitcoinj
+
+import akka.actor.{Actor, ActorLogging, Props, Terminated}
+import com.google.common.collect.ImmutableList
+import com.google.common.util.concurrent.{FutureCallback, Futures}
+import fr.acinq.bitcoin.Crypto.PublicKey
+import fr.acinq.bitcoin.Script.{pay2wsh, write}
+import fr.acinq.bitcoin.{BinaryData, Satoshi, Transaction, TxIn, TxOut}
+import fr.acinq.eclair.blockchain._
+import fr.acinq.eclair.channel.BITCOIN_PARENT_TX_CONFIRMED
+import fr.acinq.eclair.transactions.Scripts
+import fr.acinq.eclair.{Globals, fromShortId}
+import org.bitcoinj.core.{Context, Transaction => BitcoinjTransaction}
+import org.bitcoinj.kits.WalletAppKit
+import org.bitcoinj.script.Script
+
+import scala.collection.SortedMap
+import scala.concurrent.ExecutionContext
+import scala.util.{Failure, Success, Try}
+
+final case class NewConfidenceLevel(tx: Transaction, blockHeight: Int, confirmations: Int) extends BlockchainEvent
+
+/**
+  * A blockchain watcher that:
+  * - receives bitcoin events (new blocks and new txes) directly from the bitcoin network
+  * - also uses bitcoin-core rpc api, most notably for tx confirmation count and blockcount (because reorgs)
+  * Created by PM on 21/02/2016.
+  */
+class BitcoinjWatcher(val kit: WalletAppKit)(implicit ec: ExecutionContext = ExecutionContext.global) extends Actor with ActorLogging {
+
+  context.system.eventStream.subscribe(self, classOf[BlockchainEvent])
+  context.system.eventStream.subscribe(self, classOf[NewConfidenceLevel])
+
+  val broadcaster = context.actorOf(Props(new Broadcaster(kit: WalletAppKit)), name = "broadcaster")
+
+  case class TriggerEvent(w: Watch, e: WatchEvent)
+
+  def receive: Receive = watching(Set(), SortedMap(), Nil, Nil)
+
+  def watching(watches: Set[Watch], block2tx: SortedMap[Long, Seq[Transaction]], oldEvents: Seq[NewConfidenceLevel], sent: Seq[TriggerEvent]): Receive = {
+
+    case event@NewConfidenceLevel(tx, blockHeight, confirmations) =>
+      log.debug(s"analyzing txid=${tx.txid} confirmations=$confirmations tx=${Transaction.write(tx)}")
+      watches.collect {
+        case w@WatchSpentBasic(_, txid, outputIndex, _, event) if tx.txIn.exists(i => i.outPoint.txid == txid && i.outPoint.index == outputIndex) =>
+          self ! TriggerEvent(w, WatchEventSpentBasic(event))
+        case w@WatchSpent(_, txid, outputIndex, _, event) if tx.txIn.exists(i => i.outPoint.txid == txid && i.outPoint.index == outputIndex) =>
+          self ! TriggerEvent(w, WatchEventSpent(event, tx))
+        case w@WatchConfirmed(_, txId, _, minDepth, event) if txId == tx.txid && confirmations >= minDepth =>
+          self ! TriggerEvent(w, WatchEventConfirmed(event, blockHeight, 0))
+      }
+      context become watching(watches, block2tx, oldEvents.filterNot(_.tx.txid == tx.txid) :+ event, sent)
+
+    case t@TriggerEvent(w, e) if watches.contains(w) && !sent.contains(t) =>
+      log.info(s"triggering $w")
+      w.channel ! e
+      // NB: WatchSpent are permanent because we need to detect multiple spending of the funding tx
+      // They are never cleaned up but it is not a big deal for now (1 channel == 1 watch)
+      val newWatches = if (!w.isInstanceOf[WatchSpent]) watches - w else watches
+      context.become(watching(newWatches, block2tx, oldEvents, sent :+ t))
+
+    case CurrentBlockCount(count) => {
+      val toPublish = block2tx.filterKeys(_ <= count)
+      toPublish.values.flatten.map(tx => publish(tx))
+      context.become(watching(watches, block2tx -- toPublish.keys, oldEvents, sent))
+    }
+
+    case w: Watch if !watches.contains(w) =>
+      w match {
+        case w: WatchConfirmed => addHint(w.publicKeyScript)
+        case w: WatchSpent => addHint(w.publicKeyScript)
+        case w: WatchSpentBasic => addHint(w.publicKeyScript)
+        case _ => ()
+      }
+      log.debug(s"adding watch $w for $sender")
+      log.info(s"resending ${oldEvents.size} events!")
+      oldEvents.foreach(self ! _)
+      context.watch(w.channel)
+      context.become(watching(watches + w, block2tx, oldEvents, sent))
+
+    case PublishAsap(tx) =>
+      val blockCount = Globals.blockCount.get()
+      val cltvTimeout = Scripts.cltvTimeout(tx)
+      val csvTimeout = Scripts.csvTimeout(tx)
+      if (csvTimeout > 0) {
+        require(tx.txIn.size == 1, s"watcher only supports tx with 1 input, this tx has ${tx.txIn.size} inputs")
+        val parentTxid = tx.txIn(0).outPoint.txid
+        log.info(s"txid=${tx.txid} has a relative timeout of $csvTimeout blocks, watching parenttxid=$parentTxid tx=${Transaction.write(tx)}")
+        val parentPublicKey = fr.acinq.bitcoin.Script.write(fr.acinq.bitcoin.Script.pay2wsh(tx.txIn.head.witness.stack.last))
+        self ! WatchConfirmed(self, parentTxid, parentPublicKey, minDepth = 1, BITCOIN_PARENT_TX_CONFIRMED(tx))
+      } else if (cltvTimeout > blockCount) {
+        log.info(s"delaying publication of txid=${tx.txid} until block=$cltvTimeout (curblock=$blockCount)")
+        val block2tx1 = block2tx.updated(cltvTimeout, block2tx.getOrElse(cltvTimeout, Seq.empty[Transaction]) :+ tx)
+        context.become(watching(watches, block2tx1, oldEvents, sent))
+      } else publish(tx)
+
+    case WatchEventConfirmed(BITCOIN_PARENT_TX_CONFIRMED(tx), blockHeight, _) =>
+      log.info(s"parent tx of txid=${tx.txid} has been confirmed")
+      val blockCount = Globals.blockCount.get()
+      val csvTimeout = Scripts.csvTimeout(tx)
+      val absTimeout = blockHeight + csvTimeout
+      if (absTimeout > blockCount) {
+        log.info(s"delaying publication of txid=${tx.txid} until block=$absTimeout (curblock=$blockCount)")
+        val block2tx1 = block2tx.updated(absTimeout, block2tx.getOrElse(absTimeout, Seq.empty[Transaction]) :+ tx)
+        context.become(watching(watches, block2tx1, oldEvents, sent))
+      } else publish(tx)
+
+    case ParallelGetRequest(announcements) => sender ! ParallelGetResponse(announcements.map {
+      case c =>
+        log.info(s"blindly validating channel=$c")
+        val pubkeyScript = write(pay2wsh(Scripts.multiSig2of2(PublicKey(c.bitcoinKey1), PublicKey(c.bitcoinKey2))))
+        val (_, _, outputIndex) = fromShortId(c.shortChannelId)
+        val fakeFundingTx = Transaction(
+          version = 2,
+          txIn = Seq.empty[TxIn],
+          txOut = List.fill(outputIndex + 1)(TxOut(Satoshi(0), pubkeyScript)), // quick and dirty way to be sure that the outputIndex'th output is of the expected format
+          lockTime = 0)
+        IndividualResult(c, Some(fakeFundingTx), true)
+    })
+
+    case Terminated(channel) =>
+      // we remove watches associated to dead actor
+      val deprecatedWatches = watches.filter(_.channel == channel)
+      context.become(watching(watches -- deprecatedWatches, block2tx, oldEvents, sent))
+
+    case 'watches => sender ! watches
+
+  }
+
+  /**
+    * Bitcoinj needs hints to be able to detect transactions
+    *
+    * @param pubkeyScript
+    * @return
+    */
+  def addHint(pubkeyScript: BinaryData) = {
+    Context.propagate(kit.wallet.getContext)
+    val script = new Script(pubkeyScript)
+    // set creation time to 2017/09/01, so bitcoinj can still use its checkpoints optimizations
+    script.setCreationTimeSeconds(1501538400L) // 2017-09-01
+    kit.wallet().addWatchedScripts(ImmutableList.of(script))
+  }
+
+  def publish(tx: Transaction): Unit = broadcaster ! tx
+
+}
+
+object BitcoinjWatcher {
+
+  def props(kit: WalletAppKit)(implicit ec: ExecutionContext = ExecutionContext.global) = Props(new BitcoinjWatcher(kit)(ec))
+
+}
+
+class Broadcaster(kit: WalletAppKit) extends Actor with ActorLogging {
+
+  override def receive: Receive = {
+    case tx: Transaction =>
+      broadcast(tx)
+      context become waiting(Nil)
+  }
+
+  def waiting(stash: Seq[Transaction]): Receive = {
+    case BroadcastResult(tx, result) =>
+      result match {
+        case Success(_) => log.info(s"broadcast success for txid=${tx.txid}")
+        case Failure(t) => log.error(t, s"broadcast failure for txid=${tx.txid}: ")
+      }
+      stash match {
+        case head :: rest =>
+          broadcast(head)
+          context become waiting(rest)
+        case Nil => context become receive
+      }
+    case tx: Transaction =>
+      log.info(s"stashing txid=${tx.txid} for broadcast")
+      context become waiting(stash :+ tx)
+  }
+
+  case class BroadcastResult(tx: Transaction, result: Try[Boolean])
+
+  def broadcast(tx: Transaction) = {
+    Context.propagate(kit.wallet().getContext)
+    val bitcoinjTx = new org.bitcoinj.core.Transaction(kit.wallet().getParams, Transaction.write(tx))
+    log.info(s"broadcasting txid=${tx.txid}")
+    Futures.addCallback(kit.peerGroup().broadcastTransaction(bitcoinjTx).future(), new FutureCallback[BitcoinjTransaction] {
+      override def onFailure(t: Throwable): Unit = self ! BroadcastResult(tx, Failure(t))
+
+      override def onSuccess(v: BitcoinjTransaction): Unit = self ! BroadcastResult(tx, Success(true))
+    }, context.dispatcher)
+  }
+
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/electrum/ElectrumClient.scala

@@ -0,0 +1,488 @@
+package fr.acinq.eclair.blockchain.electrum
+
+import java.io.InputStream
+import java.net.InetSocketAddress
+
+import akka.actor.{Actor, ActorLogging, ActorRef, Stash, Terminated}
+import akka.io.{IO, Tcp}
+import akka.util.ByteString
+import fr.acinq.bitcoin._
+import fr.acinq.eclair.Globals
+import fr.acinq.eclair.blockchain.CurrentBlockCount
+import fr.acinq.eclair.blockchain.bitcoind.rpc.{Error, JsonRPCRequest, JsonRPCResponse}
+import org.json4s.JsonAST._
+import org.json4s.jackson.JsonMethods
+import org.json4s.{DefaultFormats, JInt, JLong, JString}
+import org.spongycastle.util.encoders.Hex
+
+import scala.annotation.tailrec
+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.duration._
+import scala.util.Random
+
+class ElectrumClient(serverAddresses: Seq[InetSocketAddress]) extends Actor with Stash with ActorLogging {
+
+  import ElectrumClient._
+  import context.system
+
+  implicit val formats = DefaultFormats
+
+  val newline = "\n"
+  val connectionFailures = collection.mutable.HashMap.empty[InetSocketAddress, Long]
+
+  val version = ServerVersion("2.1.7", "1.1")
+  // we need to regularly send a ping in order not to get disconnected
+  context.system.scheduler.schedule(30 seconds, 30 seconds, self, version)
+
+  override def unhandled(message: Any): Unit = {
+    message match {
+      case _: Tcp.ConnectionClosed =>
+        val nextAddress = nextPeer()
+        log.warning(s"connection failed, trying $nextAddress")
+        self ! Tcp.Connect(nextAddress)
+        statusListeners.map(_ ! ElectrumDisconnected)
+        context.system.eventStream.publish(ElectrumDisconnected)
+        context become disconnected
+
+      case Terminated(deadActor) =>
+        val removeMe = addressSubscriptions collect {
+          case (address, actor) if actor == deadActor => address
+        }
+        addressSubscriptions --= removeMe
+
+        val removeMe1 = scriptHashSubscriptions collect {
+          case (scriptHash, actor) if actor == deadActor => scriptHash
+        }
+        scriptHashSubscriptions --= removeMe1
+        statusListeners -= deadActor
+        headerSubscriptions -= deadActor
+
+      case _: ServerVersion => () // we only handle this when connected
+
+      case _: ServerVersionResponse => () // we just ignore these messages, they are used as pings
+
+      case _ => log.warning(s"unhandled $message")
+    }
+  }
+
+  val statusListeners = collection.mutable.HashSet.empty[ActorRef]
+
+  def send(connection: ActorRef, request: JsonRPCRequest): Unit = {
+    import org.json4s.JsonDSL._
+    import org.json4s._
+    import org.json4s.jackson.JsonMethods._
+
+    log.debug(s"sending $request")
+    val json = ("method" -> request.method) ~ ("params" -> request.params.map {
+      case s: String => new JString(s)
+      case b: BinaryData => new JString(b.toString())
+      case t: Int => new JInt(t)
+      case t: Long => new JLong(t)
+      case t: Double => new JDouble(t)
+    }) ~ ("id" -> request.id) ~ ("jsonrpc" -> request.jsonrpc)
+    val serialized = compact(render(json))
+    val bytes = (serialized + newline).getBytes
+    connection ! Tcp.Write(ByteString.fromArray(bytes))
+  }
+
+  private def nextPeer() = {
+    val nextPos = Random.nextInt(serverAddresses.size)
+    serverAddresses(nextPos)
+  }
+
+  private def updateBlockCount(blockCount: Long) = {
+    // when synchronizing we don't want to advertise previous blocks
+    if (Globals.blockCount.get() < blockCount) {
+      log.debug(s"current blockchain height=$blockCount")
+      system.eventStream.publish(CurrentBlockCount(blockCount))
+      Globals.blockCount.set(blockCount)
+    }
+  }
+
+  val addressSubscriptions = collection.mutable.HashMap.empty[String, Set[ActorRef]]
+  val scriptHashSubscriptions = collection.mutable.HashMap.empty[BinaryData, Set[ActorRef]]
+  val headerSubscriptions = collection.mutable.HashSet.empty[ActorRef]
+
+  context.system.eventStream.publish(ElectrumDisconnected)
+  self ! Tcp.Connect(serverAddresses.head)
+
+  var reqId = 0L
+
+  def receive = disconnected
+
+  def disconnected: Receive = {
+    case c: Tcp.Connect =>
+      log.info(s"connecting to $c")
+      IO(Tcp) ! c
+
+    case Tcp.Connected(remote, _) =>
+      log.info(s"connected to $remote")
+      connectionFailures.clear()
+      val connection = sender()
+      connection ! Tcp.Register(self)
+      val request = version
+      send(connection, makeRequest(request, "" + reqId))
+      reqId = reqId + 1
+      context become waitingForVersion(connection, remote)
+
+    case AddStatusListener(actor) => statusListeners += actor
+
+    case Tcp.CommandFailed(Tcp.Connect(remoteAddress, _, _, _, _)) =>
+      val nextAddress = nextPeer()
+      log.warning(s"connection to $remoteAddress failed, trying $nextAddress")
+      connectionFailures.put(remoteAddress, connectionFailures.getOrElse(remoteAddress, 0L) + 1L)
+      val count = connectionFailures.getOrElse(nextAddress, 0L)
+      val delay = Math.min(Math.pow(2.0, count), 60.0) seconds;
+      context.system.scheduler.scheduleOnce(delay, self, Tcp.Connect(nextAddress))
+  }
+
+  def waitingForVersion(connection: ActorRef, remote: InetSocketAddress): Receive = {
+    case Tcp.Received(data) =>
+      val response = parseResponse(new String(data.toArray)).right.get
+      val serverVersion = parseJsonResponse(version, response)
+      log.debug(s"serverVersion=$serverVersion")
+      val request = HeaderSubscription(self)
+      send(connection, makeRequest(request, "" + reqId))
+      headerSubscriptions += self
+      log.debug("waiting for tip")
+      reqId = reqId + 1
+      context become waitingForTip(connection, remote: InetSocketAddress)
+
+    case AddStatusListener(actor) => statusListeners += actor
+  }
+
+  def waitingForTip(connection: ActorRef, remote: InetSocketAddress): Receive = {
+    case Tcp.Received(data) =>
+      val response = parseResponse(new String(data.toArray)).right.get
+      val header = parseHeader(response.result)
+      log.debug(s"connected, tip = ${header.block_hash} $header")
+      updateBlockCount(header.block_height)
+      statusListeners.map(_ ! ElectrumReady)
+      context.system.eventStream.publish(ElectrumConnected)
+      context become connected(connection, remote, header, "", Map.empty)
+
+    case AddStatusListener(actor) => statusListeners += actor
+  }
+
+  def connected(connection: ActorRef, remoteAddress: InetSocketAddress, tip: Header, buffer: String, requests: Map[String, (Request, ActorRef)]): Receive = {
+    case AddStatusListener(actor) =>
+      statusListeners += actor
+      actor ! ElectrumReady
+
+    case HeaderSubscription(actor) =>
+      headerSubscriptions += actor
+      actor ! HeaderSubscriptionResponse(tip)
+      context watch actor
+
+    case request: Request =>
+      val curReqId = "" + reqId
+      send(connection, makeRequest(request, curReqId))
+      request match {
+        case AddressSubscription(address, actor) =>
+          addressSubscriptions.update(address, addressSubscriptions.getOrElse(address, Set()) + actor)
+          context watch actor
+        case ScriptHashSubscription(scriptHash, actor) =>
+          scriptHashSubscriptions.update(scriptHash, scriptHashSubscriptions.getOrElse(scriptHash, Set()) + actor)
+          context watch actor
+        case _ => ()
+      }
+      reqId = reqId + 1
+      context become connected(connection, remoteAddress, tip, buffer, requests + (curReqId -> (request, sender())))
+
+    case Tcp.Received(data) =>
+      val buffer1 = buffer + new String(data.toArray)
+      val (jsons, buffer2) = buffer1.split(newline) match {
+        case chunks if buffer1.endsWith(newline) => (chunks, "")
+        case chunks => (chunks.dropRight(1), chunks.last)
+      }
+      jsons.map(parseResponse(_)).map(self ! _)
+      context become connected(connection, remoteAddress, tip, buffer2, requests)
+
+    case Right(json: JsonRPCResponse) =>
+      requests.get(json.id) match {
+        case Some((request, requestor)) =>
+          val response = parseJsonResponse(request, json)
+          log.debug(s"got response for reqId=${json.id} request=$request response=$response")
+          requestor ! response
+        case None =>
+          log.warning(s"could not find requestor for reqId=${json.id} response=$json")
+      }
+      context become connected(connection, remoteAddress, tip, buffer, requests - json.id)
+
+    case Left(response: HeaderSubscriptionResponse) => headerSubscriptions.map(_ ! response)
+
+    case Left(response: AddressSubscriptionResponse) => addressSubscriptions.get(response.address).map(listeners => listeners.map(_ ! response))
+
+    case Left(response: ScriptHashSubscriptionResponse) => scriptHashSubscriptions.get(response.scriptHash).map(listeners => listeners.map(_ ! response))
+
+    case HeaderSubscriptionResponse(newtip) =>
+      log.info(s"new tip $newtip")
+      updateBlockCount(newtip.block_height)
+      context become connected(connection, remoteAddress, newtip, buffer, requests)
+  }
+
+}
+
+object ElectrumClient {
+
+  def apply(addresses: java.util.List[InetSocketAddress]): ElectrumClient = {
+    import collection.JavaConversions._
+    new ElectrumClient(addresses)
+  }
+
+  /**
+    * Utility function to converts a publicKeyScript to electrum's scripthash
+    *
+    * @param publicKeyScript public key script
+    * @return the hash of the public key script, as used by ElectrumX's hash-based methods
+    */
+  def computeScriptHash(publicKeyScript: BinaryData): BinaryData = Crypto.sha256(publicKeyScript).reverse
+
+  // @formatter:off
+  sealed trait Request
+  sealed trait Response
+
+  case class ServerVersion(clientName: String, protocolVersion: String) extends Request
+  case class ServerVersionResponse(clientName: String, protocolVersion: String) extends Response
+
+  case class GetAddressHistory(address: String) extends Request
+  case class TransactionHistoryItem(height: Long, tx_hash: BinaryData)
+  case class GetAddressHistoryResponse(address: String, history: Seq[TransactionHistoryItem]) extends Response
+
+  case class GetScriptHashHistory(scriptHash: BinaryData) extends Request
+  case class GetScriptHashHistoryResponse(scriptHash: BinaryData, history: Seq[TransactionHistoryItem]) extends Response
+
+  case class AddressListUnspent(address: String) extends Request
+  case class UnspentItem(tx_hash: BinaryData, tx_pos: Int, value: Long, height: Long) {
+    lazy val outPoint = OutPoint(tx_hash.reverse, tx_pos)
+  }
+  case class AddressListUnspentResponse(address: String, unspents: Seq[UnspentItem]) extends Response
+
+  case class ScriptHashListUnspent(scriptHash: BinaryData) extends Request
+  case class ScriptHashListUnspentResponse(scriptHash: BinaryData, unspents: Seq[UnspentItem]) extends Response
+
+  case class BroadcastTransaction(tx: Transaction) extends Request
+  case class BroadcastTransactionResponse(tx: Transaction, error: Option[Error]) extends Response
+
+  case class GetTransaction(txid: BinaryData) extends Request
+  case class GetTransactionResponse(tx: Transaction) extends Response
+
+  case class GetMerkle(txid: BinaryData, height: Long) extends Request
+  case class GetMerkleResponse(txid: BinaryData, merkle: Seq[BinaryData], block_height: Long, pos: Int) extends Response {
+    lazy val root: BinaryData = {
+      @tailrec
+      def loop(pos: Int, hashes: Seq[BinaryData]): BinaryData = {
+        if (hashes.length == 1) hashes(0).reverse
+        else {
+          val h = if (pos % 2 == 1) Crypto.hash256(hashes(1) ++ hashes(0)) else Crypto.hash256(hashes(0) ++ hashes(1))
+          loop(pos / 2, h +: hashes.drop(2))
+        }
+      }
+      loop(pos, BinaryData(txid.reverse) +: merkle.map(b => BinaryData(b.reverse)))
+    }
+  }
+
+  case class AddressSubscription(address: String, actor: ActorRef) extends Request
+  case class AddressSubscriptionResponse(address: String, status: String) extends Response
+
+  case class ScriptHashSubscription(scriptHash: BinaryData, actor: ActorRef) extends Request
+  case class ScriptHashSubscriptionResponse(scriptHash: BinaryData, status: String) extends Response
+
+  case class HeaderSubscription(actor: ActorRef) extends Request
+  case class HeaderSubscriptionResponse(header: Header) extends Response
+
+  case class Header(block_height: Long, version: Long, prev_block_hash: BinaryData, merkle_root: BinaryData, timestamp: Long, bits: Long, nonce: Long) {
+    lazy val block_hash: BinaryData = {
+      val blockHeader = BlockHeader(version, prev_block_hash.reverse, merkle_root.reverse, timestamp, bits, nonce)
+      blockHeader.hash.reverse
+    }
+  }
+
+  object Header {
+    def makeHeader(height: Long, header: BlockHeader) = ElectrumClient.Header(0, header.version, header.hashPreviousBlock, header.hashMerkleRoot, header.time, header.bits, header.nonce)
+
+    val RegtestGenesisHeader = makeHeader(0, Block.RegtestGenesisBlock.header)
+    val TestnetGenesisHeader = makeHeader(0, Block.TestnetGenesisBlock.header)
+  }
+
+  case class TransactionHistory(history: Seq[TransactionHistoryItem]) extends Response
+
+  case class AddressStatus(address: String, status: String) extends Response
+
+  case class ServerError(request: Request, error: Error) extends Response
+  case class AddStatusListener(actor: ActorRef) extends Response
+
+  sealed trait ElectrumEvent
+  case object ElectrumConnected extends ElectrumEvent
+  case object ElectrumReady extends ElectrumEvent
+  case object ElectrumDisconnected extends ElectrumEvent
+
+  // @formatter:on
+
+  def parseResponse(input: String): Either[Response, JsonRPCResponse] = {
+    implicit val formats = DefaultFormats
+    val json = JsonMethods.parse(new String(input))
+    json \ "method" match {
+      case JString(method) =>
+        // this is a jsonrpc request, i.e. a subscription response
+        val JArray(params) = json \ "params"
+        Left(((method, params): @unchecked) match {
+          case ("blockchain.headers.subscribe", header :: Nil) => HeaderSubscriptionResponse(parseHeader(header))
+          case ("blockchain.address.subscribe", JString(address) :: JNull :: Nil) => AddressSubscriptionResponse(address, "")
+          case ("blockchain.address.subscribe", JString(address) :: JString(status) :: Nil) => AddressSubscriptionResponse(address, status)
+          case ("blockchain.scripthash.subscribe", JString(scriptHashHex) :: JNull :: Nil) => ScriptHashSubscriptionResponse(BinaryData(scriptHashHex), "")
+          case ("blockchain.scripthash.subscribe", JString(scriptHashHex) :: JString(status) :: Nil) => ScriptHashSubscriptionResponse(BinaryData(scriptHashHex), status)
+        })
+      case _ => Right(parseJsonRpcResponse(json))
+    }
+  }
+
+  def parseJsonRpcResponse(json: JValue): JsonRPCResponse = {
+    implicit val formats = DefaultFormats
+    val result = json \ "result"
+    val error = json \ "error" match {
+      case JNull => None
+      case JNothing => None
+      case other =>
+        val message = other \ "message" match {
+          case JString(value) => value
+          case _ => ""
+        }
+        val code = other \ " code" match {
+          case JInt(value) => value.intValue()
+          case JLong(value) => value.intValue()
+          case _ => 0
+        }
+        Some(Error(code, message))
+    }
+    val id = json \ "id" match {
+      case JString(value) => value
+      case JInt(value) => value.toString()
+      case JLong(value) => value.toString
+      case _ => ""
+    }
+    JsonRPCResponse(result, error, id)
+  }
+
+  def longField(jvalue: JValue, field: String): Long = (jvalue \ field: @unchecked) match {
+    case JLong(value) => value.longValue()
+    case JInt(value) => value.longValue()
+  }
+
+  def intField(jvalue: JValue, field: String): Int = (jvalue \ field: @unchecked) match {
+    case JLong(value) => value.intValue()
+    case JInt(value) => value.intValue()
+  }
+
+  def parseHeader(json: JValue): Header = {
+    val block_height = longField(json, "block_height")
+    val version = longField(json, "version")
+    val timestamp = longField(json, "timestamp")
+    val bits = longField(json, "bits")
+    val nonce = longField(json, "nonce")
+    val JString(prev_block_hash) = json \ "prev_block_hash"
+    val JString(merkle_root) = json \ "merkle_root"
+    Header(block_height, version, prev_block_hash, merkle_root, timestamp, bits, nonce)
+  }
+
+  def makeRequest(request: Request, reqId: String): JsonRPCRequest = request match {
+    case ServerVersion(clientName, protocolVersion) => JsonRPCRequest(id = reqId, method = "server.version", params = clientName :: protocolVersion :: Nil)
+    case GetAddressHistory(address) => JsonRPCRequest(id = reqId, method = "blockchain.address.get_history", params = address :: Nil)
+    case GetScriptHashHistory(scripthash) => JsonRPCRequest(id = reqId, method = "blockchain.scripthash.get_history", params = scripthash.toString() :: Nil)
+    case AddressListUnspent(address) => JsonRPCRequest(id = reqId, method = "blockchain.address.listunspent", params = address :: Nil)
+    case ScriptHashListUnspent(scripthash) => JsonRPCRequest(id = reqId, method = "blockchain.scripthash.listunspent", params = scripthash.toString() :: Nil)
+    case AddressSubscription(address, _) => JsonRPCRequest(id = reqId, method = "blockchain.address.subscribe", params = address :: Nil)
+    case ScriptHashSubscription(scriptHash, _) => JsonRPCRequest(id = reqId, method = "blockchain.scripthash.subscribe", params = scriptHash.toString() :: Nil)
+    case BroadcastTransaction(tx) => JsonRPCRequest(id = reqId, method = "blockchain.transaction.broadcast", params = Hex.toHexString(Transaction.write(tx)) :: Nil)
+    case GetTransaction(txid: BinaryData) => JsonRPCRequest(id = reqId, method = "blockchain.transaction.get", params = txid :: Nil)
+    case HeaderSubscription(_) => JsonRPCRequest(id = reqId, method = "blockchain.headers.subscribe", params = Nil)
+    case GetMerkle(txid, height) => JsonRPCRequest(id = reqId, method = "blockchain.transaction.get_merkle", params = txid :: height :: Nil)
+  }
+
+  def parseJsonResponse(request: Request, json: JsonRPCResponse): Response = {
+    implicit val formats = DefaultFormats
+    json.error match {
+      case Some(error) => (request: @unchecked) match {
+        case BroadcastTransaction(tx) => BroadcastTransactionResponse(tx, Some(error)) // for this request type, error are considered a "normal" response
+        case _ => ServerError(request, error)
+      }
+      case None => (request: @unchecked) match {
+        case s: ServerVersion =>
+          val JArray(jitems) = json.result
+          val JString(clientName) = jitems(0)
+          val JString(protocolVersion) = jitems(1)
+          ServerVersionResponse(clientName, protocolVersion)
+        case GetAddressHistory(address) =>
+          val JArray(jitems) = json.result
+          val items = jitems.map(jvalue => {
+            val JString(tx_hash) = jvalue \ "tx_hash"
+            val height = longField(jvalue, "height")
+            TransactionHistoryItem(height, tx_hash)
+          })
+          GetAddressHistoryResponse(address, items)
+        case GetScriptHashHistory(scripthash) =>
+          val JArray(jitems) = json.result
+          val items = jitems.map(jvalue => {
+            val JString(tx_hash) = jvalue \ "tx_hash"
+            val height = longField(jvalue, "height")
+            TransactionHistoryItem(height, tx_hash)
+          })
+          GetScriptHashHistoryResponse(scripthash, items)
+        case AddressListUnspent(address) =>
+          val JArray(jitems) = json.result
+          val items = jitems.map(jvalue => {
+            val JString(tx_hash) = jvalue \ "tx_hash"
+            val tx_pos = intField(jvalue, "tx_pos")
+            val height = longField(jvalue, "height")
+            val value = longField(jvalue, "value")
+            UnspentItem(tx_hash, tx_pos, value, height)
+          })
+          AddressListUnspentResponse(address, items)
+        case ScriptHashListUnspent(scripthash) =>
+          val JArray(jitems) = json.result
+          val items = jitems.map(jvalue => {
+            val JString(tx_hash) = jvalue \ "tx_hash"
+            val tx_pos = intField(jvalue, "tx_pos")
+            val height = longField(jvalue, "height")
+            val value = longField(jvalue, "value")
+            UnspentItem(tx_hash, tx_pos, value, height)
+          })
+          ScriptHashListUnspentResponse(scripthash, items)
+        case GetTransaction(_) =>
+          val JString(hex) = json.result
+          GetTransactionResponse(Transaction.read(hex))
+        case AddressSubscription(address, _) => json.result match {
+          case JString(status) => AddressSubscriptionResponse(address, status)
+          case _ => AddressSubscriptionResponse(address, "")
+        }
+        case ScriptHashSubscription(scriptHash, _) => json.result match {
+          case JString(status) => ScriptHashSubscriptionResponse(scriptHash, status)
+          case _ => ScriptHashSubscriptionResponse(scriptHash, "")
+        }
+        case BroadcastTransaction(tx) =>
+          val JString(txid) = json.result
+          require(BinaryData(txid) == tx.txid)
+          BroadcastTransactionResponse(tx, None)
+        case GetMerkle(txid, height) =>
+          val JArray(hashes) = json.result \ "merkle"
+          val leaves = hashes collect { case JString(value) => BinaryData(value) }
+          val blockHeight = longField(json.result, "block_height")
+          val JInt(pos) = json.result \ "pos"
+          GetMerkleResponse(txid, leaves, blockHeight, pos.toInt)
+      }
+    }
+  }
+
+  def readServerAddresses(stream: InputStream): Seq[InetSocketAddress] = try {
+    val JObject(values) = JsonMethods.parse(stream)
+    val addresses = values.map {
+      case (name, fields) =>
+        val JString(port) = fields \ "t"
+        new InetSocketAddress(name, port.toInt)
+    }
+    val randomized = Random.shuffle(addresses)
+    randomized
+  } finally {
+    stream.close()
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/electrum/ElectrumEclairWallet.scala

@@ -0,0 +1,69 @@
+package fr.acinq.eclair.blockchain.electrum
+
+import akka.actor.{ActorRef, ActorSystem}
+import akka.pattern.ask
+import fr.acinq.bitcoin.{Base58, Base58Check, BinaryData, OP_EQUAL, OP_HASH160, OP_PUSHDATA, Satoshi, Script, Transaction, TxOut}
+import fr.acinq.eclair.blockchain.electrum.ElectrumClient.BroadcastTransaction
+import fr.acinq.eclair.blockchain.electrum.ElectrumWallet._
+import fr.acinq.eclair.blockchain.{EclairWallet, MakeFundingTxResponse}
+import grizzled.slf4j.Logging
+
+import scala.concurrent.{ExecutionContext, Future}
+
+class ElectrumEclairWallet(val wallet: ActorRef)(implicit system: ActorSystem, ec: ExecutionContext, timeout: akka.util.Timeout) extends EclairWallet with Logging {
+
+  override def getBalance = (wallet ? GetBalance).mapTo[GetBalanceResponse].map(balance => balance.confirmed + balance.unconfirmed)
+
+  override def getFinalAddress = (wallet ? GetCurrentReceiveAddress).mapTo[GetCurrentReceiveAddressResponse].map(_.address)
+
+  override def makeFundingTx(pubkeyScript: BinaryData, amount: Satoshi, feeRatePerKw: Long) = {
+    val tx = Transaction(version = 2, txIn = Nil, txOut = TxOut(amount, pubkeyScript) :: Nil, lockTime = 0)
+    (wallet ? CompleteTransaction(tx, feeRatePerKw)).mapTo[CompleteTransactionResponse].map(response => response match {
+      case CompleteTransactionResponse(tx1, None) => MakeFundingTxResponse(tx1, 0)
+      case CompleteTransactionResponse(_, Some(error)) => throw error
+    })
+  }
+
+  override def commit(tx: Transaction): Future[Boolean] =
+    (wallet ? BroadcastTransaction(tx)) flatMap {
+      case ElectrumClient.BroadcastTransactionResponse(tx, None) =>
+        //tx broadcast successfully: commit tx
+        wallet ? CommitTransaction(tx)
+      case ElectrumClient.BroadcastTransactionResponse(tx, Some(error)) if error.message.contains("transaction already in block chain") =>
+        // tx was already in the blockchain, that's weird but it is OK
+        wallet ? CommitTransaction(tx)
+      case ElectrumClient.BroadcastTransactionResponse(_, Some(error)) =>
+        //tx broadcast failed: cancel tx
+        logger.error(s"cannot broadcast tx ${tx.txid}: $error")
+        wallet ? CancelTransaction(tx)
+      case ElectrumClient.ServerError(ElectrumClient.BroadcastTransaction(tx), error) =>
+        //tx broadcast failed: cancel tx
+        logger.error(s"cannot broadcast tx ${tx.txid}: $error")
+        wallet ? CancelTransaction(tx)
+    } map {
+      case CommitTransactionResponse(_) => true
+      case CancelTransactionResponse(_) => false
+    }
+
+  def sendPayment(amount: Satoshi, address: String, feeRatePerKw: Long): Future[String] = {
+    val publicKeyScript = Base58Check.decode(address) match {
+      case (Base58.Prefix.PubkeyAddressTestnet, pubKeyHash) => Script.pay2pkh(pubKeyHash)
+      case (Base58.Prefix.ScriptAddressTestnet, scriptHash) => OP_HASH160 :: OP_PUSHDATA(scriptHash) :: OP_EQUAL :: Nil
+    }
+    val tx = Transaction(version = 2, txIn = Nil, txOut = TxOut(amount, publicKeyScript) :: Nil, lockTime = 0)
+
+    (wallet ? CompleteTransaction(tx, feeRatePerKw))
+      .mapTo[CompleteTransactionResponse]
+      .flatMap {
+        case CompleteTransactionResponse(tx, None) => commit(tx).map {
+          case true => tx.txid.toString()
+          case false => throw new RuntimeException(s"could not commit tx=${Transaction.write(tx)}")
+        }
+        case CompleteTransactionResponse(_, Some(error)) => throw error
+      }
+  }
+
+  def getMnemonics: Future[Seq[String]] = (wallet ? GetMnemonicCode).mapTo[GetMnemonicCodeResponse].map(_.mnemonics)
+
+  override def rollback(tx: Transaction): Future[Boolean] = (wallet ? CancelTransaction(tx)).map(_ => true)
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/electrum/ElectrumWallet.scala

@@ -0,0 +1,693 @@
+package fr.acinq.eclair.blockchain.electrum
+
+import java.io.File
+
+import akka.actor.{ActorRef, LoggingFSM, Props}
+import com.google.common.io.Files
+import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
+import fr.acinq.bitcoin.DeterministicWallet.{ExtendedPrivateKey, derivePrivateKey, hardened}
+import fr.acinq.bitcoin.{Base58, Base58Check, BinaryData, Block, Crypto, DeterministicWallet, MnemonicCode, OP_PUSHDATA, OutPoint, SIGHASH_ALL, Satoshi, Script, ScriptFlags, ScriptWitness, SigVersion, Transaction, TxIn, TxOut}
+import fr.acinq.eclair.blockchain.bitcoind.rpc.Error
+import fr.acinq.eclair.blockchain.electrum.ElectrumClient.{GetTransaction, GetTransactionResponse, TransactionHistoryItem, computeScriptHash}
+import fr.acinq.eclair.randomBytes
+import grizzled.slf4j.Logging
+
+import scala.annotation.tailrec
+import scala.util.{Failure, Success, Try}
+
+/**
+  * Simple electrum wallet
+  *
+  * Typical workflow:
+  *
+  * client ---- header update ----> wallet
+  * client ---- status update ----> wallet
+  * client <--- ask history   ----- wallet
+  * client ---- history       ----> wallet
+  * client <--- ask tx        ----- wallet
+  * client ---- tx            ----> wallet
+  *
+  * @param mnemonics
+  * @param client
+  * @param params
+  */
+class ElectrumWallet(mnemonics: Seq[String], client: ActorRef, params: ElectrumWallet.WalletParameters) extends LoggingFSM[ElectrumWallet.State, ElectrumWallet.Data] {
+
+  import ElectrumWallet._
+  import params._
+
+  val seed = MnemonicCode.toSeed(mnemonics, "")
+  val master = DeterministicWallet.generate(seed)
+
+  val accountMaster = accountKey(master)
+  val changeMaster = changeKey(master)
+
+  client ! ElectrumClient.AddStatusListener(self)
+
+  // disconnected --> waitingForTip --> running --
+  // ^                                            |
+  // |                                            |
+  //  --------------------------------------------
+
+  startWith(DISCONNECTED, {
+    val header = chainHash match {
+      case Block.RegtestGenesisBlock.hash => ElectrumClient.Header.RegtestGenesisHeader
+      case Block.TestnetGenesisBlock.hash => ElectrumClient.Header.TestnetGenesisHeader
+    }
+    val firstAccountKeys = (0 until params.swipeRange).map(i => derivePrivateKey(accountMaster, i)).toVector
+    val firstChangeKeys = (0 until params.swipeRange).map(i => derivePrivateKey(changeMaster, i)).toVector
+    val data = Data(params, header, firstAccountKeys, firstChangeKeys)
+    context.system.eventStream.publish(NewWalletReceiveAddress(data.currentReceiveAddress))
+    data
+  })
+
+  when(DISCONNECTED) {
+    case Event(ElectrumClient.ElectrumReady, data) =>
+      client ! ElectrumClient.HeaderSubscription(self)
+      goto(WAITING_FOR_TIP) using data
+  }
+
+  when(WAITING_FOR_TIP) {
+    case Event(ElectrumClient.HeaderSubscriptionResponse(header), data) =>
+      data.accountKeys.foreach(key => client ! ElectrumClient.ScriptHashSubscription(computeScriptHashFromPublicKey(key.publicKey), self))
+      data.changeKeys.foreach(key => client ! ElectrumClient.ScriptHashSubscription(computeScriptHashFromPublicKey(key.publicKey), self))
+      goto(RUNNING) using data.copy(tip = header)
+
+    case Event(ElectrumClient.ElectrumDisconnected, data) =>
+      log.info(s"wallet got disconnected")
+      goto(DISCONNECTED) using data
+  }
+
+  when(RUNNING) {
+    case Event(ElectrumClient.HeaderSubscriptionResponse(header), data) if data.tip == header => stay
+
+    case Event(ElectrumClient.HeaderSubscriptionResponse(header), data) =>
+      log.info(s"got new tip ${header.block_hash} at ${header.block_height}")
+      data.heights.collect {
+        case (txid, height) if height > 0 =>
+          val confirmations = computeDepth(header.block_height, height)
+          context.system.eventStream.publish(TransactionConfidenceChanged(txid, confirmations))
+      }
+      stay using data.copy(tip = header)
+
+    case Event(ElectrumClient.ScriptHashSubscriptionResponse(scriptHash, status), data) if data.status.get(scriptHash) == Some(status) => stay // we already have it
+
+    case Event(ElectrumClient.ScriptHashSubscriptionResponse(scriptHash, status), data) if !data.accountKeyMap.contains(scriptHash) && !data.changeKeyMap.contains(scriptHash) =>
+      log.warning(s"received status=$status for scriptHash=$scriptHash which does not match any of our keys")
+      stay
+
+    case Event(ElectrumClient.ScriptHashSubscriptionResponse(scriptHash, status), data) if status == "" =>
+      val data1 = data.copy(status = data.status + (scriptHash -> status)) // empty status, nothing to do
+      goto(stateName) using data1
+
+    case Event(ElectrumClient.ScriptHashSubscriptionResponse(scriptHash, status), data) =>
+      val key = data.accountKeyMap.getOrElse(scriptHash, data.changeKeyMap(scriptHash))
+      val isChange = data.changeKeyMap.contains(scriptHash)
+      log.info(s"received status=$status for scriptHash=$scriptHash key=${segwitAddress(key)} isChange=$isChange")
+
+      // let's retrieve the tx history for this key
+      client ! ElectrumClient.GetScriptHashHistory(scriptHash)
+
+      val (newAccountKeys, newChangeKeys) = data.status.get(status) match {
+        case None =>
+          // first time this script hash is used, need to generate a new key
+          val newKey = if (isChange) derivePrivateKey(changeMaster, data.changeKeys.last.path.lastChildNumber + 1) else derivePrivateKey(accountMaster, data.accountKeys.last.path.lastChildNumber + 1)
+          val newScriptHash = computeScriptHashFromPublicKey(newKey.publicKey)
+          log.info(s"generated key with index=${newKey.path.lastChildNumber} scriptHash=$newScriptHash key=${segwitAddress(newKey)} isChange=$isChange")
+          // listens to changes for the newly generated key
+          client ! ElectrumClient.ScriptHashSubscription(newScriptHash, self)
+          if (isChange) (data.accountKeys, data.changeKeys :+ newKey) else (data.accountKeys :+ newKey, data.changeKeys)
+        case Some(_) => (data.accountKeys, data.changeKeys)
+      }
+
+      val data1 = data.copy(
+        accountKeys = newAccountKeys,
+        changeKeys = newChangeKeys,
+        status = data.status + (scriptHash -> status),
+        pendingHistoryRequests = data.pendingHistoryRequests + scriptHash)
+
+      goto(stateName) using data1 // goto instead of stay because we want to fire transitions
+
+    case Event(ElectrumClient.GetScriptHashHistoryResponse(scriptHash, history), data) =>
+      log.debug(s"scriptHash=$scriptHash has history=$history")
+      val (heights1, pendingTransactionRequests1) = history.foldLeft((data.heights, data.pendingTransactionRequests)) {
+        case ((heights, hashes), item) if !data.transactions.contains(item.tx_hash) && !data.pendingTransactionRequests.contains(item.tx_hash) =>
+          // we retrieve the tx if we don't have it and haven't yet requested it
+          client ! GetTransaction(item.tx_hash)
+          (heights + (item.tx_hash -> item.height), hashes + item.tx_hash)
+        case ((heights, hashes), item) =>
+          // otherwise we just update the height
+          (heights + (item.tx_hash -> item.height), hashes)
+      }
+
+      // we now have updated height for all our transactions,
+      heights1.collect {
+        case (txid, height) =>
+          val confirmations = if (height <= 0) 0 else computeDepth(data.tip.block_height, height)
+          (data.heights.get(txid), height) match {
+            case (None, height) if height <= 0 =>
+            // height=0 => unconfirmed, height=-1 => unconfirmed and one input is unconfirmed
+            case (None, height) if height > 0 =>
+              // first time we get a height for this tx: either it was just confirmed, or we restarted the wallet
+              context.system.eventStream.publish(TransactionConfidenceChanged(txid, confirmations))
+            case (Some(previousHeight), height) if previousHeight != height =>
+              // there was a reorg
+              context.system.eventStream.publish(TransactionConfidenceChanged(txid, confirmations))
+            case (Some(previousHeight), height) if previousHeight == height =>
+            // no reorg, nothing to do
+          }
+      }
+      val data1 = data.copy(heights = heights1, history = data.history + (scriptHash -> history), pendingHistoryRequests = data.pendingHistoryRequests - scriptHash, pendingTransactionRequests = pendingTransactionRequests1)
+      goto(stateName) using data1 // goto instead of stay because we want to fire transitions
+
+    case Event(GetTransactionResponse(tx), data) =>
+      log.debug(s"received transaction ${tx.txid}")
+      data.computeTransactionDelta(tx) match {
+        case Some((received, sent, fee_opt)) =>
+          log.info(s"successfully connected txid=${tx.txid}")
+          context.system.eventStream.publish(TransactionReceived(tx, data.computeTransactionDepth(tx.txid), received, sent, fee_opt))
+          // when we have successfully processed a new tx, we retry all pending txes to see if they can be added now
+          data.pendingTransactions.foreach(self ! GetTransactionResponse(_))
+          val data1 = data.copy(transactions = data.transactions + (tx.txid -> tx), pendingTransactionRequests = data.pendingTransactionRequests - tx.txid, pendingTransactions = Nil)
+          goto(stateName) using data1 // goto instead of stay because we want to fire transitions
+        case None =>
+          // missing parents
+          log.info(s"couldn't connect txid=${tx.txid}")
+          val data1 = data.copy(pendingTransactions = data.pendingTransactions :+ tx)
+          stay using data1
+      }
+
+    case Event(CompleteTransaction(tx, feeRatePerKw), data) =>
+      Try(data.completeTransaction(tx, feeRatePerKw, minimumFee, dustLimit, allowSpendUnconfirmed)) match {
+        case Success((data1, tx1)) => stay using data1 replying CompleteTransactionResponse(tx1, None)
+        case Failure(t) => stay replying CompleteTransactionResponse(tx, Some(t))
+      }
+
+    case Event(CommitTransaction(tx), data) =>
+      log.info(s"committing txid=${tx.txid}")
+      val data1 = data.commitTransaction(tx)
+      // we use the initial state to compute the effect of the tx
+      // note: we know that computeTransactionDelta and the fee will be defined, because we built the tx ourselves so
+      // we know all the parents
+      val (received, sent, Some(fee)) = data.computeTransactionDelta(tx).get
+      // we notify here because the tx won't be downloaded again (it has been added to the state at commit)
+      context.system.eventStream.publish(TransactionReceived(tx, data1.computeTransactionDepth(tx.txid), received, sent, Some(fee)))
+      goto(stateName) using data1 replying CommitTransactionResponse(tx) // goto instead of stay because we want to fire transitions
+
+    case Event(CancelTransaction(tx), data) =>
+      log.info(s"cancelling txid=${tx.txid}")
+      stay using data.cancelTransaction(tx) replying CancelTransactionResponse(tx)
+
+    case Event(bc@ElectrumClient.BroadcastTransaction(tx), _) =>
+      log.info(s"broadcasting txid=${tx.txid}")
+      client forward bc
+      stay
+
+    case Event(ElectrumClient.ElectrumDisconnected, data) =>
+      log.info(s"wallet got disconnected")
+      goto(DISCONNECTED) using data
+  }
+
+  whenUnhandled {
+    case Event(GetMnemonicCode, _) => stay replying GetMnemonicCodeResponse(mnemonics)
+
+    case Event(GetCurrentReceiveAddress, data) => stay replying GetCurrentReceiveAddressResponse(data.currentReceiveAddress)
+
+    case Event(GetBalance, data) =>
+      val (confirmed, unconfirmed) = data.balance
+      stay replying GetBalanceResponse(confirmed, unconfirmed)
+
+    case Event(GetData, data) => stay replying GetDataResponse(data)
+
+    case Event(ElectrumClient.BroadcastTransaction(tx), _) => stay replying ElectrumClient.BroadcastTransactionResponse(tx, Some(Error(-1, "wallet is not connected")))
+  }
+
+  onTransition {
+    case _ -> _ if nextStateData.isReady(params.swipeRange) =>
+      val ready = nextStateData.readyMessage
+      log.info(s"wallet is ready with $ready")
+      context.system.eventStream.publish(ready)
+      context.system.eventStream.publish(NewWalletReceiveAddress(nextStateData.currentReceiveAddress))
+  }
+
+  initialize()
+
+}
+
+object ElectrumWallet {
+
+  // use 32 bytes seed, which will generate a 24 words mnemonic code
+  val SEED_BYTES_LENGTH = 32
+
+  def props(mnemonics: Seq[String], client: ActorRef, params: WalletParameters): Props = Props(new ElectrumWallet(mnemonics, client, params))
+
+  def props(file: File, client: ActorRef, params: WalletParameters): Props = {
+    val entropy: BinaryData = (file.exists(), file.canRead(), file.isFile) match {
+      case (true, true, true) => Files.toByteArray(file)
+      case (false, _, _) =>
+        val buffer = randomBytes(SEED_BYTES_LENGTH)
+        Files.write(buffer, file)
+        buffer
+      case _ => throw new IllegalArgumentException(s"cannot create wallet:$file exist but cannot read from")
+    }
+    val mnemonics = MnemonicCode.toMnemonics(entropy)
+    Props(new ElectrumWallet(mnemonics, client, params))
+  }
+
+  case class WalletParameters(chainHash: BinaryData, minimumFee: Satoshi = Satoshi(2000), dustLimit: Satoshi = Satoshi(546), swipeRange: Int = 10, allowSpendUnconfirmed: Boolean = true)
+
+  // @formatter:off
+  sealed trait State
+  case object DISCONNECTED extends State
+  case object WAITING_FOR_TIP extends State
+  case object RUNNING extends State
+
+  sealed trait Request
+  sealed trait Response
+
+  case object GetMnemonicCode extends RuntimeException
+  case class GetMnemonicCodeResponse(mnemonics: Seq[String]) extends Response
+
+  case object GetBalance extends Request
+  case class GetBalanceResponse(confirmed: Satoshi, unconfirmed: Satoshi) extends Response
+
+  case object GetCurrentReceiveAddress extends Request
+  case class GetCurrentReceiveAddressResponse(address: String) extends Response
+
+  case object GetData extends Request
+  case class GetDataResponse(state: Data) extends Response
+
+  case class CompleteTransaction(tx: Transaction, feeRatePerKw: Long) extends Request
+  case class CompleteTransactionResponse(tx: Transaction, error: Option[Throwable]) extends Response
+
+  case class CommitTransaction(tx: Transaction) extends Request
+  case class CommitTransactionResponse(tx: Transaction) extends Response
+
+  case class SendTransaction(tx: Transaction) extends Request
+  case class SendTransactionReponse(tx: Transaction) extends Response
+
+  case class CancelTransaction(tx: Transaction) extends Request
+  case class CancelTransactionResponse(tx: Transaction) extends Response
+
+  case object InsufficientFunds extends Response
+  case class AmountBelowDustLimit(dustLimit: Satoshi) extends Response
+
+  case class GetPrivateKey(address: String) extends Request
+  case class GetPrivateKeyResponse(address: String, key: Option[ExtendedPrivateKey]) extends Response
+
+
+  sealed trait WalletEvent
+  /**
+    *
+    * @param tx
+    * @param depth
+    * @param received
+    * @param sent
+    * @param feeOpt is set only when we know it (i.e. for outgoing transactions)
+    */
+  case class TransactionReceived(tx: Transaction, depth: Long, received: Satoshi, sent: Satoshi, feeOpt: Option[Satoshi]) extends WalletEvent
+  case class TransactionConfidenceChanged(txid: BinaryData, depth: Long) extends WalletEvent
+  case class NewWalletReceiveAddress(address: String) extends WalletEvent
+  case class WalletReady(confirmedBalance: Satoshi, unconfirmedBalance: Satoshi, height: Long) extends WalletEvent
+  // @formatter:on
+
+  /**
+    *
+    * @param key public key
+    * @return the address of the p2sh-of-p2wpkh script for this key
+    */
+  def segwitAddress(key: PublicKey): String = {
+    val script = Script.pay2wpkh(key)
+    val hash = Crypto.hash160(Script.write(script))
+    Base58Check.encode(Base58.Prefix.ScriptAddressTestnet, hash)
+  }
+
+  def segwitAddress(key: ExtendedPrivateKey): String = segwitAddress(key.publicKey)
+
+  def segwitAddress(key: PrivateKey): String = segwitAddress(key.publicKey)
+
+  /**
+    *
+    * @param key public key
+    * @return a p2sh-of-p2wpkh script for this key
+    */
+  def computePublicKeyScript(key: PublicKey) = Script.pay2sh(Script.pay2wpkh(key))
+
+  /**
+    *
+    * @param key public key
+    * @return the hash of the public key script for this key, as used by ElectrumX's hash-based methods
+    */
+  def computeScriptHashFromPublicKey(key: PublicKey): BinaryData = Crypto.sha256(Script.write(computePublicKeyScript(key))).reverse
+
+  /**
+    * use BIP49 (and not BIP44) since we use p2sh-of-p2wpkh
+    *
+    * @param master master key
+    * @return the BIP49 account key for this master key: m/49'/1'/0'/0
+    */
+  def accountKey(master: ExtendedPrivateKey) = DeterministicWallet.derivePrivateKey(master, hardened(49) :: hardened(1) :: hardened(0) :: 0L :: Nil)
+
+  /**
+    * use BIP49 (and not BIP44) since we use p2sh-of-p2wpkh
+    *
+    * @param master master key
+    * @return the BIP49 change key for this master key: m/49'/1'/0'/1
+    */
+  def changeKey(master: ExtendedPrivateKey) = DeterministicWallet.derivePrivateKey(master, hardened(49) :: hardened(1) :: hardened(0) :: 1L :: Nil)
+
+  def totalAmount(utxos: Seq[Utxo]): Satoshi = Satoshi(utxos.map(_.item.value).sum)
+
+  def totalAmount(utxos: Set[Utxo]): Satoshi = totalAmount(utxos.toSeq)
+
+  /**
+    *
+    * @param weight       transaction weight
+    * @param feeRatePerKw fee rate
+    * @return the fee for this tx weight
+    */
+  def computeFee(weight: Int, feeRatePerKw: Long): Satoshi = Satoshi((weight * feeRatePerKw) / 1000)
+
+  /**
+    *
+    * @param txIn transaction input
+    * @return Some(pubkey) if this tx input spends a p2sh-of-p2wpkh(pub), None otherwise
+    */
+  def extractPubKeySpentFrom(txIn: TxIn): Option[PublicKey] = {
+    Try {
+      // we're looking for tx that spend a pay2sh-of-p2wkph output
+      require(txIn.witness.stack.size == 2)
+      val sig = txIn.witness.stack(0)
+      val pub = txIn.witness.stack(1)
+      val OP_PUSHDATA(script, _) :: Nil = Script.parse(txIn.signatureScript)
+      val publicKey = PublicKey(pub)
+      if (Script.write(Script.pay2wpkh(publicKey)) == script) {
+        Some(publicKey)
+      } else None
+    } getOrElse None
+  }
+
+  def computeDepth(currentHeight: Long, txHeight: Long): Long = currentHeight - txHeight + 1
+
+  case class Utxo(key: ExtendedPrivateKey, item: ElectrumClient.UnspentItem) {
+    def outPoint: OutPoint = item.outPoint
+  }
+
+  /**
+    * Wallet state, which stores data returned by EletrumX servers.
+    * Most items are indexed by script hash (i.e. by pubkey script sha256 hash).
+    * Height follow ElectrumX's conventions:
+    * - h > 0 means that the tx was confirmed at block #h
+    * - 0 means unconfirmed, but all input are confirmed
+    * < 0 means unconfirmed, and sonme inputs are unconfirmed as well
+    *
+    * @param tip          current blockchain tip
+    * @param accountKeys  account keys
+    * @param changeKeys   change keys
+    * @param status       script hash -> status; "" means that the script hash has not been used
+    *                     yet
+    * @param transactions wallet transactions
+    * @param heights      transactions heights
+    * @param history      script hash -> history
+    * @param locks        transactions which lock some of our utxos.
+    */
+  case class Data(tip: ElectrumClient.Header,
+                  accountKeys: Vector[ExtendedPrivateKey],
+                  changeKeys: Vector[ExtendedPrivateKey],
+                  status: Map[BinaryData, String],
+                  transactions: Map[BinaryData, Transaction],
+                  heights: Map[BinaryData, Long],
+                  history: Map[BinaryData, Seq[ElectrumClient.TransactionHistoryItem]],
+                  locks: Set[Transaction],
+                  pendingHistoryRequests: Set[BinaryData],
+                  pendingTransactionRequests: Set[BinaryData],
+                  pendingTransactions: Seq[Transaction]) extends Logging {
+    lazy val accountKeyMap = accountKeys.map(key => computeScriptHashFromPublicKey(key.publicKey) -> key).toMap
+
+    lazy val changeKeyMap = changeKeys.map(key => computeScriptHashFromPublicKey(key.publicKey) -> key).toMap
+
+    lazy val firstUnusedAccountKeys = accountKeys.find(key => status.get(computeScriptHashFromPublicKey(key.publicKey)) == Some(""))
+
+    lazy val firstUnusedChangeKeys = changeKeys.find(key => status.get(computeScriptHashFromPublicKey(key.publicKey)) == Some(""))
+
+    lazy val publicScriptMap = (accountKeys ++ changeKeys).map(key => Script.write(computePublicKeyScript(key.publicKey)) -> key).toMap
+
+    lazy val utxos = history.keys.toSeq.map(scriptHash => getUtxos(scriptHash)).flatten
+
+    /**
+      * The wallet is ready if all current keys have an empty status, and we don't have
+      * any history/tx request pending
+      * NB: swipeRange * 2 because we have account keys and change keys
+      */
+    def isReady(swipeRange: Int) = status.filter(_._2 == "").size >= swipeRange * 2 && pendingHistoryRequests.isEmpty && pendingTransactionRequests.isEmpty
+
+    def readyMessage: WalletReady = {
+      val (confirmed, unconfirmed) = balance
+      WalletReady(confirmed, unconfirmed, tip.block_height)
+    }
+
+    /**
+      *
+      * @return the current receive key. In most cases it will be a key that has not
+      *         been used yet but it may be possible that we are still looking for
+      *         unused keys and none is available yet. In this case we will return
+      *         the latest account key.
+      */
+    def currentReceiveKey = firstUnusedAccountKeys.headOption.getOrElse {
+      // bad luck we are still looking for unused keys
+      // use the first account key
+      accountKeys.head
+    }
+
+    def currentReceiveAddress = segwitAddress(currentReceiveKey)
+
+    /**
+      *
+      * @return the current change key. In most cases it will be a key that has not
+      *         been used yet but it may be possible that we are still looking for
+      *         unused keys and none is available yet. In this case we will return
+      *         the latest change key.
+      */
+    def currentChangeKey = firstUnusedChangeKeys.headOption.getOrElse {
+      // bad luck we are still looking for unused keys
+      // use the first account key
+      changeKeys.head
+    }
+
+    def currentChangeAddress = segwitAddress(currentChangeKey)
+
+    def isMine(txIn: TxIn): Boolean = extractPubKeySpentFrom(txIn).exists(pub => publicScriptMap.contains(Script.write(computePublicKeyScript(pub))))
+
+    def isSpend(txIn: TxIn, publicKey: PublicKey): Boolean = extractPubKeySpentFrom(txIn).contains(publicKey)
+
+    /**
+      *
+      * @param txIn
+      * @param scriptHash
+      * @return true if txIn spends from an address that matches scriptHash
+      */
+    def isSpend(txIn: TxIn, scriptHash: BinaryData): Boolean = extractPubKeySpentFrom(txIn).exists(pub => computeScriptHashFromPublicKey(pub) == scriptHash)
+
+    def isReceive(txOut: TxOut, scriptHash: BinaryData): Boolean = publicScriptMap.get(txOut.publicKeyScript).exists(key => computeScriptHashFromPublicKey(key.publicKey) == scriptHash)
+
+    def isMine(txOut: TxOut): Boolean = publicScriptMap.contains(txOut.publicKeyScript)
+
+    def computeTransactionDepth(txid: BinaryData): Long = heights.get(txid).map(height => if (height > 0) computeDepth(tip.block_height, height) else 0).getOrElse(0)
+
+    /**
+      *
+      * @param scriptHash script hash
+      * @return the list of UTXOs for this script hash (including unconfirmed UTXOs)
+      */
+    def getUtxos(scriptHash: BinaryData) = {
+      history.get(scriptHash) match {
+        case None => Seq()
+        case Some(items) if items.isEmpty => Seq()
+        case Some(items) =>
+          // this is the private key for this script hash
+          val key = accountKeyMap.getOrElse(scriptHash, changeKeyMap(scriptHash))
+
+          // find all transactions that send to or receive from this script hash
+          // we use collect because we may not yet have received all transactions in the history
+          val txs = items collect { case item if transactions.contains(item.tx_hash) => transactions(item.tx_hash) }
+
+          // find all tx outputs that send to our script hash
+          val unspents = items collect { case item if transactions.contains(item.tx_hash) =>
+            val tx = transactions(item.tx_hash)
+            val outputs = tx.txOut.zipWithIndex.filter { case (txOut, index) => isReceive(txOut, scriptHash) }
+            outputs.map { case (txOut, index) => Utxo(key, ElectrumClient.UnspentItem(item.tx_hash, index, txOut.amount.toLong, item.height)) }
+          } flatten
+
+          // and remove the outputs that are being spent. this is needed because we may have unconfirmed UTXOs
+          // that are spend by unconfirmed transactions
+          unspents.filterNot(utxo => txs.exists(tx => tx.txIn.exists(_.outPoint == utxo.outPoint)))
+      }
+    }
+
+
+    /**
+      *
+      * @param scriptHash script hash
+      * @return the (confirmed, unconfirmed) balance for this script hash. This balance may not
+      *         be up-to-date if we have not received all data we've asked for yet.
+      */
+    def balance(scriptHash: BinaryData): (Satoshi, Satoshi) = {
+      history.get(scriptHash) match {
+        case None => (Satoshi(0), Satoshi(0))
+
+        case Some(items) if items.isEmpty => (Satoshi(0), Satoshi(0))
+
+        case Some(items) =>
+          val (confirmedItems, unconfirmedItems) = items.partition(_.height > 0)
+          val confirmedTxs = confirmedItems.collect { case item if transactions.contains(item.tx_hash) => transactions(item.tx_hash) }
+          val unconfirmedTxs = unconfirmedItems.collect { case item if transactions.contains(item.tx_hash) => transactions(item.tx_hash) }
+          if (confirmedTxs.size + unconfirmedTxs.size < confirmedItems.size + unconfirmedItems.size) logger.warn(s"we have not received all transactions yet, balance will not be up to date")
+
+          def findOurSpentOutputs(txs: Seq[Transaction]): Seq[TxOut] = {
+            val inputs = txs.map(_.txIn).flatten.filter(txIn => isSpend(txIn, scriptHash))
+            val spentOutputs = inputs.map(_.outPoint).map(outPoint => transactions.get(outPoint.txid).map(_.txOut(outPoint.index.toInt))).flatten
+            spentOutputs
+          }
+
+          val confirmedSpents = findOurSpentOutputs(confirmedTxs)
+          val confirmedReceived = confirmedTxs.map(_.txOut).flatten.filter(txOut => isReceive(txOut, scriptHash))
+
+          val unconfirmedSpents = findOurSpentOutputs(unconfirmedTxs)
+          val unconfirmedReceived = unconfirmedTxs.map(_.txOut).flatten.filter(txOut => isReceive(txOut, scriptHash))
+
+          val confirmedBalance = confirmedReceived.map(_.amount).sum - confirmedSpents.map(_.amount).sum
+          val unconfirmedBalance = unconfirmedReceived.map(_.amount).sum - unconfirmedSpents.map(_.amount).sum
+
+          (confirmedBalance, unconfirmedBalance)
+      }
+    }
+
+    /**
+      *
+      * @return the (confirmed, unconfirmed) balance for this wallet. This balance may not
+      *         be up-to-date if we have not received all data we've asked for yet.
+      */
+    lazy val balance: (Satoshi, Satoshi) = {
+      (accountKeyMap.keys ++ changeKeyMap.keys).map(scriptHash => balance(scriptHash)).foldLeft((Satoshi(0), Satoshi(0))) {
+        case ((confirmed, unconfirmed), (confirmed1, unconfirmed1)) => (confirmed + confirmed1, unconfirmed + unconfirmed1)
+      }
+    }
+
+    /**
+      * Computes the effect of this transaction on the wallet
+      *
+      * @param tx input transaction
+      * @return an option:
+      *         - Some(received, sent, fee) where sent if what the tx spends from us, received is what the tx sends to us,
+      *         and fee is the fee for the tx) tuple where sent if what the tx spends from us, and received is what the tx sends to us
+      *         - None if we are missing one or more parent txs
+      */
+    def computeTransactionDelta(tx: Transaction): Option[(Satoshi, Satoshi, Option[Satoshi])] = {
+      val ourInputs = tx.txIn.filter(isMine)
+      // we need to make sure that for all inputs spending an output we control, we already  have the parent tx
+      // (otherwise we can't estimate our balance)
+      val missingParent = ourInputs.exists(txIn => !transactions.contains(txIn.outPoint.txid))
+      if (missingParent) {
+        None
+      } else {
+        val sent = ourInputs.map(txIn => transactions(txIn.outPoint.txid).txOut(txIn.outPoint.index.toInt)).map(_.amount).sum
+        val received = tx.txOut.filter(isMine).map(_.amount).sum
+        // if all the inputs were ours, we can compute the fee, otherwise we can't
+        val fee_opt = if (ourInputs.size == tx.txIn.size) Some(sent - tx.txOut.map(_.amount).sum) else None
+        Some((received, sent, fee_opt))
+      }
+    }
+
+    /**
+      *
+      * @param tx           input tx that has no inputs
+      * @param feeRatePerKw fee rate per kiloweight
+      * @param minimumFee   minimum fee
+      * @param dustLimit    dust limit
+      * @return a (state, tx) tuple where state has been updated and tx is a complete,
+      *         fully signed transaction that can be broadcast.
+      *         our utxos spent by this tx are locked and won't be available for spending
+      *         until the tx has been cancelled. If the tx is committed, they will be removed
+      */
+    def completeTransaction(tx: Transaction, feeRatePerKw: Long, minimumFee: Satoshi, dustLimit: Satoshi, allowSpendUnconfirmed: Boolean): (Data, Transaction) = {
+      require(tx.txIn.isEmpty, "cannot complete a tx that already has inputs")
+      require(feeRatePerKw >= 0, "fee rate cannot be negative")
+      val amount = tx.txOut.map(_.amount).sum
+      require(amount > dustLimit, "amount to send is below dust limit")
+      val fee = {
+        val estimatedFee = computeFee(700, feeRatePerKw)
+        if (estimatedFee < minimumFee) minimumFee else estimatedFee
+      }
+
+      @tailrec
+      def select(chooseFrom: Seq[Utxo], selected: Set[Utxo]): Set[Utxo] = {
+        if (totalAmount(selected) >= amount + fee) selected
+        else if (chooseFrom.isEmpty) Set()
+        else select(chooseFrom.tail, selected + chooseFrom.head)
+      }
+
+      // select utxos that are not locked by pending txs
+      val lockedOutputs = locks.map(_.txIn.map(_.outPoint)).flatten
+      val unlocked = utxos.filterNot(utxo => lockedOutputs.contains(utxo.outPoint))
+      val unlocked1 = if (allowSpendUnconfirmed) unlocked else unlocked.filter(_.item.height > 0)
+      val selected = select(unlocked1, Set()).toSeq
+      require(totalAmount(selected) >= amount + fee, "insufficient funds")
+
+      // add inputs
+      var tx1 = tx.copy(txIn = selected.map(utxo => TxIn(utxo.outPoint, Nil, TxIn.SEQUENCE_FINAL)))
+
+      // add change output
+      val change = totalAmount(selected) - amount - fee
+      if (change >= dustLimit) tx1 = tx1.addOutput(TxOut(change, computePublicKeyScript(currentChangeKey.publicKey)))
+
+      // sign
+      for (i <- 0 until tx1.txIn.size) {
+        val key = selected(i).key
+        val sig = Transaction.signInput(tx1, i, Script.pay2pkh(key.publicKey), SIGHASH_ALL, Satoshi(selected(i).item.value), SigVersion.SIGVERSION_WITNESS_V0, key.privateKey)
+        tx1 = tx1.updateWitness(i, ScriptWitness(sig :: key.publicKey.toBin :: Nil)).updateSigScript(i, OP_PUSHDATA(Script.write(Script.pay2wpkh(key.publicKey))) :: Nil)
+      }
+      Transaction.correctlySpends(tx1, selected.map(utxo => utxo.outPoint -> TxOut(Satoshi(utxo.item.value), computePublicKeyScript(utxo.key.publicKey))).toMap, ScriptFlags.STANDARD_SCRIPT_VERIFY_FLAGS)
+
+      val data1 = this.copy(locks = this.locks + tx1)
+      (data1, tx1)
+    }
+
+    /**
+      * unlocks input locked by a pending tx. call this method if the tx will not be used after all
+      *
+      * @param tx pending transaction
+      * @return an updated state
+      */
+    def cancelTransaction(tx: Transaction): Data = this.copy(locks = this.locks - tx)
+
+    /**
+      * remove all our utxos spent by this tx. call this method if the tx was broadcast successfully
+      *
+      * @param tx pending transaction
+      * @return an updated state
+      */
+    def commitTransaction(tx: Transaction): Data = {
+      // HACK! since we base our utxos computation on the history as seen by the electrum server (so that it is
+      // reorg-proof out of the box), we need to update the history  right away if we want to be able to build chained
+      // unconfirmed transactions. A few seconds later electrum will notify us and the entry will be overwritten.
+      // Note that we need to take into account both inputs and outputs, because there may be change.
+      val history1 = (tx.txIn.filter(isMine).map(extractPubKeySpentFrom).flatten.map(computeScriptHashFromPublicKey) ++ tx.txOut.filter(isMine).map(_.publicKeyScript).map(computeScriptHash))
+        .foldLeft(this.history) {
+          case (history, scriptHash) =>
+            val entry = history.get(scriptHash) match {
+              case None => Seq(TransactionHistoryItem(0, tx.txid))
+              case Some(items) if items.map(_.tx_hash).contains(tx.txid) => items
+              case Some(items) => items :+ TransactionHistoryItem(0, tx.txid)
+            }
+            history + (scriptHash -> entry)
+        }
+      this.copy(locks = this.locks - tx, transactions = this.transactions + (tx.txid -> tx), heights = this.heights + (tx.txid -> 0L), history = history1)
+    }
+  }
+
+  object Data {
+    def apply(params: ElectrumWallet.WalletParameters, tip: ElectrumClient.Header, accountKeys: Vector[ExtendedPrivateKey], changeKeys: Vector[ExtendedPrivateKey]): Data
+    = Data(tip, accountKeys, changeKeys, Map(), Map(), Map(), Map(), Set(), Set(), Set(), Seq())
+  }
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/electrum/ElectrumWatcher.scala

@@ -0,0 +1,225 @@
+package fr.acinq.eclair.blockchain.electrum
+
+import java.net.InetSocketAddress
+
+import akka.actor.{Actor, ActorLogging, ActorRef, ActorSystem, Props, Stash, Terminated}
+import fr.acinq.bitcoin.Crypto.PublicKey
+import fr.acinq.bitcoin.{BinaryData, Satoshi, Script, Transaction, TxIn, TxOut}
+import fr.acinq.eclair.blockchain._
+import fr.acinq.eclair.blockchain.electrum.ElectrumClient._
+import fr.acinq.eclair.channel.{BITCOIN_FUNDING_DEPTHOK, BITCOIN_FUNDING_SPENT, BITCOIN_PARENT_TX_CONFIRMED}
+import fr.acinq.eclair.transactions.Scripts
+import fr.acinq.eclair.{Globals, fromShortId}
+
+import scala.collection.SortedMap
+
+
+class ElectrumWatcher(client: ActorRef) extends Actor with Stash with ActorLogging {
+
+  client ! ElectrumClient.AddStatusListener(self)
+
+  override def unhandled(message: Any): Unit = message match {
+    case ParallelGetRequest(announcements) => sender ! ParallelGetResponse(announcements.map {
+      case c =>
+        log.info(s"blindly validating channel=$c")
+        val pubkeyScript = Script.write(Script.pay2wsh(Scripts.multiSig2of2(PublicKey(c.bitcoinKey1), PublicKey(c.bitcoinKey2))))
+        val (_, _, outputIndex) = fromShortId(c.shortChannelId)
+        val fakeFundingTx = Transaction(
+          version = 2,
+          txIn = Seq.empty[TxIn],
+          txOut = List.fill(outputIndex + 1)(TxOut(Satoshi(0), pubkeyScript)), // quick and dirty way to be sure that the outputIndex'th output is of the expected format
+          lockTime = 0)
+        IndividualResult(c, Some(fakeFundingTx), true)
+    })
+    case _ => log.warning(s"unhandled message $message")
+  }
+
+  def receive = disconnected(Set.empty, Nil, SortedMap.empty)
+
+  def disconnected(watches: Set[Watch], publishQueue: Seq[PublishAsap], block2tx: SortedMap[Long, Seq[Transaction]]): Receive = {
+    case ElectrumClient.ElectrumReady =>
+      client ! ElectrumClient.HeaderSubscription(self)
+    case ElectrumClient.HeaderSubscriptionResponse(header) =>
+      watches.map(self ! _)
+      publishQueue.map(self ! _)
+      context become running(header, Set(), Map(), block2tx, Nil)
+    case watch: Watch => context become disconnected(watches + watch, publishQueue, block2tx)
+    case publish: PublishAsap => context become disconnected(watches, publishQueue :+ publish, block2tx)
+  }
+
+  def running(tip: ElectrumClient.Header, watches: Set[Watch], scriptHashStatus: Map[BinaryData, String], block2tx: SortedMap[Long, Seq[Transaction]], sent: Seq[Transaction]): Receive = {
+    case ElectrumClient.HeaderSubscriptionResponse(newtip) if tip == newtip => ()
+
+    case ElectrumClient.HeaderSubscriptionResponse(newtip) =>
+      log.info(s"new tip: ${newtip.block_hash} $newtip")
+      watches collect {
+        case watch: WatchConfirmed =>
+          val scriptHash = computeScriptHash(watch.publicKeyScript)
+          client ! ElectrumClient.GetScriptHashHistory(scriptHash)
+      }
+      val toPublish = block2tx.filterKeys(_ <= newtip.block_height)
+      toPublish.values.flatten.foreach(tx => self ! PublishAsap(tx))
+      context become running(newtip, watches, scriptHashStatus, block2tx -- toPublish.keys, sent)
+
+    case watch: Watch if watches.contains(watch) => ()
+
+    case watch@WatchSpent(_, txid, outputIndex, publicKeyScript, _) =>
+      val scriptHash = computeScriptHash(publicKeyScript)
+      log.info(s"added watch-spent on output=$txid:$outputIndex scriptHash=$scriptHash")
+      client ! ElectrumClient.ScriptHashSubscription(scriptHash, self)
+      context.watch(watch.channel)
+      context become running(tip, watches + watch, scriptHashStatus, block2tx, sent)
+
+    case watch@WatchSpentBasic(_, txid, outputIndex, publicKeyScript, _) =>
+      val scriptHash = computeScriptHash(publicKeyScript)
+      log.info(s"added watch-spent-basic on output=$txid:$outputIndex scriptHash=$scriptHash")
+      client ! ElectrumClient.ScriptHashSubscription(scriptHash, self)
+      context.watch(watch.channel)
+      context become running(tip, watches + watch, scriptHashStatus, block2tx, sent)
+
+    case watch@WatchConfirmed(_, txid, publicKeyScript, _, _) =>
+      val scriptHash = computeScriptHash(publicKeyScript)
+      log.info(s"added watch-confirmed on txid=$txid scriptHash=$scriptHash")
+      client ! ElectrumClient.GetScriptHashHistory(scriptHash)
+      context.watch(watch.channel)
+      context become running(tip, watches + watch, scriptHashStatus, block2tx, sent)
+
+    case Terminated(actor) =>
+      val watches1 = watches.filterNot(_.channel == actor)
+      context become running(tip, watches1, scriptHashStatus, block2tx, sent)
+
+    case ElectrumClient.ScriptHashSubscriptionResponse(scriptHash, status) =>
+      scriptHashStatus.get(scriptHash) match {
+        case Some(s) if s == status => log.debug(s"already have status=$status for scriptHash=$scriptHash")
+        case _ if status.isEmpty => log.info(s"empty status for scriptHash=$scriptHash")
+        case _ =>
+          log.info(s"new status=$status for scriptHash=$scriptHash")
+          client ! ElectrumClient.GetScriptHashHistory(scriptHash)
+      }
+      context become running(tip, watches, scriptHashStatus + (scriptHash -> status), block2tx, sent)
+
+    case ElectrumClient.GetScriptHashHistoryResponse(_, history) =>
+      // this is for WatchSpent/WatchSpentBasic
+      history.filter(_.height >= 0).map(item => client ! ElectrumClient.GetTransaction(item.tx_hash))
+      // this is for WatchConfirmed
+      history.collect {
+        case ElectrumClient.TransactionHistoryItem(height, tx_hash) if height > 0 => watches.collect {
+          case WatchConfirmed(_, txid, _, minDepth, _) if txid == tx_hash =>
+            val confirmations = tip.block_height - height + 1
+            log.info(s"txid=$txid was confirmed at height=$height and now has confirmations=$confirmations (currentHeight=${tip.block_height})")
+            if (confirmations >= minDepth) {
+              // we need to get the tx position in the block
+              client ! GetMerkle(tx_hash, height)
+            }
+        }
+      }
+
+    case ElectrumClient.GetMerkleResponse(tx_hash, _, height, pos) =>
+      val confirmations = tip.block_height - height + 1
+      val triggered = watches.collect {
+        case w@WatchConfirmed(channel, txid, _, minDepth, event) if txid == tx_hash && confirmations >= minDepth =>
+          log.info(s"txid=$txid had confirmations=$confirmations in block=$height pos=$pos")
+          channel ! WatchEventConfirmed(event, height.toInt, pos)
+          w
+      }
+      context become running(tip, watches -- triggered, scriptHashStatus, block2tx, sent)
+
+    case ElectrumClient.GetTransactionResponse(spendingTx) =>
+      val triggered = spendingTx.txIn.map(_.outPoint).flatMap(outPoint => watches.collect {
+        case WatchSpent(channel, txid, pos, _, event) if txid == outPoint.txid && pos == outPoint.index.toInt =>
+          log.info(s"output $txid:$pos spent by transaction ${spendingTx.txid}")
+          channel ! WatchEventSpent(event, spendingTx)
+          // NB: WatchSpent are permanent because we need to detect multiple spending of the funding tx
+          // They are never cleaned up but it is not a big deal for now (1 channel == 1 watch)
+          None
+        case w@WatchSpentBasic(channel, txid, pos, _, event) if txid == outPoint.txid && pos == outPoint.index.toInt =>
+          log.info(s"output $txid:$pos spent by transaction ${spendingTx.txid}")
+          channel ! WatchEventSpentBasic(event)
+          Some(w)
+      }).flatten
+      context become running(tip, watches -- triggered, scriptHashStatus, block2tx, sent)
+
+    case PublishAsap(tx) =>
+      val blockCount = Globals.blockCount.get()
+      val cltvTimeout = Scripts.cltvTimeout(tx)
+      val csvTimeout = Scripts.csvTimeout(tx)
+      if (csvTimeout > 0) {
+        require(tx.txIn.size == 1, s"watcher only supports tx with 1 input, this tx has ${tx.txIn.size} inputs")
+        val parentTxid = tx.txIn(0).outPoint.txid
+        log.info(s"txid=${tx.txid} has a relative timeout of $csvTimeout blocks, watching parenttxid=$parentTxid tx=${Transaction.write(tx)}")
+        val parentPublicKeyScript = WatchConfirmed.extractPublicKeyScript(tx.txIn.head.witness)
+        self ! WatchConfirmed(self, parentTxid, parentPublicKeyScript, minDepth = 1, BITCOIN_PARENT_TX_CONFIRMED(tx))
+      } else if (cltvTimeout > blockCount) {
+        log.info(s"delaying publication of txid=${tx.txid} until block=$cltvTimeout (curblock=$blockCount)")
+        val block2tx1 = block2tx.updated(cltvTimeout, block2tx.getOrElse(cltvTimeout, Seq.empty[Transaction]) :+ tx)
+        context become running(tip, watches, scriptHashStatus, block2tx1, sent)
+      } else {
+        log.info(s"publishing tx=${Transaction.write(tx)}")
+        client ! BroadcastTransaction(tx)
+        context become running(tip, watches, scriptHashStatus, block2tx, sent :+ tx)
+      }
+
+    case WatchEventConfirmed(BITCOIN_PARENT_TX_CONFIRMED(tx), blockHeight, _) =>
+      log.info(s"parent tx of txid=${tx.txid} has been confirmed")
+      val blockCount = Globals.blockCount.get()
+      val csvTimeout = Scripts.csvTimeout(tx)
+      val absTimeout = blockHeight + csvTimeout
+      if (absTimeout > blockCount) {
+        log.info(s"delaying publication of txid=${tx.txid} until block=$absTimeout (curblock=$blockCount)")
+        val block2tx1 = block2tx.updated(absTimeout, block2tx.getOrElse(absTimeout, Seq.empty[Transaction]) :+ tx)
+        context become running(tip, watches, scriptHashStatus, block2tx1, sent)
+      } else {
+        log.info(s"publishing tx=${Transaction.write(tx)}")
+        client ! BroadcastTransaction(tx)
+        context become running(tip, watches, scriptHashStatus, block2tx, sent :+ tx)
+      }
+
+    case ElectrumClient.BroadcastTransactionResponse(tx, error_opt) =>
+      error_opt match {
+        case None => log.info(s"broadcast succeeded for txid=${tx.txid} tx=${Transaction.write(tx)}")
+        case Some(error) if error.message.contains("transaction already in block chain") => log.info(s"broadcast ignored for txid=${tx.txid} tx=${Transaction.write(tx)} (tx was already in blockchain)")
+        case Some(error) => log.error(s"broadcast failed for txid=${tx.txid} tx=${Transaction.write(tx)} with error=$error")
+      }
+      context become running(tip, watches, scriptHashStatus, block2tx, sent diff Seq(tx))
+
+    case ElectrumClient.ElectrumDisconnected =>
+      // we remember watches and keep track of tx that have not yet been published
+      // we also re-send the txes that we previsouly sent but hadn't yet received the confirmation
+      context become disconnected(watches, sent.map(PublishAsap(_)), block2tx)
+  }
+
+}
+
+object ElectrumWatcher extends App {
+
+  val system = ActorSystem()
+
+  class Root extends Actor with ActorLogging {
+    val serverAddresses = Seq(new InetSocketAddress("localhost", 51000), new InetSocketAddress("localhost", 51001))
+    val client = context.actorOf(Props(new ElectrumClient(serverAddresses)), "client")
+    client ! ElectrumClient.AddStatusListener(self)
+
+    override def unhandled(message: Any): Unit = {
+      super.unhandled(message)
+      log.warning(s"unhandled message $message")
+    }
+
+    def receive = {
+      case ElectrumClient.ElectrumReady =>
+        log.info(s"starting watcher")
+        context become running(context.actorOf(Props(new ElectrumWatcher(client)), "watcher"))
+    }
+
+    def running(watcher: ActorRef): Receive = {
+      case watch: Watch => watcher forward watch
+    }
+  }
+
+  val root = system.actorOf(Props[Root], "root")
+  val scanner = new java.util.Scanner(System.in)
+  while (true) {
+    val tx = Transaction.read(scanner.nextLine())
+    root ! WatchSpent(root, tx.txid, 0, tx.txOut(0).publicKeyScript, BITCOIN_FUNDING_SPENT)
+    root ! WatchConfirmed(root, tx.txid, tx.txOut(0).publicKeyScript, 4L, BITCOIN_FUNDING_DEPTHOK)
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/fee/BitcoinCoreFeeProvider.scala

@@ -0,0 +1,44 @@
+package fr.acinq.eclair.blockchain.fee
+
+import fr.acinq.bitcoin.Btc
+import fr.acinq.eclair.blockchain.bitcoind.rpc.BitcoinJsonRPCClient
+import org.json4s.JsonAST.{JDouble, JInt}
+
+import scala.concurrent.{ExecutionContext, Future}
+
+/**
+  * Created by PM on 09/07/2017.
+  */
+class BitcoinCoreFeeProvider(rpcClient: BitcoinJsonRPCClient, defaultFeerates: FeeratesPerByte)(implicit ec: ExecutionContext) extends FeeProvider {
+
+  /**
+    * We need this to keep commitment tx fees in sync with the state of the network
+    *
+    * @param nBlocks number of blocks until tx is confirmed
+    * @return the current
+    */
+  def estimateSmartFee(nBlocks: Int): Future[Long] =
+    rpcClient.invoke("estimatesmartfee", nBlocks).map(json => {
+      json \ "feerate" match {
+        case JDouble(feerate) => Btc(feerate).toLong
+        case JInt(feerate) if feerate.toLong < 0 => feerate.toLong
+        case JInt(feerate) => Btc(feerate.toLong).toLong
+      }
+    })
+
+  override def getFeerates: Future[FeeratesPerByte] = for {
+    block_1 <- estimateSmartFee(1)
+    blocks_2 <- estimateSmartFee(2)
+    blocks_6 <- estimateSmartFee(6)
+    blocks_12 <- estimateSmartFee(12)
+    blocks_36 <- estimateSmartFee(36)
+    blocks_72 <- estimateSmartFee(72)
+  } yield FeeratesPerByte(
+    block_1 = if (block_1 > 0) block_1 else defaultFeerates.block_1,
+    blocks_2 = if (blocks_2 > 0) blocks_2 else defaultFeerates.blocks_2,
+    blocks_6 = if (blocks_6 > 0) blocks_6 else defaultFeerates.blocks_6,
+    blocks_12 = if (blocks_12 > 0) blocks_12 else defaultFeerates.blocks_12,
+    blocks_36 = if (blocks_36 > 0) blocks_36 else defaultFeerates.blocks_36,
+    blocks_72 = if (blocks_72 > 0) blocks_72 else defaultFeerates.blocks_72)
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/fee/ConstantFeeProvider.scala

@@ -0,0 +1,12 @@
+package fr.acinq.eclair.blockchain.fee
+
+import scala.concurrent.Future
+
+/**
+  * Created by PM on 09/07/2017.
+  */
+class ConstantFeeProvider(feerates: FeeratesPerByte) extends FeeProvider {
+
+  override def getFeerates: Future[FeeratesPerByte] = Future.successful(feerates)
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/fee/EarnDotComFeeProvider.scala

@@ -0,0 +1,66 @@
+package fr.acinq.eclair.blockchain.fee
+
+import akka.actor.ActorSystem
+import akka.http.scaladsl.Http
+import akka.http.scaladsl.model._
+import akka.http.scaladsl.unmarshalling.Unmarshal
+import akka.stream.ActorMaterializer
+import de.heikoseeberger.akkahttpjson4s.Json4sSupport._
+import org.json4s.JsonAST.{JArray, JInt, JValue}
+import org.json4s.{DefaultFormats, jackson}
+
+import scala.concurrent.{ExecutionContext, Future}
+
+/**
+  * Created by PM on 16/11/2017.
+  */
+class EarnDotComFeeProvider(implicit system: ActorSystem, ec: ExecutionContext) extends FeeProvider {
+
+  import EarnDotComFeeProvider._
+
+  implicit val materializer = ActorMaterializer()
+  val httpClient = Http(system)
+  implicit val serialization = jackson.Serialization
+  implicit val formats = DefaultFormats
+
+  override def getFeerates: Future[FeeratesPerByte] =
+    for {
+      httpRes <- httpClient.singleRequest(HttpRequest(uri = Uri("https://bitcoinfees.earn.com/api/v1/fees/list"), method = HttpMethods.GET))
+      json <- Unmarshal(httpRes).to[JValue]
+      feeRanges = parseFeeRanges(json)
+    } yield extractFeerates(feeRanges)
+}
+
+object EarnDotComFeeProvider {
+
+  case class FeeRange(minFee: Long, maxFee: Long, memCount: Long, minDelay: Long, maxDelay: Long)
+
+  def parseFeeRanges(json: JValue): Seq[FeeRange] = {
+    val JArray(items) = json \ "fees"
+    items.map(item => {
+      val JInt(minFee) = item \ "minFee"
+      val JInt(maxFee) = item \ "maxFee"
+      val JInt(memCount) = item \ "memCount"
+      val JInt(minDelay) = item \ "minDelay"
+      val JInt(maxDelay) = item \ "maxDelay"
+      FeeRange(minFee = minFee.toLong, maxFee = maxFee.toLong, memCount = memCount.toLong, minDelay = minDelay.toLong, maxDelay = maxDelay.toLong)
+    })
+  }
+
+  def extractFeerate(feeRanges: Seq[FeeRange], maxBlockDelay: Int): Long = {
+    // first we keep only fee ranges with a max block delay below the limit
+    val belowLimit = feeRanges.filter(_.maxDelay <= maxBlockDelay)
+    // out of all the remaining fee ranges, we select the one with the minimum higher bound
+    belowLimit.minBy(_.maxFee).maxFee
+  }
+
+  def extractFeerates(feeRanges: Seq[FeeRange]): FeeratesPerByte =
+    FeeratesPerByte(
+      block_1 = extractFeerate(feeRanges, 1),
+      blocks_2 = extractFeerate(feeRanges, 2),
+      blocks_6 = extractFeerate(feeRanges, 6),
+      blocks_12 = extractFeerate(feeRanges, 12),
+      blocks_36 = extractFeerate(feeRanges, 36),
+      blocks_72 = extractFeerate(feeRanges, 72))
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/fee/FallbackFeeProvider.scala

@@ -0,0 +1,20 @@
+package fr.acinq.eclair.blockchain.fee
+
+import scala.concurrent.{ExecutionContext, Future}
+
+/**
+  * This provider will try all child providers in sequence, until one of them works
+  */
+class FallbackFeeProvider(providers: Seq[FeeProvider])(implicit ec: ExecutionContext) extends FeeProvider {
+
+  require(providers.size >= 1, "need at least one fee provider")
+
+  def getFeerates(fallbacks: Seq[FeeProvider]): Future[FeeratesPerByte] =
+    fallbacks match {
+      case last +: Nil => last.getFeerates
+      case head +: remaining => head.getFeerates.recoverWith { case _ => getFeerates(remaining) }
+    }
+
+  override def getFeerates: Future[FeeratesPerByte] = getFeerates(providers)
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/fee/FeeProvider.scala

@@ -0,0 +1,42 @@
+package fr.acinq.eclair.blockchain.fee
+
+import fr.acinq.eclair.feerateByte2Kw
+
+import scala.concurrent.Future
+
+/**
+  * Created by PM on 09/07/2017.
+  */
+trait FeeProvider {
+
+  def getFeerates: Future[FeeratesPerByte]
+
+}
+
+case class FeeratesPerByte(block_1: Long, blocks_2: Long, blocks_6: Long, blocks_12: Long, blocks_36: Long, blocks_72: Long)
+
+case class FeeratesPerKw(block_1: Long, blocks_2: Long, blocks_6: Long, blocks_12: Long, blocks_36: Long, blocks_72: Long)
+
+object FeeratesPerKw {
+  def apply(feerates: FeeratesPerByte): FeeratesPerKw = FeeratesPerKw(
+    block_1 = feerateByte2Kw(feerates.block_1),
+    blocks_2 = feerateByte2Kw(feerates.blocks_2),
+    blocks_6 = feerateByte2Kw(feerates.blocks_6),
+    blocks_12 = feerateByte2Kw(feerates.blocks_12),
+    blocks_36 = feerateByte2Kw(feerates.blocks_36),
+    blocks_72 = feerateByte2Kw(feerates.blocks_72))
+
+  /**
+    * Used in tests
+    *
+    * @param feeratePerKw
+    * @return
+    */
+  def single(feeratePerKw: Long): FeeratesPerKw = FeeratesPerKw(
+    block_1 = feeratePerKw,
+    blocks_2 = feeratePerKw,
+    blocks_6 = feeratePerKw,
+    blocks_12 = feeratePerKw,
+    blocks_36 = feeratePerKw,
+    blocks_72 = feeratePerKw)
+}

eclair-core/src/main/scala/fr/acinq/eclair/blockchain/zmq/ZeroMQEvents.scala

@@ -1,10 +0,0 @@
-package fr.acinq.eclair.blockchain.zmq
-
-/**
-  * Created by PM on 04/04/2017.
-  */
-sealed trait ZMQEvents
-
-case object ZMQConnected extends ZMQEvents
-
-case object ZMQDisconnected extends ZMQEvents

eclair-core/src/main/scala/fr/acinq/eclair/channel/ChannelExceptions.scala

@@ -1,35 +1,43 @@
 package fr.acinq.eclair.channel
 
+import fr.acinq.bitcoin.BinaryData
+import fr.acinq.eclair.UInt64
+
 /**
   * Created by PM on 11/04/2017.
   */
 
-class ChannelException(message: String) extends RuntimeException(message)
-
-case object DebugTriggeredException extends ChannelException("debug-mode triggered failure")
-case object ClosingInProgress extends ChannelException("cannot send new htlcs, closing in progress")
-case object ClosingAlreadyInProgress extends ChannelException("closing already in progress")
-case object CannotCloseWithPendingChanges extends ChannelException("cannot close when there are pending changes")
-case object ChannelUnavailable extends ChannelException("channel is unavailable (offline or closing)")
-case object InvalidFinalScript extends ChannelException("invalid final script")
-case object HtlcTimedout extends ChannelException(s"one or more htlcs timed out")
-case class FeerateTooDifferent(localFeeratePerKw: Long, remoteFeeratePerKw: Long) extends ChannelException(s"local/remote feerates are too different: remoteFeeratePerKw=$remoteFeeratePerKw localFeeratePerKw=$localFeeratePerKw")
-case object InvalidCloseSignature extends ChannelException("cannot verify their close signature")
-case object InvalidCommitmentSignature extends ChannelException("invalid commitment signature")
-case class ForcedLocalCommit(reason: String) extends ChannelException(s"forced local commit: reason")
-case class UnexpectedHtlcId(expected: Long, actual: Long) extends ChannelException(s"unexpected htlc id: expected=$expected actual=$actual")
-case object InvalidPaymentHash extends ChannelException("invalid payment hash")
-case class ExpiryTooSmall(minimum: Long, actual: Long, blockCount: Long) extends ChannelException(s"expiry too small: required=$minimum actual=$actual blockCount=$blockCount")
-case class ExpiryCannotBeInThePast(expiry: Long, blockCount: Long) extends ChannelException(s"expiry can't be in the past: expiry=$expiry blockCount=$blockCount")
-case class HtlcValueTooSmall(minimum: Long, actual: Long) extends ChannelException(s"htlc value too small: mininmum=$minimum actual=$actual")
-case class HtlcValueTooHighInFlight(maximum: Long, actual: Long) extends ChannelException(s"in-flight htlcs hold too much value: maximum=$maximum actual=$actual")
-case class TooManyAcceptedHtlcs(maximum: Long) extends ChannelException(s"too many accepted htlcs: maximum=$maximum")
-case class InsufficientFunds(amountMsat: Long, missingSatoshis: Long, reserveSatoshis: Long, feesSatoshis: Long) extends ChannelException(s"insufficient funds: missingSatoshis=$missingSatoshis reserveSatoshis=$reserveSatoshis fees=$feesSatoshis")
-case class InvalidHtlcPreimage(id: Long) extends ChannelException(s"invalid htlc preimage for htlc id=$id")
-case class UnknownHtlcId(id: Long) extends ChannelException(s"unknown htlc id=$id")
-case object FundeeCannotSendUpdateFee extends ChannelException(s"only the funder should send update_fee messages")
-case class CannotAffordFees(missingSatoshis: Long, reserveSatoshis: Long, feesSatoshis: Long) extends ChannelException(s"can't pay the fee: missingSatoshis=$missingSatoshis reserveSatoshis=$reserveSatoshis feesSatoshis=$feesSatoshis")
-case object CannotSignWithoutChanges extends ChannelException("cannot sign when there are no changes")
-case object CannotSignBeforeRevocation extends ChannelException("cannot sign until next revocation hash is received")
-case object UnexpectedRevocation extends ChannelException("received unexpected RevokeAndAck message")
-case object InvalidRevocation extends ChannelException("invalid revocation")
+class ChannelException(channelId: BinaryData, message: String) extends RuntimeException(message)
+// @formatter:off
+case class DebugTriggeredException             (channelId: BinaryData) extends ChannelException(channelId, "debug-mode triggered failure")
+case class ChannelReserveTooHigh               (channelId: BinaryData, channelReserveSatoshis: Long, reserveToFundingRatio: Double, maxReserveToFundingRatio: Double) extends ChannelException(channelId, s"channelReserveSatoshis too high: reserve=$channelReserveSatoshis fundingRatio=$reserveToFundingRatio maxFundingRatio=$maxReserveToFundingRatio")
+case class ClosingInProgress                   (channelId: BinaryData) extends ChannelException(channelId, "cannot send new htlcs, closing in progress")
+case class ClosingAlreadyInProgress            (channelId: BinaryData) extends ChannelException(channelId, "closing already in progress")
+case class CannotCloseWithUnsignedOutgoingHtlcs(channelId: BinaryData) extends ChannelException(channelId, "cannot close when there are unsigned outgoing htlcs")
+case class ChannelUnavailable                  (channelId: BinaryData) extends ChannelException(channelId, "channel is unavailable (offline or closing)")
+case class InvalidFinalScript                  (channelId: BinaryData) extends ChannelException(channelId, "invalid final script")
+case class HtlcTimedout                        (channelId: BinaryData) extends ChannelException(channelId, s"one or more htlcs timed out")
+case class FeerateTooDifferent                 (channelId: BinaryData, localFeeratePerKw: Long, remoteFeeratePerKw: Long) extends ChannelException(channelId, s"local/remote feerates are too different: remoteFeeratePerKw=$remoteFeeratePerKw localFeeratePerKw=$localFeeratePerKw")
+case class InvalidCloseSignature               (channelId: BinaryData) extends ChannelException(channelId, "cannot verify their close signature")
+case class InvalidCommitmentSignature          (channelId: BinaryData) extends ChannelException(channelId, "invalid commitment signature")
+case class ForcedLocalCommit                   (channelId: BinaryData, reason: String) extends ChannelException(channelId, s"forced local commit: reason")
+case class UnexpectedHtlcId                    (channelId: BinaryData, expected: Long, actual: Long) extends ChannelException(channelId, s"unexpected htlc id: expected=$expected actual=$actual")
+case class InvalidPaymentHash                  (channelId: BinaryData) extends ChannelException(channelId, "invalid payment hash")
+case class ExpiryTooSmall                      (channelId: BinaryData, minimum: Long, actual: Long, blockCount: Long) extends ChannelException(channelId, s"expiry too small: required=$minimum actual=$actual blockCount=$blockCount")
+case class ExpiryCannotBeInThePast             (channelId: BinaryData, expiry: Long, blockCount: Long) extends ChannelException(channelId, s"expiry can't be in the past: expiry=$expiry blockCount=$blockCount")
+case class HtlcValueTooSmall                   (channelId: BinaryData, minimum: Long, actual: Long) extends ChannelException(channelId, s"htlc value too small: minimum=$minimum actual=$actual")
+case class HtlcValueTooHighInFlight            (channelId: BinaryData, maximum: UInt64, actual: UInt64) extends ChannelException(channelId, s"in-flight htlcs hold too much value: maximum=$maximum actual=$actual")
+case class TooManyAcceptedHtlcs                (channelId: BinaryData, maximum: Long) extends ChannelException(channelId, s"too many accepted htlcs: maximum=$maximum")
+case class InsufficientFunds                   (channelId: BinaryData, amountMsat: Long, missingSatoshis: Long, reserveSatoshis: Long, feesSatoshis: Long) extends ChannelException(channelId, s"insufficient funds: missingSatoshis=$missingSatoshis reserveSatoshis=$reserveSatoshis fees=$feesSatoshis")
+case class InvalidHtlcPreimage                 (channelId: BinaryData, id: Long) extends ChannelException(channelId, s"invalid htlc preimage for htlc id=$id")
+case class UnknownHtlcId                       (channelId: BinaryData, id: Long) extends ChannelException(channelId, s"unknown htlc id=$id")
+case class FundeeCannotSendUpdateFee           (channelId: BinaryData) extends ChannelException(channelId, s"only the funder should send update_fee messages")
+case class CannotAffordFees                    (channelId: BinaryData, missingSatoshis: Long, reserveSatoshis: Long, feesSatoshis: Long) extends ChannelException(channelId, s"can't pay the fee: missingSatoshis=$missingSatoshis reserveSatoshis=$reserveSatoshis feesSatoshis=$feesSatoshis")
+case class CannotSignWithoutChanges            (channelId: BinaryData) extends ChannelException(channelId, "cannot sign when there are no changes")
+case class CannotSignBeforeRevocation          (channelId: BinaryData) extends ChannelException(channelId, "cannot sign until next revocation hash is received")
+case class UnexpectedRevocation                (channelId: BinaryData) extends ChannelException(channelId, "received unexpected RevokeAndAck message")
+case class InvalidRevocation                   (channelId: BinaryData) extends ChannelException(channelId, "invalid revocation")
+case class CommitmentSyncError                 (channelId: BinaryData) extends ChannelException(channelId, "commitment sync error")
+case class RevocationSyncError                 (channelId: BinaryData) extends ChannelException(channelId, "revocation sync error")
+case class InvalidFailureCode                  (channelId: BinaryData) extends ChannelException(channelId, "UpdateFailMalformedHtlc message doesn't have BADONION bit set")
+// @formatter:on

eclair-core/src/main/scala/fr/acinq/eclair/channel/ChannelTypes.scala

@@ -2,12 +2,12 @@ package fr.acinq.eclair.channel
 
 import akka.actor.ActorRef
 import fr.acinq.bitcoin.Crypto.{Point, PrivateKey, PublicKey, Scalar}
-import fr.acinq.bitcoin.{BinaryData, Transaction}
-import fr.acinq.eclair.blockchain.MakeFundingTxResponse
+import fr.acinq.bitcoin.{BinaryData, OutPoint, Transaction}
+import fr.acinq.eclair.UInt64
 import fr.acinq.eclair.crypto.Sphinx
 import fr.acinq.eclair.transactions.CommitmentSpec
 import fr.acinq.eclair.transactions.Transactions.CommitTx
-import fr.acinq.eclair.wire.{AcceptChannel, ClosingSigned, FailureMessage, FundingCreated, FundingLocked, FundingSigned, Init, OpenChannel, Shutdown, UpdateAddHtlc}
+import fr.acinq.eclair.wire.{AcceptChannel, AnnouncementSignatures, ClosingSigned, FailureMessage, FundingCreated, FundingLocked, FundingSigned, Init, OpenChannel, Shutdown, UpdateAddHtlc}
 
 
 /**
@@ -31,9 +31,9 @@ case object WAIT_FOR_INIT_INTERNAL extends State
 case object WAIT_FOR_OPEN_CHANNEL extends State
 case object WAIT_FOR_ACCEPT_CHANNEL extends State
 case object WAIT_FOR_FUNDING_INTERNAL extends State
-case object WAIT_FOR_FUNDING_PARENT extends State
 case object WAIT_FOR_FUNDING_CREATED extends State
 case object WAIT_FOR_FUNDING_SIGNED extends State
+case object WAIT_FOR_FUNDING_PUBLISHED extends State
 case object WAIT_FOR_FUNDING_CONFIRMED extends State
 case object WAIT_FOR_FUNDING_LOCKED extends State
 case object NORMAL extends State
@@ -42,6 +42,8 @@ case object NEGOTIATING extends State
 case object CLOSING extends State
 case object CLOSED extends State
 case object OFFLINE extends State
+case object SYNCING extends State
+case object ERR_FUNDING_PUBLISH_FAILED extends State
 case object ERR_FUNDING_LOST extends State
 case object ERR_FUNDING_TIMEOUT extends State
 case object ERR_INFORMATION_LEAK extends State
@@ -57,7 +59,7 @@ case object ERR_INFORMATION_LEAK extends State
       8888888888     Y8P     8888888888 888    Y888     888     "Y8888P"
  */
 
-case class INPUT_INIT_FUNDER(temporaryChannelId: BinaryData, fundingSatoshis: Long, pushMsat: Long, initialFeeratePerKw: Long, localParams: LocalParams, remote: ActorRef, remoteInit: Init)
+case class INPUT_INIT_FUNDER(temporaryChannelId: BinaryData, fundingSatoshis: Long, pushMsat: Long, initialFeeratePerKw: Long, localParams: LocalParams, remote: ActorRef, remoteInit: Init, channelFlags: Byte)
 case class INPUT_INIT_FUNDEE(temporaryChannelId: BinaryData, localParams: LocalParams, remote: ActorRef, remoteInit: Init)
 case object INPUT_CLOSE_COMPLETE_TIMEOUT // when requesting a mutual close, we wait for as much as this timeout, then unilateral close
 case object INPUT_PUBLISH_LOCALCOMMIT // used in tests
@@ -66,21 +68,16 @@ case class INPUT_RECONNECTED(remote: ActorRef)
 case class INPUT_RESTORED(data: HasCommitments)
 
 sealed trait BitcoinEvent
+case object BITCOIN_FUNDING_PUBLISH_FAILED extends BitcoinEvent
 case object BITCOIN_FUNDING_DEPTHOK extends BitcoinEvent
 case object BITCOIN_FUNDING_DEEPLYBURIED extends BitcoinEvent
 case object BITCOIN_FUNDING_LOST extends BitcoinEvent
 case object BITCOIN_FUNDING_TIMEOUT extends BitcoinEvent
 case object BITCOIN_FUNDING_SPENT extends BitcoinEvent
-case object BITCOIN_HTLC_SPENT extends BitcoinEvent
-case object BITCOIN_LOCALCOMMIT_DONE extends BitcoinEvent
-case object BITCOIN_REMOTECOMMIT_DONE extends BitcoinEvent
-case object BITCOIN_NEXTREMOTECOMMIT_DONE extends BitcoinEvent
-case object BITCOIN_PENALTY_DONE extends BitcoinEvent
-case object BITCOIN_CLOSE_DONE extends BitcoinEvent
-case class BITCOIN_FUNDING_OTHER_CHANNEL_SPENT(shortChannelId: Long) extends BitcoinEvent
+case object BITCOIN_OUTPUT_SPENT extends BitcoinEvent
 case class BITCOIN_TX_CONFIRMED(tx: Transaction) extends BitcoinEvent
+case class BITCOIN_FUNDING_EXTERNAL_CHANNEL_SPENT(shortChannelId: Long) extends BitcoinEvent
 case class BITCOIN_PARENT_TX_CONFIRMED(childTx: Transaction) extends BitcoinEvent
-case class BITCOIN_INPUT_SPENT(tx: Transaction) extends BitcoinEvent
 
 /*
        .d8888b.   .d88888b.  888b     d888 888b     d888        d8888 888b    888 8888888b.   .d8888b.
@@ -126,19 +123,22 @@ trait HasCommitments extends Data {
   def channelId = commitments.channelId
 }
 
-case class LocalCommitPublished(commitTx: Transaction, claimMainDelayedOutputTx: Option[Transaction], htlcSuccessTxs: Seq[Transaction], htlcTimeoutTxs: Seq[Transaction], claimHtlcDelayedTx: Seq[Transaction])
-case class RemoteCommitPublished(commitTx: Transaction, claimMainOutputTx: Option[Transaction], claimHtlcSuccessTxs: Seq[Transaction], claimHtlcTimeoutTxs: Seq[Transaction])
-case class RevokedCommitPublished(commitTx: Transaction, claimMainOutputTx: Option[Transaction], mainPenaltyTx: Option[Transaction], claimHtlcTimeoutTxs: Seq[Transaction], htlcTimeoutTxs: Seq[Transaction], htlcPenaltyTxs: Seq[Transaction])
+case class LocalCommitPublished(commitTx: Transaction, claimMainDelayedOutputTx: Option[Transaction], htlcSuccessTxs: List[Transaction], htlcTimeoutTxs: List[Transaction], claimHtlcDelayedTx: List[Transaction], spent: Map[OutPoint, BinaryData])
+case class RemoteCommitPublished(commitTx: Transaction, claimMainOutputTx: Option[Transaction], claimHtlcSuccessTxs: List[Transaction], claimHtlcTimeoutTxs: List[Transaction], spent: Map[OutPoint, BinaryData])
+case class RevokedCommitPublished(commitTx: Transaction, claimMainOutputTx: Option[Transaction], mainPenaltyTx: Option[Transaction], claimHtlcTimeoutTxs: List[Transaction], htlcTimeoutTxs: List[Transaction], htlcPenaltyTxs: List[Transaction], spent: Map[OutPoint, BinaryData])
 
 final case class DATA_WAIT_FOR_OPEN_CHANNEL(initFundee: INPUT_INIT_FUNDEE) extends Data
 final case class DATA_WAIT_FOR_ACCEPT_CHANNEL(initFunder: INPUT_INIT_FUNDER, lastSent: OpenChannel) extends Data
 final case class DATA_WAIT_FOR_FUNDING_INTERNAL(temporaryChannelId: BinaryData, localParams: LocalParams, remoteParams: RemoteParams, fundingSatoshis: Long, pushMsat: Long, initialFeeratePerKw: Long, remoteFirstPerCommitmentPoint: Point, lastSent: OpenChannel) extends Data
-final case class DATA_WAIT_FOR_FUNDING_PARENT(fundingResponse: MakeFundingTxResponse, parentCandidates: Set[Transaction], data: DATA_WAIT_FOR_FUNDING_INTERNAL) extends Data
-final case class DATA_WAIT_FOR_FUNDING_CREATED(temporaryChannelId: BinaryData, localParams: LocalParams, remoteParams: RemoteParams, fundingSatoshis: Long, pushMsat: Long, initialFeeratePerKw: Long, remoteFirstPerCommitmentPoint: Point, lastSent: AcceptChannel) extends Data
-final case class DATA_WAIT_FOR_FUNDING_SIGNED(channelId: BinaryData, localParams: LocalParams, remoteParams: RemoteParams, fundingTx: Transaction, localSpec: CommitmentSpec, localCommitTx: CommitTx, remoteCommit: RemoteCommit, lastSent: FundingCreated) extends Data
+final case class DATA_WAIT_FOR_FUNDING_CREATED(temporaryChannelId: BinaryData, localParams: LocalParams, remoteParams: RemoteParams, fundingSatoshis: Long, pushMsat: Long, initialFeeratePerKw: Long, remoteFirstPerCommitmentPoint: Point, channelFlags: Byte, lastSent: AcceptChannel) extends Data
+final case class DATA_WAIT_FOR_FUNDING_SIGNED(channelId: BinaryData, localParams: LocalParams, remoteParams: RemoteParams, fundingTx: Transaction, localSpec: CommitmentSpec, localCommitTx: CommitTx, remoteCommit: RemoteCommit, channelFlags: Byte, lastSent: FundingCreated) extends Data
 final case class DATA_WAIT_FOR_FUNDING_CONFIRMED(commitments: Commitments, deferred: Option[FundingLocked], lastSent: Either[FundingCreated, FundingSigned]) extends Data with HasCommitments
 final case class DATA_WAIT_FOR_FUNDING_LOCKED(commitments: Commitments, lastSent: FundingLocked) extends Data with HasCommitments
-final case class DATA_NORMAL(commitments: Commitments, shortChannelId: Option[Long]) extends Data with HasCommitments
+final case class DATA_NORMAL(commitments: Commitments,
+                             shortChannelId: Option[Long],
+                             localAnnouncementSignatures: Option[AnnouncementSignatures],
+                             localShutdown: Option[Shutdown],
+                             remoteShutdown: Option[Shutdown]) extends Data with HasCommitments
 final case class DATA_SHUTDOWN(commitments: Commitments,
                                localShutdown: Shutdown, remoteShutdown: Shutdown) extends Data with HasCommitments
 final case class DATA_NEGOTIATING(commitments: Commitments,
@@ -148,30 +148,37 @@ final case class DATA_CLOSING(commitments: Commitments,
                               localCommitPublished: Option[LocalCommitPublished] = None,
                               remoteCommitPublished: Option[RemoteCommitPublished] = None,
                               nextRemoteCommitPublished: Option[RemoteCommitPublished] = None,
-                              revokedCommitPublished: Seq[RevokedCommitPublished] = Nil) extends Data with HasCommitments {
+                              revokedCommitPublished: List[RevokedCommitPublished] = Nil) extends Data with HasCommitments {
   require(mutualClosePublished.isDefined || localCommitPublished.isDefined || remoteCommitPublished.isDefined || nextRemoteCommitPublished.isDefined || revokedCommitPublished.size > 0, "there should be at least one tx published in this state")
 }
 
 final case class LocalParams(nodeId: PublicKey,
                              dustLimitSatoshis: Long,
-                             maxHtlcValueInFlightMsat: Long,
+                             maxHtlcValueInFlightMsat: UInt64,
                              channelReserveSatoshis: Long,
                              htlcMinimumMsat: Long,
                              toSelfDelay: Int,
                              maxAcceptedHtlcs: Int,
                              fundingPrivKey: PrivateKey,
                              revocationSecret: Scalar,
-                             paymentKey: PrivateKey,
+                             paymentKey: Scalar,
                              delayedPaymentKey: Scalar,
+                             htlcKey: Scalar,
                              defaultFinalScriptPubKey: BinaryData,
                              shaSeed: BinaryData,
                              isFunder: Boolean,
                              globalFeatures: BinaryData,
-                             localFeatures: BinaryData)
+                             localFeatures: BinaryData) {
+  // precomputed for performance reasons
+  val paymentBasepoint = paymentKey.toPoint
+  val delayedPaymentBasepoint = delayedPaymentKey.toPoint
+  val revocationBasepoint = revocationSecret.toPoint
+  val htlcBasepoint = htlcKey.toPoint
+}
 
 final case class RemoteParams(nodeId: PublicKey,
                               dustLimitSatoshis: Long,
-                              maxHtlcValueInFlightMsat: Long,
+                              maxHtlcValueInFlightMsat: UInt64,
                               channelReserveSatoshis: Long,
                               htlcMinimumMsat: Long,
                               toSelfDelay: Int,
@@ -180,7 +187,12 @@ final case class RemoteParams(nodeId: PublicKey,
                               revocationBasepoint: Point,
                               paymentBasepoint: Point,
                               delayedPaymentBasepoint: Point,
+                              htlcBasepoint: Point,
                               globalFeatures: BinaryData,
                               localFeatures: BinaryData)
 
+object ChannelFlags {
+  val AnnounceChannel = 0x01.toByte
+  val Empty = 0x00.toByte
+}
 // @formatter:on

eclair-core/src/main/scala/fr/acinq/eclair/channel/Commitments.scala

@@ -2,11 +2,12 @@ package fr.acinq.eclair.channel
 
 import fr.acinq.bitcoin.Crypto.{Point, PrivateKey, sha256}
 import fr.acinq.bitcoin.{BinaryData, Crypto, Satoshi, Transaction}
-import fr.acinq.eclair.Globals
 import fr.acinq.eclair.crypto.{Generators, ShaChain, Sphinx}
+import fr.acinq.eclair.payment.Origin
 import fr.acinq.eclair.transactions.Transactions._
 import fr.acinq.eclair.transactions._
 import fr.acinq.eclair.wire._
+import fr.acinq.eclair.{Globals, UInt64}
 import grizzled.slf4j.Logging
 
 // @formatter:off
@@ -16,10 +17,10 @@ case class LocalChanges(proposed: List[UpdateMessage], signed: List[UpdateMessag
 case class RemoteChanges(proposed: List[UpdateMessage], acked: List[UpdateMessage], signed: List[UpdateMessage])
 case class Changes(ourChanges: LocalChanges, theirChanges: RemoteChanges)
 case class HtlcTxAndSigs(txinfo: TransactionWithInputInfo, localSig: BinaryData, remoteSig: BinaryData)
-case class PublishableTxs(commitTx: CommitTx, htlcTxsAndSigs: Seq[HtlcTxAndSigs])
-case class LocalCommit(index: Long, spec: CommitmentSpec, publishableTxs: PublishableTxs, commit: CommitSig)
+case class PublishableTxs(commitTx: CommitTx, htlcTxsAndSigs: List[HtlcTxAndSigs])
+case class LocalCommit(index: Long, spec: CommitmentSpec, publishableTxs: PublishableTxs)
 case class RemoteCommit(index: Long, spec: CommitmentSpec, txid: BinaryData, remotePerCommitmentPoint: Point)
-case class WaitingForRevocation(nextRemoteCommit: RemoteCommit, sent: CommitSig, reSignAsap: Boolean = false)
+case class WaitingForRevocation(nextRemoteCommit: RemoteCommit, sent: CommitSig, sentAfterLocalCommitIndex: Long, reSignAsap: Boolean = false)
 // @formatter:on
 
 /**
@@ -31,15 +32,16 @@ case class WaitingForRevocation(nextRemoteCommit: RemoteCommit, sent: CommitSig,
   * theirNextCommitInfo is their next commit tx. The rest of the time, it is their next per-commitment point
   */
 case class Commitments(localParams: LocalParams, remoteParams: RemoteParams,
+                       channelFlags: Byte,
                        localCommit: LocalCommit, remoteCommit: RemoteCommit,
                        localChanges: LocalChanges, remoteChanges: RemoteChanges,
                        localNextHtlcId: Long, remoteNextHtlcId: Long,
+                       originChannels: Map[Long, Origin], // for outgoing htlcs relayed through us, the id of the previous channel
                        remoteNextCommitInfo: Either[WaitingForRevocation, Point],
-                       unackedMessages: Seq[LightningMessage],
                        commitInput: InputInfo,
                        remotePerCommitmentSecrets: ShaChain, channelId: BinaryData) {
 
-  def hasNoPendingHtlcs: Boolean = localCommit.spec.htlcs.isEmpty && remoteCommit.spec.htlcs.isEmpty
+  def hasNoPendingHtlcs: Boolean = localCommit.spec.htlcs.isEmpty && remoteCommit.spec.htlcs.isEmpty && remoteNextCommitInfo.isRight
 
   def hasTimedoutOutgoingHtlcs(blockheight: Long): Boolean =
     localCommit.spec.htlcs.exists(htlc => htlc.direction == OUT && blockheight >= htlc.add.expiry) ||
@@ -49,9 +51,7 @@ case class Commitments(localParams: LocalParams, remoteParams: RemoteParams,
 
   def addRemoteProposal(proposal: UpdateMessage): Commitments = Commitments.addRemoteProposal(this, proposal)
 
-  def addToUnackedMessages(message: LightningMessage): Commitments = this.copy(unackedMessages = unackedMessages :+ message)
-
-  def unackedShutdown(): Option[Shutdown] = this.unackedMessages.collectFirst { case d: Shutdown => d }
+  def announceChannel: Boolean = (channelFlags & 0x01) != 0
 }
 
 object Commitments extends Logging {
@@ -63,9 +63,7 @@ object Commitments extends Logging {
     * @return an updated commitment instance
     */
   private def addLocalProposal(commitments: Commitments, proposal: UpdateMessage): Commitments =
-    commitments.copy(
-      localChanges = commitments.localChanges.copy(proposed = commitments.localChanges.proposed :+ proposal),
-      unackedMessages = commitments.unackedMessages :+ proposal)
+    commitments.copy(localChanges = commitments.localChanges.copy(proposed = commitments.localChanges.proposed :+ proposal))
 
   private def addRemoteProposal(commitments: Commitments, proposal: UpdateMessage): Commitments =
     commitments.copy(remoteChanges = commitments.remoteChanges.copy(proposed = commitments.remoteChanges.proposed :+ proposal))
@@ -76,36 +74,39 @@ object Commitments extends Logging {
     * @param cmd         add HTLC command
     * @return either Left(failure, error message) where failure is a failure message (see BOLT #4 and the Failure Message class) or Right((new commitments, updateAddHtlc)
     */
-  def sendAdd(commitments: Commitments, cmd: CMD_ADD_HTLC): Either[ChannelException, (Commitments, UpdateAddHtlc)] = {
+  def sendAdd(commitments: Commitments, cmd: CMD_ADD_HTLC, origin: Origin): Either[ChannelException, (Commitments, UpdateAddHtlc)] = {
 
     if (cmd.paymentHash.size != 32) {
-      return Left(InvalidPaymentHash)
+      return Left(InvalidPaymentHash(commitments.channelId))
     }
 
     val blockCount = Globals.blockCount.get()
     if (cmd.expiry <= blockCount) {
-      return Left(ExpiryCannotBeInThePast(cmd.expiry, blockCount))
+      return Left(ExpiryCannotBeInThePast(commitments.channelId, cmd.expiry, blockCount))
     }
 
     if (cmd.amountMsat < commitments.remoteParams.htlcMinimumMsat) {
-      return Left(HtlcValueTooSmall(minimum = commitments.remoteParams.htlcMinimumMsat, actual = cmd.amountMsat))
+      return Left(HtlcValueTooSmall(commitments.channelId, minimum = commitments.remoteParams.htlcMinimumMsat, actual = cmd.amountMsat))
     }
 
     // let's compute the current commitment *as seen by them* with this change taken into account
-    val add = UpdateAddHtlc(commitments.channelId, commitments.localNextHtlcId, cmd.amountMsat, cmd.expiry, cmd.paymentHash, cmd.onion)
-    val commitments1 = addLocalProposal(commitments, add).copy(localNextHtlcId = commitments.localNextHtlcId + 1)
-    val reduced = CommitmentSpec.reduce(commitments1.remoteCommit.spec, commitments1.remoteChanges.acked, commitments1.localChanges.proposed)
+    val add = UpdateAddHtlc(commitments.channelId, commitments.localNextHtlcId, cmd.amountMsat, cmd.paymentHash, cmd.expiry, cmd.onion)
+    // we increment the local htlc index and add an entry to the origins map
+    val commitments1 = addLocalProposal(commitments, add).copy(localNextHtlcId = commitments.localNextHtlcId + 1, originChannels = commitments.originChannels + (add.id -> origin))
+    // we need to base the next current commitment on the last sig we sent, even if we didn't yet receive their revocation
+    val remoteCommit1 = commitments1.remoteNextCommitInfo.left.toOption.map(_.nextRemoteCommit).getOrElse(commitments1.remoteCommit)
+    val reduced = CommitmentSpec.reduce(remoteCommit1.spec, commitments1.remoteChanges.acked, commitments1.localChanges.proposed)
 
-    val htlcValueInFlight = reduced.htlcs.map(_.add.amountMsat).sum
+    val htlcValueInFlight = UInt64(reduced.htlcs.map(_.add.amountMsat).sum)
     if (htlcValueInFlight > commitments1.remoteParams.maxHtlcValueInFlightMsat) {
       // TODO: this should be a specific UPDATE error
-      return Left(HtlcValueTooHighInFlight(maximum = commitments1.remoteParams.maxHtlcValueInFlightMsat, actual = htlcValueInFlight))
+      return Left(HtlcValueTooHighInFlight(commitments.channelId, maximum = commitments1.remoteParams.maxHtlcValueInFlightMsat, actual = htlcValueInFlight))
     }
 
     // the HTLC we are about to create is outgoing, but from their point of view it is incoming
     val acceptedHtlcs = reduced.htlcs.count(_.direction == IN)
     if (acceptedHtlcs > commitments1.remoteParams.maxAcceptedHtlcs) {
-      return Left(TooManyAcceptedHtlcs(maximum = commitments1.remoteParams.maxAcceptedHtlcs))
+      return Left(TooManyAcceptedHtlcs(commitments.channelId, maximum = commitments1.remoteParams.maxAcceptedHtlcs))
     }
 
     // a node cannot spend pending incoming htlcs, and need to keep funds above the reserve required by the counterparty, after paying the fee
@@ -113,71 +114,60 @@ object Commitments extends Logging {
     val fees = if (commitments1.localParams.isFunder) Transactions.commitTxFee(Satoshi(commitments1.remoteParams.dustLimitSatoshis), reduced).amount else 0
     val missing = reduced.toRemoteMsat / 1000 - commitments1.remoteParams.channelReserveSatoshis - fees
     if (missing < 0) {
-      return Left(InsufficientFunds(amountMsat = cmd.amountMsat, missingSatoshis = -1 * missing, reserveSatoshis = commitments1.remoteParams.channelReserveSatoshis, feesSatoshis = fees))
+      return Left(InsufficientFunds(commitments.channelId, amountMsat = cmd.amountMsat, missingSatoshis = -1 * missing, reserveSatoshis = commitments1.remoteParams.channelReserveSatoshis, feesSatoshis = fees))
     }
 
     Right(commitments1, add)
   }
 
-  def isOldAdd(commitments: Commitments, add: UpdateAddHtlc): Boolean = {
-    add.id < commitments.remoteNextHtlcId
-  }
-
   def receiveAdd(commitments: Commitments, add: UpdateAddHtlc): Commitments = {
-    isOldAdd(commitments, add) match {
-      case true => commitments
-      case false =>
-
-        if (add.id != commitments.remoteNextHtlcId) {
-          throw UnexpectedHtlcId(expected = commitments.remoteNextHtlcId, actual = add.id)
-        }
-
-        if (add.paymentHash.size != 32) {
-          throw InvalidPaymentHash
-        }
-
-        val blockCount = Globals.blockCount.get()
-        // we need a reasonable amount of time to pull the funds before the sender can get refunded
-        val minExpiry = blockCount + 3
-        if (add.expiry < minExpiry) {
-          throw ExpiryTooSmall(minimum = minExpiry, actual = add.expiry, blockCount = blockCount)
-        }
-
-        if (add.amountMsat < commitments.localParams.htlcMinimumMsat) {
-          throw HtlcValueTooSmall(minimum = commitments.localParams.htlcMinimumMsat, actual = add.amountMsat)
-        }
-
-        // let's compute the current commitment *as seen by us* including this change
-        val commitments1 = addRemoteProposal(commitments, add).copy(remoteNextHtlcId = commitments.remoteNextHtlcId + 1)
-        val reduced = CommitmentSpec.reduce(commitments1.localCommit.spec, commitments1.localChanges.acked, commitments1.remoteChanges.proposed)
-
-        val htlcValueInFlight = reduced.htlcs.map(_.add.amountMsat).sum
-        if (htlcValueInFlight > commitments1.localParams.maxHtlcValueInFlightMsat) {
-          throw HtlcValueTooHighInFlight(maximum = commitments1.localParams.maxHtlcValueInFlightMsat, actual = htlcValueInFlight)
-        }
-
-        val acceptedHtlcs = reduced.htlcs.count(_.direction == IN)
-        if (acceptedHtlcs > commitments1.localParams.maxAcceptedHtlcs) {
-          throw TooManyAcceptedHtlcs(maximum = commitments1.localParams.maxAcceptedHtlcs)
-        }
-
-        // a node cannot spend pending incoming htlcs, and need to keep funds above the reserve required by the counterparty, after paying the fee
-        val fees = if (commitments1.localParams.isFunder) 0 else Transactions.commitTxFee(Satoshi(commitments1.localParams.dustLimitSatoshis), reduced).amount
-        val missing = reduced.toRemoteMsat / 1000 - commitments1.localParams.channelReserveSatoshis - fees
-        if (missing < 0) {
-          throw InsufficientFunds(amountMsat = add.amountMsat, missingSatoshis = -1 * missing, reserveSatoshis = commitments1.localParams.channelReserveSatoshis, feesSatoshis = fees)
-        }
-
-        commitments1
+    if (add.id != commitments.remoteNextHtlcId) {
+      throw UnexpectedHtlcId(commitments.channelId, expected = commitments.remoteNextHtlcId, actual = add.id)
     }
+
+    if (add.paymentHash.size != 32) {
+      throw InvalidPaymentHash(commitments.channelId)
+    }
+
+    val blockCount = Globals.blockCount.get()
+    // we need a reasonable amount of time to pull the funds before the sender can get refunded
+    val minExpiry = blockCount + 3
+    if (add.expiry < minExpiry) {
+      throw ExpiryTooSmall(commitments.channelId, minimum = minExpiry, actual = add.expiry, blockCount = blockCount)
+    }
+
+    if (add.amountMsat < commitments.localParams.htlcMinimumMsat) {
+      throw HtlcValueTooSmall(commitments.channelId, minimum = commitments.localParams.htlcMinimumMsat, actual = add.amountMsat)
+    }
+
+    // let's compute the current commitment *as seen by us* including this change
+    val commitments1 = addRemoteProposal(commitments, add).copy(remoteNextHtlcId = commitments.remoteNextHtlcId + 1)
+    val reduced = CommitmentSpec.reduce(commitments1.localCommit.spec, commitments1.localChanges.acked, commitments1.remoteChanges.proposed)
+
+    val htlcValueInFlight = UInt64(reduced.htlcs.map(_.add.amountMsat).sum)
+    if (htlcValueInFlight > commitments1.localParams.maxHtlcValueInFlightMsat) {
+      throw HtlcValueTooHighInFlight(commitments.channelId, maximum = commitments1.localParams.maxHtlcValueInFlightMsat, actual = htlcValueInFlight)
+    }
+
+    val acceptedHtlcs = reduced.htlcs.count(_.direction == IN)
+    if (acceptedHtlcs > commitments1.localParams.maxAcceptedHtlcs) {
+      throw TooManyAcceptedHtlcs(commitments.channelId, maximum = commitments1.localParams.maxAcceptedHtlcs)
+    }
+
+    // a node cannot spend pending incoming htlcs, and need to keep funds above the reserve required by the counterparty, after paying the fee
+    val fees = if (commitments1.localParams.isFunder) 0 else Transactions.commitTxFee(Satoshi(commitments1.localParams.dustLimitSatoshis), reduced).amount
+    val missing = reduced.toRemoteMsat / 1000 - commitments1.localParams.channelReserveSatoshis - fees
+    if (missing < 0) {
+      throw InsufficientFunds(commitments.channelId, amountMsat = add.amountMsat, missingSatoshis = -1 * missing, reserveSatoshis = commitments1.localParams.channelReserveSatoshis, feesSatoshis = fees)
+    }
+
+    commitments1
   }
 
   def getHtlcCrossSigned(commitments: Commitments, directionRelativeToLocal: Direction, htlcId: Long): Option[UpdateAddHtlc] = {
     val remoteSigned = commitments.localCommit.spec.htlcs.find(htlc => htlc.direction == directionRelativeToLocal && htlc.add.id == htlcId)
-    val localSigned = commitments.remoteNextCommitInfo match {
-      case Left(waitingForRevocation) => waitingForRevocation.nextRemoteCommit.spec.htlcs.find(htlc => htlc.direction == directionRelativeToLocal.opposite && htlc.add.id == htlcId)
-      case Right(_) => commitments.remoteCommit.spec.htlcs.find(htlc => htlc.direction == directionRelativeToLocal.opposite && htlc.add.id == htlcId)
-    }
+    val localSigned = commitments.remoteNextCommitInfo.left.toOption.map(_.nextRemoteCommit).getOrElse(commitments.remoteCommit)
+      .spec.htlcs.find(htlc => htlc.direction == directionRelativeToLocal.opposite && htlc.add.id == htlcId)
     for {
       htlc_out <- remoteSigned
       htlc_in <- localSigned
@@ -186,31 +176,39 @@ object Commitments extends Logging {
 
   def sendFulfill(commitments: Commitments, cmd: CMD_FULFILL_HTLC): (Commitments, UpdateFulfillHtlc) =
     getHtlcCrossSigned(commitments, IN, cmd.id) match {
+      case Some(htlc) if commitments.localChanges.proposed.exists {
+        case u: UpdateFulfillHtlc if htlc.id == u.id => true
+        case u: UpdateFailHtlc if htlc.id == u.id => true
+        case u: UpdateFailMalformedHtlc if htlc.id == u.id => true
+        case _ => false
+      } =>
+        // we have already sent a fail/fulfill for this htlc
+        throw UnknownHtlcId(commitments.channelId, cmd.id)
       case Some(htlc) if htlc.paymentHash == sha256(cmd.r) =>
         val fulfill = UpdateFulfillHtlc(commitments.channelId, cmd.id, cmd.r)
         val commitments1 = addLocalProposal(commitments, fulfill)
         (commitments1, fulfill)
-      case Some(htlc) => throw InvalidHtlcPreimage(cmd.id)
-      case None => throw UnknownHtlcId(cmd.id)
+      case Some(htlc) => throw InvalidHtlcPreimage(commitments.channelId, cmd.id)
+      case None => throw UnknownHtlcId(commitments.channelId, cmd.id)
     }
 
-  def isOldFulfill(commitments: Commitments, fulfill: UpdateFulfillHtlc): Boolean =
-    commitments.remoteChanges.proposed.contains(fulfill) ||
-      commitments.remoteChanges.signed.contains(fulfill) ||
-      commitments.remoteChanges.acked.contains(fulfill)
-
-  def receiveFulfill(commitments: Commitments, fulfill: UpdateFulfillHtlc): Either[Commitments, Commitments] =
-    isOldFulfill(commitments, fulfill) match {
-      case true => Left(commitments)
-      case false => getHtlcCrossSigned(commitments, OUT, fulfill.id) match {
-        case Some(htlc) if htlc.paymentHash == sha256(fulfill.paymentPreimage) => Right(addRemoteProposal(commitments, fulfill))
-        case Some(htlc) => throw InvalidHtlcPreimage(fulfill.id)
-        case None => throw UnknownHtlcId(fulfill.id)
-      }
+  def receiveFulfill(commitments: Commitments, fulfill: UpdateFulfillHtlc): Either[Commitments, (Commitments, Origin)] =
+    getHtlcCrossSigned(commitments, OUT, fulfill.id) match {
+      case Some(htlc) if htlc.paymentHash == sha256(fulfill.paymentPreimage) => Right((addRemoteProposal(commitments, fulfill), commitments.originChannels(fulfill.id)))
+      case Some(htlc) => throw InvalidHtlcPreimage(commitments.channelId, fulfill.id)
+      case None => throw UnknownHtlcId(commitments.channelId, fulfill.id)
     }
 
   def sendFail(commitments: Commitments, cmd: CMD_FAIL_HTLC, nodeSecret: PrivateKey): (Commitments, UpdateFailHtlc) =
     getHtlcCrossSigned(commitments, IN, cmd.id) match {
+      case Some(htlc) if commitments.localChanges.proposed.exists {
+        case u: UpdateFulfillHtlc if htlc.id == u.id => true
+        case u: UpdateFailHtlc if htlc.id == u.id => true
+        case u: UpdateFailMalformedHtlc if htlc.id == u.id => true
+        case _ => false
+      } =>
+        // we have already sent a fail/fulfill for this htlc
+        throw UnknownHtlcId(commitments.channelId, cmd.id)
       case Some(htlc) =>
         // we need the shared secret to build the error packet
         val sharedSecret = Sphinx.parsePacket(nodeSecret, htlc.paymentHash, htlc.onionRoutingPacket).sharedSecret
@@ -221,49 +219,52 @@ object Commitments extends Logging {
         val fail = UpdateFailHtlc(commitments.channelId, cmd.id, reason)
         val commitments1 = addLocalProposal(commitments, fail)
         (commitments1, fail)
-      case None => throw UnknownHtlcId(cmd.id)
+      case None => throw UnknownHtlcId(commitments.channelId, cmd.id)
     }
 
-  def sendFailMalformed(commitments: Commitments, cmd: CMD_FAIL_MALFORMED_HTLC): (Commitments, UpdateFailMalformedHtlc) =
+  def sendFailMalformed(commitments: Commitments, cmd: CMD_FAIL_MALFORMED_HTLC): (Commitments, UpdateFailMalformedHtlc) = {
+    // BADONION bit must be set in failure_code
+    if ((cmd.failureCode & FailureMessageCodecs.BADONION) == 0) {
+      throw InvalidFailureCode(commitments.channelId)
+    }
     getHtlcCrossSigned(commitments, IN, cmd.id) match {
+      case Some(htlc) if commitments.localChanges.proposed.exists {
+        case u: UpdateFulfillHtlc if htlc.id == u.id => true
+        case u: UpdateFailHtlc if htlc.id == u.id => true
+        case u: UpdateFailMalformedHtlc if htlc.id == u.id => true
+        case _ => false
+      } =>
+        // we have already sent a fail/fulfill for this htlc
+        throw UnknownHtlcId(commitments.channelId, cmd.id)
       case Some(htlc) =>
         val fail = UpdateFailMalformedHtlc(commitments.channelId, cmd.id, cmd.onionHash, cmd.failureCode)
         val commitments1 = addLocalProposal(commitments, fail)
         (commitments1, fail)
-      case None => throw UnknownHtlcId(cmd.id)
+      case None => throw UnknownHtlcId(commitments.channelId, cmd.id)
+    }
+  }
+
+  def receiveFail(commitments: Commitments, fail: UpdateFailHtlc): Either[Commitments, (Commitments, Origin)] =
+    getHtlcCrossSigned(commitments, OUT, fail.id) match {
+      case Some(htlc) => Right((addRemoteProposal(commitments, fail), commitments.originChannels(fail.id)))
+      case None => throw UnknownHtlcId(commitments.channelId, fail.id)
     }
 
-  def isOldFail(commitments: Commitments, fail: UpdateFailHtlc): Boolean =
-    commitments.remoteChanges.proposed.contains(fail) ||
-      commitments.remoteChanges.signed.contains(fail) ||
-      commitments.remoteChanges.acked.contains(fail)
-
-  def isOldFail(commitments: Commitments, fail: UpdateFailMalformedHtlc): Boolean =
-    commitments.remoteChanges.proposed.contains(fail) ||
-      commitments.remoteChanges.signed.contains(fail) ||
-      commitments.remoteChanges.acked.contains(fail)
-
-  def receiveFail(commitments: Commitments, fail: UpdateFailHtlc): Either[Commitments, Commitments] =
-    isOldFail(commitments, fail) match {
-      case true => Left(commitments)
-      case false => getHtlcCrossSigned(commitments, OUT, fail.id) match {
-        case Some(htlc) => Right(addRemoteProposal(commitments, fail))
-        case None => throw UnknownHtlcId(fail.id)
-      }
+  def receiveFailMalformed(commitments: Commitments, fail: UpdateFailMalformedHtlc): Either[Commitments, (Commitments, Origin)] = {
+    // A receiving node MUST fail the channel if the BADONION bit in failure_code is not set for update_fail_malformed_htlc.
+    if ((fail.failureCode & FailureMessageCodecs.BADONION) == 0) {
+      throw InvalidFailureCode(commitments.channelId)
     }
 
-  def receiveFailMalformed(commitments: Commitments, fail: UpdateFailMalformedHtlc): Either[Commitments, Commitments] =
-    isOldFail(commitments, fail) match {
-      case true => Left(commitments)
-      case false => getHtlcCrossSigned(commitments, OUT, fail.id) match {
-        case Some(htlc) => Right(addRemoteProposal(commitments, fail))
-        case None => throw UnknownHtlcId(fail.id)
-      }
+    getHtlcCrossSigned(commitments, OUT, fail.id) match {
+      case Some(htlc) => Right((addRemoteProposal(commitments, fail), commitments.originChannels(fail.id)))
+      case None => throw UnknownHtlcId(commitments.channelId, fail.id)
     }
+  }
 
   def sendFee(commitments: Commitments, cmd: CMD_UPDATE_FEE): (Commitments, UpdateFee) = {
     if (!commitments.localParams.isFunder) {
-      throw FundeeCannotSendUpdateFee
+      throw FundeeCannotSendUpdateFee(commitments.channelId)
     }
     // let's compute the current commitment *as seen by them* with this change taken into account
     val fee = UpdateFee(commitments.channelId, cmd.feeratePerKw)
@@ -275,49 +276,45 @@ object Commitments extends Logging {
     val fees = Transactions.commitTxFee(Satoshi(commitments1.remoteParams.dustLimitSatoshis), reduced).amount
     val missing = reduced.toRemoteMsat / 1000 - commitments1.remoteParams.channelReserveSatoshis - fees
     if (missing < 0) {
-      throw CannotAffordFees(missingSatoshis = -1 * missing, reserveSatoshis = commitments1.localParams.channelReserveSatoshis, feesSatoshis = fees)
+      throw CannotAffordFees(commitments.channelId, missingSatoshis = -1 * missing, reserveSatoshis = commitments1.localParams.channelReserveSatoshis, feesSatoshis = fees)
     }
 
     (commitments1, fee)
   }
 
-  def isOldFee(commitments: Commitments, fee: UpdateFee): Boolean =
-    commitments.remoteChanges.proposed.contains(fee) ||
-      commitments.remoteChanges.signed.contains(fee) ||
-      commitments.remoteChanges.acked.contains(fee)
-
-  def receiveFee(commitments: Commitments, fee: UpdateFee, maxFeerateMismatch: Double): Commitments =
-    isOldFee(commitments, fee) match {
-      case true => commitments
-      case false =>
-        if (commitments.localParams.isFunder) {
-          throw FundeeCannotSendUpdateFee
-        }
-
-        val localFeeratePerKw = Globals.feeratePerKw.get()
-        if (Helpers.isFeeDiffTooHigh(fee.feeratePerKw, localFeeratePerKw, maxFeerateMismatch)) {
-          throw FeerateTooDifferent(localFeeratePerKw = localFeeratePerKw, remoteFeeratePerKw = fee.feeratePerKw)
-        }
-
-        // NB: we check that the funder can afford this new fee even if spec allows to do it at next signature
-        // It is easier to do it here because under certain (race) conditions spec allows a lower-than-normal fee to be paid,
-        // and it would be tricky to check if the conditions are met at signing
-        // (it also means that we need to check the fee of the initial commitment tx somewhere)
-
-        // let's compute the current commitment *as seen by us* including this change
-        val commitments1 = addRemoteProposal(commitments, fee)
-        val reduced = CommitmentSpec.reduce(commitments1.localCommit.spec, commitments1.localChanges.acked, commitments1.remoteChanges.proposed)
-
-        // a node cannot spend pending incoming htlcs, and need to keep funds above the reserve required by the counterparty, after paying the fee
-        val fees = Transactions.commitTxFee(Satoshi(commitments1.remoteParams.dustLimitSatoshis), reduced).amount
-        val missing = reduced.toRemoteMsat / 1000 - commitments1.localParams.channelReserveSatoshis - fees
-        if (missing < 0) {
-          throw CannotAffordFees(missingSatoshis = -1 * missing, reserveSatoshis = commitments1.localParams.channelReserveSatoshis, feesSatoshis = fees)
-        }
-
-        commitments1
+  def receiveFee(commitments: Commitments, fee: UpdateFee, maxFeerateMismatch: Double): Commitments = {
+    if (commitments.localParams.isFunder) {
+      throw FundeeCannotSendUpdateFee(commitments.channelId)
     }
 
+    val localFeeratePerKw = Globals.feeratesPerKw.get.block_1
+    if (Helpers.isFeeDiffTooHigh(fee.feeratePerKw, localFeeratePerKw, maxFeerateMismatch)) {
+      throw FeerateTooDifferent(commitments.channelId, localFeeratePerKw = localFeeratePerKw, remoteFeeratePerKw = fee.feeratePerKw)
+    }
+
+    // NB: we check that the funder can afford this new fee even if spec allows to do it at next signature
+    // It is easier to do it here because under certain (race) conditions spec allows a lower-than-normal fee to be paid,
+    // and it would be tricky to check if the conditions are met at signing
+    // (it also means that we need to check the fee of the initial commitment tx somewhere)
+
+    // let's compute the current commitment *as seen by us* including this change
+    val commitments1 = addRemoteProposal(commitments, fee)
+    val reduced = CommitmentSpec.reduce(commitments1.localCommit.spec, commitments1.localChanges.acked, commitments1.remoteChanges.proposed)
+
+    // a node cannot spend pending incoming htlcs, and need to keep funds above the reserve required by the counterparty, after paying the fee
+    val fees = Transactions.commitTxFee(Satoshi(commitments1.remoteParams.dustLimitSatoshis), reduced).amount
+    val missing = reduced.toRemoteMsat / 1000 - commitments1.localParams.channelReserveSatoshis - fees
+    if (missing < 0) {
+      throw CannotAffordFees(commitments.channelId, missingSatoshis = -1 * missing, reserveSatoshis = commitments1.localParams.channelReserveSatoshis, feesSatoshis = fees)
+    }
+
+    commitments1
+  }
+
+  def localHasUnsignedOutgoingHtlcs(commitments: Commitments): Boolean = commitments.localChanges.proposed.collectFirst { case u: UpdateAddHtlc => u }.isDefined
+
+  def remoteHasUnsignedOutgoingHtlcs(commitments: Commitments): Boolean = commitments.remoteChanges.proposed.collectFirst { case u: UpdateAddHtlc => u }.isDefined
+
   def localHasChanges(commitments: Commitments): Boolean = commitments.remoteChanges.acked.size > 0 || commitments.localChanges.proposed.size > 0
 
   def remoteHasChanges(commitments: Commitments): Boolean = commitments.localChanges.acked.size > 0 || commitments.remoteChanges.proposed.size > 0
@@ -330,7 +327,7 @@ object Commitments extends Logging {
     import commitments._
     commitments.remoteNextCommitInfo match {
       case Right(_) if !localHasChanges(commitments) =>
-        throw CannotSignWithoutChanges
+        throw CannotSignWithoutChanges(commitments.channelId)
       case Right(remoteNextPerCommitmentPoint) =>
         // remote commitment will includes all local changes + remote acked changes
         val spec = CommitmentSpec.reduce(remoteCommit.spec, remoteChanges.acked, localChanges.proposed)
@@ -338,8 +335,8 @@ object Commitments extends Logging {
         val sig = Transactions.sign(remoteCommitTx, localParams.fundingPrivKey)
 
         val sortedHtlcTxs: Seq[TransactionWithInputInfo] = (htlcTimeoutTxs ++ htlcSuccessTxs).sortBy(_.input.outPoint.index)
-        val paymentKey = Generators.derivePrivKey(localParams.paymentKey, remoteNextPerCommitmentPoint)
-        val htlcSigs = sortedHtlcTxs.map(Transactions.sign(_, paymentKey))
+        val htlcKey = Generators.derivePrivKey(localParams.htlcKey, remoteNextPerCommitmentPoint)
+        val htlcSigs = sortedHtlcTxs.map(Transactions.sign(_, htlcKey))
 
         // don't sign if they don't get paid
         val commitSig = CommitSig(
@@ -349,147 +346,130 @@ object Commitments extends Logging {
         )
 
         val commitments1 = commitments.copy(
-          remoteNextCommitInfo = Left(WaitingForRevocation(RemoteCommit(remoteCommit.index + 1, spec, remoteCommitTx.tx.txid, remoteNextPerCommitmentPoint), commitSig)),
+          remoteNextCommitInfo = Left(WaitingForRevocation(RemoteCommit(remoteCommit.index + 1, spec, remoteCommitTx.tx.txid, remoteNextPerCommitmentPoint), commitSig, commitments.localCommit.index)),
           localChanges = localChanges.copy(proposed = Nil, signed = localChanges.proposed),
-          remoteChanges = remoteChanges.copy(acked = Nil, signed = remoteChanges.acked),
-          unackedMessages = unackedMessages :+ commitSig)
+          remoteChanges = remoteChanges.copy(acked = Nil, signed = remoteChanges.acked))
         (commitments1, commitSig)
       case Left(_) =>
-        throw CannotSignBeforeRevocation
+        throw CannotSignBeforeRevocation(commitments.channelId)
     }
   }
 
-  def isOldCommit(commitments: Commitments, commit: CommitSig): Boolean = commitments.localCommit.commit == commit
+  def receiveCommit(commitments: Commitments, commit: CommitSig): (Commitments, RevokeAndAck) = {
+    import commitments._
+    // they sent us a signature for *their* view of *our* next commit tx
+    // so in terms of rev.hashes and indexes we have:
+    // ourCommit.index -> our current revocation hash, which is about to become our old revocation hash
+    // ourCommit.index + 1 -> our next revocation hash, used by *them* to build the sig we've just received, and which
+    // is about to become our current revocation hash
+    // ourCommit.index + 2 -> which is about to become our next revocation hash
+    // we will reply to this sig with our old revocation hash preimage (at index) and our next revocation hash (at index + 1)
+    // and will increment our index
 
-  def receiveCommit(commitments: Commitments, commit: CommitSig): Either[Commitments, (Commitments, RevokeAndAck)] =
-    isOldCommit(commitments, commit) match {
-      case true => Left(commitments)
-      case false =>
-        import commitments._
-        // they sent us a signature for *their* view of *our* next commit tx
-        // so in terms of rev.hashes and indexes we have:
-        // ourCommit.index -> our current revocation hash, which is about to become our old revocation hash
-        // ourCommit.index + 1 -> our next revocation hash, used by * them * to build the sig we've just received, and which
-        // is about to become our current revocation hash
-        // ourCommit.index + 2 -> which is about to become our next revocation hash
-        // we will reply to this sig with our old revocation hash preimage (at index) and our next revocation hash (at index + 1)
-        // and will increment our index
+    if (!remoteHasChanges(commitments))
+      throw CannotSignWithoutChanges(commitments.channelId)
 
-        if (!remoteHasChanges(commitments))
-          throw CannotSignWithoutChanges
+    // check that their signature is valid
+    // signatures are now optional in the commit message, and will be sent only if the other party is actually
+    // receiving money i.e its commit tx has one output for them
 
-        // check that their signature is valid
-        // signatures are now optional in the commit message, and will be sent only if the other party is actually
-        // receiving money i.e its commit tx has one output for them
+    val spec = CommitmentSpec.reduce(localCommit.spec, localChanges.acked, remoteChanges.proposed)
+    val localPerCommitmentPoint = Generators.perCommitPoint(localParams.shaSeed, commitments.localCommit.index + 1)
+    val (localCommitTx, htlcTimeoutTxs, htlcSuccessTxs) = makeLocalTxs(localCommit.index + 1, localParams, remoteParams, commitInput, localPerCommitmentPoint, spec)
+    val sig = Transactions.sign(localCommitTx, localParams.fundingPrivKey)
 
-        val spec = CommitmentSpec.reduce(localCommit.spec, localChanges.acked, remoteChanges.proposed)
-        val localPerCommitmentPoint = Generators.perCommitPoint(localParams.shaSeed, commitments.localCommit.index + 1)
-        val (localCommitTx, htlcTimeoutTxs, htlcSuccessTxs) = makeLocalTxs(localCommit.index + 1, localParams, remoteParams, commitInput, localPerCommitmentPoint, spec)
-        val sig = Transactions.sign(localCommitTx, localParams.fundingPrivKey)
+    // TODO: should we have optional sig? (original comment: this tx will NOT be signed if our output is empty)
 
-        // TODO: should we have optional sig? (original comment: this tx will NOT be signed if our output is empty)
-
-        // no need to compute htlc sigs if commit sig doesn't check out
-        val signedCommitTx = Transactions.addSigs(localCommitTx, localParams.fundingPrivKey.publicKey, remoteParams.fundingPubKey, sig, commit.signature)
-        if (Transactions.checkSpendable(signedCommitTx).isFailure) {
-          throw InvalidCommitmentSignature
-        }
-
-        val sortedHtlcTxs: Seq[TransactionWithInputInfo] = (htlcTimeoutTxs ++ htlcSuccessTxs).sortBy(_.input.outPoint.index)
-        require(commit.htlcSignatures.size == sortedHtlcTxs.size, s"htlc sig count mismatch (received=${commit.htlcSignatures.size}, expected=${sortedHtlcTxs.size})")
-        val localPaymentKey = Generators.derivePrivKey(localParams.paymentKey, localPerCommitmentPoint)
-        val htlcSigs = sortedHtlcTxs.map(Transactions.sign(_, localPaymentKey))
-        val remotePaymentPubkey = Generators.derivePubKey(remoteParams.paymentBasepoint, localPerCommitmentPoint)
-        // combine the sigs to make signed txes
-        val htlcTxsAndSigs = (sortedHtlcTxs, htlcSigs, commit.htlcSignatures).zipped.toList.collect {
-          case (htlcTx: HtlcTimeoutTx, localSig, remoteSig) =>
-            require(Transactions.checkSpendable(Transactions.addSigs(htlcTx, localSig, remoteSig)).isSuccess, "bad sig")
-            HtlcTxAndSigs(htlcTx, localSig, remoteSig)
-          case (htlcTx: HtlcSuccessTx, localSig, remoteSig) =>
-            // we can't check that htlc-success tx are spendable because we need the payment preimage; thus we only check the remote sig
-            require(Transactions.checkSig(htlcTx, remoteSig, remotePaymentPubkey), "bad sig")
-            HtlcTxAndSigs(htlcTx, localSig, remoteSig)
-        }
-
-        // we will send our revocation preimage + our next revocation hash
-        val localPerCommitmentSecret = Generators.perCommitSecret(localParams.shaSeed, commitments.localCommit.index)
-        val localNextPerCommitmentPoint = Generators.perCommitPoint(localParams.shaSeed, commitments.localCommit.index + 2)
-        val revocation = RevokeAndAck(
-          channelId = commitments.channelId,
-          perCommitmentSecret = localPerCommitmentSecret,
-          nextPerCommitmentPoint = localNextPerCommitmentPoint
-        )
-
-        // update our commitment data
-        val ourCommit1 = LocalCommit(
-          index = localCommit.index + 1,
-          spec,
-          publishableTxs = PublishableTxs(signedCommitTx, htlcTxsAndSigs),
-          commit = commit)
-        val ourChanges1 = localChanges.copy(acked = Nil)
-        val theirChanges1 = remoteChanges.copy(proposed = Nil, acked = remoteChanges.acked ++ remoteChanges.proposed)
-        // they have received our previous revocation (otherwise they wouldn't have sent a commit)
-        // so we can acknowledge the revocation
-        val unackedMessages1 = commitments.unackedMessages.filterNot(_.isInstanceOf[RevokeAndAck]) :+ revocation
-        val commitments1 = commitments.copy(localCommit = ourCommit1, localChanges = ourChanges1, remoteChanges = theirChanges1, unackedMessages = unackedMessages1)
-
-        logger.debug(s"current commit: index=${ourCommit1.index} htlc_in=${ourCommit1.spec.htlcs.filter(_.direction == IN).size} htlc_out=${ourCommit1.spec.htlcs.filter(_.direction == OUT).size} txid=${ourCommit1.publishableTxs.commitTx.tx.txid} tx=${Transaction.write(ourCommit1.publishableTxs.commitTx.tx)}")
-
-        Right((commitments1, revocation))
+    // no need to compute htlc sigs if commit sig doesn't check out
+    val signedCommitTx = Transactions.addSigs(localCommitTx, localParams.fundingPrivKey.publicKey, remoteParams.fundingPubKey, sig, commit.signature)
+    if (Transactions.checkSpendable(signedCommitTx).isFailure) {
+      throw InvalidCommitmentSignature(commitments.channelId)
     }
 
-  def isOldRevocation(commitments: Commitments, revocation: RevokeAndAck): Boolean =
-    commitments.remoteNextCommitInfo match {
-      case Right(point) if point == revocation.nextPerCommitmentPoint => true
-      case Left(waitForRevocation) if waitForRevocation.nextRemoteCommit.remotePerCommitmentPoint == revocation.nextPerCommitmentPoint => true
-      case _ => false
+    val sortedHtlcTxs: Seq[TransactionWithInputInfo] = (htlcTimeoutTxs ++ htlcSuccessTxs).sortBy(_.input.outPoint.index)
+    require(commit.htlcSignatures.size == sortedHtlcTxs.size, s"htlc sig count mismatch (received=${commit.htlcSignatures.size}, expected=${sortedHtlcTxs.size})")
+    val localHtlcKey = Generators.derivePrivKey(localParams.htlcKey, localPerCommitmentPoint)
+    val htlcSigs = sortedHtlcTxs.map(Transactions.sign(_, localHtlcKey))
+    val remoteHtlcPubkey = Generators.derivePubKey(remoteParams.htlcBasepoint, localPerCommitmentPoint)
+    // combine the sigs to make signed txes
+    val htlcTxsAndSigs = (sortedHtlcTxs, htlcSigs, commit.htlcSignatures).zipped.toList.collect {
+      case (htlcTx: HtlcTimeoutTx, localSig, remoteSig) =>
+        require(Transactions.checkSpendable(Transactions.addSigs(htlcTx, localSig, remoteSig)).isSuccess, "bad sig")
+        HtlcTxAndSigs(htlcTx, localSig, remoteSig)
+      case (htlcTx: HtlcSuccessTx, localSig, remoteSig) =>
+        // we can't check that htlc-success tx are spendable because we need the payment preimage; thus we only check the remote sig
+        require(Transactions.checkSig(htlcTx, remoteSig, remoteHtlcPubkey), "bad sig")
+        HtlcTxAndSigs(htlcTx, localSig, remoteSig)
     }
 
-  def receiveRevocation(commitments: Commitments, revocation: RevokeAndAck): Either[Commitments, Commitments] =
-    isOldRevocation(commitments, revocation) match {
-      case true => Left(commitments)
-      case false =>
-        import commitments._
-        // we receive a revocation because we just sent them a sig for their next commit tx
-        remoteNextCommitInfo match {
-          case Left(_) if revocation.perCommitmentSecret.toPoint != remoteCommit.remotePerCommitmentPoint =>
-            throw InvalidRevocation
-          case Left(WaitingForRevocation(theirNextCommit, _, _)) =>
-            // they have received our last commitsig (otherwise they wouldn't have replied with a revocation)
-            // so we can acknowledge all our previous updates and the commitsig
-            val unackedMessages1 = commitments.unackedMessages.drop(commitments.unackedMessages.indexWhere(_.isInstanceOf[CommitSig]) + 1)
+    // we will send our revocation preimage + our next revocation hash
+    val localPerCommitmentSecret = Generators.perCommitSecret(localParams.shaSeed, commitments.localCommit.index)
+    val localNextPerCommitmentPoint = Generators.perCommitPoint(localParams.shaSeed, commitments.localCommit.index + 2)
+    val revocation = RevokeAndAck(
+      channelId = commitments.channelId,
+      perCommitmentSecret = localPerCommitmentSecret,
+      nextPerCommitmentPoint = localNextPerCommitmentPoint
+    )
 
-            val commitments1 = commitments.copy(
-              localChanges = localChanges.copy(signed = Nil, acked = localChanges.acked ++ localChanges.signed),
-              remoteChanges = remoteChanges.copy(signed = Nil),
-              remoteCommit = theirNextCommit,
-              remoteNextCommitInfo = Right(revocation.nextPerCommitmentPoint),
-              unackedMessages = unackedMessages1,
-              remotePerCommitmentSecrets = commitments.remotePerCommitmentSecrets.addHash(revocation.perCommitmentSecret, 0xFFFFFFFFFFFFL - commitments.remoteCommit.index))
+    // update our commitment data
+    val localCommit1 = LocalCommit(
+      index = localCommit.index + 1,
+      spec,
+      publishableTxs = PublishableTxs(signedCommitTx, htlcTxsAndSigs))
+    val ourChanges1 = localChanges.copy(acked = Nil)
+    val theirChanges1 = remoteChanges.copy(proposed = Nil, acked = remoteChanges.acked ++ remoteChanges.proposed)
+    // the outgoing following htlcs have been completed (fulfilled or failed) when we received this sig
+    val completedOutgoingHtlcs = commitments.localCommit.spec.htlcs.filter(_.direction == OUT).map(_.add.id) -- localCommit1.spec.htlcs.filter(_.direction == OUT).map(_.add.id)
+    // we remove the newly completed htlcs from the origin map
+    val originChannels1 = commitments.originChannels -- completedOutgoingHtlcs
+    val commitments1 = commitments.copy(localCommit = localCommit1, localChanges = ourChanges1, remoteChanges = theirChanges1, originChannels = originChannels1)
 
-            Right(commitments1)
-          case Right(_) =>
-            throw UnexpectedRevocation
-        }
+    logger.debug(s"current commit: index=${localCommit1.index} htlc_in=${localCommit1.spec.htlcs.filter(_.direction == IN).size} htlc_out=${localCommit1.spec.htlcs.filter(_.direction == OUT).size} txid=${localCommit1.publishableTxs.commitTx.tx.txid} tx=${Transaction.write(localCommit1.publishableTxs.commitTx.tx)}")
+
+    (commitments1, revocation)
+  }
+
+  def receiveRevocation(commitments: Commitments, revocation: RevokeAndAck): Commitments = {
+    import commitments._
+    // we receive a revocation because we just sent them a sig for their next commit tx
+    remoteNextCommitInfo match {
+      case Left(_) if revocation.perCommitmentSecret.toPoint != remoteCommit.remotePerCommitmentPoint =>
+        throw InvalidRevocation(commitments.channelId)
+      case Left(WaitingForRevocation(theirNextCommit, _, _, _)) =>
+        val commitments1 = commitments.copy(
+          localChanges = localChanges.copy(signed = Nil, acked = localChanges.acked ++ localChanges.signed),
+          remoteChanges = remoteChanges.copy(signed = Nil),
+          remoteCommit = theirNextCommit,
+          remoteNextCommitInfo = Right(revocation.nextPerCommitmentPoint),
+          remotePerCommitmentSecrets = commitments.remotePerCommitmentSecrets.addHash(revocation.perCommitmentSecret, 0xFFFFFFFFFFFFL - commitments.remoteCommit.index))
+
+        commitments1
+      case Right(_) =>
+        throw UnexpectedRevocation(commitments.channelId)
     }
+  }
 
   def makeLocalTxs(commitTxNumber: Long, localParams: LocalParams, remoteParams: RemoteParams, commitmentInput: InputInfo, localPerCommitmentPoint: Point, spec: CommitmentSpec): (CommitTx, Seq[HtlcTimeoutTx], Seq[HtlcSuccessTx]) = {
-    val localPubkey = Generators.derivePubKey(localParams.paymentKey.toPoint, localPerCommitmentPoint)
-    val localDelayedPubkey = Generators.derivePubKey(localParams.delayedPaymentKey.toPoint, localPerCommitmentPoint)
-    val remotePubkey = Generators.derivePubKey(remoteParams.paymentBasepoint, localPerCommitmentPoint)
+    val localPaymentPubkey = Generators.derivePubKey(localParams.paymentBasepoint, localPerCommitmentPoint)
+    val localDelayedPaymentPubkey = Generators.derivePubKey(localParams.delayedPaymentBasepoint, localPerCommitmentPoint)
+    val localHtlcPubkey = Generators.derivePubKey(localParams.htlcBasepoint, localPerCommitmentPoint)
+    val remotePaymentPubkey = Generators.derivePubKey(remoteParams.paymentBasepoint, localPerCommitmentPoint)
+    val remoteHtlcPubkey = Generators.derivePubKey(remoteParams.htlcBasepoint, localPerCommitmentPoint)
     val localRevocationPubkey = Generators.revocationPubKey(remoteParams.revocationBasepoint, localPerCommitmentPoint)
-    val commitTx = Transactions.makeCommitTx(commitmentInput, commitTxNumber, localParams.paymentKey.toPoint, remoteParams.paymentBasepoint, localParams.isFunder, Satoshi(localParams.dustLimitSatoshis), localPubkey, localRevocationPubkey, remoteParams.toSelfDelay, localDelayedPubkey, remotePubkey, spec)
-    val (htlcTimeoutTxs, htlcSuccessTxs) = Transactions.makeHtlcTxs(commitTx.tx, Satoshi(localParams.dustLimitSatoshis), localRevocationPubkey, remoteParams.toSelfDelay, localPubkey, localDelayedPubkey, remotePubkey, spec)
+    val commitTx = Transactions.makeCommitTx(commitmentInput, commitTxNumber, localParams.paymentBasepoint, remoteParams.paymentBasepoint, localParams.isFunder, Satoshi(localParams.dustLimitSatoshis), localRevocationPubkey, remoteParams.toSelfDelay, localDelayedPaymentPubkey, remotePaymentPubkey, localHtlcPubkey, remoteHtlcPubkey, spec)
+    val (htlcTimeoutTxs, htlcSuccessTxs) = Transactions.makeHtlcTxs(commitTx.tx, Satoshi(localParams.dustLimitSatoshis), localRevocationPubkey, remoteParams.toSelfDelay, localDelayedPaymentPubkey, localHtlcPubkey, remoteHtlcPubkey, spec)
     (commitTx, htlcTimeoutTxs, htlcSuccessTxs)
   }
 
   def makeRemoteTxs(commitTxNumber: Long, localParams: LocalParams, remoteParams: RemoteParams, commitmentInput: InputInfo, remotePerCommitmentPoint: Point, spec: CommitmentSpec): (CommitTx, Seq[HtlcTimeoutTx], Seq[HtlcSuccessTx]) = {
-    val localPubkey = Generators.derivePubKey(localParams.paymentKey.toPoint, remotePerCommitmentPoint)
-    val remotePubkey = Generators.derivePubKey(remoteParams.paymentBasepoint, remotePerCommitmentPoint)
-    val remoteDelayedPubkey = Generators.derivePubKey(remoteParams.delayedPaymentBasepoint, remotePerCommitmentPoint)
-    val remoteRevocationPubkey = Generators.revocationPubKey(localParams.revocationSecret.toPoint, remotePerCommitmentPoint)
-    val commitTx = Transactions.makeCommitTx(commitmentInput, commitTxNumber, remoteParams.paymentBasepoint, localParams.paymentKey.toPoint, !localParams.isFunder, Satoshi(remoteParams.dustLimitSatoshis), remotePubkey, remoteRevocationPubkey, localParams.toSelfDelay, remoteDelayedPubkey, localPubkey, spec)
-    val (htlcTimeoutTxs, htlcSuccessTxs) = Transactions.makeHtlcTxs(commitTx.tx, Satoshi(localParams.dustLimitSatoshis), remoteRevocationPubkey, localParams.toSelfDelay, remotePubkey, remoteDelayedPubkey, localPubkey, spec)
+    val localPaymentPubkey = Generators.derivePubKey(localParams.paymentBasepoint, remotePerCommitmentPoint)
+    val localHtlcPubkey = Generators.derivePubKey(localParams.htlcBasepoint, remotePerCommitmentPoint)
+    val remotePaymentPubkey = Generators.derivePubKey(remoteParams.paymentBasepoint, remotePerCommitmentPoint)
+    val remoteDelayedPaymentPubkey = Generators.derivePubKey(remoteParams.delayedPaymentBasepoint, remotePerCommitmentPoint)
+    val remoteHtlcPubkey = Generators.derivePubKey(remoteParams.htlcBasepoint, remotePerCommitmentPoint)
+    val remoteRevocationPubkey = Generators.revocationPubKey(localParams.revocationBasepoint, remotePerCommitmentPoint)
+    val commitTx = Transactions.makeCommitTx(commitmentInput, commitTxNumber, remoteParams.paymentBasepoint, localParams.paymentBasepoint, !localParams.isFunder, Satoshi(remoteParams.dustLimitSatoshis), remoteRevocationPubkey, localParams.toSelfDelay, remoteDelayedPaymentPubkey, localPaymentPubkey, remoteHtlcPubkey, localHtlcPubkey, spec)
+    val (htlcTimeoutTxs, htlcSuccessTxs) = Transactions.makeHtlcTxs(commitTx.tx, Satoshi(remoteParams.dustLimitSatoshis), remoteRevocationPubkey, localParams.toSelfDelay, remoteDelayedPaymentPubkey, remoteHtlcPubkey, localHtlcPubkey, spec)
     (commitTx, htlcTimeoutTxs, htlcSuccessTxs)
   }
 
@@ -518,9 +498,7 @@ object Commitments extends Logging {
        |        signed: ${remoteChanges.signed.map(msg2String(_)).mkString(" ")}
        |    nextHtlcId:
        |        local: $localNextHtlcId
-       |        remote: $remoteNextHtlcId
-       |    unackedMessages:
-       |        ${unackedMessages.map(msg2String(_)).mkString(" ")}""".stripMargin
+       |        remote: $remoteNextHtlcId""".stripMargin
   }
 
   def specs2String(commitments: Commitments): String = {

eclair-core/src/main/scala/fr/acinq/eclair/channel/Forwarder.scala

@@ -2,14 +2,12 @@ package fr.acinq.eclair.channel
 
 import akka.actor.{Actor, ActorLogging, ActorRef}
 import fr.acinq.eclair.NodeParams
-import fr.acinq.eclair.wire.{Error, LightningMessage}
+import fr.acinq.eclair.wire.LightningMessage
 
 /**
   * Created by fabrice on 27/02/17.
   */
 
-case class StoreAndForward(state: State, stateData: Data, outgoing: Seq[LightningMessage])
-
 class Forwarder(nodeParams: NodeParams) extends Actor with ActorLogging {
 
   // caller is responsible for sending the destination before anything else
@@ -20,18 +18,7 @@ class Forwarder(nodeParams: NodeParams) extends Actor with ActorLogging {
 
     case destination: ActorRef => context become main(destination)
 
-    case error: Error => destination ! error
+    case msg: LightningMessage => destination forward msg
 
-    case StoreAndForward(CLOSED, stateData: HasCommitments, _) =>
-      log.debug(s"deleting database record for channelId=${stateData.channelId}")
-      nodeParams.channelsDb.delete(stateData.channelId)
-
-    case StoreAndForward(_, stateData: HasCommitments, outgoing) =>
-      log.debug(s"updating database record for channelId=${stateData.channelId}")
-      nodeParams.channelsDb.put(stateData.channelId, stateData)
-      outgoing.foreach(destination forward _)
-
-    case StoreAndForward(_, _, outgoing) =>
-      outgoing.foreach(destination forward _)
   }
 }

eclair-core/src/main/scala/fr/acinq/eclair/channel/Helpers.scala

@@ -3,16 +3,17 @@ package fr.acinq.eclair.channel
 import fr.acinq.bitcoin.Crypto.{Point, PublicKey, Scalar, sha256}
 import fr.acinq.bitcoin.Script._
 import fr.acinq.bitcoin.{OutPoint, _}
+import fr.acinq.eclair.blockchain.EclairWallet
 import fr.acinq.eclair.crypto.Generators
+import fr.acinq.eclair.router.Announcements
 import fr.acinq.eclair.transactions.Scripts._
 import fr.acinq.eclair.transactions.Transactions._
 import fr.acinq.eclair.transactions._
-import fr.acinq.eclair.Features.CHANNELS_PUBLIC_BIT
-import fr.acinq.eclair.blockchain.MakeFundingTxResponse
-import fr.acinq.eclair.wire.{ClosingSigned, LightningMessage, UpdateAddHtlc, UpdateFulfillHtlc}
-import fr.acinq.eclair.{Features, Globals, NodeParams}
+import fr.acinq.eclair.wire.{AnnouncementSignatures, ClosingSigned, UpdateAddHtlc, UpdateFulfillHtlc}
+import fr.acinq.eclair.{Globals, NodeParams}
 import grizzled.slf4j.Logging
 
+import scala.concurrent.Await
 import scala.util.{Failure, Success, Try}
 
 /**
@@ -32,43 +33,28 @@ object Helpers {
     case d: DATA_WAIT_FOR_OPEN_CHANNEL => d.initFundee.temporaryChannelId
     case d: DATA_WAIT_FOR_ACCEPT_CHANNEL => d.initFunder.temporaryChannelId
     case d: DATA_WAIT_FOR_FUNDING_INTERNAL => d.temporaryChannelId
-    case d: DATA_WAIT_FOR_FUNDING_PARENT => d.data.temporaryChannelId
     case d: DATA_WAIT_FOR_FUNDING_CREATED => d.temporaryChannelId
     case d: DATA_WAIT_FOR_FUNDING_SIGNED => d.channelId
     case d: HasCommitments => d.channelId
   }
 
-  def validateParamsFunder(nodeParams: NodeParams, channelReserveSatoshis: Long, fundingSatoshis: Long): Unit = {
+  def validateParamsFunder(temporaryChannelId: BinaryData, nodeParams: NodeParams, channelReserveSatoshis: Long, fundingSatoshis: Long): Unit = {
     val reserveToFundingRatio = channelReserveSatoshis.toDouble / fundingSatoshis
-    require(reserveToFundingRatio <= nodeParams.maxReserveToFundingRatio, s"channelReserveSatoshis too high: ratio=$reserveToFundingRatio max=${nodeParams.maxReserveToFundingRatio}")
-  }
-
-  def validateParamsFundee(nodeParams: NodeParams, channelReserveSatoshis: Long, fundingSatoshis: Long, chainHash: BinaryData): Unit = {
-    require(nodeParams.chainHash == chainHash, s"invalid chain hash $chainHash (we are on ${nodeParams.chainHash})")
-    validateParamsFunder(nodeParams, channelReserveSatoshis, fundingSatoshis)
-  }
-
-  def extractOutgoingMessages(currentState: State, nextState: State, currentData: Data, nextData: Data): Seq[LightningMessage] = {
-    (currentState, nextState, currentData, nextData) match {
-      case (_, OFFLINE, _, _) => Nil // we are not connected anymore (or not yet connected after a restore), we will re-send messages when we leave OFFLINE state
-      case (OFFLINE, _, _, d: DATA_WAIT_FOR_FUNDING_CONFIRMED) => d.lastSent.right.toSeq // NB: if we re-send the message and the other party didn't receive it before, they will return an error because channel wasn't stored (see #120), and that's ok
-      case (OFFLINE, _, _, d: DATA_WAIT_FOR_FUNDING_LOCKED) => d.lastSent :: Nil
-      case (OFFLINE, _, _: HasCommitments, d2: HasCommitments) => d2.commitments.unackedMessages
-      case (OFFLINE, _, _, _) => Nil
-      case (_, _, _, d: DATA_CLOSING) => Nil
-      case (_, _, _, d: DATA_WAIT_FOR_OPEN_CHANNEL) => Nil
-      case (_, _, _, d: DATA_WAIT_FOR_ACCEPT_CHANNEL) => d.lastSent :: Nil
-      case (_, _, _, d: DATA_WAIT_FOR_FUNDING_INTERNAL) => Nil
-      case (_, _, _, d: DATA_WAIT_FOR_FUNDING_PARENT) => Nil
-      case (_, _, _, d: DATA_WAIT_FOR_FUNDING_CREATED) => d.lastSent :: Nil
-      case (_, _, _, d: DATA_WAIT_FOR_FUNDING_SIGNED) => d.lastSent :: Nil
-      case (_, _, _, d: DATA_WAIT_FOR_FUNDING_CONFIRMED) => d.lastSent.right.toOption.map(_ :: Nil).getOrElse(Nil)
-      case (_, _, _, d: DATA_WAIT_FOR_FUNDING_LOCKED) => d.lastSent :: Nil
-      case (_, _, d1: HasCommitments, d2: HasCommitments) => d2.commitments.unackedMessages diff d1.commitments.unackedMessages
-      case _ => ??? // eg: goto(CLOSING)
+    if (reserveToFundingRatio > nodeParams.maxReserveToFundingRatio) {
+      throw new ChannelReserveTooHigh(temporaryChannelId, channelReserveSatoshis, reserveToFundingRatio, nodeParams.maxReserveToFundingRatio)
     }
   }
 
+  def validateParamsFundee(temporaryChannelId: BinaryData, nodeParams: NodeParams, channelReserveSatoshis: Long, fundingSatoshis: Long, chainHash: BinaryData, initialFeeratePerKw: Long): Unit = {
+    require(nodeParams.chainHash == chainHash, s"invalid chain hash $chainHash (we are on ${nodeParams.chainHash})")
+    val localFeeratePerKw = Globals.feeratesPerKw.get.block_1
+    // we are fundee => initialFeeratePerKw has been set by remote
+    if (isFeeDiffTooHigh(initialFeeratePerKw, localFeeratePerKw, nodeParams.maxFeerateMismatch)) {
+      throw new FeerateTooDifferent(temporaryChannelId, localFeeratePerKw, initialFeeratePerKw)
+    }
+    validateParamsFunder(temporaryChannelId, nodeParams, channelReserveSatoshis, fundingSatoshis)
+  }
+
   /**
     *
     * @param remoteFeeratePerKw remote fee rate per kiloweight
@@ -95,6 +81,23 @@ object Helpers {
     remoteFeeratePerKw > 0 && feeRateMismatch(remoteFeeratePerKw, localFeeratePerKw) > maxFeerateMismatchRatio
   }
 
+  def makeAnnouncementSignatures(nodeParams: NodeParams, commitments: Commitments, shortChannelId: Long) = {
+    // TODO: empty features
+    val features = BinaryData("")
+    val (localNodeSig, localBitcoinSig) = Announcements.signChannelAnnouncement(nodeParams.chainHash, shortChannelId, nodeParams.privateKey, commitments.remoteParams.nodeId, commitments.localParams.fundingPrivKey, commitments.remoteParams.fundingPubKey, features)
+    AnnouncementSignatures(commitments.channelId, shortChannelId, localNodeSig, localBitcoinSig)
+  }
+
+  def getFinalScriptPubKey(wallet: EclairWallet): BinaryData = {
+    import scala.concurrent.duration._
+    val finalAddress = Await.result(wallet.getFinalAddress, 40 seconds)
+    val finalScriptPubKey = Base58Check.decode(finalAddress) match {
+      case (Base58.Prefix.PubkeyAddressTestnet, hash) => Script.write(OP_DUP :: OP_HASH160 :: OP_PUSHDATA(hash) :: OP_EQUALVERIFY :: OP_CHECKSIG :: Nil)
+      case (Base58.Prefix.ScriptAddressTestnet, hash) => Script.write(OP_HASH160 :: OP_PUSHDATA(hash) :: OP_EQUAL :: Nil)
+    }
+    finalScriptPubKey
+  }
+
   object Funding {
 
     def makeFundingInputInfo(fundingTxId: BinaryData, fundingTxOutputIndex: Int, fundingSatoshis: Satoshi, fundingPubkey1: PublicKey, fundingPubkey2: PublicKey): InputInfo = {
@@ -114,24 +117,21 @@ object Helpers {
       * @param remoteFirstPerCommitmentPoint
       * @return (localSpec, localTx, remoteSpec, remoteTx, fundingTxOutput)
       */
-    def makeFirstCommitTxs(localParams: LocalParams, remoteParams: RemoteParams, fundingSatoshis: Long, pushMsat: Long, initialFeeratePerKw: Long, fundingTxHash: BinaryData, fundingTxOutputIndex: Int, remoteFirstPerCommitmentPoint: Point, maxFeerateMismatch: Double): (CommitmentSpec, CommitTx, CommitmentSpec, CommitTx) = {
+    def makeFirstCommitTxs(temporaryChannelId: BinaryData, localParams: LocalParams, remoteParams: RemoteParams, fundingSatoshis: Long, pushMsat: Long, initialFeeratePerKw: Long, fundingTxHash: BinaryData, fundingTxOutputIndex: Int, remoteFirstPerCommitmentPoint: Point, maxFeerateMismatch: Double): (CommitmentSpec, CommitTx, CommitmentSpec, CommitTx) = {
       val toLocalMsat = if (localParams.isFunder) fundingSatoshis * 1000 - pushMsat else pushMsat
       val toRemoteMsat = if (localParams.isFunder) pushMsat else fundingSatoshis * 1000 - pushMsat
 
-      val localSpec = CommitmentSpec(Set.empty[Htlc], feeratePerKw = initialFeeratePerKw, toLocalMsat = toLocalMsat, toRemoteMsat = toRemoteMsat)
-      val remoteSpec = CommitmentSpec(Set.empty[Htlc], feeratePerKw = initialFeeratePerKw, toLocalMsat = toRemoteMsat, toRemoteMsat = toLocalMsat)
+      val localSpec = CommitmentSpec(Set.empty[DirectedHtlc], feeratePerKw = initialFeeratePerKw, toLocalMsat = toLocalMsat, toRemoteMsat = toRemoteMsat)
+      val remoteSpec = CommitmentSpec(Set.empty[DirectedHtlc], feeratePerKw = initialFeeratePerKw, toLocalMsat = toRemoteMsat, toRemoteMsat = toLocalMsat)
 
-      // TODO: should we check the fees sooner in the process?
       if (!localParams.isFunder) {
-        // they are funder, we need to make sure that they can pay the fee is reasonable, and that they can afford to pay it
-        val localFeeratePerKw = Globals.feeratePerKw.get()
-        if (isFeeDiffTooHigh(initialFeeratePerKw, localFeeratePerKw, maxFeerateMismatch)) {
-          throw new RuntimeException(s"local/remote feerates are too different: remoteFeeratePerKw=$initialFeeratePerKw localFeeratePerKw=$localFeeratePerKw")
+        // they are funder, therefore they pay the fee: we need to make sure they can afford it!
+        val toRemoteMsat = remoteSpec.toLocalMsat
+        val fees = Transactions.commitTxFee(Satoshi(remoteParams.dustLimitSatoshis), remoteSpec).amount
+        val missing = toRemoteMsat / 1000 - localParams.channelReserveSatoshis - fees
+        if (missing < 0) {
+          throw CannotAffordFees(temporaryChannelId, missingSatoshis = -1 * missing, reserveSatoshis = localParams.channelReserveSatoshis, feesSatoshis = fees)
         }
-        val toRemote = MilliSatoshi(remoteSpec.toLocalMsat)
-        val reserve = Satoshi(localParams.channelReserveSatoshis)
-        val fees = Transactions.commitTxFee(Satoshi(remoteParams.dustLimitSatoshis), remoteSpec)
-        require(toRemote >= reserve + fees, s"remote cannot pay the fees for the initial commit tx: toRemote=$toRemote reserve=$reserve fees=$fees")
       }
 
       val commitmentInput = makeFundingInputInfo(fundingTxHash, fundingTxOutputIndex, Satoshi(fundingSatoshis), localParams.fundingPrivKey.publicKey, remoteParams.fundingPubKey)
@@ -142,51 +142,6 @@ object Helpers {
       (localSpec, localCommitTx, remoteSpec, remoteCommitTx)
     }
 
-    def announceChannel(localLocalFeatures: BinaryData, remoteLocalFeature: BinaryData): Boolean =
-      Features.isSet(localLocalFeatures, CHANNELS_PUBLIC_BIT) && Features.isSet(remoteLocalFeature, CHANNELS_PUBLIC_BIT)
-
-    /**
-      *
-      * @param fundingTxResponse funding transaction response, which includes a funding tx, its parent, and the private key
-      *                          that we need to re-sign the funding
-      * @param newParentTx       new parent tx
-      * @return an updated funding transaction response where the funding tx now spends from newParentTx
-      */
-    def replaceParent(fundingTxResponse: MakeFundingTxResponse, newParentTx: Transaction): MakeFundingTxResponse = {
-      // find the output that we are spending from
-      val utxo = newParentTx.txOut(fundingTxResponse.fundingTx.txIn(0).outPoint.index.toInt)
-
-      // check that it matches what we expect, which is a P2WPKH output to our public key
-      require(utxo.publicKeyScript == Script.write(Script.pay2sh(Script.pay2wpkh(fundingTxResponse.priv.publicKey))))
-
-      // update our tx input we the hash of the new parent
-      val input = fundingTxResponse.fundingTx.txIn(0)
-      val input1 = input.copy(outPoint = input.outPoint.copy(hash = newParentTx.hash))
-      val unsignedFundingTx = fundingTxResponse.fundingTx.copy(txIn = Seq(input1))
-
-      // and re-sign it
-      Helpers.Funding.sign(MakeFundingTxResponse(newParentTx, unsignedFundingTx, fundingTxResponse.fundingTxOutputIndex, fundingTxResponse.priv))
-    }
-
-    /**
-      *
-      * @param fundingTxResponse a funding tx response
-      * @return an updated funding tx response that is properly sign
-      */
-    def sign(fundingTxResponse: MakeFundingTxResponse): MakeFundingTxResponse = {
-      // find the output that we are spending from
-      val utxo = fundingTxResponse.parentTx.txOut(fundingTxResponse.fundingTx.txIn(0).outPoint.index.toInt)
-
-      val pub = fundingTxResponse.priv.publicKey
-      val pubKeyScript = Script.pay2pkh(pub)
-      val sig = Transaction.signInput(fundingTxResponse.fundingTx, 0, pubKeyScript, SIGHASH_ALL, utxo.amount, SigVersion.SIGVERSION_WITNESS_V0, fundingTxResponse.priv)
-      val witness = ScriptWitness(Seq(sig, pub.toBin))
-      val fundingTx1 = fundingTxResponse.fundingTx.updateSigScript(0, OP_PUSHDATA(Script.write(Script.pay2wpkh(pub))) :: Nil).updateWitness(0, witness)
-
-      Transaction.correctlySpends(fundingTx1, fundingTxResponse.parentTx :: Nil, ScriptFlags.STANDARD_SCRIPT_VERIFY_FLAGS)
-      fundingTxResponse.copy(fundingTx = fundingTx1)
-    }
-
   }
 
   object Closing extends Logging {
@@ -208,8 +163,8 @@ object Helpers {
         // this is just to estimate the weight, it depends on size of the pubkey scripts
         val dummyClosingTx = Transactions.makeClosingTx(commitInput, localScriptPubkey, remoteScriptPubkey, localParams.isFunder, Satoshi(0), Satoshi(0), localCommit.spec)
         val closingWeight = Transaction.weight(Transactions.addSigs(dummyClosingTx, localParams.fundingPrivKey.publicKey, remoteParams.fundingPubKey, "aa" * 71, "bb" * 71).tx)
-        // we use our local fee estimate
-        val feeratePerKw = Globals.feeratePerKw.get()
+        // no need to use a very high fee here
+        val feeratePerKw = Globals.feeratesPerKw.get.blocks_6
         logger.info(s"using feeratePerKw=$feeratePerKw for closing tx")
         Transactions.weight2fee(feeratePerKw, closingWeight)
       }
@@ -237,12 +192,7 @@ object Helpers {
       Transactions.checkSpendable(signedClosingTx).map(x => signedClosingTx.tx)
     }
 
-    def nextClosingFee(localClosingFee: Satoshi, remoteClosingFee: Satoshi): Satoshi = {
-      ((localClosingFee + remoteClosingFee) / 4) * 2 match {
-        case value if value == localClosingFee => value + Satoshi(2) // TODO: why +2 sat?
-        case value => value
-      }
-    }
+    def nextClosingFee(localClosingFee: Satoshi, remoteClosingFee: Satoshi): Satoshi = ((localClosingFee + remoteClosingFee) / 4) * 2
 
     def generateTx(desc: String)(attempt: Try[TransactionWithInputInfo]): Option[TransactionWithInputInfo] = {
       attempt match {
@@ -271,12 +221,12 @@ object Helpers {
       val localRevocationPubkey = Generators.revocationPubKey(remoteParams.revocationBasepoint, localPerCommitmentPoint)
       val localDelayedPrivkey = Generators.derivePrivKey(localParams.delayedPaymentKey, localPerCommitmentPoint)
 
-      // for now we use the current commit's fee rate, it should be up-to-date
-      val feeratePerKw = localCommit.spec.feeratePerKw
+      // no need to use a high fee rate for delayed transactions (we are the only one who can spend them)
+      val feeratePerKwDelayed = Globals.feeratesPerKw.get.blocks_6
 
       // first we will claim our main output as soon as the delay is over
       val mainDelayedTx = generateTx("main-delayed-output")(Try {
-        val claimDelayed = Transactions.makeClaimDelayedOutputTx(tx, localRevocationPubkey, localParams.toSelfDelay, localDelayedPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKw)
+        val claimDelayed = Transactions.makeClaimDelayedOutputTx(tx, Satoshi(localParams.dustLimitSatoshis), localRevocationPubkey, localParams.toSelfDelay, localDelayedPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKwDelayed)
         val sig = Transactions.sign(claimDelayed, localDelayedPrivkey)
         Transactions.addSigs(claimDelayed, sig)
       })
@@ -292,7 +242,7 @@ object Helpers {
             Transactions.addSigs(txinfo, localSig, remoteSig, preimage)
           })
 
-        // NB: regarding htlc for which we don't have the preimage: nothing to do, it will timeout eventually and they will get their funds back
+        // (incoming htlc for which we don't have the preimage: nothing to do, it will timeout eventually and they will get their funds back)
 
         // outgoing htlc: they may or may not have the preimage, the only thing to do is try to get back our funds after timeout
         case HtlcTxAndSigs(txinfo: HtlcTimeoutTx, localSig, remoteSig) =>
@@ -305,7 +255,7 @@ object Helpers {
       val htlcDelayedTxes = htlcTxes.map {
         case txinfo: TransactionWithInputInfo => generateTx("claim-delayed-output")(Try {
           // TODO: we should use the current fee rate, not the initial fee rate that we get from localParams
-          val claimDelayed = Transactions.makeClaimDelayedOutputTx(txinfo.tx, localRevocationPubkey, localParams.toSelfDelay, localDelayedPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKw)
+          val claimDelayed = Transactions.makeClaimDelayedOutputTx(txinfo.tx, Satoshi(localParams.dustLimitSatoshis), localRevocationPubkey, localParams.toSelfDelay, localDelayedPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKwDelayed)
           val sig = Transactions.sign(claimDelayed, localDelayedPrivkey)
           Transactions.addSigs(claimDelayed, sig)
         })
@@ -320,7 +270,8 @@ object Helpers {
         claimMainDelayedOutputTx = mainDelayedTx.map(_.tx),
         htlcSuccessTxs = htlcTxes.collect { case c: HtlcSuccessTx => c.tx },
         htlcTimeoutTxs = htlcTxes.collect { case c: HtlcTimeoutTx => c.tx },
-        claimHtlcDelayedTx = htlcDelayedTxes.map(_.tx))
+        claimHtlcDelayedTx = htlcDelayedTxes.map(_.tx),
+        spent = Map.empty)
     }
 
     /**
@@ -336,20 +287,23 @@ object Helpers {
       val (remoteCommitTx, htlcTimeoutTxs, htlcSuccessTxs) = Commitments.makeRemoteTxs(remoteCommit.index, localParams, remoteParams, commitInput, remoteCommit.remotePerCommitmentPoint, remoteCommit.spec)
       require(remoteCommitTx.tx.txid == tx.txid, "txid mismatch, cannot recompute the current remote commit tx")
 
-      val remotePubkey = Generators.derivePubKey(remoteParams.paymentBasepoint, remoteCommit.remotePerCommitmentPoint)
-      val localPrivkey = Generators.derivePrivKey(localParams.paymentKey, remoteCommit.remotePerCommitmentPoint)
+      val localPaymentPrivkey = Generators.derivePrivKey(localParams.paymentKey, remoteCommit.remotePerCommitmentPoint)
+      val localHtlcPrivkey = Generators.derivePrivKey(localParams.htlcKey, remoteCommit.remotePerCommitmentPoint)
+      val remoteHtlcPubkey = Generators.derivePubKey(remoteParams.htlcBasepoint, remoteCommit.remotePerCommitmentPoint)
       val localPerCommitmentPoint = Generators.perCommitPoint(localParams.shaSeed, commitments.localCommit.index.toInt)
       val localRevocationPubKey = Generators.revocationPubKey(remoteParams.revocationBasepoint, localPerCommitmentPoint)
-      val remoteRevocationPubkey = Generators.revocationPubKey(localParams.revocationSecret.toPoint, remoteCommit.remotePerCommitmentPoint)
+      val remoteRevocationPubkey = Generators.revocationPubKey(localParams.revocationBasepoint, remoteCommit.remotePerCommitmentPoint)
 
-      // for now we use the same fee rate they used, it should be up-to-date
-      val feeratePerKw = remoteCommit.spec.feeratePerKw
+      // no need to use a high fee rate for our main output (we are the only one who can spend it)
+      val feeratePerKwMain = Globals.feeratesPerKw.get.blocks_6
+      // we need to use a rather high fee for htlc-claim because we compete with the counterparty
+      val feeratePerKwHtlc = Globals.feeratesPerKw.get.block_1
 
       // first we will claim our main output right away
       val mainTx = generateTx("claim-p2wpkh-output")(Try {
-        val claimMain = Transactions.makeClaimP2WPKHOutputTx(tx, localPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKw)
-        val sig = Transactions.sign(claimMain, localPrivkey)
-        Transactions.addSigs(claimMain, localPrivkey.publicKey, sig)
+        val claimMain = Transactions.makeClaimP2WPKHOutputTx(tx, Satoshi(localParams.dustLimitSatoshis), localPaymentPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKwMain)
+        val sig = Transactions.sign(claimMain, localPaymentPrivkey)
+        Transactions.addSigs(claimMain, localPaymentPrivkey.publicKey, sig)
       })
 
       // those are the preimages to existing received htlcs
@@ -358,17 +312,19 @@ object Helpers {
       // remember we are looking at the remote commitment so IN for them is really OUT for us and vice versa
       val txes = commitments.remoteCommit.spec.htlcs.collect {
         // incoming htlc for which we have the preimage: we spend it directly
-        case Htlc(OUT, add: UpdateAddHtlc, _) if preimages.exists(r => sha256(r) == add.paymentHash) => generateTx("claim-htlc-success")(Try {
+        case DirectedHtlc(OUT, add: UpdateAddHtlc) if preimages.exists(r => sha256(r) == add.paymentHash) => generateTx("claim-htlc-success")(Try {
           val preimage = preimages.find(r => sha256(r) == add.paymentHash).get
-          val tx = Transactions.makeClaimHtlcSuccessTx(remoteCommitTx.tx, localPrivkey.publicKey, remotePubkey, remoteRevocationPubkey, localParams.defaultFinalScriptPubKey, add, feeratePerKw)
-          val sig = Transactions.sign(tx, localPrivkey)
+          val tx = Transactions.makeClaimHtlcSuccessTx(remoteCommitTx.tx, Satoshi(localParams.dustLimitSatoshis), localHtlcPrivkey.publicKey, remoteHtlcPubkey, remoteRevocationPubkey, localParams.defaultFinalScriptPubKey, add, feeratePerKwHtlc)
+          val sig = Transactions.sign(tx, localHtlcPrivkey)
           Transactions.addSigs(tx, sig, preimage)
         })
-        // NB: incoming htlc for which we don't have the preimage: nothing to do, it will timeout eventually and they will get their funds back
+
+        // (incoming htlc for which we don't have the preimage: nothing to do, it will timeout eventually and they will get their funds back)
+
         // outgoing htlc: they may or may not have the preimage, the only thing to do is try to get back our funds after timeout
-        case Htlc(IN, add: UpdateAddHtlc, _) => generateTx("claim-htlc-timeout")(Try {
-          val tx = Transactions.makeClaimHtlcTimeoutTx(remoteCommitTx.tx, localPrivkey.publicKey, remotePubkey, remoteRevocationPubkey, localParams.defaultFinalScriptPubKey, add, feeratePerKw)
-          val sig = Transactions.sign(tx, localPrivkey)
+        case DirectedHtlc(IN, add: UpdateAddHtlc) => generateTx("claim-htlc-timeout")(Try {
+          val tx = Transactions.makeClaimHtlcTimeoutTx(remoteCommitTx.tx, Satoshi(localParams.dustLimitSatoshis), localHtlcPrivkey.publicKey, remoteHtlcPubkey, remoteRevocationPubkey, localParams.defaultFinalScriptPubKey, add, feeratePerKwHtlc)
+          val sig = Transactions.sign(tx, localHtlcPrivkey)
           Transactions.addSigs(tx, sig)
         })
       }.toSeq.flatten
@@ -379,8 +335,9 @@ object Helpers {
       RemoteCommitPublished(
         commitTx = tx,
         claimMainOutputTx = mainTx.map(_.tx),
-        claimHtlcSuccessTxs = txes.collect { case c: ClaimHtlcSuccessTx => c.tx },
-        claimHtlcTimeoutTxs = txes.collect { case c: ClaimHtlcTimeoutTx => c.tx }
+        claimHtlcSuccessTxs = txes.toList.collect { case c: ClaimHtlcSuccessTx => c.tx },
+        claimHtlcTimeoutTxs = txes.toList.collect { case c: ClaimHtlcTimeoutTx => c.tx },
+        spent = Map.empty
       )
 
     }
@@ -399,7 +356,7 @@ object Helpers {
       require(tx.txIn.size == 1, "commitment tx should have 1 input")
       val obscuredTxNumber = Transactions.decodeTxNumber(tx.txIn(0).sequence, tx.lockTime)
       // this tx has been published by remote, so we need to invert local/remote params
-      val txnumber = Transactions.obscuredCommitTxNumber(obscuredTxNumber, !localParams.isFunder, remoteParams.paymentBasepoint, localParams.paymentKey.toPoint)
+      val txnumber = Transactions.obscuredCommitTxNumber(obscuredTxNumber, !localParams.isFunder, remoteParams.paymentBasepoint, localParams.paymentBasepoint)
       require(txnumber <= 0xffffffffffffL, "txnumber must be lesser than 48 bits long")
       logger.warn(s"counterparty has published revoked commit txnumber=$txnumber")
       // now we know what commit number this tx is referring to, we can derive the commitment point from the shachain
@@ -408,16 +365,18 @@ object Helpers {
         .map { remotePerCommitmentSecret =>
           val remotePerCommitmentPoint = remotePerCommitmentSecret.toPoint
 
-          val remoteDelayedPubkey = Generators.derivePubKey(remoteParams.delayedPaymentBasepoint, remotePerCommitmentPoint)
+          val remoteDelayedPaymentPubkey = Generators.derivePubKey(remoteParams.delayedPaymentBasepoint, remotePerCommitmentPoint)
           val remoteRevocationPrivkey = Generators.revocationPrivKey(localParams.revocationSecret, remotePerCommitmentSecret)
           val localPrivkey = Generators.derivePrivKey(localParams.paymentKey, remotePerCommitmentPoint)
 
-          // for now we use the current commit's fee rate, it should be up-to-date
-          val feeratePerKw = localCommit.spec.feeratePerKw
+          // no need to use a high fee rate for our main output (we are the only one who can spend it)
+          val feeratePerKwMain = Globals.feeratesPerKw.get.blocks_6
+          // we need to use a high fee here for punishment txes because after a delay they can be spent by the counterparty
+          val feeratePerKwPenalty = Globals.feeratesPerKw.get.block_1
 
           // first we will claim our main output right away
           val mainTx = generateTx("claim-p2wpkh-output")(Try {
-            val claimMain = Transactions.makeClaimP2WPKHOutputTx(tx, localPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKw)
+            val claimMain = Transactions.makeClaimP2WPKHOutputTx(tx, Satoshi(localParams.dustLimitSatoshis), localPrivkey.publicKey, localParams.defaultFinalScriptPubKey, feeratePerKwMain)
             val sig = Transactions.sign(claimMain, localPrivkey)
             Transactions.addSigs(claimMain, localPrivkey.publicKey, sig)
           })
@@ -425,7 +384,7 @@ object Helpers {
           // then we punish them by stealing their main output
           val mainPenaltyTx = generateTx("main-penalty")(Try {
             // TODO: we should use the current fee rate, not the initial fee rate that we get from localParams
-            val txinfo = Transactions.makeMainPenaltyTx(tx, remoteRevocationPrivkey.publicKey, localParams.defaultFinalScriptPubKey, remoteParams.toSelfDelay, remoteDelayedPubkey, feeratePerKw)
+            val txinfo = Transactions.makeMainPenaltyTx(tx, Satoshi(localParams.dustLimitSatoshis), remoteRevocationPrivkey.publicKey, localParams.defaultFinalScriptPubKey, remoteParams.toSelfDelay, remoteDelayedPaymentPubkey, feeratePerKwPenalty)
             val sig = Transactions.sign(txinfo, remoteRevocationPrivkey)
             Transactions.addSigs(txinfo, sig)
           })
@@ -442,11 +401,145 @@ object Helpers {
             mainPenaltyTx = mainPenaltyTx.map(_.tx),
             claimHtlcTimeoutTxs = Nil,
             htlcTimeoutTxs = Nil,
-            htlcPenaltyTxs = Nil
+            htlcPenaltyTxs = Nil,
+            spent = Map.empty
           )
         }
     }
 
+    /**
+      * In CLOSING state, when we are notified that a transaction has been confirmed, we check if this tx belongs in the
+      * local commit scenario and keep track of it.
+      *
+      * We need to keep track of all transactions spending the outputs of the commitment tx, because some outputs can be
+      * spent both by us and our counterparty. Because of that, some of our transactions may never confirm and we don't
+      * want to wait forever before declaring that the channel is CLOSED.
+      *
+      * @param localCommitPublished
+      * @return
+      */
+    def updateLocalCommitPublished(localCommitPublished: LocalCommitPublished, tx: Transaction) = {
+      // even if our txes only have one input, maybe our counterparty uses a different scheme so we need to iterate
+      // over all of them to check if they are relevant
+      val relevantOutpoints = tx.txIn.map(_.outPoint).filter { outPoint =>
+        // is this the commit tx itself ? (we could do this outside of the loop...)
+        val isCommitTx = localCommitPublished.commitTx.txid == tx.txid
+        // does the tx spend an output of the local commitment tx?
+        val spendsTheCommitTx = localCommitPublished.commitTx.txid == outPoint.txid
+        // is the tx one of our 3rd stage delayed txes? (a 3rd stage tx is a tx spending the output of an htlc tx, which
+        // is itself spending the output of the commitment tx)
+        val is3rdStageDelayedTx = localCommitPublished.claimHtlcDelayedTx.map(_.txid).contains(outPoint.txid)
+        isCommitTx || spendsTheCommitTx || is3rdStageDelayedTx
+      }
+      // then we add the relevant outpoints to the map keeping track of which txid spends which outpoint
+      localCommitPublished.copy(spent = localCommitPublished.spent ++ relevantOutpoints.map(o => (o -> tx.txid)).toMap)
+    }
+
+    /**
+      * In CLOSING state, when we are notified that a transaction has been confirmed, we check if this tx belongs in the
+      * remote commit scenario and keep track of it.
+      *
+      * We need to keep track of all transactions spending the outputs of the commitment tx, because some outputs can be
+      * spent both by us and our counterparty. Because of that, some of our transactions may never confirm and we don't
+      * want to wait forever before declaring that the channel is CLOSED.
+      *
+      * @param remoteCommitPublished
+      * @return
+      */
+    def updateRemoteCommitPublished(remoteCommitPublished: RemoteCommitPublished, tx: Transaction) = {
+      // even if our txes only have one input, maybe our counterparty uses a different scheme so we need to iterate
+      // over all of them to check if they are relevant
+      val relevantOutpoints = tx.txIn.map(_.outPoint).filter { outPoint =>
+        // is this the commit tx itself ? (we could do this outside of the loop...)
+        val isCommitTx = remoteCommitPublished.commitTx.txid == tx.txid
+        // does the tx spend an output of the local commitment tx?
+        val spendsTheCommitTx = remoteCommitPublished.commitTx.txid == outPoint.txid
+        // TODO: we don't currently spend htlc transactions
+        isCommitTx || spendsTheCommitTx
+      }
+      // then we add the relevant outpoints to the map keeping track of which txid spends which outpoint
+      remoteCommitPublished.copy(spent = remoteCommitPublished.spent ++ relevantOutpoints.map(o => (o -> tx.txid)).toMap)
+    }
+
+    /**
+      * In CLOSING state, when we are notified that a transaction has been confirmed, we check if this tx belongs in the
+      * revoked commit scenario and keep track of it.
+      *
+      * We need to keep track of all transactions spending the outputs of the commitment tx, because some outputs can be
+      * spent both by us and our counterparty. Because of that, some of our transactions may never confirm and we don't
+      * want to wait forever before declaring that the channel is CLOSED.
+      *
+      * @param revokedCommitPublished
+      * @return
+      */
+    def updateRevokedCommitPublished(revokedCommitPublished: RevokedCommitPublished, tx: Transaction) = {
+      // even if our txes only have one input, maybe our counterparty uses a different scheme so we need to iterate
+      // over all of them to check if they are relevant
+      val relevantOutpoints = tx.txIn.map(_.outPoint).filter { outPoint =>
+        // is this the commit tx itself ? (we could do this outside of the loop...)
+        val isCommitTx = revokedCommitPublished.commitTx.txid == tx.txid
+        // does the tx spend an output of the local commitment tx?
+        val spendsTheCommitTx = revokedCommitPublished.commitTx.txid == outPoint.txid
+        isCommitTx || spendsTheCommitTx
+      }
+      // then we add the relevant outpoints to the map keeping track of which txid spends which outpoint
+      revokedCommitPublished.copy(spent = revokedCommitPublished.spent ++ relevantOutpoints.map(o => (o -> tx.txid)).toMap)
+    }
+
+    /**
+      * A local commit is considered done when:
+      * - all commitment tx outputs that we can spend have been spent and confirmed (even if the spending tx was not ours)
+      * - all 3rd stage txes (txes spending htlc txes) have been confirmed
+      *
+      * @param localCommitPublished
+      * @return
+      */
+    def isLocalCommitDone(localCommitPublished: LocalCommitPublished) = {
+      // is the commitment tx buried? (we need to check this because we may not have nay outputs)
+      val isCommitTxConfirmed = localCommitPublished.spent.values.toSet.contains(localCommitPublished.commitTx.txid)
+      // are there remaining spendable outputs from the commitment tx? we just substract all known spent outputs from the ones we control
+      val commitOutputsSpendableByUs = (localCommitPublished.claimMainDelayedOutputTx.toSeq ++ localCommitPublished.htlcSuccessTxs ++ localCommitPublished.htlcTimeoutTxs)
+        .flatMap(_.txIn.map(_.outPoint)).toSet -- localCommitPublished.spent.keys
+      // which htlc delayed txes can we expect to be confirmed?
+      val unconfirmedHtlcDelayedTxes = localCommitPublished.claimHtlcDelayedTx
+        .filter(tx => (tx.txIn.map(_.outPoint.txid).toSet -- localCommitPublished.spent.values).isEmpty) // only the txes which parents are already confirmed may get confirmed (note that this also eliminates outputs that have been double-spent by a competing tx)
+        .filterNot(tx => localCommitPublished.spent.values.toSet.contains(tx.txid)) // has the tx already been confirmed?
+      isCommitTxConfirmed && commitOutputsSpendableByUs.isEmpty && unconfirmedHtlcDelayedTxes.isEmpty
+    }
+
+    /**
+      * A remote commit is considered done when all commitment tx outputs that we can spend have been spent and confirmed
+      * (even if the spending tx was not ours).
+      *
+      * @param remoteCommitPublished
+      * @return
+      */
+    def isRemoteCommitDone(remoteCommitPublished: RemoteCommitPublished) = {
+      // is the commitment tx buried? (we need to check this because we may not have nay outputs)
+      val isCommitTxConfirmed = remoteCommitPublished.spent.values.toSet.contains(remoteCommitPublished.commitTx.txid)
+      // are there remaining spendable outputs from the commitment tx?
+      val commitOutputsSpendableByUs = (remoteCommitPublished.claimMainOutputTx.toSeq ++ remoteCommitPublished.claimHtlcSuccessTxs ++ remoteCommitPublished.claimHtlcTimeoutTxs)
+        .flatMap(_.txIn.map(_.outPoint)).toSet -- remoteCommitPublished.spent.keys
+      isCommitTxConfirmed && commitOutputsSpendableByUs.isEmpty
+    }
+
+    /**
+      * A remote commit is considered done when all commitment tx outputs that we can spend have been spent and confirmed
+      * (even if the spending tx was not ours).
+      *
+      * @param revokedCommitPublished
+      * @return
+      */
+    def isRevokedCommitDone(revokedCommitPublished: RevokedCommitPublished) = {
+      // is the commitment tx buried? (we need to check this because we may not have nay outputs)
+      val isCommitTxConfirmed = revokedCommitPublished.spent.values.toSet.contains(revokedCommitPublished.commitTx.txid)
+      // are there remaining spendable outputs from the commitment tx?
+      val commitOutputsSpendableByUs = (revokedCommitPublished.claimMainOutputTx.toSeq ++ revokedCommitPublished.mainPenaltyTx)
+        .flatMap(_.txIn.map(_.outPoint)).toSet -- revokedCommitPublished.spent.keys
+      // TODO: we don't currently spend htlc transactions
+      isCommitTxConfirmed && commitOutputsSpendableByUs.isEmpty
+    }
+
   }
 
 }

eclair-core/src/main/scala/fr/acinq/eclair/channel/Register.scala

@@ -1,9 +1,9 @@
 package fr.acinq.eclair.channel
 
 import akka.actor.Status.Failure
-import akka.actor.{Actor, ActorContext, ActorLogging, ActorPath, ActorRef, Terminated}
+import akka.actor.{Actor, ActorLogging, ActorRef, Terminated}
 import fr.acinq.bitcoin.BinaryData
-import fr.acinq.eclair.channel.Register.{Forward, ForwardShortId}
+import fr.acinq.eclair.channel.Register.{Forward, ForwardFailure, ForwardShortId, ForwardShortIdFailure}
 
 /**
   * Created by PM on 26/01/2016.
@@ -42,23 +42,27 @@ class Register extends Actor with ActorLogging {
 
     case 'shortIds => sender ! shortIds
 
-    case Forward(channelId, msg) =>
+    case fwd@Forward(channelId, msg) =>
       channels.get(channelId) match {
         case Some(channel) => channel forward msg
-        case None => sender ! Failure(new RuntimeException(s"channel $channelId not found"))
+        case None => sender ! Failure(ForwardFailure(fwd))
       }
 
-    case ForwardShortId(shortChannelId, msg) =>
+    case fwd@ForwardShortId(shortChannelId, msg) =>
       shortIds.get(shortChannelId).flatMap(channels.get(_)) match {
         case Some(channel) => channel forward msg
-        case None => sender ! Failure(new RuntimeException(s"channel $shortChannelId not found"))
+        case None => sender ! Failure(ForwardShortIdFailure(fwd))
       }
   }
 }
 
 object Register {
+
   // @formatter:off
-  case class Forward(channelId: BinaryData, message: Any)
-  case class ForwardShortId(shortChannelId: Long, message: Any)
+  case class Forward[T](channelId: BinaryData, message: T)
+  case class ForwardShortId[T](shortChannelId: Long, message: T)
+
+  case class ForwardFailure[T](fwd: Forward[T]) extends RuntimeException(s"channel ${fwd.channelId} not found")
+  case class ForwardShortIdFailure[T](fwd: ForwardShortId[T]) extends RuntimeException(s"channel ${fwd.shortChannelId} not found")
   // @formatter:on
 }

eclair-core/src/main/scala/fr/acinq/eclair/crypto/BitStream.scala

@@ -0,0 +1,161 @@
+package fr.acinq.eclair.crypto
+
+import org.spongycastle.util.encoders.Hex
+
+import scala.annotation.tailrec
+
+/**
+  * Bit stream that can be written to and read at both ends (i.e. you can read from the end or the beginning of the stream)
+  *
+  * @param bytes    bits packed as bytes, the last byte is padded with 0s
+  * @param offstart offset at which the first bit is in the first byte
+  * @param offend   offset at which the last bit is in the last byte
+  */
+case class BitStream(bytes: Vector[Byte], offstart: Int, offend: Int) {
+
+  // offstart: 0 1 2 3 4 5 6 7
+  // offend: 7 6 5 4 3 2 1 0
+  import BitStream._
+
+  def bitCount = 8 * bytes.length - offstart - offend
+
+  def isEmpty = bitCount == 0
+
+  /**
+    * append a byte to a bitstream
+    *
+    * @param input byte to append
+    * @return an updated bitstream
+    */
+  def writeByte(input: Byte): BitStream = offend match {
+    case 0 => this.copy(bytes = this.bytes :+ input)
+    case shift =>
+      val input1 = input & 0xff
+      val last = ((bytes.last | (input1 >>> (8 - shift))) & 0xff).toByte
+      val next = ((input1 << shift) & 0xff).toByte
+      this.copy(bytes = bytes.dropRight(1) ++ Vector(last, next))
+  }
+
+  /**
+    * append bytes to a bitstream
+    *
+    * @param input bytes to append
+    * @return an udpdate bitstream
+    */
+  def writeBytes(input: Seq[Byte]): BitStream = input.foldLeft(this) { case (bs, b) => bs.writeByte(b) }
+
+  /**
+    * append a bit to a bistream
+    *
+    * @param bit bit to append
+    * @return an update bitstream
+    */
+  def writeBit(bit: Bit): BitStream = offend match {
+    case 0 if bit =>
+      BitStream(bytes :+ 0x80.toByte, offstart, 7)
+    case 0 =>
+      BitStream(bytes :+ 0x00.toByte, offstart, 7)
+    case n if bit =>
+      val last = (bytes.last + (1 << (offend - 1))).toByte
+      BitStream(bytes.updated(bytes.length - 1, last), offstart, offend - 1)
+    case n =>
+      BitStream(bytes, offstart, offend - 1)
+  }
+
+  /**
+    * append bits to a bistream
+    *
+    * @param input bits to append
+    * @return an update bitstream
+    */
+  def writeBits(input: Seq[Bit]): BitStream = input.foldLeft(this) { case (bs, b) => bs.writeBit(b) }
+
+  /**
+    * read the last bit from a bitstream
+    *
+    * @return a (stream, bit) pair where stream is an updated bitstream and bit is the last bit
+    */
+  def popBit: (BitStream, Bit) = offend match {
+    case 7 => BitStream(bytes.dropRight(1), offstart, 0) -> lastBit
+    case n =>
+      val shift = n + 1
+      val last = (bytes.last >>> shift) << shift
+      BitStream(bytes.updated(bytes.length - 1, last.toByte), offstart, offend + 1) -> lastBit
+  }
+
+  /**
+    * read the last byte from a bitstream
+    *
+    * @return a (stream, byte) pair where stream is an updated bitstream and byte is the last byte
+    */
+  def popByte: (BitStream, Byte) = offend match {
+    case 0 => BitStream(bytes.dropRight(1), offstart, offend) -> bytes.last
+    case shift =>
+      val a = bytes(bytes.length - 2) & 0xff
+      val b = bytes(bytes.length - 1) & 0xff
+      val byte = ((a << (8 - shift)) | (b >>> shift)) & 0xff
+      val a1 = (a >>> shift) << shift
+      BitStream(bytes.dropRight(2) :+ a1.toByte, offstart, offend) -> byte.toByte
+  }
+
+  def popBytes(n: Int): (BitStream, Seq[Byte]) = {
+    @tailrec
+    def loop(stream: BitStream, acc: Seq[Byte]): (BitStream, Seq[Byte]) =
+      if (acc.length == n) (stream, acc) else {
+        val (stream1, value) = stream.popByte
+        loop(stream1, acc :+ value)
+      }
+
+    loop(this, Nil)
+  }
+
+  /**
+    * read the first bit from a bitstream
+    *
+    * @return
+    */
+  def readBit: (BitStream, Bit) = offstart match {
+    case 7 => BitStream(bytes.tail, 0, offend) -> firstBit
+    case _ => BitStream(bytes, offstart + 1, offend) -> firstBit
+  }
+
+  def readBits(count: Int): (BitStream, Seq[Bit]) = {
+    @tailrec
+    def loop(stream: BitStream, acc: Seq[Bit]): (BitStream, Seq[Bit]) = if (acc.length == count) (stream, acc) else {
+      val (stream1, bit) = stream.readBit
+      loop(stream1, acc :+ bit)
+    }
+
+    loop(this, Nil)
+  }
+
+  /**
+    * read the first byte from a bitstream
+    *
+    * @return
+    */
+  def readByte: (BitStream, Byte) = {
+    val byte = ((bytes(0) << offstart) | (bytes(1) >>> (7 - offstart))) & 0xff
+    BitStream(bytes.tail, offstart, offend) -> byte.toByte
+  }
+
+  def isSet(pos: Int): Boolean = {
+    val pos1 = pos + offstart
+    (bytes(pos1 / 8) & (1 << (7 - (pos1 % 8)))) != 0
+  }
+
+  def firstBit = (bytes.head & (1 << (7 - offstart))) != 0
+
+  def lastBit = (bytes.last & (1 << offend)) != 0
+
+  def toBinString: String = "0b" + (for (i <- 0 until bitCount) yield if (isSet(i)) '1' else '0').mkString
+
+  def toHexString: String = "0x" + Hex.toHexString(bytes.toArray).toLowerCase
+}
+
+object BitStream {
+  type Bit = Boolean
+  val Zero = false
+  val One = true
+  val empty = BitStream(Vector.empty[Byte], 0, 0)
+}

eclair-core/src/main/scala/fr/acinq/eclair/crypto/ChaCha20Poly1305.scala

@@ -4,8 +4,8 @@ import java.nio.ByteOrder
 
 import fr.acinq.bitcoin.{BinaryData, Protocol}
 import grizzled.slf4j.Logging
-import org.bouncycastle.crypto.engines.{ChaCha7539Engine, ChaChaEngine}
-import org.bouncycastle.crypto.params.{KeyParameter, ParametersWithIV}
+import org.spongycastle.crypto.engines.{ChaCha7539Engine, ChaChaEngine}
+import org.spongycastle.crypto.params.{KeyParameter, ParametersWithIV}
 
 /**
   * Poly1305 authenticator
@@ -20,7 +20,7 @@ object Poly1305 {
     */
   def mac(key: BinaryData, data: BinaryData): BinaryData = {
     val out = new Array[Byte](16)
-    val poly = new org.bouncycastle.crypto.macs.Poly1305()
+    val poly = new org.spongycastle.crypto.macs.Poly1305()
     poly.init(new KeyParameter(key))
     poly.update(data, 0, data.length)
     poly.doFinal(out, 0)

eclair-core/src/main/scala/fr/acinq/eclair/crypto/Noise.scala

@@ -3,12 +3,12 @@ package fr.acinq.eclair.crypto
 import java.math.BigInteger
 import java.nio.ByteOrder
 
-import fr.acinq.eclair.randomBytes
 import fr.acinq.bitcoin.{BinaryData, Crypto, Protocol}
+import fr.acinq.eclair.randomBytes
 import grizzled.slf4j.Logging
-import org.bouncycastle.crypto.digests.SHA256Digest
-import org.bouncycastle.crypto.macs.HMac
-import org.bouncycastle.crypto.params.KeyParameter
+import org.spongycastle.crypto.digests.SHA256Digest
+import org.spongycastle.crypto.macs.HMac
+import org.spongycastle.crypto.params.KeyParameter
 
 /**
   * see http://noiseprotocol.org/

eclair-core/src/main/scala/fr/acinq/eclair/crypto/ShaChain.scala

@@ -97,7 +97,7 @@ object ShaChain {
   }
 
 
-  val shaChainCodec: Codec[ShaChain] =  {
+  val shaChainCodec: Codec[ShaChain] = {
     import scodec.Codec
     import scodec.bits.BitVector
     import scodec.codecs._
@@ -106,7 +106,7 @@ object ShaChain {
     val entryCodec = vectorOfN(uint16, bool) ~ LightningMessageCodecs.varsizebinarydata
 
     // codec for a Map[Vector[Boolean], BinaryData]: write all k ->v pairs using the codec defined above
-    val mapCodec: Codec[Map[Vector[Boolean], BinaryData]] = Codec[Map[Vector[Boolean], BinaryData]] (
+    val mapCodec: Codec[Map[Vector[Boolean], BinaryData]] = Codec[Map[Vector[Boolean], BinaryData]](
       (m: Map[Vector[Boolean], BinaryData]) => vectorOfN(uint16, entryCodec).encode(m.toVector),
       (b: BitVector) => vectorOfN(uint16, entryCodec).decode(b).map(_.map(_.toMap))
     )

eclair-core/src/main/scala/fr/acinq/eclair/crypto/Sphinx.scala

@@ -5,10 +5,11 @@ import java.nio.ByteOrder
 
 import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
 import fr.acinq.bitcoin.{BinaryData, Crypto, Protocol}
-import fr.acinq.eclair.wire.{ChannelUpdate, FailureMessage, FailureMessageCodecs, LightningMessageCodecs}
-import org.bouncycastle.crypto.digests.SHA256Digest
-import org.bouncycastle.crypto.macs.HMac
-import org.bouncycastle.crypto.params.KeyParameter
+import fr.acinq.eclair.wire.{FailureMessage, FailureMessageCodecs}
+import grizzled.slf4j.Logging
+import org.spongycastle.crypto.digests.SHA256Digest
+import org.spongycastle.crypto.macs.HMac
+import org.spongycastle.crypto.params.KeyParameter
 import scodec.bits.BitVector
 
 import scala.annotation.tailrec
@@ -17,8 +18,8 @@ import scala.annotation.tailrec
   * Created by fabrice on 13/01/17.
   * see https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md
   */
-object Sphinx {
-  val Version = 1.toByte
+object Sphinx extends Logging {
+  val Version = 0.toByte
 
   // length of a MAC
   val MacLength = 32
@@ -31,7 +32,7 @@ object Sphinx {
 
   // onion packet length
   val PacketLength = 1 + 33 + MacLength + MaxHops * (PayloadLength + MacLength)
-  
+
   // last packet (all zeroes except for the version byte)
   val LAST_PACKET = Packet(Version, zeroes(33), zeroes(MacLength), zeroes(MaxHops * (PayloadLength + MacLength)))
 
@@ -268,9 +269,10 @@ object Sphinx {
     +----------------+----------------------------------+-----------------+----------------------+-----+
     | HMAC(32 bytes) | failure message length (2 bytes) | failure message | pad length (2 bytes) | pad |
     +----------------+----------------------------------+-----------------+----------------------+-----+
-    with failure message length + pad length = 128
+    with failure message length + pad length = 256
    */
-  val ErrorPacketLength = MacLength + 128 + 2 + 2
+  val MaxErrorPayloadLength = 256
+  val ErrorPacketLength = MacLength + MaxErrorPayloadLength + 2 + 2
 
   /**
     *
@@ -281,10 +283,13 @@ object Sphinx {
     */
   def createErrorPacket(sharedSecret: BinaryData, failure: FailureMessage): BinaryData = {
     val message: BinaryData = FailureMessageCodecs.failureMessageCodec.encode(failure).require.toByteArray
-    require(message.length <= 128, s"error message length is ${message.length}, it must be less than 128")
+    require(message.length <= MaxErrorPayloadLength, s"error message length is ${message.length}, it must be less than $MaxErrorPayloadLength")
     val um = Sphinx.generateKey("um", sharedSecret)
-    val padlen = 128 - message.length
+    val padlen = MaxErrorPayloadLength - message.length
     val payload = Protocol.writeUInt16(message.length, ByteOrder.BIG_ENDIAN) ++ message ++ Protocol.writeUInt16(padlen, ByteOrder.BIG_ENDIAN) ++ Sphinx.zeroes(padlen)
+    logger.debug(s"um key: $um")
+    logger.debug(s"error payload: ${BinaryData(payload)}")
+    logger.debug(s"raw error packet: ${BinaryData(Sphinx.mac(um, payload) ++ payload)}")
     forwardErrorPacket(Sphinx.mac(um, payload) ++ payload, sharedSecret)
   }
 
@@ -297,7 +302,7 @@ object Sphinx {
     require(packet.length == ErrorPacketLength, s"invalid error packet length ${packet.length}, must be $ErrorPacketLength")
     val (mac, payload) = packet.splitAt(Sphinx.MacLength)
     val len = Protocol.uint16(payload, ByteOrder.BIG_ENDIAN)
-    require((len >= 0) && (len <= 128), "message length must be less than 128")
+    require((len >= 0) && (len <= MaxErrorPayloadLength), s"message length must be less than $MaxErrorPayloadLength")
     FailureMessageCodecs.failureMessageCodec.decode(BitVector(payload.drop(2).take(len))).require.value
   }
 
@@ -311,6 +316,8 @@ object Sphinx {
     require(packet.length == ErrorPacketLength, s"invalid error packet length ${packet.length}, must be $ErrorPacketLength")
     val key = generateKey("ammag", sharedSecret)
     val stream = generateStream(key, ErrorPacketLength)
+    logger.debug(s"ammag key: $key")
+    logger.debug(s"error stream: $stream")
     Sphinx.xor(packet, stream)
   }
 

eclair-core/src/main/scala/fr/acinq/eclair/crypto/TransportHandler.scala

@@ -2,16 +2,18 @@ package fr.acinq.eclair.crypto
 
 import java.nio.ByteOrder
 
-import akka.actor.{Actor, ActorRef, FSM, Terminated}
+import akka.actor.{Actor, ActorRef, FSM, Props, Terminated}
 import akka.io.Tcp.{PeerClosed, _}
 import akka.util.ByteString
 import fr.acinq.bitcoin.Crypto.PublicKey
 import fr.acinq.bitcoin.{BinaryData, Protocol}
 import fr.acinq.eclair.crypto.Noise._
+import fr.acinq.eclair.io.WriteAckSender
+import scodec.bits.BitVector
+import scodec.{Attempt, Codec, DecodeResult}
 
 import scala.annotation.tailrec
 import scala.reflect.ClassTag
-import scala.util.{Failure, Success, Try}
 
 /**
   * see BOLT #8
@@ -22,16 +24,20 @@ import scala.util.{Failure, Success, Try}
   * Once the initial handshake has been completed successfully, the handler will create a listener actor with the
   * provided factory, and will forward it all decrypted messages
   *
-  * @param keyPair private/public key pair for this node
-  * @param rs      remote node static public key (which must be known before we initiate communication)
-  * @param connection    actor that represents the other node's
+  * @param keyPair    private/public key pair for this node
+  * @param rs         remote node static public key (which must be known before we initiate communication)
+  * @param connection actor that represents the other node's
   */
-class TransportHandler[T: ClassTag](keyPair: KeyPair, rs: Option[BinaryData], connection: ActorRef, serializer: TransportHandler.Serializer[T]) extends Actor with FSM[TransportHandler.State, TransportHandler.Data] {
+class TransportHandler[T: ClassTag](keyPair: KeyPair, rs: Option[BinaryData], connection: ActorRef, codec: Codec[T]) extends Actor with FSM[TransportHandler.State, TransportHandler.Data] {
 
   import TransportHandler._
 
   connection ! akka.io.Tcp.Register(self)
 
+  val out = context.actorOf(Props(new WriteAckSender(connection)))
+
+  def buf(message: BinaryData): ByteString = ByteString.fromArray(message)
+
   // it means we initiate the dialog
   val isWriter = rs.isDefined
 
@@ -41,7 +47,7 @@ class TransportHandler[T: ClassTag](keyPair: KeyPair, rs: Option[BinaryData], co
     val state = makeWriter(keyPair, rs.get)
     val (state1, message, None) = state.write(BinaryData.empty)
     log.debug(s"sending prefix + $message")
-    connection ! Write(TransportHandler.prefix +: message)
+    out ! buf(TransportHandler.prefix +: message)
     state1
   } else {
     makeReader(keyPair)
@@ -49,10 +55,9 @@ class TransportHandler[T: ClassTag](keyPair: KeyPair, rs: Option[BinaryData], co
 
   def sendToListener(listener: ActorRef, plaintextMessages: Seq[BinaryData]) = {
     plaintextMessages.map(plaintext => {
-      Try(serializer.deserialize(plaintext)) match {
-        case Success(message) => listener ! message
-        case Failure(t) =>
-          log.error(t, s"cannot deserialize $plaintext")
+      codec.decode(BitVector(plaintext.data)) match {
+        case Attempt.Successful(DecodeResult(message, _)) => listener ! message
+        case Attempt.Failure(err) => log.error(s"cannot deserialize $plaintext: $err")
       }
     })
   }
@@ -82,11 +87,11 @@ class TransportHandler[T: ClassTag](keyPair: KeyPair, rs: Option[BinaryData], co
                 // we're still in the middle of the handshake process and the other end must first received our next
                 // message before they can reply
                 require(remainder.isEmpty, "unexpected additional data received during handshake")
-                connection ! Write(TransportHandler.prefix +: message)
+                out ! buf(TransportHandler.prefix +: message)
                 stay using HandshakeData(reader1, remainder)
               }
               case (_, message, Some((enc, dec, ck))) => {
-                connection ! Write(TransportHandler.prefix +: message)
+                out ! buf(TransportHandler.prefix +: message)
                 val remoteNodeId = PublicKey(writer.rs)
                 context.parent ! HandshakeCompleted(self, remoteNodeId)
                 val nextStateData = WaitingForListenerData(ExtendedCipherState(enc, ck), ExtendedCipherState(dec, ck), remainder)
@@ -118,9 +123,9 @@ class TransportHandler[T: ClassTag](keyPair: KeyPair, rs: Option[BinaryData], co
       stay using nextStateData
 
     case Event(t: T, WaitingForCyphertextData(enc, dec, length, buffer, listener)) =>
-      val blob = serializer.serialize(t)
+      val blob = codec.encode(t).require.toByteArray
       val (enc1, ciphertext) = TransportHandler.encrypt(enc, blob)
-      connection ! Write(ByteString.fromArray(ciphertext.toArray))
+      out ! buf(ciphertext)
       stay using WaitingForCyphertextData(enc1, dec, length, buffer, listener)
   }
 
@@ -210,6 +215,7 @@ object TransportHandler {
   // @formatter:on
 
   case class Listener(listener: ActorRef)
+
   case class HandshakeCompleted(transport: ActorRef, remoteNodeId: PublicKey)
 
   sealed trait Data
@@ -283,16 +289,4 @@ object TransportHandler {
     }
   }
 
-  trait Serializer[T] {
-    def serialize(t: T): BinaryData
-
-    def deserialize(bin: BinaryData): T
-  }
-
-  object Noop extends Serializer[BinaryData] {
-    override def serialize(t: BinaryData): BinaryData = t
-
-    override def deserialize(bin: BinaryData): BinaryData = bin
-  }
-
 }

eclair-core/src/main/scala/fr/acinq/eclair/db/ChannelsDb.scala

@@ -0,0 +1,14 @@
+package fr.acinq.eclair.db
+
+import fr.acinq.bitcoin.BinaryData
+import fr.acinq.eclair.channel.HasCommitments
+
+trait ChannelsDb {
+
+  def addOrUpdateChannel(state: HasCommitments)
+
+  def removeChannel(channelId: BinaryData)
+
+  def listChannels(): List[HasCommitments]
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/Dbs.scala

@@ -1,59 +0,0 @@
-package fr.acinq.eclair.db
-
-import fr.acinq.bitcoin.BinaryData
-import fr.acinq.bitcoin.Crypto.PublicKey
-import fr.acinq.eclair.channel.Data
-import fr.acinq.eclair.crypto.TransportHandler.Serializer
-import fr.acinq.eclair.io.{LightningMessageSerializer, PeerRecord}
-import fr.acinq.eclair.wire.LightningMessage
-
-/**
-  * Created by PM on 28/02/2017.
-  */
-object Dbs {
-
-  def makeChannelDb(db: SimpleDb): SimpleTypedDb[BinaryData, Data] = {
-    def channelid2String(id: BinaryData) = s"channel-$id"
-
-    def string2channelid(s: String) = if (s.startsWith("channel-")) Some(BinaryData(s.stripPrefix("channel-"))) else None
-
-    new SimpleTypedDb[BinaryData, Data](
-      channelid2String,
-      string2channelid,
-      new Serializer[Data] {
-        override def serialize(t: Data): BinaryData = JavaSerializer.serialize(t)
-
-        override def deserialize(bin: BinaryData): Data = JavaSerializer.deserialize[Data](bin)
-      },
-      db
-    )
-  }
-
-  def makeAnnouncementDb(db: SimpleDb): SimpleTypedDb[String, LightningMessage] = {
-    // we use a single key: router.state
-    new SimpleTypedDb[String, LightningMessage](
-      s => s,
-      s => if (s.startsWith("ann-")) Some(s) else None,
-      LightningMessageSerializer,
-      db
-    )
-  }
-
-  def makePeerDb(db: SimpleDb): SimpleTypedDb[PublicKey, PeerRecord] = {
-    def peerid2String(id: PublicKey) = s"peer-$id"
-
-    def string2peerid(s: String) = if (s.startsWith("peer-")) Some(PublicKey(BinaryData(s.stripPrefix("peer-")))) else None
-
-    new SimpleTypedDb[PublicKey, PeerRecord](
-      peerid2String,
-      string2peerid,
-      new Serializer[PeerRecord] {
-        override def serialize(t: PeerRecord): BinaryData = JavaSerializer.serialize(t)
-
-        override def deserialize(bin: BinaryData): PeerRecord = JavaSerializer.deserialize[PeerRecord](bin)
-      },
-      db
-    )
-  }
-
-}

eclair-core/src/main/scala/fr/acinq/eclair/db/JavaSerializer.scala

@@ -1,24 +0,0 @@
-package fr.acinq.eclair.db
-
-import java.io.{ByteArrayInputStream, ByteArrayOutputStream, ObjectInputStream, ObjectOutputStream}
-
-import fr.acinq.bitcoin.BinaryData
-
-
-/**
-  * Created by fabrice on 17/02/17.
-  */
-object JavaSerializer {
-  def serialize[T](cs: T): BinaryData = {
-    val bos = new ByteArrayOutputStream()
-    val oos = new ObjectOutputStream(bos)
-    oos.writeObject(cs)
-    bos.toByteArray
-  }
-
-  def deserialize[T](input: BinaryData): T = {
-    val bis = new ByteArrayInputStream(input)
-    val osi = new ObjectInputStream(bis)
-    osi.readObject().asInstanceOf[T]
-  }
-}

eclair-core/src/main/scala/fr/acinq/eclair/db/NetworkDb.scala

@@ -0,0 +1,34 @@
+package fr.acinq.eclair.db
+
+import fr.acinq.bitcoin.Crypto.PublicKey
+import fr.acinq.eclair.wire.{ChannelAnnouncement, ChannelUpdate, NodeAnnouncement}
+
+trait NetworkDb {
+
+  def addNode(n: NodeAnnouncement)
+
+  def updateNode(n: NodeAnnouncement)
+
+  def removeNode(nodeId: PublicKey)
+
+  def listNodes(): List[NodeAnnouncement]
+
+  def addChannel(c: ChannelAnnouncement)
+
+  /**
+    * This method removes 1 channel announcement and 2 channel updates (at both ends of the same channel)
+    *
+    * @param shortChannelId
+    * @return
+    */
+  def removeChannel(shortChannelId: Long)
+
+  def listChannels(): List[ChannelAnnouncement]
+
+  def addChannelUpdate(u: ChannelUpdate)
+
+  def updateChannelUpdate(u: ChannelUpdate)
+
+  def listChannelUpdates(): List[ChannelUpdate]
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/PeersDb.scala

@@ -0,0 +1,15 @@
+package fr.acinq.eclair.db
+
+import java.net.InetSocketAddress
+
+import fr.acinq.bitcoin.Crypto.PublicKey
+
+trait PeersDb {
+
+  def addOrUpdatePeer(nodeId: PublicKey, address: InetSocketAddress)
+
+  def removePeer(nodeId: PublicKey)
+
+  def listPeers(): List[(PublicKey, InetSocketAddress)]
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/PreimagesDb.scala

@@ -0,0 +1,25 @@
+package fr.acinq.eclair.db
+
+import fr.acinq.bitcoin.BinaryData
+
+/**
+  * This database stores the preimages that we have received from downstream
+  * (either directly via UpdateFulfillHtlc or by extracting the value from the
+  * blockchain).
+  *
+  * This means that this database is only used in the context of *relaying* payments.
+  *
+  * We need to be sure that if downstream is able to pulls funds from us, we can always
+  * do the same from upstream, otherwise we lose money. Hence the need for persistence
+  * to handle all corner cases.
+  *
+  */
+trait PreimagesDb {
+
+  def addPreimage(channelId: BinaryData, htlcId: Long, paymentPreimage: BinaryData)
+
+  def removePreimage(channelId: BinaryData, htlcId: Long)
+
+  def listPreimages(channelId: BinaryData): List[(BinaryData, Long, BinaryData)]
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/SimpleDb.scala

@@ -1,29 +0,0 @@
-package fr.acinq.eclair.db
-
-import fr.acinq.bitcoin.BinaryData
-import fr.acinq.eclair.crypto.TransportHandler
-
-/**
-  * Created by fabrice on 25/02/17.
-  */
-
-trait SimpleDb {
-  // @formatter:off
-  def put(k: String, v: BinaryData) : Unit
-  def get(k: String) : Option[BinaryData]
-  def delete(k: String) : Boolean
-  def keys: Seq[String]
-  def values: Seq[BinaryData]
-  // @formatter:on
-}
-
-class SimpleTypedDb[K, V](id2string: K => String, string2id: String => Option[K], serializer: TransportHandler.Serializer[V], db: SimpleDb) {
-  // @formatter:off
-  def put(k: K, v: V) = db.put(id2string(k), serializer.serialize(v))
-  def get(k: K): Option[V] = db.get(id2string(k)).map(serializer.deserialize)
-  def delete(k: K) : Boolean = db.delete(id2string(k))
-  def keys: Seq[K] = db.keys.map(string2id).flatten
-  def values: Seq[V] = keys.map(get).flatten
-  // @formatter:on
-}
-

eclair-core/src/main/scala/fr/acinq/eclair/db/SimpleFileDb.scala

@@ -1,30 +0,0 @@
-package fr.acinq.eclair.db
-
-import java.io.File
-import java.nio.file.{Files, Paths}
-
-import fr.acinq.bitcoin.BinaryData
-import grizzled.slf4j.Logging
-
-import scala.util.Try
-
-/**
-  * Created by fabrice on 25/02/17.
-  */
-case class SimpleFileDb(root: File) extends SimpleDb with Logging {
-
-  root.mkdirs()
-
-  override def put(key: String, value: BinaryData): Unit = {
-    logger.debug(s"put $key -> $value")
-    Files.write(Paths.get(root.getPath, key), value)
-  }
-
-  override def get(key: String): Option[BinaryData] = Try(Files.readAllBytes(Paths.get(root.getPath, key))).toOption.map(a => BinaryData(a))
-
-  override def delete(key: String): Boolean = Paths.get(root.getPath, key).toFile.delete()
-
-  override def keys: Seq[String] = root.list()
-
-  override def values: Seq[BinaryData] = keys.map(get).flatten
-}

eclair-core/src/main/scala/fr/acinq/eclair/db/sqlite/SqliteChannelsDb.scala

@@ -0,0 +1,46 @@
+package fr.acinq.eclair.db.sqlite
+
+import java.sql.Connection
+
+import fr.acinq.bitcoin.BinaryData
+import fr.acinq.eclair.channel.HasCommitments
+import fr.acinq.eclair.db.ChannelsDb
+import fr.acinq.eclair.wire.ChannelCodecs.stateDataCodec
+
+class SqliteChannelsDb(sqlite: Connection) extends ChannelsDb {
+
+  import SqliteUtils._
+
+  {
+    val statement = sqlite.createStatement
+    statement.executeUpdate("CREATE TABLE IF NOT EXISTS local_channels (channel_id BLOB NOT NULL PRIMARY KEY, data BLOB NOT NULL)")
+  }
+
+  override def addOrUpdateChannel(state: HasCommitments): Unit = {
+    val data = stateDataCodec.encode(state).require.toByteArray
+    val update = sqlite.prepareStatement("UPDATE local_channels SET data=? WHERE channel_id=?")
+    update.setBytes(1, data)
+    update.setBytes(2, state.channelId)
+    if (update.executeUpdate() == 0) {
+      val statement = sqlite.prepareStatement("INSERT INTO local_channels VALUES (?, ?)")
+      statement.setBytes(1, state.channelId)
+      statement.setBytes(2, data)
+      statement.executeUpdate()
+    }
+  }
+
+  override def removeChannel(channelId: BinaryData): Unit = {
+    val statement1 = sqlite.prepareStatement("DELETE FROM preimages WHERE channel_id=?")
+    statement1.setBytes(1, channelId)
+    statement1.executeUpdate()
+
+    val statement2 = sqlite.prepareStatement("DELETE FROM local_channels WHERE channel_id=?")
+    statement2.setBytes(1, channelId)
+    statement2.executeUpdate()
+  }
+
+  override def listChannels(): List[HasCommitments] = {
+    val rs = sqlite.createStatement.executeQuery("SELECT data FROM local_channels")
+    codecList(rs, stateDataCodec)
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/sqlite/SqliteNetworkDb.scala

@@ -0,0 +1,90 @@
+package fr.acinq.eclair.db.sqlite
+
+import java.sql.Connection
+
+import fr.acinq.bitcoin.Crypto
+import fr.acinq.eclair.db.NetworkDb
+import fr.acinq.eclair.router.Announcements
+import fr.acinq.eclair.wire.LightningMessageCodecs.{channelAnnouncementCodec, channelUpdateCodec, nodeAnnouncementCodec}
+import fr.acinq.eclair.wire.{ChannelAnnouncement, ChannelUpdate, NodeAnnouncement}
+
+class SqliteNetworkDb(sqlite: Connection) extends NetworkDb {
+
+  import SqliteUtils._
+
+  {
+    val statement = sqlite.createStatement
+    statement.execute("PRAGMA foreign_keys = ON")
+    statement.executeUpdate("CREATE TABLE IF NOT EXISTS nodes (node_id BLOB NOT NULL PRIMARY KEY, data BLOB NOT NULL)")
+    statement.executeUpdate("CREATE TABLE IF NOT EXISTS channels (short_channel_id INTEGER NOT NULL PRIMARY KEY, data BLOB NOT NULL)")
+    statement.executeUpdate("CREATE TABLE IF NOT EXISTS channel_updates (short_channel_id INTEGER NOT NULL, node_flag INTEGER NOT NULL, data BLOB NOT NULL, PRIMARY KEY(short_channel_id, node_flag), FOREIGN KEY(short_channel_id) REFERENCES channels(short_channel_id))")
+    statement.executeUpdate("CREATE INDEX IF NOT EXISTS channel_updates_idx ON channel_updates(short_channel_id)")
+  }
+
+  override def addNode(n: NodeAnnouncement): Unit = {
+    val statement = sqlite.prepareStatement("INSERT OR IGNORE INTO nodes VALUES (?, ?)")
+    statement.setBytes(1, n.nodeId.toBin)
+    statement.setBytes(2, nodeAnnouncementCodec.encode(n).require.toByteArray)
+    statement.executeUpdate()
+  }
+
+  override def updateNode(n: NodeAnnouncement): Unit = {
+    val statement = sqlite.prepareStatement("UPDATE nodes SET data=? WHERE node_id=?")
+    statement.setBytes(1, nodeAnnouncementCodec.encode(n).require.toByteArray)
+    statement.setBytes(2, n.nodeId.toBin)
+    statement.executeUpdate()
+  }
+
+  override def removeNode(nodeId: Crypto.PublicKey): Unit = {
+    val statement = sqlite.prepareStatement("DELETE FROM nodes WHERE node_id=?")
+    statement.setBytes(1, nodeId.toBin)
+    statement.executeUpdate()
+  }
+
+  override def listNodes(): List[NodeAnnouncement] = {
+    val rs = sqlite.createStatement.executeQuery("SELECT data FROM nodes")
+    codecList(rs, nodeAnnouncementCodec)
+  }
+
+  override def addChannel(c: ChannelAnnouncement): Unit = {
+    val statement = sqlite.prepareStatement("INSERT OR IGNORE INTO channels VALUES (?, ?)")
+    statement.setLong(1, c.shortChannelId)
+    statement.setBytes(2, channelAnnouncementCodec.encode(c).require.toByteArray)
+    statement.executeUpdate()
+  }
+
+  override def removeChannel(shortChannelId: Long): Unit = {
+    val statement = sqlite.createStatement
+    statement.execute("BEGIN TRANSACTION")
+    statement.executeUpdate(s"DELETE FROM channel_updates WHERE short_channel_id=$shortChannelId")
+    statement.executeUpdate(s"DELETE FROM channels WHERE short_channel_id=$shortChannelId")
+    statement.execute("COMMIT TRANSACTION")
+  }
+
+  override def listChannels(): List[ChannelAnnouncement] = {
+    val rs = sqlite.createStatement.executeQuery("SELECT data FROM channels")
+    codecList(rs, channelAnnouncementCodec)
+  }
+
+  override def addChannelUpdate(u: ChannelUpdate): Unit = {
+    val statement = sqlite.prepareStatement("INSERT OR IGNORE INTO channel_updates VALUES (?, ?, ?)")
+    statement.setLong(1, u.shortChannelId)
+    statement.setBoolean(2, Announcements.isNode1(u.flags))
+    statement.setBytes(3, channelUpdateCodec.encode(u).require.toByteArray)
+    statement.executeUpdate()
+  }
+
+  override def updateChannelUpdate(u: ChannelUpdate): Unit = {
+    val statement = sqlite.prepareStatement("UPDATE channel_updates SET data=? WHERE short_channel_id=? AND node_flag=?")
+    statement.setBytes(1, channelUpdateCodec.encode(u).require.toByteArray)
+    statement.setLong(2, u.shortChannelId)
+    statement.setBoolean(3, Announcements.isNode1(u.flags))
+    statement.executeUpdate()
+  }
+
+  override def listChannelUpdates(): List[ChannelUpdate] = {
+    val rs = sqlite.createStatement.executeQuery("SELECT data FROM channel_updates")
+    codecList(rs, channelUpdateCodec)
+  }
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/sqlite/SqlitePeersDb.scala

@@ -0,0 +1,46 @@
+package fr.acinq.eclair.db.sqlite
+
+import java.net.InetSocketAddress
+import java.sql.Connection
+
+import fr.acinq.bitcoin.Crypto
+import fr.acinq.bitcoin.Crypto.PublicKey
+import fr.acinq.eclair.db.PeersDb
+import fr.acinq.eclair.wire.LightningMessageCodecs.socketaddress
+import scodec.bits.BitVector
+
+class SqlitePeersDb(sqlite: Connection) extends PeersDb {
+
+  {
+    val statement = sqlite.createStatement
+    statement.executeUpdate("CREATE TABLE IF NOT EXISTS peers (node_id BLOB NOT NULL PRIMARY KEY, data BLOB NOT NULL)")
+  }
+
+  override def addOrUpdatePeer(nodeId: Crypto.PublicKey, address: InetSocketAddress): Unit = {
+    val data = socketaddress.encode(address).require.toByteArray
+    val update = sqlite.prepareStatement("UPDATE peers SET data=? WHERE node_id=?")
+    update.setBytes(1, data)
+    update.setBytes(2, nodeId.toBin)
+    if (update.executeUpdate() == 0) {
+      val statement = sqlite.prepareStatement("INSERT INTO peers VALUES (?, ?)")
+      statement.setBytes(1, nodeId.toBin)
+      statement.setBytes(2, data)
+      statement.executeUpdate()
+    }
+  }
+
+  override def removePeer(nodeId: Crypto.PublicKey): Unit = {
+    val statement = sqlite.prepareStatement("DELETE FROM peers WHERE node_id=?")
+    statement.setBytes(1, nodeId.toBin)
+    statement.executeUpdate()
+  }
+
+  override def listPeers(): List[(PublicKey, InetSocketAddress)] = {
+    val rs = sqlite.createStatement.executeQuery("SELECT node_id, data FROM peers")
+    var l: List[(PublicKey, InetSocketAddress)] = Nil
+    while (rs.next()) {
+      l = l :+ (PublicKey(rs.getBytes("node_id")), socketaddress.decode(BitVector(rs.getBytes("data"))).require.value)
+    }
+    l
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/sqlite/SqlitePreimagesDb.scala

@@ -0,0 +1,41 @@
+package fr.acinq.eclair.db.sqlite
+
+import java.sql.Connection
+
+import fr.acinq.bitcoin.BinaryData
+import fr.acinq.eclair.db.PreimagesDb
+
+class SqlitePreimagesDb(sqlite: Connection) extends PreimagesDb {
+
+  {
+    val statement = sqlite.createStatement
+    // note: should we use a foreign key to local_channels table here?
+    statement.executeUpdate("CREATE TABLE IF NOT EXISTS preimages (channel_id BLOB NOT NULL, htlc_id INTEGER NOT NULL, preimage BLOB NOT NULL, PRIMARY KEY(channel_id, htlc_id))")
+  }
+
+  override def addPreimage(channelId: BinaryData, htlcId: Long, paymentPreimage: BinaryData): Unit = {
+    val statement = sqlite.prepareStatement("INSERT OR IGNORE INTO preimages VALUES (?, ?, ?)")
+    statement.setBytes(1, channelId)
+    statement.setLong(2, htlcId)
+    statement.setBytes(3, paymentPreimage)
+    statement.executeUpdate()
+  }
+
+  override def removePreimage(channelId: BinaryData, htlcId: Long): Unit = {
+    val statement = sqlite.prepareStatement("DELETE FROM preimages WHERE channel_id=? AND htlc_id=?")
+    statement.setBytes(1, channelId)
+    statement.setLong(2, htlcId)
+    statement.executeUpdate()
+  }
+
+  override def listPreimages(channelId: BinaryData): List[(BinaryData, Long, BinaryData)] = {
+    val statement = sqlite.prepareStatement("SELECT htlc_id, preimage FROM preimages WHERE channel_id=?")
+    statement.setBytes(1, channelId)
+    val rs = statement.executeQuery()
+    var l: List[(BinaryData, Long, BinaryData)] = Nil
+    while (rs.next()) {
+      l = l :+ (channelId, rs.getLong("htlc_id"), BinaryData(rs.getBytes("preimage")))
+    }
+    l
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/db/sqlite/SqliteUtils.scala

@@ -0,0 +1,27 @@
+package fr.acinq.eclair.db.sqlite
+
+import java.sql.ResultSet
+
+import scodec.Codec
+import scodec.bits.BitVector
+
+object SqliteUtils {
+
+  /**
+    * This helper assumes that there is a "data" column available, decodable with the provided codec
+    *
+    * TODO: we should use an scala.Iterator instead
+    *
+    * @param rs
+    * @param codec
+    * @tparam T
+    * @return
+    */
+  def codecList[T](rs: ResultSet, codec: Codec[T]): List[T] = {
+    var l: List[T] = Nil
+    while (rs.next()) {
+      l = l :+ codec.decode(BitVector(rs.getBytes("data"))).require.value
+    }
+    l
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/io/Client.scala

@@ -6,11 +6,11 @@ import akka.actor.{Props, _}
 import akka.io.Tcp.SO.KeepAlive
 import akka.io.{IO, Tcp}
 import fr.acinq.bitcoin.Crypto.PublicKey
-import fr.acinq.eclair.{Globals, NodeParams}
+import fr.acinq.eclair.NodeParams
 import fr.acinq.eclair.crypto.Noise.KeyPair
 import fr.acinq.eclair.crypto.TransportHandler
 import fr.acinq.eclair.crypto.TransportHandler.HandshakeCompleted
-import fr.acinq.eclair.wire.LightningMessage
+import fr.acinq.eclair.wire.{LightningMessage, LightningMessageCodecs}
 
 /**
   * Created by PM on 27/10/2015.
@@ -35,7 +35,7 @@ class Client(nodeParams: NodeParams, switchboard: ActorRef, address: InetSocketA
           KeyPair(nodeParams.privateKey.publicKey.toBin, nodeParams.privateKey.toBin),
           Some(remoteNodeId),
           connection = connection,
-          serializer = LightningMessageSerializer)))
+          codec = LightningMessageCodecs.lightningMessageCodec)))
       context watch transport
       context become authenticating(transport)
   }

eclair-core/src/main/scala/fr/acinq/eclair/io/LightningMessageSerializer.scala

@@ -1,25 +0,0 @@
-package fr.acinq.eclair.io
-
-import fr.acinq.bitcoin.BinaryData
-import fr.acinq.eclair.crypto.TransportHandler
-import fr.acinq.eclair.wire.{LightningMessageCodecs, LightningMessage}
-import scodec.bits.BitVector
-import scodec.{Attempt, DecodeResult}
-
-/**
-  * Created by fabrice on 16/01/17.
-  */
-object LightningMessageSerializer extends TransportHandler.Serializer[LightningMessage] {
-
-  override def serialize(t: LightningMessage): BinaryData =
-    LightningMessageCodecs.lightningMessageCodec.encode(t) match {
-      case Attempt.Successful(bitVector) => BinaryData(bitVector.toByteArray)
-      case Attempt.Failure(cause) => throw new RuntimeException(s"serialization error: $cause")
-    }
-
-  override def deserialize(bin: BinaryData): LightningMessage =
-    LightningMessageCodecs.lightningMessageCodec.decode(BitVector(bin.data)) match {
-      case Attempt.Successful(DecodeResult(msg, _)) => msg
-      case Attempt.Failure(cause) => throw new RuntimeException(s"deserialization error: $cause")
-    }
-}

eclair-core/src/main/scala/fr/acinq/eclair/io/Peer.scala

@@ -2,15 +2,16 @@ package fr.acinq.eclair.io
 
 import java.net.InetSocketAddress
 
-import akka.actor.{ActorRef, FSM, LoggingFSM, OneForOneStrategy, PoisonPill, Props, SupervisorStrategy, Terminated}
+import akka.actor.{ActorRef, LoggingFSM, OneForOneStrategy, PoisonPill, Props, SupervisorStrategy, Terminated}
 import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
 import fr.acinq.bitcoin.{BinaryData, Crypto, DeterministicWallet}
+import fr.acinq.eclair._
+import fr.acinq.eclair.blockchain.EclairWallet
 import fr.acinq.eclair.channel._
 import fr.acinq.eclair.crypto.TransportHandler.{HandshakeCompleted, Listener}
 import fr.acinq.eclair.io.Switchboard.{NewChannel, NewConnection}
 import fr.acinq.eclair.router.{Rebroadcast, SendRoutingState}
 import fr.acinq.eclair.wire._
-import fr.acinq.eclair._
 
 import scala.concurrent.duration._
 import scala.util.Random
@@ -29,7 +30,7 @@ case class TemporaryChannelId(id: BinaryData) extends ChannelId
 case class FinalChannelId(id: BinaryData) extends ChannelId
 
 sealed trait Data
-case class DisconnectedData(offlineChannels: Set[OfflineChannel]) extends Data
+case class DisconnectedData(offlineChannels: Set[OfflineChannel], attempts: Int = 0) extends Data
 case class InitializingData(transport: ActorRef, offlineChannels: Set[OfflineChannel]) extends Data
 case class ConnectedData(transport: ActorRef, remoteInit: Init, channels: Map[ChannelId, ActorRef]) extends Data
 
@@ -38,81 +39,63 @@ case object DISCONNECTED extends State
 case object INITIALIZING extends State
 case object CONNECTED extends State
 
-case class PeerRecord(id: PublicKey, address: InetSocketAddress)
-
 // @formatter:on
 
 /**
   * Created by PM on 26/08/2016.
   */
-class Peer(nodeParams: NodeParams, remoteNodeId: PublicKey, address_opt: Option[InetSocketAddress], watcher: ActorRef, router: ActorRef, relayer: ActorRef, defaultFinalScriptPubKey: BinaryData) extends LoggingFSM[State, Data] {
+class Peer(nodeParams: NodeParams, remoteNodeId: PublicKey, address_opt: Option[InetSocketAddress], watcher: ActorRef, router: ActorRef, relayer: ActorRef, wallet: EclairWallet, storedChannels: Set[HasCommitments]) extends LoggingFSM[State, Data] {
 
   import Peer._
 
-  startWith(DISCONNECTED, DisconnectedData(Set.empty))
+  val RECONNECT_TIMER = "reconnect"
 
-  when(DISCONNECTED, stateTimeout = if (nodeParams.autoReconnect) 60 seconds else null) {
-    case Event(state: HasCommitments, d@DisconnectedData(offlineChannels)) =>
-      val channel = spawnChannel(nodeParams, context.system.deadLetters)
-      channel ! INPUT_RESTORED(state)
-      self ! Reconnect
-      stay using d.copy(offlineChannels = offlineChannels + HotChannel(FinalChannelId(state.channelId), channel))
+  startWith(DISCONNECTED, DisconnectedData(offlineChannels = storedChannels.map { state =>
+    val channel = spawnChannel(nodeParams, context.system.deadLetters)
+    channel ! INPUT_RESTORED(state)
+    HotChannel(FinalChannelId(state.channelId), channel)
+  }, attempts = 0))
 
-    case Event(c: NewChannel, d@DisconnectedData(offlineChannels)) =>
-      self ! Reconnect
+  when(DISCONNECTED) {
+    case Event(c: NewChannel, d@DisconnectedData(offlineChannels, _)) =>
       stay using d.copy(offlineChannels = offlineChannels + BrandNewChannel(c))
 
-    case Event(Reconnect, _) =>
+    case Event(Reconnect, d@DisconnectedData(_, attempts)) =>
       address_opt match {
-        case Some(address) => context.parent forward NewConnection(remoteNodeId, address, None)
-        case None => {}
+        case None => stay // no-op (this peer didn't initiate the connection and doesn't have the ip of the counterparty)
+        case _ if d.offlineChannels.size == 0 => stay // no-op (no more channels with this peer)
+        case Some(address) =>
+          context.parent forward NewConnection(remoteNodeId, address, None)
+          // exponential backoff retry with a finite max
+          setTimer(RECONNECT_TIMER, Reconnect, Math.min(Math.pow(2, attempts), 60) seconds, repeat = false)
+          stay using d.copy(attempts = attempts + 1)
       }
-      stay
 
-    case Event(HandshakeCompleted(transport, _), DisconnectedData(offlineChannels)) =>
+    case Event(HandshakeCompleted(transport, _), DisconnectedData(offlineChannels, _)) =>
       log.info(s"registering as a listener to $transport")
       transport ! Listener(self)
       context watch transport
       transport ! Init(globalFeatures = nodeParams.globalFeatures, localFeatures = nodeParams.localFeatures)
+      // we store the ip upon successful connection, keeping only the most recent one
+      address_opt.map(address => nodeParams.peersDb.addOrUpdatePeer(remoteNodeId, address))
       goto(INITIALIZING) using InitializingData(transport, offlineChannels)
 
-    case Event(Terminated(actor), d@DisconnectedData(offlineChannels)) if offlineChannels.collect { case h: HotChannel if h.a == actor => h }.size >= 0 =>
+    case Event(Terminated(actor), d@DisconnectedData(offlineChannels, _)) if offlineChannels.collect { case h: HotChannel if h.a == actor => h }.size >= 0 =>
       val h = offlineChannels.collect { case h: HotChannel if h.a == actor => h }
       log.info(s"channel closed: channelId=${h.map(_.channelId).mkString("/")}")
       stay using d.copy(offlineChannels = offlineChannels -- h)
 
-    case Event(_: Rebroadcast, _) => stay // ignored
-
-    case Event("connected", _) => stay // ignored
-
-    case Event(StateTimeout, d: DisconnectedData) if d.offlineChannels.size == 0 =>
-      log.info(s"reconnect timeout triggered, but peer doesn't have any channels: closing the peer")
-      // NB: there is a possibility of a race with concurrent NewChannel requests because peer do not explicitly acknowledge them
-      // in that case the NewChannel request would simply go to DeadLetters (meaning: ignored)
-      stop(FSM.Normal)
-
-    case Event(StateTimeout, _) =>
-      log.info(s"attempting a reconnect")
-      self ! Reconnect
-      stay
+    case Event(_: Rebroadcast | "connected", _) => stay // ignored
   }
 
   when(INITIALIZING) {
-    case Event(state: HasCommitments, d@InitializingData(_, offlineChannels)) =>
-      val channel = spawnChannel(nodeParams, context.system.deadLetters)
-      channel ! INPUT_RESTORED(state)
-      stay using d.copy(offlineChannels = offlineChannels + HotChannel(FinalChannelId(state.channelId), channel))
-
     case Event(c: NewChannel, d@InitializingData(_, offlineChannels)) =>
       stay using d.copy(offlineChannels = offlineChannels + BrandNewChannel(c))
 
     case Event(remoteInit: Init, InitializingData(transport, offlineChannels)) =>
-      // we store the ip upon successful connection
-      address_opt.foreach(address => nodeParams.peersDb.put(remoteNodeId, PeerRecord(remoteNodeId, address)))
-      import fr.acinq.eclair.Features._
-      log.info(s"$remoteNodeId has features: channelPublic=${Features.isSet(remoteInit.localFeatures, CHANNELS_PUBLIC_BIT)} initialRoutingSync=${Features.isSet(remoteInit.localFeatures, INITIAL_ROUTING_SYNC_BIT)}")
+      log.info(s"$remoteNodeId has features: initialRoutingSync=${Features.initialRoutingSync(remoteInit.localFeatures)}")
       if (Features.areSupported(remoteInit.localFeatures)) {
-        if (Features.isSet(remoteInit.localFeatures, INITIAL_ROUTING_SYNC_BIT)) {
+        if (Features.initialRoutingSync(remoteInit.localFeatures)) {
           router ! SendRoutingState(transport)
         }
         // let's bring existing/requested channels online
@@ -161,12 +144,6 @@ class Peer(nodeParams: NodeParams, remoteNodeId: PublicKey, address_opt: Option[
       log.debug(s"received pong with ${data.length} bytes")
       stay
 
-    case Event(state: HasCommitments, d@ConnectedData(transport, _, channels)) =>
-      val channel = spawnChannel(nodeParams, context.system.deadLetters)
-      channel ! INPUT_RESTORED(state)
-      channel ! INPUT_RECONNECTED(transport)
-      stay using d.copy(channels = channels + (FinalChannelId(state.channelId) -> channel))
-
     case Event(err@Error(channelId, reason), ConnectedData(transport, _, channels)) if channelId == CHANNELID_ZERO =>
       log.error(s"connection-level error, failing all channels! reason=${new String(reason)}")
       channels.values.foreach(_ forward err)
@@ -201,7 +178,8 @@ class Peer(nodeParams: NodeParams, remoteNodeId: PublicKey, address_opt: Option[
       log.info(s"requesting a new channel to $remoteNodeId with fundingSatoshis=${c.fundingSatoshis} and pushMsat=${c.pushMsat}")
       val (channel, localParams) = createChannel(nodeParams, transport, funder = true, c.fundingSatoshis.toLong)
       val temporaryChannelId = randomBytes(32)
-      channel ! INPUT_INIT_FUNDER(temporaryChannelId, c.fundingSatoshis.amount, c.pushMsat.amount, Globals.feeratePerKw.get, localParams, transport, remoteInit)
+      val networkFeeratePerKw = Globals.feeratesPerKw.get.block_1
+      channel ! INPUT_INIT_FUNDER(temporaryChannelId, c.fundingSatoshis.amount, c.pushMsat.amount, networkFeeratePerKw, localParams, transport, remoteInit, c.channelFlags.getOrElse(nodeParams.channelFlags))
       stay using d.copy(channels = channels + (TemporaryChannelId(temporaryChannelId) -> channel))
 
     case Event(msg: OpenChannel, d@ConnectedData(transport, remoteInit, channels)) if !channels.contains(TemporaryChannelId(msg.temporaryChannelId)) =>
@@ -241,16 +219,30 @@ class Peer(nodeParams: NodeParams, remoteNodeId: PublicKey, address_opt: Option[
       }
       stay using d.copy(channels = channels -- channelIds)
 
+    case Event(h: HandshakeCompleted, ConnectedData(oldTransport, _, channels)) =>
+      log.info(s"got new transport while already connected, switching to new transport")
+      context unwatch oldTransport
+      oldTransport ! PoisonPill
+      channels.values.foreach(_ ! INPUT_DISCONNECTED)
+      val c: Set[OfflineChannel] = channels.map(c => HotChannel(c._1, c._2)).toSet
+      self ! h
+      goto(DISCONNECTED) using DisconnectedData(c)
+  }
+
+  onTransition {
+    case _ -> DISCONNECTED if nodeParams.autoReconnect && address_opt.isDefined => setTimer(RECONNECT_TIMER, Reconnect, 1 second, repeat = false)
+    case DISCONNECTED -> _ if nodeParams.autoReconnect && address_opt.isDefined => cancelTimer(RECONNECT_TIMER)
   }
 
   def createChannel(nodeParams: NodeParams, transport: ActorRef, funder: Boolean, fundingSatoshis: Long): (ActorRef, LocalParams) = {
+    val defaultFinalScriptPubKey = Helpers.getFinalScriptPubKey(wallet)
     val localParams = makeChannelParams(nodeParams, defaultFinalScriptPubKey, funder, fundingSatoshis)
     val channel = spawnChannel(nodeParams, transport)
     (channel, localParams)
   }
 
   def spawnChannel(nodeParams: NodeParams, transport: ActorRef): ActorRef = {
-    val channel = context.actorOf(Channel.props(nodeParams, remoteNodeId, watcher, router, relayer))
+    val channel = context.actorOf(Channel.props(nodeParams, wallet, remoteNodeId, watcher, router, relayer))
     context watch channel
     channel
   }
@@ -266,7 +258,7 @@ object Peer {
 
   val CHANNELID_ZERO = BinaryData("00" * 32)
 
-  def props(nodeParams: NodeParams, remoteNodeId: PublicKey, address_opt: Option[InetSocketAddress], watcher: ActorRef, router: ActorRef, relayer: ActorRef, defaultFinalScriptPubKey: BinaryData) = Props(new Peer(nodeParams, remoteNodeId, address_opt, watcher, router, relayer, defaultFinalScriptPubKey))
+  def props(nodeParams: NodeParams, remoteNodeId: PublicKey, address_opt: Option[InetSocketAddress], watcher: ActorRef, router: ActorRef, relayer: ActorRef, wallet: EclairWallet, storedChannels: Set[HasCommitments]) = Props(new Peer(nodeParams, remoteNodeId, address_opt, watcher, router, relayer, wallet: EclairWallet, storedChannels))
 
   def generateKey(nodeParams: NodeParams, keyPath: Seq[Long]): PrivateKey = DeterministicWallet.derivePrivateKey(nodeParams.extendedPrivateKey, keyPath).privateKey
 
@@ -286,8 +278,9 @@ object Peer {
       revocationSecret = generateKey(nodeParams, keyIndex :: 1L :: Nil),
       paymentKey = generateKey(nodeParams, keyIndex :: 2L :: Nil),
       delayedPaymentKey = generateKey(nodeParams, keyIndex :: 3L :: Nil),
+      htlcKey = generateKey(nodeParams, keyIndex :: 4L :: Nil),
       defaultFinalScriptPubKey = defaultFinalScriptPubKey,
-      shaSeed = Crypto.sha256(generateKey(nodeParams, keyIndex :: 4L :: Nil).toBin), // TODO: check that
+      shaSeed = Crypto.sha256(generateKey(nodeParams, keyIndex :: 5L :: Nil).toBin), // TODO: check that
       isFunder = isFunder,
       globalFeatures = nodeParams.globalFeatures,
       localFeatures = nodeParams.localFeatures)

eclair-core/src/main/scala/fr/acinq/eclair/io/Server.scala

@@ -2,14 +2,14 @@ package fr.acinq.eclair.io
 
 import java.net.InetSocketAddress
 
-import akka.actor.{Actor, ActorLogging, ActorRef, OneForOneStrategy, Props, Status, SupervisorStrategy, Terminated}
+import akka.actor.{Actor, ActorLogging, ActorRef, OneForOneStrategy, Props, SupervisorStrategy}
 import akka.io.Tcp.SO.KeepAlive
 import akka.io.{IO, Tcp}
 import fr.acinq.eclair.NodeParams
 import fr.acinq.eclair.crypto.Noise.KeyPair
 import fr.acinq.eclair.crypto.TransportHandler
 import fr.acinq.eclair.crypto.TransportHandler.HandshakeCompleted
-import fr.acinq.eclair.wire.LightningMessage
+import fr.acinq.eclair.wire.{LightningMessage, LightningMessageCodecs}
 
 import scala.concurrent.Promise
 
@@ -40,7 +40,7 @@ class Server(nodeParams: NodeParams, switchboard: ActorRef, address: InetSocketA
           KeyPair(nodeParams.privateKey.publicKey.toBin, nodeParams.privateKey.toBin),
           None,
           connection = connection,
-          serializer = LightningMessageSerializer)))
+          codec = LightningMessageCodecs.lightningMessageCodec)))
 
     case h: HandshakeCompleted =>
       log.info(s"handshake completed with ${h.remoteNodeId}")

eclair-core/src/main/scala/fr/acinq/eclair/io/Switchboard.scala

@@ -4,8 +4,9 @@ import java.net.InetSocketAddress
 
 import akka.actor.{Actor, ActorLogging, ActorRef, OneForOneStrategy, Props, Status, SupervisorStrategy, Terminated}
 import fr.acinq.bitcoin.Crypto.PublicKey
-import fr.acinq.bitcoin.{BinaryData, MilliSatoshi, Satoshi}
+import fr.acinq.bitcoin.{MilliSatoshi, Satoshi}
 import fr.acinq.eclair.NodeParams
+import fr.acinq.eclair.blockchain.EclairWallet
 import fr.acinq.eclair.channel.HasCommitments
 import fr.acinq.eclair.crypto.TransportHandler.HandshakeCompleted
 import fr.acinq.eclair.router.Rebroadcast
@@ -14,24 +15,30 @@ import fr.acinq.eclair.router.Rebroadcast
   * Ties network connections to peers.
   * Created by PM on 14/02/2017.
   */
-class Switchboard(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, relayer: ActorRef, defaultFinalScriptPubKey: BinaryData) extends Actor with ActorLogging {
+class Switchboard(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, relayer: ActorRef, wallet: EclairWallet) extends Actor with ActorLogging {
 
   import Switchboard._
 
-  def receive: Receive = main(Map(), Map())
+  // we load peers and channels from database
+  val initialPeers = {
+    val channels = nodeParams.channelsDb.listChannels().toList.groupBy(_.commitments.remoteParams.nodeId)
+    val peers = nodeParams.peersDb.listPeers().toMap
+    channels
+      .map {
+        case (remoteNodeId, states) => (remoteNodeId, states, peers.get(remoteNodeId))
+      }
+      .map {
+        case (remoteNodeId, states, address_opt) =>
+          // we might not have an address if we didn't initiate the connection in the first place
+          val peer = createOrGetPeer(Map(), remoteNodeId, address_opt, states.toSet)
+          (remoteNodeId -> peer)
+      }.toMap
+  }
+
+  def receive: Receive = main(initialPeers, Map())
 
   def main(peers: Map[PublicKey, ActorRef], connections: Map[PublicKey, ActorRef]): Receive = {
 
-    case PeerRecord(remoteNodeId, address) =>
-      val peer = createOrGetPeer(peers, remoteNodeId, Some(address))
-      context become main(peers + (remoteNodeId -> peer), connections)
-
-    case channelState: HasCommitments =>
-      val remoteNodeId = channelState.commitments.remoteParams.nodeId
-      val peer = createOrGetPeer(peers, remoteNodeId, None)
-      peer forward channelState
-      context become main(peers + (remoteNodeId -> peer), connections)
-
     case NewConnection(publicKey, _, _) if publicKey == nodeParams.privateKey.publicKey =>
       sender ! Status.Failure(new RuntimeException("cannot open connection with oneself"))
 
@@ -47,7 +54,7 @@ class Switchboard(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, r
           context watch (connection)
           connection
       }
-      val peer = createOrGetPeer(peers, remoteNodeId, Some(address))
+      val peer = createOrGetPeer(peers, remoteNodeId, Some(address), Set.empty)
       newChannel_opt.foreach(peer forward _)
       context become main(peers + (remoteNodeId -> peer), connections + (remoteNodeId -> connection))
 
@@ -59,11 +66,11 @@ class Switchboard(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, r
     case Terminated(actor) if peers.values.toSet.contains(actor) =>
       log.info(s"$actor is dead, removing from peers/connections/db")
       val remoteNodeId = peers.find(_._2 == actor).get._1
-      nodeParams.peersDb.delete(remoteNodeId)
+      nodeParams.peersDb.removePeer(remoteNodeId)
       context become main(peers - remoteNodeId, connections - remoteNodeId)
 
     case h@HandshakeCompleted(_, remoteNodeId) =>
-      val peer = createOrGetPeer(peers, remoteNodeId, None)
+      val peer = createOrGetPeer(peers, remoteNodeId, None, Set.empty)
       peer forward h
       context become main(peers + (remoteNodeId -> peer), connections)
 
@@ -75,11 +82,11 @@ class Switchboard(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, r
 
   }
 
-  def createOrGetPeer(peers: Map[PublicKey, ActorRef], remoteNodeId: PublicKey, address_opt: Option[InetSocketAddress]) = {
+  def createOrGetPeer(peers: Map[PublicKey, ActorRef], remoteNodeId: PublicKey, address_opt: Option[InetSocketAddress], offlineChannels: Set[HasCommitments]) = {
     peers.get(remoteNodeId) match {
       case Some(peer) => peer
       case None =>
-        val peer = context.actorOf(Peer.props(nodeParams, remoteNodeId, address_opt, watcher, router, relayer, defaultFinalScriptPubKey), name = s"peer-$remoteNodeId")
+        val peer = context.actorOf(Peer.props(nodeParams, remoteNodeId, address_opt, watcher, router, relayer, wallet, offlineChannels), name = s"peer-$remoteNodeId")
         context watch (peer)
         peer
     }
@@ -91,10 +98,10 @@ class Switchboard(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, r
 
 object Switchboard {
 
-  def props(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, relayer: ActorRef, defaultFinalScriptPubKey: BinaryData) = Props(new Switchboard(nodeParams, watcher, router, relayer, defaultFinalScriptPubKey))
+  def props(nodeParams: NodeParams, watcher: ActorRef, router: ActorRef, relayer: ActorRef, wallet: EclairWallet) = Props(new Switchboard(nodeParams, watcher, router, relayer, wallet))
 
   // @formatter:off
-  case class NewChannel(fundingSatoshis: Satoshi, pushMsat: MilliSatoshi)
+  case class NewChannel(fundingSatoshis: Satoshi, pushMsat: MilliSatoshi, channelFlags: Option[Byte])
   case class NewConnection(remoteNodeId: PublicKey, address: InetSocketAddress, newChannel_opt: Option[NewChannel])
   // @formatter:on
 

eclair-core/src/main/scala/fr/acinq/eclair/io/WriteAckSender.scala

@@ -0,0 +1,48 @@
+package fr.acinq.eclair.io
+
+import akka.actor.{Actor, ActorLogging, ActorRef, PoisonPill}
+import akka.io.Tcp
+import akka.util.ByteString
+
+
+/**
+  * This implements an ACK-based throttling mechanism
+  * See https://doc.akka.io/docs/akka/snapshot/scala/io-tcp.html#throttling-reads-and-writes
+  */
+class WriteAckSender(connection: ActorRef) extends Actor with ActorLogging {
+
+  // Note: this actor should be killed if connection dies
+
+  case object Ack extends Tcp.Event
+
+  override def receive = idle
+
+  def idle: Receive = {
+    case data: ByteString =>
+      connection ! Tcp.Write(data, Ack)
+      context become buffering(Vector.empty[ByteString])
+  }
+
+  def buffering(buffer: Vector[ByteString]): Receive = {
+    case _: ByteString if buffer.size > MAX_BUFFERED =>
+      log.warning(s"buffer overrun, closing connection")
+      connection ! PoisonPill
+    case data: ByteString =>
+      log.debug(s"buffering write $data")
+      context become buffering(buffer :+ data)
+    case Ack =>
+      buffer.headOption match {
+        case Some(data) =>
+          connection ! Tcp.Write(data, Ack)
+          context become buffering(buffer.drop(1))
+        case None =>
+          log.debug(s"got last ack, back to idle")
+          context become idle
+      }
+  }
+
+  override def unhandled(message: Any): Unit = log.warning(s"unhandled message $message")
+
+  val MAX_BUFFERED = 100000L
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/package.scala

@@ -56,12 +56,12 @@ package object eclair {
   }
 
   /**
-    * Converts fee-rate-per-kB to fee-rate-per-kw, *based on a standard commit tx*
+    * Converts feerate in satoshi-per-bytes to feerate in satoshi-per-kw
     *
-    * @param feeratePerKB
-    * @return
+    * @param feeratePerByte feerate in satoshi-per-bytes
+    * @return feerate in satoshi-per-kw
     */
-  def feerateKB2Kw(feeratePerKB: Long): Long = feeratePerKB / 2
+  def feerateByte2Kw(feeratePerByte: Long): Long = feeratePerByte * 1024 / 4
 
 
 }

eclair-core/src/main/scala/fr/acinq/eclair/payment/LocalPaymentHandler.scala

@@ -2,9 +2,9 @@ package fr.acinq.eclair.payment
 
 import akka.actor.{Actor, ActorLogging, Props, Status}
 import fr.acinq.bitcoin.{BinaryData, Crypto, MilliSatoshi}
-import fr.acinq.eclair.{Globals, NodeParams, randomBytes}
-import fr.acinq.eclair.channel.{CMD_FAIL_HTLC, CMD_FULFILL_HTLC, ExpiryTooSmall}
+import fr.acinq.eclair.channel.{CMD_FAIL_HTLC, CMD_FULFILL_HTLC}
 import fr.acinq.eclair.wire._
+import fr.acinq.eclair.{NodeParams, randomBytes}
 
 import scala.util.{Failure, Success, Try}
 
@@ -17,11 +17,11 @@ class LocalPaymentHandler(nodeParams: NodeParams) extends Actor with ActorLoggin
 
   def run(h2r: Map[BinaryData, (BinaryData, PaymentRequest)]): Receive = {
 
-    case ReceivePayment(amount) =>
+    case ReceivePayment(amount, desc) =>
       Try {
-        val r = randomBytes(32)
-        val h = Crypto.sha256(r)
-        (r, h, new PaymentRequest(nodeParams.privateKey.publicKey, amount, h))
+        val paymentPreimage = randomBytes(32)
+        val paymentHash = Crypto.sha256(paymentPreimage)
+        (paymentPreimage, paymentHash, PaymentRequest(nodeParams.chainHash, Some(amount), paymentHash, nodeParams.privateKey, desc))
       } match {
         case Success((r, h, pr)) =>
           log.debug(s"generated payment request=${PaymentRequest.write(pr)} from amount=$amount")
@@ -32,18 +32,21 @@ class LocalPaymentHandler(nodeParams: NodeParams) extends Actor with ActorLoggin
       }
 
     case htlc: UpdateAddHtlc =>
-     if (h2r.contains(htlc.paymentHash)) {
+      if (h2r.contains(htlc.paymentHash)) {
         val r = h2r(htlc.paymentHash)._1
         val pr = h2r(htlc.paymentHash)._2
         // The htlc amount must be equal or greater than the requested amount. A slight overpaying is permitted, however
         // it must not be greater than two times the requested amount.
         // see https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#failure-messages
-        if (pr.amount.amount <= htlc.amountMsat && htlc.amountMsat <= (2 * pr.amount.amount)) {
-          sender ! CMD_FULFILL_HTLC(htlc.id, r, commit = true)
-          context.system.eventStream.publish(PaymentReceived(MilliSatoshi(htlc.amountMsat), htlc.paymentHash))
-          context.become(run(h2r - htlc.paymentHash))
-        } else {
-          sender ! CMD_FAIL_HTLC(htlc.id, Right(IncorrectPaymentAmount), commit = true)
+        pr.amount match {
+          case Some(amount) if MilliSatoshi(htlc.amountMsat) < amount => sender ! CMD_FAIL_HTLC(htlc.id, Right(IncorrectPaymentAmount), commit = true)
+          case Some(amount) if MilliSatoshi(htlc.amountMsat) > amount * 2 => sender ! CMD_FAIL_HTLC(htlc.id, Right(IncorrectPaymentAmount), commit = true)
+          case _ =>
+            log.info(s"received payment for paymentHash=${htlc.paymentHash} amountMsat=${htlc.amountMsat}")
+            // amount is correct or was not specified in the payment request
+            sender ! CMD_FULFILL_HTLC(htlc.id, r, commit = true)
+            context.system.eventStream.publish(PaymentReceived(MilliSatoshi(htlc.amountMsat), htlc.paymentHash))
+            context.become(run(h2r - htlc.paymentHash))
         }
       } else {
         sender ! CMD_FAIL_HTLC(htlc.id, Right(UnknownPaymentHash), commit = true)

eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentEvents.scala

@@ -11,6 +11,6 @@ sealed trait PaymentEvent {
 
 case class PaymentSent(amount: MilliSatoshi, feesPaid: MilliSatoshi, paymentHash: BinaryData) extends PaymentEvent
 
-case class PaymentRelayed(amount: MilliSatoshi, feesEarned: MilliSatoshi, paymentHash: BinaryData) extends PaymentEvent
+case class PaymentRelayed(amountIn: MilliSatoshi, amountOut: MilliSatoshi, paymentHash: BinaryData) extends PaymentEvent
 
 case class PaymentReceived(amount: MilliSatoshi, paymentHash: BinaryData) extends PaymentEvent

eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentHop.scala

@@ -0,0 +1,46 @@
+package fr.acinq.eclair.payment
+
+import fr.acinq.bitcoin.Crypto.PublicKey
+import fr.acinq.eclair.payment.PaymentRequest.ExtraHop
+import fr.acinq.eclair.wire.ChannelUpdate
+
+
+object PaymentHop {
+  /**
+    *
+    * @param baseMsat     fixed fee
+    * @param proportional proportional fee
+    * @param msat         amount in millisatoshi
+    * @return the fee (in msat) that a node should be paid to forward an HTLC of 'amount' millisatoshis
+    */
+  def nodeFee(baseMsat: Long, proportional: Long, msat: Long): Long = baseMsat + (proportional * msat) / 1000000
+
+  /**
+    *
+    * @param reversePath sequence of Hops from recipient to a start of assisted path
+    * @param msat        an amount to send to a payment recipient
+    * @return a sequence of extra hops with a pre-calculated fee for a given msat amount
+    */
+  def buildExtra(reversePath: Seq[Hop], msat: Long): Seq[ExtraHop] = (List.empty[ExtraHop] /: reversePath) {
+    case (Nil, hop) => ExtraHop(hop.nodeId, hop.shortChannelId, hop.nextFee(msat), hop.cltvExpiryDelta) :: Nil
+    case (head :: rest, hop) => ExtraHop(hop.nodeId, hop.shortChannelId, hop.nextFee(msat + head.fee), hop.cltvExpiryDelta) :: head :: rest
+  }
+}
+
+trait PaymentHop {
+  def nextFee(msat: Long): Long
+
+  def shortChannelId: Long
+
+  def cltvExpiryDelta: Int
+
+  def nodeId: PublicKey
+}
+
+case class Hop(nodeId: PublicKey, nextNodeId: PublicKey, lastUpdate: ChannelUpdate) extends PaymentHop {
+  def nextFee(msat: Long): Long = PaymentHop.nodeFee(lastUpdate.feeBaseMsat, lastUpdate.feeProportionalMillionths, msat)
+
+  def cltvExpiryDelta: Int = lastUpdate.cltvExpiryDelta
+
+  def shortChannelId: Long = lastUpdate.shortChannelId
+}

eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentInitiator.scala

@@ -1,7 +1,6 @@
 package fr.acinq.eclair.payment
 
 import akka.actor.{Actor, ActorLogging, ActorRef, Props}
-import fr.acinq.bitcoin.BinaryData
 import fr.acinq.bitcoin.Crypto.PublicKey
 
 /**

eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentLifecycle.scala

@@ -1,7 +1,6 @@
 package fr.acinq.eclair.payment
 
-import akka.actor.Status.Failure
-import akka.actor.{ActorRef, FSM, LoggingFSM, Props}
+import akka.actor.{ActorRef, FSM, LoggingFSM, Props, Status}
 import fr.acinq.bitcoin.Crypto.PublicKey
 import fr.acinq.bitcoin.{BinaryData, MilliSatoshi}
 import fr.acinq.eclair._
@@ -13,17 +12,21 @@ import fr.acinq.eclair.wire._
 import scodec.Attempt
 
 // @formatter:off
-case class ReceivePayment(amountMsat: MilliSatoshi)
-case class SendPayment(amountMsat: Long, paymentHash: BinaryData, targetNodeId: PublicKey, maxAttempts: Int = 5)
+case class ReceivePayment(amountMsat: MilliSatoshi, description: String)
+case class SendPayment(amountMsat: Long, paymentHash: BinaryData, targetNodeId: PublicKey, minFinalCltvExpiry: Long = PaymentLifecycle.defaultMinFinalCltvExpiry, maxAttempts: Int = 5)
 
 sealed trait PaymentResult
-case class PaymentSucceeded(paymentPreimage: BinaryData) extends PaymentResult
-case class PaymentFailed(paymentHash: BinaryData, error: Option[ErrorPacket]) extends PaymentResult
+case class PaymentSucceeded(route: Seq[Hop], paymentPreimage: BinaryData) extends PaymentResult
+sealed trait PaymentFailure
+case class LocalFailure(t: Throwable) extends PaymentFailure
+case class RemoteFailure(route: Seq[Hop], e: ErrorPacket) extends PaymentFailure
+case class UnreadableRemoteFailure(route: Seq[Hop]) extends PaymentFailure
+case class PaymentFailed(paymentHash: BinaryData, failures: Seq[PaymentFailure]) extends PaymentResult
 
 sealed trait Data
 case object WaitingForRequest extends Data
-case class WaitingForRoute(sender: ActorRef, c: SendPayment, attempts: Int) extends Data
-case class WaitingForComplete(sender: ActorRef, c: SendPayment, cmd: CMD_ADD_HTLC, attempts: Int, sharedSecrets: Seq[(BinaryData, PublicKey)], ignoreNodes: Set[PublicKey], ignoreChannels: Set[Long], hops: Seq[Hop]) extends Data
+case class WaitingForRoute(sender: ActorRef, c: SendPayment, failures: Seq[PaymentFailure]) extends Data
+case class WaitingForComplete(sender: ActorRef, c: SendPayment, cmd: CMD_ADD_HTLC, failures: Seq[PaymentFailure], sharedSecrets: Seq[(BinaryData, PublicKey)], ignoreNodes: Set[PublicKey], ignoreChannels: Set[Long], hops: Seq[Hop]) extends Data
 
 sealed trait State
 case object WAITING_FOR_REQUEST extends State
@@ -44,20 +47,25 @@ class PaymentLifecycle(sourceNodeId: PublicKey, router: ActorRef, register: Acto
   when(WAITING_FOR_REQUEST) {
     case Event(c: SendPayment, WaitingForRequest) =>
       router ! RouteRequest(sourceNodeId, c.targetNodeId)
-      goto(WAITING_FOR_ROUTE) using WaitingForRoute(sender, c, attempts = 0)
+      goto(WAITING_FOR_ROUTE) using WaitingForRoute(sender, c, failures = Nil)
   }
 
   when(WAITING_FOR_ROUTE) {
-    case Event(RouteResponse(hops, ignoreNodes, ignoreChannels), WaitingForRoute(s, c, attempts)) =>
-      log.info(s"route found: attempt=$attempts/${c.maxAttempts} route=${hops.map(_.nextNodeId).mkString("->")}")
+    case Event(RouteResponse(hops, ignoreNodes, ignoreChannels), WaitingForRoute(s, c, failures)) =>
+      log.info(s"route found: attempt=${failures.size + 1}/${c.maxAttempts} route=${hops.map(_.nextNodeId).mkString("->")} channels=${hops.map(_.lastUpdate.shortChannelId.toHexString).mkString("->")}")
       val firstHop = hops.head
-      val finalExpiry = Globals.blockCount.get().toInt + defaultHtlcExpiry
+      val finalExpiry = Globals.blockCount.get().toInt + c.minFinalCltvExpiry.toInt
       val (cmd, sharedSecrets) = buildCommand(c.amountMsat, finalExpiry, c.paymentHash, hops)
-      register ! Register.ForwardShortId(firstHop.lastUpdate.shortChannelId, cmd)
-      goto(WAITING_FOR_PAYMENT_COMPLETE) using WaitingForComplete(s, c, cmd, attempts + 1, sharedSecrets, ignoreNodes, ignoreChannels, hops)
+      // TODO: HACK!!!! see Router.scala (we actually store the first node id in the sig)
+      if (firstHop.lastUpdate.signature.size == 32) {
+        register ! Register.Forward(firstHop.lastUpdate.signature, cmd)
+      } else {
+        register ! Register.ForwardShortId(firstHop.lastUpdate.shortChannelId, cmd)
+      }
+      goto(WAITING_FOR_PAYMENT_COMPLETE) using WaitingForComplete(s, c, cmd, failures, sharedSecrets, ignoreNodes, ignoreChannels, hops)
 
-    case Event(f@Failure(t), WaitingForRoute(s, c, _)) =>
-      s ! f
+    case Event(Status.Failure(t), WaitingForRoute(s, c, failures)) =>
+      s ! PaymentFailed(c.paymentHash, failures = failures :+ LocalFailure(t))
       stop(FSM.Normal)
   }
 
@@ -65,55 +73,81 @@ class PaymentLifecycle(sourceNodeId: PublicKey, router: ActorRef, register: Acto
     case Event("ok", _) => stay()
 
     case Event(fulfill: UpdateFulfillHtlc, w: WaitingForComplete) =>
-      w.sender ! PaymentSucceeded(fulfill.paymentPreimage)
+      w.sender ! PaymentSucceeded(w.hops, fulfill.paymentPreimage)
       context.system.eventStream.publish(PaymentSent(MilliSatoshi(w.c.amountMsat), MilliSatoshi(w.cmd.amountMsat - w.c.amountMsat), w.cmd.paymentHash))
       stop(FSM.Normal)
 
-    case Event(fail: UpdateFailHtlc, WaitingForComplete(s, c, _, attempts, sharedSecrets, ignoreNodes, ignoreChannels, hops)) =>
+    case Event(fail: UpdateFailHtlc, WaitingForComplete(s, c, _, failures, sharedSecrets, ignoreNodes, ignoreChannels, hops)) =>
       Sphinx.parseErrorPacket(fail.reason, sharedSecrets) match {
-        case e@Some(ErrorPacket(nodeId, failureMessage)) if nodeId == c.targetNodeId =>
-          // TODO: spec says: that MAY retry the payment in certain conditions, see https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#receiving-failure-codes
-          log.warning(s"received an error message from target nodeId=$nodeId, failing the payment (failure=$failureMessage)")
-          s ! PaymentFailed(c.paymentHash, error = e)
+        case None =>
+          log.warning(s"cannot parse returned error ${fail.reason}")
+          s ! PaymentFailed(c.paymentHash, failures = failures :+ UnreadableRemoteFailure(hops))
           stop(FSM.Normal)
-        case Some(ErrorPacket(nodeId, failureMessage: Node)) =>
-          // TODO: spec says: If the PERM bit is not set, the origin node SHOULD restore the channels as it sees new channel_updates.
+        case Some(e@ErrorPacket(nodeId, failureMessage)) if nodeId == c.targetNodeId =>
+          log.warning(s"received an error message from target nodeId=$nodeId, failing the payment (failure=$failureMessage)")
+          s ! PaymentFailed(c.paymentHash, failures = failures :+ RemoteFailure(hops, e))
+          stop(FSM.Normal)
+        case Some(e@ErrorPacket(nodeId, failureMessage)) if failures.size + 1 >= c.maxAttempts =>
+          log.info(s"received an error message from nodeId=$nodeId (failure=$failureMessage)")
+          log.warning(s"too many failed attempts, failing the payment")
+          s ! PaymentFailed(c.paymentHash, failures = failures :+ RemoteFailure(hops, e))
+          stop(FSM.Normal)
+        case Some(e@ErrorPacket(nodeId, failureMessage: Node)) =>
           log.info(s"received an error message from nodeId=$nodeId, trying to route around it (failure=$failureMessage)")
           // let's try to route around this node
           router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes + nodeId, ignoreChannels)
-          goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, attempts)
-        case e@Some(ErrorPacket(nodeId, failureMessage: Update)) =>
-          // TODO: spec says: if UPDATE is set, and the channel_update is valid *and more recent* than the channel_update used to send the payment
+          goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, failures :+ RemoteFailure(hops, e))
+        case Some(e@ErrorPacket(nodeId, failureMessage: Update)) =>
           log.info(s"received 'Update' type error message from nodeId=$nodeId, retrying payment (failure=$failureMessage)")
-          // TODO: should check that signature of the update is valid? Router will currently ignore it and send us an Error message
-          router ! failureMessage.update
-          // let's try again, router will have updated its state
-          router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes, ignoreChannels)
-          goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, attempts)
-        case e@Some(ErrorPacket(nodeId, failureMessage)) if attempts < c.maxAttempts =>
-          // TODO: If the PERM bit is not set, the origin node SHOULD restore the channel as it sees a new channel_update.
+          if (Announcements.checkSig(failureMessage.update, nodeId)) {
+            // note that we check the sig, but we don't make sure that this update was for the exact channel we required
+            // the reason is that we don't want to prevent relaying nodes to use another channel to the same N+1 node if they deem necessary
+            failureMessage match {
+              case _: TemporaryChannelFailure =>
+                // node indicates that its outgoing channel is experiencing a transient issue (eg. channel capacity reached, too many in-flight htlc)
+                hops.find(_.nodeId == nodeId).map(_.lastUpdate) match {
+                  case Some(u) if u.copy(signature = BinaryData.empty, timestamp = 0) == failureMessage.update.copy(signature = BinaryData.empty, timestamp = 0) =>
+                    // node returned the exact same update we used: in that case, let's temporarily exclude the channel from future routes, giving it time to recover
+                    val nextNodeId = hops.find(_.nodeId == nodeId).get.nextNodeId
+                    router ! ExcludeChannel(ChannelDesc(failureMessage.update.shortChannelId, nodeId, nextNodeId))
+                  case _ => // node returned a different update, maybe the payment will go through next time...
+                }
+              case _ => {}
+            }
+            // in any case, we forward the update to the router
+            router ! failureMessage.update
+            // let's try again, router will have updated its state
+            router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes, ignoreChannels)
+          } else {
+            // this node is fishy, it gave us a bad sig!! let's filter it out
+            log.warning(s"got bad signature from node=$nodeId update=${failureMessage.update}")
+            router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes + nodeId, ignoreChannels)
+          }
+          goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, failures :+ RemoteFailure(hops, e))
+        case Some(e@ErrorPacket(nodeId, failureMessage)) =>
           log.info(s"received an error message from nodeId=$nodeId, trying to use a different channel (failure=$failureMessage)")
           // let's try again without the channel outgoing from nodeId
-          router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes, ignoreChannels ++ hops.find(hop => hop.nodeId == nodeId.toBin).map(_.lastUpdate.shortChannelId).toSet)
-          goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, attempts)
-        case e@Some(ErrorPacket(nodeId, failureMessage)) =>
-          log.warning(s"too many failed attempts, failing the payment (attempts=$attempts)")
-          s ! PaymentFailed(c.paymentHash, error = e)
-          stop(FSM.Normal)
-        case None =>
-          log.warning(s"cannot parse returned error ${fail.reason}")
-          s ! PaymentFailed(c.paymentHash, error = None)
-          stop(FSM.Normal)
+          val faultyChannel = hops.find(_.nodeId == nodeId).map(_.lastUpdate.shortChannelId)
+          router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes, ignoreChannels ++ faultyChannel.toSet)
+          goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, failures :+ RemoteFailure(hops, e))
       }
 
-    case Event(failure@Failure(cause), WaitingForComplete(s, c, _, attempts, _, ignoreNodes, ignoreChannels, hops)) =>
-      if (attempts < c.maxAttempts) {
-        log.info(s"received an error message from local, trying to use a different channel (failure=${cause.getMessage})")
-        router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes, ignoreChannels + hops.head.lastUpdate.shortChannelId)
-        goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, attempts)
+    case Event(fail: UpdateFailMalformedHtlc, _) =>
+      log.info(s"first node in the route couldn't parse our htlc: fail=$fail")
+      // this is a corner case, that can only happen when the *first* node in the route cannot parse the onion
+      // (if this happens higher up in the route, the error would be wrapped in an UpdateFailHtlc and handled above)
+      // let's consider it a local error and treat is as such
+      self ! Status.Failure(new RuntimeException("first hop returned an UpdateFailMalformedHtlc message"))
+      stay
+
+    case Event(Status.Failure(t), WaitingForComplete(s, c, _, failures, _, ignoreNodes, ignoreChannels, hops)) =>
+      if (failures.size + 1 >= c.maxAttempts) {
+        s ! PaymentFailed(c.paymentHash, failures :+ LocalFailure(t))
+        stop(FSM.Normal)
       } else {
-        s ! failure
-        stop(FSM.Failure(failure.cause))
+        log.info(s"received an error message from local, trying to use a different channel (failure=${t.getMessage})")
+        router ! RouteRequest(sourceNodeId, c.targetNodeId, ignoreNodes, ignoreChannels + hops.head.lastUpdate.shortChannelId)
+        goto(WAITING_FOR_ROUTE) using WaitingForRoute(s, c, failures :+ LocalFailure(t))
       }
 
   }
@@ -125,48 +159,36 @@ object PaymentLifecycle {
 
   def props(sourceNodeId: PublicKey, router: ActorRef, register: ActorRef) = Props(classOf[PaymentLifecycle], sourceNodeId, router, register)
 
-  /**
-    *
-    * @param baseMsat     fixed fee
-    * @param proportional proportional fee
-    * @param msat         amount in millisatoshi
-    * @return the fee (in msat) that a node should be paid to forward an HTLC of 'amount' millisatoshis
-    */
-  def nodeFee(baseMsat: Long, proportional: Long, msat: Long): Long = baseMsat + (proportional * msat) / 1000000
-
-  def buildOnion(nodes: Seq[BinaryData], payloads: Seq[PerHopPayload], associatedData: BinaryData): Sphinx.PacketAndSecrets = {
+  def buildOnion(nodes: Seq[PublicKey], payloads: Seq[PerHopPayload], associatedData: BinaryData): Sphinx.PacketAndSecrets = {
     require(nodes.size == payloads.size)
     val sessionKey = randomKey
-    val pubkeys = nodes.map(PublicKey(_))
     val payloadsbin: Seq[BinaryData] = payloads
       .map(LightningMessageCodecs.perHopPayloadCodec.encode(_))
       .map {
         case Attempt.Successful(bitVector) => BinaryData(bitVector.toByteArray)
         case Attempt.Failure(cause) => throw new RuntimeException(s"serialization error: $cause")
       }
-    Sphinx.makePacket(sessionKey, pubkeys, payloadsbin, associatedData)
+    Sphinx.makePacket(sessionKey, nodes, payloadsbin, associatedData)
   }
 
   /**
     *
     * @param finalAmountMsat the final htlc amount in millisatoshis
-    * @param finalExpiry the final htlc expiry in number of blocks
-    * @param hops            the hops as computed by the router
+    * @param finalExpiry     the final htlc expiry in number of blocks
+    * @param hops            the hops as computed by the router + extra routes from payment request
     * @return a (firstAmountMsat, firstExpiry, payloads) tuple where:
     *         - firstAmountMsat is the amount for the first htlc in the route
     *         - firstExpiry is the cltv expiry for the first htlc in the route
     *         - a sequence of payloads that will be used to build the onion
     */
-  def buildPayloads(finalAmountMsat: Long, finalExpiry: Int, hops: Seq[Hop]): (Long, Int, Seq[PerHopPayload]) =
+  def buildPayloads(finalAmountMsat: Long, finalExpiry: Int, hops: Seq[PaymentHop]): (Long, Int, Seq[PerHopPayload]) =
     hops.reverse.foldLeft((finalAmountMsat, finalExpiry, PerHopPayload(0L, finalAmountMsat, finalExpiry) :: Nil)) {
       case ((msat, expiry, payloads), hop) =>
-        val feeMsat = nodeFee(hop.lastUpdate.feeBaseMsat, hop.lastUpdate.feeProportionalMillionths, msat)
-        val expiryDelta = hop.lastUpdate.cltvExpiryDelta
-        (msat + feeMsat, expiry + expiryDelta, PerHopPayload(hop.lastUpdate.shortChannelId, msat, expiry) +: payloads)
+        (msat + hop.nextFee(msat), expiry + hop.cltvExpiryDelta, PerHopPayload(hop.shortChannelId, msat, expiry) +: payloads)
     }
 
-  // TODO: set correct initial expiry
-  val defaultHtlcExpiry = 10
+  // this is defined in BOLT 11
+  val defaultMinFinalCltvExpiry = 9
 
   def buildCommand(finalAmountMsat: Long, finalExpiry: Int, paymentHash: BinaryData, hops: Seq[Hop]): (CMD_ADD_HTLC, Seq[(BinaryData, PublicKey)]) = {
     val (firstAmountMsat, firstExpiry, payloads) = buildPayloads(finalAmountMsat, finalExpiry, hops.drop(1))

eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentRequest.scala

@@ -1,43 +1,513 @@
 package fr.acinq.eclair.payment
 
-import fr.acinq.bitcoin.{BinaryData, MilliSatoshi}
-import grizzled.slf4j.Logging
+import java.math.BigInteger
+import java.nio.ByteOrder
 
-import scala.util.{Failure, Success, Try}
+import fr.acinq.bitcoin.Bech32.Int5
+import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
+import fr.acinq.bitcoin.{BinaryData, MilliSatoshi, _}
+import fr.acinq.eclair.crypto.BitStream
+import fr.acinq.eclair.crypto.BitStream.Bit
+import fr.acinq.eclair.payment.PaymentRequest.{Amount, RoutingInfoTag, Timestamp}
+
+import scala.annotation.tailrec
+import scala.util.Try
 
 /**
-  * Created by DPA on 11/04/2017.
+  * Lightning Payment Request
+  * see https://github.com/lightningnetwork/lightning-rfc/pull/183
+  *
+  * @param prefix    currency prefix; lnbc for bitcoin, lntb for bitcoin testnet
+  * @param amount    amount to pay (empty string means no amount is specified)
+  * @param timestamp request timestamp (UNIX format)
+  * @param nodeId    id of the node emitting the payment request
+  * @param tags      payment tags; must include a single PaymentHash tag
+  * @param signature request signature that will be checked against node id
   */
-object PaymentRequest extends Logging {
+case class PaymentRequest(prefix: String, amount: Option[MilliSatoshi], timestamp: Long, nodeId: PublicKey, tags: List[PaymentRequest.Tag], signature: BinaryData) {
 
-  // https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#adding-an-htlc-update_add_htlc
-  val maxAmountMsat = 4294967296L
+  amount.map(a => require(a > MilliSatoshi(0) && a <= PaymentRequest.maxAmount, s"amount is not valid"))
+  require(tags.collect { case _: PaymentRequest.PaymentHashTag => {} }.size == 1, "there must be exactly one payment hash tag")
+  require(tags.collect { case PaymentRequest.DescriptionTag(_) | PaymentRequest.DescriptionHashTag(_) => {} }.size == 1, "there must be exactly one description tag or one description hash tag")
 
-  def write(pr: PaymentRequest): String = {
-    s"${pr.nodeId.toString}:${pr.amount.amount}:${pr.paymentHash.toString}"
+  /**
+    *
+    * @return the payment hash
+    */
+  def paymentHash = tags.collectFirst { case p: PaymentRequest.PaymentHashTag => p }.get.hash
+
+  /**
+    *
+    * @return the description of the payment, or its hash
+    */
+  def description: Either[String, BinaryData] = tags.collectFirst {
+    case PaymentRequest.DescriptionTag(d) => Left(d)
+    case PaymentRequest.DescriptionHashTag(h) => Right(h)
+  }.get
+
+  /**
+    *
+    * @return the fallback address if any. It could be a script address, pubkey address, ..
+    */
+  def fallbackAddress(): Option[String] = tags.collectFirst {
+    case PaymentRequest.FallbackAddressTag(17, hash) if prefix == "lnbc" => Base58Check.encode(Base58.Prefix.PubkeyAddress, hash)
+    case PaymentRequest.FallbackAddressTag(18, hash) if prefix == "lnbc" => Base58Check.encode(Base58.Prefix.ScriptAddress, hash)
+    case PaymentRequest.FallbackAddressTag(17, hash) if prefix == "lntb" => Base58Check.encode(Base58.Prefix.PubkeyAddressTestnet, hash)
+    case PaymentRequest.FallbackAddressTag(18, hash) if prefix == "lntb" => Base58Check.encode(Base58.Prefix.ScriptAddressTestnet, hash)
+    case PaymentRequest.FallbackAddressTag(version, hash) if prefix == "lnbc" => Bech32.encodeWitnessAddress("bc", version, hash)
+    case PaymentRequest.FallbackAddressTag(version, hash) if prefix == "lntb" => Bech32.encodeWitnessAddress("tb", version, hash)
+  }
+
+  def routingInfo(): Seq[RoutingInfoTag] = tags.collect { case t: RoutingInfoTag => t }
+
+  def expiry: Option[Long] = tags.collectFirst {
+    case PaymentRequest.ExpiryTag(seconds) => seconds
+  }
+
+  def minFinalCltvExpiry: Option[Long] = tags.collectFirst {
+    case PaymentRequest.MinFinalCltvExpiryTag(expiry) => expiry
   }
 
   /**
-    * Parse a string and if the string is correctly formatted, returns a PaymentRequest object.
-    * Otherwise, throws and exception
     *
-    * @param pr payment request string, should look like <pre>node:amount:hash</pre>
-    * @return a PaymentRequest object
+    * @return a representation of this payment request, without its signature, as a bit stream. This is what will be signed.
     */
-  def read(pr: String): PaymentRequest = {
-    Try {
-      val Array(nodeId, amount, hash) = pr.split(":")
-      PaymentRequest(BinaryData(nodeId), MilliSatoshi(amount.toLong), BinaryData(hash))
-    } match {
-      case Success(s) => s
-      case Failure(t) =>
-        logger.debug(s"could not parse payment request: ${t.getMessage}")
-        throw t
-    }
+  def stream: BitStream = {
+    val stream = BitStream.empty
+    val int5s = Timestamp.encode(timestamp) ++ (tags.map(_.toInt5s).flatten)
+    val stream1 = int5s.foldLeft(stream)(PaymentRequest.write5)
+    stream1
+  }
+
+  /**
+    *
+    * @return the hash of this payment request
+    */
+  def hash: BinaryData = Crypto.sha256(s"${prefix}${Amount.encode(amount)}".getBytes("UTF-8") ++ stream.bytes)
+
+  /**
+    *
+    * @param priv private key
+    * @return a signed payment request
+    */
+  def sign(priv: PrivateKey): PaymentRequest = {
+    val (r, s) = Crypto.sign(hash, priv)
+    val (pub1, pub2) = Crypto.recoverPublicKey((r, s), hash)
+    val recid = if (nodeId == pub1) 0.toByte else 1.toByte
+    val signature = PaymentRequest.Signature.encode(r, s, recid)
+    this.copy(signature = signature)
   }
 }
 
-case class PaymentRequest(nodeId: BinaryData, amount: MilliSatoshi, paymentHash: BinaryData) {
-  require(amount.amount > 0 && amount.amount < PaymentRequest.maxAmountMsat,
-    f"amount is not valid: must be > 0 and < ${PaymentRequest.maxAmountMsat}%,d msat (~${PaymentRequest.maxAmountMsat / 1e11}%.3f BTC)")
+object PaymentRequest {
+
+  // https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#adding-an-htlc-update_add_htlc
+  val maxAmount = MilliSatoshi(4294967296L)
+
+  def apply(chainHash: BinaryData, amount: Option[MilliSatoshi], paymentHash: BinaryData, privateKey: PrivateKey,
+            description: String, fallbackAddress: Option[String] = None, expirySeconds: Option[Long] = None,
+            extraHops: Seq[Seq[ExtraHop]] = Nil, timestamp: Long = System.currentTimeMillis() / 1000L): PaymentRequest = {
+
+    val prefix = chainHash match {
+      case Block.RegtestGenesisBlock.hash => "lntb"
+      case Block.TestnetGenesisBlock.hash => "lntb"
+      case Block.LivenetGenesisBlock.hash => "lnbc"
+    }
+
+    PaymentRequest(
+      prefix = prefix,
+      amount = amount,
+      timestamp = timestamp,
+      nodeId = privateKey.publicKey,
+      tags = List(
+        Some(PaymentHashTag(paymentHash)),
+        Some(DescriptionTag(description)),
+        expirySeconds.map(ExpiryTag(_))
+      ).flatten ++ extraHops.map(RoutingInfoTag(_)),
+      signature = BinaryData.empty)
+      .sign(privateKey)
+  }
+
+  sealed trait Tag {
+    def toInt5s: Seq[Int5]
+  }
+
+  /**
+    * Payment Hash Tag
+    *
+    * @param hash payment hash
+    */
+  case class PaymentHashTag(hash: BinaryData) extends Tag {
+    override def toInt5s = {
+      val ints = Bech32.eight2five(hash)
+      Seq(Bech32.map('p'), (ints.length / 32).toByte, (ints.length % 32).toByte) ++ ints
+    }
+  }
+
+  /**
+    * Description Tag
+    *
+    * @param description a free-format string that will be included in the payment request
+    */
+  case class DescriptionTag(description: String) extends Tag {
+    override def toInt5s = {
+      val ints = Bech32.eight2five(description.getBytes("UTF-8"))
+      Seq(Bech32.map('d'), (ints.length / 32).toByte, (ints.length % 32).toByte) ++ ints
+    }
+  }
+
+  /**
+    * Hash Tag
+    *
+    * @param hash hash that will be included in the payment request, and can be checked against the hash of a
+    *             long description, an invoice, ...
+    */
+  case class DescriptionHashTag(hash: BinaryData) extends Tag {
+    override def toInt5s = {
+      val ints = Bech32.eight2five(hash)
+      Seq(Bech32.map('h'), (ints.length / 32).toByte, (ints.length % 32).toByte) ++ ints
+    }
+  }
+
+
+  /**
+    * Fallback Payment Tag that specifies a fallback payment address to be used if LN payment cannot be processed
+    *
+    * @param version address version; valid values are
+    *                - 17 (pubkey hash)
+    *                - 18 (script hash)
+    *                - 0 (segwit hash: p2wpkh (20 bytes) or p2wsh (32 bytes))
+    * @param hash    address hash
+    */
+  case class FallbackAddressTag(version: Byte, hash: BinaryData) extends Tag {
+    override def toInt5s = {
+      val ints = version +: Bech32.eight2five(hash)
+      Seq(Bech32.map('f'), (ints.length / 32).toByte, (ints.length % 32).toByte) ++ ints
+    }
+  }
+
+  object FallbackAddressTag {
+    /**
+      *
+      * @param address valid base58 or bech32 address
+      * @return a FallbackAddressTag instance
+      */
+    def apply(address: String): FallbackAddressTag = {
+      Try(fromBase58Address(address)).orElse(Try(fromBech32Address(address))).get
+    }
+
+    def fromBase58Address(address: String): FallbackAddressTag = {
+      val (prefix, hash) = Base58Check.decode(address)
+      prefix match {
+        case Base58.Prefix.PubkeyAddress => FallbackAddressTag(17, hash)
+        case Base58.Prefix.PubkeyAddressTestnet => FallbackAddressTag(17, hash)
+        case Base58.Prefix.ScriptAddress => FallbackAddressTag(18, hash)
+        case Base58.Prefix.ScriptAddressTestnet => FallbackAddressTag(18, hash)
+      }
+    }
+
+    def fromBech32Address(address: String): FallbackAddressTag = {
+      val (prefix, hash) = Bech32.decodeWitnessAddress(address)
+      FallbackAddressTag(prefix, hash)
+    }
+  }
+
+  /**
+    * Extra hop contained in RoutingInfoTag
+    *
+    * @param nodeId          node id
+    * @param shortChannelId  channel id
+    * @param fee             node fee
+    * @param cltvExpiryDelta node cltv expiry delta
+    */
+  case class ExtraHop(nodeId: PublicKey, shortChannelId: Long, fee: Long, cltvExpiryDelta: Int) extends PaymentHop {
+    def pack: Seq[Byte] = nodeId.toBin ++ Protocol.writeUInt64(shortChannelId, ByteOrder.BIG_ENDIAN) ++
+      Protocol.writeUInt64(fee, ByteOrder.BIG_ENDIAN) ++ Protocol.writeUInt16(cltvExpiryDelta, ByteOrder.BIG_ENDIAN)
+
+    // Fee is already pre-calculated for extra hops
+    def nextFee(msat: Long): Long = fee
+  }
+
+  /**
+    * Routing Info Tag
+    *
+    * @param path one or more entries containing extra routing information for a private route
+    */
+  case class RoutingInfoTag(path: Seq[ExtraHop]) extends Tag {
+    override def toInt5s = {
+      val ints = Bech32.eight2five(path.flatMap(_.pack))
+      Seq(Bech32.map('r'), (ints.length / 32).toByte, (ints.length % 32).toByte) ++ ints
+    }
+  }
+
+  object RoutingInfoTag {
+    def parse(data: Seq[Byte]) = {
+      val pubkey = data.slice(0, 33)
+      val shortChannelId = Protocol.uint64(data.slice(33, 33 + 8), ByteOrder.BIG_ENDIAN)
+      val fee = Protocol.uint64(data.slice(33 + 8, 33 + 8 + 8), ByteOrder.BIG_ENDIAN)
+      val cltv = Protocol.uint16(data.slice(33 + 8 + 8, chunkLength), ByteOrder.BIG_ENDIAN)
+      ExtraHop(PublicKey(pubkey), shortChannelId, fee, cltv)
+    }
+
+    def parseAll(data: Seq[Byte]): Seq[ExtraHop] =
+      data.grouped(chunkLength).map(parse).toList
+
+    val chunkLength: Int = 33 + 8 + 8 + 2
+  }
+
+  /**
+    * Expiry Date
+    *
+    * @param seconds expiry data for this payment request
+    */
+  case class ExpiryTag(seconds: Long) extends Tag {
+    override def toInt5s = {
+      val ints = writeUnsignedLong(seconds)
+      Bech32.map('x') +: (writeSize(ints.size) ++ ints)
+    }
+  }
+
+  /**
+    * Min final CLTV expiry
+    *
+    * @param blocks min final cltv expiry, in blocks
+    */
+  case class MinFinalCltvExpiryTag(blocks: Long) extends Tag {
+    override def toInt5s = {
+      val ints = writeUnsignedLong(blocks)
+      Bech32.map('c') +: (writeSize(ints.size) ++ ints)
+    }
+  }
+
+  object Amount {
+
+    /**
+      * @param amount
+      * @return the unit allowing for the shortest representation possible
+      */
+    def unit(amount: MilliSatoshi): Char = amount.amount * 10 match { // 1 milli-satoshis == 10 pico-bitcoin
+      case pico if pico % 1000 > 0 => 'p'
+      case pico if pico % 1000000 > 0 => 'n'
+      case pico if pico % 1000000000 > 0 => 'u'
+      case _ => 'm'
+    }
+
+    def decode(input: String): Option[MilliSatoshi] =
+      input match {
+        case "" => None
+        case a if a.last == 'p' => Some(MilliSatoshi(a.dropRight(1).toLong / 10L)) // 1 pico-bitcoin == 10 milli-satoshis
+        case a if a.last == 'n' => Some(MilliSatoshi(a.dropRight(1).toLong * 100L))
+        case a if a.last == 'u' => Some(MilliSatoshi(a.dropRight(1).toLong * 100000L))
+        case a if a.last == 'm' => Some(MilliSatoshi(a.dropRight(1).toLong * 100000000L))
+      }
+
+    def encode(amount: Option[MilliSatoshi]): String = {
+      amount match {
+        case None => ""
+        case Some(amt) if unit(amt) == 'p' => s"${amt.amount * 10L}p" // 1 pico-bitcoin == 10 milli-satoshis
+        case Some(amt) if unit(amt) == 'n' => s"${amt.amount / 100L}n"
+        case Some(amt) if unit(amt) == 'u' => s"${amt.amount / 100000L}u"
+        case Some(amt) if unit(amt) == 'm' => s"${amt.amount / 100000000L}m"
+      }
+    }
+  }
+
+  object Tag {
+    def parse(input: Seq[Byte]): Tag = {
+      val tag = input(0)
+      val len = input(1) * 32 + input(2)
+      tag match {
+        case p if p == Bech32.map('p') =>
+          val hash = Bech32.five2eight(input.drop(3).take(52))
+          PaymentHashTag(hash)
+        case d if d == Bech32.map('d') =>
+          val description = new String(Bech32.five2eight(input.drop(3).take(len)).toArray, "UTF-8")
+          DescriptionTag(description)
+        case h if h == Bech32.map('h') =>
+          val hash: BinaryData = Bech32.five2eight(input.drop(3).take(len))
+          DescriptionHashTag(hash)
+        case f if f == Bech32.map('f') =>
+          val version = input(3)
+          val prog = Bech32.five2eight(input.drop(4).take(len - 1))
+          version match {
+            case v if v >= 0 && v <= 16 =>
+              FallbackAddressTag(version, prog)
+            case 17 | 18 =>
+              FallbackAddressTag(version, prog)
+          }
+        case r if r == Bech32.map('r') =>
+          val data = Bech32.five2eight(input.drop(3).take(len))
+          val path = RoutingInfoTag.parseAll(data)
+          RoutingInfoTag(path)
+        case x if x == Bech32.map('x') =>
+          val expiry = readUnsignedLong(len, input.drop(3).take(len))
+          ExpiryTag(expiry)
+        case c if c == Bech32.map('c') =>
+          val expiry = readUnsignedLong(len, input.drop(3).take(len))
+          MinFinalCltvExpiryTag(expiry)
+      }
+    }
+  }
+
+  object Timestamp {
+    def decode(data: Seq[Int5]): Long = data.take(7).foldLeft(0L)((a, b) => a * 32 + b)
+
+    def encode(timestamp: Long, acc: Seq[Int5] = Nil): Seq[Int5] = if (acc.length == 7) acc else {
+      encode(timestamp / 32, (timestamp % 32).toByte +: acc)
+    }
+  }
+
+  object Signature {
+    /**
+      *
+      * @param signature 65-bytes signatyre: r (32 bytes) | s (32 bytes) | recid (1 bytes)
+      * @return a (r, s, recoveryId)
+      */
+    def decode(signature: BinaryData): (BigInteger, BigInteger, Byte) = {
+      require(signature.length == 65)
+      val r = new BigInteger(1, signature.take(32).toArray)
+      val s = new BigInteger(1, signature.drop(32).take(32).toArray)
+      val recid = signature.last
+      (r, s, recid)
+    }
+
+    /**
+      *
+      * @return a 65 bytes representation of (r, s, recid)
+      */
+    def encode(r: BigInteger, s: BigInteger, recid: Byte): BinaryData = {
+      Crypto.fixSize(r.toByteArray.dropWhile(_ == 0.toByte)) ++ Crypto.fixSize(s.toByteArray.dropWhile(_ == 0.toByte)) :+ recid
+    }
+  }
+
+  def toBits(value: Int5): Seq[Bit] = Seq((value & 16) != 0, (value & 8) != 0, (value & 4) != 0, (value & 2) != 0, (value & 1) != 0)
+
+  /**
+    * write a 5bits integer to a stream
+    *
+    * @param stream stream to write to
+    * @param value  a 5bits value
+    * @return an upated stream
+    */
+  def write5(stream: BitStream, value: Int5): BitStream = stream.writeBits(toBits(value))
+
+  /**
+    * read a 5bits value from a stream
+    *
+    * @param stream stream to read from
+    * @return a (stream, value) pair
+    */
+  def read5(stream: BitStream): (BitStream, Int5) = {
+    val (stream1, bits) = stream.readBits(5)
+    val value = (if (bits(0)) 1 << 4 else 0) + (if (bits(1)) 1 << 3 else 0) + (if (bits(2)) 1 << 2 else 0) + (if (bits(3)) 1 << 1 else 0) + (if (bits(4)) 1 << 0 else 0)
+    (stream1, (value & 0xff).toByte)
+  }
+
+  /**
+    * splits a bit stream into 5bits values
+    *
+    * @param stream
+    * @param acc
+    * @return a sequence of 5bits values
+    */
+  @tailrec
+  def toInt5s(stream: BitStream, acc: Seq[Int5] = Nil): Seq[Int5] = if (stream.bitCount == 0) acc else {
+    val (stream1, value) = read5(stream)
+    toInt5s(stream1, acc :+ value)
+  }
+
+  /**
+    * prepend an unsigned long value to a sequence of Int5s
+    *
+    * @param value input value
+    * @param acc   sequence of Int5 values
+    * @return an update sequence of Int5s
+    */
+  @tailrec
+  def writeUnsignedLong(value: Long, acc: Seq[Int5] = Nil): Seq[Int5] = {
+    require(value >= 0)
+    if (value == 0) acc
+    else writeUnsignedLong(value / 32, (value % 32).toByte +: acc)
+  }
+
+  /**
+    * convert a tag data size to a sequence of Int5s. It * must * fit on a sequence
+    * of 2 Int5 values
+    *
+    * @param size data size
+    * @return size as a sequence of exactly 2 Int5 values
+    */
+  def writeSize(size: Long): Seq[Int5] = {
+    val output = writeUnsignedLong(size)
+    // make sure that size is encoded on 2 int5 values
+    output.length match {
+      case 0 => Seq(0.toByte, 0.toByte)
+      case 1 => 0.toByte +: output
+      case 2 => output
+      case n => throw new IllegalArgumentException("tag data length field must be encoded on 2 5-bits integers")
+    }
+  }
+
+  /**
+    * reads an unsigned long value from a sequence of Int5s
+    *
+    * @param length length of the sequence
+    * @param ints   sequence of Int5s
+    * @return an unsigned long value
+    */
+  def readUnsignedLong(length: Int, ints: Seq[Int5]): Long = ints.take(length).foldLeft(0L) { case (acc, i) => acc * 32 + i }
+
+  /**
+    *
+    * @param input bech32-encoded payment request
+    * @return a payment request
+    */
+  def read(input: String): PaymentRequest = {
+    val (hrp, data) = Bech32.decode(input)
+    val stream = data.foldLeft(BitStream.empty)(write5)
+    require(stream.bitCount >= 65 * 8, "data is too short to contain a 65 bytes signature")
+    val (stream1, sig) = stream.popBytes(65)
+
+    val data0 = toInt5s(stream1)
+    val timestamp = Timestamp.decode(data0)
+    val data1 = data0.drop(7)
+
+    @tailrec
+    def loop(data: Seq[Int5], tags: Seq[Seq[Int5]] = Nil): Seq[Seq[Int5]] = if (data.isEmpty) tags else {
+      // 104 is the size of a signature
+      val len = 1 + 2 + 32 * data(1) + data(2)
+      loop(data.drop(len), tags :+ data.take(len))
+    }
+
+    val rawtags = loop(data1)
+    val tags = rawtags.map(Tag.parse)
+    val signature = sig.reverse
+    val r = new BigInteger(1, signature.take(32).toArray)
+    val s = new BigInteger(1, signature.drop(32).take(32).toArray)
+    val recid = signature.last
+    val message: BinaryData = hrp.getBytes ++ stream1.bytes
+    val (pub1, pub2) = Crypto.recoverPublicKey((r, s), Crypto.sha256(message))
+    val pub = if (recid % 2 != 0) pub2 else pub1
+    val prefix = hrp.take(4)
+    val amount_opt = Amount.decode(hrp.drop(4))
+    val pr = PaymentRequest(prefix, amount_opt, timestamp, pub, tags.toList, signature)
+    val validSig = Crypto.verifySignature(Crypto.sha256(message), (r, s), pub)
+    require(validSig, "invalid signature")
+    pr
+  }
+
+  /**
+    *
+    * @param pr payment request
+    * @return a bech32-encoded payment request
+    */
+  def write(pr: PaymentRequest): String = {
+    // currency unit is Satoshi, but we compute amounts in Millisatoshis
+    val hramount = Amount.encode(pr.amount)
+    val hrp = s"${pr.prefix}$hramount"
+    val stream = pr.stream.writeBytes(pr.signature)
+    val checksum = Bech32.checksum(hrp, toInt5s(stream))
+    hrp + "1" + new String((toInt5s(stream) ++ checksum).map(i => Bech32.pam(i)).toArray)
+  }
 }
+

eclair-core/src/main/scala/fr/acinq/eclair/payment/Relayer.scala

@@ -1,15 +1,12 @@
 package fr.acinq.eclair.payment
 
-import akka.actor.{Actor, ActorLogging, ActorRef, Props}
-import fr.acinq.bitcoin.Crypto.{PrivateKey, ripemd160, sha256}
-import fr.acinq.bitcoin.{BinaryData, Crypto, MilliSatoshi, ScriptWitness, Transaction}
-import fr.acinq.eclair.Globals
-import fr.acinq.eclair.blockchain.WatchEventSpent
+import akka.actor.{Actor, ActorLogging, ActorRef, Props, Status}
+import fr.acinq.bitcoin.{BinaryData, Crypto, MilliSatoshi}
 import fr.acinq.eclair.channel._
 import fr.acinq.eclair.crypto.Sphinx
-import fr.acinq.eclair.crypto.Sphinx.ParsedPacket
 import fr.acinq.eclair.router.Announcements
 import fr.acinq.eclair.wire._
+import fr.acinq.eclair.{Globals, NodeParams}
 import scodec.bits.BitVector
 import scodec.{Attempt, DecodeResult}
 
@@ -18,15 +15,16 @@ import scala.util.{Failure, Success, Try}
 // @formatter:off
 
 sealed trait Origin
-case class Local(sender: ActorRef) extends Origin
-case class Relayed(upstream: ActorRef, htlcIn: UpdateAddHtlc) extends Origin
+case class Local(sender: Option[ActorRef]) extends Origin // we don't persist reference to local actors
+case class Relayed(originChannelId: BinaryData, originHtlcId: Long, amountMsatIn: Long, amountMsatOut: Long) extends Origin
 
-case class AddHtlcSucceeded(add: UpdateAddHtlc, origin: Origin)
-case class AddHtlcFailed(add: CMD_ADD_HTLC, error: ChannelException)
 case class ForwardAdd(add: UpdateAddHtlc)
-case class ForwardFulfill(fulfill: UpdateFulfillHtlc)
-case class ForwardFail(fail: UpdateFailHtlc)
-case class ForwardFailMalformed(fail: UpdateFailMalformedHtlc)
+case class ForwardFulfill(fulfill: UpdateFulfillHtlc, to: Origin)
+case class ForwardLocalFail(error: Throwable, to: Origin) // happens when the failure happened in a local channel (and not in some downstream channel)
+case class ForwardFail(fail: UpdateFailHtlc, to: Origin)
+case class ForwardFailMalformed(fail: UpdateFailMalformedHtlc, to: Origin)
+
+case class AckFulfillCmd(channelId: BinaryData, htlcId: Long)
 
 // @formatter:on
 
@@ -34,43 +32,33 @@ case class ForwardFailMalformed(fail: UpdateFailMalformedHtlc)
 /**
   * Created by PM on 01/02/2017.
   */
-class Relayer(nodeSecret: PrivateKey, paymentHandler: ActorRef) extends Actor with ActorLogging {
+class Relayer(nodeParams: NodeParams, register: ActorRef, paymentHandler: ActorRef) extends Actor with ActorLogging {
+
+  import nodeParams.preimagesDb
 
   context.system.eventStream.subscribe(self, classOf[ChannelStateChanged])
-  context.system.eventStream.subscribe(self, classOf[ShortChannelIdAssigned])
 
-  override def receive: Receive = main(Map(), Map(), Map(), Map())
+  override def receive: Receive = main(Map())
 
-  def shortId2Channel(channels: Map[BinaryData, ActorRef], shortIds: Map[Long, BinaryData], shortId: Long): Option[ActorRef] = shortIds.get(shortId).flatMap(channels.get(_))
+  def main(channelUpdates: Map[Long, ChannelUpdate]): Receive = {
 
-  def main(channels: Map[BinaryData, ActorRef], shortIds: Map[Long, BinaryData], bindings: Map[UpdateAddHtlc, Origin], channelUpdates: Map[Long, ChannelUpdate]): Receive = {
-
-    case ChannelStateChanged(channel, _, _, _, NORMAL, d: DATA_NORMAL) =>
-      import d.commitments.channelId
-      log.info(s"adding channel $channelId to available channels")
-      context become main(channels + (channelId -> channel), shortIds, bindings, channelUpdates)
-
-    case ChannelStateChanged(_, _, _, _, NEGOTIATING, d: DATA_NEGOTIATING) =>
-      import d.commitments.channelId
-      log.info(s"removing channel $channelId from available channels")
-      // TODO: cleanup bindings
-      context become main(channels - channelId, shortIds, bindings, channelUpdates)
-
-    case ChannelStateChanged(_, _, _, _, CLOSING, d: DATA_CLOSING) =>
-      import d.commitments.channelId
-      log.info(s"removing channel $channelId from available channels")
-      // TODO: cleanup bindings
-      context become main(channels - channelId, shortIds, bindings, channelUpdates)
-
-    case ShortChannelIdAssigned(_, channelId, shortChannelId) =>
-      context become main(channels, shortIds + (shortChannelId -> channelId), bindings, channelUpdates)
+    case ChannelStateChanged(channel, _, _, _, NORMAL | SHUTDOWN | CLOSING, d: HasCommitments) =>
+      import d.channelId
+      preimagesDb.listPreimages(channelId) match {
+        case Nil => {}
+        case preimages =>
+          log.info(s"re-sending ${preimages.size} unacked fulfills to channel $channelId")
+          preimages.map(p => CMD_FULFILL_HTLC(p._2, p._3, commit = false)).foreach(channel ! _)
+          // better to sign once instead of after each fulfill
+          channel ! CMD_SIGN
+      }
 
     case channelUpdate: ChannelUpdate =>
       log.info(s"updating relay parameters with channelUpdate=$channelUpdate")
-      context become main(channels, shortIds, bindings, channelUpdates + (channelUpdate.shortChannelId -> channelUpdate))
+      context become main(channelUpdates + (channelUpdate.shortChannelId -> channelUpdate))
 
     case ForwardAdd(add) =>
-      Try(Sphinx.parsePacket(nodeSecret, add.paymentHash, add.onionRoutingPacket))
+      Try(Sphinx.parsePacket(nodeParams.privateKey, add.paymentHash, add.onionRoutingPacket))
         .map {
           case Sphinx.ParsedPacket(payload, nextPacket, sharedSecret) => (LightningMessageCodecs.perHopPayloadCodec.decode(BitVector(payload.data)), nextPacket, sharedSecret)
         } match {
@@ -81,34 +69,28 @@ class Relayer(nodeSecret: PrivateKey, paymentHandler: ActorRef) extends Actor wi
               sender ! CMD_FAIL_HTLC(add.id, Right(FinalIncorrectHtlcAmount(add.amountMsat)), commit = true)
             case PerHopPayload(_, _, finalOutgoingCltvValue) if finalOutgoingCltvValue != add.expiry =>
               sender ! CMD_FAIL_HTLC(add.id, Right(FinalIncorrectCltvExpiry(add.expiry)), commit = true)
-            case _  if add.expiry < Globals.blockCount.get() + 3 => // TODO: check hardcoded value
+            case _ if add.expiry < Globals.blockCount.get() + 3 => // TODO: check hardcoded value
               sender ! CMD_FAIL_HTLC(add.id, Right(FinalExpiryTooSoon), commit = true)
             case _ =>
               paymentHandler forward add
           }
         case Success((Attempt.Successful(DecodeResult(perHopPayload, _)), nextPacket, _)) =>
-          shortId2Channel(channels, shortIds, perHopPayload.channel_id) match {
-            case Some(downstream) =>
-              val channelUpdate_opt = channelUpdates.get(perHopPayload.channel_id)
-              channelUpdate_opt match {
-                case None =>
-                  // TODO: clarify what we're supposed to to in the specs
-                  sender ! CMD_FAIL_HTLC(add.id, Right(TemporaryNodeFailure), commit = true)
-                case Some(channelUpdate) if !Announcements.isEnabled(channelUpdate.flags) =>
-                  sender ! CMD_FAIL_HTLC(add.id, Right(ChannelDisabled(channelUpdate.flags, channelUpdate)), commit = true)
-                case Some(channelUpdate) if add.amountMsat < channelUpdate.htlcMinimumMsat =>
-                  sender ! CMD_FAIL_HTLC(add.id, Right(AmountBelowMinimum(add.amountMsat, channelUpdate)), commit = true)
-                case Some(channelUpdate) if add.expiry != perHopPayload.outgoing_cltv_value + channelUpdate.cltvExpiryDelta =>
-                  sender ! CMD_FAIL_HTLC(add.id, Right(IncorrectCltvExpiry(add.expiry, channelUpdate)), commit = true)
-                case Some(channelUpdate) if add.expiry < Globals.blockCount.get() + 3 => // TODO: hardcoded value
-                  sender ! CMD_FAIL_HTLC(add.id, Right(ExpiryTooSoon(channelUpdate)), commit = true)
-                case _ =>
-                  log.info(s"forwarding htlc #${add.id} to downstream=$downstream")
-                  downstream forward CMD_ADD_HTLC(perHopPayload.amt_to_forward, add.paymentHash, perHopPayload.outgoing_cltv_value, nextPacket.serialize, upstream_opt = Some(add), commit = true)
-              }
+          val channelUpdate_opt = channelUpdates.get(perHopPayload.channel_id)
+          channelUpdate_opt match {
             case None =>
-              log.warning(s"couldn't resolve downstream channel ${perHopPayload.channel_id}, failing htlc #${add.id}")
-              sender ! CMD_FAIL_HTLC(add.id, Right(UnknownNextPeer), commit = true)
+              // TODO: clarify what we're supposed to do in the specs
+              sender ! CMD_FAIL_HTLC(add.id, Right(TemporaryNodeFailure), commit = true)
+            case Some(channelUpdate) if !Announcements.isEnabled(channelUpdate.flags) =>
+              sender ! CMD_FAIL_HTLC(add.id, Right(ChannelDisabled(channelUpdate.flags, channelUpdate)), commit = true)
+            case Some(channelUpdate) if add.amountMsat < channelUpdate.htlcMinimumMsat =>
+              sender ! CMD_FAIL_HTLC(add.id, Right(AmountBelowMinimum(add.amountMsat, channelUpdate)), commit = true)
+            case Some(channelUpdate) if add.expiry != perHopPayload.outgoingCltvValue + channelUpdate.cltvExpiryDelta =>
+              sender ! CMD_FAIL_HTLC(add.id, Right(IncorrectCltvExpiry(add.expiry, channelUpdate)), commit = true)
+            case Some(channelUpdate) if add.expiry < Globals.blockCount.get() + 3 => // TODO: hardcoded value
+              sender ! CMD_FAIL_HTLC(add.id, Right(ExpiryTooSoon(channelUpdate)), commit = true)
+            case _ =>
+              log.info(s"forwarding htlc #${add.id} to shortChannelId=${perHopPayload.channel_id}")
+              register forward Register.ForwardShortId(perHopPayload.channel_id, CMD_ADD_HTLC(perHopPayload.amtToForward, add.paymentHash, perHopPayload.outgoingCltvValue, nextPacket.serialize, upstream_opt = Some(add), commit = true))
           }
         case Success((Attempt.Failure(cause), _, _)) =>
           log.error(s"couldn't parse payload: $cause")
@@ -119,108 +101,53 @@ class Relayer(nodeSecret: PrivateKey, paymentHandler: ActorRef) extends Actor wi
           sender ! CMD_FAIL_MALFORMED_HTLC(add.id, Crypto.sha256(add.onionRoutingPacket), failureCode = FailureMessageCodecs.BADONION, commit = true)
       }
 
-    case AddHtlcSucceeded(downstream, origin) =>
-      origin match {
-        case Local(_) => log.info(s"we are the origin of htlc ${downstream.channelId}/${downstream.id}")
-        case Relayed(_, upstream) => log.info(s"relayed htlc ${upstream.channelId}/${upstream.id} to ${downstream.channelId}/${downstream.id}")
-      }
-      context become main(channels, shortIds, bindings + (downstream -> origin), channelUpdates)
+    case Register.ForwardShortIdFailure(Register.ForwardShortId(shortChannelId, CMD_ADD_HTLC(_, _, _, _, Some(add), _))) =>
+      log.warning(s"couldn't resolve downstream channel $shortChannelId, failing htlc #${add.id}")
+      register ! Register.Forward(add.channelId, CMD_FAIL_HTLC(add.id, Right(UnknownNextPeer), commit = true))
 
-    case AddHtlcFailed(CMD_ADD_HTLC(_, _, _, _, Some(updateAddHtlc), _), error) if channels.contains(updateAddHtlc.channelId) =>
-      val upstream = channels(updateAddHtlc.channelId)
-      val channelUpdate_opt = for {
-        channelId <- channels.map(_.swap).get(sender)
-        shortId <- shortIds.map(_.swap).get(channelId)
-        update <- channelUpdates.get(shortId)
-      } yield update
-      // detail errors should have been catched earlier (when relayer picks the next channel), so here we just answer with generic error messages
-      channelUpdate_opt match {
-        case None =>
-          // TODO: clarify what we're supposed to to in the specs
-          upstream ! CMD_FAIL_HTLC(updateAddHtlc.id, Right(TemporaryNodeFailure), commit = true)
-        case Some(channelUpdate) =>
-          upstream ! CMD_FAIL_HTLC(updateAddHtlc.id, Right(TemporaryChannelFailure(channelUpdate)), commit = true)
-      }
+    case ForwardFulfill(fulfill, Local(Some(sender))) =>
+      sender ! fulfill
 
-    case ForwardFulfill(fulfill) =>
-      bindings.find(b => b._1.channelId == fulfill.channelId && b._1.id == fulfill.id) match {
-        case Some((htlcOut, Relayed(upstream, htlcIn))) =>
-          upstream ! CMD_FULFILL_HTLC(htlcIn.id, fulfill.paymentPreimage, commit = true)
-          context.system.eventStream.publish(PaymentRelayed(MilliSatoshi(htlcIn.amountMsat), MilliSatoshi(htlcIn.amountMsat - htlcOut.amountMsat), htlcIn.paymentHash))
-          context become main(channels, shortIds, bindings - htlcOut, channelUpdates)
-        case Some((htlcOut, Local(origin))) =>
-          log.info(s"we were the origin payer for htlc #${fulfill.id}")
-          origin ! fulfill
-          context become main(channels, shortIds, bindings - htlcOut, channelUpdates)
-        case None =>
-          log.warning(s"no origin found for htlc ${fulfill.channelId}/${fulfill.id}")
-      }
+    case ForwardFulfill(fulfill, Relayed(originChannelId, originHtlcId, amountMsatIn, amountMsatOut)) =>
+      val cmd = CMD_FULFILL_HTLC(originHtlcId, fulfill.paymentPreimage, commit = true)
+      register ! Register.Forward(originChannelId, cmd)
+      context.system.eventStream.publish(PaymentRelayed(MilliSatoshi(amountMsatIn), MilliSatoshi(amountMsatOut), Crypto.sha256(fulfill.paymentPreimage)))
+      // we also store the preimage in a db (note that this happens *after* forwarding the fulfill to the channel, so we don't add latency)
+      preimagesDb.addPreimage(originChannelId, originHtlcId, fulfill.paymentPreimage)
 
-    case ForwardFail(fail) =>
-      bindings.find(b => b._1.channelId == fail.channelId && b._1.id == fail.id) match {
-        case Some((htlcOut, Relayed(upstream, htlcIn))) =>
-          upstream ! CMD_FAIL_HTLC(htlcIn.id, Left(fail.reason), commit = true)
-          context become main(channels, shortIds, bindings - htlcOut, channelUpdates)
-        case Some((htlcOut, Local(origin))) =>
-          log.info(s"we were the origin payer for htlc #${fail.id}")
-          origin ! fail
-          context become main(channels, shortIds, bindings - htlcOut, channelUpdates)
-        case None =>
-          log.warning(s"no origin found for htlc ${fail.channelId}/${fail.id}")
-      }
+    case AckFulfillCmd(channelId, htlcId) =>
+      log.debug(s"fulfill acked for channelId=$channelId htlcId=$htlcId")
+      preimagesDb.removePreimage(channelId, htlcId)
 
-    case ForwardFailMalformed(fail) =>
-      bindings.find(b => b._1.channelId == fail.channelId && b._1.id == fail.id) match {
-        case Some((htlcOut, Relayed(upstream, htlcIn))) =>
-          upstream ! CMD_FAIL_MALFORMED_HTLC(htlcIn.id, fail.onionHash, fail.failureCode, commit = true)
-          context become main(channels, shortIds, bindings - htlcOut, channelUpdates)
-        case Some((htlcOut, Local(origin))) =>
-          log.info(s"we were the origin payer for htlc #${fail.id}")
-          origin ! fail
-          context become main(channels, shortIds, bindings - htlcOut, channelUpdates)
-        case None =>
-          log.warning(s"no origin found for htlc ${fail.channelId}/${fail.id}")
-      }
+    case ForwardLocalFail(error, Local(Some(sender))) =>
+      sender ! Status.Failure(error)
 
-    case w@WatchEventSpent(BITCOIN_HTLC_SPENT, tx) =>
-      // when a remote or local commitment tx containing outgoing htlcs is published on the network,
-      // we watch it in order to extract payment preimage if funds are pulled by the counterparty
-      // we can then use these preimages to fulfill origin htlcs
-      log.warning(s"processing BITCOIN_HTLC_SPENT with txid=${tx.txid} tx=${Transaction.write(tx)}")
-      require(tx.txIn.size == 1, s"htlc tx should only have 1 input")
-      val witness = tx.txIn(0).witness
-      val extracted = witness match {
-        case ScriptWitness(Seq(localSig, paymentPreimage, htlcOfferedScript)) if paymentPreimage.size == 32 =>
-          log.warning(s"extracted preimage=$paymentPreimage from tx=${Transaction.write(tx)} (claim-htlc-success)")
-          paymentPreimage
-        case ScriptWitness(Seq(BinaryData.empty, remoteSig, localSig, paymentPreimage, htlcReceivedScript)) if paymentPreimage.size == 32 =>
-          log.warning(s"extracted preimage=$paymentPreimage from tx=${Transaction.write(tx)} (htlc-success)")
-          paymentPreimage
-        case ScriptWitness(Seq(BinaryData.empty, remoteSig, localSig, BinaryData.empty, htlcOfferedScript)) =>
-          val paymentHash160 = BinaryData(htlcOfferedScript.slice(109, 109 + 20))
-          log.warning(s"extracted paymentHash160=$paymentHash160 from tx=${Transaction.write(tx)} (htlc-timeout)")
-          paymentHash160
-        case ScriptWitness(Seq(remoteSig, BinaryData.empty, htlcReceivedScript)) =>
-          val paymentHash160 = BinaryData(htlcReceivedScript.slice(69, 69 + 20))
-          log.warning(s"extracted paymentHash160=$paymentHash160 from tx=${Transaction.write(tx)} (claim-htlc-timeout)")
-          paymentHash160
+    case ForwardLocalFail(error, Relayed(originChannelId, originHtlcId, _, _)) =>
+      // TODO: clarify what we're supposed to do in the specs depending on the error
+      val failure = error match {
+        case HtlcTimedout(_) => PermanentChannelFailure
+        case _ => TemporaryNodeFailure
       }
-      val htlcsOut = bindings.collect {
-        case b@(htlcOut, Relayed(upstream, htlcIn)) if htlcIn.paymentHash == sha256(extracted) =>
-          log.warning(s"found a match between preimage=$extracted and origin htlc=$htlcIn")
-          upstream ! CMD_FULFILL_HTLC(htlcIn.id, extracted, commit = true)
-          htlcOut
-        case b@(htlcOut, Relayed(upstream, htlcIn)) if ripemd160(htlcIn.paymentHash) == extracted =>
-          log.warning(s"found a match between paymentHash160=$extracted and origin htlc=$htlcIn")
-          upstream ! CMD_FAIL_HTLC(htlcIn.id, Right(PermanentChannelFailure), commit = true)
-          htlcOut
-      }
-      context become main(channels, shortIds, bindings -- htlcsOut, channelUpdates)
+      val cmd = CMD_FAIL_HTLC(originHtlcId, Right(failure), commit = true)
+      register ! Register.Forward(originChannelId, cmd)
 
-    case 'channels => sender ! channels
+    case ForwardFail(fail, Local(Some(sender))) =>
+      sender ! fail
+
+    case ForwardFail(fail, Relayed(originChannelId, originHtlcId, _, _)) =>
+      val cmd = CMD_FAIL_HTLC(originHtlcId, Left(fail.reason), commit = true)
+      register ! Register.Forward(originChannelId, cmd)
+
+    case ForwardFailMalformed(fail, Local(Some(sender))) =>
+      sender ! fail
+
+    case ForwardFailMalformed(fail, Relayed(originChannelId, originHtlcId, _, _)) =>
+      val cmd = CMD_FAIL_MALFORMED_HTLC(originHtlcId, fail.onionHash, fail.failureCode, commit = true)
+      register ! Register.Forward(originChannelId, cmd)
   }
+
 }
 
 object Relayer {
-  def props(nodeSecret: PrivateKey, paymentHandler: ActorRef) = Props(classOf[Relayer], nodeSecret: PrivateKey, paymentHandler)
+  def props(nodeParams: NodeParams, register: ActorRef, paymentHandler: ActorRef) = Props(classOf[Relayer], nodeParams, register, paymentHandler)
 }

eclair-core/src/main/scala/fr/acinq/eclair/router/Announcements.scala

@@ -17,27 +17,27 @@ import scala.compat.Platform
   */
 object Announcements {
 
-  def channelAnnouncementWitnessEncode(shortChannelId: Long, nodeId1: BinaryData, nodeId2: BinaryData, bitcoinKey1: BinaryData, bitcoinKey2: BinaryData, features: BinaryData): BinaryData =
-    sha256(sha256(serializationResult(LightningMessageCodecs.channelAnnouncementWitnessCodec.encode(shortChannelId :: nodeId1 :: nodeId2 :: bitcoinKey1 :: bitcoinKey2 :: features :: HNil))))
+  def channelAnnouncementWitnessEncode(chainHash: BinaryData, shortChannelId: Long, nodeId1: PublicKey, nodeId2: PublicKey, bitcoinKey1: PublicKey, bitcoinKey2: PublicKey, features: BinaryData): BinaryData =
+    sha256(sha256(serializationResult(LightningMessageCodecs.channelAnnouncementWitnessCodec.encode(features :: chainHash :: shortChannelId :: nodeId1 :: nodeId2 :: bitcoinKey1 :: bitcoinKey2 :: HNil))))
 
-  def nodeAnnouncementWitnessEncode(timestamp: Long, nodeId: BinaryData, rgbColor: (Byte, Byte, Byte), alias: String, features: BinaryData, addresses: List[InetSocketAddress]): BinaryData =
-    sha256(sha256(serializationResult(LightningMessageCodecs.nodeAnnouncementWitnessCodec.encode(timestamp :: nodeId :: rgbColor :: alias :: features :: addresses :: HNil))))
+  def nodeAnnouncementWitnessEncode(timestamp: Long, nodeId: PublicKey, rgbColor: (Byte, Byte, Byte), alias: String, features: BinaryData, addresses: List[InetSocketAddress]): BinaryData =
+    sha256(sha256(serializationResult(LightningMessageCodecs.nodeAnnouncementWitnessCodec.encode(features :: timestamp :: nodeId :: rgbColor :: alias :: addresses :: HNil))))
 
-  def channelUpdateWitnessEncode(shortChannelId: Long, timestamp: Long, flags: BinaryData, cltvExpiryDelta: Int, htlcMinimumMsat: Long, feeBaseMsat: Long, feeProportionalMillionths: Long): BinaryData =
-    sha256(sha256(serializationResult(LightningMessageCodecs.channelUpdateWitnessCodec.encode(shortChannelId :: timestamp :: flags :: cltvExpiryDelta :: htlcMinimumMsat :: feeBaseMsat :: feeProportionalMillionths :: HNil))))
+  def channelUpdateWitnessEncode(chainHash: BinaryData, shortChannelId: Long, timestamp: Long, flags: BinaryData, cltvExpiryDelta: Int, htlcMinimumMsat: Long, feeBaseMsat: Long, feeProportionalMillionths: Long): BinaryData =
+    sha256(sha256(serializationResult(LightningMessageCodecs.channelUpdateWitnessCodec.encode(chainHash :: shortChannelId :: timestamp :: flags :: cltvExpiryDelta :: htlcMinimumMsat :: feeBaseMsat :: feeProportionalMillionths :: HNil))))
 
-  def signChannelAnnouncement(shortChannelId: Long, localNodeSecret: PrivateKey, remoteNodeId: PublicKey, localFundingPrivKey: PrivateKey, remoteFundingKey: PublicKey, features: BinaryData): (BinaryData, BinaryData) = {
+  def signChannelAnnouncement(chainHash: BinaryData, shortChannelId: Long, localNodeSecret: PrivateKey, remoteNodeId: PublicKey, localFundingPrivKey: PrivateKey, remoteFundingKey: PublicKey, features: BinaryData): (BinaryData, BinaryData) = {
     val witness = if (isNode1(localNodeSecret.publicKey.toBin, remoteNodeId.toBin)) {
-      channelAnnouncementWitnessEncode(shortChannelId, localNodeSecret.publicKey, remoteNodeId, localFundingPrivKey.publicKey, remoteFundingKey, features)
+      channelAnnouncementWitnessEncode(chainHash, shortChannelId, localNodeSecret.publicKey, remoteNodeId, localFundingPrivKey.publicKey, remoteFundingKey, features)
     } else {
-      channelAnnouncementWitnessEncode(shortChannelId, remoteNodeId, localNodeSecret.publicKey, remoteFundingKey, localFundingPrivKey.publicKey, features)
+      channelAnnouncementWitnessEncode(chainHash, shortChannelId, remoteNodeId, localNodeSecret.publicKey, remoteFundingKey, localFundingPrivKey.publicKey, features)
     }
     val nodeSig = Crypto.encodeSignature(Crypto.sign(witness, localNodeSecret)) :+ 1.toByte
     val bitcoinSig = Crypto.encodeSignature(Crypto.sign(witness, localFundingPrivKey)) :+ 1.toByte
     (nodeSig, bitcoinSig)
   }
 
-  def makeChannelAnnouncement(shortChannelId: Long, localNodeId: PublicKey, remoteNodeId: PublicKey, localFundingKey: PublicKey, remoteFundingKey: PublicKey, localNodeSignature: BinaryData, remoteNodeSignature: BinaryData, localBitcoinSignature: BinaryData, remoteBitcoinSignature: BinaryData): ChannelAnnouncement = {
+  def makeChannelAnnouncement(chainHash: BinaryData, shortChannelId: Long, localNodeId: PublicKey, remoteNodeId: PublicKey, localFundingKey: PublicKey, remoteFundingKey: PublicKey, localNodeSignature: BinaryData, remoteNodeSignature: BinaryData, localBitcoinSignature: BinaryData, remoteBitcoinSignature: BinaryData): ChannelAnnouncement = {
     val (nodeId1, nodeId2, bitcoinKey1, bitcoinKey2, nodeSignature1, nodeSignature2, bitcoinSignature1, bitcoinSignature2) =
       if (isNode1(localNodeId.toBin, remoteNodeId.toBin)) {
         (localNodeId, remoteNodeId, localFundingKey, remoteFundingKey, localNodeSignature, remoteNodeSignature, localBitcoinSignature, remoteBitcoinSignature)
@@ -54,7 +54,8 @@ object Announcements {
       nodeId2 = nodeId2,
       bitcoinKey1 = bitcoinKey1,
       bitcoinKey2 = bitcoinKey2,
-      features = BinaryData("")
+      features = BinaryData(""),
+      chainHash = chainHash
     )
   }
 
@@ -78,6 +79,7 @@ object Announcements {
     * The creating node MUST set node-id-1 and node-id-2 to the public keys of the
     * two nodes who are operating the channel, such that node-id-1 is the numerically-lesser
     * of the two DER encoded keys sorted in ascending numerical order,
+    *
     * @return true if localNodeId is node1
     */
   def isNode1(localNodeId: BinaryData, remoteNodeId: BinaryData) = LexicographicalOrdering.isLessThan(localNodeId, remoteNodeId)
@@ -86,6 +88,7 @@ object Announcements {
     * BOLT 7:
     * The creating node [...] MUST set the direction bit of flags to 0 if
     * the creating node is node-id-1 in that message, otherwise 1.
+    *
     * @return true if the node who sent these flags is node1
     */
   def isNode1(flags: BinaryData) = !BitVector(flags.data).reverse.get(0)
@@ -93,19 +96,21 @@ object Announcements {
   /**
     * A node MAY create and send a channel_update with the disable bit set to
     * signal the temporary unavailability of a channel
+    *
     * @return
     */
   def isEnabled(flags: BinaryData) = !BitVector(flags.data).reverse.get(1)
 
   def makeFlags(isNode1: Boolean, enable: Boolean): BinaryData = BitVector.bits(!enable :: !isNode1 :: Nil).padLeft(16).toByteArray
 
-  def makeChannelUpdate(nodeSecret: PrivateKey, remoteNodeId: PublicKey, shortChannelId: Long, cltvExpiryDelta: Int, htlcMinimumMsat: Long, feeBaseMsat: Long, feeProportionalMillionths: Long, enable: Boolean = true, timestamp: Long = Platform.currentTime / 1000): ChannelUpdate = {
+  def makeChannelUpdate(chainHash: BinaryData, nodeSecret: PrivateKey, remoteNodeId: PublicKey, shortChannelId: Long, cltvExpiryDelta: Int, htlcMinimumMsat: Long, feeBaseMsat: Long, feeProportionalMillionths: Long, enable: Boolean = true, timestamp: Long = Platform.currentTime / 1000): ChannelUpdate = {
     val flags = makeFlags(isNode1 = isNode1(nodeSecret.publicKey.toBin, remoteNodeId.toBin), enable = enable)
     require(flags.size == 2, "flags must be a 2-bytes field")
-    val witness = channelUpdateWitnessEncode(shortChannelId, timestamp, flags, cltvExpiryDelta, htlcMinimumMsat, feeBaseMsat, feeProportionalMillionths)
+    val witness = channelUpdateWitnessEncode(chainHash, shortChannelId, timestamp, flags, cltvExpiryDelta, htlcMinimumMsat, feeBaseMsat, feeProportionalMillionths)
     val sig = Crypto.encodeSignature(Crypto.sign(witness, nodeSecret)) :+ 1.toByte
     ChannelUpdate(
       signature = sig,
+      chainHash = chainHash,
       shortChannelId = shortChannelId,
       timestamp = timestamp,
       flags = flags,
@@ -117,21 +122,21 @@ object Announcements {
   }
 
   def checkSigs(ann: ChannelAnnouncement): Boolean = {
-    val witness = channelAnnouncementWitnessEncode(ann.shortChannelId, ann.nodeId1, ann.nodeId2, ann.bitcoinKey1, ann.bitcoinKey2, ann.features)
-    verifySignature(witness, ann.nodeSignature1, PublicKey(ann.nodeId1)) &&
-      verifySignature(witness, ann.nodeSignature2, PublicKey(ann.nodeId2)) &&
-      verifySignature(witness, ann.bitcoinSignature1, PublicKey(ann.bitcoinKey1)) &&
-      verifySignature(witness, ann.bitcoinSignature2, PublicKey(ann.bitcoinKey2))
+    val witness = channelAnnouncementWitnessEncode(ann.chainHash, ann.shortChannelId, ann.nodeId1, ann.nodeId2, ann.bitcoinKey1, ann.bitcoinKey2, ann.features)
+    verifySignature(witness, ann.nodeSignature1, ann.nodeId1) &&
+      verifySignature(witness, ann.nodeSignature2, ann.nodeId2) &&
+      verifySignature(witness, ann.bitcoinSignature1, ann.bitcoinKey1) &&
+      verifySignature(witness, ann.bitcoinSignature2, ann.bitcoinKey2)
   }
 
   def checkSig(ann: NodeAnnouncement): Boolean = {
     val witness = nodeAnnouncementWitnessEncode(ann.timestamp, ann.nodeId, ann.rgbColor, ann.alias, ann.features, ann.addresses)
-    verifySignature(witness, ann.signature, PublicKey(ann.nodeId))
+    verifySignature(witness, ann.signature, ann.nodeId)
   }
 
-  def checkSig(ann: ChannelUpdate, nodeId: BinaryData): Boolean = {
-    val witness = channelUpdateWitnessEncode(ann.shortChannelId, ann.timestamp, ann.flags, ann.cltvExpiryDelta, ann.htlcMinimumMsat, ann.feeBaseMsat, ann.feeProportionalMillionths)
-    verifySignature(witness, ann.signature, PublicKey(nodeId))
+  def checkSig(ann: ChannelUpdate, nodeId: PublicKey): Boolean = {
+    val witness = channelUpdateWitnessEncode(ann.chainHash, ann.shortChannelId, ann.timestamp, ann.flags, ann.cltvExpiryDelta, ann.htlcMinimumMsat, ann.feeBaseMsat, ann.feeProportionalMillionths)
+    verifySignature(witness, ann.signature, nodeId)
   }
 
 }

eclair-core/src/main/scala/fr/acinq/eclair/router/NetworkEvents.scala

@@ -1,6 +1,7 @@
 package fr.acinq.eclair.router
 
-import fr.acinq.bitcoin.{BinaryData, Satoshi}
+import fr.acinq.bitcoin.Crypto.PublicKey
+import fr.acinq.bitcoin.Satoshi
 import fr.acinq.eclair.wire.{ChannelAnnouncement, ChannelUpdate, NodeAnnouncement}
 
 /**
@@ -12,7 +13,7 @@ case class NodeDiscovered(ann: NodeAnnouncement) extends NetworkEvent
 
 case class NodeUpdated(ann: NodeAnnouncement) extends NetworkEvent
 
-case class NodeLost(nodeId: BinaryData) extends NetworkEvent
+case class NodeLost(nodeId: PublicKey) extends NetworkEvent
 
 case class ChannelDiscovered(ann: ChannelAnnouncement, capacity: Satoshi) extends NetworkEvent
 

eclair-core/src/main/scala/fr/acinq/eclair/router/Router.scala

@@ -11,38 +11,47 @@ import fr.acinq.eclair._
 import fr.acinq.eclair.blockchain._
 import fr.acinq.eclair.channel._
 import fr.acinq.eclair.io.Peer
+import fr.acinq.eclair.payment.Hop
 import fr.acinq.eclair.transactions.Scripts
 import fr.acinq.eclair.wire._
 import org.jgrapht.alg.shortestpath.DijkstraShortestPath
 import org.jgrapht.ext._
 import org.jgrapht.graph.{DefaultDirectedGraph, DefaultEdge, SimpleGraph}
-import scodec.bits.BitVector
 
 import scala.collection.JavaConversions._
+import scala.compat.Platform
+import scala.concurrent.duration._
 import scala.concurrent.{ExecutionContext, Future}
-import scala.util.{Success, Try}
+import scala.util.{Random, Success, Try}
 
 // @formatter:off
 
-case class ChannelDesc(id: Long, a: BinaryData, b: BinaryData)
-case class Hop(nodeId: BinaryData, nextNodeId: BinaryData, lastUpdate: ChannelUpdate)
-case class RouteRequest(source: BinaryData, target: BinaryData, ignoreNodes: Set[PublicKey] = Set.empty, ignoreChannels: Set[Long] = Set.empty)
+case class ChannelDesc(id: Long, a: PublicKey, b: PublicKey)
+case class RouteRequest(source: PublicKey, target: PublicKey, ignoreNodes: Set[PublicKey] = Set.empty, ignoreChannels: Set[Long] = Set.empty)
 case class RouteResponse(hops: Seq[Hop], ignoreNodes: Set[PublicKey], ignoreChannels: Set[Long]) { require(hops.size > 0, "route cannot be empty") }
+case class ExcludeChannel(desc: ChannelDesc) // this is used when we get a TemporaryChannelFailure, to give time for the channel to recover (note that exclusions are directed)
+case class LiftChannelExclusion(desc: ChannelDesc)
 case class SendRoutingState(to: ActorRef)
 case class Rebroadcast(ann: Seq[RoutingMessage], origins: Map[RoutingMessage, ActorRef])
 
-case class Data(nodes: Map[BinaryData, NodeAnnouncement],
+case class Data(nodes: Map[PublicKey, NodeAnnouncement],
                   channels: Map[Long, ChannelAnnouncement],
                   updates: Map[ChannelDesc, ChannelUpdate],
                   rebroadcast: Seq[RoutingMessage],
                   stash: Seq[RoutingMessage],
                   awaiting: Seq[ChannelAnnouncement],
-                  origins: Map[RoutingMessage, ActorRef])
+                  origins: Map[RoutingMessage, ActorRef],
+                  localChannels: Map[BinaryData, PublicKey],
+                  excludedChannels: Set[ChannelDesc]) // those channels are temporarily excluded from route calculation, because their node returned a TemporaryChannelFailure
 
 sealed trait State
 case object NORMAL extends State
 case object WAITING_FOR_VALIDATION extends State
 
+case object TickBroadcast
+case object TickValidate
+case object TickPruneStaleChannels
+
 // @formatter:on
 
 /**
@@ -55,14 +64,31 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
 
   import ExecutionContext.Implicits.global
 
-  setTimer("broadcast", 'tick_broadcast, nodeParams.routerBroadcastInterval, repeat = true)
-  setTimer("validate", 'tick_validate, nodeParams.routerValidateInterval, repeat = true)
+  context.system.eventStream.subscribe(self, classOf[ChannelStateChanged])
 
-  startWith(NORMAL, Data(Map.empty, Map.empty, Map.empty, Nil, Nil, Nil, Map.empty))
+  setTimer(TickBroadcast.toString, TickBroadcast, nodeParams.routerBroadcastInterval, repeat = true)
+  setTimer(TickValidate.toString, TickValidate, nodeParams.routerValidateInterval, repeat = true)
+  setTimer(TickPruneStaleChannels.toString, TickPruneStaleChannels, 1 day, repeat = true)
+
+  val db = nodeParams.networkDb
+
+  // Note: We go through the whole validation process instead of directly loading into memory, because the channels
+  // could have been closed while we were shutdown, and if someone connects to us right after startup we don't want to
+  // advertise invalid channels. We could optimize this (at least not fetch txes from the blockchain, and not check sigs)
+  log.info(s"loading network announcements from db...")
+  db.listChannels().map(self ! _)
+  db.listNodes().map(self ! _)
+  db.listChannelUpdates().map(self ! _)
+  if (db.listChannels().size > 0) {
+    val nodeAnn = Announcements.makeNodeAnnouncement(nodeParams.privateKey, nodeParams.alias, nodeParams.color, nodeParams.publicAddresses, Platform.currentTime / 1000)
+    self ! nodeAnn
+  }
+  log.info(s"starting state machine")
+
+  startWith(NORMAL, Data(Map.empty, Map.empty, Map.empty, Nil, Nil, Nil, Map.empty, Map.empty, Set.empty))
 
   when(NORMAL) {
-
-    case Event('tick_validate, d) =>
+    case Event(TickValidate, d) =>
       require(d.awaiting.size == 0)
       var i = 0
       // we extract a batch of channel announcements from the stash
@@ -80,7 +106,6 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
   }
 
   when(WAITING_FOR_VALIDATION) {
-
     case Event(ParallelGetResponse(results), d) =>
       val validated = results.map {
         case IndividualResult(c, Some(tx), true) =>
@@ -95,16 +120,18 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
             log.error(s"invalid script for shortChannelId=${c.shortChannelId} txid=${tx.txid} ann=$c")
             None
           } else {
-            watcher ! WatchSpentBasic(self, tx.txid, outputIndex, BITCOIN_FUNDING_OTHER_CHANNEL_SPENT(c.shortChannelId))
+            watcher ! WatchSpentBasic(self, tx, outputIndex, BITCOIN_FUNDING_EXTERNAL_CHANNEL_SPENT(c.shortChannelId))
             // TODO: check feature bit set
             log.debug(s"added channel channelId=${c.shortChannelId}")
             context.system.eventStream.publish(ChannelDiscovered(c, tx.txOut(outputIndex).amount))
-            nodeParams.announcementsDb.put(channelKey(c.shortChannelId), c)
+            db.addChannel(c)
             Some(c)
           }
         case IndividualResult(c, Some(tx), false) =>
           // TODO: vulnerability if they flood us with spent funding tx?
           log.warning(s"ignoring shortChannelId=${c.shortChannelId} tx=${tx.txid} (funding tx not found in utxo)")
+          // there may be a record if we have just restarted
+          db.removeChannel(c.shortChannelId)
           None
         case IndividualResult(c, None, _) =>
           // TODO: blacklist?
@@ -113,7 +140,7 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
       }.flatten
       // we reprocess node and channel-update announcements that may have been validated
       val (resend, stash1) = d.stash.partition {
-        case n: NodeAnnouncement => results.exists(r => isRelatedTo(r.c, n))
+        case n: NodeAnnouncement => results.exists(r => isRelatedTo(r.c, n.nodeId))
         case u: ChannelUpdate => results.exists(r => r.c.shortChannelId == u.shortChannelId)
         case _ => false
       }
@@ -122,51 +149,59 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
   }
 
   whenUnhandled {
+    case Event(ChannelStateChanged(_, _, _, _, channel.NORMAL, d: DATA_NORMAL), d1) =>
+      stay using d1.copy(localChannels = d1.localChannels + (d.commitments.channelId -> d.commitments.remoteParams.nodeId))
 
-    case Event(SendRoutingState(remote), Data(nodes, channels, updates, _, _, _, _)) =>
+    case Event(ChannelStateChanged(_, _, _, channel.NORMAL, _, d: DATA_NEGOTIATING), d1) =>
+      stay using d1.copy(localChannels = d1.localChannels - d.commitments.channelId)
+
+    case Event(_: ChannelStateChanged, _) => stay
+
+    case Event(SendRoutingState(remote), Data(nodes, channels, updates, _, _, _, _, _, _)) =>
       log.debug(s"info sending all announcements to $remote: channels=${channels.size} nodes=${nodes.size} updates=${updates.size}")
-      channels.values.foreach(remote ! _)
-      nodes.values.foreach(remote ! _)
-      updates.values.foreach(remote ! _)
+      // we group and add delays to leave room for channel messages
+      context.actorOf(ThrottleForwarder.props(remote, channels.values ++ nodes.values ++ updates.values, 100, 100 millis))
       stay
 
     case Event(c: ChannelAnnouncement, d) =>
       log.debug(s"received channel announcement for shortChannelId=${c.shortChannelId} nodeId1=${c.nodeId1} nodeId2=${c.nodeId2}")
-      if (!Announcements.checkSigs(c)) {
+      if (d.channels.containsKey(c.shortChannelId) || d.awaiting.exists(_.shortChannelId == c.shortChannelId) || d.stash.contains(c)) {
+        log.debug(s"ignoring $c (duplicate)")
+        stay
+      } else if (!Announcements.checkSigs(c)) {
         log.error(s"bad signature for announcement $c")
         sender ! Error(Peer.CHANNELID_ZERO, "bad announcement sig!!!".getBytes())
         stay
-      } else if (d.channels.containsKey(c.shortChannelId) || d.awaiting.exists(_.shortChannelId == c.shortChannelId) || d.stash.contains(c)) {
-        log.debug(s"ignoring $c (duplicate)")
-        stay
       } else {
         log.debug(s"stashing $c")
         stay using d.copy(stash = d.stash :+ c, origins = d.origins + (c -> sender))
       }
 
     case Event(n: NodeAnnouncement, d: Data) =>
-      if (!Announcements.checkSig(n)) {
+      if (d.nodes.containsKey(n.nodeId) && d.nodes(n.nodeId).timestamp >= n.timestamp) {
+        log.debug(s"ignoring announcement $n (old timestamp or duplicate)")
+        stay
+      } else if (!Announcements.checkSig(n)) {
         log.error(s"bad signature for announcement $n")
         sender ! Error(Peer.CHANNELID_ZERO, "bad announcement sig!!!".getBytes())
         stay
-      } else if (d.nodes.containsKey(n.nodeId) && d.nodes(n.nodeId).timestamp >= n.timestamp) {
-        log.debug(s"ignoring announcement $n (old timestamp or duplicate)")
-        stay
       } else if (d.nodes.containsKey(n.nodeId)) {
         log.debug(s"updated node nodeId=${n.nodeId}")
         context.system.eventStream.publish(NodeUpdated(n))
-        nodeParams.announcementsDb.put(nodeKey(n.nodeId), n)
+        db.updateNode(n)
         stay using d.copy(nodes = d.nodes + (n.nodeId -> n), rebroadcast = d.rebroadcast :+ n, origins = d.origins + (n -> sender))
-      } else if (d.channels.values.exists(c => isRelatedTo(c, n))) {
+      } else if (d.channels.values.exists(c => isRelatedTo(c, n.nodeId))) {
         log.debug(s"added node nodeId=${n.nodeId}")
         context.system.eventStream.publish(NodeDiscovered(n))
-        nodeParams.announcementsDb.put(nodeKey(n.nodeId), n)
+        db.addNode(n)
         stay using d.copy(nodes = d.nodes + (n.nodeId -> n), rebroadcast = d.rebroadcast :+ n, origins = d.origins + (n -> sender))
-      } else if (d.awaiting.exists(c => isRelatedTo(c, n)) || d.stash.collectFirst { case c: ChannelAnnouncement if isRelatedTo(c, n) => c }.isDefined) {
+      } else if (d.awaiting.exists(c => isRelatedTo(c, n.nodeId)) || d.stash.collectFirst { case c: ChannelAnnouncement if isRelatedTo(c, n.nodeId) => c }.isDefined) {
         log.debug(s"stashing $n")
         stay using d.copy(stash = d.stash :+ n, origins = d.origins + (n -> sender))
       } else {
         log.warning(s"ignoring $n (no related channel found)")
+        // there may be a record if we have just restarted
+        db.removeNode(n.nodeId)
         stay
       }
 
@@ -174,18 +209,23 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
       if (d.channels.contains(u.shortChannelId)) {
         val c = d.channels(u.shortChannelId)
         val desc = getDesc(u, c)
-        if (!Announcements.checkSig(u, getDesc(u, d.channels(u.shortChannelId)).a)) {
+        if (d.updates.contains(desc) && d.updates(desc).timestamp >= u.timestamp) {
+          log.debug(s"ignoring $u (old timestamp or duplicate)")
+          stay
+        } else if (!Announcements.checkSig(u, getDesc(u, d.channels(u.shortChannelId)).a)) {
           // TODO: (dirty) this will make the origin channel close the connection
           log.error(s"bad signature for announcement $u")
           sender ! Error(Peer.CHANNELID_ZERO, "bad announcement sig!!!".getBytes())
           stay
-        } else if (d.updates.contains(desc) && d.updates(desc).timestamp >= u.timestamp) {
-          log.debug(s"ignoring $u (old timestamp or duplicate)")
-          stay
-        } else {
-          log.debug(s"added/updated $u")
+        } else if (d.updates.contains(desc)) {
+          log.debug(s"updated $u")
           context.system.eventStream.publish(ChannelUpdateReceived(u))
-          nodeParams.announcementsDb.put(channelUpdateKey(u.shortChannelId, u.flags), u)
+          db.updateChannelUpdate(u)
+          stay using d.copy(updates = d.updates + (desc -> u), rebroadcast = d.rebroadcast :+ u, origins = d.origins + (u -> sender))
+        } else {
+          log.debug(s"added $u")
+          context.system.eventStream.publish(ChannelUpdateReceived(u))
+          db.addChannelUpdate(u)
           stay using d.copy(updates = d.updates + (desc -> u), rebroadcast = d.rebroadcast :+ u, origins = d.origins + (u -> sender))
         }
       } else if (d.awaiting.exists(c => c.shortChannelId == u.shortChannelId) || d.stash.collectFirst { case c: ChannelAnnouncement if c.shortChannelId == u.shortChannelId => c }.isDefined) {
@@ -196,31 +236,28 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
         stay
       }
 
-    case Event(WatchEventSpentBasic(BITCOIN_FUNDING_OTHER_CHANNEL_SPENT(shortChannelId)), d)
+    case Event(WatchEventSpentBasic(BITCOIN_FUNDING_EXTERNAL_CHANNEL_SPENT(shortChannelId)), d)
       if d.channels.containsKey(shortChannelId) =>
       val lostChannel = d.channels(shortChannelId)
-      log.debug(s"funding tx of channelId=$shortChannelId has been spent")
-      log.debug(s"removed channel channelId=$shortChannelId")
+      log.info(s"funding tx of channelId=$shortChannelId has been spent")
+      // we need to remove nodes that aren't tied to any channels anymore
+      val channels1 = d.channels - lostChannel.shortChannelId
+      val lostNodes = Seq(lostChannel.nodeId1, lostChannel.nodeId2).filterNot(nodeId => hasChannels(nodeId, channels1.values))
+      // let's clean the db and send the events
+      log.info(s"pruning shortChannelId=$shortChannelId (spent)")
+      db.removeChannel(shortChannelId) // NB: this also removes channel updates
       context.system.eventStream.publish(ChannelLost(shortChannelId))
-
-      def isNodeLost(nodeId: BinaryData): Option[BinaryData] = {
-        // has nodeId still open channels?
-        if ((d.channels - shortChannelId).values.filter(c => c.nodeId1 == nodeId || c.nodeId2 == nodeId).isEmpty) {
+      lostNodes.foreach {
+        case nodeId =>
+          log.info(s"pruning nodeId=$nodeId (spent)")
+          db.removeNode(nodeId)
           context.system.eventStream.publish(NodeLost(nodeId))
-          log.debug(s"removed node nodeId=$nodeId")
-          Some(nodeId)
-        } else None
       }
-
-      val lostNodes = isNodeLost(lostChannel.nodeId1).toSeq ++ isNodeLost(lostChannel.nodeId2).toSeq
-      nodeParams.announcementsDb.delete(channelKey(shortChannelId))
-      d.updates.values.filter(_.shortChannelId == shortChannelId).foreach(u => nodeParams.announcementsDb.delete(channelUpdateKey(u.shortChannelId, u.flags)))
-      lostNodes.foreach(id => nodeParams.announcementsDb.delete(s"ann-node-$id"))
       stay using d.copy(nodes = d.nodes -- lostNodes, channels = d.channels - shortChannelId, updates = d.updates.filterKeys(_.id != shortChannelId))
 
-    case Event('tick_validate, d) => stay // ignored
+    case Event(TickValidate, d) => stay // ignored
 
-    case Event('tick_broadcast, d) =>
+    case Event(TickBroadcast, d) =>
       d.rebroadcast match {
         case Nil => stay using d.copy(origins = Map.empty)
         case _ =>
@@ -229,6 +266,39 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
           stay using d.copy(rebroadcast = Nil, origins = Map.empty)
       }
 
+    case Event(TickPruneStaleChannels, d) =>
+      // first we select channels that we will prune
+      val staleChannels = getStaleChannels(d.channels, d.updates)
+      // then we clean up the related channel updates
+      val staleUpdates = d.updates.keys.filter(desc => staleChannels.contains(desc.id))
+      // finally we remove nodes that aren't tied to any channels anymore
+      val channels1 = d.channels -- staleChannels
+      val staleNodes = d.nodes.keys.filterNot(nodeId => hasChannels(nodeId, channels1.values))
+      // let's clean the db and send the events
+      staleChannels.foreach {
+        case shortChannelId =>
+          log.info(s"pruning shortChannelId=$shortChannelId (stale)")
+          db.removeChannel(shortChannelId) // NB: this also removes channel updates
+          context.system.eventStream.publish(ChannelLost(shortChannelId))
+      }
+      staleNodes.foreach {
+        case nodeId =>
+          log.info(s"pruning nodeId=$nodeId (stale)")
+          db.removeNode(nodeId)
+          context.system.eventStream.publish(NodeLost(nodeId))
+      }
+      stay using d.copy(nodes = d.nodes -- staleNodes, channels = channels1, updates = d.updates -- staleUpdates)
+
+    case Event(ExcludeChannel(desc@ChannelDesc(shortChannelId, nodeId, _)), d) =>
+      val banDuration = nodeParams.channelExcludeDuration
+      log.info(s"excluding shortChannelId=$shortChannelId from nodeId=$nodeId for duration=$banDuration")
+      context.system.scheduler.scheduleOnce(banDuration, self, LiftChannelExclusion(desc))
+      stay using d.copy(excludedChannels = d.excludedChannels + desc)
+
+    case Event(LiftChannelExclusion(desc@ChannelDesc(shortChannelId, nodeId, _)), d) =>
+      log.info(s"reinstating shortChannelId=$shortChannelId from nodeId=$nodeId")
+      stay using d.copy(excludedChannels = d.excludedChannels - desc)
+
     case Event('nodes, d) =>
       sender ! d.nodes.values
       stay
@@ -246,8 +316,24 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
       stay
 
     case Event(RouteRequest(start, end, ignoreNodes, ignoreChannels), d) =>
-      log.info(s"finding a route $start->$end with ignoreNodes=${ignoreNodes.map(_.toBin).mkString(",")} ignoreChannels=${ignoreChannels.mkString(",")}")
-      findRoute(start, end, filterUpdates(d.updates, ignoreNodes, ignoreChannels)).map(r => RouteResponse(r, ignoreNodes, ignoreChannels)) pipeTo sender
+      val localNodeId = nodeParams.privateKey.publicKey
+      // TODO: HACK!!!!! the following is a workaround to make our routing work with private/not-yet-announced channels, that do not have a channelUpdate
+      val fakeUpdates = d.localChannels.map { case (channelId, remoteNodeId) =>
+        // note that this id is deterministic, otherwise filterUpdates would not work
+        val fakeShortId = BigInt(channelId.take(7).toArray).toLong
+        val channelDesc = ChannelDesc(fakeShortId, localNodeId, remoteNodeId)
+        // note that we store the channelId in the sig, other values are not used because if it is selected this will be the first channel in the route
+        val channelUpdate = ChannelUpdate(signature = channelId, chainHash = nodeParams.chainHash, fakeShortId, 0, "0000", 0, 0, 0, 0)
+        (channelDesc -> channelUpdate)
+      }
+      // we replace local channelUpdates (we have them for regular public already-announced channels) by the ones we just generated
+      val updates1 = d.updates.filterKeys(_.a != localNodeId) ++ fakeUpdates
+      // we then filter out the currently excluded channels
+      val updates2 = updates1.filterKeys(!d.excludedChannels.contains(_))
+      // we also filter out  excluded channels
+      val updates3 = filterUpdates(updates2, ignoreNodes, ignoreChannels)
+      log.info(s"finding a route $start->$end with ignoreNodes=${ignoreNodes.map(_.toBin).mkString(",")} ignoreChannels=${ignoreChannels.map(_.toHexString).mkString(",")}")
+      findRoute(start, end, updates3).map(r => RouteResponse(r, ignoreNodes, ignoreChannels)) pipeTo sender
       stay
   }
 
@@ -261,13 +347,6 @@ class Router(nodeParams: NodeParams, watcher: ActorRef) extends FSM[State, Data]
 
 object Router {
 
-  // TODO: temporary, required because we stored all three types of announcements in the same key-value database
-  // @formatter:off
-  def nodeKey(nodeId: BinaryData) = s"ann-node-$nodeId"
-  def channelKey(shortChannelId: Long) = s"ann-channel-$shortChannelId"
-  def channelUpdateKey(shortChannelId: Long, flags: BinaryData) = s"ann-update-$shortChannelId-$flags"
-  // @formatter:on
-
   val MAX_PARALLEL_JSONRPC_REQUESTS = 50
 
   def props(nodeParams: NodeParams, watcher: ActorRef) = Props(new Router(nodeParams, watcher))
@@ -278,19 +357,39 @@ object Router {
     if (Announcements.isNode1(u.flags)) ChannelDesc(u.shortChannelId, channel.nodeId1, channel.nodeId2) else ChannelDesc(u.shortChannelId, channel.nodeId2, channel.nodeId1)
   }
 
-  def isRelatedTo(c: ChannelAnnouncement, n: NodeAnnouncement) = n.nodeId == c.nodeId1 || n.nodeId == c.nodeId2
+  def isRelatedTo(c: ChannelAnnouncement, nodeId: PublicKey) = nodeId == c.nodeId1 || nodeId == c.nodeId2
 
+  def hasChannels(nodeId: PublicKey, channels: Iterable[ChannelAnnouncement]): Boolean = channels.exists(c => isRelatedTo(c, nodeId))
+
+  def getStaleChannels(channels: Map[Long, ChannelAnnouncement], updates: Map[ChannelDesc, ChannelUpdate]): Iterable[Long] = {
+    // BOLT 7: "nodes MAY prune channels should the timestamp of the latest channel_update be older than 2 weeks (1209600 seconds)"
+    // but we don't want to prune brand new channels for which we didn't yet receive a channel update
+    // so we consider stale a channel that:
+    // (1) is older than 2 weeks (2*7*144 = 2016 blocks)
+    //  AND
+    // (2) didn't have an update during the last 2 weeks
+    val staleThresholdSeconds = Platform.currentTime / 1000 - 1209600
+    val staleThresholdBlocks = Globals.blockCount.get() - 2016
+    val staleChannels = channels
+      .filterKeys(shortChannelId => fromShortId(shortChannelId)._1 < staleThresholdBlocks) // consider only channels older than 2 weeks
+      .filterKeys(shortChannelId => !updates.values.exists(u => u.shortChannelId == shortChannelId && u.timestamp >= staleThresholdSeconds)) // no update in the past 2 weeks
+    staleChannels.keys
+  }
+
+  /**
+    * This method is used after a payment failed, and we want to exclude some nodes/channels that we know are failing
+    */
   def filterUpdates(updates: Map[ChannelDesc, ChannelUpdate], ignoreNodes: Set[PublicKey], ignoreChannels: Set[Long]) =
     updates
       .filterNot(u => ignoreNodes.map(_.toBin).contains(u._1.a) || ignoreNodes.map(_.toBin).contains(u._1.b))
       .filterNot(u => ignoreChannels.contains(u._1.id))
       .filterNot(u => !Announcements.isEnabled(u._2.flags))
 
-  def findRouteDijkstra(localNodeId: BinaryData, targetNodeId: BinaryData, channels: Iterable[ChannelDesc]): Seq[ChannelDesc] = {
+  def findRouteDijkstra(localNodeId: PublicKey, targetNodeId: PublicKey, channels: Iterable[ChannelDesc]): Seq[ChannelDesc] = {
     if (localNodeId == targetNodeId) throw CannotRouteToSelf
     case class DescEdge(desc: ChannelDesc) extends DefaultEdge
-    val g = new DefaultDirectedGraph[BinaryData, DescEdge](classOf[DescEdge])
-    channels.foreach(d => {
+    val g = new DefaultDirectedGraph[PublicKey, DescEdge](classOf[DescEdge])
+    Random.shuffle(channels).foreach(d => {
       g.addVertex(d.a)
       g.addVertex(d.b)
       g.addEdge(d.a, d.b, new DescEdge(d))
@@ -301,35 +400,35 @@ object Router {
     }
   }
 
-  def findRoute(localNodeId: BinaryData, targetNodeId: BinaryData, updates: Map[ChannelDesc, ChannelUpdate])(implicit ec: ExecutionContext): Future[Seq[Hop]] = Future {
+  def findRoute(localNodeId: PublicKey, targetNodeId: PublicKey, updates: Map[ChannelDesc, ChannelUpdate])(implicit ec: ExecutionContext): Future[Seq[Hop]] = Future {
     findRouteDijkstra(localNodeId, targetNodeId, updates.keys)
       .map(desc => Hop(desc.a, desc.b, updates(desc)))
   }
 
-  def graph2dot(nodes: Map[BinaryData, NodeAnnouncement], channels: Map[Long, ChannelAnnouncement])(implicit ec: ExecutionContext): Future[String] = Future {
+  def graph2dot(nodes: Map[PublicKey, NodeAnnouncement], channels: Map[Long, ChannelAnnouncement])(implicit ec: ExecutionContext): Future[String] = Future {
     case class DescEdge(channelId: Long) extends DefaultEdge
-    val g = new SimpleGraph[BinaryData, DescEdge](classOf[DescEdge])
+    val g = new SimpleGraph[PublicKey, DescEdge](classOf[DescEdge])
     channels.foreach(d => {
       g.addVertex(d._2.nodeId1)
       g.addVertex(d._2.nodeId2)
       g.addEdge(d._2.nodeId1, d._2.nodeId2, new DescEdge(d._1))
     })
-    val vertexIDProvider = new ComponentNameProvider[BinaryData]() {
-      override def getName(nodeId: BinaryData): String = "\"" + nodeId.toString() + "\""
+    val vertexIDProvider = new ComponentNameProvider[PublicKey]() {
+      override def getName(nodeId: PublicKey): String = "\"" + nodeId.toString() + "\""
     }
     val edgeLabelProvider = new ComponentNameProvider[DescEdge]() {
       override def getName(e: DescEdge): String = e.channelId.toString
     }
-    val vertexAttributeProvider = new ComponentAttributeProvider[BinaryData]() {
+    val vertexAttributeProvider = new ComponentAttributeProvider[PublicKey]() {
 
-      override def getComponentAttributes(nodeId: BinaryData): java.util.Map[String, String] =
+      override def getComponentAttributes(nodeId: PublicKey): java.util.Map[String, String] =
 
         nodes.get(nodeId) match {
           case Some(ann) => Map("label" -> ann.alias, "color" -> f"#${ann.rgbColor._1}%02x${ann.rgbColor._2}%02x${ann.rgbColor._3}%02x")
           case None => Map.empty[String, String]
         }
     }
-    val exporter = new DOTExporter[BinaryData, DescEdge](vertexIDProvider, null, edgeLabelProvider, vertexAttributeProvider, null)
+    val exporter = new DOTExporter[PublicKey, DescEdge](vertexIDProvider, null, edgeLabelProvider, vertexAttributeProvider, null)
     val writer = new StringWriter()
     try {
       exporter.exportGraph(g, writer)

eclair-core/src/main/scala/fr/acinq/eclair/router/RouterExceptions.scala

@@ -7,4 +7,5 @@ package fr.acinq.eclair.router
 class RouterException(message: String) extends RuntimeException(message)
 
 object RouteNotFound extends RouterException("Route not found")
+
 object CannotRouteToSelf extends RouterException("Cannot route to self")

eclair-core/src/main/scala/fr/acinq/eclair/router/ThrottleForwarder.scala

@@ -0,0 +1,47 @@
+package fr.acinq.eclair.router
+
+import akka.actor.{Actor, ActorLogging, ActorRef, Props}
+
+import scala.concurrent.duration.{FiniteDuration, _}
+
+/**
+  * This actor forwards messages to another actor, but groups them and introduces
+  * delays between each groups.
+  *
+  * If A wants to send a lot of lower importance messages to B, it is useful to let
+  * higher importance messages go in the stream.
+  */
+class ThrottleForwarder(target: ActorRef, messages: Iterable[Any], chunkSize: Int, delay: FiniteDuration) extends Actor with ActorLogging {
+
+  import ThrottleForwarder.Tick
+
+  import scala.concurrent.ExecutionContext.Implicits.global
+
+  val clock = context.system.scheduler.schedule(0 second, delay, self, Tick)
+
+  log.debug(s"sending messages=${messages.size} with chunkSize=$chunkSize and delay=$delay")
+
+  override def receive = group(messages)
+
+  def group(messages: Iterable[Any]): Receive = {
+    case Tick =>
+      messages.splitAt(chunkSize) match {
+        case (Nil, _) =>
+          clock.cancel()
+          log.debug(s"sent messages=${messages.size} with chunkSize=$chunkSize and delay=$delay")
+          context stop self
+        case (chunk, rest) =>
+          chunk.foreach(target ! _)
+          context become group(rest)
+      }
+  }
+
+}
+
+object ThrottleForwarder {
+
+  def props(target: ActorRef, messages: Iterable[Any], groupSize: Int, delay: FiniteDuration) = Props(new ThrottleForwarder(target, messages, groupSize, delay))
+
+  case object Tick
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/transactions/CommitmentSpec.scala

@@ -1,6 +1,5 @@
 package fr.acinq.eclair.transactions
 
-import fr.acinq.bitcoin.BinaryData
 import fr.acinq.eclair.wire._
 
 /**
@@ -13,9 +12,9 @@ case object IN extends Direction { def opposite = OUT }
 case object OUT extends Direction { def opposite = IN }
 // @formatter:on
 
-case class Htlc(direction: Direction, add: UpdateAddHtlc, val previousChannelId: Option[BinaryData])
+case class DirectedHtlc(direction: Direction, add: UpdateAddHtlc)
 
-final case class CommitmentSpec(htlcs: Set[Htlc], feeratePerKw: Long, toLocalMsat: Long, toRemoteMsat: Long) {
+final case class CommitmentSpec(htlcs: Set[DirectedHtlc], feeratePerKw: Long, toLocalMsat: Long, toRemoteMsat: Long) {
   val totalFunds = toLocalMsat + toRemoteMsat + htlcs.toSeq.map(_.add.amountMsat).sum
 }
 
@@ -26,7 +25,7 @@ object CommitmentSpec {
   })
 
   def addHtlc(spec: CommitmentSpec, direction: Direction, update: UpdateAddHtlc): CommitmentSpec = {
-    val htlc = Htlc(direction, update, previousChannelId = None)
+    val htlc = DirectedHtlc(direction, update)
     direction match {
       case OUT => spec.copy(toLocalMsat = spec.toLocalMsat - htlc.add.amountMsat, htlcs = spec.htlcs + htlc)
       case IN => spec.copy(toRemoteMsat = spec.toRemoteMsat - htlc.add.amountMsat, htlcs = spec.htlcs + htlc)
@@ -34,20 +33,20 @@ object CommitmentSpec {
   }
 
   // OUT means we are sending an UpdateFulfillHtlc message which means that we are fulfilling an HTLC that they sent
-  def fulfillHtlc(spec: CommitmentSpec, direction: Direction, update: UpdateFulfillHtlc): CommitmentSpec = {
-    spec.htlcs.find(htlc => htlc.direction != direction && htlc.add.id == update.id) match {
+  def fulfillHtlc(spec: CommitmentSpec, direction: Direction, htlcId: Long): CommitmentSpec = {
+    spec.htlcs.find(htlc => htlc.direction != direction && htlc.add.id == htlcId) match {
       case Some(htlc) if direction == OUT => spec.copy(toLocalMsat = spec.toLocalMsat + htlc.add.amountMsat, htlcs = spec.htlcs - htlc)
       case Some(htlc) if direction == IN => spec.copy(toRemoteMsat = spec.toRemoteMsat + htlc.add.amountMsat, htlcs = spec.htlcs - htlc)
-      case None => throw new RuntimeException(s"cannot find htlc id=${update.id}")
+      case None => throw new RuntimeException(s"cannot find htlc id=${htlcId}")
     }
   }
 
   // OUT means we are sending an UpdateFailHtlc message which means that we are failing an HTLC that they sent
-  def failHtlc(spec: CommitmentSpec, direction: Direction, update: UpdateFailHtlc): CommitmentSpec = {
-    spec.htlcs.find(htlc => htlc.direction != direction && htlc.add.id == update.id) match {
+  def failHtlc(spec: CommitmentSpec, direction: Direction, htlcId: Long): CommitmentSpec = {
+    spec.htlcs.find(htlc => htlc.direction != direction && htlc.add.id == htlcId) match {
       case Some(htlc) if direction == OUT => spec.copy(toRemoteMsat = spec.toRemoteMsat + htlc.add.amountMsat, htlcs = spec.htlcs - htlc)
       case Some(htlc) if direction == IN => spec.copy(toLocalMsat = spec.toLocalMsat + htlc.add.amountMsat, htlcs = spec.htlcs - htlc)
-      case None => throw new RuntimeException(s"cannot find htlc id=${update.id}")
+      case None => throw new RuntimeException(s"cannot find htlc id=${htlcId}")
     }
   }
 
@@ -61,13 +60,15 @@ object CommitmentSpec {
       case (spec, _) => spec
     }
     val spec3 = localChanges.foldLeft(spec2) {
-      case (spec, u: UpdateFulfillHtlc) => fulfillHtlc(spec, OUT, u)
-      case (spec, u: UpdateFailHtlc) => failHtlc(spec, OUT, u)
+      case (spec, u: UpdateFulfillHtlc) => fulfillHtlc(spec, OUT, u.id)
+      case (spec, u: UpdateFailHtlc) => failHtlc(spec, OUT, u.id)
+      case (spec, u: UpdateFailMalformedHtlc) => failHtlc(spec, OUT, u.id)
       case (spec, _) => spec
     }
     val spec4 = remoteChanges.foldLeft(spec3) {
-      case (spec, u: UpdateFulfillHtlc) => fulfillHtlc(spec, IN, u)
-      case (spec, u: UpdateFailHtlc) => failHtlc(spec, IN, u)
+      case (spec, u: UpdateFulfillHtlc) => fulfillHtlc(spec, IN, u.id)
+      case (spec, u: UpdateFailHtlc) => failHtlc(spec, IN, u.id)
+      case (spec, u: UpdateFailMalformedHtlc) => failHtlc(spec, IN, u.id)
       case (spec, _) => spec
     }
     val spec5 = (localChanges ++ remoteChanges).foldLeft(spec4) {

eclair-core/src/main/scala/fr/acinq/eclair/transactions/Scripts.scala

@@ -1,8 +1,8 @@
 package fr.acinq.eclair.transactions
 
-import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey, ripemd160}
+import fr.acinq.bitcoin.Crypto.{PublicKey, ripemd160}
 import fr.acinq.bitcoin.Script._
-import fr.acinq.bitcoin.{BinaryData, Crypto, LexicographicalOrdering, LockTimeThreshold, OP_0, OP_1, OP_1NEGATE, OP_2, OP_2DROP, OP_ADD, OP_CHECKLOCKTIMEVERIFY, OP_CHECKMULTISIG, OP_CHECKSEQUENCEVERIFY, OP_CHECKSIG, OP_DROP, OP_DUP, OP_ELSE, OP_ENDIF, OP_EQUAL, OP_EQUALVERIFY, OP_HASH160, OP_IF, OP_NOTIF, OP_PUSHDATA, OP_SIZE, OP_SWAP, OutPoint, SIGHASH_ALL, Satoshi, Script, ScriptElt, ScriptWitness, Transaction, TxIn, TxOut}
+import fr.acinq.bitcoin.{BinaryData, LexicographicalOrdering, LockTimeThreshold, OP_0, OP_1, OP_1NEGATE, OP_2, OP_2DROP, OP_ADD, OP_CHECKLOCKTIMEVERIFY, OP_CHECKMULTISIG, OP_CHECKSEQUENCEVERIFY, OP_CHECKSIG, OP_DROP, OP_DUP, OP_ELSE, OP_ENDIF, OP_EQUAL, OP_EQUALVERIFY, OP_HASH160, OP_IF, OP_NOTIF, OP_PUSHDATA, OP_SIZE, OP_SWAP, Satoshi, Script, ScriptElt, ScriptWitness, Transaction, TxIn}
 
 /**
   * Created by PM on 02/12/2016.
@@ -156,13 +156,13 @@ object Scripts {
     else tx.txIn.map(_.sequence).map(sequenceToBlockHeight).max
   }
 
-  def toLocalDelayed(revocationPubkey: PublicKey, toSelfDelay: Int, localDelayedPubkey: PublicKey) = {
+  def toLocalDelayed(revocationPubkey: PublicKey, toSelfDelay: Int, localDelayedPaymentPubkey: PublicKey) = {
     // @formatter:off
     OP_IF ::
       OP_PUSHDATA(revocationPubkey) ::
     OP_ELSE ::
       encodeNumber(toSelfDelay) :: OP_CHECKSEQUENCEVERIFY :: OP_DROP ::
-      OP_PUSHDATA(localDelayedPubkey) ::
+      OP_PUSHDATA(localDelayedPaymentPubkey) ::
     OP_ENDIF ::
     OP_CHECKSIG :: Nil
     // @formatter:on
@@ -181,17 +181,17 @@ object Scripts {
   def witnessToLocalDelayedWithRevocationSig(revocationSig: BinaryData, toLocalScript: BinaryData) =
     ScriptWitness(revocationSig :: BinaryData("01") :: toLocalScript :: Nil)
 
-  def htlcOffered(localPubkey: PublicKey, remotePubkey: PublicKey, revocationPubKey: PublicKey, paymentHash: BinaryData): Seq[ScriptElt] = {
+  def htlcOffered(localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, revocationPubKey: PublicKey, paymentHash: BinaryData): Seq[ScriptElt] = {
     // @formatter:off
     // To you with revocation key
     OP_DUP :: OP_HASH160 :: OP_PUSHDATA(revocationPubKey.hash160) :: OP_EQUAL ::
     OP_IF ::
         OP_CHECKSIG ::
     OP_ELSE ::
-        OP_PUSHDATA(remotePubkey) :: OP_SWAP  :: OP_SIZE :: encodeNumber(32) :: OP_EQUAL ::
+        OP_PUSHDATA(remoteHtlcPubkey) :: OP_SWAP  :: OP_SIZE :: encodeNumber(32) :: OP_EQUAL ::
         OP_NOTIF ::
             // To me via HTLC-timeout transaction (timelocked).
-            OP_DROP :: OP_2 :: OP_SWAP :: OP_PUSHDATA(localPubkey) :: OP_2 :: OP_CHECKMULTISIG ::
+            OP_DROP :: OP_2 :: OP_SWAP :: OP_PUSHDATA(localHtlcPubkey) :: OP_2 :: OP_CHECKMULTISIG ::
         OP_ELSE ::
             OP_HASH160 :: OP_PUSHDATA(paymentHash) :: OP_EQUALVERIFY ::
             OP_CHECKSIG ::
@@ -213,18 +213,18 @@ object Scripts {
   def witnessClaimHtlcSuccessFromCommitTx(localSig: BinaryData, paymentPreimage: BinaryData, htlcOfferedScript: BinaryData) =
     ScriptWitness(localSig :: paymentPreimage :: htlcOfferedScript :: Nil)
 
-  def htlcReceived(localKey: PublicKey, remotePubkey: PublicKey, revocationPubKey: PublicKey, paymentHash: BinaryData, lockTime: Long) = {
+  def htlcReceived(localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, revocationPubKey: PublicKey, paymentHash: BinaryData, lockTime: Long) = {
     // @formatter:off
     // To you with revocation key
     OP_DUP :: OP_HASH160 :: OP_PUSHDATA(revocationPubKey.hash160) :: OP_EQUAL ::
     OP_IF ::
         OP_CHECKSIG ::
     OP_ELSE ::
-        OP_PUSHDATA(remotePubkey) :: OP_SWAP :: OP_SIZE :: encodeNumber(32) :: OP_EQUAL ::
+        OP_PUSHDATA(remoteHtlcPubkey) :: OP_SWAP :: OP_SIZE :: encodeNumber(32) :: OP_EQUAL ::
         OP_IF ::
             // To me via HTLC-success transaction.
             OP_HASH160 :: OP_PUSHDATA(paymentHash) :: OP_EQUALVERIFY ::
-            OP_2 :: OP_SWAP :: OP_PUSHDATA(localKey) :: OP_2 :: OP_CHECKMULTISIG ::
+            OP_2 :: OP_SWAP :: OP_PUSHDATA(localHtlcPubkey) :: OP_2 :: OP_CHECKMULTISIG ::
         OP_ELSE ::
             // To you after timeout.
             OP_DROP :: encodeNumber(lockTime) :: OP_CHECKLOCKTIMEVERIFY :: OP_DROP ::

eclair-core/src/main/scala/fr/acinq/eclair/transactions/Transactions.scala

@@ -73,7 +73,7 @@ object Transactions {
 
   def weight2fee(feeratePerKw: Long, weight: Int) = Satoshi((feeratePerKw * weight) / 1000)
 
-  def trimOfferedHtlcs(dustLimit: Satoshi, spec: CommitmentSpec): Seq[Htlc] = {
+  def trimOfferedHtlcs(dustLimit: Satoshi, spec: CommitmentSpec): Seq[DirectedHtlc] = {
     val htlcTimeoutFee = weight2fee(spec.feeratePerKw, htlcTimeoutWeight)
     spec.htlcs
       .filter(_.direction == OUT)
@@ -81,7 +81,7 @@ object Transactions {
       .toSeq
   }
 
-  def trimReceivedHtlcs(dustLimit: Satoshi, spec: CommitmentSpec): Seq[Htlc] = {
+  def trimReceivedHtlcs(dustLimit: Satoshi, spec: CommitmentSpec): Seq[DirectedHtlc] = {
     val htlcSuccessFee = weight2fee(spec.feeratePerKw, htlcSuccessWeight)
     spec.htlcs
       .filter(_.direction == IN)
@@ -142,7 +142,7 @@ object Transactions {
 
   def decodeTxNumber(sequence: Long, locktime: Long) = ((sequence & 0xffffffL) << 24) + (locktime & 0xffffffL)
 
-  def makeCommitTx(commitTxInput: InputInfo, commitTxNumber: Long, localPaymentBasePoint: Point, remotePaymentBasePoint: Point, localIsFunder: Boolean, localDustLimit: Satoshi, localPubKey: PublicKey, localRevocationPubkey: PublicKey, toLocalDelay: Int, localDelayedPubkey: PublicKey, remotePubkey: PublicKey, spec: CommitmentSpec): CommitTx = {
+  def makeCommitTx(commitTxInput: InputInfo, commitTxNumber: Long, localPaymentBasePoint: Point, remotePaymentBasePoint: Point, localIsFunder: Boolean, localDustLimit: Satoshi, localRevocationPubkey: PublicKey, toLocalDelay: Int, localDelayedPaymentPubkey: PublicKey, remotePaymentPubkey: PublicKey, localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, spec: CommitmentSpec): CommitTx = {
 
     val commitFee = commitTxFee(localDustLimit, spec)
 
@@ -151,13 +151,13 @@ object Transactions {
       case false => (millisatoshi2satoshi(MilliSatoshi(spec.toLocalMsat)), millisatoshi2satoshi(MilliSatoshi(spec.toRemoteMsat)) - commitFee)
     } // NB: we don't care if values are < 0, they will be trimmed if they are < dust limit anyway
 
-    val toLocalDelayedOutput_opt = if (toLocalAmount >= localDustLimit) Some(TxOut(toLocalAmount, pay2wsh(toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPubkey)))) else None
-    val toRemoteOutput_opt = if (toRemoteAmount >= localDustLimit) Some(TxOut(toRemoteAmount, pay2wpkh(remotePubkey))) else None
+    val toLocalDelayedOutput_opt = if (toLocalAmount >= localDustLimit) Some(TxOut(toLocalAmount, pay2wsh(toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPaymentPubkey)))) else None
+    val toRemoteOutput_opt = if (toRemoteAmount >= localDustLimit) Some(TxOut(toRemoteAmount, pay2wpkh(remotePaymentPubkey))) else None
 
     val htlcOfferedOutputs = trimOfferedHtlcs(localDustLimit, spec)
-      .map(htlc => TxOut(MilliSatoshi(htlc.add.amountMsat), pay2wsh(htlcOffered(localPubKey, remotePubkey, localRevocationPubkey, ripemd160(htlc.add.paymentHash)))))
+      .map(htlc => TxOut(MilliSatoshi(htlc.add.amountMsat), pay2wsh(htlcOffered(localHtlcPubkey, remoteHtlcPubkey, localRevocationPubkey, ripemd160(htlc.add.paymentHash)))))
     val htlcReceivedOutputs = trimReceivedHtlcs(localDustLimit, spec)
-      .map(htlc => TxOut(MilliSatoshi(htlc.add.amountMsat), pay2wsh(htlcReceived(localPubKey, remotePubkey, localRevocationPubkey, ripemd160(htlc.add.paymentHash), htlc.add.expiry))))
+      .map(htlc => TxOut(MilliSatoshi(htlc.add.amountMsat), pay2wsh(htlcReceived(localHtlcPubkey, remoteHtlcPubkey, localRevocationPubkey, ripemd160(htlc.add.paymentHash), htlc.add.expiry))))
 
     val txnumber = obscuredCommitTxNumber(commitTxNumber, localIsFunder, localPaymentBasePoint, remotePaymentBasePoint)
     val (sequence, locktime) = encodeTxNumber(txnumber)
@@ -170,131 +170,127 @@ object Transactions {
     CommitTx(commitTxInput, LexicographicalOrdering.sort(tx))
   }
 
-  def makeHtlcTimeoutTx(commitTx: Transaction, localRevocationPubkey: PublicKey, toLocalDelay: Int, localPubKey: PublicKey, localDelayedPubkey: PublicKey, remotePubkey: PublicKey, feeratePerKw: Long, htlc: UpdateAddHtlc): HtlcTimeoutTx = {
+  def makeHtlcTimeoutTx(commitTx: Transaction, localDustLimit: Satoshi, localRevocationPubkey: PublicKey, toLocalDelay: Int, localDelayedPaymentPubkey: PublicKey, localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, feeratePerKw: Long, htlc: UpdateAddHtlc): HtlcTimeoutTx = {
     val fee = weight2fee(feeratePerKw, htlcTimeoutWeight)
-    val redeemScript = htlcOffered(localPubKey, remotePubkey, localRevocationPubkey, ripemd160(htlc.paymentHash))
+    val redeemScript = htlcOffered(localHtlcPubkey, remoteHtlcPubkey, localRevocationPubkey, ripemd160(htlc.paymentHash))
     val pubkeyScript = write(pay2wsh(redeemScript))
     val outputIndex = findPubKeyScriptIndex(commitTx, pubkeyScript)
-    require(outputIndex >= 0, "output not found")
+    require(outputIndex >= 0, "output not found (was trimmed?)")
+    val amount = MilliSatoshi(htlc.amountMsat) - fee
+    require(amount >= localDustLimit, "amount lesser than dust limit")
     val input = InputInfo(OutPoint(commitTx, outputIndex), commitTx.txOut(outputIndex), write(redeemScript))
     HtlcTimeoutTx(input, Transaction(
       version = 2,
       txIn = TxIn(input.outPoint, Array.emptyByteArray, 0) :: Nil,
-      txOut = TxOut(MilliSatoshi(htlc.amountMsat) - fee, pay2wsh(toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPubkey))) :: Nil,
+      txOut = TxOut(amount, pay2wsh(toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPaymentPubkey))) :: Nil,
       lockTime = htlc.expiry))
   }
 
-  def makeHtlcSuccessTx(commitTx: Transaction, localRevocationPubkey: PublicKey, toLocalDelay: Int, localPubkey: PublicKey, localDelayedPubkey: PublicKey, remotePubkey: PublicKey, feeratePerKw: Long, htlc: UpdateAddHtlc): HtlcSuccessTx = {
+  def makeHtlcSuccessTx(commitTx: Transaction, localDustLimit: Satoshi, localRevocationPubkey: PublicKey, toLocalDelay: Int, localDelayedPaymentPubkey: PublicKey, localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, feeratePerKw: Long, htlc: UpdateAddHtlc): HtlcSuccessTx = {
     val fee = weight2fee(feeratePerKw, htlcSuccessWeight)
-    val redeemScript = htlcReceived(localPubkey, remotePubkey, localRevocationPubkey, ripemd160(htlc.paymentHash), htlc.expiry)
+    val redeemScript = htlcReceived(localHtlcPubkey, remoteHtlcPubkey, localRevocationPubkey, ripemd160(htlc.paymentHash), htlc.expiry)
     val pubkeyScript = write(pay2wsh(redeemScript))
     val outputIndex = findPubKeyScriptIndex(commitTx, pubkeyScript)
-    require(outputIndex >= 0, "output not found")
+    require(outputIndex >= 0, "output not found (was trimmed?)")
+    val amount = MilliSatoshi(htlc.amountMsat) - fee
+    require(amount >= localDustLimit, "amount lesser than dust limit")
     val input = InputInfo(OutPoint(commitTx, outputIndex), commitTx.txOut(outputIndex), write(redeemScript))
     HtlcSuccessTx(input, Transaction(
       version = 2,
       txIn = TxIn(input.outPoint, Array.emptyByteArray, 0) :: Nil,
-      txOut = TxOut(MilliSatoshi(htlc.amountMsat) - fee, pay2wsh(toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPubkey))) :: Nil,
+      txOut = TxOut(amount, pay2wsh(toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPaymentPubkey))) :: Nil,
       lockTime = 0), htlc.paymentHash)
   }
 
-  def makeHtlcTxs(commitTx: Transaction, localDustLimit: Satoshi, localRevocationPubkey: PublicKey, toLocalDelay: Int, localPubkey: PublicKey, localDelayedPubkey: PublicKey, remotePubkey: PublicKey, spec: CommitmentSpec): (Seq[HtlcTimeoutTx], Seq[HtlcSuccessTx]) = {
+  def makeHtlcTxs(commitTx: Transaction, localDustLimit: Satoshi, localRevocationPubkey: PublicKey, toLocalDelay: Int, localDelayedPaymentPubkey: PublicKey, localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, spec: CommitmentSpec): (Seq[HtlcTimeoutTx], Seq[HtlcSuccessTx]) = {
     val htlcTimeoutTxs = trimOfferedHtlcs(localDustLimit, spec)
-      .map(htlc => makeHtlcTimeoutTx(commitTx, localRevocationPubkey, toLocalDelay, localPubkey, localDelayedPubkey, remotePubkey, spec.feeratePerKw, htlc.add))
+      .map(htlc => makeHtlcTimeoutTx(commitTx, localDustLimit, localRevocationPubkey, toLocalDelay, localDelayedPaymentPubkey, localHtlcPubkey, remoteHtlcPubkey, spec.feeratePerKw, htlc.add))
     val htlcSuccessTxs = trimReceivedHtlcs(localDustLimit, spec)
-      .map(htlc => makeHtlcSuccessTx(commitTx, localRevocationPubkey, toLocalDelay, localPubkey, localDelayedPubkey, remotePubkey, spec.feeratePerKw, htlc.add))
+      .map(htlc => makeHtlcSuccessTx(commitTx, localDustLimit, localRevocationPubkey, toLocalDelay, localDelayedPaymentPubkey, localHtlcPubkey, remoteHtlcPubkey, spec.feeratePerKw, htlc.add))
     (htlcTimeoutTxs, htlcSuccessTxs)
   }
 
-  def makeClaimHtlcSuccessTx(commitTx: Transaction, localPubkey: PublicKey, remotePubkey: PublicKey, remoteRevocationPubkey: PublicKey, localFinalScriptPubKey: BinaryData, htlc: UpdateAddHtlc, feeratePerKw: Long): ClaimHtlcSuccessTx = {
+  def makeClaimHtlcSuccessTx(commitTx: Transaction, localDustLimit: Satoshi, localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, remoteRevocationPubkey: PublicKey, localFinalScriptPubKey: BinaryData, htlc: UpdateAddHtlc, feeratePerKw: Long): ClaimHtlcSuccessTx = {
     val fee = weight2fee(feeratePerKw, claimHtlcSuccessWeight)
-    val redeemScript = htlcOffered(remotePubkey, localPubkey, remoteRevocationPubkey, ripemd160(htlc.paymentHash))
+    val redeemScript = htlcOffered(remoteHtlcPubkey, localHtlcPubkey, remoteRevocationPubkey, ripemd160(htlc.paymentHash))
     val pubkeyScript = write(pay2wsh(redeemScript))
     val outputIndex = findPubKeyScriptIndex(commitTx, pubkeyScript)
-    require(outputIndex >= 0, "output not found")
+    require(outputIndex >= 0, "output not found (was trimmed?)")
     val input = InputInfo(OutPoint(commitTx, outputIndex), commitTx.txOut(outputIndex), write(redeemScript))
+    val amount = input.txOut.amount - fee
+    require(amount >= localDustLimit, "amount lesser than dust limit")
     ClaimHtlcSuccessTx(input, Transaction(
       version = 2,
       txIn = TxIn(input.outPoint, Array.emptyByteArray, 0xffffffffL) :: Nil,
-      txOut = TxOut(input.txOut.amount - fee, localFinalScriptPubKey) :: Nil,
+      txOut = TxOut(amount, localFinalScriptPubKey) :: Nil,
       lockTime = 0))
   }
 
-  def makeClaimHtlcTimeoutTx(commitTx: Transaction, localPubkey: PublicKey, remotePubkey: PublicKey, remoteRevocationPubkey: PublicKey, localFinalScriptPubKey: BinaryData, htlc: UpdateAddHtlc, feeratePerKw: Long): ClaimHtlcTimeoutTx = {
+  def makeClaimHtlcTimeoutTx(commitTx: Transaction, localDustLimit: Satoshi, localHtlcPubkey: PublicKey, remoteHtlcPubkey: PublicKey, remoteRevocationPubkey: PublicKey, localFinalScriptPubKey: BinaryData, htlc: UpdateAddHtlc, feeratePerKw: Long): ClaimHtlcTimeoutTx = {
     val fee = weight2fee(feeratePerKw, claimHtlcTimeoutWeight)
-    val redeemScript = htlcReceived(remotePubkey, localPubkey, remoteRevocationPubkey, ripemd160(htlc.paymentHash), htlc.expiry)
+    val redeemScript = htlcReceived(remoteHtlcPubkey, localHtlcPubkey, remoteRevocationPubkey, ripemd160(htlc.paymentHash), htlc.expiry)
     val pubkeyScript = write(pay2wsh(redeemScript))
     val outputIndex = findPubKeyScriptIndex(commitTx, pubkeyScript)
-    require(outputIndex >= 0, "output not found")
+    require(outputIndex >= 0, "output not found (was trimmed?)")
     val input = InputInfo(OutPoint(commitTx, outputIndex), commitTx.txOut(outputIndex), write(redeemScript))
+    val amount = input.txOut.amount - fee
+    require(amount >= localDustLimit, "amount lesser than dust limit")
     ClaimHtlcTimeoutTx(input, Transaction(
       version = 2,
       txIn = TxIn(input.outPoint, Array.emptyByteArray, 0x00000000L) :: Nil,
-      txOut = TxOut(input.txOut.amount - fee, localFinalScriptPubKey) :: Nil,
+      txOut = TxOut(amount, localFinalScriptPubKey) :: Nil,
       lockTime = htlc.expiry))
   }
 
-  def makeClaimP2WPKHOutputTx(delayedOutputTx: Transaction, localPubkey: PublicKey, localFinalScriptPubKey: BinaryData, feeratePerKw: Long): ClaimP2WPKHOutputTx = {
+  def makeClaimP2WPKHOutputTx(delayedOutputTx: Transaction, localDustLimit: Satoshi, localPaymentPubkey: PublicKey, localFinalScriptPubKey: BinaryData, feeratePerKw: Long): ClaimP2WPKHOutputTx = {
     val fee = weight2fee(feeratePerKw, claimP2WPKHOutputWeight)
-    val redeemScript = Script.pay2pkh(localPubkey)
-    val pubkeyScript = write(pay2wpkh(localPubkey))
+    val redeemScript = Script.pay2pkh(localPaymentPubkey)
+    val pubkeyScript = write(pay2wpkh(localPaymentPubkey))
     val outputIndex = findPubKeyScriptIndex(delayedOutputTx, pubkeyScript)
-    require(outputIndex >= 0, "output not found")
+    require(outputIndex >= 0, "output not found (was trimmed?)")
     val input = InputInfo(OutPoint(delayedOutputTx, outputIndex), delayedOutputTx.txOut(outputIndex), write(redeemScript))
+    val amount = input.txOut.amount - fee
+    require(amount >= localDustLimit, "amount lesser than dust limit")
     ClaimP2WPKHOutputTx(input, Transaction(
       version = 2,
       txIn = TxIn(input.outPoint, Array.emptyByteArray, 0x00000000L) :: Nil,
-      txOut = TxOut(input.txOut.amount - fee, localFinalScriptPubKey) :: Nil,
+      txOut = TxOut(amount, localFinalScriptPubKey) :: Nil,
       lockTime = 0))
   }
 
-  def makeClaimDelayedOutputTx(delayedOutputTx: Transaction, localRevocationPubkey: PublicKey, toLocalDelay: Int, localDelayedPubkey: PublicKey, localFinalScriptPubKey: BinaryData, feeratePerKw: Long): ClaimDelayedOutputTx = {
+  def makeClaimDelayedOutputTx(delayedOutputTx: Transaction, localDustLimit: Satoshi, localRevocationPubkey: PublicKey, toLocalDelay: Int, localDelayedPaymentPubkey: PublicKey, localFinalScriptPubKey: BinaryData, feeratePerKw: Long): ClaimDelayedOutputTx = {
     val fee = weight2fee(feeratePerKw, claimHtlcDelayedWeight)
-    val redeemScript = toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPubkey)
+    val redeemScript = toLocalDelayed(localRevocationPubkey, toLocalDelay, localDelayedPaymentPubkey)
     val pubkeyScript = write(pay2wsh(redeemScript))
     val outputIndex = findPubKeyScriptIndex(delayedOutputTx, pubkeyScript)
-    require(outputIndex >= 0, "output not found")
+    require(outputIndex >= 0, "output not found (was trimmed?)")
     val input = InputInfo(OutPoint(delayedOutputTx, outputIndex), delayedOutputTx.txOut(outputIndex), write(redeemScript))
+    val amount = input.txOut.amount - fee
+    require(amount >= localDustLimit, "amount lesser than dust limit")
     ClaimDelayedOutputTx(input, Transaction(
       version = 2,
       txIn = TxIn(input.outPoint, Array.emptyByteArray, toLocalDelay) :: Nil,
-      txOut = TxOut(input.txOut.amount - fee, localFinalScriptPubKey) :: Nil,
+      txOut = TxOut(amount, localFinalScriptPubKey) :: Nil,
       lockTime = 0))
   }
 
-  def makeMainPenaltyTx(commitTx: Transaction, remoteRevocationPubkey: PublicKey, localFinalScriptPubKey: BinaryData, toRemoteDelay: Int, remoteDelayedPubkey: PublicKey, feeratePerKw: Long): MainPenaltyTx = {
+  def makeMainPenaltyTx(commitTx: Transaction, localDustLimit: Satoshi, remoteRevocationPubkey: PublicKey, localFinalScriptPubKey: BinaryData, toRemoteDelay: Int, remoteDelayedPaymentPubkey: PublicKey, feeratePerKw: Long): MainPenaltyTx = {
     val fee = weight2fee(feeratePerKw, mainPenaltyWeight)
-    val redeemScript = toLocalDelayed(remoteRevocationPubkey, toRemoteDelay, remoteDelayedPubkey)
+    val redeemScript = toLocalDelayed(remoteRevocationPubkey, toRemoteDelay, remoteDelayedPaymentPubkey)
     val pubkeyScript = write(pay2wsh(redeemScript))
     val outputIndex = findPubKeyScriptIndex(commitTx, pubkeyScript)
-    require(outputIndex >= 0, "output not found")
+    require(outputIndex >= 0, "output not found (was trimmed?)")
     val input = InputInfo(OutPoint(commitTx, outputIndex), commitTx.txOut(outputIndex), write(redeemScript))
+    val amount = input.txOut.amount - fee
+    require(amount >= localDustLimit, "amount lesser than dust limit")
     MainPenaltyTx(input, Transaction(
       version = 2,
       txIn = TxIn(input.outPoint, Array.emptyByteArray, 0xffffffffL) :: Nil,
-      txOut = TxOut(input.txOut.amount - fee, localFinalScriptPubKey) :: Nil,
+      txOut = TxOut(amount, localFinalScriptPubKey) :: Nil,
       lockTime = 0))
   }
 
-  def makeHtlcPenaltyTx(commitTx: Transaction): HtlcPenaltyTx = ???
-
-  /**
-    * This generates a partial transaction that will be completed by bitcoind using a 'fundrawtransaction' rpc call.
-    * Since bitcoind may add a change output, we return the pubkeyScript so that we can do a lookup afterwards.
-    *
-    * @param amount
-    * @param localFundingPubkey
-    * @param remoteFundingPubkey
-    * @return (partialTx, pubkeyScript)
-    */
-  def makePartialFundingTx(amount: Satoshi, localFundingPubkey: PublicKey, remoteFundingPubkey: PublicKey): (Transaction, BinaryData) = {
-    val pubkeyScript = write(pay2wsh(Scripts.multiSig2of2(localFundingPubkey, remoteFundingPubkey)))
-    (Transaction(
-      version = 2,
-      txIn = Seq.empty[TxIn],
-      txOut = TxOut(amount, pubkeyScript) :: Nil,
-      lockTime = 0), pubkeyScript)
-  }
+  def makeHtlcPenaltyTx(commitTx: Transaction, localDustLimit: Satoshi): HtlcPenaltyTx = ???
 
   def makeClosingTx(commitTxInput: InputInfo, localScriptPubKey: BinaryData, remoteScriptPubKey: BinaryData, localIsFunder: Boolean, dustLimit: Satoshi, closingFee: Satoshi, spec: CommitmentSpec): ClosingTx = {
     require(spec.htlcs.size == 0, "there shouldn't be any pending htlcs")
@@ -366,8 +362,8 @@ object Transactions {
     claimHtlcTimeoutTx.copy(tx = claimHtlcTimeoutTx.tx.updateWitness(0, witness))
   }
 
-  def addSigs(claimP2WPKHOutputTx: ClaimP2WPKHOutputTx, localPubkey: BinaryData, localSig: BinaryData): ClaimP2WPKHOutputTx = {
-    val witness = ScriptWitness(Seq(localSig, localPubkey))
+  def addSigs(claimP2WPKHOutputTx: ClaimP2WPKHOutputTx, localPaymentPubkey: BinaryData, localSig: BinaryData): ClaimP2WPKHOutputTx = {
+    val witness = ScriptWitness(Seq(localSig, localPaymentPubkey))
     claimP2WPKHOutputTx.copy(tx = claimP2WPKHOutputTx.tx.updateWitness(0, witness))
   }
 
@@ -381,6 +377,9 @@ object Transactions {
     closingTx.copy(tx = closingTx.tx.updateWitness(0, witness))
   }
 
+  def checkSpendable(parent: Transaction, child: Transaction): Try[Unit] =
+    Try(Transaction.correctlySpends(child, parent :: Nil, ScriptFlags.STANDARD_SCRIPT_VERIFY_FLAGS))
+
   def checkSpendable(txinfo: TransactionWithInputInfo): Try[Unit] =
     Try(Transaction.correctlySpends(txinfo.tx, Map(txinfo.tx.txIn(0).outPoint -> txinfo.input.txOut), ScriptFlags.STANDARD_SCRIPT_VERIFY_FLAGS))
 

eclair-core/src/main/scala/fr/acinq/eclair/wire/ChannelCodecs.scala

@@ -0,0 +1,235 @@
+package fr.acinq.eclair.wire
+
+import fr.acinq.bitcoin.{BinaryData, OutPoint, Transaction, TxOut}
+import fr.acinq.eclair.channel._
+import fr.acinq.eclair.crypto.ShaChain
+import fr.acinq.eclair.payment.{Local, Origin, Relayed}
+import fr.acinq.eclair.transactions.Transactions._
+import fr.acinq.eclair.transactions._
+import fr.acinq.eclair.wire.LightningMessageCodecs._
+import scodec.bits.{BitVector, ByteVector}
+import scodec.codecs._
+import scodec.{Attempt, Codec}
+
+/**
+  * Created by PM on 02/06/2017.
+  */
+object ChannelCodecs {
+
+  val localParamsCodec: Codec[LocalParams] = (
+    ("nodeId" | publicKey) ::
+      ("dustLimitSatoshis" | uint64) ::
+      ("maxHtlcValueInFlightMsat" | uint64ex) ::
+      ("channelReserveSatoshis" | uint64) ::
+      ("htlcMinimumMsat" | uint64) ::
+      ("toSelfDelay" | uint16) ::
+      ("maxAcceptedHtlcs" | uint16) ::
+      ("fundingPrivKey" | privateKey) ::
+      ("revocationSecret" | scalar) ::
+      ("paymentKey" | scalar) ::
+      ("delayedPaymentKey" | scalar) ::
+      ("htlcKey" | scalar) ::
+      ("defaultFinalScriptPubKey" | varsizebinarydata) ::
+      ("shaSeed" | varsizebinarydata) ::
+      ("isFunder" | bool) ::
+      ("globalFeatures" | varsizebinarydata) ::
+      ("localFeatures" | varsizebinarydata)).as[LocalParams]
+
+  val remoteParamsCodec: Codec[RemoteParams] = (
+    ("nodeId" | publicKey) ::
+      ("dustLimitSatoshis" | uint64) ::
+      ("maxHtlcValueInFlightMsat" | uint64ex) ::
+      ("channelReserveSatoshis" | uint64) ::
+      ("htlcMinimumMsat" | uint64) ::
+      ("toSelfDelay" | uint16) ::
+      ("maxAcceptedHtlcs" | uint16) ::
+      ("fundingPubKey" | publicKey) ::
+      ("revocationBasepoint" | point) ::
+      ("paymentBasepoint" | point) ::
+      ("delayedPaymentBasepoint" | point) ::
+      ("htlcBasepoint" | point) ::
+      ("globalFeatures" | varsizebinarydata) ::
+      ("localFeatures" | varsizebinarydata)).as[RemoteParams]
+
+  val directionCodec: Codec[Direction] = Codec[Direction](
+    (dir: Direction) => bool.encode(dir == IN),
+    (wire: BitVector) => bool.decode(wire).map(_.map(b => if (b) IN else OUT))
+  )
+
+  val htlcCodec: Codec[DirectedHtlc] = (
+    ("direction" | directionCodec) ::
+      ("add" | updateAddHtlcCodec)).as[DirectedHtlc]
+
+  def setCodec[T](codec: Codec[T]): Codec[Set[T]] = Codec[Set[T]](
+    (elems: Set[T]) => listOfN(uint16, codec).encode(elems.toList),
+    (wire: BitVector) => listOfN(uint16, codec).decode(wire).map(_.map(_.toSet))
+  )
+
+  val commitmentSpecCodec: Codec[CommitmentSpec] = (
+    ("htlcs" | setCodec(htlcCodec)) ::
+      ("feeratePerKw" | uint32) ::
+      ("toLocalMsat" | uint64) ::
+      ("toRemoteMsat" | uint64)).as[CommitmentSpec]
+
+  def outPointCodec: Codec[OutPoint] = variableSizeBytes(uint16, bytes.xmap(d => OutPoint.read(d.toArray), d => ByteVector(OutPoint.write(d).data)))
+
+  def txOutCodec: Codec[TxOut] = variableSizeBytes(uint16, bytes.xmap(d => TxOut.read(d.toArray), d => ByteVector(TxOut.write(d).data)))
+
+  def txCodec: Codec[Transaction] = variableSizeBytes(uint16, bytes.xmap(d => Transaction.read(d.toArray), d => ByteVector(Transaction.write(d).data)))
+
+  val inputInfoCodec: Codec[InputInfo] = (
+    ("outPoint" | outPointCodec) ::
+      ("txOut" | txOutCodec) ::
+      ("redeemScript" | varsizebinarydata)).as[InputInfo]
+
+  val txWithInputInfoCodec: Codec[TransactionWithInputInfo] = discriminated[TransactionWithInputInfo].by(uint16)
+    .typecase(0x01, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[CommitTx])
+    .typecase(0x02, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec) :: ("paymentHash" | binarydata(32))).as[HtlcSuccessTx])
+    .typecase(0x03, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[HtlcTimeoutTx])
+    .typecase(0x04, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[ClaimHtlcSuccessTx])
+    .typecase(0x05, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[ClaimHtlcTimeoutTx])
+    .typecase(0x06, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[ClaimP2WPKHOutputTx])
+    .typecase(0x07, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[ClaimDelayedOutputTx])
+    .typecase(0x08, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[MainPenaltyTx])
+    .typecase(0x09, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[HtlcPenaltyTx])
+    .typecase(0x10, (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[ClosingTx])
+
+  val htlcTxAndSigsCodec: Codec[HtlcTxAndSigs] = (
+    ("txinfo" | txWithInputInfoCodec) ::
+      ("localSig" | varsizebinarydata) ::
+      ("remoteSig" | varsizebinarydata)).as[HtlcTxAndSigs]
+
+  val publishableTxsCodec: Codec[PublishableTxs] = (
+    ("commitTx" | (("inputInfo" | inputInfoCodec) :: ("tx" | txCodec)).as[CommitTx]) ::
+      ("htlcTxsAndSigs" | listOfN(uint16, htlcTxAndSigsCodec))).as[PublishableTxs]
+
+  val localCommitCodec: Codec[LocalCommit] = (
+    ("index" | uint64) ::
+      ("spec" | commitmentSpecCodec) ::
+      ("publishableTxs" | publishableTxsCodec)).as[LocalCommit]
+
+  val remoteCommitCodec: Codec[RemoteCommit] = (
+    ("index" | uint64) ::
+      ("spec" | commitmentSpecCodec) ::
+      ("txid" | binarydata(32)) ::
+      ("remotePerCommitmentPoint" | point)).as[RemoteCommit]
+
+  val updateMessageCodec: Codec[UpdateMessage] = lightningMessageCodec.narrow(f => Attempt.successful(f.asInstanceOf[UpdateMessage]), g => g)
+
+  val localChangesCodec: Codec[LocalChanges] = (
+    ("proposed" | listOfN(uint16, updateMessageCodec)) ::
+      ("signed" | listOfN(uint16, updateMessageCodec)) ::
+      ("acked" | listOfN(uint16, updateMessageCodec))).as[LocalChanges]
+
+  val remoteChangesCodec: Codec[RemoteChanges] = (
+    ("proposed" | listOfN(uint16, updateMessageCodec)) ::
+      ("acked" | listOfN(uint16, updateMessageCodec)) ::
+      ("signed" | listOfN(uint16, updateMessageCodec))).as[RemoteChanges]
+
+  val waitingForRevocationCodec: Codec[WaitingForRevocation] = (
+    ("nextRemoteCommit" | remoteCommitCodec) ::
+      ("sent" | commitSigCodec) ::
+      ("sentAfterLocalCommitIndex" | uint64) ::
+      ("reSignAsap" | bool)).as[WaitingForRevocation]
+
+  val relayedCodec: Codec[Relayed] = (
+    ("originChannelId" | binarydata(32)) ::
+      ("originHtlcId" | int64) ::
+      ("amountMsatIn" | uint64) ::
+      ("amountMsatOut" | uint64)).as[Relayed]
+
+  val originCodec: Codec[Origin] = discriminated[Origin].by(uint16)
+    .typecase(0x01, provide(Local(None)))
+    .typecase(0x02, relayedCodec)
+
+  val originsListCodec: Codec[List[(Long, Origin)]] = listOfN(uint16, int64 ~ originCodec)
+
+  val originsMapCodec: Codec[Map[Long, Origin]] = Codec[Map[Long, Origin]](
+    (map: Map[Long, Origin]) => originsListCodec.encode(map.toList),
+    (wire: BitVector) => originsListCodec.decode(wire).map(_.map(_.toMap))
+  )
+
+  val commitmentsCodec: Codec[Commitments] = (
+    ("localParams" | localParamsCodec) ::
+      ("remoteParams" | remoteParamsCodec) ::
+      ("channelFlags" | byte) ::
+      ("localCommit" | localCommitCodec) ::
+      ("remoteCommit" | remoteCommitCodec) ::
+      ("localChanges" | localChangesCodec) ::
+      ("remoteChanges" | remoteChangesCodec) ::
+      ("localNextHtlcId" | uint64) ::
+      ("remoteNextHtlcId" | uint64) ::
+      ("originChannels" | originsMapCodec) ::
+      ("remoteNextCommitInfo" | either(bool, waitingForRevocationCodec, point)) ::
+      ("commitInput" | inputInfoCodec) ::
+      ("remotePerCommitmentSecrets" | ShaChain.shaChainCodec) ::
+      ("channelId" | binarydata(32))).as[Commitments]
+
+  val localCommitPublishedCodec: Codec[LocalCommitPublished] = (
+    ("commitTx" | txCodec) ::
+      ("claimMainDelayedOutputTx" | optional(bool, txCodec)) ::
+      ("htlcSuccessTxs" | listOfN(uint16, txCodec)) ::
+      ("htlcTimeoutTxs" | listOfN(uint16, txCodec)) ::
+      ("claimHtlcDelayedTx" | listOfN(uint16, txCodec)) ::
+      ("spent" | provide(Map.empty[OutPoint, BinaryData]))).as[LocalCommitPublished]
+
+  val remoteCommitPublishedCodec: Codec[RemoteCommitPublished] = (
+    ("commitTx" | txCodec) ::
+      ("claimMainOutputTx" | optional(bool, txCodec)) ::
+      ("claimHtlcSuccessTxs" | listOfN(uint16, txCodec)) ::
+      ("claimHtlcTimeoutTxs" | listOfN(uint16, txCodec)) ::
+      ("spent" | provide(Map.empty[OutPoint, BinaryData]))).as[RemoteCommitPublished]
+
+  val revokedCommitPublishedCodec: Codec[RevokedCommitPublished] = (
+    ("commitTx" | txCodec) ::
+      ("claimMainOutputTx" | optional(bool, txCodec)) ::
+      ("mainPenaltyTx" | optional(bool, txCodec)) ::
+      ("claimHtlcTimeoutTxs" | listOfN(uint16, txCodec)) ::
+      ("htlcTimeoutTxs" | listOfN(uint16, txCodec)) ::
+      ("htlcPenaltyTxs" | listOfN(uint16, txCodec)) ::
+      ("spent" | provide(Map.empty[OutPoint, BinaryData]))).as[RevokedCommitPublished]
+
+  val DATA_WAIT_FOR_FUNDING_CONFIRMED_Codec: Codec[DATA_WAIT_FOR_FUNDING_CONFIRMED] = (
+    ("commitments" | commitmentsCodec) ::
+      ("deferred" | optional(bool, fundingLockedCodec)) ::
+      ("lastSent" | either(bool, fundingCreatedCodec, fundingSignedCodec))).as[DATA_WAIT_FOR_FUNDING_CONFIRMED]
+
+  val DATA_WAIT_FOR_FUNDING_LOCKED_Codec: Codec[DATA_WAIT_FOR_FUNDING_LOCKED] = (
+    ("commitments" | commitmentsCodec) ::
+      ("lastSent" | fundingLockedCodec)).as[DATA_WAIT_FOR_FUNDING_LOCKED]
+
+  val DATA_NORMAL_Codec: Codec[DATA_NORMAL] = (
+    ("commitments" | commitmentsCodec) ::
+      ("shortChannelId" | optional(bool, uint64)) ::
+      ("localAnnouncementSignatures" | optional(bool, announcementSignaturesCodec)) ::
+      ("localShutdown" | optional(bool, shutdownCodec)) ::
+      ("remoteShutdown" | optional(bool, shutdownCodec))).as[DATA_NORMAL]
+
+  val DATA_SHUTDOWN_Codec: Codec[DATA_SHUTDOWN] = (
+    ("commitments" | commitmentsCodec) ::
+      ("localShutdown" | shutdownCodec) ::
+      ("remoteShutdown" | shutdownCodec)).as[DATA_SHUTDOWN]
+
+  val DATA_NEGOTIATING_Codec: Codec[DATA_NEGOTIATING] = (
+    ("commitments" | commitmentsCodec) ::
+      ("localShutdown" | shutdownCodec) ::
+      ("remoteShutdown" | shutdownCodec) ::
+      ("localClosingSigned" | closingSignedCodec)).as[DATA_NEGOTIATING]
+
+  val DATA_CLOSING_Codec: Codec[DATA_CLOSING] = (
+    ("commitments" | commitmentsCodec) ::
+      ("mutualClosePublished" | optional(bool, txCodec)) ::
+      ("localCommitPublished" | optional(bool, localCommitPublishedCodec)) ::
+      ("remoteCommitPublished" | optional(bool, remoteCommitPublishedCodec)) ::
+      ("nextRemoteCommitPublished" | optional(bool, remoteCommitPublishedCodec)) ::
+      ("revokedCommitPublished" | listOfN(uint16, revokedCommitPublishedCodec))).as[DATA_CLOSING]
+
+  val stateDataCodec: Codec[HasCommitments] = ("version" | constant(0x00)) ~> discriminated[HasCommitments].by(uint16)
+    .typecase(0x01, DATA_WAIT_FOR_FUNDING_CONFIRMED_Codec)
+    .typecase(0x02, DATA_WAIT_FOR_FUNDING_LOCKED_Codec)
+    .typecase(0x03, DATA_NORMAL_Codec)
+    .typecase(0x04, DATA_SHUTDOWN_Codec)
+    .typecase(0x05, DATA_NEGOTIATING_Codec)
+    .typecase(0x06, DATA_CLOSING_Codec)
+
+}

eclair-core/src/main/scala/fr/acinq/eclair/wire/FailureMessage.scala

@@ -1,7 +1,7 @@
 package fr.acinq.eclair.wire
 
 import fr.acinq.bitcoin.BinaryData
-import fr.acinq.eclair.wire.LightningMessageCodecs.{binarydata, channelUpdateCodec}
+import fr.acinq.eclair.wire.LightningMessageCodecs.{binarydata, channelUpdateCodec, uint64}
 import scodec.Codec
 import scodec.codecs._
 
@@ -61,8 +61,8 @@ object FailureMessageCodecs {
     .typecase(PERM | 8, provide(PermanentChannelFailure))
     .typecase(PERM | 9, provide(RequiredChannelFeatureMissing))
     .typecase(PERM | 10, provide(UnknownNextPeer))
-    .typecase(UPDATE | 11, (("amountMsat" | uint32) :: ("channelUpdate" | channelUpdateCodec)).as[AmountBelowMinimum])
-    .typecase(UPDATE | 12, (("amountMsat" | uint32) :: ("channelUpdate" | channelUpdateCodec)).as[FeeInsufficient])
+    .typecase(UPDATE | 11, (("amountMsat" | uint64) :: ("channelUpdate" | channelUpdateCodec)).as[AmountBelowMinimum])
+    .typecase(UPDATE | 12, (("amountMsat" | uint64) :: ("channelUpdate" | channelUpdateCodec)).as[FeeInsufficient])
     .typecase(UPDATE | 13, (("expiry" | uint32) :: ("channelUpdate" | channelUpdateCodec)).as[IncorrectCltvExpiry])
     .typecase(UPDATE | 14, (("channelUpdate" | channelUpdateCodec)).as[ExpiryTooSoon])
     .typecase(UPDATE | 20, (("flags" | binarydata(2)) :: ("channelUpdate" | channelUpdateCodec)).as[ChannelDisabled])

eclair-core/src/main/scala/fr/acinq/eclair/wire/FixedSizeStrictCodec.scala

@@ -0,0 +1,59 @@
+package fr.acinq.eclair.wire
+
+import scodec.bits.{BitVector, ByteVector}
+import scodec.{Attempt, Codec, DecodeResult, Err, SizeBound, codecs}
+
+/**
+  *
+  * REMOVE THIS A NEW VERSION OF SCODEC IS RELEASED THAT INCLUDES CHANGES MADE IN
+  * https://github.com/scodec/scodec/pull/99/files
+  *
+  * Created by PM on 02/06/2017.
+  */
+final class FixedSizeStrictCodec[A](size: Long, codec: Codec[A]) extends Codec[A] {
+
+  override def sizeBound = SizeBound.exact(size)
+
+  override def encode(a: A) = for {
+    encoded <- codec.encode(a)
+    result <- {
+      if (encoded.size != size)
+        Attempt.failure(Err(s"[$a] requires ${encoded.size} bits but field is fixed size of exactly $size bits"))
+      else
+        Attempt.successful(encoded.padTo(size))
+    }
+  } yield result
+
+  override def decode(buffer: BitVector) = {
+    if (buffer.size == size) {
+      codec.decode(buffer.take(size)) map { res =>
+        DecodeResult(res.value, buffer.drop(size))
+      }
+    } else {
+      Attempt.failure(Err(s"expected exactly $size bits but got ${buffer.size} bits"))
+    }
+  }
+
+  override def toString = s"fixedSizeBitsStrict($size, $codec)"
+}
+
+object FixedSizeStrictCodec {
+  /**
+    * Encodes by returning the supplied byte vector if its length is `size` bytes, otherwise returning error;
+    * decodes by taking `size * 8` bits from the supplied bit vector and converting to a byte vector.
+    *
+    * @param size number of bits to encode/decode
+    * @group bits
+    */
+  def bytesStrict(size: Int): Codec[ByteVector] = new Codec[ByteVector] {
+    private val codec = new FixedSizeStrictCodec(size * 8L, codecs.bits).xmap[ByteVector](_.toByteVector, _.toBitVector)
+
+    def sizeBound = codec.sizeBound
+
+    def encode(b: ByteVector) = codec.encode(b)
+
+    def decode(b: BitVector) = codec.decode(b)
+
+    override def toString = s"bytesStrict($size)"
+  }
+}

eclair-core/src/main/scala/fr/acinq/eclair/wire/LightningMessageCodecs.scala

@@ -3,10 +3,11 @@ package fr.acinq.eclair.wire
 import java.math.BigInteger
 import java.net.{Inet4Address, Inet6Address, InetAddress, InetSocketAddress}
 
-import fr.acinq.bitcoin.Crypto.{Point, PublicKey, Scalar}
+import fr.acinq.bitcoin.Crypto.{Point, PrivateKey, PublicKey, Scalar}
 import fr.acinq.bitcoin.{BinaryData, Crypto}
 import fr.acinq.eclair.crypto.{Generators, Sphinx}
-import fr.acinq.eclair.wire
+import fr.acinq.eclair.wire.FixedSizeStrictCodec.bytesStrict
+import fr.acinq.eclair.{UInt64, wire}
 import scodec.bits.{BitVector, ByteVector}
 import scodec.codecs._
 import scodec.{Attempt, Codec, Err}
@@ -21,7 +22,9 @@ object LightningMessageCodecs {
   // (for something smarter see https://github.com/yzernik/bitcoin-scodec/blob/master/src/main/scala/io/github/yzernik/bitcoinscodec/structures/UInt64.scala)
   val uint64: Codec[Long] = int64.narrow(l => if (l >= 0) Attempt.Successful(l) else Attempt.failure(Err(s"overflow for value $l")), l => l)
 
-  def binarydata(size: Int): Codec[BinaryData] = bytes(size).xmap(d => BinaryData(d.toArray), d => ByteVector(d.data))
+  val uint64ex: Codec[UInt64] = bytes(8).xmap(b => UInt64(b.toArray), a => ByteVector(a.underlying.toByteArray).takeRight(8).padLeft(8))
+
+  def binarydata(size: Int): Codec[BinaryData] = limitedSizeBytes(size, bytesStrict(size).xmap(d => BinaryData(d.toArray), d => ByteVector(d.data)))
 
   def varsizebinarydata: Codec[BinaryData] = variableSizeBytes(uint16, bytes.xmap(d => BinaryData(d.toArray), d => ByteVector(d.data)))
 
@@ -37,7 +40,10 @@ object LightningMessageCodecs {
       .typecase(2, ipv6address) ~ uint16)
       .xmap(x => new InetSocketAddress(x._1, x._2), x => (x.getAddress, x.getPort))
 
-  def listofsocketaddresses: Codec[List[InetSocketAddress]] = listOfN(uint16, socketaddress)
+  // this one is a bit different from most other codecs: the first 'len' element is * not * the number of items
+  // in the list but rather the  number of bytes of the encoded list. The rationale is once we've read this
+  // number of bytes we can just skip to the next field
+  def listofsocketaddresses: Codec[List[InetSocketAddress]] = variableSizeBytes(uint16, list(socketaddress))
 
   def signature: Codec[BinaryData] = Codec[BinaryData](
     (der: BinaryData) => bytes(64).encode(ByteVector(der2wire(der).toArray)),
@@ -50,12 +56,17 @@ object LightningMessageCodecs {
   )
 
   def point: Codec[Point] = Codec[Point](
-    (point: Point) => bytes(33).encode(ByteVector(point.toBin(true).toArray)),
+    (point: Point) => bytes(33).encode(ByteVector(point.toBin(compressed = true).toArray)),
     (wire: BitVector) => bytes(33).decode(wire).map(_.map(b => Point(b.toArray)))
   )
 
+  def privateKey: Codec[PrivateKey] = Codec[PrivateKey](
+    (priv: PrivateKey) => bytes(32).encode(ByteVector(priv.value.toBin.toArray)),
+    (wire: BitVector) => bytes(32).decode(wire).map(_.map(b => PrivateKey(b.toArray, compressed = true)))
+  )
+
   def publicKey: Codec[PublicKey] = Codec[PublicKey](
-    (pub: PublicKey) => bytes(33).encode(ByteVector(pub.value.toBin(true).toArray)),
+    (pub: PublicKey) => bytes(33).encode(ByteVector(pub.value.toBin(compressed = true).toArray)),
     (wire: BitVector) => bytes(33).decode(wire).map(_.map(b => PublicKey(b.toArray)))
   )
 
@@ -102,15 +113,20 @@ object LightningMessageCodecs {
   val pongCodec: Codec[Pong] =
     ("data" | varsizebinarydata).as[Pong]
 
+  val channelReestablishCodec: Codec[ChannelReestablish] = (
+    ("channelId" | binarydata(32)) ::
+      ("nextLocalCommitmentNumber" | uint64) ::
+      ("nextRemoteRevocationNumber" | uint64)).as[ChannelReestablish]
+
   val openChannelCodec: Codec[OpenChannel] = (
     ("chainHash" | binarydata(32)) ::
       ("temporaryChannelId" | binarydata(32)) ::
       ("fundingSatoshis" | uint64) ::
       ("pushMsat" | uint64) ::
       ("dustLimitSatoshis" | uint64) ::
-      ("maxHtlcValueInFlightMsat" | uint64) ::
+      ("maxHtlcValueInFlightMsat" | uint64ex) ::
       ("channelReserveSatoshis" | uint64) ::
-      ("htlcMinimumMsat" | uint32) ::
+      ("htlcMinimumMsat" | uint64) ::
       ("feeratePerKw" | uint32) ::
       ("toSelfDelay" | uint16) ::
       ("maxAcceptedHtlcs" | uint16) ::
@@ -118,21 +134,24 @@ object LightningMessageCodecs {
       ("revocationBasepoint" | point) ::
       ("paymentBasepoint" | point) ::
       ("delayedPaymentBasepoint" | point) ::
-      ("firstPerCommitmentPoint" | point)).as[OpenChannel]
+      ("htlcBasepoint" | point) ::
+      ("firstPerCommitmentPoint" | point) ::
+      ("channelFlags" | byte)).as[OpenChannel]
 
   val acceptChannelCodec: Codec[AcceptChannel] = (
     ("temporaryChannelId" | binarydata(32)) ::
       ("dustLimitSatoshis" | uint64) ::
-      ("maxHtlcValueInFlightMsat" | uint64) ::
+      ("maxHtlcValueInFlightMsat" | uint64ex) ::
       ("channelReserveSatoshis" | uint64) ::
+      ("htlcMinimumMsat" | uint64) ::
       ("minimumDepth" | uint32) ::
-      ("htlcMinimumMsat" | uint32) ::
       ("toSelfDelay" | uint16) ::
       ("maxAcceptedHtlcs" | uint16) ::
       ("fundingPubkey" | publicKey) ::
       ("revocationBasepoint" | point) ::
       ("paymentBasepoint" | point) ::
       ("delayedPaymentBasepoint" | point) ::
+      ("htlcBasepoint" | point) ::
       ("firstPerCommitmentPoint" | point)).as[AcceptChannel]
 
   val fundingCreatedCodec: Codec[FundingCreated] = (
@@ -161,9 +180,9 @@ object LightningMessageCodecs {
   val updateAddHtlcCodec: Codec[UpdateAddHtlc] = (
     ("channelId" | binarydata(32)) ::
       ("id" | uint64) ::
-      ("amountMsat" | uint32) ::
-      ("expiry" | uint32) ::
+      ("amountMsat" | uint64) ::
       ("paymentHash" | binarydata(32)) ::
+      ("expiry" | uint32) ::
       ("onionRoutingPacket" | binarydata(Sphinx.PacketLength))).as[UpdateAddHtlc]
 
   val updateFulfillHtlcCodec: Codec[UpdateFulfillHtlc] = (
@@ -204,12 +223,13 @@ object LightningMessageCodecs {
       ("bitcoinSignature" | signature)).as[AnnouncementSignatures]
 
   val channelAnnouncementWitnessCodec = (
-    ("shortChannelId" | int64) ::
-      ("nodeId1" | binarydata(33)) ::
-      ("nodeId2" | binarydata(33)) ::
-      ("bitcoinKey1" | binarydata(33)) ::
-      ("bitcoinKey2" | binarydata(33)) ::
-      ("features" | varsizebinarydata))
+    ("features" | varsizebinarydata) ::
+      ("chainHash" | binarydata(32)) ::
+      ("shortChannelId" | int64) ::
+      ("nodeId1" | publicKey) ::
+      ("nodeId2" | publicKey) ::
+      ("bitcoinKey1" | publicKey) ::
+      ("bitcoinKey2" | publicKey))
 
   val channelAnnouncementCodec: Codec[ChannelAnnouncement] = (
     ("nodeSignature1" | signature) ::
@@ -219,11 +239,11 @@ object LightningMessageCodecs {
       channelAnnouncementWitnessCodec).as[ChannelAnnouncement]
 
   val nodeAnnouncementWitnessCodec = (
-    ("timestamp" | uint32) ::
-      ("nodeId" | binarydata(33)) ::
+    ("features" | varsizebinarydata) ::
+      ("timestamp" | uint32) ::
+      ("nodeId" | publicKey) ::
       ("rgbColor" | rgb) ::
       ("alias" | zeropaddedstring(32)) ::
-      ("features" | varsizebinarydata) ::
       ("addresses" | listofsocketaddresses))
 
   val nodeAnnouncementCodec: Codec[NodeAnnouncement] = (
@@ -231,11 +251,12 @@ object LightningMessageCodecs {
       nodeAnnouncementWitnessCodec).as[NodeAnnouncement]
 
   val channelUpdateWitnessCodec = (
-    ("shortChannelId" | int64) ::
+    ("chainHash" | binarydata(32)) ::
+      ("shortChannelId" | int64) ::
       ("timestamp" | uint32) ::
       ("flags" | binarydata(2)) ::
       ("cltvExpiryDelta" | uint16) ::
-      ("htlcMinimumMsat" | uint32) ::
+      ("htlcMinimumMsat" | uint64) ::
       ("feeBaseMsat" | uint32) ::
       ("feeProportionalMillionths" | uint32))
 
@@ -263,6 +284,7 @@ object LightningMessageCodecs {
     .typecase(133, revokeAndAckCodec)
     .typecase(134, updateFeeCodec)
     .typecase(135, updateFailMalformedHtlcCodec)
+    .typecase(136, channelReestablishCodec)
     .typecase(256, channelAnnouncementCodec)
     .typecase(257, nodeAnnouncementCodec)
     .typecase(258, channelUpdateCodec)
@@ -271,8 +293,8 @@ object LightningMessageCodecs {
   val perHopPayloadCodec: Codec[PerHopPayload] = (
     ("realm" | constant(ByteVector.fromByte(0))) ::
       ("channel_id" | uint64) ::
-      ("amt_to_forward" | uint32) ::
+      ("amt_to_forward" | uint64) ::
       ("outgoing_cltv_value" | int32) :: // we use a signed int32, it is enough to store cltv for 40 000 years
-      ("unused_with_v0_version_on_header" | ignore(8 * 16))).as[PerHopPayload]
+      ("unused_with_v0_version_on_header" | ignore(8 * 12))).as[PerHopPayload]
 
 }

eclair-core/src/main/scala/fr/acinq/eclair/wire/LightningMessageTypes.scala

@@ -4,6 +4,7 @@ import java.net.InetSocketAddress
 
 import fr.acinq.bitcoin.BinaryData
 import fr.acinq.bitcoin.Crypto.{Point, PublicKey, Scalar}
+import fr.acinq.eclair.UInt64
 
 /**
   * Created by PM on 15/11/2016.
@@ -30,12 +31,17 @@ case class Ping(pongLength: Int, data: BinaryData) extends SetupMessage
 
 case class Pong(data: BinaryData) extends SetupMessage
 
+case class ChannelReestablish(
+                               channelId: BinaryData,
+                               nextLocalCommitmentNumber: Long,
+                               nextRemoteRevocationNumber: Long) extends ChannelMessage with HasChannelId
+
 case class OpenChannel(chainHash: BinaryData,
                        temporaryChannelId: BinaryData,
                        fundingSatoshis: Long,
                        pushMsat: Long,
                        dustLimitSatoshis: Long,
-                       maxHtlcValueInFlightMsat: Long,
+                       maxHtlcValueInFlightMsat: UInt64,
                        channelReserveSatoshis: Long,
                        htlcMinimumMsat: Long,
                        feeratePerKw: Long,
@@ -45,20 +51,23 @@ case class OpenChannel(chainHash: BinaryData,
                        revocationBasepoint: Point,
                        paymentBasepoint: Point,
                        delayedPaymentBasepoint: Point,
-                       firstPerCommitmentPoint: Point) extends ChannelMessage with HasTemporaryChannelId
+                       htlcBasepoint: Point,
+                       firstPerCommitmentPoint: Point,
+                       channelFlags: Byte) extends ChannelMessage with HasTemporaryChannelId
 
 case class AcceptChannel(temporaryChannelId: BinaryData,
                          dustLimitSatoshis: Long,
-                         maxHtlcValueInFlightMsat: Long,
+                         maxHtlcValueInFlightMsat: UInt64,
                          channelReserveSatoshis: Long,
-                         minimumDepth: Long,
                          htlcMinimumMsat: Long,
+                         minimumDepth: Long,
                          toSelfDelay: Int,
                          maxAcceptedHtlcs: Int,
                          fundingPubkey: PublicKey,
                          revocationBasepoint: Point,
                          paymentBasepoint: Point,
                          delayedPaymentBasepoint: Point,
+                         htlcBasepoint: Point,
                          firstPerCommitmentPoint: Point) extends ChannelMessage with HasTemporaryChannelId
 
 case class FundingCreated(temporaryChannelId: BinaryData,
@@ -82,8 +91,8 @@ case class ClosingSigned(channelId: BinaryData,
 case class UpdateAddHtlc(channelId: BinaryData,
                          id: Long,
                          amountMsat: Long,
-                         expiry: Long,
                          paymentHash: BinaryData,
+                         expiry: Long,
                          onionRoutingPacket: BinaryData) extends HtlcMessage with UpdateMessage with HasChannelId
 
 case class UpdateFulfillHtlc(channelId: BinaryData,
@@ -119,23 +128,25 @@ case class ChannelAnnouncement(nodeSignature1: BinaryData,
                                nodeSignature2: BinaryData,
                                bitcoinSignature1: BinaryData,
                                bitcoinSignature2: BinaryData,
+                               features: BinaryData,
+                               chainHash: BinaryData,
                                shortChannelId: Long,
-                               nodeId1: BinaryData,
-                               nodeId2: BinaryData,
-                               bitcoinKey1: BinaryData,
-                               bitcoinKey2: BinaryData,
-                               features: BinaryData) extends RoutingMessage
+                               nodeId1: PublicKey,
+                               nodeId2: PublicKey,
+                               bitcoinKey1: PublicKey,
+                               bitcoinKey2: PublicKey) extends RoutingMessage
 
 case class NodeAnnouncement(signature: BinaryData,
+                            features: BinaryData,
                             timestamp: Long,
-                            nodeId: BinaryData,
+                            nodeId: PublicKey,
                             rgbColor: (Byte, Byte, Byte),
                             alias: String,
-                            features: BinaryData,
                             // TODO: check address order + support padding data (type 0)
                             addresses: List[InetSocketAddress]) extends RoutingMessage
 
 case class ChannelUpdate(signature: BinaryData,
+                         chainHash: BinaryData,
                          shortChannelId: Long,
                          timestamp: Long,
                          flags: BinaryData,
@@ -145,5 +156,5 @@ case class ChannelUpdate(signature: BinaryData,
                          feeProportionalMillionths: Long) extends RoutingMessage
 
 case class PerHopPayload(channel_id: Long,
-                         amt_to_forward: Long,
-                         outgoing_cltv_value: Int)
+                         amtToForward: Long,
+                         outgoingCltvValue: Int)

eclair-core/src/test/java/fr/acinq/eclair/crypto/Curve25519.java

@@ -27,13 +27,13 @@ import java.util.Arrays;
 
 /**
  * Implementation of the Curve25519 elliptic curve algorithm.
- *
+ * <p>
  * This implementation is based on that from arduinolibs:
  * https://github.com/rweather/arduinolibs
- *
+ * <p>
  * Differences in this version are due to using 26-bit limbs for the
  * representation instead of the 8/16/32-bit limbs in the original.
- *
+ * <p>
  * References: http://cr.yp.to/ecdh.html, RFC 7748
  */
 public final class Curve25519 {
@@ -61,25 +61,24 @@ public final class Curve25519 {
     /**
      * Constructs the temporary state holder for Curve25519 evaluation.
      */
-    private Curve25519()
-    {
+    private Curve25519() {
         // Allocate memory for all of the temporary variables we will need.
-        x_1 = new int [NUM_LIMBS_255BIT];
-        x_2 = new int [NUM_LIMBS_255BIT];
-        x_3 = new int [NUM_LIMBS_255BIT];
-        z_2 = new int [NUM_LIMBS_255BIT];
-        z_3 = new int [NUM_LIMBS_255BIT];
-        A = new int [NUM_LIMBS_255BIT];
-        B = new int [NUM_LIMBS_255BIT];
-        C = new int [NUM_LIMBS_255BIT];
-        D = new int [NUM_LIMBS_255BIT];
-        E = new int [NUM_LIMBS_255BIT];
-        AA = new int [NUM_LIMBS_255BIT];
-        BB = new int [NUM_LIMBS_255BIT];
-        DA = new int [NUM_LIMBS_255BIT];
-        CB = new int [NUM_LIMBS_255BIT];
-        t1 = new long [NUM_LIMBS_510BIT];
-        t2 = new int [NUM_LIMBS_510BIT];
+        x_1 = new int[NUM_LIMBS_255BIT];
+        x_2 = new int[NUM_LIMBS_255BIT];
+        x_3 = new int[NUM_LIMBS_255BIT];
+        z_2 = new int[NUM_LIMBS_255BIT];
+        z_3 = new int[NUM_LIMBS_255BIT];
+        A = new int[NUM_LIMBS_255BIT];
+        B = new int[NUM_LIMBS_255BIT];
+        C = new int[NUM_LIMBS_255BIT];
+        D = new int[NUM_LIMBS_255BIT];
+        E = new int[NUM_LIMBS_255BIT];
+        AA = new int[NUM_LIMBS_255BIT];
+        BB = new int[NUM_LIMBS_255BIT];
+        DA = new int[NUM_LIMBS_255BIT];
+        CB = new int[NUM_LIMBS_255BIT];
+        t1 = new long[NUM_LIMBS_510BIT];
+        t2 = new int[NUM_LIMBS_510BIT];
     }
 
 
@@ -102,8 +101,8 @@ public final class Curve25519 {
         Arrays.fill(BB, 0);
         Arrays.fill(DA, 0);
         Arrays.fill(CB, 0);
-        Arrays.fill(t1,  0L);
-        Arrays.fill(t2,  0);
+        Arrays.fill(t1, 0L);
+        Arrays.fill(t2, 0);
     }
 
     /**
@@ -112,8 +111,7 @@ public final class Curve25519 {
      *
      * @param x The number to reduce, and the result.
      */
-    private void reduceQuick(int[] x)
-    {
+    private void reduceQuick(int[] x) {
         int index, carry;
 
         // Perform a trial subtraction of (2^255 - 19) from "x" which is
@@ -142,12 +140,11 @@ public final class Curve25519 {
      * Reduce a number modulo 2^255 - 19.
      *
      * @param result The result.
-     * @param x The value to be reduced.  This array will be
-     * modified during the reduction.
-     * @param size The number of limbs in the high order half of x.
+     * @param x      The value to be reduced.  This array will be
+     *               modified during the reduction.
+     * @param size   The number of limbs in the high order half of x.
      */
-    private void reduce(int[] result, int[] x, int size)
-    {
+    private void reduce(int[] result, int[] x, int size) {
         int index, limb, carry;
 
         // Calculate (x mod 2^255) + ((x / 2^255) * 19) which will
@@ -198,11 +195,10 @@ public final class Curve25519 {
      * Multiplies two numbers modulo 2^255 - 19.
      *
      * @param result The result.
-     * @param x The first number to multiply.
-     * @param y The second number to multiply.
+     * @param x      The first number to multiply.
+     * @param y      The second number to multiply.
      */
-    private void mul(int[] result, int[] x, int[] y)
-    {
+    private void mul(int[] result, int[] x, int[] y) {
         int i, j;
 
         // Multiply the two numbers to create the intermediate result.
@@ -220,10 +216,10 @@ public final class Curve25519 {
 
         // Propagate carries and convert back into 26-bit words.
         v = t1[0];
-        t2[0] = ((int)v) & 0x03FFFFFF;
+        t2[0] = ((int) v) & 0x03FFFFFF;
         for (i = 1; i < NUM_LIMBS_510BIT; ++i) {
             v = (v >> 26) + t1[i];
-            t2[i] = ((int)v) & 0x03FFFFFF;
+            t2[i] = ((int) v) & 0x03FFFFFF;
         }
 
         // Reduce the result modulo 2^255 - 19.
@@ -234,10 +230,9 @@ public final class Curve25519 {
      * Squares a number modulo 2^255 - 19.
      *
      * @param result The result.
-     * @param x The number to square.
+     * @param x      The number to square.
      */
-    private void square(int[] result, int[] x)
-    {
+    private void square(int[] result, int[] x) {
         mul(result, x, x);
     }
 
@@ -245,19 +240,18 @@ public final class Curve25519 {
      * Multiplies a number by the a24 constant, modulo 2^255 - 19.
      *
      * @param result The result.
-     * @param x The number to multiply by a24.
+     * @param x      The number to multiply by a24.
      */
-    private void mulA24(int[] result, int[] x)
-    {
+    private void mulA24(int[] result, int[] x) {
         long a24 = 121665;
         long carry = 0;
         int index;
         for (index = 0; index < NUM_LIMBS_255BIT; ++index) {
             carry += a24 * x[index];
-            t2[index] = ((int)carry) & 0x03FFFFFF;
+            t2[index] = ((int) carry) & 0x03FFFFFF;
             carry >>= 26;
         }
-        t2[NUM_LIMBS_255BIT] = ((int)carry) & 0x03FFFFFF;
+        t2[NUM_LIMBS_255BIT] = ((int) carry) & 0x03FFFFFF;
         reduce(result, t2, 1);
     }
 
@@ -265,11 +259,10 @@ public final class Curve25519 {
      * Adds two numbers modulo 2^255 - 19.
      *
      * @param result The result.
-     * @param x The first number to add.
-     * @param y The second number to add.
+     * @param x      The first number to add.
+     * @param y      The second number to add.
      */
-    private void add(int[] result, int[] x, int[] y)
-    {
+    private void add(int[] result, int[] x, int[] y) {
         int index, carry;
         carry = x[0] + y[0];
         result[0] = carry & 0x03FFFFFF;
@@ -284,11 +277,10 @@ public final class Curve25519 {
      * Subtracts two numbers modulo 2^255 - 19.
      *
      * @param result The result.
-     * @param x The first number to subtract.
-     * @param y The second number to subtract.
+     * @param x      The first number to subtract.
+     * @param y      The second number to subtract.
      */
-    private void sub(int[] result, int[] x, int[] y)
-    {
+    private void sub(int[] result, int[] x, int[] y) {
         int index, borrow;
 
         // Subtract y from x to generate the intermediate result.
@@ -316,11 +308,10 @@ public final class Curve25519 {
      * Conditional swap of two values.
      *
      * @param select Set to 1 to swap, 0 to leave as-is.
-     * @param x The first value.
-     * @param y The second value.
+     * @param x      The first value.
+     * @param y      The second value.
      */
-    private static void cswap(int select, int[] x, int[] y)
-    {
+    private static void cswap(int select, int[] x, int[] y) {
         int dummy;
         select = -select;
         for (int index = 0; index < NUM_LIMBS_255BIT; ++index) {
@@ -334,10 +325,9 @@ public final class Curve25519 {
      * Raise x to the power of (2^250 - 1).
      *
      * @param result The result.  Must not overlap with x.
-     * @param x The argument.
+     * @param x      The argument.
      */
-    private void pow250(int[] result, int[] x)
-    {
+    private void pow250(int[] result, int[] x) {
         int i, j;
 
         // The big-endian hexadecimal expansion of (2^250 - 1) is:
@@ -375,10 +365,9 @@ public final class Curve25519 {
      * Computes the reciprocal of a number modulo 2^255 - 19.
      *
      * @param result The result.  Must not overlap with x.
-     * @param x The argument.
+     * @param x      The argument.
      */
-    private void recip(int[] result, int[] x)
-    {
+    private void recip(int[] result, int[] x) {
         // The reciprocal is the same as x ^ (p - 2) where p = 2^255 - 19.
         // The big-endian hexadecimal expansion of (p - 2) is:
         // 7FFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFEB
@@ -401,8 +390,7 @@ public final class Curve25519 {
      *
      * @param s The 32-byte secret key.
      */
-    private void evalCurve(byte[] s)
-    {
+    private void evalCurve(byte[] s) {
         int sposn = 31;
         int sbit = 6;
         int svalue = s[sposn] | 0x40;
@@ -411,7 +399,7 @@ public final class Curve25519 {
 
         // Iterate over all 255 bits of "s" from the highest to the lowest.
         // We ignore the high bit of the 256-bit representation of "s".
-        for (;;) {
+        for (; ; ) {
             // Conditional swaps on entry to this bit but only if we
             // didn't swap on the previous bit.
             select = (svalue >> sbit) & 0x01;
@@ -464,14 +452,13 @@ public final class Curve25519 {
     /**
      * Evaluates the Curve25519 curve.
      *
-     * @param result Buffer to place the result of the evaluation into.
-     * @param offset Offset into the result buffer.
+     * @param result     Buffer to place the result of the evaluation into.
+     * @param offset     Offset into the result buffer.
      * @param privateKey The private key to use in the evaluation.
-     * @param publicKey The public key to use in the evaluation, or null
-     * if the base point of the curve should be used.
+     * @param publicKey  The public key to use in the evaluation, or null
+     *                   if the base point of the curve should be used.
      */
-    public static void eval(byte[] result, int offset, byte[] privateKey, byte[] publicKey)
-    {
+    public static void eval(byte[] result, int offset, byte[] privateKey, byte[] publicKey) {
         Curve25519 state = new Curve25519();
         try {
             // Unpack the public key value.  If null, use 9 as the base point.
@@ -501,11 +488,11 @@ public final class Curve25519 {
             }
 
             // Initialize the other temporary variables.
-            Arrays.fill(state.x_2, 0);			// x_2 = 1
+            Arrays.fill(state.x_2, 0);            // x_2 = 1
             state.x_2[0] = 1;
-            Arrays.fill(state.z_2, 0);			// z_2 = 0
+            Arrays.fill(state.z_2, 0);            // z_2 = 0
             System.arraycopy(state.x_1, 0, state.x_3, 0, state.x_1.length);  // x_3 = x_1
-            Arrays.fill(state.z_3, 0);			// z_3 = 1
+            Arrays.fill(state.z_3, 0);            // z_3 = 1
             state.z_3[0] = 1;
 
             // Evaluate the curve for every bit of the private key.
@@ -520,9 +507,9 @@ public final class Curve25519 {
                 int bit = (index * 8) % 26;
                 int word = (index * 8) / 26;
                 if (bit <= (26 - 8))
-                    result[offset + index] = (byte)(state.x_2[word] >> bit);
+                    result[offset + index] = (byte) (state.x_2[word] >> bit);
                 else
-                    result[offset + index] = (byte)((state.x_2[word] >> bit) | (state.x_2[word + 1] << (26 - bit)));
+                    result[offset + index] = (byte) ((state.x_2[word] >> bit) | (state.x_2[word + 1] << (26 - bit)));
             }
         } finally {
             // Clean up all temporary state before we exit.

eclair-core/src/test/resources/logback-test.xml

@@ -26,8 +26,7 @@
         </encoder>
     </appender-->
 
-    <logger name="fr.acinq.eclair.Pipe" level="DEBUG" />
-    <logger name="fr.acinq.eclair.crypto.TransportHandler" level="DEBUG" />
+    <!--logger name="fr.acinq.eclair.channel" level="DEBUG"/-->
 
     <root level="INFO">
         <!--appender-ref ref="FILE"/>

eclair-core/src/test/scala/fr/acinq/eclair/FeaturesSpec.scala

@@ -1,5 +1,8 @@
 package fr.acinq.eclair
 
+import java.nio.ByteOrder
+
+import fr.acinq.bitcoin.Protocol
 import fr.acinq.eclair.Features._
 import org.junit.runner.RunWith
 import org.scalatest.FunSuite
@@ -11,25 +14,15 @@ import org.scalatest.junit.JUnitRunner
 @RunWith(classOf[JUnitRunner])
 class FeaturesSpec extends FunSuite {
 
-  test("'channel_public' feature") {
-    assert(isSet("", CHANNELS_PUBLIC_BIT) == false)
-    assert(isSet("00", CHANNELS_PUBLIC_BIT) == false)
-    assert(isSet("01", CHANNELS_PUBLIC_BIT) == true)
-    assert(isSet("a602", CHANNELS_PUBLIC_BIT) == false)
-  }
-
   test("'initial_routing_sync' feature") {
-    assert(isSet("", INITIAL_ROUTING_SYNC_BIT) == false)
-    assert(isSet("00", INITIAL_ROUTING_SYNC_BIT) == false)
-    assert(isSet("04", INITIAL_ROUTING_SYNC_BIT) == true)
-    assert(isSet("05", INITIAL_ROUTING_SYNC_BIT) == true)
+    assert(initialRoutingSync("08"))
   }
 
   test("features compatibility") {
-    for (i <- 0 until 16) assert(areSupported(Array[Byte](i.toByte)) == true)
+    assert(!areSupported(Protocol.writeUInt64(1L << INITIAL_ROUTING_SYNC_BIT_MANDATORY, ByteOrder.BIG_ENDIAN)))
+    assert(areSupported(Protocol.writeUInt64(1l << INITIAL_ROUTING_SYNC_BIT_OPTIONAL, ByteOrder.BIG_ENDIAN)))
     assert(areSupported("14") == false)
     assert(areSupported("0141") == false)
-    assert(areSupported("02af") == true)
   }
 
 }

eclair-core/src/test/scala/fr/acinq/eclair/Pipe.scala

@@ -2,7 +2,6 @@ package fr.acinq.eclair
 
 import akka.actor.{Actor, ActorLogging, ActorRef, Stash}
 import fr.acinq.eclair.channel.Commitments.msg2String
-import fr.acinq.eclair.channel.{INPUT_DISCONNECTED, INPUT_RECONNECTED}
 import fr.acinq.eclair.wire.LightningMessage
 
 /**
@@ -18,7 +17,7 @@ class Pipe extends Actor with Stash with ActorLogging {
       unstashAll()
       context become connected(a, b)
 
-    case msg => stash()
+    case _ => stash()
   }
 
   def connected(a: ActorRef, b: ActorRef): Receive = {
@@ -28,27 +27,5 @@ class Pipe extends Actor with Stash with ActorLogging {
     case msg: LightningMessage if sender() == b =>
       log.debug(f"A <--${msg2String(msg)}%-6s--- B")
       a forward msg
-    case msg@INPUT_DISCONNECTED =>
-      log.debug("DISCONNECTED")
-      // used for fuzzy testing (eg: send Disconnected messages)
-      a forward msg
-      b forward msg
-      context become disconnected(a, b)
-  }
-
-  def disconnected(a: ActorRef, b: ActorRef): Receive = {
-    case msg: LightningMessage if sender() == a =>
-      // dropped
-      log.info(f"A ---${msg2String(msg)}%-6s-X")
-    case msg: LightningMessage if sender() == b =>
-      // dropped
-      log.debug(f"  X-${msg2String(msg)}%-6s--- B")
-    case msg@INPUT_RECONNECTED(r) =>
-      log.debug(s"RECONNECTED with $r")
-      // used for fuzzy testing (eg: send Disconnected messages)
-      a forward msg
-      b forward msg
-      r ! (a, b)
-
   }
 }

eclair-core/src/test/scala/fr/acinq/eclair/TestBitcoinClient.scala

@@ -1,16 +1,12 @@
 package fr.acinq.eclair
 
 import akka.actor.ActorSystem
-import fr.acinq.bitcoin.Crypto.{PrivateKey, PublicKey}
-import fr.acinq.bitcoin.{BinaryData, Block, Crypto, OP_PUSHDATA, OutPoint, Satoshi, Script, Transaction, TxIn, TxOut}
-import fr.acinq.eclair.blockchain.ExtendedBitcoinClient.SignTransactionResponse
-import fr.acinq.eclair.blockchain.rpc.BitcoinJsonRPCClient
+import fr.acinq.bitcoin.{Block, Transaction}
 import fr.acinq.eclair.blockchain._
-import fr.acinq.eclair.transactions.Scripts
+import fr.acinq.eclair.blockchain.bitcoind.rpc.{BitcoinJsonRPCClient, ExtendedBitcoinClient}
 
 import scala.concurrent.duration._
 import scala.concurrent.{ExecutionContext, Future}
-import scala.util.Try
 
 /**
   * Created by PM on 26/04/2016.
@@ -25,9 +21,6 @@ class TestBitcoinClient()(implicit system: ActorSystem) extends ExtendedBitcoinC
     override def run(): Unit = system.eventStream.publish(NewBlock(DUMMY_BLOCK)) // blocks are not actually interpreted
   })
 
-  override def makeFundingTx(ourCommitPub: PublicKey, theirCommitPub: PublicKey, amount: Satoshi, feeRatePerKw: Long)(implicit ec: ExecutionContext): Future[MakeFundingTxResponse] =
-    Future.successful(TestBitcoinClient.makeDummyFundingTx(MakeFundingTx(ourCommitPub, theirCommitPub, amount, feeRatePerKw)))
-
   override def publishTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[String] = {
     system.eventStream.publish(NewTransaction(tx))
     Future.successful(tx.txid.toString())
@@ -37,38 +30,6 @@ class TestBitcoinClient()(implicit system: ActorSystem) extends ExtendedBitcoinC
 
   override def getTransaction(txId: String)(implicit ec: ExecutionContext): Future[Transaction] = ???
 
-  override def fundTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[ExtendedBitcoinClient.FundTransactionResponse] = ???
-
-  override def signTransaction(tx: Transaction)(implicit ec: ExecutionContext): Future[SignTransactionResponse] = ???
-
   override def getTransactionShortId(txId: String)(implicit ec: ExecutionContext): Future[(Int, Int)] = Future.successful((400000, 42))
 
-}
-
-object TestBitcoinClient {
-
-  def makeDummyFundingTx(makeFundingTx: MakeFundingTx): MakeFundingTxResponse = {
-    val priv = PrivateKey(BinaryData("01" * 32), compressed = true)
-    val parentTx = Transaction(version = 2,
-      txIn = TxIn(OutPoint("42" * 32, 42), signatureScript = Nil, sequence = TxIn.SEQUENCE_FINAL) :: Nil,
-      txOut = TxOut(makeFundingTx.amount, Script.pay2sh(Script.pay2wpkh(priv.publicKey))) :: Nil,
-      lockTime = 0)
-    val anchorTx = Transaction(version = 2,
-      txIn = TxIn(OutPoint(parentTx, 0), signatureScript = Nil, sequence = TxIn.SEQUENCE_FINAL) :: Nil,
-      txOut = TxOut(makeFundingTx.amount, Script.pay2wsh(Scripts.multiSig2of2(makeFundingTx.localCommitPub, makeFundingTx.remoteCommitPub))) :: Nil,
-      lockTime = 0)
-    MakeFundingTxResponse(parentTx, anchorTx, 0, priv)
-  }
-
-  def malleateTx(tx: Transaction): Transaction = {
-    val inputs1 = tx.txIn.map(input => Script.parse(input.signatureScript) match {
-      case OP_PUSHDATA(sig, _) :: OP_PUSHDATA(pub, _) :: Nil if pub.length == 33 && Try(Crypto.decodeSignature(sig)).isSuccess =>
-        val (r, s) = Crypto.decodeSignature(sig)
-        val s1 = Crypto.curve.getN.subtract(s)
-        val sig1 = Crypto.encodeSignature(r, s1)
-        input.copy(signatureScript = Script.write(OP_PUSHDATA(sig1) :: OP_PUSHDATA(pub) :: Nil))
-    })
-    val tx1 = tx.copy(txIn = inputs1)
-    tx1
-  }
 }

eclair-core/src/test/scala/fr/acinq/eclair/TestConstants.scala

@@ -1,11 +1,14 @@
 package fr.acinq.eclair
 
 import java.net.InetSocketAddress
+import java.sql.DriverManager
 
 import fr.acinq.bitcoin.Crypto.PrivateKey
 import fr.acinq.bitcoin.{BinaryData, Block, DeterministicWallet, Script}
-import fr.acinq.eclair.db.{Dbs, DummyDb}
+import fr.acinq.eclair.NodeParams.BITCOIND
+import fr.acinq.eclair.db.sqlite.{SqliteChannelsDb, SqliteNetworkDb, SqlitePeersDb, SqlitePreimagesDb}
 import fr.acinq.eclair.io.Peer
+
 import scala.concurrent.duration._
 
 /**
@@ -20,17 +23,19 @@ object TestConstants {
     val seed = BinaryData("01" * 32)
     val master = DeterministicWallet.generate(seed)
     val extendedPrivateKey = DeterministicWallet.derivePrivateKey(master, DeterministicWallet.hardened(46) :: DeterministicWallet.hardened(0) :: Nil)
-    val db = new DummyDb()
-    val nodeParams = NodeParams(
+
+    def sqlite = DriverManager.getConnection("jdbc:sqlite::memory:")
+
+    def nodeParams = NodeParams(
       extendedPrivateKey = extendedPrivateKey,
       privateKey = extendedPrivateKey.privateKey,
       alias = "alice",
       color = (1: Byte, 2: Byte, 3: Byte),
-      address = new InetSocketAddress("localhost", 9731),
+      publicAddresses = new InetSocketAddress("localhost", 9731) :: Nil,
       globalFeatures = "",
-      localFeatures = "00", // no announcement
-      dustLimitSatoshis = 542,
-      maxHtlcValueInFlightMsat = 150000000,
+      localFeatures = "00",
+      dustLimitSatoshis = 500,
+      maxHtlcValueInFlightMsat = UInt64(150000000),
       maxAcceptedHtlcs = 100,
       expiryDeltaBlocks = 144,
       htlcMinimumMsat = 0,
@@ -41,20 +46,26 @@ object TestConstants {
       feeProportionalMillionth = 10,
       reserveToFundingRatio = 0.01, // note: not used (overriden below)
       maxReserveToFundingRatio = 0.05,
-      channelsDb = Dbs.makeChannelDb(db),
-      peersDb = Dbs.makePeerDb(db),
-      announcementsDb = Dbs.makeAnnouncementDb(db),
+      channelsDb = new SqliteChannelsDb(sqlite),
+      peersDb = new SqlitePeersDb(sqlite),
+      networkDb = new SqliteNetworkDb(sqlite),
+      preimagesDb = new SqlitePreimagesDb(sqlite),
       routerBroadcastInterval = 60 seconds,
       routerValidateInterval = 2 seconds,
       pingInterval = 30 seconds,
       maxFeerateMismatch = 1.5,
       updateFeeMinDiffRatio = 0.1,
       autoReconnect = false,
-      chainHash = Block.RegtestGenesisBlock.blockId)
-    val id = nodeParams.privateKey.publicKey
-    val channelParams = Peer.makeChannelParams(
+      chainHash = Block.RegtestGenesisBlock.hash,
+      channelFlags = 1,
+      channelExcludeDuration = 5 seconds,
+      watcherType = BITCOIND)
+
+    def id = nodeParams.privateKey.publicKey
+
+    def channelParams = Peer.makeChannelParams(
       nodeParams = nodeParams,
-      defaultFinalScriptPubKey = Script.write(Script.pay2wpkh(PrivateKey(Array.fill[Byte](32)(5), compressed = true).publicKey)),
+      defaultFinalScriptPubKey = Script.write(Script.pay2wpkh(PrivateKey(Array.fill[Byte](32)(4), compressed = true).publicKey)),
       isFunder = true,
       fundingSatoshis).copy(
       channelReserveSatoshis = 10000 // Bob will need to keep that much satoshis as direct payment
@@ -65,17 +76,19 @@ object TestConstants {
     val seed = BinaryData("02" * 32)
     val master = DeterministicWallet.generate(seed)
     val extendedPrivateKey = DeterministicWallet.derivePrivateKey(master, DeterministicWallet.hardened(46) :: DeterministicWallet.hardened(0) :: Nil)
-    val db = new DummyDb()
-    val nodeParams = NodeParams(
+
+    def sqlite = DriverManager.getConnection("jdbc:sqlite::memory:")
+
+    def nodeParams = NodeParams(
       extendedPrivateKey = extendedPrivateKey,
       privateKey = extendedPrivateKey.privateKey,
       alias = "bob",
       color = (4: Byte, 5: Byte, 6: Byte),
-      address = new InetSocketAddress("localhost", 9732),
+      publicAddresses = new InetSocketAddress("localhost", 9732) :: Nil,
       globalFeatures = "",
       localFeatures = "00", // no announcement
-      dustLimitSatoshis = 542,
-      maxHtlcValueInFlightMsat = Long.MaxValue, // Bob has no limit on the combined max value of in-flight htlcs
+      dustLimitSatoshis = 1000,
+      maxHtlcValueInFlightMsat = UInt64.MaxValue, // Bob has no limit on the combined max value of in-flight htlcs
       maxAcceptedHtlcs = 30,
       expiryDeltaBlocks = 144,
       htlcMinimumMsat = 1000,
@@ -86,20 +99,26 @@ object TestConstants {
       feeProportionalMillionth = 10,
       reserveToFundingRatio = 0.01, // note: not used (overriden below)
       maxReserveToFundingRatio = 0.05,
-      channelsDb = Dbs.makeChannelDb(db),
-      peersDb = Dbs.makePeerDb(db),
-      announcementsDb = Dbs.makeAnnouncementDb(db),
+      channelsDb = new SqliteChannelsDb(sqlite),
+      peersDb = new SqlitePeersDb(sqlite),
+      networkDb = new SqliteNetworkDb(sqlite),
+      preimagesDb = new SqlitePreimagesDb(sqlite),
       routerBroadcastInterval = 60 seconds,
       routerValidateInterval = 2 seconds,
       pingInterval = 30 seconds,
       maxFeerateMismatch = 1.0,
       updateFeeMinDiffRatio = 0.1,
       autoReconnect = false,
-      chainHash = Block.RegtestGenesisBlock.blockId)
-    val id = nodeParams.privateKey.publicKey
-    val channelParams = Peer.makeChannelParams(
+      chainHash = Block.RegtestGenesisBlock.hash,
+      channelFlags = 1,
+      channelExcludeDuration = 5 seconds,
+      watcherType = BITCOIND)
+
+    def id = nodeParams.privateKey.publicKey
+
+    def channelParams = Peer.makeChannelParams(
       nodeParams = nodeParams,
-      defaultFinalScriptPubKey = Script.write(Script.pay2wpkh(PrivateKey(Array.fill[Byte](32)(15), compressed = true).publicKey)),
+      defaultFinalScriptPubKey = Script.write(Script.pay2wpkh(PrivateKey(Array.fill[Byte](32)(5), compressed = true).publicKey)),
       isFunder = false,
       fundingSatoshis).copy(
       channelReserveSatoshis = 20000 // Alice will need to keep that much satoshis as direct payment