This marks the first major release in the v0.11.0 series!
Verifying the Release
In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:
curl https://keybase.io/bitconner/pgp_keys.asc | gpg --import
Once you have the required PGP keys, you can verify the release (assuming manifest-v0.11.0-beta.rc1.txt and manifest-v0.11.0-beta.rc1.txt.sig are in the current directory) with:
gpg --verify manifest-v0.11.0-beta.rc1.txt.sig
You should see the following if the verification was successful:
gpg: assuming signed data in 'manifest-v0.11.0-beta.rc1.txt'
gpg: Signature made Wed Jul 29 21:21:50 2020 PDT
gpg: using RSA key 9C8D61868A7C492003B2744EE7D737B67FA592C7
gpg: Good signature from "Conner Fromknecht <conner@lightning.engineering>" [ultimate]
That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.
Verifying the Release Binaries
Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.14.6, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.
The make release command can be used to ensure one rebuilds with all the same flags used for the release. If one wishes to build for only a single platform, then make release sys=<OS-ARCH> tag=<tag> can be used.
Finally, you can also verify the tag itself with the following command:
git verify-tag v0.11.0-beta.rc1
Building the Contained Release
Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.11.0-beta.rc1.tar.gz are in the current directory, follow these steps:
tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.11.0-beta.rc1.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.11.0-beta.rc1" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.11.0-beta.rc1" ./cmd/lncli
The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.
Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:
make release sys="linux-arm64 darwin-amd64"
⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️
Release Notes
WIP, release will be up shortly!
Wumbo Channel Support
This version of lnd supports accepting and creating wumbo channels! A wumbo channel is a channel whose total capcity is greater than 0.16 BTC (the existing channel limit). Activating wumbo channel features is opt in meaning users will need to set a name flag (--protocol.wumbo-channels) in order to use it. Most users can likely get by w/o wumbo channels, but larger nodes or exchanges/services may really benefit from the ability to manage a fewer set of larger channels.
As is, there's no limit w.r.t the size of a channel that a node will accept. However, nodes may want to limit the size of channel they'll accept in order to limt their exposure. This can be done by using a ChannelAcceptor call back to reject all in-bound funding requests greater than a certain value.
To coincide with this feature, the existing payment limits have been lifted in the payment related RPC calls. The max payment is now bounded by what the reciever can receive, and what the sender can route over using MPP.
New Autopilot Heuristic
In this version of lnd, we've added a new autopilot heuristic that is more robust and resistant to manipulation compared to the existing preferential attachment heuristic. This new heuristic is based on the concept betweenness centrality. The betweenness centrality metric is a essentially a measure of the number of shortest paths that pass through a particular node/vertex. The new heuristic uses this value rather than the relative number of channels a node has to make attachment (channel creation) decisions. This new heuristic should be more effective for both clients and routing nodes as as it's more indicative of a nodes potential as a routing node than simply the number of channels it has.
This new mode can be activated by passing a new argument: --autopilot.heuristic=top_centrality:1.0 to lnd. Note that it's possible to blend the old and new heuristics by specifying a another linear combination of heuristics with their total weight summing up to zero. The current heuristic still maintains the stochastic nature of the existing attachment heuristics. In later releases, we aim to improve on this, by doing a mini simulation to identify the best N new channels a node can add to maximize the gain of its betweenness centrality metrics. Once this is complete, then this metric will actually attempt to actively better position a potential routing node in the network, rather than try to connect to the node w/ the "best" value for a given metric.
Experimental Support for new Storage Backend: etcd
In this release of etcd, we've added experimental support for etcd as a
new storage backend for lnd. etcd differs from bbolt (the current default storage backend) in that it's a replicated database which can be run in a clustered manner. The main advantage over bbolt is that the replicated nature eliminates the current single point of failure that is channel.db. When activated, lnd will store data that can be reconstructed easily locally in bbolt (such as the channel graph), while critical data such as the current state of a channel will be replicated and stored in etcd.
As this mode is still experimental, in order to test out this new DB mode, users will need to compile lnd with the new kvdb_etcd build tag: make install tags=kvdb_etcd. Before running etcd, users should check out the new documentation for information concerning the recommended etcd configuration set up. Future releases will further optimize our usage of the new database backend, eventually adding support to run lnd in a clustered mode by by using etcd's leader election APIs.
Protocol/Spec Compliance
Sporadic Force Close Issue Resolution
This release fixes three distinct cases of erroneous force closes. All were related to the code that handles restarting channels, so nodes would regularly hit this if one of them flapped. Nodes should see increased stability across the network with this release.
ChannelAnnouncement Feature Bit Decoding
Short Channel ID Decoding
A bug has been fixed that would otherwise cuase lnd incorrectly decode a zero value short channel ID.
Support for Dynamic IPs and Domain Resolution for Node Addrs
In this version of lnd, a new flag has been added to allow users that either have a dynamic IP or want to advertise the resolved IP address of a domain name: --external-hosts. This new flag can be set to a series of external hosts. If this flag is set, then lnd will peridically attempt to resolve the target host, advetising the new IP address. We'll also keep track of the prior IPs reported by that domain name, and will send out a new advertisement with the most up to date IP addresses if we detect a change.
RPC Enhancement and Bug Fixes
New External Coin Selection API
In this release, we've added a new set of calls to the WalletKit sub-server that allows clients to perform custom coin selection. Along-side the existing SignRPC sub-server calls, this set of APIs allows clients to manually construct, sign and broadcast custom transactions. This may also be combined with the new PSBT APIs to carry out complex multi-party protocols.
Three new RPC calls have been added:
ListUnspent: a copy of the existingListUnspentRPC inLightningClient, which has been marked as deprecated in favor of using thewalletrpcone.LeaseOutput: lease an output for a predetermined amount of time bound to a unique ID to ensure no other applications can lease the same output.ReleaseOutput: if the lease for an output is no longer needed, it can be returned to the set of spendable UTXOs manually through this RPC, or it will be done automatically by the wallet once the lease expires.
New HTLC Interception API
This new release of lnd contains a new RPC API which is essentially the forwarded HTLC analgogue of the existing "hodl" invoice API. This new API allows for custom forward policies to be implemented. With the new API, for each forwarded HTLC, a call-back implemented as a streaming RPC will be called. With that call back, users can either signal they want the HTLC to immediately be settled, held, or instantly failed. This new API allows for a number of cool use cases such as notifying off-line clients of a new incoming payment (HTLC mailbox) and also opening a channel to a client in order to satisfy an incoming HTLC forwaard event.
This new API also allows user to examine any custom records which may have been dropped off to intermediate nodes by the sender. This intermedaite node custom records have no "official" protocl use, but some interesting meta-protocol are possible with its inclusion.
Hodl Keysend Payments
With this new version of lnd, we've added a new option to lnd to hold received keysend payments in the accepted state for a configured period of time. During this time, an application can inspect the payment parameters and decide whether to cancel or settle this payment.
Example: a keysend payment with an embedded order comes in. The payment is held and an external application checks that the paid amount is sufficient for the ordered goods. If not, the payment is canceled without the need to refund anything. If the amount is sufficient, the payment is settled and the order processed.
This PR makes development of applications like tlvshop.com possible without forking lnd.
Users can use the existing SettleInvoice call to manually settle inbound keysend payments if the --keysend-hold-time flag is set.
Updated Close Summary
The CloseChannelSummary returned by the ClosedChannels endpoint has been updated to include a list of on chain actions that were taken to resolve channels that were force closed. The on chain actions taken depend on the type of commitment the channel had, and the party that force closed.
- Commit sweep: Present if our node force closed the channel, represents sweeping of our own balance, which is behind a timelock for local force closes, back to our own wallet.
- Incoming HTLC: Incoming htlcs offer to pay your node funds in exchange for reveal of a preimage. If your node had knowledge of the preimage, these outputs will reflect as having been claimed, which means your node swept the funds to itself. If your node force closed, this claim will have two on chain resolutions, a success transaction followed by a sweep. Otherwise, the remote party will have timed out the htlc.
- Outgoing HTLC: Outgoing htlcs are offers that you make to pay a node in exchange for a preimage. If the remote party has knowledge of the preimage, they htlc will reflect as having been claimed, and the funds are swept by the remote party. If they do not provide the preimage before the timeout, our node will sweep the funds back to itself, and the htlc will reflect as having been timed out. If your node force closed, this timeout will have two on chain resolutions, a timeout transaction followed by a sweep.
- Anchor: Present if the channel was created using the experimental anchors commitment format. These outputs can either be swept back to the wallet by our node, or claimed by another party if we did not sweep it within 16 blocks.
New List Sweeps API
A new ListSweeps endpoint that provides a list of all the on chain transactions that lnd has published to sweep funds back to its own wallet has been added to the walletrpc subserver. By default, the endpoint returns a list of transaction IDs. The verbose option can be used to provide full transaction details, identical to those provided in the existing GetTransactions endpoint.
Height Parameters for GetTransactions
The GetTransactions endpoint was updated to take a start and end height parameter, both values are inclusive. If the start height provided is greater than the end height, transactions will be listed in reverse order (from highest to lowest block). To include unconfirmed transactions in your query, the end height should be set to -1, and a list of transactions from start height to chain tip will be returned. This option is set by default to maintain backwards compatibility with the old API.
Transaction Labelling
On chain transactions broadcast by LND can now be labelled with strings up to 500 characters long. These labels can be added on transaction creation when using the various on chain sending APIs, or retrospectively using the LabelTransaction endpoint in the walletkit. If a transaction already has a label, you will need to explicitly set an overwrite flag.These labels are visible in the output of GetTransactions.
Payments Indexing
Payments in lnd are uniquely identified by a payment index which is auto-incremented as we dispatch payment attempts. Previously, calls to ListPayments allowed pagination using these indexes, but required that we fetch all payments from the database and sift through them to find payments lying in the correct range. This caused performance issues for nodes with large numbers of payments. This release adds an index by sequence number which allows efficient lookup of payments in a range. The pagination API itself has not changed, but node operators with a lot of payments can expect a performance increase.
Multi-Channel Outgoing Payment Restriction
The payment RPCs now support sticking to a set of specified channels during path finding. This allows for more fine-grained path contorl, as if one want sto rebalance a specific set of channels, they can use this along with the existing allow/deny lists to force lnd to take a specific route.
REST API
With a series of pull requests the REST API has been overhauled quite notably. First of all, REST is now enabled on all (experimental) subservers. This allows users of the REST API to make use of the new MPP enabled payment calls (Router.SendPaymentV2) for example.
Then, to make the many server-streaming RPC calls available to REST users, a WebSocket proxy was added that transforms streaming responses from single line JSON strings on a keep-alive HTTP response into proper WebSocket messages.
Our online API documentation now also shows a JavaScript example of how to use the WebSockets.
To make our RPC protobuf definitions easier to use and compile by other projects, we removed the Google annotations from our rpc.proto file and instead extracted all REST definitions into the file lnrpc/rest-annotations.yaml.
Bug Fixes
We'll now properly parse both possible KeyDescriptor fields for all calls to the signrpc sub-server.
Pagination for the ListInvoices was updated to return invoices when called with the reversed flag set, and the last index offset was greater than the number of invoices on disk. Previously, this endpoint would return no invoices if the API was called with the reversed flag and did not have the precise number of invoices as its offset.
htlcswitch Improvements
We'll now log each time an HTLC is succesfully forwarded on the least verbose logging level (info). This allows users to monitor their logs to detect sueccesfully forwarded HTLCs.
Payments initiated by the routing will now be directly handed to the link, rather than going through the switch first. This results in lower latency for MPP payments, and also ensures that a link is able to actually carry the HTLC before the router moves onto the next payment shard.
AMP Progress
As a stepping stone towards enabling Atomic Multi-path Payments, 0.11.0 now indexes all newly created invoices by payment_addr in addition to the typical payment_hash index. Unlike MPP, the payment_hash of AMP HTLCs is not known to the sender at the time of invoice creation. Additionally, the payment_hash of each HTLC is expected to be unique, which creates an n-to-1 mapping of payment_hash to invoice. In the future, payment_addr will serve as a universal identifier for both MPP and AMP invoices, as it provides a consistent 1-to-1 mapping for both payment types. The remaining pieces of the AMP saga are expected to appear in a later release, at which point users will be able to query and/or interact with invoices via RPC using payment_addr as the primary identifier. As of now, the plan is to continue to support RPC interactions via payment_hash for MPP invoices as is done today.
Build process
Writing software that compiles and runs on all major operating systems and hardware architectures has become increasingly involved. Our Continuous Integration platform Travis only lets us run a certain number of parallel tasks which limited what we could automatically compile and test on each new commit. Offloading some of those tasks to run as GitHub Workflows has allowed us to add cross-compilation checks for all major operating systems. Travis now only runs a very basic sanity check and then the integration tests. Splitting the work between GitHub and Travis also freed up two build slots where we now run integration tests on ARM and on Windows.
A number of small improvements to our integration tests in general reduced the flakiness of those tests significantly.
Production binaries built from source via make install are now stripped of DWARF symbols and are noticeably smaller.
Full-Node Compatability Updates
This new version of lnd now properly supports bitcoind v0.20.
The latest master branch of btcd is now also supported as well.
Safety Improvements
SCB Recovery
This release of lnd fixes an existing bug that would at times replace the on-disk SCB state with fresh information, if lnd was started in SCB recovery mode (a clean slate) with an existing SCB file on disk. With this new version of lnd, we'll now always combine our in memory state with what's on disk, ensuring that we never override the information on disk.
We'll also no longer insert a recovered SCB channel into the graph database. This resolves a number of edge cases, and also makes the recovery process much simpler.
PSBT Channel Funding
The recently introduced PSBT channel funding flow was found to be risky when used to open multiple channels in one batch transaction. It can happen that the funding transaction is published too early and some of the channel funds get stuck. To mitigate that risk, we added the --no-publish flag to the PSBT funding flow.
On-Chain Contract Handling
We'll now properly clean up local state within the database after a local channel breach.
- Add
--dry-run-migrationconfig option #2668, which will preventlndfrom committing migrations on startup. This provides two useful features:- Avoids unexpected migrations that would otherwise prevent downgrading to a prior database version.
- Allows users to aid in testing migrations for correctness and to guage performance on real-world databases.
Refactoring
This release sees the addition of the lnwallet/chancloser and peer packages. This modularization will help with testing now that these components are properly isolated and can be imported into other packages.
Bug fixes
- Fixed sporadic
tls: internal error: failed to clone hasherror that was caused by a faulty library import. - Added a sanity check to the Tor configuration options that disallows setting Tor socks and control port to the same value.
- Fixes several
subscribeinvoicebugs #4298:- Backlogs are now delivered properly even if no invoices have been settled.
- Two separate race conditions that may have lead to duplicate notifications are now resolved.
- Fixed properly annotating certain classes of closed channels in the database
- Fixed a panic that coul darise if a notification of the
ChainNotifiersub-server is cancelled as a new block arrives - We'll now log the full raw TX hex of any broadcasted funding transactions
- A bug has been fixed to prevent a panic when accepting an incompatible output index in the wallet
- A bug that could arise when a client switches between multiple towers has been fixed
- We'll now ignore reconnection for unadvertised inbound peers over Tor
- Bad features will no longer cause path finding to fail early
The full list of changes since v0.10.0-beta can be found here:
Contributors (Alphabetical Order)
Andreas M. Antonopoulos
András Bánki-Horváth
Anthony Ronning
Bjarne Magnussen
Brian Mancini
CandleHater
Carla Kirk-Cohen
Conner Fromknecht
cryptagoras
ErikEk
Eugene Siegel
fguisso
Gabriel Comte
Jake Sylvestre
Jasper Rädisch
Johan T. Halseth
Joost Jager
Markus Fix
Matheus Degiovani
Olaoluwa Osuntokun
Oliver Gugger
Roei Erez
Sebastian Falbesoner
vctt94
Wilmer Paulino
yyforyongyu