cculianu/Fulcrum v2.0.0

Fulcrum 2.0

on GitHub

What's New:

This is a major release of Fulcrum which adds much-needed reliability and performance improvements.

The database format has been completely redone and now Fulcrum provides reliability guarantees such that Fulcrum's db can no longer get corrupted. In other words, you should never see the dreaded "your database is corrupt, please resynch"-style messages, even if Fulcrum is killed at an inopportune time.

Since the database format has been completely redone, the database is no longer backward compatible with older Fulcrum versions, but you can upgrade your existing database to the new format by passing the one-time flag --db-upgrade the first time you run Fulcrum 2.0 when upgrading from a 1.x install.

Brief Summary: What's New in 2.0

A brief summary of what Fulcrum 2.0 offers vs the 1.x series:

• Database Reliability - one of the main weaknesses of Fulcrum 1.x was that should the process be killed at an inopportune time, or should abrupt powerloss or kernel-level crash occur at an inopportune time, there was always a risk that the datadir would get corrupted and the admin would be forced to do a full database resynch. This is no longer the case!
  • Fulcrum 2.x series will no longer suffer from this problem and the process can be killed at any time (including abrupt powerloss), without any database corruption. At worst the last few blocks worth of data is rolled-back and Fulcrum will re-synch from the rollback point.
  • It is still recommended that admins gracefully shut-down Fulcrum, however, via SIGINT/SIGTERM/SIGQUIT (either from their init/systemd scripts or via kill -TERM <pid>, etc), and wait for it to terminate. This maximizes performance and minimizes startup times for Fulcrum the next time it is restarted.
• Platform Neutrality - database format is now 100% platform-neutral. This means it should be possible to always copy datadirs around regardless if transferring them from Windows, Linux, BSD, macOS, and regardless of platform architecture (including big endian platforms).
• Improved database performance.

Upgrading from the Fulcrum 1.x Series

Upgrade Instructions

Fulcrum 2.0 has a different database format than the 1.x series. As such, you have two options for upgrading:

• Create a blank new datadir and just re-synch from block 0 (slow, but 100% reliable)
• Upgrade your existing 1.x datadir (faster, but irreversible and destructice)

To upgade your existing database, be sure to pass the one-time --db-upgrade flag to Fulcrum. It will refuse to start up if it detects that your datadir is in the old format and you did not pass this flag. This ensures that admins know what they are getting into.

Note: The upgrade process takes seconds to minutes on BCH and BTC testnet(s), maybe ~15 minutes on BCH mainnet, and around an hour or more (depending on hardware) on BTC mainnet.

--utxo-cache Removed

The --utxo-cache option has been completely removed. If you want to temporarily leverage memory to help out initial synch, use a temporary --db_mem (CLI) or db_mem= setting in the conf file to give Fulcrum more memory for initial synch, stop Fulcrum, then set it down again to a smaller value for continued operations.

Critical Note On Upgrading the DB from 1.x -> 2.0

Let it run to completion!

While Fulcrum 2.0 is 100% reliable and may be killed at any time during its operation -- there is one exception to this rule: the upgrade process that converts 1.x databases -> 2.0. As such please let the upgrade process run to completion and do not kill, stop, molest, or otherwise abuse the Fulcrum process while it's upgrading the DB!!. If you do, the old 1.x database will be lost and your 2.0 database will also be corrupted.

You have been warned!

Detailed Summary of Changes:

• Completely redid the database format to leverage rockskdb atomicity and reliability guarantees.
• Added --db-upgrade CLI arg (required to upgrade an existing DB from Fulcrum 1.x to 2.0 format)
• Added some performance improvements to the rocksdb database
  • Made some tables use point-lookup optimizations
  • Made all tables use 2-level indexing which is more appropriate for the way Fulcrum data is laid out
  • Made some of the tables' keys be properly sorted (such as scripthash_history) -- they were not always completely sorted in an optimal fashion before, which cost a few cycles sometimes.
  • Changes to make better use of db_mem db memory (should also help slightly reduce memory consumption)
  • More forcefully enforce rocksdb kNoCompression for all levels.
• Made the database format have a specifiec endian-ness for all data and so the database is completely cross-platform. You should now be able to copy any Fulcrum datadir from any OS and architecture to any other OS and architecture, regardless of endianness, word size, etc.
• Got rid of the entire --utxo-cache mechanism (it's no longer compatible with Fulcrum's newfound data reliability guarantees).
• db_mem now defauls to 2048 GiB or 25% of total physical memory, whichever is smaller
• db_max_open_files now defaults to 1000 (was: 40)
• Internal misc. other performance fixups
• Internal code quality improvements
• Removal of some dead/unused code
• Redid some aspects of the internal unit test system, and added some unit tests
• Various build system and misc. compile issue fixes

Pre-compiled Binaries for Linux

I have provided two pre-built binaries for Linux:

• Fulcrum-2.0.0-x86_64-linux.tar.gz, which is compiled on an Ubuntu 18.04 system using Docker.
• Fulcrum-2.0.0-arm64-linux.tar.gz, which is compiled on an Ubuntu 22.04 system (for ARM64 bit) using Docker.

All of the above binaries contain jemalloc, libzmq, libminiupnpc, Qt5Core, Qt5Network (from Qt 5.15.6), OpenSSL 1.1.1 and libstdc++ statically linked. They still require libz2, and the right libc version as dynamic libs on your system (but those are usually present if you are on a recent system).

Pre-compiled Binaries for Windows

• Fulcrum-2.0.0-win64.zip - Pre-built, statically linked Windows version.
  • It should "just work" on any Windows 10 or above 64-bit system.
  • Includes is a statically-linked Fulcrum.exe, built with jemalloc, libzmq, libminiupnpc, Qt 5.15.13, OpenSSL 3.3.0, and gcc 13.2.0.
  • Additionally, FulcrumAdmin.exe is included which is the python script, but made into a onefile .exe using PyInstaller.
    • NOTE Windows virus scanners have been known to erroneously identify FulcrumAdmin.exe as containing a virus. This is because many heuristic scanners dislike the use of PyInstaller onefile .exe's. See: #203. Please ignore this -- we contain no viruses! Also please complain to your virus software vendor about this.

See the .asc file for signatures; I signed the sha256 hashes of all the release binaries. My gpg public key can be obtained here: https://github.com/Electron-Cash/keys-n-hashes/blob/master/pubkeys/calinkey.txt

Binary builds for macOS coming soon -- Until then you can always build from source!