This Advisory is sent on behalf of BSV blockchain Infrastructure Team. The goal is to provide necessary technical support to exchanges and wallets for them better operating BSV blockchain nodes with reasonable cost and high security.
Do Exchanges and Wallets Have to Run Node Software?
Business entities on BSV blockchain are not required to run the SV Node software except miners, who use it to produce and validate blocks. Today, however, many exchanges and wallets across the world that receive frequent payment are still running the node software as blockchain listeners for more independent security and quicker verification.
What Should Exchanges and Wallets Do?
The BSV blockchain Infrastructure Team keeps developing and delivering node software with better performance to handle large volume of transactions. We highly recommend node operators to always update their nodes to the latest version.
All the technical description below assumes that the node operator is running SV Node v1.0.11 which was released in April 2022.
Learn more: https://www.bitcoinsv.io/releases/bitcoin-sv-node-software-v1-0-11
This advisory is only for non-mining node operators (blockchain listeners), covering four parts:
- The minimum and recommended system requirements.
- Recommended parameter values of the mandatory settings.
- Use External data service to crosscheck the blockchain.
- Frequent technical issues when running the node.
Contact Us
Should you have any support questions, please direct them via support@bitcoinsv.io or via telegram at https://t.me/bitcoinsvsupport
Thank you for your continued support of BSV blockchain.
System Requirements
Below are the recommended system requirements for the bitcoind node software v1.0.11 for non-mining.
Development environment
To only follow the longest chain and handle small volumes of other tasks e.g. RPC requests:
Production environment – minimum requirements
To handle a medium volume of workload while maintaining real-time sync with the current chain tip.
Requirements for the STN varies depending on the use it is being put to. Typically, the hardware requirements to run on the STN are at least as high as for Mainnet. We have seen sequential 4+GB block sizes and blocks with transaction counts exceeding 2-3 million transactions on the STN.
Mandatory Settings and Mandatory Parameter Values
There is no default setting for the maximum block size; it is a “mandatory consensus parameter”. The maximum script memory usage is also a mandatory consensus parameter. The SV Node software v1.0.11 also introduced a 3rd mandatory configuration parameter related to fees.
The values for these three parameters must be manually configured in the software by the system administrator. Below provides information on these parameters and recommendations on how to choose the required values.
Method to choose mandatory consensus parameter values
The capacity of the BSV blockchain network is determined by the miners. Miners will analyze the state of the blockchain, the capability of the software, and other factors and determine values for the mandatory consensus parameters. Miners will publish the values that have been chosen.
The recommended method for determining the values of the mandatory consensus parameters is to survey the values that have been published by miners, taking account of the capabilities of the miner. If you are mining, use similar values. If you are not mining, use higher values. Note that miners may change the values that they use so a regular review of the settings is recommended.
Mining nodes vs non-mining nodes
For a non-mining (blockchain listener) node, we recommend setting the values at least twice as high as that of the miners so that when miners increase their settings in the future, non-miners are unlikely to be forked from the network by having a setting lower than theirs.
What if the value is set too low?
If your setting is lower than the majority of miners and (for example) a block is mined that exceeds your settings, your BSV blockchain node will reject that block and all blocks mined on top of it, effectively forking off the network.
However, if you are not mining, it is likely that your fork will not be extended, and your instance will simply cease following the longest chain. In this case, the remedy is simply to increase the values of those settings and restart your BSV blockchain instance. It will then accept the failed blocks and catch up to the rest of the network.
Choosing the ‘unlimited’ option
It is possible to set either of the maximum block size and the maximum script memory usage parameters to effectively unlimited, by choosing a value of “0”. If this option is chosen, there is no risk of forking off the network. However, in the event of an extremely large block being mined or a transaction being broadcast that requires a large amount of memory to validate, it is possible your node could run out of memory and crash. If you have followed best practices and allocated a large swap file and have minimum recommended memory, this is only likely, in an attack scenario. If this happens the remedy is to set your limits similarly to the majority of miners and restart the node.
Maximum block size
This mandatory consensus parameter defines the maximum size, in bytes, of a block. The software will not retrieve or validate any blocks larger than the value of this parameter.
This parameter can be configured using the configuration option “-excessiveblocksize”. This option can be specified on the command line or in the configuration file.
This parameter can also be configured while the software is running using the “bitcoin-cli setexcessiveblock” command. It is important to note that the value of this setting is not persisted when it is set using the “bitcoin-cli” command. If the software is restarted the value of the parameter will revert to the value set in the configuration file or defined on the command line.
Please note that “-excessiveblocksize” is the maximum size block the node will accept. The maximum size block a Miner will attempt to produce is governed by a different setting “-blockmaxsize” which is usually set to a lower value than “-excessiveblocksize”.
The value 0 for “-excessiveblocksize” means unlimited.
It is currently recommended that -excessiveblocksize be set to 10GB.
For example:
excessiveblocksize=10GB
Maximum stack memory usage
This mandatory consensus parameter defines the maximum amount of stack memory that a single script can use during evaluation. If a script attempts to use more than this amount of memory during evaluation, then the evaluation will be terminated, and the script will fail. A script failure will cause a transaction to be deemed invalid and if that transaction is contained in a block, the block will also be deemed invalid.
This parameter can be configured by “-maxstackmemoryusageconsensus”. This option can be specified on the command line or in the configuration file.
It is currently recommended that -maxstackmemoryusageconsensus be set to 200MB.
For example:
maxstackmemoryusageconsensus=200MB
Mimimum mining transaction fee
This mandatory parameter sets the lowest fee rate (in BSV/kB) for transactions to be included in block creation. Note that this paramaeter is still mandatory even for non-mining nodes and can be configured using the option -minminingtxfee.
The recommended value for -minminingtxfee is currently 50 satoshis.
For example:
minminingtxfee=0.00000050
Summary of mandatory settings
Based on the current known miners’ settings, the following values are suggested and would be deemed safe:
Additional Settings
With the increasing adoption of BSV blockchain the transaction volume continues to rise. With the explosive use of data transactions (OP_RETURN), it is possible your BSV blockchain node will not be able to handle the volume of traffic reaching your mempool or be inundated with computationally heavy requests. As a result, the node will drop transactions to allow higher fee-paying ones in, increasing computation at a later point, or worse, cease to function.
A solution is to increase the following values from their defaults to allow the node to remain efficient under high load situations. These situations include re-orgs, which require the node to go back and reconsider transactions or blocks it has most probably already seen. A re-org can be the reason your node spikes from 32GB ram use to 100GB or more, if this is too much, your operating system may choose to end the process (stop bitcoind), or your node will crash with an “Out of Memory” error code.
Since re-orgs and orphans are a part of the Bitcoin system and should be expected, it would be wise to prepare your environment for such situations. The default impaired settings and concepts inherited in bitcoind are too small for the volumes we see during operations on the STN or during a stress test on mainnet.
With this in mind, we suggest increasing a few default settings on your bitcoind node:
- maxmempool=25GB
Your mempool size allowance should be set to 25GB or more. This tells the node how much memory it should assign to storing unconfirmed transactions. This is done by adding the following to your bitcoin.conf file.
- maxsigcachesize=256MiB
- maxscriptcachesize=256MiB
We also suggest increasing the signature and script cache. This tells the node how many accepted transactions in megabytes we can keep in our cache (RAM) improving performance by reducing expensive calls to recalculate signatures and scripts on the fly. We suggest setting these to 256MiB or more to improve performance. This is done by adding the following to your bitcoin.conf file.
Please be aware that setting all three of the mentioned settings will add an additional memory requirement of 25.5GB on your node just for this aspect of bitcoind’s operation.
- maxorphantxsize=64MB
Lastly, we suggest adding -maxorphantxsize to your bitcoin.conf as well. This value specifies how large the orphan transaction pool can grow in memory. This helps if your node is receiving a child transaction whose parent has not been confirmed in the blockchain. This means that the node will remember the child until it sees the parent, or it exceeds its expiration time of 20 minutes. This is done by adding the above to your bitcoin.conf file.
If you have the ram/swap available, you can increase this number to avoid any dropped orphans which may have parents that your node has not seen yet.
Summary of optional settings
If you have the memory available, you may want to change the following optional settings to improve the efficiency of the node operation.
Use External Data Service to Crosscheck the Blockchain
If you would like to use external data services for crosschecking BSV blockchain, we recommend WhatsOnChain.com (WoC). WoC is a leading blockchain explorer and blockchain data provider for BSV runs independent full BSV blockchain nodes across the globe allowing its users to better access historical data. It is free to use and provides REST API and Websocket endpoints that can benefit exchanges, wallets, miners, financial institutions, and research companies that want to utilize the best available analytics to inform their on-chain activities.
Along with receiving real-time BSV data, WoC users also gain access to historic blocks, transactions, address activity, and data. Engineers will find the following endpoints to be useful in displaying customer balances and transactions:
- Transaction History – This endpoint retrieves confirmed and unconfirmed script transactions.
- Address History – This endpoint retrieves confirmed and unconfirmed address transactions.
- Unspent Transactions – This endpoint retrieves an ordered list of UTXOs.
- Bulk unspent transactions – Fetch UTXOs for multiple addresses in a single request.
WoC serves tens of millions of API calls a month and its endpoints are accessible at https://developers.whatsonchain.com/ for free.
For more than 3 calls per second, users can ask for their free API key by contacting the TAAL team through the WoC Telegram channel: https://t.me/joinchat/FfE6-EjZhoTHwhDhZH6F-w.
Learn more about WoC features: https://www.taal.com/blog/2022/whatsonchain-connects-developers-to-essential-bsv-data-and-tools/
Common Issues and Suggested Remedies
The following are details of some commonly encountered issues and suggestions for how they have sometimes been resolved in the past.
Unable to complete Initial Block Download (IBD) or other block download problems
Most issues related to IBD or other block sync/download problems are the result of an insufficiently powerful system, not enough available disk space, running old versions of the node software, or configuration issues. See the sections above on system requirements and mandatory/recommended configuration settings, in particular the “-excessiveblocksize” parameter.
- Inadequate network performance
Older versions of the node software had a hardcoded timeout for block downloads (10 minutes). If a block could not be downloaded in that time the download fails and is aborted.
A 4GB block requires both the source server and the network to sustain at least 56MB/s per connection. On rare occasions a new block may not found for 30 minutes or more. In that case, the size of the block will be roughly 3x the size of a standard block. If 4GB blocks become the norm, the source server and network may need to sustain at least 168MB per second per connection. If your system does not meet that requirement, your node may timeout downloading the block.
The latest version of the node software has parameters:
“-blockdownloadtimeoutbasepercent=<% of 10 minutes>”
“-blockdownloadtimeoutbaseibdpercent=<% of 10 minutes>”
These two can be used to change the timeout. The first parameter controls the timeout during normal node operation while the second is specific to IBD.
For example, to set the timeout to 60 minutes during IBD and 20 minutes thereafter:
- blockdownloadtimeoutbaseibdpercent=600
- blockdownloadtimeoutbasepercent=200
- Insufficient disk space and pruning
The node uses separate threads to download blocks over the network and validate these downloaded blocks. In older versions of the software the the thread downloading the blocks could race ahead of validation and cause the disk to fill up, even if a pruning target has been set.
The latest version of the node software fixes this particular problem and ensures the node sticks much more closely to any pruning target that has been configured. Note however that the following points still need to be considered:
- The node will keep at least 288 blocks no matter what the pruning setting is.
- Pruning is incompatible with configuration “txindex=1” and “rescan”.
- Reverting “txindex=1” or “rescan” will only take effect after restarting the node which in this case will repeat the whole IBD process from scratch.
If you are running with “-txindex=1” then as of mid 2022 you will require disk storage >4TB. This is only going to increase as time goes on. To minimize disruption it is important therefore to have a strategy in mind for adding storage as the blockchain grows. For example, elastic file system, lvm, raid, etc.
- Slow disks
For a node you are interfacing to frequently, the speed of access to data on disk can become important. In such cases the minimum data transfer rate from the disk should ideally be 500MB/s SSD, preferably more (consider upgrading to NVMe).
- Insufficient memory
If your node’s process has mysteriously vanished it may be that it has run out of memory and been terminated by the operating system. Check the syslog file for an entry similar to the following:
/var/log/syslog:Jan 17 20:08:16 ubuntu kernel: [95858.667098] Out of memory: Killed process 3502 (main) total-vm:11606368kB, anon-rss:4790248kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12492kB oom_score_adj:0
- Using “loadblock” to resolve download issues with a specific block
If you have access to another working node and your download fails at a specific block height, it is possible to copy the blkXXXXX.dat file for the specific block across to your local block
repository and use the “loadblock” configuration option to get the node to load the version of the block stored locally on disk rather than perform a download over the network. Note that you need to ensure that you trust the source of your blkXXXXX.dat files.
To be able to associate a blk.dat file to the hash of the block in question the debug option “debug=bench” needs to be set. You can then look for a message in the log file of the form:
Accepted block hash=000000000000000008b0254ec892b3a218fa53372eee99e1840afa25ed9285df,
height=745232, size=15715292, num_tx=2674, block-time=2022-06-23 16:36:48, file=blk21741.dat
Sometimes all the subsequent blocks have already been successfully downloaded and the problem is related to just one block. In such cases forcing a load with “loadblock” can be a quick solution.
Corrupted block database
If your node instance crashes or is killed by the operating system (for example due to lack of memory) it can sometimes happen that when you try to restart the node it reports an error loading the block index and aborts:
Error loading block database.
Please restart with -reindex or –reindex-chainstate to recover.
If you want to try and avoid the time it takes to do a full reindex, a quick fix that sometimes works is to delete the last 1- 3 blkXXXXX.dat files from the blocks directory and try restarting again.
RPC timeouts and other issues
JSON is not a good choice for requesting large amounts of data over the RPC interface for “-getblock” or similar calls. In such cases it can require large amounts of memory to construct the response and trigger the out of memory issues already mentioned. It can also take a considerable amount of time to send the resulting large JSON string, possibly resulting in a queue of pending RPC calls building up and eventually causing some calls to timeout.
The timeout for RPC calls can be increased with the “-rpcservertimeout” configuration parameter, but if you are using the RPC interface to request details for large blocks or transactions a much better option is to use the binary REST interface instead. See the documentation in the “doc/REST-interface.md” file for more details.
Other ways to reduce the cost of running the node
An alternative solution to the increasingly costly and complex challenges of maintaining an operating a node may be outsourcing. You can collaborate with mining nodes. For example, Taal, as a prominent miner on the BSV Blockchain, offers enterprise grade node-as-a-service solutions. Taal is also planning to release a new service that offers the functionality of a node at a fraction of the current cost. Contact Taal via sales@taal.com.