My Blog

Meet ARC – the new and improved way to broadcast

Showcasing BSV Transaction Processing Service ARC

On October 4, 2023, Thomas Giacomo, Director of Utilisation, and Darren Kellenschwiler, Utilisation Technical Lead hosted the webinar “Meet ARC – the new and improved way to broadcast”. The webinar aimed to introduce ARC’s features and advantages, highlight the differences from mAPI and improve transparency within the ecosystem regarding upcoming infrastructure changes.

ARC was initially developed by Simon Ordish, Co-Founder of WhatsOnChain, and Siggi Oskarsson, Co-Developer of BUX. Currently, it is being implemented by TAAL and GorillaPool. ARC is designed to integrate with the upcoming Teranode upgrade of the BSV ecosystem.

What is a Transaction Processing Service

Darren Kellenschwiler introduced the topic by explaining the concept of transaction processing services, focusing on the challenges of using the SVNode network. He explained that the SVNode network is a peer-to-peer network, which only sometimes acts lucidly due to its decentralised nature. Nodes come and go, and it’s difficult to identify them. The network operates on a best-effort basis for communication, including transaction propagation and block announcements. Nodes request data from each other to achieve a global network consensus within about 2 seconds.

At times, this can be challenging for users, especially for those on the move or with unstable internet connections. Some nodes are prioritised based on their role, and whitelisting helps maintain important connections. To simplify this complexity for users, Kellenschwiler suggests using a transaction processing API, which is a straightforward HTTPS web API with a request-response model. It ensures that transactions are broadcast to the network until they are mined, making it easier for users to interact with the network.

The ultimate goal of transaction processing services is to allow developers and businesses to focus on their specific problems, leaving the complexities of the blockchain and transaction handling to the miners. Users and businesses should not have to worry about these technical aspects.

What ARC is and how it works

ARC is designed to connect to every mining node on the network and includes peering logic, retry logic for transaction tracking, transaction validation, and an API for clients. It also calculates Merkle paths for broadcasted transactions. ARC’s microservices include the API server, validator (with the ability to scale for increased workload), metamorph for managing changing transaction statuses, and a peer manager to handle connections. Additionally, ARC stores a new format of blocks containing transaction IDs rather than full transaction data.

This architecture aims to enhance transaction reliability and efficiency in the Bitcoin network. ARC consists of four microservices: the API, Metamorph, BlockTX and Callbacker.

ARC API

The ARC API is the user interface for interacting with the ARC system. The API provides a few key endpoints, such as a policy for setting requirements, notably the mining fee in satoshis per byte. The tx submission endpoint is exposed for processing transactions, either in parallel or serial. These endpoints receive transactions in an “extended format,” allowing for script validation and fee checks, as Kellenschwiler explains.

The extended format transaction concept was developed to overcome potential bottlenecks caused by looking up UTXOs using only raw transaction data. It includes additional information, like the unlocking script and the locking script of the previous transaction. Kellenschwiler encourages the originator of a transaction to pass this information to the recipient for broadcasting. However, he remarked that the concept depends on a UTXO lookup service, which is still in development.

The goal is to change the validation process from being purely performative to genuinely checking the validity of transactions before broadcasting to other peers. This added data enables transaction validation before reaching the Metamorph stage, improving the reliability of transaction processing within ARC.

Metamorph

The primary purpose of Metamorph is to take pre-validated transactions and place them on the blockchain using the peer-to-peer network. This process may involve connecting to multiple instances of Metamorph, which are designed to be horizontally scalable, especially when dealing with a large number of transactions. Load balancing, specifically round-robin load balancing, is used to distribute the workload among multiple instances. Metamorph is responsible for processing the transactions received from the API, monitoring their status, and forwarding relevant updates as necessary.

BlockTx

The BlockTx microservice listens to peer-to-peer messages and specifically monitors block announcements. When it detects a block announcement, BlockTx retrieves the necessary data and actively processes it as it comes in. Unlike a core node, ARC doesn’t assemble blocks or require full transaction data. Instead, it focuses on ensuring that the transaction IDs are genuinely part of the Merkle root within the block header.

BlockTx operates by hashing each transaction as it arrives, producing the transaction ID, and discarding the rest of the data. This creates a data structure consisting of a block header and a list of transaction IDs. BlockTx’s role is to calculate the Merkle path for the transaction IDs relevant to the connected Metamorph instances.

Each Metamorph instance is only interested in a subset of these transaction IDs, particularly those submitted to it. BlockTx updates the status of the transactions it’s responsible for to “mined” and stores their corresponding Merkle paths in a format described in BRC 71, an open repository for Bitcoin improvement proposals.

ARC: Callbacker

The Callbacker allows users to monitor the progress and status of their transactions. Users can check the status of their transactions at any time by hitting the “get tx” endpoint through the API. The Callback service is responsible for tracking users who want to be notified when a transaction’s status changes. It also allows them to specify which status updates they wish to receive.

Users can specify a URL and, if desired, a token to receive notifications when the transaction status changes. The callback service will send data, including the mining status and Merkle path, to the provided URL. This data is identical to what is included in the transaction status.

ARC workflow

Darren Kellenschwiler outlined the overall flow of the transaction processing system within ARC. He uses a diagram to showcase ARC’s workflow, there are two sides to consider:

Submission Transaction via P2P (Left Side):

  • The client initiates a transaction submission through the API server.
  • Validation is performed.
  • The transaction goes through the Metamorph system.
  • Within approximately five seconds, the transaction reaches its final status.
  • In most cases, these processes occur within seconds, and a positive response (e.g., “announced,” “sent,” “seen”) is received. The default behaviour is to respond within 5 seconds or earlier if the transaction reaches “seen” status before then.

Process Block via P2P (Right Side):

  • The Bitcoin network initiates an event by announcing a new block with its hash.
  • ARC retrieves the block, calculates the Merkle paths, and delivers transaction IDs and status updates to clients.
  • Extended format transactions are delivered to the API, undergo validation, and are passed to Metamorph.
  • The transaction is registered with a store, and its status becomes “stored.”
  • The transaction ID is announced via the peer handler to the Bitcoin Network, updating the status to “announced.”
  • Other Bitcoin nodes retrieve the transaction data from ARC through peer-to-peer messages, and ARC updates the status to “sent.”
  • When a node in the network validates the transaction, it announces a valid transaction via a text message to its peers, and ARC updates the status to “seen.”
  • At this point, a response is sent to the original client through the API.
  • This flow demonstrates how transactions move through the system, from submission to reaching a final status, with real-time updates on their progress.

A closer look at the block hash message

Darren Kellenschwiler explained how the block hash message functions when a node in the network finds a new block. ARC retrieves the block data, streams it in, extracts the transaction data, and stores the IDs and Merkle paths for every transaction in the block. This approach is taken to handle situations where a user reconnects to the API after being offline and sees a status of “mined” for a transaction they didn’t send themselves. Storing all transaction IDs in the block ensures the accuracy of this process.

Kellenschwiler anticipates that the Merkle path calculation service may become more sophisticated in the future. It is worth noting that this is the first Merkle service separate from a node, allowing it to calculate Merkle paths independently, which is a key differentiator from previous iterations, as Kellenschwiler outlines.

Differences from SVNode

Darren Kellenschwiler clarifies that ARC is not intended to be a mere upgrade of SVNode software, often referred to as “SVNode 2.0.” Instead, it represents an entirely different approach. While both ARC and SVNode serve the same user base, ARC is designed to be compatible with Teranode.

Kellenschwiler provided an overview of the SVNode, which is (at the time of the webinar) still the official implementation for the BSV blockchain. SVNode is responsible for numerous tasks, including block assembly, transaction assembly in blocks, transaction and block validation, timestamping, peer-to-peer network setup, and serving as a wallet with an embedded JSON-RPC wallet. He emphasises that the inclusion of wallet functionality within the node software made sense during Bitcoin’s early days, as it was an all-in-one software handling everything.

ARC, on the other hand, represents a different approach, focusing on improving and enhancing transaction processing services in the BSV ecosystem. It is not merely an upgraded version of the SVNode software but rather a complementary solution designed to serve the same user base more effectively.

mAPI 2.0?

Darren Kellenschwiler touched on whether ARC can be seen as a new version of mAPI (merchant API). mAPI is a service that extends the functionality of the user wallet built into the SVNode. mAPI exposes a public-facing transaction processing API that uses the local node’s JSON-RPC client to broadcast transactions as if they were coming from the node itself. In the early days of Bitcoin, this was a valuable solution for dealing with the complexities of the peer-to-peer network, making it easier for application builders.

mAPI allows users to submit transactions via a standard web API and receive a response signed by a miner on the network. It also provided features like double-spend notifications and Merkle-proof webhooks.

Issues with mAPI and how ARC solves these issues

However, these services relied on the node’s JSON-RBC client, and over time, as transaction volumes increased significantly, the resource drain on the node led to issues. The node prioritises maintaining the network, validating transactions, and assembling blocks, which sometimes causes it to ignore local wallet JSON-RBC client requests during heavy load, leading to 500 errors for mAPI users. This bottleneck became a problem in high-traffic scenarios.

Darren Kellenschwiler said that ARC doesn’t face the same issues as mAPI, because it can adapt to increased demand by spinning up multiple Metamorph instances. While the BSV network may experience delays in reaching consensus and confirming transactions into blocks, ARC continuously manages the status of these transactions. Even under heavy load, a transaction may remain in the “announced on the network” status for hours, but eventually, it will get mined and update its status.

In contrast to mAPI, ARC handles the retry logic and the complexities of the peer-to-peer network on behalf of applications. This separation of responsibilities allows applications to proceed without dealing with these inherent complexities, providing a smoother experience for users.

Context: Infrastructure upgrade with Teranode

Darren Kellenschwiler delved into the need to transition from the SVNode to Teranode in the coming year. As Teranode begins to come online, considerations arise regarding how to manage transactions within the Teranode system. One major difference between these systems is that Teranode lacks a built-in wallet, meaning the same methods used for transactions in SVNode won’t apply when Teranode takes over the network’s load.

This transition is part of a larger strategy aimed at separating and specialising different components of the original software. Kellenschwiler referred to his work on the LiteClient, which focuses on Simplified Payment Verification (SPV), and how Teranode specialises in mining, block assembly, and transaction validation. The separation of concerns between these components underscores the need for a new approach, as the reliance on the node wallet is no longer feasible in this evolving ecosystem.

Darren Kellenschwiler highlights the difference in approach between ARC and the previous systems. ARC is designed to broadcast transactions to all nodes in the network simultaneously over the peer-to-peer network. When considering Teranode, IPv6 multicast broadcasting might be used, which isn’t necessarily straightforward, especially in mobile environments where it depends on router configurations.

ARC simplifies this process by allowing users to make outbound connections to a transaction processing service (ARC), which handles the complexities of the new technology and hyperscale IPv6 multicast. This approach provides a user-friendly interface for mobile connectivity and transaction broadcasting while removing the challenging technical aspects, allowing businesses to focus on their core activities.

The advantages of using ARC

Darren Kellenschwiler summarises the advantages of switching to ARC:

  • Transactions broadcast over P2P network: ARC broadcasts transactions over the peer-to-peer network, avoiding bottlenecks associated with the use of the node wallet.
  • Avoids bottlenecks: ARC doesn’t rely on the node wallet, extended format transaction validation, or external data. It provides all necessary data for validation, making it more scalable and a step towards Simplified Payment Verification (SPV).
  • Horizontal scalability: ARC scales horizontally as demand increases, making it suitable for high-traffic scenarios.
  • Granular transaction status: Metamorph offers detailed transaction propagation status, providing real-time information about the transaction’s journey to being mined into a block.
  • TxBlock confirmation: TxBlock provides final status confirmation of being mined into a block and includes Merkle paths, allowing for the construction of SPV payment protocols or record generation.
  • Asynchronous status updates: The callback feature can inform users of asynchronous status updates, providing real-time information on transaction progress.

Who is using ARC and how to use it

Currently, ARC is installed by the BSV mining pools TAAL and GorillaPool, two of the main miners on the BSV network. The documentation for these implementations is specific to each miner, but from a user perspective, they function similarly. TAAL requires registration on their console to obtain an API key, while GorillaPool does not require it at the moment. Both miners have been running these implementations for a couple of months, and they are currently in public beta.

Users might encounter challenges if they use unusual transaction formats, as there may be some issues. Kellenschwiler mentions that they are receiving great feedback from power users and gives a shoutout to David Case, who has been valuable in finding and addressing bugs in the software. Currently, the software architecture has a limit of around 3,000 transactions per second. TAAL is working on a Kubernetes configuration to handle higher throughput on their instance.

Transaction Processing Roadmap

In terms of the roadmap for ARC, the plan for the rest of this month involves continued feedback collection to identify and fix errors, particularly those associated with the Go BT Library and the script interpreter. Issues related to unformatted or returned data have been addressed.

From October to December, there will be a phasing-out process of mAPI. This involves encouraging users to switch to ARC, as mAPI will not be adapted to work with Teranode. In 2024, mAPI will be deprecated, with the specific transition timeline being subject to miners’ decisions.

In the future, there may be a divergence between two different ARC versions. The first version will be a highly scalable Kubernetes configuration of ARC designed for miners to run and provide additional services. Kellenschwiler projects the second version to be an independent business version of what ARC does, which he refers to as an overlay node. Companies within the ecosystem can run their own instances of overlay nodes, customised to meet specific needs, such as managing tokens. Efforts will be made to facilitate peering logic for running an overlay node with SVNode, and a reference implementation will be provided for creating self-service overlay nodes.

Overall, development will be split into these two products: the miner-oriented ARC configuration and the self-service overlay node for businesses to manage their transactions more effectively.

Michael Sprick

, , , , , , ,
, , , , , , , ,