One of the biggest announcements to come out of the CoinGeek Zurich Conference was the publication of the Merkle proof standardised format by the Bitcoin SV Technical Standards Committee (TSC).
During their presentation at the event, nChain CTO and TSC chairman Steve Shadders announced together with Two Hop Ventures founder and TSC founding member Alex Fauvel that the public review process for the Merkle proof standardised format had been completed and the standard has now been published under the OpenBSV licence.
This is big news for Bitcoin SV developers and users alike, as it enables far greater interoperability for BSV payments and provides a stable and reliable platform for building simple payment verification (SPV) services on the blockchain. SPV is an integral component of Bitcoin SV’s core functionality, as it is the blockchain that most closely aligns with the original Bitcoin white paper published by Satoshi Nakamoto in 2008, which describes the application of SPV to enable users to receive and validate payments without running a full Bitcoin node.
The Technical Standards Committee aims to improve interoperability within the growing Bitcoin SV ecosystem by working with developers and stakeholders to develop common standards that are compatible across applications. It is important to note that the TSC does not determine standards – it provides the framework and process for the development of technical standards to improve interoperability and accelerate the adoption of Bitcoin SV.
‘The TSC is not here to decide what standards are. It is really here to help industry decide what those standards are themselves,’ Shadders explains.
The Merkle proof standardised format has been through the committee’s comprehensive and open standardisation process, which includes consideration, drafting, internal review and public review, and it has now been published for adoption.
Merkle proofs and how they work with SPV
To understand the benefit of a standardised format for Merkle proofs, one must first understand the importance of this data structure to simple payment verification (SPV). SPV allows a merchant or user to validate a Bitcoin SV transaction without needing to download the whole blockchain. Instead, they can receive a block header of the Bitcoin block containing that transaction from a reliable node. This block header contains the Merkle root, a hash of all the transactions within that block.
Bitcoin transactions are stored in a Merkle tree data structure, which acts similarly to a binary tree, except that the leaves of the structure comprise hashes of each transaction, which are then hashed with their neighbours up each layer until the root of the tree – the Merkle root. The Merkle root, therefore, contains the proof for each transaction contained within that Bitcoin block, making it possible to determine the existence of a transaction on the blockchain using this proof and the transaction ID or hash.
The technical properties of this validation mean that users of Bitcoin SV can safely run an SPV ‘light node’ on lightweight and inexpensive hardware to receive transactions from others, with the confidence that they can easily check whether those transactions are valid – no complete blockchain required.
‘Aside from a Bitcoin block header, a Merkle proof is probably one of the most fundamental data structures in Bitcoin. It’s what allows you to prove that a transaction is connected to a block, i.e., prove that miners have accepted that transaction, so there are a lot of different reasons why you would want to exchange a Merkle proof between parties,’ Shadders tells Bitcoin Association.
‘It is critical to peer-to-peer interaction because part of that interaction means sending bits of information with Merkle proofs attached, so that means many different wallets need to support it. So, if everyone is implementing it in a different way, then every time you want to attach to a service, you’ve got to work out a new way of doing it.’
With the launch of the Merkle proof standardised format, these crucial proofs can be shared between different services and users with assurance that they will be compatible with their wallets or SPV nodes, making the entire process of transaction validation more autonomous and accessible.
The importance of technical standards
The Merkle proof standardised format is the first standard from the TSC to reach the publication stage, and while it was fleshed out considerably by the time it reached the public review stage, stakeholders still significantly improved aspects of the standard as they submitted feedback prior to publication.
Standards are an important part of fostering an ecosystem of innovation and interoperability, Shadders explains, as their prevalence across traditional industries demonstrates. They also do not reduce competition, but instead improve accessibility for both competitors and users. He uses the example of the DVD industry to portray how standards can enable competition and improve user experience.
‘Let’s say I had a Sony DVD player and I buy a Samsung DVD player. Imagine if they had different formats – I’d have to go out and buy a whole new DVD collection to replace all my existing ones. [This standard] basically solves that problem,’ he says.
‘Every wallet can implement the existing standard for Merkle proofs – that means they are able to instantly communicate with all of the other wallets that have implemented it, some of which they might not even have thought of. That includes wallets that do not exist yet and may be built in the future.’
Without standards like this, industries can run the danger of presenting siloed, disparate ecosystems to users and reducing overall accessibility to the underlying technology and applications built upon it. By addressing the standardisation of the Merkle proof format, the Bitcoin SV community has ensured that interoperability will be a foundational component to competition and innovation among SPV wallets and data providers.
‘The fundamental element of SPV itself is the inclusion proof – the Merkle proof. If we didn’t have this standard, anyone who wanted to implement proper SPV in a wallet would have to make up a data structure and then the second person that came along would have to decide whether to adopt that data structure or to be incompatible, in which case you have two SPV wallets that have siloed ecosystems,’ Shadders explains.
He stresses that the standards published by the TSC are reflective of community input rather than the views of committee members, adding that once these standards are published, they remain subject to public scrutiny and may be updated, modified or even withdrawn if better solutions are found or the surrounding technology changes.
‘The TSC only works in collaboration with the industry, with us playing the role of being the facilitators and the service providers and other people actually providing the input,’ Shadders says.
‘I think it’s probably important to draw a distinction when it comes to TSC members; TSC members, when they operate in their capacity as a member of the committee itself, they are independent and are there to act in a facilitation role. Outside of that, of course, they’re all industry participants and people who have interests in certain things so they can also become members of workgroups.’
If a standard needs updating at a point in the future, the TSC’s standardisation process is designed to accommodate this, and it is possible that many standards will require updates and modifications as new requirements arise or the ecosystem changes.
‘We try to encourage the participants to think about forward compatibility so that they’re not proscriptive about changes that might happen in the future. But it’s quite possible that a standard might simply become unfeasible or obsolete because the surrounding technology has changed, in which case that standard will more than likely be withdrawn and replaced by a new one,’ Shadders says.
‘It’s simply a matter of somebody articulating an industry need for either an update or a replacement of a standard, and that gets submitted and will be evaluated by stakeholders, not just by the TSC.’
The Merkle proof standardised format
Now that the Merkle proof standardised format has been published, details of the standard are available on a dedicated page on the TSC website.
The format specifies the data structure in which the Merkle proof information is stored when it is transferred between users, such as SPV wallets, which the preamble to the technical specification notes as crucially reliant on the efficient and interoperable exchange of this information.
The Merkle proof standardised format comprises two components:
- A representation of the proposed data structure format in binary and JavaScript Object Notation (JSON).
- An explanation of the algorithm used to validate transactions against the Merkle proof once received in this format.
The scope of this standardised format for Merkle proofs relates only to simple proofs, such as those required by SPV wallets validating single transactions upon reception, with composite proofs falling under the umbrella of a future extended standard. The definition of standardised API calls is also omitted, as this is unrelated to the core data structure and will benefit from further deliberation.
Under the standardised format, Merkle proofs are stored in a structure the includes a single flags byte, the positional index of the transaction in the Merkle tree, a list of 32 bytes hashes and a transaction ID or hash. This information can be stored and provided in either binary or JSON format, with the flags byte describing additional functionality. Those using this standard to check a transaction against the Merkle proof of a block can choose whether to include the original transaction or just the transaction ID, the target type or final element, the proof type (branch or tree), and whether this proof is part of a composite proof set.
The process of Merkle proof validation is also described and includes several checks to account for the flags mentioned above, as well as the validation of nodes with no neighbour in the Merkle tree data structure.
During its internal and public review processes, the TSC received comments regarding the complexity and default options of the standard data structure format. Most notably, many stakeholders suggested that the common use case (of validating single transactions using common data fields) should not require interaction with the more complex aspects of the format.
‘In order to address this concern, changes have been made, primarily to the JSON use case, such that using default options results in a much simpler JSON object. This retains the possibility of using the alternate options without breaking the standard, whilst hiding the optional typing fields in the default case,’ the specification states.
A full technical description of the Merkle proof standardised format can be found on the TSC website.
How a Bitcoin SV standard is made
This debut standard demonstrates the capabilities of the TSC as a vehicle for community development to improve the environment that Bitcoin SV developers, start-ups and enterprises are operating within. The internal and public review of the Merkle proof standardised format led to the creation of an easier-to-use data structure format and validation algorithm while retaining flexibility and forward-looking support for tasks like validating composite proofs.
The TSC follows an in-depth standardisation process that comprises several steps which ensure interaction with the community and stakeholders. Ideas for Bitcoin SV standards can be proposed by anybody, and they are considered by the committee before reaching the next stage – drafting of the technical standards.
Once the standard has been drafted, it undergoes a series of internal reviews which include a review of the intellectual property and a legal review to determine any concerns that may arise from the publication of the draft standard. While this is taking place, an NDA is effective, ensuring that the proposed standard remains secured within the environment of the committee and stakeholders involved.
Once these review processes are completed, the standard is either moved back to the drafting phase if more work is needed or it enters the public review phase. During the public review, any interested parties can provide feedback on the proposed standard, all of which is considered and if proven to be relevant, the standard may be adapted to accommodate this feedback.
Only once this public review has been passed is the standard published, although the work does not stop there. Following publication of a standard, the TSC and stakeholders inspect the industry reaction and adoption and after a sufficient period of time, they may recommend or withdraw the standard from publication.
‘At the TSC we developed a standard creation process that consists of three phases. We have the first stage, which is the submission phase, which allows people from outside of the TSC to submit something that they would like to see standardised, whether it’s in their industry or not. We would prefer them to be a stakeholder in that industry so that they will actually adopt it and increase the need for it in the actual industry,’ explains founding member of the TSC, Alex Fauvel.
‘The second stage is where we have the drafting area. Effectively we make the first draft, then we review it internally and then we get IP specialists to look at it. And then if it’s required, we get technical specialists to look at it.’
‘Once we’re happy with the first draft and that’s done several cycles, we then publish it into the public where the public has a chance to comment on it. That is a two-month period and once we’ve incorporated that feedback, if it’s needed, we will publish it, which is the third stage. We then monitor the uptake of it in the industry to either rubber-stamp it and say we recommend the standard or we withdraw it, which we hope to not be doing!’
The TSC is leveraging this process as it continues to facilitate the creation of useful and powerful standards for Bitcoin SV development. This includes the envelope specification, which is currently in the public review phase and open to comment. This specification deals with the way data is stored within Bitcoin transactions by creating a general-purpose data envelope that allows for efficient processing of information and interoperability with existing and future data protocols.
To participate in the development of technical standards for Bitcoin SV, visit the TSC website to propose a new standard, provide feedback on draft standards in public review, or submit suggestions on the draft roadmap.