The First Step of Ethereum Modularization: Basic Introduction to Proto-danksharding and Its Working Principle

Ethereum has successfully completed the transition from proof of work to proof of stake consensus protocol. The next important project is EIP-4844, also known as "raw data availability". This code change aims to enhance the scalability of rollups built on Ethereum.

EIP-4844 introduces a new transaction type called blob. It increases the data and storage requirements of Ethereum blocks and creates a new fee market that separates the pricing of blobs from regular transactions.

Rollup is a protocol that relies on Layer 2 blockchain ( such as Ethereum ) for data availability ( DA ). Typically, smart contract-based rollups rely not only on Ethereum's DA but also on Ethereum for transaction settlement. These rollups read data from DA layers such as Ethereum and execute valid transactions and smart contract code.

The cost of publishing large amounts of data to Ethereum is usually high, partly because the network permanently stores data as part of the transaction history. With EIP-4844, each block will create an additional 512kB or 768kB of data space for rollups. This data will be stored for about three weeks. In theory, the cost of publishing data to Ethereum through rollups will be greatly reduced.

Proto-danksharding is the precursor and "prototype" of fully danksharding, which will enable Ethereum nodes to download fragments of blob data to determine the availability of the complete blob.

This article delves into the details of how EIP-4844 works, the limitations of the original danksharding, the evolution plan of EIP-4844 towards complete danksharding, and the direct benefits to end users and dapp developers.

Background

EIP-4844 is seen as a scalability upgrade for Ethereum. However, it is important to note that this code change does not substantively increase or improve the transaction capacity of Ethereum itself. Proto-danksharding reduces the cost of publishing large amounts of data to Ethereum, thereby lowering the operational costs of rollups. EIP-4844 is considered to enhance the scalability of Ethereum, as it makes Layer 2 networks built on Ethereum more cost-effective, but the code change does not improve Ethereum's scalability as a general-purpose blockchain.

In recent years, trading activity of Ethereum rollups such as Arbitrum, Optimism, StarkNet, zkSync, and Polygon zkEVM has been increasing. L2Beat.com estimates that the total transaction throughput achieved by all Layer 2 networks is 3.8 times the average daily TPS of Ethereum.

According to data provided by Blockworks Research through Dune Analytics, rollups save end users and dapp developers over 99% in gas fees compared to the costs of directly deploying code and transacting on Ethereum.

As of June 13, 2023, the cost of sending transactions on the two most popular Ethereum rollups, Optimism and Arbitrum, is about $0.03 to $0.05. However, during times of high on-chain activity and network congestion, these costs can sometimes spike to over $1.

The goal of EIP-4844 is to further reduce rollup costs by introducing blob transactions. Below is a step-by-step description of the blob transaction lifecycle as defined by EIP-4844:

1. The user submits a transaction to the rollup
2. The Rollup sequencer packages transactions into batches
3. The sorter creates a blob transaction, encoding the packaged transaction data into a blob.
4. Blob transaction submitted to Ethereum mempool
5. Validators include blob transactions in blocks
6. The block is broadcast to the network and eventually confirmed.
7. Blob data is stored in the beacon chain for about 3 weeks.
8. Blob data will be deleted after 3 weeks.
9. Rollup can still access the cryptographic commitment of transaction data.

EIP-4844 will not affect how regular transactions are included in blocks, nor will it influence the fee market that determines the price of Ethereum block space, but it does increase the storage requirements for Ethereum blocks. The additional data space is meant to attach blob transactions to the blocks. Blobs are like sidecars that can be attached to Ethereum blocks without affecting or taking up existing block space.

Blob block space will be auctioned according to its own fee market, modeled after the fee market design of EIP-1559. Initially, blob transactions will have almost no cost. Subsequently, for every confirmed block, if more than half of the blob block space is utilized, the cost of blob transactions will increase by 12.5%. For each block where the blob block space is underutilized, the cost of blobs will decrease by 12.5%.

Blob transactions will not be stored indefinitely on Ethereum, but rather on Ethereum's consensus layer (CL), and will be discarded from the CL nodes after three weeks. Proto-danksharding will allow each block to have up to 4 blobs, with each blob capable of containing up to 128kB of additional data. The maximum limit of 512kB blob space per block may change based on the ongoing tests of EIP-4844. Developers are actively discussing the possibility of increasing this limit from 4 blobs to 6.

Approximately 7,094 blocks are generated daily on Ethereum. After EIP-4844, assuming a limit of 4 blobs per block, a maximum of 28,376 blobs can be processed daily. ( This is a theoretical maximum, and due to the dynamic changes in blob fees, it may never be reached in practice. )

In the past six months, the sequencer running on Optimism submitted approximately 3,126 transaction batches to Ethereum daily. The transaction volume confirmed by Arbitrum is about twice that of Optimism. On Optimism, over 90% of the fees come from Layer 1 CALLDATA fees.

The introduction of dedicated data storage space, no matter how small it is at the beginning, is aimed at reducing the cost of using Ethereum as the DA layer for all Ethereum-based rollups. Conservatively, rollup developers expect that after the activation of EIP-4844, rollup fees will decrease by 100% to 900%. However, these estimates may change based on the increase in rollup adoption and activity in the months before and after the activation of the original database.

The cost of blob transactions, although initially cheaper than regular transactions when EIP-4844 is activated, may quickly rise if the number of rollups built on Ethereum increases. Furthermore, while each blob is designed to provide a single sorter the opportunity to publish up to 128kB of data, rollup sorters have the potential to coordinate to make a single blob contain data from multiple rollups. Ethereum developers are aware that due to the limited number of each block, and that a single batch of transactions may not fully utilize the entire 128kB data space of each blob transaction, a secondary market for pricing blobs may emerge.

Proto-danksharding lays the foundation for introducing more advanced technologies to further reduce blob costs without increasing node computational load. Known as complete danksharding, the full vision for blobs is to increase the maximum number of blobs per block from 4 to 64.

Complete Danksharding

Four blobs increase the Ethereum block size by 512kB. Six blobs will further increase the Ethereum block size by an additional 768kB. As mentioned earlier, the extra block space is strictly used for blob transactions and does not permanently store data like regular block space. The full vision of EIP-4844 is to introduce up to 64 blobs for Ethereum while achieving this without significantly increasing the computational load on nodes for block validation. To achieve complete danksharding, Ethereum needs to implement two technologies: Data Availability Sampling (DAS) and erasure coding.

Data Availability Sampling ( DAS )

In the context of validating Layer 2 rollup transactions, the goal of DAS is to ensure that all data fragments bundled by the sequencer have been published on-chain. Full nodes are randomly selected to download a piece of data from the blob and generate a data availability proof. The more times a full node samples the data, the greater the probabilistic certainty that all data has been provided by the sequencer without withholding important data. For nodes, the process of sampling data is less computationally intensive than downloading the entire blob data, but theoretically provides the same data availability guarantees. Similar to proto-danksharding, blob data sampling under full danksharding will ensure that transactions from the sequencer have been verified and published on-chain for any user or network stakeholders to assess.

Through DAS, Ethereum developers are confident in increasing the number and volume of blobs published to Ethereum without adding computational load to the nodes. In addition, developers plan to further reduce the computational load on nodes in future upgrades by implementing suggestions such as historical expiration. According to Ethereum researcher Dankrad Feist, over time, Ethereum will become like "a public bulletin board rather than an archival system," shifting the responsibility of retaining complete copies of transaction history to network stakeholders who frequently use this data, such as Layer 2 rollups and blockchain infrastructure companies like Infura, Alchemy, and Blockdaemon. While EIP-4844 introduces blobs, this is an early example of how all transactions might one day become blob storage on Ethereum.

Erase Encoding

Erasure coding technology enhances the sampling capability of data. If a malicious sorter withholds a few blocks of data, anywhere between 1% and 49% of the blob data, transaction sampling may probabilistically result in certain samples proving to be correct from the outset, rather than incorrect. Erasure coding ensures that if at least half of the blobs are verified, the remaining blobs can be reconstructed. This technology is only effective when data is represented as a polynomial, that is, an expression of two or more algebraic terms. The most common form of erasure coding relies on Reed-Solomon(RS) codes, which are an advanced mathematical formula that can resolve missing data based on sufficient known data fragments. Intuitively, relying solely on sampling may not effectively guarantee the availability of large amounts of data, especially under the assumption that a malicious sorter has withheld a single piece of data from a blob. Erasure coding introduces data redundancy for blobs, so that a malicious sorter would need to withhold a significant portion of blob data to withhold any amount of data.

The coupling of DAS and erasure coding is the foundation of the complete danksharding technology. These technologies are also behind certain DA layers, such as Polygon Avail and Celestia. In many ways, the vision of supporting modular blockchain computation is being tested on a small scale through other blockchain projects, and will be tested at scale on Ethereum, partly through proto-danksharding and rigorously through complete danksharding.

KZG Commitment

Before DAS and erasure coding can be implemented on Ethereum, full danksharding requires the implementation of the KZG commitment scheme. Kate Zaverucha Goldberg(KZG) commitment is a zero-knowledge(ZK) proof system that allows the evaluation of a polynomial without revealing the entire polynomial. Large data objects, such as blobs, can be efficiently manipulated and proven by first representing them as polynomials. EIP-4844 importantly introduces KZG commitments as part of the verification and proof generation process for blobs. For Layer 2 rollups that rely on ZK proofs, KZG commitments can represent the computation of transactions and the state of the rollup protocol. In the context of EIP-4844, KZG commitments can check the properties of blobs without needing to read the entire contents of the blob transactions.

KZG Ceremony

The KZG commitment scheme used for verifying blobs relies on secret values generated once through a trusted setup. Some cryptographic protocols, such as Zcash, Tornado Cash, and Filecoin, depend on trusted setups to securely generate a secret value for repeated on-chain computations. The trusted setup ceremony for EIP-4844 began in January 2023. It has received nearly 100,000 contributions from the Ethereum community. As background, a trusted setup ceremony is a one-time procedure that generates a piece of data for cryptographic protocols from contributions from multiple parties. The goal of combining entropy from multiple contributions in the trusted setup ceremony is to produce a secret value that is nearly impossible to regenerate or guess. Crucially, the most produced from the trusted setup ceremony is the...