Creating a DLP is the best way for highly motivated builders to get involved and rewarded. It takes a few hours to deploy a template, and 1-2 days to modify the template for your chosen data source.
The Data Liquidity Layer is where data is contributed, validated, and recorded to the network into data liquidity pools (DLPs). Here, DLP creators deploy DLP smart contracts with specific data contribution objectives, including the DLP’s purpose, validation method, and data contribution parameters. DLPs on Vana are analogous to subnets on Bittensor.
The top DLPs earn data liquidity pool rewards, so DLP slots are competitive. There 16 slots available on mainnet. This limit is intended to incentivize quality over quantity.
To register your DLP or provide updates to be added to the leaderboard, submit here.
To see other DLPs, view the DLP Leaderboard
Join our Discord server to present your DLP and chat with other devs creating DLPs
A data liquidity pool consists of the following components:
A DLP smart contract that issues rewards and communicates back to the connectome.
A proof-of-contribution function to measure data quality. This is run by the validators
An optional UI for data contributors to add data to a liquidity pool
A validator runs a proof-of-contribution function that measures data quality. There are two kinds of Validators that DLPs can use, TEE Validators (recommended) and DLP Validators.
Validators use of the vana framework to interact with all the components of the Vana network. A starting point can be found here:
Vana framework with core components: https://github.com/vana-com/vana-framework
Template for creating DLP validators https://github.com/vana-com/vana-dlp-hotdog
Data value and validation depends on the data source. DLPs have the flexibility to validate data according to their unique requirements. Regardless of the specific verification methods employed, a DLP should strive to ensure that each data point passes the following criteria, which is implemented through a proof-of-contribution function:
Meaningfulness: The data should contribute novel and valuable insights to the DLP's overall dataset. It should enhance the richness and diversity of the available information.
Validity: The data must adhere to a schema that is consistent with the existing data within the DLP. Conforming to a standardized structure ensures compatibility and facilitates efficient data processing.
Authenticity and Contributor's rights: The data must be genuine and, if required, originate from a legitimate account. This ensures the integrity and reliability of the information. The data contributor submitting the data must have the necessary rights and permissions to do so.
This proof-of-contribution function checks data validity and maps the diverse data submissions to a score that can be used to reward fungible DLP-specific tokens. A template is included in the DLP template to make these easier to implement.
See the ChatGPT DLP for a detailed example of proof-of-contribution
A DLP smart contract is responsible for recording consensus values among network nodes and can help orchestrate how proof-of-contribution works in the DLP. You can use the template below, swapping out the DLP token to something specific to your DLP (example: $TDAT for Twitter data).
The Smart Contract repo contains a DLPT (DLP Token) contract, that is an ERC20 token with additional features such as minting permissions, an admin role, and a blocklist for addresses. It leverages OpenZeppelin's ERC20, ERC20Permit, ERC20Votes, and Ownable modules for extended functionality.
The Data Liquidity Pool smart contract is designed to manage the registration of validators. This contract ensures that validators can participate in maintaining the network's integrity and security while earning rewards for their contributions.
Some DLPs use a UI for data contributors to upload their data to a DLP. It's up to the DLP to decide what is best for the DLP's contributors to be able to write data transactions onchain. A starting point for a UI, which implements client-side encryption, is here:
Generic UI for uploading data to DLPs: https://github.com/vana-com/vana-dlp-ui gpt data dao: https://www.gptdatadao.org/ reddit data dao: https://rdatadao.org/
Other DLPs are using chrome extensions or scrapers for users to contribute data.
Now that you're familiar with the components of a DLP, here's a conceptual step-by-step guide to launch a DLP of your own.
Follow the ChatGPT DLP tutorial for a step-by-step guide on creating a DLP
Choose a valuable data source.
Current community requests: stack overflow, telegram, youtube, email, google drive
To see what others are building, visit the DLP Leaderboard
Tell users how to access that data source.
For example, through a GDPR or CCPA data export request, through scraping their own data, or through an API.
Optionally, build an UI that makes it easy for data contributors to add their data via your smart contract.
Implement a DLP validator template to validate and value data.
Data validation and value depends on the data source.
Once decided, implement your incentives and validation checks.
We recommend rolling this out iteratively in phases to test the incentives.
Register as a DLP creator. This requires meeting a minimum staking threshold.
Deploy your DLP smart contract.
Start up a DLP validator and recruit other existing validators to participate in the DLP as well.
Congrats, you're up and running!
To qualify for block rewards, get voted into the top 16 DLP slots based on data value and participation.
Satori Testnet is for Testing Purposes Only
Ensure you create a different configuration for data validation on testnet versus mainnet.
Once a DLP has been built, we are ready to deploy it. DLP nodes form a peer-to-peer network, and can be deployed to any infrastructure of your choice (ex, AWS, Google Cloud Platform, Azure, etc). Ideally, nodes have a static IP to more easily communicate with each other. Each node must have a wallet used to register the node to the DLP.
Once a DLP Smart Contract is deployed to the Vana network, reach out to us and we will register this contract as a DLP. We will be manually approving DLP registration at the beginning of the testnet before moving to voting-based approval.
Clone the DLP smart contract from a template, modify it for the needs of your specific DLP, and deploy it to the network following the readme.
A DLP Smart contract will be able to accept new nodes into the DLP. To register a new node, create a wallet for it and call the registration function in the DLP smart contract. More information can be found in Become a DLP Validator.
Once DLP nodes are registered, they must be deployed to a location where they can be reached by other DLP nodes. This can mean deploying to your favorite cloud vendor of choice or running a node on your own hardware and setting up appropriate networking rules. Once the node is running, they can begin serving proof-of-contribution requests within the DLP.
The Vana network utilizes independent Trusted Execution Environment (TEE) validators to perform data validation requests for DLPs, simplifying their architecture by eliminating the need for DLPs to operate their own validator nodes. Each TEE validator node is a sealed, encrypted virtual machine running on an independently operated server with a bare metal install of Intel SGX, an advanced hardware-level isolation technology.
This setup ensures that neither the node operators nor any other parties can access the interior of the processor or the data being processed. The TEE environment guarantees that deployed programs within it are immutable and private, maintaining operational integrity and preventing unauthorized changes while processing user data.
Verify user data by running Proof-of-Contribution to assess data legitimacy and value.
Attest to the validity of the data and write the attestation back on-chain.
TEE Validators are a work in progress. Code samples and technical documentation will be added soon.
A detailed data flow diagram of how TEE Validators operate can be found here.
DLPs may choose to rely on a network of DLP Validators to run their DLP's proof-of-contribution (PoC). After running PoC, these validators form a consensus with each other and write the proof-of-contribution assessment back on-chain. In this model, DLPs are responsible for deploying and maintaining their validators. DLP Validators earn DLP token rewards for accurate and consistent data evaluations.
Verify user data within DLPs according to standards set by DLP owners.
Use Vana’s Proof-of-Contribution system to assess data legitimacy and value.
Attest to the validity of the data and write the attestation back on-chain
Participate in the Nagoya Consensus to ensure consistent and accurate data scoring.
Perform accurate and consistent data evaluations and disincentivize bad actors through slashing.
Evaluate the performance of other validators to maintain network integrity.
Back evaluations with personal stake to ensure accuracy and reliability.
Respond to queries from data consumers, including decrypting data and validating results.
Each DLP owner is responsible for deploying a smart contract specific to the DLP's needs. We provide contract templates that offer a starting point for registering DLP validators, recording and verifying data transactions written on-chain, and validators reaching consensus through Nagoya consensus. We also provide a template implementation for the corresponding validators.
The provided templates include:
A smart contract: https://github.com/vana-com/vana-dlp-smart-contracts
A sample validator node (transacts with contract): https://github.com/vana-com/vana-dlp-hotdog
The Vana framework (used by validators): https://github.com/vana-com/vana-framework
The Vana Framework is a library designed to streamline the process of building a DLP.
This object encapsulates interactions with the blockchain.
The state contains information about the current state of the DLP, including nodes in the network (and how they can be reached), scores of the nodes, when it was last synced, current block number, etc.
The Vana framework provides a node abstraction that simplifies the creation and management of a peer-to-peer network that operates the DLP.
A node is a network participant responsible for validating, querying, scoring, or performing any arbitrary task necessary for the DLP to perform proof-of-contribution. A node can be a validator tasked with ensuring a data point belongs to the data contributor and is not fraudulent. A node can also be a miner responsible for aggregating data points to respond to a data query. A DLP is responsible for defining who the DLP participants are, and how they're incentivized for good behavior and penalized for bad.
Nodes can communicate with each other by encapsulating information in a Message object, and sending that object back and forth using a client-server relationship over HTTP.
A NodeClient is responsible for building the inputs of a Message object, and sending it to one or more NodeServers.
The NodeServer runs a FastAPI server that is responsible for responding to API requests sent from a NodeClient. It will perform a task, then fill the outputs of the Message object and send it back to the NodeClient that requested it.
The Messages object is sent back and forth between nodes, providing a vehicle for communication between nodes. It wraps the inputs and outputs of a communication exchange sent between nodes.
Writer's note: We had to design a new consensus mechanism to handle the fuzziness of data contributions. For example, if I believe your data deserves a score of 100, and another validator believes your data deserves a score of 102, we could both be pretty much right. Neither of us as validators are acting maliciously or incorrectly. But generally crypto consensus mechanisms are designed for exact consensus only. Bittensor proposed an early version of this fuzzy consensus, which we have modified to work for private data and proof of contribution.
To reach a state of agreement on data contributions and disincentivize malicious validators, the Proof-of-Contribution system employs Nagoya Consensus. In Nagoya Consensus, each DLP Validator expresses their perspective on the quality and value of data contributions as a rating. Validators then use their rating to score other validators through a set of ratings weighted by stake.
Nagoya Consensus rewards validators for producing data contribution scores that are in agreement with the evaluations of other validators. This disincentivizes divergence from the consensus majority while incentivizing validators to converge on honest assessments of data contribution.
By requiring validators to put stake behind their evaluations and rewarding convergence weighted by stake, Nagoya Consensus makes it economically unfavorable for even a significant minority of validators to collude and manipulate the state of the DLP. As long as an honest majority of stake-weighted validators participate, the system can come to consensus on data contribution scores that accurately reflect the quality and value of data in the DLP.
Validate user data for specific Data Liquidity Pools
Start to validate user data for specific Data Liquidity Pools.
Please join our Discord to get an overview of all existing DLPs and how to access them.
If you need to meet specific minimum staking requirements for a DLP, please reach out to the community in our #testnet-help channel for support as well.
You can run a validator on your own hardware or on a cloud provider like GCP and AWS, ensuring the quality of data in the pool and earning rewards accordingly.
Minimum hardware requirements: 1 CPU, 8GB RAM, 10GB free disk space
See example integration of a Validator here.
Choose the DLP you'd like to run a validator for.
You can run validators in multiple DLPs
Register as a validator through the DLP via its smart contract.
You must meet the minimum staking requirements for the DLP
Wait for your registration request to be approved by the DLP.
Run the validator node specific to the DLP. Confirm that your validator is running correctly. Your logs should look something like this, which will vary by DLP:
See DLP-specific instructions for running a validator node
Congratulations, your validator is up and running! You can keep track of your stats and trust score by looking onchain.
Validators earn rewards for validating uploaded files. For a given data validation request, each validator scores data based on metrics that are relevant to the data type. The scores are aggregated and written onchain, but how does the DLP decide how to reward its validators?
Every 1800 blocks (~3 hours), a DLP epoch concludes and the DLP contract sends a chunk of rewards to its validators. The precise amount of rewards a given validator receives depends is determined by the Nagoya consensus process.
In Nagoya consensus, each validator submits a score for every other validator to the DLP smart contract. Validators score each other based on the quality of their assessments and their operational performance. For instance, if a validator is perceived as wrongly giving an uploaded file that appears fraudulent or low quality a high score, it may receive low scores from other validators.
Somewhat more formally: a validator peer's emissions amount is based on calculations for their rank and consensus. It is calculated by multiplying these two scores, integrating both the peer's individual valuation by the network (rank) and the collective agreement on that valuation (consensus). This multiplication ensures that emissions are allocated in a manner that considers both the quality of contributions and the degree of communal support for those contributions.
When an epoch concludes, the consensus process converts the most recent validator scores into a distribution for that epoch's emissions, and the rewards are distributed to the validators accordingly. Low-ranking validators quickly realize fewer rewards for their contributions, ensuring that an honest majority of validators is able to out-earn dishonest actors and therefore uphold the integrity of the DLP over time.
