4. Create Data Refiner

Now we will define a Refiner for your DataDAO's data. This is a job that will turn each data contribution into a queryable libSQL database, pinned on IPFS, that the Vana Protocol can use. Like Proof-of-Contribution jobs, refiners run in secure compute environments.

1. Fork the Data Refinement Template
   1. Fork the template repository in GitHub here.
   2. Open the Actions tab for your fork and enable GitHub Actions (if prompted).
   3. In a local terminal, download your fork, replacing YOUR_GITHUB_USERNAME:

git clone https://github.com/YOUR_GITHUB_USERNAME/vana-data-refinement-template.git
cd vana-data-refinement-template

2. Create.env

# Get this key from chain (see next step)
REFINEMENT_ENCRYPTION_KEY=0x...

# Optional: IPFS pinning for local test
PINATA_API_KEY=your_key
PINATA_API_SECRET=your_secret

Where do I get REFINEMENT_ENCRYPTION_KEY?

1. Navigate to the dlpPubKeys(dlpId) method on Vanascan.
2. Input your dlpId that you saved in 2. Register DataDAO , and copy the returned key.
3. If the result is an empty string "", wait a few minutes and try again. If you still get an empty string after 30 minutes, please open a support ticket in the Vana Builders Discord.

Where do I get PINATA_API_KEY and PINATA_API_SECRET?

1. Go to pinata.cloud and log in.
2. In the sidebar, click "API Keys"
3. Click "New Key"
4. Turn on the Admin toggle, give it a name (e.g. "test"), and click Create
5. Copy the generated API Key and Secret into your .env

⚠️

Tip

These .env parameters are only used for local testing — in production, the actual values will be injected automatically when the Contributor UI calls Satya via /refine method.

You're doing this now to generate and preview the schema your Refiner produces — this is what data consumers will later use to query your DataDAO’s data.

3. Update Schema Metadata

   Before running the refinement test, make a quick edit to your schema’s metadata — this is required to trigger the GitHub pipeline later.

   • Open refiner/config.py
   • Edit SCHEMA_NAME to something like: "Google Drive Analytics of QuickstartDAO"

4. Run a Local Test

docker build -t refiner .
docker run \
  --rm \
  -v $(pwd)/input:/input \
  -v $(pwd)/output:/output \
  --env-file .env \
  refiner

This will produce two key files in output/:

• db.libsql.pgp: the input data refined into a database, encrypted and pinned to IPFS
• schema.json: table structure for queries

5. Upload Your Schema to IPFS

Now we need to make your schema.json accessible to public.

1. Go to pinata.cloud and log in
2. In the sidebar, click "Files" → "Add" → "File Upload"
3. Upload the schema.json from your local /output directory
4. After the upload completes, click on the file and copy it’s URL

You’ll use this in the next step as the schemaDefinitionUrl when registering your Refiner.

6. Commit Changes

git add .
git commit -m "Set up refiner for the DataDAO"
git push origin main

Then, go to your GitHub repo → Actions tab, and you’ll see how the build pipeline automatically builds your refiner and compresses it into a .tar.gz artifact.

7. Get the Artifact URL

Once the GitHub Action completes:

• Navigate to your repo’s Releases section
• You’ll find a .tar.gz file — this is your compiled PoC image
• Copy the public URL on this file — you’ll need it for the next step as refinementInstructionUrl. Save it for later too, you'll need it in the Contributor UI.

8. Register the Refiner Onchain

Once registered, your schema becomes part of the Vana data access layer, enabling apps to query refined data with your approval.

• Navigate to the addRefiner method on Vanascan.
• Fill in the fields:
  1. dlpId: your dlpId, saved from 1. Deploy Smart Contracts (no need to convert from wei with ×10^18).
  2. name: a label for your Refiner, e.g. "Quickstart Refiner".
  3. schemaDefinitionUrl: the IPFS URL of your schema from step (5) above.
  4. refinementInstructionUrl: the URL to your .tar.gz refiner file from step (7) above.
  5. publicKey: the REFINEMENT_ENCRYPTION_KEY from above.
• Connect your wallet (same as OWNER_ADDRESS)
• Submit the transaction

9. Get Your Refiner ID

After submitting the addRefiner transaction:

• Open your transaction on Vanascan
• Go to the Logs tab
• Find the RefinerAdded event and save your refinerId for the next section.

At this point, you’ve got everything needed to make your data query-ready on the Vana network.