Filebase
HomepageFilebase DashboardDiscordBlog
  • 👋Welcome to Filebase!
  • Getting Started
    • FAQ
    • Service Limits
    • Getting Started Guides
      • Pin Your First File To IPFS
      • Developer Quick Start Guide
  • IPFS Concepts
    • What is IPFS?
    • What is an IPFS CID?
    • What is IPFS Pinning?
    • What is an IPFS Gateway?
  • IPFS Pinning
    • Pinning Files
    • Listing Files
    • Deleting Files
    • Event Notifications
  • IPNS Names
    • Managing Names
  • Dedicated IPFS Gateways
    • Managing Dedicated Gateways
    • Access Controls
    • Image Optimizations
    • Adding a Custom Domain
  • API Documentation
    • IPFS RPC API
    • S3-Compatible API
      • Cross Origin Resource Sharing (CORS)
        • Create and Apply a CORS Rule to a Filebase Bucket
        • Deep Dive: Cross Origin Resource Sharing (CORS)
      • Pre-Signed URLs
    • Filebase Platform APIs
    • IPFS Pinning Service API
  • Code Development + SDKs
    • Code Development
      • Apache Libcloud - Python
      • Apostrophe CMS
      • AWS Lambda - NodeJS
      • AWS Lambda - Python
      • AWS SDK - Go (Golang)
      • AWS SDK - Java
      • AWS SDK - JavaScript
      • AWS SDK - .NET
      • AWS SDK - PHP
      • AWS SDK - Python
      • AWS SDK - Ruby
      • Code Examples
        • How To Utilize Filebase with NodeJS
      • DataCamp
      • Dart / Flutter
      • Django
      • Docker
        • Docker Volume Backup
      • Elixir Phoenix
      • Filebase NPM Package
      • Fog.io - Ruby
      • Google App Scripts
      • Integrated Haskell Platform
      • Laravel
      • Nuxt
      • Paperspace
      • Postman
      • Reading a JSON File
      • UNA
      • Unity
      • Uppy AWS S3 Plugin
      • Vue
      • Watcher for NodeJS
      • Webpack S3 Plugin
      • Xamarin
    • SDK Examples: Pinning Files and Folders to IPFS
      • AWS SDK for .NET
      • AWS SDK for Go
      • AWS SDK for JavaScript
      • AWS SDK for PHP
      • AWS SDK for Python
      • AWS SDK for Ruby
  • Archive
    • Content Archive
      • IPFS Getting Started Guide
      • Web Console Getting Started Guide
      • IPFS Tools
        • IPFS CLI
        • IPFS Desktop
        • IPFS Pin Sync
        • IPFS Pinning Service API
        • IPFS3up
      • Third Party Tools and Clients
        • Backup Client Configurations
          • AhsayCBS
          • BackupAssist Classic
          • BackupAssist ER
          • BackupNinja
          • BackupSheep
          • Bacula Enterprise Edition
          • CloudBacko
          • CloudBerry Backup
          • Cloudron
          • cPanel
          • Comet
          • Commvault
          • Duplicacy
          • Ghost IPFS Storage Adapter
          • IPFS Pinning GitHub Action
          • JetBackup
          • Kopia
          • MoveBot
          • MSP360 Online Backup
          • oDrive
          • Photos+ Cloud Library
          • qBackup
          • S3 Uploader for GitHub Actions
          • SimpleBackups
          • SnapShooter
          • Strapi Provider Plugin
          • Veeam
          • Wordpress
            • Media Cloud
            • XCloner
          • Zerto
        • CLI Tools
          • Ansible
          • Apache Pulsar
          • AWS CLI
            • How To Delete Data with AWS CLI
            • What is Multipart Upload?
          • Bash
            • Backup Files to IPFS with Bash
            • Laravel Backup with Bash
            • MongoDB Backup with Bash
            • PostgreSQL Backup with Bash
            • Wordpress Backup with Bash
          • cURL
          • Elasticsearch
          • IPFS-CAR
          • IPFScrape
          • IPGet
          • Jenkins
          • JFrog Artifactory
          • Kubernetes
            • Backup and Restore InFluxDB to Filebase with TrilioVault
            • CSI-S3
            • Kasten K10
            • Kerberos Vault
            • Longhorn.io
            • Stash for Kubernetes
            • Velero
          • Litestream
          • Minty
          • MongoDB
          • MoSMB
          • MySQL
          • Next.js .CAR File Upload
          • NFT Image Generator
          • NGINX S3 Gateway
          • Pinning Docker Images to IPFS
          • Pinning Encrypted Files to IPFS
          • PowerShell
            • Calculate the Size of Filebase Buckets using PowerShell
          • Rclone
            • Backing Up DigitalOcean Spaces to Filebase using Rclone
          • Restic
          • S3cmd
          • S3Express
          • S3FS-FUSE
          • S3QL
          • S3Surfer
          • S4cmd
          • SeaweedFS
          • Tableland
        • Content Delivery Networks
          • Bunny CDN
          • CloudFront CDN
          • Fastly CDN
        • File Management Client Configurations
          • Airbyte
          • Arq
          • Astiga
          • AWS Rekognition
          • AWS S3 Manager - iOS App
          • BucketAnywhere for S3 - Android App
          • CentreStack
          • CloudFlyer
          • Cloudfser
          • Couchdrop
          • CrossFTP
          • CyberDuck
            • How To Delete Data with CyberDuck
          • Dropshare
          • Duplicati
          • FileBrowserGO
          • Flexify.IO
          • ForkLift
          • Goofys
          • Joplin
          • LucidLink
          • MASV
          • Matrix Synapse
          • MinIO Gateway for S3
          • Mountain Duck
          • NetDrive
          • Nexfs
          • NextCloud
          • Nodeum
          • ownCloud
          • Plesk
          • Pure Storage FlashBlade
          • RaiDrive
          • S3 Browser
          • ShareX
          • SmartFTP
          • StableBit Cloud Drive
          • Storage Made Easy Enterprise File Fabric
          • WinSCP
        • NAS Device Configurations
          • Buffalo TeraStation
          • Datadobi DobiProtect
          • Netapp ONTAP Select
          • OpenDrives Atlas
          • Synology Hyper Backup
          • TrueNAS CORE
      • Knowledge Base
        • Deep Dives
          • Deep Dive: Blockchains
          • Deep Dive: Decentralized Compute
          • Deep Dive: Decentralized Digital Identity
          • Deep Dive: Decentralized Storage
          • Deep Dive: Erasure Coding
          • Deep Dive: Geo-Redundancy
          • Deep Dive: Metadata
          • Deep Dive: Metaverse
          • Deep Dive: NFTs
          • Deep Dive: Web3
        • Filebase eBooks
        • Filebase One-Pagers
        • Filebase Whitepapers
        • Web3 Tutorials
          • Alchemy
            • Alchemy: Build a dApp That Provides Real-Time Ethereum Transaction Notifications
            • Alchemy: Create a Full-Stack dApp
            • Alchemy: Create a Hello World Smart Contract
            • Alchemy: Create Your Own ERC20 Cryptocurrency
            • Alchemy: Decentralized Youtube
            • Alchemy: How to Create and Mint an NFT using Filebase
            • Alchemy: How to Mint an NFT Using Web3.js
            • Alchemy: Using The Alchemy SDK with NFTs Minted Through thirdweb
          • Agoric
            • Agoric: Create a DeFi dApp Using Agoric That’s Stored on Filebase
          • AirSwap
            • AirSwap: Track NFT Contract Addresses with AirSwap CLI
          • ArcBlock
            • ArcBlock: Running an ArcBlock Blocket Server on IPFS
          • Ankr
            • Ankr: Create a Truffle Project with Ankr and Filebase
            • Ankr: Deploy a Smart Contract on Polygon using Ankr that is backed up to Filebase
          • Avalanche
            • Avalanche: How To Launch a Generative NFT Collection With Avalanche and IPFS
          • Backing Up NFTs
          • Brownie
            • Brownie: Create and Mint an NFT Using Brownie
          • Bueno
            • Bueno: How to Create a Generative NFT Collection with Bueno
          • Cardano
            • Cardano: Submit Cardano Blockchain Transactions with Embedded Metadata Stored on Filebase
          • Ceramic
            • Ceramic: How to Host a Ceramic Node Using Decentralized Storage
          • Create-IPFS-app
          • Cosmos
            • Cosmos: Storing Cosmos Chain Governance Metadata on IPFS
          • DeCommerce
          • Ethereum Name Service
            • ENS: Configure an ENS Domain to use a Filebase IPFS Dedicated Gateway
          • Figment Datahub
            • Figment Datahub and Avalanche: Make an e-Voting dApp Using Figment Datahub, Avalanche, and Filebase
            • Figment Datahub and Celo Network: Create an ERC1155 NFT on the Celo Network using Figment Datahub and Objects Stored on Filebase
          • Flow
            • Flow: How to Create an NFT Marketplace with Filebase + Flow
          • Fauna
            • Fauna: Host an Application on IPFS with IPFS Dedicated Gateways
          • Ganache
            • Ganache: Create a dApp Hosted on IPFS
          • GUN
            • GUN: Create a Decentralized Chat App with GUN and IPFS
          • Hardhat
            • Hardhat: Creating an NFT Contract using Hardhat and Setting NFT Metadata using IPFS on Filebase
          • Harmony
            • Harmony: Deploy an HRC721 NFT on Harmony with IPFS
          • Hosting a Form on IPFS
          • iExec
            • iExec: Using iExec and Tee Worker to Create Apps that Use Datasets Stored on Filebase
          • Infura
            • Infura: Build dApp Frontends with Infura, React, and IPFS
            • Infura: Create an NFT Contract Factory with Metadata stored on IPFS
          • Lens Protocol
            • Lens Protocol: Build a Decentralized Social Media Network Stored on IPFS
          • LIT Protocol
            • LIT Protocol: Create a MintLIT NFT with LIT Protocol and IPFS
          • LivePeer
            • LivePeer: Mint a Video NFT with LivePeer
          • Macrometa
            • Macrometa: Track IPFS Files with Macrometa
          • Mina Protocol
            • Mina Protocol: Create a Simple zkApp with Mina Protocol and IPFS
          • NEAR Protocol
            • NEAR Protocol: Storing Off-Chain Data on IPFS using Filebase
          • NFTPort
            • NFTPort: Create an NFT Collection with NFTPort
          • Ocean Protocol
            • Ocean Protocol: Publish Data NFTs Stored on IPFS using Ocean Protocol
          • Pin Tezos Tokens Tool
          • Polkadot
            • Polkadot: Deploy a Polkadot dApp on Decentralized Storage
          • Polygon
            • Polygon: Building an App NFT With Polygon
            • Polygon: Make a Donation App with IPFS and Polygon
          • Python
            • Generating NFT Metadata with Python
          • QuickNode
            • QuickNode: Create a Token dApp using QuickNode
          • Remix
            • Remix: Create a Web App with Remix to Upload to Decentralized Storage
          • Remix IDE
            • Remix IDE: Creating Interactive NFTs with IPFS and Remix IDE
          • Secret Network
            • Secret Network: Create an NFT on Secret Network with Data Stored on IPFS
          • Stargaze
            • Stargaze: Create an NFT Collection Using IPFS and Stargaze
          • Starknet
            • Starknet: Create a HardHat Project Using A Starknet Plugin Hosted On IPFS
          • Studio 721
            • Studio 721: Create an NFT Collection with Studio 721 and IPFS
          • Solana
            • Solana: Minting an NFT Using Sugar, Candy Machine, and Filebase
          • Subsquid
            • Subsquid: Querying NFT Data using Subsquid and a Filebase IPFS Dedicated Gateway
          • Tailwind CSS
            • Tailwind CSS: Build an Image Gallery App with IPFS and Tailwind CSS
          • Tatum
            • Tatum: How To Mint NFTs on Solana with Tatum
          • Tezos
            • Tezos: Create an NFT on the Tezos Network using IPFS on Filebase
          • thirdweb
            • thirdweb: Build an NFT Loot Box with thirdweb and IPFS
            • thirdweb: Build an NFT Minting Page with thirdweb, IPFS, RainbowKit, and WAGMI
            • thirdweb: Create a Discord Bot That Gives Roles to NFT Holders
            • thirdweb: Create a Gated Website Using NFTs and IPFS
            • thirdweb: Create an NFT Marketplace with thirdweb and IPFS
            • thirdweb: Release an NFT Drop Using thirdweb and IPFS
          • useDApp
            • useDApp: Create a dApp using useDApp and IPFS
          • Unstoppable Domains
            • Unstoppable Domains: Create a Decentralized Website Using Unstoppable Domains and IPFS Folders
            • Unstoppable Domains: Deploy a Decentralized Blog Using Unstoppable Domains, Akash, and IPFS
            • Unstoppable Domains: IPFS Configuration
          • Vultr
            • Vultr: Store Bedrock Minecraft Worlds on Decentralized Storage
            • Vultr: Store Forge Minecraft Worlds on Decentralized Storage
            • Vultr: Store PaperSpigot Minecraft Worlds on Decentralized Storage
            • Vultr: Store Vanilla Minecraft Worlds on Decentralized Storage
          • Waffle
            • Waffle: Deploy a Smart Contract with Waffle That’s Stored on IPFS
          • Walt.id
            • Walt.id: Mint an NFT with Walt.id and Filebase
          • Web3 Toolbox
            • Web3 Toolbox: Building an NFT Drop With Web3 Toolbox
Powered by GitBook
On this page
  • Centralized Storage
  • Decentralized Storage
  • Steps to a Decentralized Storage File Upload
  • Data Integrity
  • Security
  • Reliability
  • Transitioning From Centralized to Decentralized Storage

Was this helpful?

  1. Archive
  2. Content Archive
  3. Knowledge Base
  4. Deep Dives

Deep Dive: Decentralized Storage

Learn about what decentralized storage is, how it works, and how to transition from centralized storage to decentralized storage.

PreviousDeep Dive: Decentralized Digital IdentityNextDeep Dive: Erasure Coding

Last updated 10 months ago

Was this helpful?

To understand decentralized storage, you first need to understand how centralized storage works.

Centralized Storage

Centralized storage is the type of storage that most people currently use in their daily lives. Mobile phones, laptops, and tablets are all forms of centralized storage. The hard drives or SD cards in these devices are forms of centralized storage since everything stored on these devices is stored in one place with one copy.

Data centers are also forms of centralized storage. Data files that are stored on servers housed in data centers are stored in one geographical location on one single server in the data center. Data is not spread amongst different servers within the data center unless explicit data replication is configured, such as RAID, but even then each copy of each data file is stored in one location.

This means that when data from devices such as phones or laptops are backed up to cloud storage, while there might be two copies of the data, they are both stored in two different forms of centralized storage. If something happens to your phone that compromises or destroys the data, then you have to rely on the backup to retrieve your data. If something has happened to that data backup, such as it being destroyed by a fire, natural disaster, or if it's affected by an outage, your data is inaccessible despite the fact you were diligent and backed it up in case of these situations.

The problem with centralized storage is if something happens to it, it’s gone unless you have a backup plan and active backup method. If you’ve lost your phone and never backed it up to iCloud, or if you did and it was backed up once 8 months before you lost the phone, all the data on that phone that’s been added since that backup is gone since it was stored in a centralized location. Then if you go to iCloud and try to retrieve the backed-up data, if it's inaccessible or corrupt you’ve lost not just 8 months of data, but everything.

This is a huge weakness of centralized storage since despite how vigilant you are in backing up your data regularly, it can still be lost if the cloud storage provider is hit by an outage or disaster.

Decentralized Storage

Decentralized storage can be visualized as the backbone of the Web3 ecosystem. To use or view data, it must be stored somewhere that it can be retrieved from. Data must be stored somewhere to be retrieved, edited, viewed, or used. Keeping data in a centralized manner directly goes against the values of Web3, which include decentralization at its core.

Decentralized storage networks are peer-to-peer networks made up of nodes that provide storage resources for the network to use for transactions and data storage. Each node on the network is an individual entity, such as a home computer or a dedicated server, that has been added to the network using unique software. Using this configuration, decentralized storage networks can utilize data that already exists across the globe that is otherwise unused. Since already existing storage is used, there are no additional costs regarding adding new hard drives, building and maintaining data centers, or employing data center employees. As a result, decentralized storage can be offered at a significantly cheaper price than storage provided by a centralized provider.

On a decentralized storage network, files are split into multiple chunks using technology, then stored across the node on the network worldwide. A node never has access to the complete pieces of a file, providing an innate layer of security. The only way for a file to be retrieved is if the user who uploaded the file requests that the file be downloaded. The file is then reconstructed using pieces stored across the globe, then sent to the user to be viewed or downloaded.

The best way to visualize how decentralized storage works is to think about how online orders are processed and shipped.

Say you place an order on the website Chewy, a pet supplies marketplace with warehouses all over the United States. Your order contains three different items; a dog toy, a dog treat, and some dog food. Since there are warehouses all over the country, it's unlikely that each warehouse has all three of these items in stock at the same time. To fulfill your order, each item gets shipped from whatever warehouse has the item in stock.

When you store a file on a decentralized storage network, the file gets broken apart into a number of different pieces. That number varies based on what network you’re using, if you store on the Sia network, the file gets broken into 30 pieces. This process is called erasure coding. Each piece is then individually encrypted with a special algorithm, then stored across the world in a wide variety of locations, which also will vary based on the network you store data on. When you go to access or download your file, it's like when your order gets shipped from Chewy - your file is pieced together from each location it's stored at, then sent to you. But unlike a Chewy order that takes a few days to arrive, your file is ready to access in just a few seconds.

Steps to a Decentralized Storage File Upload

Below are the steps that happen when a user goes directly to a decentralized storage network, such as Sia, themselves.

  • Before a file is uploaded to a decentralized storage network, it is always best practice that the file is encrypted by the user who maintains their own encryption key. This is optional but highly recommended for maximum data security. The user who is uploading the file is known as the ‘renter' since they are considered to be 'renting' the storage from the network.

  • The renter uploads the file to the decentralized storage network. The file is divided into multiple pieces, then expanded with parity blocks according to the network’s erasure coding algorithm configuration.

  • Each piece of a file is known as a shard, which are then stored across multiple nodes on the decentralized network. These nodes are referred to as ‘providers’ or ‘farmers.

  • Each provider that receives a data shard is unable to access or view the content of that shard. This provides protection for the data if a bad actor compromises the node.

  • When the user requests their file be downloaded, the data gets reconstructed using the minimum number of shards determined through the erasure coding algorithm. Each provider node sends the shard that they store, which is authenticated using the network’s hash table and the user’s credentials which are often in the format of an access key pair.

  • Lastly, the user decrypts their file with their encryption key.

Data Integrity

When the file is broken apart into multiple pieces and erasure-coded, not all of those pieces are required to reassemble the file to be accessed or downloaded. This is a feature that’s in place specifically to ensure data integrity. For example, on the Sia network where files are broken into 30 pieces, only 10 of those chunks are needed to be pieced together to access or download that file. That means that ⅔ of the file chunks can be offline, corrupt, or otherwise inaccessible, but you can still access your file. You won’t even know that there are offline or corrupt chunks, it won’t change how you access your file at all. No file can be accessed without the minimum number of other pieces, which only you can access due to that special algorithm that gets applied during the erasure coding process, so there's no concern about the owner of the node that stores a piece of the file being able to access the file.

So going back to our Chewy example, this is like when you place an order, but one of the items in your order has been inventoried incorrectly, so when the warehouse staff goes to pull the item, it's actually out of stock there. The warehouse staff routes your request to another warehouse that has the item, so the item still gets delivered to you. They don’t cancel your entire order for a missing item, they simply send it from another warehouse. It’s the same concept with decentralized storage - if one file chunk can’t be accessed, the file can still be accessed and downloaded with no interruption to you, they just use a chunk stored in another location to piece together the file and send it to you.

Security

One of the biggest advantages is the innate security that comes with decentralized storage. Since each file is erasure coded, encrypted, and stored across the globe, and you need at least ⅓ of the pieces to access the file, files are secure not only in integrity and accessibility but also in data privacy. No one can access the data file’s chunks besides the user who uploaded them. The exception to this is data stored on IPFS. This is because data stored on these networks is public by default since IPFS files are accessed with their content identifier through an IPFS gateway address.

Reliability

Another advantage of decentralized storage is reliability. Since decentralized storage is just that, decentralized, it has no single point of failure that would take the network down and make the data inaccessible. This means there are no more crippling outages that disrupt workflows or result in a loss of business.

Transitioning From Centralized to Decentralized Storage

In the past, it has been hard to transition from centralized to decentralized storage, but Filebase is intended to be an easy on-ramp for everyone to make the transition from centralized to decentralized storage. Traditionally, users would have to manage things like contracts and cryptocurrency to use decentralized storage. At Filebase, we manage all of that for you and give you the ability to store data across different decentralized networks. We currently support IPFS and Sia. Filebase also doesn’t impose any restrictions that you’d face when storing directly on any of these decentralized networks, such as minimum file size or data retention limitations.

Filebase is the first S3-compatible decentralized storage platform, which means that almost any product, tool, or piece of code that works with Amazon S3 can be configured with Filebase with extreme ease, which makes the transition for developers and enterprises super seamless, but also for the everyday user. You can use Filebase from our easy-to-use web dashboard, or you can configure your favorite backup tool to point to Filebase. So before Filebase, the transition was hard, but today, we aim to be the on-ramp to help transition from Web2 centralized storage to Web3 decentralized storage.

erasure coding