What is IPFS Pinning?

Learn more about IPFS pinning and how to pin data to IPFS

IPFS pinning refers to the process of specifying data to be retained and persisted on one or more IPFS nodes. Pinning assures that data is accessible indefinitely, and will not be removed during the IPFS garbage collection process.

Understanding the Garbage Collection Process

When files and data are stored on the IPFS network, nodes on the network cache the files that they download and keep those files available for other nodes on the network. Since storage on these nodes is finite, the cache for each node must be cleared periodically to make room for new files to be cached and made available. The process of clearing the cache for IPFS nodes is referred to as the IPFS garbage collection process.

Garbage collection is an automatic process that is used to manage resources, such as IPFS node disk space. The process is designed to remove cached data that it thinks is no longer needed, though if your IPFS CID refers to a file that is vital to your workflow or environment, having this file removed can be detrimental. So how do you prevent your file from being removed during this process?

IPFS Pinning

To protect data from the garbage collection process, data must be pinned on the IPFS network. This ensures that data is retained indefinitely and is always accessible. Pinning is useful for a variety of workflows, such as accessing data files from around the world without managing sharing permissions. All data uploaded to IPFS is public by default since all you need to access it is the file’s CID. There are no permissions, user accounts, or other security settings tied to IPFS CIDs.

One of the most popular workflows utilizing IPFS pinning right now are NFT collections. These collections have a variety of files involved, including NFT image files and their associated metadata files. If these files aren’t pinned on IPFS and they get removed by the IPFS garbage collection process, this can result in an “NFT Rug Pull”, which means that the NFT ceases to exist, and is no longer accessible or transferable.

Pinning Services

IPFS pinning can be configured on locally hosted IPFS nodes, but for external, long-term storage, that’s where pinning services such as Filebase come in.

How To Pin New Data on IPFS With Filebase

Files uploaded to an IPFS bucket on Filebase are automatically pinned to IPFS and stored with 3x replication across the Filebase infrastructure by default, at no extra cost to you. This means your data is accessible and reliable in the event of a disaster or outage, and won't be affected by the IPFS garbage collection process.

Uploading a File to IPFS Through The Filebase Web Dashboard

1. Start by clicking on the ‘Buckets’ option from the menu to open the Buckets dashboard.

2. Select your IPFS Bucket.

3. After clicking on the bucket name, you will see any previously uploaded files. To upload another file, select 'Upload', then select 'File' from the options.

4. Select the file you want to upload to the IPFS.

5. Once uploaded, you will be able to view and copy the IPFS CID from the 'CID' category, as seen below.

Uploading a Folder to IPFS Through The Filebase Web Console

1. Start by clicking on the ‘Buckets’ option from the menu to open the Buckets dashboard.

2. Select your IPFS Bucket.

3. After clicking on the bucket name, you will see any previously uploaded files. To upload another file, select 'Upload', then select 'Folder' from the options.

4. Select the folder you'd like to upload to IPFS.

5. Once uploaded, the folder will look similar to IPFS individual files.

6. Copy the IPFS CID for your folder, then navigate to https://ipfs.filebase.io/ipfs/[CID]. The contents of your folder will be listed.

Uploading a File to IPFS Using the S3-Compatible API

If you're using the S3 compatible API, the CID will be returned in the response of a PutObject call.

For example, if we run the following AWS CLI command:

aws --endpoint https://s3.filebase.com s3 cp test-images/7FIMFhlMf6A.jpg s3://ipfs-test --debug

For more information on AWS CLI, see here.

The response is shown below. For convenience, we've highlighted the respective response header:

You can also call the HeadObject API to fetch the CID at any time as well:

aws --endpoint https://s3.filebase.com s3api head-object --bucket ipfs-test --key 7FIMFhlMf6A.jpg

Uploading a Folder to IPFS Using IPFS Desktop then Pinning it Using Filebase

1. Start by downloading the IPFS Desktop GUI client, or navigating to the IPFS Web UI.

2. Select 'Files' from the left side bar menu.

1. Select 'Import'

1. Select 'Folder' from the list of options.

1. Select the folder you'd like to upload. Once it has been imported into IPFS Desktop, select the three dots to the right of the screen.

1. From the list of options, select 'Copy CID'.

1. Next, navigate to the Filebase Web Console dashboard. Click on the ‘Buckets’ option from the menu to open the Buckets dashboard.

8. Once at the Buckets dashboard, create a new IPFS bucket by clicking the ‘Create Bucket’ option in the top right corner.

9. Enter a bucket name and choose the IPFS network.

Bucket names must be unique across all Filebase users, be between 3 and 63 characters long, and can contain only lowercase characters, numbers, and dashes.

10. Then select your new IPFS Bucket.

11. After clicking on the bucket name, select 'Upload' from the top right corner, then select 'CID'.

Pin by CID is a paid feature that requires a paid Filebase IPFS subscription plan.

12. Then enter your IPFS CID that you copied from IPFS Desktop and a custom human-readable name to associate with your CID.

13. Select 'Search and Pin' to pin your CID to IPFS through Filebase.

Note: The IPFS network is large and it may take some time for Filebase's IPFS nodes to locate and fetch the CID.

Uploading a Folder to IPFS Through the S3-Compatible API

A CAR file is a type of compressed archive file that contains multiple files, similar to a ZIP file. CAR files are used by the FileCoin and IPFS networks, which utilize the metadata fields to include the IPFS CIDs of each file within the archive.

Filebase supports uploading CAR files to IPFS using the S3-compatible API through the PutObject or MultipartUpload methods.

For instructions on how to create a CAR file, check out our guide here:

IPFS-CAR

When you upload a CAR file, you can pass an S3 metadata header of x-amz-meta-import: car along with the request. Filebase will then import the file as a CAR, and return the resulting CID.

Below is an example using the AWS CLI, which adds the x-amz-meta prefix automatically:

aws --endpoint https://s3.filebase.com s3 cp pictures.car s3://ipfs-test --debug --metadata 'import=car'

With the specified –debug flag, the response headers are shown, which reveal the CID:

2022-06-14 19:51:41,400 - ThreadPoolExecutor-0_2 - botocore.parsers - DEBUG - Response headers: 
{
    'Date': 'Tue, 14 Jun 2022 23:51:41 GMT', 
    'Content-Type': 'application/xml', 
    'Transfer-Encoding': 'chunked', 
    'Connection': 'keep-alive', 
    'x-amz-meta-cid': 'bafybeieurpeyzighqrvwjqyj3szuvbqvrbijm7cdair5a422ipf2d5qnlq', 
    'ETag': 'W/"d8cad258a3d9bbe03a26a13a3ec43b21"',
    'x-amz-request-id': '144e0415-8162-45cd-b071-e51dada956ae', 
    'x-amz-id-2': 'ZmlsZWJhc2UtNmQ3ZjQ5OGZmZC14ejk1Mg==',
    'Server': 'Filebase'
}

How to Re-Pin Existing IPFS CIDs With Filebase

Re-Pinning Existing IPFS CIDs from the Filebase Web Console

Pin by CID is a paid feature that requires a paid Filebase IPFS subscription plan.

1. Start by clicking on the ‘Buckets’ option from the menu to open the Buckets dashboard.

2. Once at the Buckets dashboard, create a new IPFS bucket by clicking the ‘Create Bucket’ option in the top right corner.

3. Enter a bucket name and choose the IPFS network.

Bucket names must be unique across all Filebase users, be between 3 and 63 characters long, and can contain only lowercase characters, numbers, and dashes.

4. Then select your new IPFS Bucket.

5. After clicking on the bucket name, select 'Upload' from the top right corner, then select 'CID'.

6. Then enter your IPFS CID and a custom human-readable name to associate with your CID.

7. Select 'Search and Pin' to pin your CID to IPFS through Filebase.

Note: The IPFS network is large and it may take some time for Filebase's IPFS nodes to locate and fetch the CID.

Setting Filebase as a Pinning Service in the IPFS Desktop GUI or IPFS Web UI and Importing Existing CIDs

1. Start by downloading the IPFS Desktop GUI client, or navigating to the IPFS Web UI.

2. From the left side-bar menu, select 'Settings'.

3. Select 'Add Service' under 'Pinning Services'.

4. Select 'Filebase' when prompted to select a pinning service provider.

5. Enter the following configuration information to add Filebase as your pinning service provider:

• Secret Access Token: Navigate to the Filebase Access Keys page, then view the IPFS Pinning Service API Endpoint. Click the drop down menu for 'Choose Bucket to Generate Token', then choose the IPFS Filebase Bucket you want to use.

Then copy the generated Secret Access Token:

6. Select 'Save'. Your Filebase pinning service configuration will now be listed under 'Pinning Services'.

7. Next, select 'Files' from the left side-bar menu, then select 'Import'.

8. Select 'From IPFS'.

9. Input your IPFS CID, appended with /ipfs/ and provide a name to associate with your CID, then select 'Import'. This CID can be for an individual file or a folder.

10. Once imported, your IPFS asset will be listed. Select the 3 dots from the right side menu to bring up options to interact with your IPFS asset.

11. Select 'Set pinning'.

12. Select the check box next to 'Filebase', then select 'Apply'.

13. The 'Pin Status' for this IPFS asset will now show an icon with a cloud and a thumbtack, indicating it has been pinned to the configured pinning service.

14. Navigate to the Filebase web console, then select the bucket you used to generate your Secret Access Token earlier, and you will see your pinned IPFS asset, which will show the status 'Pinning', meaning it is in the process of being pinned to IPFS through Filebase. Once finished, this will read 'Pinned'.

Re-Pinning Existing IPFS CIDs with Filebase Using the IPFS CLI

1. Start by downloading and installing the Go-IPFS CLI tool.

2. Next, add Filebase as a pinning service with the following command:

ipfs pin remote service add Filebase https://api.filebase.io/v1/ipfs SECRET-ACCESS-TOKEN

If you intend to use more than one bucket, you will need to repeat this command with the secret access token for each bucket. In this workflow, the nickname should be Filebase(Bucket-Name) in order to identify different buckets.

Replace the following value to match your configuration:

• SECRET-ACCESS-TOKEN: Navigate to the Filebase Access Keys page, then view the IPFS PInning Service API Endpoint. Click the drop down menu for 'Choose Bucket to Generate Token', then choose the IPFS Filebase Bucket you want to use.

  

  Then copy the generated Secret Access Token:

  

3. Use the following command to pin an existing IPFS CID to Filebase:

ipfs pin remote add --service=Filebase --name=ASSET-NAME IPFS-CID

Replace ASSET-NAME with your desired human-readable value to be associated with your pinned CID, and replace IPFS-CID with the CID of your IPFS asset. This can be either a file or a folder.

ipfs pin remote add --service=Filebase --name=my-nft-collection bafybeif5x2gihgepga2vtjlx3lfdwgxquo2sbv4a77rsedokf6poqrrcse

4. Navigate to the Filebase web console, then select the bucket you used to generate your Secret Access Token earlier, and you will see your pinned IPFS asset, which will show the status 'Pinning', meaning it is in the process of being pinned to IPFS through Filebase. Once finished, this will read 'Pinned'.

Your terminal will also return the following output:

CID: bafybeif5x2gihgepga2vtjlx3lfdwgxquo2sbv4a77rsedokf6poqrrcse
Name: my-nft-collection
Status: pinned

How to Retrieve Pinned CIDs Using The AWS S3 SDK for JavaScript

You can also upload and retrieve the IPFS CID using different AWS S3 SDKs. The example below uses the JavaScript SDK:

AWS SDK v2:

const AWS = require('aws-sdk');
const fs = require('fs');

const s3 = new AWS.S3({
    endpoint: 'https://s3.filebase.com',
    region: 'us-east-1',
    signatureVersion: 'v4',
});

fs.readFile('image.png', (err, data) => {
    if (err) {
        console.error(err);
        return;
    }
    
    const params = {
        Bucket: 'my-ipfs-bucket',
        Key: 'test/image.png',
        ContentType: 'image/png',
        Body: data,
        Metadata: { import: "car" }
    };
    
    const request = s3.putObject(params);
    request.on('httpHeaders', (statusCode, headers) => {
        console.log(`CID: ${headers['x-amz-meta-cid']}`);
    });
    request.send();
});

AWS SDK v3:

const command = new PutObjectCommand({...});

command.middlewareStack.add(
  (next) => async (args) => {
    // Check if request is incoming as middleware works both ways
    const response = await next(args);
    if (!response.response.statusCode) return response;

    // Get cid from headers
    const cid = response.response.headers["x-amz-meta-cid"];
    console.log(cid);
    return response;
  },
  {
    step: "build",
    name: "addCidToOutput",
  },
);

const res = await client.send(command);