iExec: Using iExec and Tee Worker to Create Apps that Use Datasets Stored on Filebase

Learn how to use iExec and Tee Worker to create apps that use datasets stored on Filebase.

What is iExec?

The iExec network enables decentralized docker app deployment and monetization on the blockchain by connecting cloud resource sellers with cloud resource buyers and encourages an ecosystem of decentralized, autonomous, privacy-preserving applications.

Read below to learn how to use iExec and its Tee Worker Application to create apps that use datasets stored on Filebase.

Prerequisites:

Download and install Docker 17.05 or higher on the daemon and client.
Download and install Nodejs 12.0.0 or higher.
Sign up for a free Filebase account.
Have your Filebase Access and Secret Keys. Learn how to view your access keys here.

1. Install the iExec SDK CLI package with the following command:

npm i -g iexec

2. Create a new Wallet file:

iexec wallet create

You will be prompted to set a password on this wallet file. Take note of this password since there is no way to recover it if forgotten.

3. Initialize your iExec project:

mkdir ~/iexec-projects

cd ~/iexec-projects

iexec init --skip-wallet

4. We will use the iExec testnet, called Viviani, for our project. Next, initialize the default remote storage:

iexec storage init --chain viviani

5. Now initialize your app:

iexec app init

The default app is the public docker image iexechub/python-hello-world.

6. Then deploy this default app on the Viviani network:

iexec app deploy --chain viviani

7. Confirm it was deployed successfully by checking your list of deployed apps:

iexec app show --chain viviani

8. Now, to run your app you will need RLC tokens.

Since we’re running on the testnet, we can get some test RLC with the command:

iexec wallet get-RLC --chain viviani

9. Confirm the transaction was successful by checking your wallet’s balance:

iexec wallet show --chain viviani

10. Now it's time to run your application. Use the following command:

iexec app run --args APP_NAME --watch --chain viviani

Replace APP_NAME with your desired Application name.

11. Download the result of your task with the command:

iexec task show TASK_ID --download my-result --chain viviani

Replace TASK_ID with the Task ID value from the last command’s output.

12. View the contents of your task result file:

unzip my-result.zip -d my-result

cat my-result/result.txt

13. Up until now, we’ve been using the default storage configuration for iExec, which is IPFS managed by iExec.

For private datasets and assets, this method is not recommended, so we’ll now switch to using Filebase for our private storage backend.

14. For this tutorial, we’ll use a sample dataset that we’ll initiate with the command:

iexec dataset init --encrypted

This command makes 3 new directories:

datasets/encrypted
datasets/original
.secrets/datasets

A new section will be appended to the iexec.json file called ‘datasets’.

15. Now we’ll create a sample dataset with the command:

echo "Hello Filebase!" > datasets/original/filebase-test-dataset.txt

16. Then encrypt your dataset:

iexec dataset encrypt

Take note of the checksum value that is returned from this command.

This command created the file datasets/encrypted/filebase-test-dataset.txt.enc. Let’s upload this file to Filebase.

17. You can upload files to Filebase using two methods: the Filebase Web Console, or the Filebase S3-compatible API.

Either method will require an IPFS bucket which must be created from the Web Console.

18. To use the web console, navigate to console.filebase.com.

If you don’t have an account already, sign up, then log in.

Select ‘Create Bucket’ in the top right corner to create a new bucket.

20. Enter a bucket name and choose the IPFS storage network to create the bucket.

Bucket names must be unique across all Filebase users, be between 3 and 63 characters long, and can contain only lowercase characters, numbers, and dashes.

21. Next, select the bucket from your list of buckets, then select ‘Upload’ in the top right corner to upload your files.

22. Select your files to be uploaded.

Once uploaded, they will be listed in the bucket.

23. You can view the IPFS CID from the bucket object list in the CID column, or you can click on each uploaded object to display the metadata for the object to view the IPFS CID.

Take note of this CID, since we will use it in the next step.

24. Next, open the `iexec.json` file, navigate to the dataset section, and edit the following lines:

"dataset": {
    "owner": "0x-your-wallet-address",
    "name": "Encrypted Filebase dataset",
    "multiaddr": "https://ipfs.filebase.io/ipfs/[IPFS_CID]",
    "checksum": "0x0000000000000000000000000000000000000000000000000000000000000000"
  }

Replace the IPFS_CID with the IPFS CID from the uploaded dataset in your Filebase bucket.

25. Deploy your dataset on the network with the command:

iexec dataset deploy --chain viviani

26. Now we need to push the encryption key to the Secret Management Service so that the deployed dataset can be used for authorized applications.

To get the encryption key for this dataset and push it to the testnet, use the following commands:

Note: This workflow is fine for debugging purposes on the testnet, but is not recommended to store secrets for production environments.

sed -i 's|"viviani": {},|"viviani": { "sms": "<https://v7.sms.debug-tee-services.viviani.iex.ec>" },|g' chain.json

iexec dataset push-secret --chain viviani

27. Now let’s deploy an application that uses this dataset.

First, make a new directory for this project, initiate iExec inside this directory, and create some empty files that we’ll need later.

mkdir my-tee-dataset-app && cd my-tee-dataset-app

iexec init --skip-wallet

mkdir src

touch src/app.py

touch Dockerfile

touch sconify.sh

28. In the `src/` folder, open the new file called `app.py` and input the following code:

import json
import os

from pyfiglet import Figlet

iexec_out = os.environ['IEXEC_OUT']
iexec_in = os.environ['IEXEC_IN']
dataset_filename = os.environ['IEXEC_DATASET_FILENAME']

text = ''

# Check the confidential file exists and open it
try:
    dataset_file = open(iexec_in + '/' + dataset_filename, 'r')
    dataset = dataset_file.read()
    text = Figlet().renderText(dataset)
except OSError:
    print('confidential file does not exists')
    exit(1)

print(text)

# Append some results in /iexec_out/
with open(iexec_out + '/result.txt', 'w+') as fout:
    fout.write(text)

# Declare everything is computed
with open(iexec_out + '/computed.json', 'w+') as f:
    json.dump({"deterministic-output-path": iexec_out + '/result.txt'}, f)

29. Then build the TEE docker image by editing the Dockerfile we created previously to reflect the following content:

FROM python:3.7.3-alpine3.10
RUN pip3 install pyfiglet
COPY ./src /app
ENTRYPOINT ["python3", "/app/app.py"]

30. Open the `sconify.sh` file and input the following code:

#!/bin/bash

# declare the app entrypoint
ENTRYPOINT="python /app/app.py"
# declare an image name
IMG_NAME=python-dataset-app

IMG_FROM=${IMG_NAME}:temp-non-tee
IMG_TO=${IMG_NAME}:tee-debug

# build the regular non-TEE image
docker build . -t ${IMG_FROM}

# run the sconifier to build the TEE image based on the non-TEE image
docker run -it \\
            -v /var/run/docker.sock:/var/run/docker.sock \\
            registry.scontain.com:5050/scone-production/iexec-sconify-image:5.3.7 \\
            sconify_iexec \\
            --name=${IMG_NAME} \\
            --from=${IMG_FROM} \\
            --to=${IMG_TO} \\
            --binary-fs \\
            --fs-dir=/app \\
            --host-path=/etc/hosts \\
            --host-path=/etc/resolv.conf \\
            --binary=/usr/local/bin/python3.7 \\
            --heap=1G \\
            --dlopen=2 \\
            --no-color \\
            --verbose \\
            --command=${ENTRYPOINT} \\
            && echo -e "\\n------------------\\n" \\
            && echo "successfully built TEE docker image => ${IMG_TO}" \\
            && echo "application mrenclave.fingerprint is $(docker run -it --rm -e SCONE_HASH=1 ${IMG_TO})"

31. Then run the `sconify.sh` script to build the TEE app:

./sconify.sh

32. Now let’s deploy the TEE app on the iExec network. Start by initializing the application:

iexec app init --tee

33. Get the enclave fingerprint for your TEE image:

docker run -it --rm -e SCONE_HASH=1 python-dataset-app:tee-debug

Take note of this, we will use it in the next step.

34. Then edit the `iexec.json` file to add the following configuration details:

"mrenclave": {
      "provider": "SCONE", // TEE provider (keep default value)
      "version": "v5", // Scone version (keep default value)
      "entrypoint": "node /app/app.py",
      "heapSize": 1073741824, // heap size in bytes (1GB)
      "fingerprint": "ENCLAVE_FINGERPRINT" 
    }

Replace the ENCLAVE_FINGERPRINT value with the enclave fingerprint you took note of in the last step.

35. Deploy the app on the iExec testnet: