Apache Pulsar

Learn how to configure Apache Pulsar for use with Filebase.

What is Apache Pulsar?

Apache Pulsar is a high-performance, multi-tenant solution for server-to-server messaging. Pulsar is highly scalable and uses a simple client API that integrates with Java, Go, Python, and C++. Pulsar’s tiered storage feature offloads data to long term storage such as S3-compatible

Read below to learn how to use Apache Pulsar with Filebase.

Prerequisites:

• Download and install Apache Pulsar.

• Download and install the Apache Pulsar Offloader package.

• Sign up for a free Filebase account.

• Have your Filebase Access and Secret Keys. Learn how to view your access keys here.

• Create a Filebase Bucket. Learn how to create a bucket here.

1. Navigate into your Apache Pulsar installation directory. Open the ./conf/standalone.conf and enter the following lines:

managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=FILEBASE_BUCKET_NAME
s3ManagedLedgerOffloadRegion=us-east-1
s3ManagedLedgerOffloadServiceEndpoint=https://s3.filebase.com

Replace the following value to match your configuration:

• FILEBASE_BUCKET_NAME: Filebase Bucket Name

2. Set the following configurations in the ./conf/pulsar_env.sh file:

export AWS_ACCESS_KEY_ID=FILEBASE_ACCESS_KEY

export AWS_SECRET_ACCESS_KEY=FILEBASE_SECRET_KEY

3. Start Pulsar standalone with the following command:

./bin/pulsar standalone -a 127.0.0.1

4. Configure a data retention policy with the following command:

./bin/pulsar-admin namespaces set-retention public/default --size -10G --time 3d

5. Produce data using pulsar-perf:

./bin/pulsar-perf produce -r 1000 -s 2048 test-topic

6. Check the pulsar-admin status with the following command:

./bin/pulsar-admin topics stats-internal test-topic

7. Trigger the offload process with the command:

./bin/pulsar-admin topics offload --size-threshold 10M public/default/test-topic

8. Check the offloading operation status with the command:

./bin/pulsar-admin topics offload-status -w public/default/test-topic