Apache Pulsar
Learn how to configure Apache Pulsar for use with Filebase.

What is Apache Pulsar?

Apache Pulsar is a high-performance, multi-tenant solution for server-to-server messaging. Pulsar is highly scalable and uses a simple client API that integrates with Java, Go, Python, and C++. Pulsar’s tiered storage feature offloads data to long term storage such as S3-compatible
Read below to learn how to use Apache Pulsar with Filebase.

Prerequisites:

  • Download and install Apache Pulsar.
  • Download and install the Apache Pulsar Offloader package.
  • Sign up for a free Filebase account.
  • Have your Filebase Access and Secret Keys. Learn how to view your access keys here.
  • Create a Filebase Bucket. Learn how to create a bucket here.

1. Navigate into your Apache Pulsar installation directory. Open the ./conf/standalone.conf and enter the following lines:

managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=FILEBASE_BUCKET_NAME
s3ManagedLedgerOffloadRegion=us-east-1
s3ManagedLedgerOffloadServiceEndpoint=https://s3.filebase.com
Replace the following value to match your configuration:
  • FILEBASE_BUCKET_NAME: Filebase Bucket Name

2. Set the following configurations in the ./conf/pulsar_env.sh file:

export AWS_ACCESS_KEY_ID=FILEBASE_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=FILEBASE_SECRET_KEY

3. Start Pulsar standalone with the following command:

./bin/pulsar standalone -a 127.0.0.1

4. Configure a data retention policy with the following command:

./bin/pulsar-admin namespaces set-retention public/default --size -10G --time 3d

5. Produce data using pulsar-perf:

./bin/pulsar-perf produce -r 1000 -s 2048 test-topic

6. Check the pulsar-admin status with the following command:

./bin/pulsar-admin topics stats-internal test-topic

7. Trigger the offload process with the command:

./bin/pulsar-admin topics offload --size-threshold 10M public/default/test-topic

8. Check the offloading operation status with the command:

./bin/pulsar-admin topics offload-status -w public/default/test-topic
If you have any questions, please join our Discord server, or send us an email at [email protected]
Last modified 1mo ago
Copy link
On this page
What is Apache Pulsar?
1. Navigate into your Apache Pulsar installation directory. Open the ./conf/standalone.conf and enter the following lines:
2. Set the following configurations in the ./conf/pulsar_env.sh file:
3. Start Pulsar standalone with the following command:
4. Configure a data retention policy with the following command:
5. Produce data using pulsar-perf:
6. Check the pulsar-admin status with the following command:
7. Trigger the offload process with the command:
8. Check the offloading operation status with the command: