Apache Pulsar

Learn how to configure Apache Pulsar for use with Filebase.

What is Apache Pulsar?
Apache Pulsar is a high-performance, multi-tenant solution for server-to-server messaging. Pulsar is highly scalable and uses a simple client API that integrates with Java, Go, Python, and C++. Pulsar’s tiered storage feature offloads data to long term storage such as S3-compatible
Read below to learn how to use Apache Pulsar with Filebase.
Prerequisites:
​Download and install Apache Pulsar.
​Download and install the Apache Pulsar Offloader package.
​Sign up for a free Filebase account. 
Have your Filebase Access and Secret Keys. Learn how to view your access keys here.
Create a Filebase Bucket. Learn how to create a bucket here.
1. Navigate into your Apache Pulsar installation directory. Open the ./conf/standalone.conf and enter the following lines:
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=FILEBASE_BUCKET_NAME
s3ManagedLedgerOffloadRegion=us-east-1
s3ManagedLedgerOffloadServiceEndpoint=https://s3.filebase.com
Replace the following value to match your configuration:
FILEBASE_BUCKET_NAME: Filebase Bucket Name
2. Set the following configurations in the ./conf/pulsar_env.sh file:
export AWS_ACCESS_KEY_ID=FILEBASE_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=FILEBASE_SECRET_KEY
3. Start Pulsar standalone with the following command:
./bin/pulsar standalone -a 127.0.0.1
4. Configure a data retention policy with the following command:
./bin/pulsar-admin namespaces set-retention public/default --size -10G --time 3d
5. Produce data using pulsar-perf:
./bin/pulsar-perf produce -r 1000 -s 2048 test-topic
6. Check the pulsar-admin status with the following command:
./bin/pulsar-admin topics stats-internal test-topic
7. Trigger the offload process with the command:
./bin/pulsar-admin topics offload --size-threshold 10M public/default/test-topic
8. Check the offloading operation status with the command:
./bin/pulsar-admin topics offload-status -w public/default/test-topic
If you have any questions, please join our Discord server, or send us an email at [email protected]​

Ansible

AWS CLI

Last modified 1yr ago

Apache Pulsar

What is Apache Pulsar?

Prerequisites:

1. Navigate into your Apache Pulsar installation directory. Open the `./conf/standalone.conf` and enter the following lines:

2. Set the following configurations in the `./conf/pulsar_env.sh` file:

3. Start Pulsar standalone with the following command:

4. Configure a data retention policy with the following command:

5. Produce data using pulsar-perf:

6. Check the pulsar-admin status with the following command:

7. Trigger the offload process with the command:

8. Check the offloading operation status with the command: