Use Ceph for Deep Storage

NVIDIA Minerva keeps historical time-series data in deep storage. If your site doesn't have an S3 object store, you can run Ceph on your own hosts and disks. You then use the Ceph Object Gateway (RGW), which provides an S3-compatible API, as NVIDIA Minerva deep storage.

Ceph is open-source software that you install and operate. NVIDIA Minerva does not install, upgrade, or monitor Ceph. NVIDIA Minerva has qualified the configuration on this page: Ceph v19.2.3 (Squid) deployed with cephadm, RGW over HTTP on port 80, and three-way replication across hosts.

The setup has these steps:

1. Set up the Ceph cluster and RGW.

2. Create an S3 user and a bucket.

3. Install NVIDIA Minerva with s3-compatible time-series storage.

4. Verify that data reaches the bucket.

Requirements

The qualified reference cluster has three hosts with three 11.6 TB NVMe devices each (nine OSDs, 105 TiB raw, about 31 TiB usable) and 25 GbE networking.

Performance Requirements

The Ceph cluster must meet these minimums:

• The network sets the throughput limit. Aggregate write throughput is about the sum of the hosts' NIC bandwidth divided by three (the replication factor). Adding disks doesn't raise this limit. Adding hosts or faster NICs does.

• Keep disk utilization at or below 70%, so the remaining OSDs can absorb the data of a failed one.

Set Up Ceph

Run cephadm bootstrap and the SSH key step on the host itself. Run every other ceph and radosgw-admin command on this page inside sudo cephadm shell on a Ceph host.

Use the exact image quay.io/ceph/ceph:v19.2.3, not the floating v19 tag. Version 19.2.3 is the qualified release. Later 19.2.x releases are expected to work, but they aren't qualified.

Step 1 — Install Podman and cephadm

On each Ceph host, install the container runtime, chrony, and the standalone cephadm script. On Ubuntu, run:

apt-get update && apt-get install -y podman chrony
curl -fsSL https://download.ceph.com/rpm-19.2.3/el9/noarch/cephadm -o /usr/sbin/cephadm && chmod +x /usr/sbin/cephadm

Step 2 — Bootstrap the Cluster

On the first Ceph host, run:

cephadm --image quay.io/ceph/ceph:v19.2.3 bootstrap --mon-ip <ceph-host-1-ip>

The bootstrap also deploys Ceph's monitoring stack. NVIDIA Minerva doesn't monitor Ceph, so use this stack or your own alerting to find out when the cluster is degraded.

Step 3 — Add the Other Hosts

1. On the first Ceph host, install the cluster's public key in root's authorized_keys on each other Ceph host. Go through a user with sudo rights, because ssh-copy-id to root needs password authentication, which prohibit-password turns off:

   Bash

   cat /etc/ceph/ceph.pub | ssh <user>@<ceph-host-2-ip> 'sudo mkdir -p /root/.ssh && sudo tee -a /root/.ssh/authorized_keys >/dev/null'
   

   Repeat the command for each other Ceph host.

2. Inside sudo cephadm shell, add each host to the cluster:

   Bash

   ceph orch host add <ceph-host-2-name> <ceph-host-2-ip>
   ceph orch host add <ceph-host-3-name> <ceph-host-3-ip>
   

   If you have a fourth host, add it the same way.

Step 4 — Create the OSDs

Create an OSD on every data device. Ceph excludes the boot device automatically.

ceph orch device ls
ceph orch apply osd --all-available-devices

If ceph orch device ls reports Has GPT headers for a data device, zap the device first with ceph orch device zap <host> <device> --force, and then run ceph orch apply osd --all-available-devices again.

On hosts with less than 32 GB of RAM, set the OSD memory target, in bytes:

ceph config set osd osd_memory_target <bytes>

Step 5 — Deploy the Control Plane and RGW

Scale the monitors and managers, deploy one RGW on each host on port 80, and set three-way replication:

ceph orch apply mon 3 ; ceph orch apply mgr 2
ceph orch host label add <ceph-host-name> rgw          # each host
ceph orch apply rgw minerva --placement="count-per-host:1 label:rgw" --port=80
ceph config set global osd_pool_default_size 3

Then check the cluster:

ceph -s ; ceph df

The cluster is ready when ceph -s shows HEALTH_OK, three monitors, and all OSDs up. In ceph df, usable capacity is about the raw capacity divided by three.

Step 6 — Create the S3 User and Bucket

1. Create the S3 user that NVIDIA Minerva uses:

   Bash

   radosgw-admin user create --uid=druid --display-name="Minerva Druid"
   

   Record the access_key and secret_key from the output. You enter them in the installer later.

2. Create the deep storage bucket. You can use any host that has the AWS CLI and can reach RGW. Set the new user's keys, and then create the bucket:

   Bash

   export AWS_ACCESS_KEY_ID=<access-key>
   export AWS_SECRET_ACCESS_KEY=<secret-key>
   aws --endpoint-url http://<ceph-host-1-ip>:80 s3 mb s3://minerva-deep-storage
   

Create the bucket before you install NVIDIA Minerva. Neither the installer nor NVIDIA Minerva creates it. Don't add lifecycle expiration rules to the bucket, and leave versioning off.

This page uses the bucket name minerva-deep-storage. If you choose another name, use it in the installer and in the commands that follow.

Install NVIDIA Minerva with Ceph Deep Storage

Run the installer on the manager node as described in Install on a Single Node or Install on Multiple Nodes. Answer the time-series storage prompts as shown below. For every installer prompt, see Installer Prompts.

The endpoint is a single RGW host:port. The qualified configuration uses one RGW endpoint.

With these answers, the installer configures the NVIDIA Minerva time-series service to use the RGW endpoint, including the path-style access that RGW requires. You don't need to edit any configuration files. The runtime S3 credentials are kept separate from any AWS credentials on the host and don't replace them.

For a single-node installation, bring up the stacks when the installer finishes:

cd ~/minerva
./svm stack-up all

For a multi-node installation, complete the steps in Multi-Node Installations first.

If the installer stops with Cannot write custom S3 endpoint config: <directory> is not writable by <user>, make that directory writable by the user who runs the installer, and then run the installer again. The default directory is ~/opt/minerva/var/tsservice/config.

Multi-Node Installations

In a multi-node installation, the time-series services can run on worker nodes, and every one of them needs the endpoint settings that the installer writes on the manager node.

1. Answer the S3-compatible prompts on the manager node only.

2. Mount a shared NFS directory at the same path on every node that runs a time-series service.

3. In ~/minerva/deploy-cfg.yml, set the data_config volume of all five time-series roles to the same folder in the shared directory:

   YAML

   broker:
       volume:
           data_config: /nfs-share/data/service/tsservice/config
   coordinator:
       volume:
           data_config: /nfs-share/data/service/tsservice/config
   router:
       volume:
           data_config: /nfs-share/data/service/tsservice/config
   historical:
       volume:
           data_config: /nfs-share/data/service/tsservice/config
   middlemanager:
       volume:
           data_config: /nfs-share/data/service/tsservice/config
   

4. On the manager node, write the configuration, and then check that tsservice.conf is in the shared folder:

   Bash

   cd ~/minerva
   ./svm pre-startup
   ls /nfs-share/data/service/tsservice/config/tsservice.conf
   

5. Bring up the stacks:

   Bash

   ./svm stack-up all
   

Use HTTPS

The qualified configuration uses plain HTTP on port 80. The setup steps on this page deploy RGW that way.

If your RGW endpoint uses HTTPS and you answer https at the protocol prompt, the installer turns off TLS certificate checking for the time-series service's S3 connections. Traffic is encrypted, but the server certificate is not verified, so the connection isn't protected against on-path interception.

Verifying the RGW certificate against your own certificate authority isn't part of the qualified configuration, and the installer doesn't set it up. If you need it, contact your NVIDIA representative for the additional steps.

Plan Capacity

Size the cluster from your ingestion rate and retention:

usable bytes = datapoints per minute × 1,440 × 1.5 bytes × retention days
raw bytes    = usable bytes × 3

The 1.5 bytes per datapoint is a rounded, observed size after compression. Provision at least 1.5 times the raw figure, to leave headroom for compaction.

NVIDIA Minerva deletes expired segments from deep storage every day, so deep storage use stays bounded by the retention period.

Verify and Operate

After the stacks are up, check that deep storage is receiving data. Inside sudo cephadm shell on a Ceph host, run:

radosgw-admin bucket stats --bucket=minerva-deep-storage

The object count and size grow as NVIDIA Minerva writes time-series segments to the bucket.

Use these commands for routine checks. Run chronyc tracking directly on each host. Run the others inside sudo cephadm shell.

For other Ceph operations, see the Ceph documentation.

Last updated: November 02, 2026