In this guide, we will cover the installation and configuration of a fully functional Ceph Storage Cluster on Ubuntu 18.04 LTS server. Ceph is an open source storage solution that implements distributed object storage cluster, and provides interfaces for an object, block, and file-level storage.
Components of Ceph storage cluster
The basic components of a Ceph storage cluster
- Monitors: A Ceph Monitor (
ceph-mon) maintains maps of the cluster state, including the monitor map, manager map, the OSD map, and the CRUSH map - Ceph OSDs: A Ceph OSD (object storage daemon,
ceph-osd) stores data, handles data replication, recovery, rebalancing, and provides some monitoring information to Ceph Monitors and Managers by checking other Ceph OSD Daemons for a heartbeat. At least 3 Ceph OSDs are normally required for redundancy and high availability. - MDSs: A Ceph Metadata Server (MDS,
ceph-mds) stores metadata on behalf of the Ceph Filesystem (i.e., Ceph Block Devices and Ceph Object Storage do not use MDS). Ceph Metadata Servers allow POSIX file system users to execute basic commands (like,ls, findetc.) without placing an enormous burden on the Ceph Storage Cluster.\ - Managers: A Ceph Manager daemon (
ceph-mgr) is responsible for keeping track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.
Ceph Cluster on Ubuntu 18.04 Architecture
My Ceph cluster deployment is based on the following system diagram
The hostnames of each server should be set accordingly, and /etc/hosts file configured to look like below on all Ceph Cluster Nodes:
192.168.18.70 rgw.example.com rgw
192.168.18.71 mon01.example.com mon01
192.168.18.72 mon02.example.com mon02
192.168.18.73 mon03.example.com mon03
192.168.18.75 osd01.example.com osd01
192.168.18.76 osd02.example.com osd02
192.168.18.77 osd03.example.com osd03Change the hostnames and IP addresses to fit your setup. As a pre-requisite, ensure your system is updated
sudo apt update
sudo apt -y upgrade
sudo rebootOnce the systems are rebooted, you can begin Ceph cluster deployment. It is recommended to have an additional node, whose work is deploying ceph on all Ceph nodes. This node is called admin node
192.168.18.74 ceph-admin.example.com ceph-adminPrepare the Ceph Admin Node
You need to add Ceph repositories to the ceph-deploy admin node, then, install ceph-deploy package. My admin node is on IP 192.168.18.74
Import repository key
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -Add the Ceph repository to your system. This installation will do Ceph nautilus
echo deb https://download.ceph.com/debian-nautilus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.listUpdate your repository and install ceph-deploy:
sudo apt update
sudo apt -y install ceph-deployPrepare Ceph Nodes
The admin node must have password-less SSH access to Ceph nodes. When ceph-deploy logs in to a Ceph node as a user, that particular user must have passwordless sudo privileges.
Add SSH user on All Ceph Nodes – rgw,osd nodes & Monitors
export USER_NAME="ceph-admin"
export USER_PASS="[email protected]"
sudo useradd --create-home -s /bin/bash ${USER_NAME}
echo "${USER_NAME}:${USER_PASS}"|sudo chpasswd
echo "${USER_NAME} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/${USER_NAME}
sudo chmod 0440 /etc/sudoers.d/${USER_NAME}Confirm that the user can run sudo without a password prompt
[email protected]:~$ su - ceph-admin
Password:
[email protected]:~$ sudo su -
[email protected]:~#
Generate the SSH keys on ceph-admin node, but do not use sudo or the root user. Leave the passphrase empty:
# su - ceph-admin
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph-admin/.ssh/id_rsa):
Created directory '/home/ceph-admin/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ceph-admin/.ssh/id_rsa.
Your public key has been saved in /home/ceph-admin/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:DZZdfRS1Yo+unWAkDum7juShEF67pm7VdSkfWlsCpbA [email protected]
The key's randomart image is:
+---[RSA 2048]----+
| . .. .. o=|
| o..o . . o|
| E .= o o o |
| +.O .. + |
| . .. .oS.*. . . |
|. o.....ooo . |
| o.. o . . o . |
| ...= o . . + . |
|oooo o.+. . o |
+----[SHA256]-----+
$ ls /home/ceph-admin/.ssh/
config id_rsa id_rsa.pub known_hosts
Configure your ~/.ssh/config
$ cat /home/ceph-admin/.ssh/config
Host osd01
Hostname osd01
User ceph-admin
Host osd02
Hostname osd02
User ceph-admin
Host osd03
Hostname osd03
User ceph-admin
Host osd01
Hostname osd01
User ceph-admin
Host mon01
Hostname mon01
User ceph-admin
Host mon02
Hostname mon02
User ceph-admin
Host mon03
Hostname mon03
User ceph-admin
Host rgw
Hostname rgw
User ceph-admin
Copy the key to each Ceph Node ( Do this from Ceph Admin Node as the ceph-admin user)
for i in rgw mon01 mon02 mon03 osd01 osd02 osd03; do
ssh-copy-id $i
done
If your remote user on all Ceph Nodes is not,ceph-admin replace it with correct username.
Deploy Ceph Storage Cluster
Ceph is sensitive to Clock drift, so ensure you setup NTP on all Ceph Nodes, especially Monitors.
sudo apt install ntp
We’re going to create a three Ceph Node cluster with one Ceph Monitor and three Ceph OSD Daemons. Once the cluster reaches a active + clean state, expand it by adding a Metadata Server and two more Ceph Monitors
Let’s start by creating a directory on our admin node for maintaining the configuration files and keys that ceph-deploy generates for the cluster.
mkdir ceph-deploy
cd ceph-deploy
The ceph-deploy utility will output files to the current directory. Ensure you are in this directory when executing ceph-deploy.
Step 1: Initialize ceph monitor nodes
Run the following commands on your admin node from the ceph-deploy directory you created for holding your configuration details
$ ceph-deploy new mon01 mon02 mon03
Sample output
[email protected]:~/ceph-deploy$ ceph-deploy new mon01 mon02 mon03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy new mon01
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4c0720a950>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['mon01']
[ceph_deploy.cli][INFO ] func : <function new at 0x7f4c07456d70>
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[mon01][DEBUG ] connected to host: ceph-admin
[mon01][INFO ] Running command: ssh -CT -o BatchMode=yes mon01
[mon01][DEBUG ] connection detected need for sudo
[mon01][DEBUG ] connected to host: mon01
[mon01][DEBUG ] detect platform information from remote host
[mon01][DEBUG ] detect machine type
[mon01][DEBUG ] find the location of an executable
[mon01][INFO ] Running command: sudo /bin/ip link show
[mon01][INFO ] Running command: sudo /bin/ip addr show
[mon01][DEBUG ] IP addresses found: [u'192.168.18.71', u'192.168.19.71', u'192.168.121.23']
[ceph_deploy.new][DEBUG ] Resolving host mon01
[ceph_deploy.new][DEBUG ] Monitor mon01 at 192.168.18.71
[ceph_deploy.new][DEBUG ] Monitor initial members are ['mon01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.18.71']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf..Step 2: Install Ceph packages:
Install Ceph Packages on all nodes.
ceph-deploy install mon01 mon02 mon03 osd01 osd02 osd03 rgwThe ceph-deploy utility will install Ceph on each node.
Sample output
..........................
[osd03][DEBUG ] Setting up ceph-base (13.2.2-1bionic) ...
[osd03][DEBUG ] Setting up python-pecan (1.2.1-2) ...
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode
[osd03][DEBUG ] update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode
[osd03][DEBUG ] Setting up ceph-osd (13.2.2-1bionic) ...
[osd03][DEBUG ] chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target.
[osd03][DEBUG ] Setting up ceph-mds (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target.
[osd03][DEBUG ] Setting up ceph-mon (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target.
[osd03][DEBUG ] Setting up ceph-mgr (13.2.2-1bionic) ...
[osd03][DEBUG ] Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target.
[osd03][DEBUG ] Setting up ceph (13.2.2-1bionic) ...
[osd03][DEBUG ] Processing triggers for libc-bin (2.27-3ubuntu1) ...
[osd03][DEBUG ] Processing triggers for ureadahead (0.100.0-20) ...
[osd03][DEBUG ] Processing triggers for systemd (237-3ubuntu10) ...
[osd03][INFO ] Running command: sudo ceph --version
[osd03][DEBUG ] ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)Step 3: Deploy the initial monitor(s) and gather the keys:
Create initial monitors by running the command:
ceph-deploy mon create-initialA number of keyrings will be placed in your working directory.
Deploy a manager daemon:
ceph-deploy mgr create mon01 mon02 mon03Add a Metadata Servers:
ceph-deploy mds create mon01 mon02 mon03Step 4: Copy Ceph Admin Key
Copy the configuration file and admin key to your admin node and your Ceph Nodes:
ceph-deploy admin mon01 mon02 mon03 osd01 osd02 osd03Step 5: Add three OSDs
I assume you have an unused disk in each node called /dev/device. Mine look like below:
[email protected]:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 32G 0 disk
|-sda1 8:1 0 487M 0 part /boot
|-sda2 8:2 0 1.9G 0 part [SWAP]
`-sda3 8:3 0 29.6G 0 part /
vdb 252:0 0 5G 0 disk
vdc 252:16 0 5G 0 disk
vdd 252:32 0 5G 0 disk
I have three devices to use:
- /dev/vda
- dev/vdb
- dev/vdc
Be sure that the device is not currently in use and does not contain any important data. The syntax to use is:
ceph-deploy osd create --data {device} {ceph-node}
In my case, I’ll run:
for i in vdb vdc vdd; do
for j in osd01 osd02 osd03; do
ceph-deploy osd create --data /dev/$i $j
done
doneNote: If you are creating an OSD on an LVM volume, the argument to --data must be volume_group/lv_name, rather than the path to the volume’s block device.
The output of the lsblkcommand should have changed
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 32G 0 disk
|-sda1 8:1 0 487M 0 part /boot
|-sda2 8:2 0 1.9G 0 part [SWAP]
`-sda3 8:3 0 29.6G 0 part /
vdb 252:0 0 5G 0 disk
`-ceph--908c8792--04e8--414f--8430--faa78e9b18eb-osd--block--275c9d8b--3825--4898--9b3b--5ea080fd7137 253:0 0 5G 0 lvm
vdc 252:16 0 5G 0 disk
`-ceph--c79a5159--3980--47e8--b649--ed0c44d32d51-osd--block--a50c2ebc--8d65--4d16--9196--6f741606b3a2 253:1 0 5G 0 lvm
vdd 252:32 0 5G 0 disk
`-ceph--594ff477--943e--49d8--8e08--addf5dbccca3-osd--block--5b71bad9--7fa8--41af--a3af--48af1219aafd 253:2 0 5G 0 lvm
Step 6: Check your cluster’s health.
Check Ceph Cluster status
[email protected]:~# sudo ceph health
HEALTH_OK
[email protected]:~# sudo ceph status
cluster:
id: 5594514c-4d70-4ec8-a74e-e54ac2c1994f
health: HEALTH_OK
services:
mon: 1 daemons, quorum mon01
mgr: osd01(active)
osd: 9 osds: 9 up, 9 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 9.0 GiB used, 36 GiB / 45 GiB avail
pgs:
Once you have added your new Ceph Monitors, Ceph will begin synchronizing the monitors and form a quorum. You can check the quorum status by executing the following:
ceph quorum_status --format json-pretty
Step 7: Enable Ceph Dashboard
Enable the Ceph Dashboard module:
sudo ceph mgr module enable dashboard
sudo ceph mgr module lsGenerate self signed certificates for the dashboard:
sudo ceph dashboard create-self-signed-certCreate a user for Dashboard:
sudo ceph dashboard ac-user-create admin '[email protected]' administratorEnabling the Object Gateway Management Frontend:
$ sudo radosgw-admin user create --uid=admin --display-name='Ceph Admin' --systemFinally, provide the credentials to the dashboard:
sudo ceph dashboard set-rgw-api-access-key <api-access-key>
sudo ceph dashboard set-rgw-api-secret-key <api-secret-key>If you are using a self-signed certificate in your Object Gateway setup, then you should disable certificate verification:
sudo ceph dashboard set-rgw-api-ssl-verify FalseStep 8: Add Rados Gateway
To use the Ceph Object Gateway component of Ceph, you must deploy an instance of RGW. Execute the following to create a new instance of Rados Gateway:
$ ceph-deploy rgw create {gateway-node}Example:
$ ceph-deploy rgw create rgwBy default, the RGW instance will listen on port 7480. This can be changed by editing ceph.conf on the node running the RGW as follows:
[client]
rgw frontends = civetweb port=80
Resetting your Ceph Cluster
If at any point you run into trouble and you want to start over, execute the following to purge the Ceph packages, and erase all its data and configuration:
ceph-deploy purge {ceph-node} [{ceph-node}]
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
rm ceph.*If you execute purge, you must re-install Ceph. The last rm command removes any files that were written out by ceph-deploy locally during a previous installation.
In our next article, I’ll cover how to store and retrieve objects on Ceph and configuring Ceph Clients.
