科技改變生活 · 科技引領未來
科技改變生活 · 科技引領未來
目錄2.1cephadm介紹2.2相關概念介紹2.3放置規范介紹3.1部署環境要求3.2本次實驗環境5.1下載cephadm腳本5.2引導單群集Ceph安裝5.3添加主機5.4添加OSD5.5查看Ceph部署服務5.6部署RGW5.7部署C
1.前言
在一年前的寫的文章中我提到了cephadm安裝工具，那會剛出來有不少功能還無法安裝，經過一年的時間的等待，一個月前發再ceph 16版本基本功能都差不多了，就開始了cephadm的研究，但因為是想寫一個比較全面的文章，不像現有網上的文章只有基本的一些功能，所以在研究iscsi和ingress安裝的時候遇到了些問題，安裝過數十次也未解決，在前段時候ceph出了最新的ceph 16.2.5版本，終于所有功能都正常了，才有了本篇文章。
PS:個人感悟，如沒有必要還是不要研究比較新的開源的軟件，太折騰太累，新的東西總可能有些BUG。
2.cephadm介紹
22.1cephadm介紹
Cephadm 是隨著 Ceph 新版本 v15.2.0(Octopus)發布的安裝工具，并且不支持 Ceph的舊版本，Ceph中已經Cephadm 不依賴于外部配置工具，如 Ansible、 Rook 和 Salt，它通過 SSH 將管理器守護進程連接到主機來實現這一點。管理器守護進程可以添加、刪除和更新 Ceph 容器。
Redhat Ceph 5和SUSE Ceph 7的最新版本中也已經使用了Cephadm，所以Cephadm是Ceph的安裝工具的未來，并且現在Cephadm也差不多已經成熟可用了，像NFS iSCSI服務官方也宣稱已經穩定可用，未來也會加入CIFS等服務，其他功能也在不停地完善中，所以是時候系統的學習下Cephadm了。
通過下圖可以看出ceph新版本中的部署架構，通過一個編排接口，可以向下對接rook和cephadm兩種編排工具，向上通過命令行"ceph orch"或ceph dashboard實現編排。
其實這個編排和我們理解中的容器編排是相同的，我們可以理解成Cephadm就是一個迷你版本的kubernetes,當然rook有點不同，rook可以理解成一種對接kubernetes編排的中間層，實際編排還是由kubernetes實現，而Cephadm是自己就實現了一些調度編排的邏輯。
cephadm繪圖-第 4 頁
兩種編排工具支持的功能上是有些差別的，具體差別如下表所示：
Cepdm 管理 Ceph 集群的整個生命周期。這個生命周期從引導過程開始，當 cephdm 在單個節點上創建一個單節點的 Ceph 集群時。這個集群由一個MON和一個MGR組成。
在創建了單節點的Ceph集群后，Cephadm 然后會使用編排接口(“ day 2”命令)擴展集群，添加相應主機并部署相應的 Ceph 守護進程（daemons ）和服務（ services）。這個動作可以通過 Ceph 命令行界面(CLI)或者通過儀表板(GUI)來執行。
cephadm還在開發完善中，有些功能是文檔方面沒有很完善，比如，RWG，有些功能方面是沒有最終確定思路，將來可能有大的變化，比如，ingress(之前叫rgw-ha)，cephfs-mirror等。
32.2 相關概念介紹
上文中提到了兩個名詞servicet和daemon，這兩個概念在cephadm中很重要，會貫穿整個ceph的生命周期的管理中。在之前的ceph-deploy中沒有這個概念，當我們要部署三個MON的話，我們會在三臺主機上分別部署三個MON進程，在cephadm中由于引入了編排，當我們部署一個高可用的MON服務時，會說我要部署一個mon service,但實際還是得由三個容器去承載mon，這里還引出另一個問題，這三個承載MON的容器是啟動在一臺主機上還是啟動在三臺主機上，這個就是由“放置規范”決定的了（后面我們會將放置規范），servcie和deamon的關系其實就如下圖所示：
cephadm-servcie-第 2 頁
42.3放置規范介紹
上文中提示放置規范也就是部署一個Servcie,會有幾個Deamon,這幾個Deamon是在哪幾臺主機上，這就是放置規范需要規定的內容，編排調度再根據放置規范具體調度，cephadm總共有5種放置規范，如下：
4456
讓我們實現看下Cephadm服務的展示，下圖中PLACEMENT 中*表示部署在所有主機， count:1 表示使用了指定數量匹配，ceph1;ceph2;ceph3 表示直接使用了明確指定匹配， RUNNING表示實現期望運行的Deamon和正在運行的Deamon比例，如mon顯示3/5 表示mon默認指定了5個Deamon,但現在只有3臺主機，所有只有3個在運行，當然這個期望數量也是可以修改的， PORTS 表示該Service暴露的IP和端口。
[root@ceph1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 4m ago 3d count:1 crash 3/3 4m ago 3d * grafana ?:3000 1/1 4m ago 3d count:1 ingress.nfs.nfs 192.168.149.201:2050,1968 6/6 4m ago 6h count:3 ingress.rgw.rgw 192.168.149.200:8080,1967 6/6 4m ago 6h count:3 iscsi.gw 3/3 4m ago 7h ceph1;ceph2;ceph3 mds.cephfs 3/3 4m ago 7h count:3 mgr 2/2 4m ago 3d count:2 mon 3/5 4m ago 3d count:5 nfs.nfs 3/3 4m ago 7h count:3 node-exporter ?:9100 3/3 4m ago 3d * osd.all-available-devices 9/12 4m ago 3d * prometheus ?:9095 1/1 4m ago 3d count:1 rbd-mirror 3/3 4m ago 7h count:3 rgw.rgw ?:80 3/3 4m ago 7h count:3 注意事項
如果想修改默認mon數量，可以使用ceph orch appy mon 3,如果想單獨指定mon部署到某幾臺主機上，想禁用自動部署，可以使用ceph orch apply mon --unmanaged
3.部署環境要求
53.1部署環境要求
Cehpadm需要有如下軟件安裝在系統中。
63.2 本次實驗環境
序號 類型 版本 1 OS CentOS 8.3(mini) 2 Podman 3.0.2-dev 3 Ceph Pacific (16.2.5) 4 Python 3.6.8
序號 主機名 磁盤(20GB) 角色 1 ceph1 sdb,sdd,sdc cephadm,mon,mgr,osd,rgw,nfs,cephfs,iscsi,prometheus,grafana,rbd-mirror,cephfs-mirror,nfs-ingress,rgw-ingress 2 ceph2 sdb,sdd,sdc mon,mgr,osd,rgw,nfs,cephfs,iscsi,prometheus,grafana,rbd-mirror,cephfs-mirror,nfs-ingress,rgw-ingress 3 ceph3 sdb,sdd,sdc mon,mgr,osd,rgw,nfs,cephfs,iscsi,prometheus,grafana,rbd-mirror,cephfs-mirror,nfs-ingress,rgw-ingress
4.部署準備工作
lvm因為系統自帶的都有，所以就不用單獨安裝了，
#dnf install epel-release -y #dnf install python3 -y #dnf install podman -y #dnf install -y chrony #systemctl start chronyd && systemctl enable chronyd 注意事項
chrony時間服務為必須安裝，具體有2點原因：1為不安裝在添加主機的時候會報錯，2為即使安裝成功ceph -s會也提示時間不同步！
關閉防火墻和SELinux
#systemctl start chronyd && systemctl enable chronyd #systemctl disable firewalld && systemctl stop firewalld #setenforce 0 #sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config (可選)如果在安裝系統的時候就設置過短主機名，就可以不用這一步驟。
#hostnamectl set-hostname ceph1 #hostnamectl set-hostname ceph2 #hostnamectl set-hostname ceph3 注意事項
cephadm需要主機名為短主機名，不能為FQDN，否則在添加主機會報錯！
添加hosts文件中的主機名和IP關系，主機名需要和上面一致
# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.149.128 ceph1 192.168.149.145 ceph2 192.168.149.146 ceph3 配置時間同步
配置Ceph1節點為ntp server。
#vi /etc/chrony.conf allow 192.168.123.0/24 #systemctl restart chronyd 配置其它Ceph2,Ceph3節點為ntp client。
#vi /etc/chrony.conf server ceph-admin iburst #systemctl restart chronyd 在Ceph2和Ceph3確認配置是否成功。
#chronyc sources 210 Number of sources = 4 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* ceph-admin 3 9 0 68m +39us[ +45us] +/- 18ms ^- time.cloudflare.com 3 10 352 49m +5653us[+5653us] +/- 71ms ^? de-user.deepinid.deepin.> 3 10 21 21m -2286us[-2286us] +/- 97ms ^? 2402:f000:1:416:101:6:6:> 0 6 0 - +0ns[ +0ns] +/- 0ns 5.部署Ceph
75.1下載cephadm腳本
下載cephadm腳本,并安裝相應版本的容器源。
#curl --silent --remote-name --location https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm #chmod +x cephadm #./cephadm add-repo --release pacific #./cephadm install #./cephadm install ceph-common 注意事項
官方文檔中還提到了另一種安裝cephadm方式，就是通過dnf install -y cephadm安裝，實踐證明最好不要使用這種方式，這種方式安裝的cephadm可能不是最新版本的，但cephadm去拉的容器版本又是最新的，會導致兩個版本不一致！
查看編排后端是cephadm,如果是使用的rook這里后端顯示的就是rook。
[root@ceph1 ~]# ceph orch status Backend: cephadm Available: Yes Paused: No 85.2引導單群集Ceph安裝
開始通過cephadm “引導”一個單節點Ceph集群起來，示意圖如下：
333
[root@ceph1 ~]# cephadm bootstrap --mon-ip 192.168.149.128 Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman|docker (/usr/bin/podman) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 36e7a21c-e3f7-11eb-8960-000c299df6ef Verifying IP 192.168.149.128 port 3300 ... Verifying IP 192.168.149.128 port 6789 ... Mon IP `192.168.149.128` is in CIDR network `192.168.149.0/24` - internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Pulling container image docker.io/ceph/ceph:v16... Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network to 192.168.149.0/24 Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 9283 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to /etc/ceph/ceph.pub Adding key to root@localhost authorized_keys... Adding host ceph1... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Enabling mgr prometheus module... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling the dashboard module... Waiting for the mgr to restart... Waiting for mgr epoch 13... mgr epoch 13 is available Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... Ceph Dashboard is now available at: URL: https://ceph1:8443/ User: admin Password: dqhiov5x4v Enabling client.admin keyring and conf on hosts with "admin" label You can access the Ceph CLI with: sudo /usr/sbin/cephadm shell --fsid 36e7a21c-e3f7-11eb-8960-000c299df6ef -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/pacific/mgr/telemetry/ Bootstrap complete. 引導完成一個單節點群集，程序會做如下事情：
完成后記錄以上了IP以及用戶和密碼，打開Ceph Dashboard并根據提示修改密碼，打開后提示要激活計量模塊。
(可選)如果忘記記錄密碼可以通過以下方法重置密碼(將密碼寫入password文件中，通過命令導入密碼)
ceph dashboard ac-user-set-password admin -i password {"username": "admin", "password": "$2b$12$6oFrEpssXCzLnKTWQy5fM.YZwlHjn8CuQRdeSSJR9hBGgVuwGCxoa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1620495653, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false} image-20210717120935815
根據向導激活完成。
image-20210717121059540
如果Ceph Dashboard中錯過了啟用，也可以使用命令啟用，命令是“ceph telemetry on --license sharing-1-0”。
95.3 添加主機
上文中提示了在引導成功單節點Ceph群集后會引導程序會將public key的副本寫入/etc/ceph/ceph.pub，在添加主機節點前需要將該key分發到要加入群集的主機上，示意圖如下所示：
444
[root@ceph1 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub"The authenticity of host 'ceph2 (192.168.149.145)' can't be established.ECDSA key fingerprint is SHA256:1fioQmugbtBCiRuwNNKr/aa3Z/hm5zeqUrIfOZi2nS8.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesroot@ceph2's password: Number of key(s) added: 1Now try logging into the machine, with: "ssh 'root@ceph2'"and check to make sure that only the key(s) you wanted were added.[root@ceph1 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ceph/ceph.pub"The authenticity of host 'ceph3 (192.168.149.146)' can't be established.ECDSA key fingerprint is SHA256:eBmb4q2ptVYS55njTzmQYCNo4p3yguNi85nHyAuR4XU.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesroot@ceph3's password: Number of key(s) added: 1Now try logging into the machine, with: "ssh 'root@ceph3'"and check to make sure that only the key(s) you wanted were added.
111
[root@ceph1 ~]# ceph orch host add ceph2 192.168.149.145Added host 'ceph2' with addr '192.168.149.145'[root@ceph1 ~]# ceph orch host add ceph3 192.168.149.146Added host 'ceph3' with addr '192.168.149.146'
注意事項
注意這里添加主機有時候不用寫IP地址是不能添加的，網上有些文章是沒寫IP的，但有些情況下不加IP會報錯，所以還是直接加上，并且官方文檔中也是加的IP的！
[root@ceph1 ~]# ceph orch host ls HOST ADDR LABELS STATUS ceph1 192.168.149.128 _admin ceph2 192.168.149.145 ceph3 192.168.149.146 105.4 添加OSD
添加OSD需求滿足以下所有條件：
添加OSD有2種方式，1為自動添加所有滿足條件的OSD。
#ceph orch apply osd --all-available-devices 2為通過手工指定的方式添加OSD。
#ceph orch daemon add osd ceph1:/dev/sdb 222
本次使用第一種自動部署的方式，部署完成后查看設備列表，顯示為NO就完成了。
[root@ceph1 ~]# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available ceph1 /dev/sdb hdd 21.4G Unknown N/A N/A No ceph1 /dev/sdc hdd 21.4G Unknown N/A N/A No ceph1 /dev/sdd hdd 21.4G Unknown N/A N/A No ceph2 /dev/sdb hdd 21.4G Unknown N/A N/A No ceph2 /dev/sdc hdd 21.4G Unknown N/A N/A No ceph2 /dev/sdd hdd 21.4G Unknown N/A N/A No ceph3 /dev/sdb hdd 21.4G Unknown N/A N/A No ceph3 /dev/sdc hdd 21.4G Unknown N/A N/A No ceph3 /dev/sdd hdd 21.4G Unknown N/A N/A No 115.5 查看Ceph部署服務
命令行查看Ceph狀態正常。
[root@ceph1 ~]# ceph -s cluster: id: 36e7a21c-e3f7-11eb-8960-000c299df6ef health: HEALTH_OK services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 8s) mgr: ceph1.nwbihh(active, since 3d), standbys: ceph2.ednijf osd: 9 osds: 9 up (since 3s), 9 in (since 3d) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 48 MiB used, 180 GiB / 180 GiB avail pgs: 1 active+clean 我們打開dashboard看下，監控而不能顯示。
image-20210717162056574
其實這個原因是使用了https，又沒有安全證書，所有不能顯示。
image-20210717162132879
我們只需求通過瀏覽器直接打開grafana地址，手工點擊“接受風險并繼續”再回到Ceph Dashboard中再查看就正常了。
image-20210717162237683
可以看到監控顯示正常了。
image-20210717162315766
可以看到告警模塊也自動集成了，這點不會像之前Ceph-deploy安裝方式使用那復雜的步驟手工集成了。
image-20210717162350611
iSCSI還是沒有自動集成。
image-20210717162437984
rbd-mirror服務沒有啟用。
image-20210717162509603
NFS也沒有集成。
image-20210717162528974
Cephfs也沒有部署。
image-20210717162545960
RGW也沒有集成。
image-20210717162606723
下面我們逐步一個個部署集成這些服務。
125.6 部署RGW
使用指定數量匹配模式部署。
#ceph orch apply rgw rgw --placement=3 333
通過Service查看命令ceph orch ls查看該服務狀態。
[root@ceph1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 10m ago 3d count:1 crash 3/3 10m ago 3d * grafana ?:3000 1/1 10m ago 3d count:1 mgr 2/2 10m ago 3d count:2 mon 3/5 10m ago 3d count:5 node-exporter ?:9100 3/3 10m ago 3d * osd.all-available-devices 9/12 10m ago 3d * prometheus ?:9095 1/1 10m ago 3d count:1 rgw.rgw ?:80 3/3 1s ago 12s count:3 通過Deamon查看命令ceph orch ps查看該進程狀態。
[root@ceph1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID ConTAINER ID alertmanager.ceph1 ceph1 *:9093,9094 running (3d) 30s ago 3d 25.4M - 0.20.0 0881eb8f169f 32812d14049d crash.ceph1 ceph1 running (3d) 30s ago 3d 2675k - 16.2.5 6933c2a0b7dd 4e28e82a2c92 crash.ceph2 ceph2 running (3d) 34s ago 3d 12.3M - 16.2.5 6933c2a0b7dd 45e02925199a crash.ceph3 ceph3 running (3d) 34s ago 3d 12.9M - 16.2.5 6933c2a0b7dd e98dc6157ba2 grafana.ceph1 ceph1 *:3000 running (3d) 30s ago 3d 54.0M - 6.7.4 ae5c36c3d3cd df3a2a73271c mgr.ceph1.nwbihh ceph1 *:9283 running (3d) 30s ago 3d 428M - 16.2.5 6933c2a0b7dd 8210247b8cef mgr.ceph2.ednijf ceph2 *:8443,9283 running (3d) 34s ago 3d 399M - 16.2.5 6933c2a0b7dd cf556f5d2527 mon.ceph1 ceph1 running (3d) 30s ago 3d 475M 2048M 16.2.5 6933c2a0b7dd eef3a01cca72 mon.ceph2 ceph2 running (3d) 34s ago 3d 293M 2048M 16.2.5 6933c2a0b7dd 4130307d82f2 mon.ceph3 ceph3 running (3d) 34s ago 3d 261M 2048M 16.2.5 6933c2a0b7dd fa0cdd7af8a9 node-exporter.ceph1 ceph1 *:9100 running (3d) 30s ago 3d 14.9M - 0.18.1 e5a616e4b9cf 3fc396d01969 node-exporter.ceph2 ceph2 *:9100 running (3d) 34s ago 3d 11.3M - 0.18.1 e5a616e4b9cf 2b0081864a94 node-exporter.ceph3 ceph3 *:9100 running (3d) 34s ago 3d 11.2M - 0.18.1 e5a616e4b9cf 73a7bbe0831c osd.0 ceph2 running (3d) 34s ago 3d 41.2M 4096M 16.2.5 6933c2a0b7dd 2046f2cc358e osd.1 ceph3 running (3d) 34s ago 3d 45.4M 4096M 16.2.5 6933c2a0b7dd 09058507fe6e osd.2 ceph1 running (3d) 30s ago 3d 33.0M 4096M 16.2.5 6933c2a0b7dd 80d58366f0dc osd.3 ceph2 running (3d) 34s ago 3d 42.5M 4096M 16.2.5 6933c2a0b7dd 63654f9d8082 osd.4 ceph3 running (3d) 34s ago 3d 43.3M 4096M 16.2.5 6933c2a0b7dd d3a82429878d osd.5 ceph1 running (3d) 30s ago 3d 31.3M 4096M 16.2.5 6933c2a0b7dd ebfc2a71bc3c osd.6 ceph2 running (3d) 34s ago 3d 38.8M 4096M 16.2.5 6933c2a0b7dd 235189a4bd54 osd.7 ceph3 running (3d) 34s ago 3d 44.3M 4096M 16.2.5 6933c2a0b7dd b7f8a457a3b1 osd.8 ceph1 running (3d) 30s ago 3d 33.2M 4096M 16.2.5 6933c2a0b7dd eb1bf3e567fd prometheus.ceph1 ceph1 *:9095 running (3d) 30s ago 3d 81.5M - 2.18.1 de242295e225 4da5a8d98259 rgw.rgw.ceph1.lgcvfw ceph1 *:80 running (43s) 30s ago 42s 48.0M - 16.2.5 6933c2a0b7dd 20fb488d35ad rgw.rgw.ceph2.eqykjt ceph2 *:80 running (40s) 34s ago 40s 40.1M - 16.2.5 6933c2a0b7dd e9c9538b064e rgw.rgw.ceph3.eybvwe ceph3 *:80 running (37s) 34s ago 37s 41.8M - 16.2.5 6933c2a0b7dd 75e93ef56bb7 集成到dashboard
#radosgw-admin user create --uid=rgw --display-name=rgw --system "keys": [ { "user": "rgw", "access_key": "M0XRR80H4AGGE4PP0A5B", "secret_key": "Tbln48sfIceDGNill5muCrX0oMCHrQcl2oC9OURe" } ], ...... 記錄access_key和secret_key的值保存為 access_key.txt 和secret_key.txt ，通過命令集成到dahsboard中。
#ceph dashboard set-rgw-api-access-key -i access_key.txt Option RGW_API_ACCESS_KEY updated #ceph dashboard set-rgw-api-secret-key -i secret_key.txt Option RGW_API_SECRET_KEY updated 在Ceph Dashboard中看到RGW已經集成成功（看官方文檔未來Cephadm安裝會自動集成到Ceph Dashboard中，不像現在還需要手動集成）。
image-20210717164724797
image-20210717164738030
image-20210717164753742
135.7 部署Cephfs
部署cephfs服務并創建cepfs，創建cephfs有兩種方式，一種是使用的是ceph fs命令該命令會自動創建相應的池，另一種手工創建池并創建Service，下面方法任選一種。
#ceph fs volume create cephfs --placement=3 #ceph osd pool create cephfs_data 32 #ceph osd pool create cephfs_metadata 32 #ceph fs new cephfs cephfs_metadata cephfs_data #ceph orch apply mds cephfs --placement=3 444
查看Service狀態。
[root@ceph1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 1s ago 3d count:1 crash 3/3 5s ago 3d * grafana ?:3000 1/1 1s ago 3d count:1 mds.cephfs 3/3 5s ago 15s count:3 mgr 2/2 5s ago 3d count:2 mon 3/5 5s ago 3d count:5 node-exporter ?:9100 3/3 5s ago 3d * osd.all-available-devices 9/12 5s ago 3d * prometheus ?:9095 1/1 1s ago 3d count:1 rgw.rgw ?:80 3/3 5s ago 21m count:3 查看Deamon狀態。
[root@ceph1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID ConTAINER ID alertmanager.ceph1 ceph1 *:9093,9094 running (7m) 32s ago 3d 31.9M - 0.20.0 0881eb8f169f 66918f633189 crash.ceph1 ceph1 running (7m) 32s ago 3d 6287k - 16.2.5 6933c2a0b7dd dbe00b18ef37 crash.ceph2 ceph2 running (7m) 36s ago 3d 7889k - 16.2.5 6933c2a0b7dd c24dcd762121 crash.ceph3 ceph3 running (7m) 36s ago 3d 10.1M - 16.2.5 6933c2a0b7dd eed1c74352f0 grafana.ceph1 ceph1 *:3000 running (7m) 32s ago 3d 70.5M - 6.7.4 ae5c36c3d3cd cbbfae0f90e4 mds.cephfs.ceph1.ooraiz ceph1 running (44s) 32s ago 43s 24.0M - 16.2.5 6933c2a0b7dd f668ab1ad9bb mds.cephfs.ceph2.qfmprj ceph2 running (41s) 36s ago 41s 19.5M - 16.2.5 6933c2a0b7dd d8bb9979aca2 mds.cephfs.ceph3.ifskba ceph3 running (39s) 36s ago 39s 19.8M - 16.2.5 6933c2a0b7dd d0fd9a78c1d8 mgr.ceph1.nwbihh ceph1 *:9283 running (7m) 32s ago 3d 480M - 16.2.5 6933c2a0b7dd 90bd2d0704b3 mgr.ceph2.ednijf ceph2 *:8443,9283 running (7m) 36s ago 3d 468M - 16.2.5 6933c2a0b7dd 9b69c446355f mon.ceph1 ceph1 running (7m) 32s ago 3d 139M 2048M 16.2.5 6933c2a0b7dd 736fd1630a06 mon.ceph2 ceph2 running (7m) 36s ago 3d 108M 2048M 16.2.5 6933c2a0b7dd 629bc53c8992 mon.ceph3 ceph3 running (7m) 36s ago 3d 125M 2048M 16.2.5 6933c2a0b7dd 1b56b8a0d9ac node-exporter.ceph1 ceph1 *:9100 running (7m) 32s ago 3d 23.3M - 0.18.1 e5a616e4b9cf 3664470b1641 node-exporter.ceph2 ceph2 *:9100 running (7m) 36s ago 3d 24.9M - 0.18.1 e5a616e4b9cf 140f7e84ba38 node-exporter.ceph3 ceph3 *:9100 running (7m) 36s ago 3d 24.8M - 0.18.1 e5a616e4b9cf de98304ceaea osd.0 ceph2 running (7m) 36s ago 3d 59.2M 4096M 16.2.5 6933c2a0b7dd 6ebce33417c2 osd.1 ceph3 running (7m) 36s ago 3d 82.7M 4096M 16.2.5 6933c2a0b7dd 6d43f1b7bfde osd.2 ceph1 running (7m) 32s ago 3d 56.0M 4096M 16.2.5 6933c2a0b7dd 07d1183306d1 osd.3 ceph2 running (7m) 36s ago 3d 67.5M 4096M 16.2.5 6933c2a0b7dd a77d771d3e6d osd.4 ceph3 running (7m) 36s ago 3d 48.7M 4096M 16.2.5 6933c2a0b7dd 3d82752a8fb1 osd.5 ceph1 running (7m) 32s ago 3d 61.6M 4096M 16.2.5 6933c2a0b7dd 6f7b5df090d5 osd.6 ceph2 running (7m) 36s ago 3d 59.2M 4096M 16.2.5 6933c2a0b7dd 7f769655adc7 osd.7 ceph3 running (7m) 36s ago 3d 51.9M 4096M 16.2.5 6933c2a0b7dd 24d63cd18dde osd.8 ceph1 running (7m) 32s ago 3d 44.8M 4096M 16.2.5 6933c2a0b7dd 667fb8831e53 prometheus.ceph1 ceph1 *:9095 running (7m) 32s ago 3d 97.0M - 2.18.1 de242295e225 f67fbc035cba rgw.rgw.ceph1.lgcvfw ceph1 *:80 running (7m) 32s ago 22m 71.1M - 16.2.5 6933c2a0b7dd c8e1c9701010 rgw.rgw.ceph2.eqykjt ceph2 *:80 running (7m) 36s ago 22m 82.9M - 16.2.5 6933c2a0b7dd d8ba326c22b8 rgw.rgw.ceph3.eybvwe ceph3 *:80 running (7m) 36s ago 22m 81.4M - 16.2.5 6933c2a0b7dd d89c2b87e8e4 查看Ceph Dashboard中狀態。
image-20210717165017471
145.8 部署NFS
先創建nfs所需求的池。
#ceph osd pool create ganesha_data 32 #ceph osd pool application enable ganesha_data nfs 555
部署nfs Service。
#ceph orch apply nfs nfs ganesha_data --placement=3 查看Service狀態。
[root@ceph1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 3s ago 3d count:1 crash 3/3 8s ago 3d * grafana ?:3000 1/1 3s ago 3d count:1 mds.cephfs 3/3 8s ago 3m count:3 mgr 2/2 8s ago 3d count:2 mon 3/5 8s ago 3d count:5 nfs.nfs 3/3 8s ago 28s count:3 node-exporter ?:9100 3/3 8s ago 3d * osd.all-available-devices 9/12 8s ago 3d * prometheus ?:9095 1/1 3s ago 3d count:1 rgw.rgw ?:80 3/3 8s ago 25m count:3 查看Deamon狀態。
[root@ceph1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID ConTAINER ID alertmanager.ceph1 ceph1 *:9093,9094 running (11m) 27s ago 3d 28.9M - 0.20.0 0881eb8f169f 66918f633189 crash.ceph1 ceph1 running (11m) 27s ago 3d 5628k - 16.2.5 6933c2a0b7dd dbe00b18ef37 crash.ceph2 ceph2 running (10m) 33s ago 3d 7889k - 16.2.5 6933c2a0b7dd c24dcd762121 crash.ceph3 ceph3 running (10m) 33s ago 3d 10.1M - 16.2.5 6933c2a0b7dd eed1c74352f0 grafana.ceph1 ceph1 *:3000 running (11m) 27s ago 3d 70.6M - 6.7.4 ae5c36c3d3cd cbbfae0f90e4 mds.cephfs.ceph1.ooraiz ceph1 running (3m) 27s ago 3m 26.6M - 16.2.5 6933c2a0b7dd f668ab1ad9bb mds.cephfs.ceph2.qfmprj ceph2 running (3m) 33s ago 3m 21.4M - 16.2.5 6933c2a0b7dd d8bb9979aca2 mds.cephfs.ceph3.ifskba ceph3 running (3m) 33s ago 3m 21.7M - 16.2.5 6933c2a0b7dd d0fd9a78c1d8 mgr.ceph1.nwbihh ceph1 *:9283 running (11m) 27s ago 3d 489M - 16.2.5 6933c2a0b7dd 90bd2d0704b3 mgr.ceph2.ednijf ceph2 *:8443,9283 running (10m) 33s ago 3d 468M - 16.2.5 6933c2a0b7dd 9b69c446355f mon.ceph1 ceph1 running (11m) 27s ago 3d 126M 2048M 16.2.5 6933c2a0b7dd 736fd1630a06 mon.ceph2 ceph2 running (10m) 33s ago 3d 119M 2048M 16.2.5 6933c2a0b7dd 629bc53c8992 mon.ceph3 ceph3 running (10m) 33s ago 3d 135M 2048M 16.2.5 6933c2a0b7dd 1b56b8a0d9ac nfs.nfs.0.0.ceph1.ufsfpf ceph1 *:2049 running (47s) 27s ago 47s 50.5M - 3.5 6933c2a0b7dd 8af0f7d10e2b nfs.nfs.1.0.ceph2.blgraz ceph2 *:2049 running (42s) 33s ago 41s 27.4M - 3.5 6933c2a0b7dd d2e2238859e8 nfs.nfs.2.0.ceph3.edjscz ceph3 *:2049 running (36s) 33s ago 36s 27.2M - 3.5 6933c2a0b7dd 31e148631f9f node-exporter.ceph1 ceph1 *:9100 running (11m) 27s ago 3d 22.6M - 0.18.1 e5a616e4b9cf 3664470b1641 node-exporter.ceph2 ceph2 *:9100 running (11m) 33s ago 3d 24.9M - 0.18.1 e5a616e4b9cf 140f7e84ba38 node-exporter.ceph3 ceph3 *:9100 running (10m) 33s ago 3d 25.0M - 0.18.1 e5a616e4b9cf de98304ceaea osd.0 ceph2 running (10m) 33s ago 3d 62.4M 4096M 16.2.5 6933c2a0b7dd 6ebce33417c2 osd.1 ceph3 running (10m) 33s ago 3d 85.9M 4096M 16.2.5 6933c2a0b7dd 6d43f1b7bfde osd.2 ceph1 running (10m) 27s ago 3d 52.4M 4096M 16.2.5 6933c2a0b7dd 07d1183306d1 osd.3 ceph2 running (10m) 33s ago 3d 71.3M 4096M 16.2.5 6933c2a0b7dd a77d771d3e6d osd.4 ceph3 running (10m) 33s ago 3d 52.1M 4096M 16.2.5 6933c2a0b7dd 3d82752a8fb1 osd.5 ceph1 running (10m) 27s ago 3d 56.3M 4096M 16.2.5 6933c2a0b7dd 6f7b5df090d5 osd.6 ceph2 running (10m) 33s ago 3d 61.5M 4096M 16.2.5 6933c2a0b7dd 7f769655adc7 osd.7 ceph3 running (10m) 33s ago 3d 54.7M 4096M 16.2.5 6933c2a0b7dd 24d63cd18dde osd.8 ceph1 running (10m) 27s ago 3d 46.7M 4096M 16.2.5 6933c2a0b7dd 667fb8831e53 prometheus.ceph1 ceph1 *:9095 running (11m) 27s ago 3d 97.2M - 2.18.1 de242295e225 f67fbc035cba rgw.rgw.ceph1.lgcvfw ceph1 *:80 running (11m) 27s ago 25m 62.2M - 16.2.5 6933c2a0b7dd c8e1c9701010 rgw.rgw.ceph2.eqykjt ceph2 *:80 running (11m) 33s ago 25m 83.3M - 16.2.5 6933c2a0b7dd d8ba326c22b8 rgw.rgw.ceph3.eybvwe ceph3 *:80 running (10m) 33s ago 25m 81.9M - 16.2.5 6933c2a0b7dd d89c2b87e8e4 查看Ceph Dashboard狀態。
image-20210717165333310
155.9 部署iSCSi
創建iscsi所需求的池。
#ceph osd pool create iscsi_pool 32 32 #ceph osd pool application enable iscsi_pool iscsi 前面幾個我們都是通過命令指定放置規范，由于iscsi配置的參數有點多，所以部署iscsi我們換YAM方式（當初iscsi也支持命令指定方式）。
#vi iscsi.yaml service_type: iscsi service_id: gw placement: hosts: - ceph1 - ceph2 - ceph3 spec: pool: iscsi_pool trusted_ip_list: "192.168.149.128,192.168.149.145,192.168.149.146" api_user: admin api_password: admin api_secure: false 通過apply命令部署，看到apply熟悉kuernetes就很明白他的意思了，cephadm也是聲明式的，所以如果想修改配置參數只需要直接修改YAML文件，再apply就可以了。
[root@ceph1 ~]# ceph orch apply -i iscsi.yaml Scheduled iscsi.gw update... 666
查看Service狀態。
[root@ceph1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 5s ago 3d count:1 crash 3/3 9s ago 3d * grafana ?:3000 1/1 5s ago 3d count:1 iscsi.gw 3/3 9s ago 21s ceph1;ceph2;ceph3 mds.cephfs 3/3 9s ago 8m count:3 mgr 2/2 9s ago 3d count:2 mon 3/5 9s ago 3d count:5 nfs.nfs 3/3 9s ago 5m count:3 node-exporter ?:9100 3/3 9s ago 3d * osd.all-available-devices 9/12 9s ago 3d * prometheus ?:9095 1/1 5s ago 3d count:1 rgw.rgw ?:80 3/3 9s ago 30m count:3 查看Deamon狀態。
[root@ceph1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID ConTAINER ID alertmanager.ceph1 ceph1 *:9093,9094 running (15m) 28s ago 3d 23.6M - 0.20.0 0881eb8f169f 66918f633189 crash.ceph1 ceph1 running (15m) 28s ago 3d 4072k - 16.2.5 6933c2a0b7dd dbe00b18ef37 crash.ceph2 ceph2 running (15m) 32s ago 3d 7470k - 16.2.5 6933c2a0b7dd c24dcd762121 crash.ceph3 ceph3 running (15m) 32s ago 3d 10.1M - 16.2.5 6933c2a0b7dd eed1c74352f0 grafana.ceph1 ceph1 *:3000 running (15m) 28s ago 3d 47.3M - 6.7.4 ae5c36c3d3cd cbbfae0f90e4 iscsi.gw.ceph1.esngze ceph1 running (42s) 28s ago 41s 64.2M - 3.5 6933c2a0b7dd 96f1250ba7f1 iscsi.gw.ceph2.ypjkrx ceph2 running (39s) 32s ago 39s 61.9M - 3.5 6933c2a0b7dd 9cc090aa85ec iscsi.gw.ceph3.hntxjs ceph3 running (36s) 32s ago 36s 28.2M - 3.5 6933c2a0b7dd d9a6906671a4 mds.cephfs.ceph1.ooraiz ceph1 running (8m) 28s ago 8m 22.1M - 16.2.5 6933c2a0b7dd f668ab1ad9bb mds.cephfs.ceph2.qfmprj ceph2 running (8m) 32s ago 8m 20.3M - 16.2.5 6933c2a0b7dd d8bb9979aca2 mds.cephfs.ceph3.ifskba ceph3 running (8m) 32s ago 8m 22.9M - 16.2.5 6933c2a0b7dd d0fd9a78c1d8 mgr.ceph1.nwbihh ceph1 *:9283 running (15m) 28s ago 3d 465M - 16.2.5 6933c2a0b7dd 90bd2d0704b3 mgr.ceph2.ednijf ceph2 *:8443,9283 running (15m) 32s ago 3d 434M - 16.2.5 6933c2a0b7dd 9b69c446355f mon.ceph1 ceph1 running (15m) 28s ago 3d 128M 2048M 16.2.5 6933c2a0b7dd 736fd1630a06 mon.ceph2 ceph2 running (15m) 32s ago 3d 115M 2048M 16.2.5 6933c2a0b7dd 629bc53c8992 mon.ceph3 ceph3 running (15m) 32s ago 3d 161M 2048M 16.2.5 6933c2a0b7dd 1b56b8a0d9ac nfs.nfs.0.0.ceph1.ufsfpf ceph1 *:2049 running (5m) 28s ago 5m 70.3M - 3.5 6933c2a0b7dd 8af0f7d10e2b nfs.nfs.1.0.ceph2.blgraz ceph2 *:2049 running (5m) 32s ago 5m 77.7M - 3.5 6933c2a0b7dd d2e2238859e8 nfs.nfs.2.0.ceph3.edjscz ceph3 *:2049 running (5m) 32s ago 5m 80.7M - 3.5 6933c2a0b7dd 31e148631f9f node-exporter.ceph1 ceph1 *:9100 running (15m) 28s ago 3d 16.0M - 0.18.1 e5a616e4b9cf 3664470b1641 node-exporter.ceph2 ceph2 *:9100 running (15m) 32s ago 3d 23.5M - 0.18.1 e5a616e4b9cf 140f7e84ba38 node-exporter.ceph3 ceph3 *:9100 running (15m) 32s ago 3d 27.5M - 0.18.1 e5a616e4b9cf de98304ceaea osd.0 ceph2 running (15m) 32s ago 3d 60.7M 4096M 16.2.5 6933c2a0b7dd 6ebce33417c2 osd.1 ceph3 running (15m) 32s ago 3d 90.3M 4096M 16.2.5 6933c2a0b7dd 6d43f1b7bfde osd.2 ceph1 running (15m) 28s ago 3d 49.1M 4096M 16.2.5 6933c2a0b7dd 07d1183306d1 osd.3 ceph2 running (15m) 32s ago 3d 71.8M 4096M 16.2.5 6933c2a0b7dd a77d771d3e6d osd.4 ceph3 running (15m) 32s ago 3d 55.8M 4096M 16.2.5 6933c2a0b7dd 3d82752a8fb1 osd.5 ceph1 running (15m) 28s ago 3d 51.5M 4096M 16.2.5 6933c2a0b7dd 6f7b5df090d5 osd.6 ceph2 running (15m) 32s ago 3d 62.1M 4096M 16.2.5 6933c2a0b7dd 7f769655adc7 osd.7 ceph3 running (15m) 32s ago 3d 57.4M 4096M 16.2.5 6933c2a0b7dd 24d63cd18dde osd.8 ceph1 running (15m) 28s ago 3d 49.2M 4096M 16.2.5 6933c2a0b7dd 667fb8831e53 prometheus.ceph1 ceph1 *:9095 running (15m) 28s ago 3d 80.4M - 2.18.1 de242295e225 f67fbc035cba rgw.rgw.ceph1.lgcvfw ceph1 *:80 running (15m) 28s ago 30m 55.8M - 16.2.5 6933c2a0b7dd c8e1c9701010 rgw.rgw.ceph2.eqykjt ceph2 *:80 running (15m) 32s ago 30m 74.9M - 16.2.5 6933c2a0b7dd d8ba326c22b8 rgw.rgw.ceph3.eybvwe ceph3 *:80 running (15m) 32s ago 30m 83.8M - 16.2.5 6933c2a0b7dd d89c2b87e8e4 查看Ceph Dashboard狀態。
image-20210717165822644
image-20210717165848575
165.10 添加rbd-mirror
部署rbd-mirror Service.
[root@ceph-node1 ~]# ceph orch apply rbd-mirror --placement=3 Scheduled rbd-mirror update... 777
查看Service狀態。
[root@ceph1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 9s ago 3d count:1 crash 3/3 12s ago 3d * grafana ?:3000 1/1 9s ago 3d count:1 iscsi.gw 3/3 12s ago 3m ceph1;ceph2;ceph3 mds.cephfs 3/3 12s ago 11m count:3 mgr 2/2 12s ago 3d count:2 mon 3/5 12s ago 3d count:5 nfs.nfs 3/3 12s ago 8m count:3 node-exporter ?:9100 3/3 12s ago 3d * osd.all-available-devices 9/12 12s ago 3d * prometheus ?:9095 1/1 9s ago 3d count:1 rbd-mirror 3/3 12s ago 23s count:3 rgw.rgw ?:80 3/3 12s ago 32m count:3 查看Deamon狀態。
[root@ceph1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID ConTAINER ID alertmanager.ceph1 ceph1 *:9093,9094 running (18m) 34s ago 3d 24.5M - 0.20.0 0881eb8f169f 66918f633189 crash.ceph1 ceph1 running (18m) 34s ago 3d 3309k - 16.2.5 6933c2a0b7dd dbe00b18ef37 crash.ceph2 ceph2 running (18m) 37s ago 3d 11.6M - 16.2.5 6933c2a0b7dd c24dcd762121 crash.ceph3 ceph3 running (18m) 37s ago 3d 10.1M - 16.2.5 6933c2a0b7dd eed1c74352f0 grafana.ceph1 ceph1 *:3000 running (18m) 34s ago 3d 60.2M - 6.7.4 ae5c36c3d3cd cbbfae0f90e4 iscsi.gw.ceph1.esngze ceph1 running (3m) 34s ago 3m 53.1M - 3.5 6933c2a0b7dd 96f1250ba7f1 iscsi.gw.ceph2.ypjkrx ceph2 running (3m) 37s ago 3m 66.1M - 3.5 6933c2a0b7dd 9cc090aa85ec iscsi.gw.ceph3.hntxjs ceph3 running (3m) 37s ago 3m 59.7M - 3.5 6933c2a0b7dd d9a6906671a4 mds.cephfs.ceph1.ooraiz ceph1 running (11m) 34s ago 11m 18.4M - 16.2.5 6933c2a0b7dd f668ab1ad9bb mds.cephfs.ceph2.qfmprj ceph2 running (11m) 37s ago 11m 20.1M - 16.2.5 6933c2a0b7dd d8bb9979aca2 mds.cephfs.ceph3.ifskba ceph3 running (11m) 37s ago 11m 23.4M - 16.2.5 6933c2a0b7dd d0fd9a78c1d8 mgr.ceph1.nwbihh ceph1 *:9283 running (18m) 34s ago 3d 456M - 16.2.5 6933c2a0b7dd 90bd2d0704b3 mgr.ceph2.ednijf ceph2 *:8443,9283 running (18m) 37s ago 3d 423M - 16.2.5 6933c2a0b7dd 9b69c446355f mon.ceph1 ceph1 running (18m) 34s ago 3d 129M 2048M 16.2.5 6933c2a0b7dd 736fd1630a06 mon.ceph2 ceph2 running (18m) 37s ago 3d 133M 2048M 16.2.5 6933c2a0b7dd 629bc53c8992 mon.ceph3 ceph3 running (18m) 37s ago 3d 158M 2048M 16.2.5 6933c2a0b7dd 1b56b8a0d9ac nfs.nfs.0.0.ceph1.ufsfpf ceph1 *:2049 running (8m) 34s ago 8m 71.9M - 3.5 6933c2a0b7dd 8af0f7d10e2b nfs.nfs.1.0.ceph2.blgraz ceph2 *:2049 running (8m) 37s ago 8m 86.8M - 3.5 6933c2a0b7dd d2e2238859e8 nfs.nfs.2.0.ceph3.edjscz ceph3 *:2049 running (8m) 37s ago 8m 80.9M - 3.5 6933c2a0b7dd 31e148631f9f node-exporter.ceph1 ceph1 *:9100 running (18m) 34s ago 3d 22.2M - 0.18.1 e5a616e4b9cf 3664470b1641 node-exporter.ceph2 ceph2 *:9100 running (18m) 37s ago 3d 22.0M - 0.18.1 e5a616e4b9cf 140f7e84ba38 node-exporter.ceph3 ceph3 *:9100 running (18m) 37s ago 3d 27.6M - 0.18.1 e5a616e4b9cf de98304ceaea osd.0 ceph2 running (18m) 37s ago 3d 58.4M 4096M 16.2.5 6933c2a0b7dd 6ebce33417c2 osd.1 ceph3 running (18m) 37s ago 3d 91.3M 4096M 16.2.5 6933c2a0b7dd 6d43f1b7bfde osd.2 ceph1 running (18m) 34s ago 3d 42.7M 4096M 16.2.5 6933c2a0b7dd 07d1183306d1 osd.3 ceph2 running (18m) 37s ago 3d 67.0M 4096M 16.2.5 6933c2a0b7dd a77d771d3e6d osd.4 ceph3 running (18m) 37s ago 3d 56.9M 4096M 16.2.5 6933c2a0b7dd 3d82752a8fb1 osd.5 ceph1 running (18m) 34s ago 3d 58.5M 4096M 16.2.5 6933c2a0b7dd 6f7b5df090d5 osd.6 ceph2 running (18m) 37s ago 3d 60.2M 4096M 16.2.5 6933c2a0b7dd 7f769655adc7 osd.7 ceph3 running (18m) 37s ago 3d 58.0M 4096M 16.2.5 6933c2a0b7dd 24d63cd18dde osd.8 ceph1 running (18m) 34s ago 3d 47.8M 4096M 16.2.5 6933c2a0b7dd 667fb8831e53 prometheus.ceph1 ceph1 *:9095 running (18m) 34s ago 3d 95.1M - 2.18.1 de242295e225 f67fbc035cba rbd-mirror.ceph1.rkchvq ceph1 running (46s) 34s ago 46s 32.6M - 16.2.5 6933c2a0b7dd 46c56c1528f0 rbd-mirror.ceph2.zdhnvt ceph2 running (44s) 37s ago 44s 33.7M - 16.2.5 6933c2a0b7dd de9df26682c7 rbd-mirror.ceph3.mssyuu ceph3 running (42s) 37s ago 41s 29.9M - 16.2.5 6933c2a0b7dd 679eabd8dd5c rgw.rgw.ceph1.lgcvfw ceph1 *:80 running (18m) 34s ago 33m 50.1M - 16.2.5 6933c2a0b7dd c8e1c9701010 rgw.rgw.ceph2.eqykjt ceph2 *:80 running (18m) 37s ago 33m 73.9M - 16.2.5 6933c2a0b7dd d8ba326c22b8 rgw.rgw.ceph3.eybvwe ceph3 *:80 running (18m) 37s ago 33m 84.4M - 16.2.5 6933c2a0b7dd d89c2b87e8e4 查看Ceph Dashboard狀態。
image-20210717170114782
175.11 部署Cephfs-mirror
部署cephfs-mirror Service.
#ceph orch apply cephfs-mirror --placement=3 Scheduled cephfs-mirror update... 888
查看Service狀態。
[root@ceph1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 22s ago 4d count:1 cephfs-mirror 3/3 25s ago 52s count:3 crash 3/3 25s ago 4d * grafana ?:3000 1/1 22s ago 4d count:1 ingress.nfs.nfs 192.168.149.201:2050,1968 6/6 25s ago 7h count:3 ingress.rgw.rgw 192.168.149.200:8080,1967 6/6 25s ago 7h count:3 iscsi.gw 3/3 25s ago 8h ceph1;ceph2;ceph3 mds.cephfs 3/3 25s ago 8h count:3 mgr 2/2 25s ago 4d count:2 mon 3/5 25s ago 4d count:5 nfs.nfs 3/3 25s ago 8h count:3 node-exporter ?:9100 3/3 25s ago 4d * osd.all-available-devices 9/12 25s ago 3d * prometheus ?:9095 1/1 22s ago 4d count:1 rbd-mirror 3/3 25s ago 8h count:3 rgw.rgw ?:80 3/3 25s ago 8h count:3 查看Deamon狀態。
[root@ceph1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID ConTAINER ID alertmanager.ceph1 ceph1 *:9093,9094 running (7h) 61s ago 4d 29.5M - 0.20.0 0881eb8f169f b9640288c91f cephfs-mirror.ceph1.cclxku ceph1 running (86s) 61s ago 85s 26.1M - 16.2.5 6933c2a0b7dd 88b6f8918aa8 cephfs-mirror.ceph2.ozwgqg ceph2 running (116s) 64s ago 116s 30.8M - 16.2.5 6933c2a0b7dd dcd8c6dcc4f0 cephfs-mirror.ceph3.iefmko ceph3 running (2m) 64s ago 2m 30.5M - 16.2.5 6933c2a0b7dd 581ba61bad28 crash.ceph1 ceph1 running (7h) 61s ago 4d 1975k - 16.2.5 6933c2a0b7dd 116ff5ce3646 crash.ceph2 ceph2 running (7h) 64s ago 3d 5448k - 16.2.5 6933c2a0b7dd 354b8903892b crash.ceph3 ceph3 running (7h) 64s ago 3d 5737k - 16.2.5 6933c2a0b7dd a5c223e5362c grafana.ceph1 ceph1 *:3000 running (7h) 61s ago 4d 50.8M - 6.7.4 ae5c36c3d3cd d284162e0da4 haproxy.nfs.nfs.ceph1.bdmrwh ceph1 *:2050,1968 running (7h) 61s ago 7h 1899k - 2.3.12-b99e499 b2284eda2221 3d9c640d828e haproxy.nfs.nfs.ceph2.ryaaaq ceph2 *:2050,1968 running (7h) 64s ago 7h 1879k - 2.3.12-b99e499 b2284eda2221 cffaa94c6c4a haproxy.nfs.nfs.ceph3.ysfamh ceph3 *:2050,1968 running (7h) 64s ago 7h 1644k - 2.3.12-b99e499 b2284eda2221 0e48492c6233 haproxy.rgw.rgw.ceph1.nmvtig ceph1 *:8080,1967 running (7h) 61s ago 7h 7574k - 2.3.12-b99e499 b2284eda2221 5e521b875eba haproxy.rgw.rgw.ceph2.bbcnso ceph2 *:8080,1967 running (7h) 64s ago 7h 4512k - 2.3.12-b99e499 b2284eda2221 cbd03c96a1fd haproxy.rgw.rgw.ceph3.bcvvxl ceph3 *:8080,1967 running (7h) 64s ago 7h 5775k - 2.3.12-b99e499 b2284eda2221 535a37749437 iscsi.gw.ceph1.esngze ceph1 running (7h) 61s ago 8h 23.8M - 3.5 6933c2a0b7dd 04d1bfd141a9 iscsi.gw.ceph2.ypjkrx ceph2 running (7h) 64s ago 8h 21.2M - 3.5 6933c2a0b7dd 7f78b9c0deb0 iscsi.gw.ceph3.hntxjs ceph3 running (7h) 64s ago 8h 28.5M - 3.5 6933c2a0b7dd 2fe9bb0b7bfe keepalived.nfs.nfs.ceph1.jprcvw ceph1 running (7h) 61s ago 7h 3862k - 2.0.5 073e0c3cd1b9 8e337e634255 keepalived.nfs.nfs.ceph2.oiynik ceph2 running (7h) 64s ago 7h 3493k - 2.0.5 073e0c3cd1b9 a6d10588190e keepalived.nfs.nfs.ceph3.guulfc ceph3 running (7h) 64s ago 7h 1853k - 2.0.5 073e0c3cd1b9 23680bf04f55 keepalived.rgw.rgw.ceph1.keriqf ceph1 running (7h) 61s ago 7h 2902k - 2.0.5 073e0c3cd1b9 a11d221ae1af keepalived.rgw.rgw.ceph2.gxshhg ceph2 running (7h) 64s ago 7h 2461k - 2.0.5 073e0c3cd1b9 59026981ef10 keepalived.rgw.rgw.ceph3.tqaixq ceph3 running (7h) 64s ago 7h 4294k - 2.0.5 073e0c3cd1b9 25cfdab23bfe mds.cephfs.ceph1.ooraiz ceph1 running (7h) 61s ago 8h 14.6M - 16.2.5 6933c2a0b7dd 64246a825e21 mds.cephfs.ceph2.qfmprj ceph2 running (7h) 64s ago 8h 9323k - 16.2.5 6933c2a0b7dd f0faa5b9a7d5 mds.cephfs.ceph3.ifskba ceph3 running (7h) 64s ago 8h 9072k - 16.2.5 6933c2a0b7dd 7f294af518f7 mgr.ceph1.nwbihh ceph1 *:9283 running (7h) 61s ago 4d 161M - 16.2.5 6933c2a0b7dd f7b2470c5797 mgr.ceph2.ednijf ceph2 *:8443,9283 running (7h) 64s ago 3d 28.2M - 16.2.5 6933c2a0b7dd e24b40d99e6d mon.ceph1 ceph1 running (7h) 61s ago 4d 483M 2048M 16.2.5 6933c2a0b7dd 9936a8b6587b mon.ceph2 ceph2 running (7h) 64s ago 3d 878M 2048M 16.2.5 6933c2a0b7dd abe7eed2d100 mon.ceph3 ceph3 running (7h) 64s ago 3d 881M 2048M 16.2.5 6933c2a0b7dd 52d747e3d011 nfs.nfs.0.0.ceph1.ufsfpf ceph1 *:2049 running (7h) 61s ago 8h 25.4M - 3.5 6933c2a0b7dd fff5327b8415 nfs.nfs.1.0.ceph2.blgraz ceph2 *:2049 running (7h) 64s ago 8h 33.0M - 3.5 6933c2a0b7dd ea09e2429950 nfs.nfs.2.0.ceph3.edjscz ceph3 *:2049 running (7h) 64s ago 8h 41.8M - 3.5 6933c2a0b7dd dcc6fd5aa85f node-exporter.ceph1 ceph1 *:9100 running (7h) 61s ago 4d 13.8M - 0.18.1 e5a616e4b9cf 69e5d1f3e310 node-exporter.ceph2 ceph2 *:9100 running (7h) 64s ago 3d 15.8M - 0.18.1 e5a616e4b9cf cd41893212ca node-exporter.ceph3 ceph3 *:9100 running (7h) 64s ago 3d 15.3M - 0.18.1 e5a616e4b9cf 9ab2b99dabd6 osd.0 ceph2 running (7h) 64s ago 3d 31.6M 4096M 16.2.5 6933c2a0b7dd 0c9dc80a1d74 osd.1 ceph3 running (7h) 64s ago 3d 47.2M 4096M 16.2.5 6933c2a0b7dd c698fa4c50c0 osd.2 ceph1 running (7h) 61s ago 3d 39.3M 4096M 16.2.5 6933c2a0b7dd 69501861396e osd.3 ceph2 running (7h) 64s ago 3d 50.0M 4096M 16.2.5 6933c2a0b7dd fa9549c63716 osd.4 ceph3 running (7h) 64s ago 3d 37.8M 4096M 16.2.5 6933c2a0b7dd d1fc644bd8f6 osd.5 ceph1 running (7h) 61s ago 3d 38.5M 4096M 16.2.5 6933c2a0b7dd ed81fef0dd1c osd.6 ceph2 running (7h) 64s ago 3d 29.7M 4096M 16.2.5 6933c2a0b7dd 68d0dcad316b osd.7 ceph3 running (7h) 64s ago 3d 28.4M 4096M 16.2.5 6933c2a0b7dd d3dc04b1d1ff osd.8 ceph1 running (7h) 61s ago 3d 38.4M 4096M 16.2.5 6933c2a0b7dd 13f0d2b109ff prometheus.ceph1 ceph1 *:9095 running (7h) 61s ago 4d 131M - 2.18.1 de242295e225 e5fb2f5703d5 rbd-mirror.ceph1.rkchvq ceph1 running (7h) 61s ago 8h 7453k - 16.2.5 6933c2a0b7dd e5ca469f9184 rbd-mirror.ceph2.zdhnvt ceph2 running (7h) 64s ago 8h 6505k - 16.2.5 6933c2a0b7dd 2933e22fd669 rbd-mirror.ceph3.mssyuu ceph3 running (7h) 64s ago 8h 10.5M - 16.2.5 6933c2a0b7dd 862ef07ae291 rgw.rgw.ceph1.lgcvfw ceph1 *:80 running (7h) 61s ago 8h 66.9M - 16.2.5 6933c2a0b7dd 9002673bd55a rgw.rgw.ceph2.eqykjt ceph2 *:80 running (7h) 64s ago 8h 67.9M - 16.2.5 6933c2a0b7dd dbdddd0f10cd rgw.rgw.ceph3.eybvwe ceph3 *:80 running (7h) 64s ago 8h 69.0M - 16.2.5 6933c2a0b7dd b896b40ef1d8 命令行查看部署的哪些服務。
[root@ceph1 ~]# ceph -s cluster: id: 36e7a21c-e3f7-11eb-8960-000c299df6ef health: HEALTH_WARN clock skew detected on mon.ceph2, mon.ceph3 services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 58m) mgr: ceph1.nwbihh(active, since 7h), standbys: ceph2.ednijf mds: 1/1 daemons up, 2 standby osd: 9 osds: 9 up (since 7h), 9 in (since 3d) cephfs-mirror: 3 daemons active (3 hosts) rbd-mirror: 3 daemons active (3 hosts) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 241 pgs objects: 233 objects, 9.4 KiB usage: 323 MiB used, 180 GiB / 180 GiB avail pgs: 241 active+clean io: client: 4.1 KiB/s rd, 4 op/s rd, 0 op/s wr Ceph Dashboard中查看部署了哪些服務。
image-20210717173522825
185.12 添加ingress
在前幾個步驟中我們添加了RGW和NFS，并且都部署了3個Deamon，但其實三個還是獨立服務的，也就是前端沒有負載均衡實現統一訪問，在Cephadm中將haproxy和keepalived兩個封裝成了ingress服務，rgw-ingress架構示意圖如下：
HAProxy_for_RGW
編寫rgw-ingress放置規范及配置參數，并部署。
[root@ceph1 ~]# vi rgw-ingress.yaml service_type: ingress service_id: rgw.rgw placement: count: 3 spec: backend_service: rgw.rgw virtual_ip: 192.168.149.200/24 frontend_port: 8080 monitor_port: 1967 #ceph orch apply -i rgw-ingress.yaml 注意事項
注意這里的backend_service一定要通過ceph orch ls查看到實際的名稱為準， frontend_port為VIP的端口，注意到這里有個 monitor_port這個的意思是haproxy 狀態頁端口。
999
比如想驗證 monitor_port的意思，可以進入haproxy容器中，查看haproxy配置文件frontend stats段落中有1967端口以及用戶名和密碼。
#podman exec -it 5e521b875eba bash # cat /var/lib/haproxy/haproxy.cfg root@ceph1:/var/lib/haproxy# cat haproxy.cfg # This file is generated by cephadm. global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/lib/haproxy/haproxy.pid maxconn 8000 daemo
馬陽東