Ceph RadosGW
云存储 Object Storage System 即 OSS,将每个文件无论多大都单独存储为一个对象,通过RESTful API分格协议进行存储。所有的数据都在同一个平面下。Ceph的API兼容aws的S3 API和Swift API。
S3:User、Bucket、Object
Swift:Account、User、Container、Object
RGW:tenant、User/subuser、Bucket、Object
ACL:read、write、readwrite、full-control
启用RadosGW
启用RadosGW:需要运行radosgw服务,如果一个节点不够可以多部署几个,radosgw为无状态服务。
]$ ceph-deploy rgw create ceph01
访问如下这个地址:可以看到输出一段xml信息
curl http://192.168.1.60:7480/
启用后会自动创建出如下几个存储池:
]$ ceph osd pool ls | grep rgw .rgw.root default.rgw.control default.rgw.meta default.rgw.log
修改监听端口
Ceph在0.8版本之前使用的是apache+fastcgi 来提供radosgw服务的,ceph0.8版本开始由civetweb来提供web服务,默认监听在7480端口上。
修改监听的端口:在配置文件ceph.conf 中添加如下配置,多个节点定义多个配置段
[client.rgw.ceph01] rgw_host = ceph01 rgw_frontends = "civetweb port=8080"
再将配置文件推送到所有节点:
ceph-deploy --overwrite-conf config push ceph0{1,2,3}
重启radosgw服务:
systemctl restart ceph-radosgw@rgw.ceph01.service
查看端口是否监听:
ss -tnlp | grep radosgw
使用ssl加密通信
创建目录:
cd /etc/ceph mkdir ssl
创建证书:
openssl genrsa -out civetweb.key 2048 openssl req -new -x509 -key civetweb.key -out civetweb.crt -days 5000 -subj "/CN=ceph01.local"
这里需要的是pem格式的证书:需要把key和crt放到一个文件中
cat civetweb.key civetweb.crt > civetweb.pem
使用ssl通信的配置:
port=8443s 后面的“s”表示启用ssl服务。
ssl_certificate:证书位置。
num_threads=500,服务的线程数,默认50。
request_timeout_ms=60000,超时时间,单位毫秒。
rgw_dns_name = ceph01.local,提供服务的DNS域名。
rgw_host = ceph01,主机名
[client.rgw.ceph01] rgw_host = ceph01 rgw_frontends = "civetweb port=8443s ssl_certificate=/etc/ceph/ssl/civetweb.pem"
同时启用ssl和http服务:
[client.rgw.ceph01] rgw_host = ceph01 rgw_frontends = "civetweb port=7480+8443s ssl_certificate=/etc/ceph/ssl/civetweb.pem"
重启服务:
systemctl restart ceph-radosgw@rgw.ceph01.service
查看端口是否监听:可以看到7480和8443端口都已监听。
ss -tnlp | grep radosgw
访问服务:
curl ceph01:7480
使用RadosGW
创建bucket和访问资源时需要用到范域名解析功能,能解析如 *.ceph01.local 这样的域名,所以需要namd服务,并使用该namd服务为客户端提供解析。
配置named服务:
vim /etc/named.rfc1912.zones zone "ceph01.local" IN { type master; file "ceph01.local.zone"; }; vim /var/named/ceph01.local.zone $TTL 1D $ORIGIN ceph01.local. @ IN SOA ns.ceph01.local. admin.ceph01.local. ( 0 ; serial 1D ; refresh 1H ; retry 1W ; expire 3H ) ; minimum IN NS ns ns IN A 192.168.1.6 @ IN A 192.168.1.30 * IN A 192.168.1.30
检查namd配置:如果无错误则重新加载配置。
named-checkconf rndc reload
解析测试:
dig -t A 111.ceph04.local @192.168.0.6
配置DNS:
vim /etc/resolv.conf nameserver 192.168.0.6 nameserver 233.5.5.5
修改配置ceph.conf:需要添加提供服务的dns域名。单个radosgw服务配置一个rgw配置段,多个radosgw服务配置多个rgw配置段。
[client.rgw.ceph01] rgw_host = ceph01 rgw_frontends = "civetweb port=7480 num_threads=500 request_timeout_ms=60000" rgw_dns_name = ceph01.local [client.rgw.ceph02] rgw_host = ceph02 rgw_frontends = "civetweb port=7480 num_threads=500 request_timeout_ms=60000" rgw_dns_name = ceph02.local
重启radosgw服务:
systemctl restart ceph-radosgw@rgw.ceph01.service
创建用户账号:包含客户端访问资源使用的秘钥信息
]# radosgw-admin user create --uid "s3user" --display-name "S3 Testing User" { "user_id": "s3user", "display_name": "S3 Testing User", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "s3user", "access_key": "28QDPPLZTW1NF7V3EOH3", "secret_key": "BkacX4PUzgVEYwyuyCBJsWVC3VKV2LL4MN2RPs9h" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] }
在客户端安装s3cmd工具:s3cmd为模拟aws存储的客户端工具
yum install s3cmd -y
配置客户端工具:
]# s3cmd --configure Access Key: 28QDPPLZTW1NF7V3EOH3 Secret Key: BkacX4PUzgVEYwyuyCBJsWVC3VKV2LL4MN2RPs9h Default Region [US]: S3 Endpoint [s3.amazonaws.com]: ceph01.local:7480 DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.ceph01.local:7480 Encryption password: Path to GPG program [/usr/bin/gpg]: Use HTTPS protocol [Yes]: No HTTP Proxy server name: New settings: Access Key: 28QDPPLZTW1NF7V3EOH3 Secret Key: BkacX4PUzgVEYwyuyCBJsWVC3VKV2LL4MN2RPs9h Default Region: US S3 Endpoint: ceph01.local:7480 DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.ceph01.local:7480 Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Save settings? [y/N] y
创建bucket:
]# s3cmd mb s3://images Bucket 's3://images/' created
查看bucket:
]# s3cmd ls 2020-02-25 14:38 s3://images
上传文件:目录名称anyname是映射名称,不是真正的目录,可任意名称。
]# s3cmd put /usr/share/backgrounds/morning.jpg s3://images/anyname/morning.jpg upload: '/usr/share/backgrounds/morning.jpg' -> 's3://images/anyname/morning.jpg' [1 of 1] 980265 of 980265 100% in 2s 370.20 kB/s done
查看上传的文件对象:
~]# s3cmd ls s3://images/ DIR s3://images/anyname/ ~]# s3cmd ls s3://images/anyname/ 2020-02-25 14:47 980265 s3://images/anyname/morning.jpg
下载文件:
~]# s3cmd get s3://images/anyname/morning.jpg download: 's3://images/anyname/morning.jpg' -> './morning.jpg' [1 of 1] 980265 of 980265 100% in 0s 61.65 MB/s done
通过web访问资源:这里的images为bucket名称。
http://images.ceph.local:7480/kitty.jpg
或者在路径中带上bucket的名称:
http://ceph.local:7480/images/kitty.jpg
访问地址返回的是一段报错的xml信息,没权限,可以通过调用接口来生成可访问的URL地址,格式类似如下:
http://ceph.local:7480/images/kitty.jpg?AWSAccessKeyId=08XE7K29QLXE8081F5AF&Expires=1597324815&Signature=bPFjuaJagd0X14nx%2B4xtP7acTt4%3D
可以生成访问连接的客户端:
https://github.com/qiyang-true/ceph-oss-client