What is the minimum number of Ceph OSD hosts needed for using the erasure code pool?
The best practice of using Ceph storage is to set the pool failure domain larger than the “host”. A data pool using a host as the failure domain instructs Ceph to write chunks distributedly to different hosts for ensuring data can be recovered for other chunks saved on other healthy hosts. Users can still configure the failure domain to “disk drive,” which protects data specifically against disk failures, similar to traditional disk arrays.
Here are the recommendations for the minimum number of hosts required to use an erasure code pool.
Best configuration:Best configuration: Number of servers ≧ K + M +1
- This configuration enables Ceph to distribute every EC chunk across K+M hosts.
- When one of the hosts fails, you have enough hosts to restore the lost chunk
Second best configuration: Number of servers = K + M
- This configuration allows each EC chunk to be distributed across K+M hosts.
- When a host fails, you have no other host to restore the lost EC chunk. It is crucial to repair it promptly to restore the system to a healthy state as quickly as possible.
Budget-limited configuration:Number of servers ≧ (K + M) / M
This configuration directs Ceph to distribute multiple erasure coding (EC) chunks across the specified number of hosts. When a host fails, at most M chunks of an object will be lost.
This configuration allows
- Any number of disks can fail simultaneously in one host.
- Or the cluster can lose at most one host.
- Or only one disk in each of M hosts can fail simultaneously.
For the example of K+M = 4+2; the minimum number of hosts required is 6/2 = 3. You can lose only one host but you won't have enough healthy hosts in the cluster to re-heal the lost chunks.