Difference between revisions of "Swift/erasure-codes/"
< Swift
(Created page with "Today Swift uses replication for data durability and overall performance. However, in some use cases, the low-latency and high availability of replica storage is not as import...") |
|||
| Line 3: | Line 3: | ||
To implement erasure codes into Swift, we need to answer the following questions: | To implement erasure codes into Swift, we need to answer the following questions: | ||
| − | * How does auditing work for EC data? | + | * How does auditing work for EC data? [[Swift/erasure-codes/auditing]] |
| − | * How are missing pieces of data detected and reconstructed? (ie How is drive failure managed?) | + | * How are missing pieces of data detected and reconstructed? (ie How is drive failure managed?) [[Swift/erasure-codes/reconstructor]] |
| − | * How is the entire EC object validated to assure that the chunks are not corrupted? | + | * How is the entire EC object validated to assure that the chunks are not corrupted? [[Swift/erasure-codes/validator]] |
| − | * How are orphaned chunks handled? | + | * How are orphaned chunks handled? [[Swift/erasure-codes/orphans]] |
| − | * What is the data flow for reads? | + | * What is the data flow for reads? [[Swift/erasure-codes/reads]] |
| − | * What is the data flow for writes? | + | * What is the data flow for writes? [[Swift/erasure-codes/writes]] |
* What is the granularity of EC data? (ie cluster, account, container, object) [[Swift/erasure-codes/granularity]] | * What is the granularity of EC data? (ie cluster, account, container, object) [[Swift/erasure-codes/granularity]] | ||
Revision as of 00:20, 1 July 2013
Today Swift uses replication for data durability and overall performance. However, in some use cases, the low-latency and high availability of replica storage is not as important as reducing the overall storage footprint. These use cases are normally characterized by having at least one of the following properties: 1) Large aggregate size 2) Large individual object sizes 3) Infrequent access. One example of this use case is a cluster that is used as storage capacity for backups.
To implement erasure codes into Swift, we need to answer the following questions:
- How does auditing work for EC data? Swift/erasure-codes/auditing
- How are missing pieces of data detected and reconstructed? (ie How is drive failure managed?) Swift/erasure-codes/reconstructor
- How is the entire EC object validated to assure that the chunks are not corrupted? Swift/erasure-codes/validator
- How are orphaned chunks handled? Swift/erasure-codes/orphans
- What is the data flow for reads? Swift/erasure-codes/reads
- What is the data flow for writes? Swift/erasure-codes/writes
- What is the granularity of EC data? (ie cluster, account, container, object) Swift/erasure-codes/granularity
