Difference between revisions of "Swift for File Systems w/o Extended Attributes"
Tejas Sumant (talk | contribs) m |
m (Grammar fix) |
||
| (17 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
==Summary== | ==Summary== | ||
| − | Today | + | Today, Swift assumes that the file system underneath supports extended attributes. This restricts the use of file systems without extended attributes with Swift. |
==Release Date== | ==Release Date== | ||
Post Havana | Post Havana | ||
==Rationale== | ==Rationale== | ||
| − | + | If we can conquer the limitations of file systems without extended attributes, that would add flexibility for end users to choose their file system(s) and thereby increase adoption of Swift and Openstack. | |
| + | |||
==User Stories== | ==User Stories== | ||
User wants to deploy Openstack swift on EXT2 file system. | User wants to deploy Openstack swift on EXT2 file system. | ||
==Design== | ==Design== | ||
| + | ===Alternative to extended attributes=== | ||
| + | Swift creates one file in file system per Swift object. Each file has a set of extended attributes associated with it. To remove dependency on extended attributes, it is proposed to have one shadow file per Swift created directory. The shadow file will contain all extended attributes in key value form for all the files in a given directory. | ||
| + | |||
| + | When Swift shuffles data on various nodes, rsync is used. Swift issues rsync commands on the directory and not files. So it is inherently ensured, along with Swift files, the shadow file also moves. This change ensures consistency of Swift data. | ||
| + | |||
| + | ===Extended attribute writing to file=== | ||
| + | The ''pickle'' package would be used to manipulate attribute value pairs (extended attributes). | ||
==Implementation== | ==Implementation== | ||
| + | ===Code Changes=== | ||
| + | ====read_metadata and write_metadata functions in server.py==== | ||
| + | These are the key functions to change, to either invoke a system call to get and set extended attributes, or call ''pickle'' based on configuration. | ||
| + | Swift will use standard system calls, if it was known when it started that the underlying filesystem supports extended attributes. | ||
| + | Otherwise, all get and set extended attribute calls are converted to read and write from the shadow file. | ||
| + | ====Other changes==== | ||
| + | Wherever call to read_metadata and write_metadata are made, a check is required to pass the correct file handle. | ||
| + | If extended attributes are supported, then file handle uses the target file, else file handle to shadow file needs to be passed. | ||
| + | |||
| + | ==Test Plan== | ||
| + | Standard Swift tests should pass with this feature in. | ||
| + | Swift should pass tests on file system with extended attribute support as well as file systems without extended attributes support. | ||
| + | At given time, all file systems underneath should either support extended attribute or should not support extended attributes. | ||
==Demo Plan== | ==Demo Plan== | ||
| + | Demo to be uploaded on Calsoft website (www.calsoftinc.com) | ||
Latest revision as of 20:56, 4 November 2013
Contents
Summary
Today, Swift assumes that the file system underneath supports extended attributes. This restricts the use of file systems without extended attributes with Swift.
Release Date
Post Havana
Rationale
If we can conquer the limitations of file systems without extended attributes, that would add flexibility for end users to choose their file system(s) and thereby increase adoption of Swift and Openstack.
User Stories
User wants to deploy Openstack swift on EXT2 file system.
Design
Alternative to extended attributes
Swift creates one file in file system per Swift object. Each file has a set of extended attributes associated with it. To remove dependency on extended attributes, it is proposed to have one shadow file per Swift created directory. The shadow file will contain all extended attributes in key value form for all the files in a given directory.
When Swift shuffles data on various nodes, rsync is used. Swift issues rsync commands on the directory and not files. So it is inherently ensured, along with Swift files, the shadow file also moves. This change ensures consistency of Swift data.
Extended attribute writing to file
The pickle package would be used to manipulate attribute value pairs (extended attributes).
Implementation
Code Changes
read_metadata and write_metadata functions in server.py
These are the key functions to change, to either invoke a system call to get and set extended attributes, or call pickle based on configuration. Swift will use standard system calls, if it was known when it started that the underlying filesystem supports extended attributes. Otherwise, all get and set extended attribute calls are converted to read and write from the shadow file.
Other changes
Wherever call to read_metadata and write_metadata are made, a check is required to pass the correct file handle. If extended attributes are supported, then file handle uses the target file, else file handle to shadow file needs to be passed.
Test Plan
Standard Swift tests should pass with this feature in. Swift should pass tests on file system with extended attribute support as well as file systems without extended attributes support. At given time, all file systems underneath should either support extended attribute or should not support extended attributes.
Demo Plan
Demo to be uploaded on Calsoft website (www.calsoftinc.com)
