Jump to: navigation, search

Difference between revisions of "Swift/ideas"

(→‎Ideas)
 
(61 intermediate revisions by 18 users not shown)
Line 1: Line 1:
 
= Ideas for OpenStack Swift =
 
= Ideas for OpenStack Swift =
  
−
This is a list of stuff you might be interested in working on. Most of these are smaller, self-contained projects that probably won't overlap too much with other ongoing work. This is a great place to get some ideas of things to work on.
+
This page is a collection of "brain dumps" for ideas about features in OpenStack Swift. If you're working on something, it's a very good idea to write down what you're thinking about. This lets others get up to speed, helps you collaborate, and serves as a great record for future reference. Write down your thoughts somewhere and put a link to it here. It doesn't matter what form your thoughts are in; use whatever is best for you. Your document should include why your idea is needed and your thoughts on particular design choices and tradeoffs. '''Please include some contact information''' (ideally, your IRC nick) so that people can collaborate with you.
  
−
* change default ports - follow-on to https://review.openstack.org/#/c/118200/
+
''Historic "specs" are available at https://specs.openstack.org/openstack/swift-specs/''
−
* reverse listings - https://review.openstack.org/#/c/120709/
+
 
−
* stdin streaming to swiftclient
+
''Etherpads from previous meetups/hackathons/Design Summits/PTGs are available at [[Swift/Etherpads]]''
−
* deep health check to test all the way to drives (? or at least storage servers)
+
 
−
* better ring deployment inside of swift itself
+
== Ideas ==
−
* need internal net sec enforcement - https://blueprints.launchpad.net/swift/+spec/secure-internal-network-requests
+
 
−
* change the ssync (object-replicator) design to be run as another process for concurrency
+
Format: '''Idea''' -- ''link to your brain dump''
−
* durability simulator based on swift specification (e.g. # of replicas, # of devices and network speed to replicate) (inprogress by Christian)
+
 
−
* in the situation where a proxy needs to talk to a storage node on the same server, call the storage node method(s) directly instead of putting bytes on the network (somewhat in progress)
+
* '''Small file optimization''' -- [[Swift/ideas/small files]]
−
* skeleton middleware, well-commented
+
** '''Small files experimentations''' [[Swift/ideas/small files/experimentations]]
−
* DiskFile entry point
+
** '''Small files implementation''' [[Swift/ideas/small files/implementation]] -- [https://docs.google.com/document/d/1KoLsqWiXv9u2rnMZ5Rh73Js-lwP5h2qRj9CuFJ2IJJc LOSF Design doc]
−
* policy descriptions, exactly how https://review.openstack.org/#/c/120424/ is not implemented (see the review comments)
+
* '''Reduce memcache lookups''' -- [[Swift/ideas/memoize lookups]]
−
* allow multiple storage policy names for one policy
+
* '''Improve internal network security''' -- [[Swift/ideas/network_security]]
−
* rewrite server-side-copy (COPY and PUT with x-copy-from) as middleware (in progress https://review.openstack.org/#/c/156923/)
+
* '''Metrics around rate-limiting''' -- [[Swift/ideas/ratelimiting_metrics]]
−
* utility to "undelete" accounts, as described in http://docs.openstack.org/developer/swift/overview_reaper.html
+
* '''Hierarchical keymaster''' -- [[Swift/ideas/hierarchical_keymaster]]
−
* add account-to-account server-side copy docs. actually just add server-side copy docs
+
* '''Container sharding improvements''' - https://etherpad.openstack.org/p/sharding-improvements [https://etherpad.openstack.org/p/swift-auto-sharding auto-sharding-thoughts]
−
* Exposed allowed and denied headers on tempurl requests in /info. See http://docs.openstack.org/developer/swift/middleware.html#tempurl
+
** Auto-Sharding:
−
* expose container sync enablement in /info
+
*** [https://docs.google.com/document/d/15yG9zOatwPyXCA3YLNLdNIue9qDR4wLyMPIbypcbALc/edit?usp=sharing auto sharding 2.0]
−
* API endpoint for invalidating cached auth tokens
+
*** Election algorithm -- https://docs.google.com/document/d/17NllKQmH6tfTsKm5nAx3KCKUvs0zs_qamXtkreOQDWg/edit#
−
* specify the storage policy for functests to use. or test every exposed policy
+
*** Shardrange audit -- https://etherpad.opendev.org/p/swift-autosharding-shardrange-audit
−
* container reaper that also sets a header on the container to show it's being deleted
+
* '''High-latency media / Tape support for Swift''' -- [[Swift/HighLatencyMedia]]
−
* COPY with ranges: send 1 or more ranges with a copy request that allows for those ranges to be copied to a new file
+
* '''The Archival Storage extension to the Swift API''' -- [https://etherpad.openstack.org/p/high-latency-storage-policy]
−
* deprecate threads_per_port setting
+
* '''Metadata index (Elasticsearch)''' -- [[Swift/ideas/metadata-sync]]
−
* use flat xor erasure codes for testing to eliminate a testing dependency on jerasure or isa-l (maybe. there is still an outstanding question of if this is a good idea)
+
* '''Keymaster v2''' -- [[Swift/ideas/keymaster_v2]]
−
* allow some sort of "None" or empty referrer ACL as valid. eg allow these referrers and empty ones but not others
+
* '''oslo.config''' -- https://etherpad.openstack.org/p/swift_oslo-config
−
* Use O_TMPFILE under Linux during object creation: https://review.openstack.org/162243
+
* '''Auto Tiering Basic Infra work''' -- https://etherpad.openstack.org/p/swift_tiering_foundational_work
−
* Allow listings to be sorted/fetched by last-modified time
+
* '''txt lookup middleware''' -- https://wiki.openstack.org/wiki/Swift/ideas/txt_lookup_middleware
 +
* '''Tasks execution''' - [[Swift/ideas/task-execution]] [https://docs.google.com/document/d/11sBbB6pBvLYNeM9wjTdvvsJIu8Dl8i13UH2NfRVNOqg/edit?usp=sharing Expiring Objects - Task Queue]
 +
* '''Moar Better Faster Rebalance (tsync)''' -- https://etherpad.openstack.org/p/swift-rebalance
 +
* '''More disks in saio''' -- https://etherpad.openstack.org/p/more-disks-in-saio
 +
* '''Update python-swiftclient with Keystone session support''' -- [[Swift/ideas/swiftclient-keystone-session]]
 +
* '''Enable 'atime' for objects''' -- https://etherpad.openstack.org/p/atime_for_Swift
 +
* '''Probe tests with more devices''' -- [[Swift/ideas/more-testing-devices]]
 +
* '''Automatically set db_prealloc for account/containers based on if the drive is HDD or SDD'''
 +
* '''Object copy directly between object-servers''' -- [[Swift/ideas/object-copy-between-object-servers]]
 +
* '''Account quota for number of files''' -- [[Swift/ideas/account-quota-files]]
 +
* '''Use SO_REUSEPORT''' -- https://etherpad.openstack.org/p/PDumlTZ8uX
 +
* '''Third party transfers between Swift installations''' -- [[Swift/ideas/third-party-transfers]]
 +
* '''Add domains in ACLs''' -- [[Swift/ideas/acl-domains]]
 +
Done:
 +
 
 +
* <s>'''friendly tempurl timestamps''' -- https://blueprints.launchpad.net/swift/+spec/human-readable-tempurl-timestamp
 +
* '''Composite Rings''' -- https://etherpad.openstack.org/p/composite_rings
 +
</s>
 +
* <s>'''Symlinks''' -- https://etherpad.openstack.org/p/swift_symlinks</s>
 +
* <s>'''Container Sharding''' -- [https://docs.google.com/document/d/1-0wUnaRf2TTA0KQGk301BPR8-5wzuyA1r4tLSEh-cbw/edit?usp=sharing container-sharding.odt] [https://etherpad.openstack.org/p/container-sharding-SAT-2016 Etherpad notes] [https://trello.com/b/z6oKKI4Q/container-sharding trello] [https://docs.google.com/document/d/16rPdDrT-4G6hcen_6no8cr8KbxUqbcvsSkOfxxu0EBE/edit?usp=sharing Initial locking db shard approach idea] [https://docs.google.com/document/d/1dsROCHrgM0S_95bNp1BqD9DUUuQbkuoBxKkrhsNNeX0/edit?usp=sharing Sharder 2.0]
 +
** https://etherpad.openstack.org/p/deep-containers</s>
 +
* <s>'''Enable object path obfuscation in logs e.g. replace with hash''' - suggested here https://etherpad.openstack.org/p/BOS-Swift-ops-feedback-session</s>
 +
** Implemented in https://github.com/openstack/swift/commit/a4cc353375264e4bcd8f9aef2f0701ae800e0e73
  
 
----
 
----
  
 
[[Category:ObjectStorage]]
 
[[Category:ObjectStorage]]

Latest revision as of 00:07, 1 May 2020

Ideas for OpenStack Swift

This page is a collection of "brain dumps" for ideas about features in OpenStack Swift. If you're working on something, it's a very good idea to write down what you're thinking about. This lets others get up to speed, helps you collaborate, and serves as a great record for future reference. Write down your thoughts somewhere and put a link to it here. It doesn't matter what form your thoughts are in; use whatever is best for you. Your document should include why your idea is needed and your thoughts on particular design choices and tradeoffs. Please include some contact information (ideally, your IRC nick) so that people can collaborate with you.

Historic "specs" are available at https://specs.openstack.org/openstack/swift-specs/

Etherpads from previous meetups/hackathons/Design Summits/PTGs are available at Swift/Etherpads

Ideas

Format: Idea -- link to your brain dump

Done:

Retrieved from "https://wiki.openstack.org/w/index.php?title=Swift/ideas&oldid=174722"