Latest revision as of 00:07, 1 May 2020

Ideas for OpenStack Swift

This page is a collection of "brain dumps" for ideas about features in OpenStack Swift. If you're working on something, it's a very good idea to write down what you're thinking about. This lets others get up to speed, helps you collaborate, and serves as a great record for future reference. Write down your thoughts somewhere and put a link to it here. It doesn't matter what form your thoughts are in; use whatever is best for you. Your document should include why your idea is needed and your thoughts on particular design choices and tradeoffs. Please include some contact information (ideally, your IRC nick) so that people can collaborate with you.

Historic "specs" are available at https://specs.openstack.org/openstack/swift-specs/

Etherpads from previous meetups/hackathons/Design Summits/PTGs are available at Swift/Etherpads

Ideas

Format: Idea -- link to your brain dump

• Small file optimization -- Swift/ideas/small files
  • Small files experimentations Swift/ideas/small files/experimentations
  • Small files implementation Swift/ideas/small files/implementation -- LOSF Design doc
• Reduce memcache lookups -- Swift/ideas/memoize lookups
• Improve internal network security -- Swift/ideas/network_security
• Metrics around rate-limiting -- Swift/ideas/ratelimiting_metrics
• Hierarchical keymaster -- Swift/ideas/hierarchical_keymaster
• Container sharding improvements - https://etherpad.openstack.org/p/sharding-improvements auto-sharding-thoughts
  • Auto-Sharding:
    • auto sharding 2.0
    • Election algorithm -- https://docs.google.com/document/d/17NllKQmH6tfTsKm5nAx3KCKUvs0zs_qamXtkreOQDWg/edit#
    • Shardrange audit -- https://etherpad.opendev.org/p/swift-autosharding-shardrange-audit
• High-latency media / Tape support for Swift -- Swift/HighLatencyMedia
• The Archival Storage extension to the Swift API -- [1]
• Metadata index (Elasticsearch) -- Swift/ideas/metadata-sync
• Keymaster v2 -- Swift/ideas/keymaster_v2
• oslo.config -- https://etherpad.openstack.org/p/swift_oslo-config
• Auto Tiering Basic Infra work -- https://etherpad.openstack.org/p/swift_tiering_foundational_work
• txt lookup middleware -- https://wiki.openstack.org/wiki/Swift/ideas/txt_lookup_middleware
• Tasks execution - Swift/ideas/task-execution Expiring Objects - Task Queue
• Moar Better Faster Rebalance (tsync) -- https://etherpad.openstack.org/p/swift-rebalance
• More disks in saio -- https://etherpad.openstack.org/p/more-disks-in-saio
• Update python-swiftclient with Keystone session support -- Swift/ideas/swiftclient-keystone-session
• Enable 'atime' for objects -- https://etherpad.openstack.org/p/atime_for_Swift
• Probe tests with more devices -- Swift/ideas/more-testing-devices
• Automatically set db_prealloc for account/containers based on if the drive is HDD or SDD
• Object copy directly between object-servers -- Swift/ideas/object-copy-between-object-servers
• Account quota for number of files -- Swift/ideas/account-quota-files
• Use SO_REUSEPORT -- https://etherpad.openstack.org/p/PDumlTZ8uX
• Third party transfers between Swift installations -- Swift/ideas/third-party-transfers
• Add domains in ACLs -- Swift/ideas/acl-domains

Done:

• friendly tempurl timestamps -- https://blueprints.launchpad.net/swift/+spec/human-readable-tempurl-timestamp
• Composite Rings -- https://etherpad.openstack.org/p/composite_rings

• Symlinks -- https://etherpad.openstack.org/p/swift_symlinks
• Container Sharding -- container-sharding.odt Etherpad notes trello Initial locking db shard approach idea Sharder 2.0
  • https://etherpad.openstack.org/p/deep-containers
• Enable object path obfuscation in logs e.g. replace with hash - suggested here https://etherpad.openstack.org/p/BOS-Swift-ops-feedback-session
  • Implemented in https://github.com/openstack/swift/commit/a4cc353375264e4bcd8f9aef2f0701ae800e0e73