Meetings/Storlets

Meeting Time: Every Wednesday at 13:00 UTC in #openstack-storlets

The meeting on Wed January 27th starts at 12 UTC.

Agenda:

• Austin talk.
• Swift Hackathon
• bugs.

Austin:

• We currently have the following abstract
  • Title: From Analytic to Image Processing in Swift with Openstack Storlets
  • Openstack Stolets is an emerging Openstack project aimed at pushing down data centric compute jobs to the object store. Storlets offer a server-less programming model enabling developers to quickly develop and deploy data path computations without worrying about the underlying platform details.
  • Storlets facilitate several interesting use cases such as:
    • Performing image processing in the storage, allowing to process large objects without downloading it.
    • Spark analytic push down to the store, saving on both bandwidth and memory in the analytic cluster.
    • Data obfuscation and anonymization at the source. Thus, preventing sensitive data from ever leaving the store.
  • In this talk we will present the project, discuss and demonstrate the use cases and present a PoC about an effective architecture for image processing using Storlets done at NTT.

Swift Hackathon

• The purpose of the session is to update the community on what's happening in the openstack-storlets project and get feedback on number of subjects (the intention is not to discuss pushing the middleware to Swift at this point).
  • Updates:
    • Unit tests (Thanks NTT)
      • storlets GET flow: https://review.openstack.org/#/c/263533/
      • storlets PUT flow: https://review.openstack.org/#/c/264520/
      • uploading storlets/dependency: https://review.openstack.org/#/c/264638/
    • Copy semantics (I got a working prototype)
    • Storlets samples (thumbnail extraction, more to come), what else?
  • Points for discussion:
    • Adding a non voting gate job for testing Storlets
    • Use footers for adding metadata
      • Can we also add system header like Content-Type?
    • Resource limitation (just ideas)
      • Limit the number of containers or processes running on one node
      • Limit resource consumptions in each containers
      • stop unnecessary containers/processes?
    • To-be-fixed thing in swift
      • chunked transferring support in proxy-server: https://review.openstack.org/#/c/256201/

Bugs

• Range handling
  • https://bugs.launchpad.net/storlets/+bug/1534414
  • https://bugs.launchpad.net/storlets/+bug/1534455
• FD leak
  • https://bugs.launchpad.net/storlets/+bug/1537982