Revision as of 04:06, 24 February 2016

Meeting Time: Every Wednesday at 13:00 UTC in #openstack-storlets

Agenda:

• Prioritize patches (suggestion):

* https://review.openstack.org/#/c/277322/ (refactor test_storlet_handler)...merged
* https://review.openstack.org/#/c/280488/ (refactor validation methods)...merged
* https://review.openstack.org/#/c/276543/ (use logger exception).............merged
* https://review.openstack.org/#/c/280038/ (change info to debug logs).....merged
* COPY functionality - Still need to post the patch
* https://review.openstack.org/#/c/276545/ (Improve timeout handling)
* https://review.openstack.org/#/c/282135/ (Refactor unit tests about storlet docker gateway)
* https://review.openstack.org/#/c/280570/ (Implement readline/readlines in IterLike, depends on 282135)
* Putting the last two after COPY does not mean I will not review them before.

• Storlet in Hackathon etherpad
  • I would like to add another analytics related question.
  • What timeframe do we think we need?

Austin:

• We currently have the following abstract
  • Title: From Analytic to Image Processing in Swift with Openstack Storlets
  • Openstack Stolets is an emerging Openstack project aimed at pushing down data centric compute jobs to the object store. Storlets offer a server-less programming model enabling developers to quickly develop and deploy data path computations without worrying about the underlying platform details.
  • Storlets facilitate several interesting use cases such as:
    • Performing image processing in the storage, allowing to process large objects without downloading it.
    • Spark analytic push down to the store, saving on both bandwidth and memory in the analytic cluster.
    • Data obfuscation and anonymization at the source. Thus, preventing sensitive data from ever leaving the store.
  • In this talk we will present the project, discuss and demonstrate the use cases and present a PoC about an effective architecture for image processing using Storlets done at NTT.

Swift Hackathon

Swift Community Session

• See the following etherpad
  • https://etherpad.openstack.org/p/swift-hackathon-feb-2016-storlets

Storlets Community Session(s)

• Resource limitation (just ideas)
  • Limit the number of containers or processes running on one node
  • Limit resource consumptions in each containers
    • ex. core binding of docker container
  • stop unnecessary containers/processes
  • Set resource parameter about jvm: https://review.openstack.org/#/c/275083/ (patch should be fixed)
• Can we also add system header like Content-Type?
• Unit tests
  • for python modules
  • for Java modules
    • How can we integrate them into openstack gate testing?
• Documentation
• Refactoring ideas

Bugs

• Range handling: on going
  • https://bugs.launchpad.net/storlets/+bug/1534414
  • https://bugs.launchpad.net/storlets/+bug/1534455
• FD leak
  • FD leak in Storlet GET: https://bugs.launchpad.net/storlets/+bug/1537982
    • https://review.openstack.org/#/c/272379/
  • There are other possible FD leaks, which may happen in timeout/exception case.
    • Use context manager to make sure all FDs are closed: https://review.openstack.org/#/c/282211/ (patch should be fixed)