Difference between revisions of "Meetings/Storlets"
< Meetings
(→Agenda:: remove merged items) |
|||
| Line 3: | Line 3: | ||
=Agenda:= | =Agenda:= | ||
* Prioritize patches (suggestion): | * Prioritize patches (suggestion): | ||
| − | ** | + | ** No patches now ready for review |
| − | + | ** Incoming patches | |
| − | ** | + | *** Use contextmanager to manager file descriptors: https://review.openstack.org/#/c/282211/ |
| − | *** | + | **** COPY should be fixed |
| − | + | *** Refactor directory structure about python modules: https://review.openstack.org/#/c/285947/ | |
| − | ** | ||
| − | ** | ||
| − | *** | ||
| − | |||
| − | |||
**** Needs more work about fixing setup.py and rebasing | **** Needs more work about fixing setup.py and rebasing | ||
**** We should decide when we rebase/merge this, to avoid many merge conflicts caused by it. [takashi] | **** We should decide when we rebase/merge this, to avoid many merge conflicts caused by it. [takashi] | ||
| + | **** Move swift specific functionality from gateway to middleware | ||
* Storlet in Hackathon etherpad | * Storlet in Hackathon etherpad | ||
| Line 35: | Line 31: | ||
=Bugs= | =Bugs= | ||
| − | * Range handling: | + | * Range handling: fix released |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
* FD leak | * FD leak | ||
** FD leak in Storlet GET: https://bugs.launchpad.net/storlets/+bug/1537982 (Fix Released) | ** FD leak in Storlet GET: https://bugs.launchpad.net/storlets/+bug/1537982 (Fix Released) | ||
| Line 47: | Line 37: | ||
** There are other possible FD leaks, which may happen in timeout/exception case. | ** There are other possible FD leaks, which may happen in timeout/exception case. | ||
*** Use context manager to make sure all FDs are closed: https://review.openstack.org/#/c/282211/ (patch should be fixed) | *** Use context manager to make sure all FDs are closed: https://review.openstack.org/#/c/282211/ (patch should be fixed) | ||
| + | * Validate metadata in POST/COPY about starlet object: https://bugs.launchpad.net/storlets/+bug/1555411 | ||
| + | ** POST is already fixed: https://review.openstack.org/#/c/290952/ | ||
| + | ** How can we validate metadata when COPYing object outside starlet container into starlet container? | ||
Revision as of 11:31, 16 March 2016
Meeting Time: Every Wednesday at 13:00 UTC in #openstack-storlets
Contents
Agenda:
- Prioritize patches (suggestion):
- No patches now ready for review
- Incoming patches
- Use contextmanager to manager file descriptors: https://review.openstack.org/#/c/282211/
- COPY should be fixed
- Refactor directory structure about python modules: https://review.openstack.org/#/c/285947/
- Needs more work about fixing setup.py and rebasing
- We should decide when we rebase/merge this, to avoid many merge conflicts caused by it. [takashi]
- Move swift specific functionality from gateway to middleware
- Use contextmanager to manager file descriptors: https://review.openstack.org/#/c/282211/
- Storlet in Hackathon etherpad
- I would like to add another analytics related question.
- What timeframe do we think we need?
Austin:
- We currently have the following abstract
- Title: From Analytic to Image Processing in Swift with Openstack Storlets
- Openstack Stolets is an emerging Openstack project aimed at pushing down data centric compute jobs to the object store. Storlets offer a server-less programming model enabling developers to quickly develop and deploy data path computations without worrying about the underlying platform details.
- Storlets facilitate several interesting use cases such as:
- Performing image processing in the storage, allowing to process large objects without downloading it.
- Spark analytic push down to the store, saving on both bandwidth and memory in the analytic cluster.
- Data obfuscation and anonymization at the source. Thus, preventing sensitive data from ever leaving the store.
- In this talk we will present the project, discuss and demonstrate the use cases and present a PoC about an effective architecture for image processing using Storlets done at NTT.
Swift Hackathon
- See the following etherpad
Bugs
- Range handling: fix released
- FD leak
- FD leak in Storlet GET: https://bugs.launchpad.net/storlets/+bug/1537982 (Fix Released)
- https://review.openstack.org/#/c/272379/ ... merged
- There are other possible FD leaks, which may happen in timeout/exception case.
- Use context manager to make sure all FDs are closed: https://review.openstack.org/#/c/282211/ (patch should be fixed)
- FD leak in Storlet GET: https://bugs.launchpad.net/storlets/+bug/1537982 (Fix Released)
- Validate metadata in POST/COPY about starlet object: https://bugs.launchpad.net/storlets/+bug/1555411
- POST is already fixed: https://review.openstack.org/#/c/290952/
- How can we validate metadata when COPYing object outside starlet container into starlet container?
