Jump to: navigation, search

Milk

Revision as of 21:13, 4 June 2014 by Sarob (talk | contribs)

ETC Cloud API Prototype project

Project Lead Sean Roberts (sarob)

purpose

The Entertainment Technology Center @ the University of Southern California (ETC) formally launched “Production in the Cloud,” a new project that brings together a core group of key media and cloud-resource leaders to develop guidelines and accelerate innovation and adoption of next-gen cloud-based content creation, production, and distribution tools and processes. Senior executives from the six major studios in coordination with Rackspace, EMC, EVault, Front Porch Digital, DAX, Google and other cloud companies convened recently to serve as governing body to collectively guide this process. The project is looking at the life cycle of film and media production, from pre-production collaboration, production, post production and through to archiving.

This specific part of the effort is around

  • Bringing together various competitive organizations to work on a common goal
  • Goal of developing an interoperable cloud framework
  • First steps of socializing well underway
  • Next important step of creating a prototype was discussed in October
  • Now we need to execute on design and implementation

timeline

  • 15 October 2013: Start design COMPLETED
  • November 2013: Setup Continuous Integration (CI) system COMPLETED
  • December 2013: Prototype development starts INPROGRESS
  • 07-08 April 2014: Present at NAB
  • May 2014 start design on next stage ETC clould projects framework, archive, transport, security, metadata
  • Aug 2014 CineGrid Brazil n


group structure

design

High Level Goals

  • Ingesting data from the many changes throughout the media pipeline
  • Metadata from the data stored and the pipeline activities
  • URL/URN marker identification for assets

details

  • look to openstack project Trove as the first step to database services layer, API supported now
  • ApiaryIO frame
  • push WSME, Pecan code with dummy verbs into repo
  • Create API with WSME/Pecan support metadata return function; PUT, POST, GET

reference.

  1. metadata: first pass at identification process and markers File:ETC process examples.pdf. We will use some parts of the EIGR format for the data object URI.
  2. archive: TBD
  3. transport: TBD
  4. security: TBD
  5. framework: this project
C4 as a code base for some of the services

C4 is a Desktop/Cloud image production platform currently in development at Studio Pyxis. It is built on top of a Domain Specific Language (DSL) for high preferment cloud integrated imaging applications that will be made open source in the near future.

The C4 DSL includes an API interface to communicate with remote resources that may be useful as a reference for the ETC cloud project. Following are some highlights of the API.

Resources that are accessible through a REST API are grouped into abstract categories.

  • Identity - Abstraction of "account" for individual users, and organizations
    • User
    • Company
    • Group - AKA friends, club, team, etc.
    • Family
    • Agent - AKA parent, representative, lawyer, etc.
  • Assets - Open ended abstract asset class identified by the unique SHA256 digest of the asset in question.
    • Image
    • Video
    • Sound
    • Document
  • Groups - Groups are usually, but not always hierarchical
    • File System - AKA folder or directory
    • Project
    • Portfolio
    • Tag
    • Store
    • Collection
    • Archive
    • Time Line - AKA clip
    • Version
    • Stream
    • Transformation - Geometric transformation, i.e. Translate, Rotate, Scale, etc.
  • Actions - Operations, usually version controlled, that can be applied to other resources.
    • Process - Data transformation operations such as image processing, mixing, etc.
    • Movement - Operations that physically modify the location of a resource on the network, copy, move, delete, etc.
    • Human - A human only process on an asset such as approve, comment on, create or modify
    • Cryptographic
    • Interactive
  • Resources
    • Storage
    • CPU
    • GPU
    • Transport Channel
  • Links - Relationships between resources
    • Permissions
    • Relationship ("agent", "friend", "employee", "team member", etc)
    • Attributes
    • Foreign Account

Contributing

  1. Create a Launchpad account
    https://login.launchpad.net/+new_account
  2. Join the OpenStack Foundation free
    https://www.openstack.org/join
    (Use the email address you plan to use with git for code contributions)
  3. Agree to the the Individual Contributor License agreement:
    https://review.openstack.org/#/settings/
    Look under "Agreements" on the left menu.
  4. Add an SSH public key to your launchpad account
    https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair
  5. Add the same SSH key to your OpenStack Foundation / Gerrit Identity as well
    https://review.openstack.org/#/settings/
    Look under "SSH Public Keys" on the left menu.
  6. Add the same SSH keys to your GitHub account
    Be sure to configure git on your development machine as well if you haven't already.
    $ git config --global user.name "Firstname Lastname"
    $ git config --global user.email "your_email@youremail.com"
  7. Install git-review. (In addition to python3, and pip3 )
    $ sudo pip install git-review
  8. Clone the GitHub project
    $ git clone https://github.com/stackforge/milk
  9. Work on the project (fix bugs, implement requirement, contribute code, etc)
  10. Submit your patch for review
    $ git review
  11. Track your contributions
    https://review.openstack.org/#/
  12. If necessary manually add a reviewer
    Click on your patch listed from #10 above enter one of the following names or emails in the "Add Reviewer" field, and click add.
    "Sean Robers"
    "Joshua Kolden"

Conceptual Services To build

API / daemon names function
IO API client generate checksum, client will post projectID, data type, original date&time, data binary, auth, extension for client defined metadata in XML format
IO API auth through keystone API, passes to IO daemon
IO daemon support 15-20 data types, generate reference URN, listens to API, put raw binary into swift API
state daemon zookeeper storage, monitor workflow, trigger alert on workflow timeouts, errors
transform daemon trigger on IO swift change, check on data type, if match some data types, execute transform, FFMBC for transform, support collaborative standards like scripte XML transform
String metadata API register service against asset ID, register metadata (?), register user/service provider (for now, use keystone later), register service, auth, lookup asset ID, add parent/peer Spring service, configure synchronization set
String metadata API client ...
concept description
Example Example

IO API

verb URI description
Example Example Example

metadata API

verb URI description
Example Example Example

Work to support APIs from

  • DAX, PIX, Critique, 5thKind, CineGlass, OpenStack, FIMS, Scripte, Shotgun, C4

Reference

  • 12nov2013 ETC production in the cloud board meeting

  • etcc-design-meeting-17dec2013

  • ETC process examples

  • String Server API draft

ETCc Gdrive with background documents


Next Stage projects likely will be built on top of the OpenStack projects Congress and Solum

source

Link here to Milk stackforge repository

review status

Link here to Milk gerrit review page

project activity

overview of project contributors and activity to date

Last edit 54923 by Sarob on 20140604211305 (year+month+day+UTC hour+seconds)
Retrieved from "https://wiki.openstack.org/w/index.php?title=Milk&oldid=54923"