Jump to: navigation, search

Difference between revisions of "Solum/FeatureBlueprints/GitIntegration"

(→‎Implementation)
m (→‎Implementation)
 
(42 intermediate revisions by 5 users not shown)
Line 1: Line 1:
−
[https://blueprints.launchpad.net/solum/+spec/git-integration/+edit Blueprint]
+
Blueprint: https://blueprints.launchpad.net/solum/+spec/git-integration
  
−
== Git Integration ==
+
== Git Deployment==
  
 
=== Functionality ===
 
=== Functionality ===
−
As an application developer, I want to push my code to my git repository, and have the platform automatically detect the code push and generate a running application using the new code.
+
A '''deployment unit (DU)''' is a ''unit of deployment'' for the application being deployed. For example, if Solum is configured to use containers for deploying application code, then a DU would refer to a container instance. <br /><br />
  
−
I would be able to use git push for deploying a new application, or to update an existing application.
+
=== Story 1: generate new app (minimal implementation) ===
 +
As an application developer, I want to push my code to the master branch of my git repository, and have the platform automatically detect the code push and generate a running application (i.e. an assembly) using the new code.
  
−
I would be able to specify the the meta-data for the application (AppName, app instance size, instance count, auto scale policies, etc) via an application manifest file. The application manifest file will be stored at the root location of my code project's directory, and will be automatically picked up by the platform in response to a git push.
+
I would be able to specify the application characteristics (AppName, size and number of deployment units, auto scale policies, etc) while deploying my app.  <br />
−
 
+
 
−
=== Implementation ===
+
Following are the application characteristics that can be specified (this is an indicative list, not intended to be a comprehensive list).
 +
 
 +
'''Application Characteristics'''
 +
{| class="wikitable"
 +
|-
 +
! Attribute name !! Description !! Required?
 +
|-
 +
| DU Count|| Initial number of deployment units.  || Optional (default = 1)
 +
|-
 +
| Flavor|| Size of each DU (as determined by RAM, CPU, disk space, allocated to each DU). Small, medium, large, extra large, etc. the sizing characteristics of what is small and what is large is configurable by the platform service provider || Optional (if no flavor is provided, then a default flavor will be used. The default flavor is configurable by the service provider)
 +
|-
 +
| AutoScale|| Should the application auto-scale (Y/N)|| Optional (default = N)
 +
|-
 +
| MinInstances || Minimum number of DU's (lower threshold for the auto scale policy)|| Optional (default = 1)
 +
|-
 +
| MaxInstances|| Maximum number of DU's (upper threshold for the auto scale policy)|| Optional (default = specified DU Count during app creation)
 +
|-
 +
| ShellEnvironment|| Set application specific shell environment variables at app start up|| Optional
 +
|-
 +
| LanguagePack|| User could specify the language pack (say MyPythonPack) to be used during app creation.  If no language pack is specified, then the platform would auto-detect the default language pack associated with the code being pushed. || Optional
 +
|-
 +
| Services|| List of services that the application will bind to (e.g.Trove, NewRelic). The services list would also optionally contain any service initialization parameters if the service support it. An example of this will be an SQL script passed to a database service to create the custom application scheme.  || Optional
 +
|-
 +
|}
 +
 
 +
The general workflow for a git based deployment is as follows:<br />
 +
1. Precondition: the user is already registered as an OpenStack user, and has been associated with the Solum service (see the blueprint "user authentication" for details)<br />
 +
2. Register[noun]: registers the new application with Solum (generates a git repository in Solum and associates it with the new app).<br />
 +
:* Inputs: Name (Optional) of the application, if no Name is provided, Solum will generate one
 +
:* Returns: URL of the remote git repository, and the URL associated with the app <br />
 +
3. Git push (deploys and generates a running app). Builds the app binary from source.
 +
Git push will use the default application characteristics (per the default values specified in the Application Characteristics table above). The default values can be overridden by specifying the values in an application meta-data file that is stored along with the application source.
 +
 
 +
While implementing this story, the build step and deploy step will be self-contained steps that can either be invoked together, or separately. This is needed to support "build once, deploy multiple" scenarios, for example when we implement environment support.
 +
 
 +
'''Side note:''' there will be multiple deployment paths supported by Solum. Deployments can happen via
 +
* Git (this blueprint)
 +
* CLI (where register and deploy will be combined to a single CLI operation exposed to the user, that packages code from a directory network accessible from the developer's workstation)
 +
* IDE Plugins
 +
* SDK (that calls into the REST API)
 +
* REST API
 +
All the deployment paths above will converge at the REST API: i.e. all the steps end up calling the REST API where all the business logic for build and deploy resides. All the other deployment mechanisms are "hooks" to do the necessarily pre-processing before calling the REST API. More details on the multiple deployment approaches will be documented in a separate blueprint (TBD).
 +
 
 +
=== Story 2: update app ===
 +
I would be able to use git push for updating an existing application. The update can include changes to the application code, or changes to the application characteristics (refer table in Story 1 above). I would like the update to happen with zero downtime (or at least near zero time) for my running app.
 +
 
 +
=== Story 3: specify git branch ===
 +
By default, what we implemented in Story 1 will always build from the master branch. As part of this story, I would like to be able to specify a specific branch to use for the build. I should be able to specify the branch in two ways -
 +
- specify the branch as part of each deployment request
 +
- configure a default branch to be used for each subsequent deployment
  
−
We posit that the git integration workflow would work as follows. To setup the git repository within Solum for a user's application, we propose that Solum API provide a 'register' resource. A user would invoke this resource and pass in the application name (AppName) as part of the request body. The Solum system will create a git repository for that user and the specified AppName. A 'post-receive-hook' will be created in this git repository (contents described below). The git repository URL will be sent back to the user in the response of the 'register' call. Users will have to add this URL as a 'remote' for their local repository. Let's call this as an application's solum remote. Adding a solum remote will be a one time step, which will need to be performed only once per application. Once the solum remote is setup, whenever the user pushes the code to it, the configured post-receive-hook will get triggered and perform actions which will lead to deployment of the application in the Solum system. The URL to access the application will be sent back to the user as part of the response of git push.
+
=== Story 4: Auto deploy Yes/No ===
 +
Auto deploy: I would be able to specify if a git push operation just performs a build, or performs a build+deploy to generate a running application (default behavior will be build+deploy)
  
−
At a high-level, the steps that need to happen for creating a running application from the pushed code include, (a) detecting the application's type (Java vs. Python, etc.), (b) compiling the application (if required), (c) adding/updating any required data services (for instance, a database), (d) deploying the application code to the appropriate runtime environment (like a Docker container) after setting up the required application specific environment on it.
+
Solum will implement Environments/Application Life Cycle management as part of the [[Solum/HighLevelRoadmap|roadmap]]. However, implementation of environments is out of scope of this blueprint.
  
−
In designing the contents of the post-receive-hook we have to answer the following question: which of the above steps are good candidates to be included within a post-receive-hook? We argue that steps (a) and (b) can potentially be performed in a post-receive-hook. The remaining steps (step (c) and step (d)) require knowledge about parts of the Solum system, such as available container resources, that may not be easily accessible within a post-receive-hook. Also, from design perspective post-receive-hook does not seem to be the right place to be performing steps (c) and (d).
+
=== Flow Boundaries ===
 +
Diagram for component areas and potential abstraction boundaries:
  
−
Now, even if steps (a) and (b) can be done in a post-receive-hook, one of the proposed Solum requirements is that users would be able to deploy their code to the Solum system directly, without having to register the application first (Put the deploy application API and blueprint link). This means that, in the workflow where users choose not to first register their applications with the Solum system, steps (a) and (b) need to occur as part of the implementation of the deploy application API call. So, if we implement steps (a) and (b) in the post-receive-hook, it will lead to duplication of functionality. This leads to the decision that neither of step (a) or (b) are good candidates for being included in a post-receive-hook as well.
+
[[File:Solum r01 flow.jpeg|framed|center]]
  
−
So, if neither of the above steps can be part of a post-receive-hook then what should be in it?
+
=== Implementation ===
  
−
The answer to this is provided by the fact that Solum will be supporting deploy application resource. The post-receive-hook can then just be a call to invoke this resource. This call will be exactly similar to the call that a user would make in order to directly invoke the deploy application resource (without going through the git integration workflow). The deploy application API call is discussed in blueprint (put deploy app blueprint link here).
+
[[https://wiki.openstack.org/w/images/a/ab/Solum_git-pull_flow.svg Sequence diagram for Git-Pull workflow]] based on discussions during Git workgroup [http://irclogs.solum.io/2014/solum.2014-02-05-17.03.log.html meeting on 02/05/2013]

Latest revision as of 07:51, 12 February 2014

Blueprint: https://blueprints.launchpad.net/solum/+spec/git-integration

Git Deployment

Functionality

A deployment unit (DU) is a unit of deployment for the application being deployed. For example, if Solum is configured to use containers for deploying application code, then a DU would refer to a container instance.

Story 1: generate new app (minimal implementation)

As an application developer, I want to push my code to the master branch of my git repository, and have the platform automatically detect the code push and generate a running application (i.e. an assembly) using the new code.

I would be able to specify the application characteristics (AppName, size and number of deployment units, auto scale policies, etc) while deploying my app.

Following are the application characteristics that can be specified (this is an indicative list, not intended to be a comprehensive list).

Application Characteristics

Attribute name Description Required?
DU Count Initial number of deployment units. Optional (default = 1)
Flavor Size of each DU (as determined by RAM, CPU, disk space, allocated to each DU). Small, medium, large, extra large, etc. the sizing characteristics of what is small and what is large is configurable by the platform service provider Optional (if no flavor is provided, then a default flavor will be used. The default flavor is configurable by the service provider)
AutoScale Should the application auto-scale (Y/N) Optional (default = N)
MinInstances Minimum number of DU's (lower threshold for the auto scale policy) Optional (default = 1)
MaxInstances Maximum number of DU's (upper threshold for the auto scale policy) Optional (default = specified DU Count during app creation)
ShellEnvironment Set application specific shell environment variables at app start up Optional
LanguagePack User could specify the language pack (say MyPythonPack) to be used during app creation. If no language pack is specified, then the platform would auto-detect the default language pack associated with the code being pushed. Optional
Services List of services that the application will bind to (e.g.Trove, NewRelic). The services list would also optionally contain any service initialization parameters if the service support it. An example of this will be an SQL script passed to a database service to create the custom application scheme. Optional

The general workflow for a git based deployment is as follows:
1. Precondition: the user is already registered as an OpenStack user, and has been associated with the Solum service (see the blueprint "user authentication" for details)
2. Register[noun]: registers the new application with Solum (generates a git repository in Solum and associates it with the new app).

  • Inputs: Name (Optional) of the application, if no Name is provided, Solum will generate one
  • Returns: URL of the remote git repository, and the URL associated with the app

3. Git push (deploys and generates a running app). Builds the app binary from source. Git push will use the default application characteristics (per the default values specified in the Application Characteristics table above). The default values can be overridden by specifying the values in an application meta-data file that is stored along with the application source.

While implementing this story, the build step and deploy step will be self-contained steps that can either be invoked together, or separately. This is needed to support "build once, deploy multiple" scenarios, for example when we implement environment support.

Side note: there will be multiple deployment paths supported by Solum. Deployments can happen via

  • Git (this blueprint)
  • CLI (where register and deploy will be combined to a single CLI operation exposed to the user, that packages code from a directory network accessible from the developer's workstation)
  • IDE Plugins
  • SDK (that calls into the REST API)
  • REST API

All the deployment paths above will converge at the REST API: i.e. all the steps end up calling the REST API where all the business logic for build and deploy resides. All the other deployment mechanisms are "hooks" to do the necessarily pre-processing before calling the REST API. More details on the multiple deployment approaches will be documented in a separate blueprint (TBD).

Story 2: update app

I would be able to use git push for updating an existing application. The update can include changes to the application code, or changes to the application characteristics (refer table in Story 1 above). I would like the update to happen with zero downtime (or at least near zero time) for my running app.

Story 3: specify git branch

By default, what we implemented in Story 1 will always build from the master branch. As part of this story, I would like to be able to specify a specific branch to use for the build. I should be able to specify the branch in two ways - 

- specify the branch as part of each deployment request
- configure a default branch to be used for each subsequent deployment

Story 4: Auto deploy Yes/No

Auto deploy: I would be able to specify if a git push operation just performs a build, or performs a build+deploy to generate a running application (default behavior will be build+deploy)

Solum will implement Environments/Application Life Cycle management as part of the roadmap. However, implementation of environments is out of scope of this blueprint.

Flow Boundaries

Diagram for component areas and potential abstraction boundaries:

Solum r01 flow.jpeg

Implementation

[Sequence diagram for Git-Pull workflow] based on discussions during Git workgroup meeting on 02/05/2013

Retrieved from "https://wiki.openstack.org/w/index.php?title=Solum/FeatureBlueprints/GitIntegration&oldid=41985"