Latest revision as of 07:10, 19 October 2015

RACK (Real Application Centric Kernel)

Project Resources

Resource	English	Japanese
Wiki	https://wiki.openstack.org/wiki/RACK	https://wiki.openstack.org/wiki/RACK/ja
Source code	https://github.com/openstack/rack
Deployment guide	https://github.com/openstack/rack/tree/master/tools/setup	https://github.com/openstack/rack/blob/master/tools/setup/README_ja.md
Sample applications	https://github.com/ctc-g/rack-sample-apps/tree/master/montecarlo https://github.com/ctc-g/rack-sample-apps/tree/master/shell-like	https://github.com/ctc-g/rack-sample-apps/tree/master/montecarlo/README_ja.md https://github.com/ctc-g/rack-sample-apps/tree/master/shell-like/README_ja.md
Python RACK client	https://github.com/openstack/python-rackclient

The concept of RACK: "OpenStack Native Application"

OpenStack Native Application is the software, which uses OpenStack resource (eg. VM or VNET) directly from application. Recent popular applications are designed before the cloud computing age, so affinity with cloud is not considered. In order to make those applications work on OpenStack, tools such as Chef, Puppet and other tools are required, and it makes systems very complex in design.

RACK provides the mechanism to create “After the Cloud” applications. Programmer can write codes that are scalable and migratable on OpenStack platform without cooperating with the external systems.

Concepts of RACK are as follows:

RACK handles VM with "functions" as a single execution binary file. “Functions” here means OS, middleware and programs that are necessary for application to function. The programs here are made in such a way as to call and operate RACK API.
When this execution binary is deployed onto OpenStack, the VM will behave like a Linux process and then finish its own task.
This process is based on the descriptions in the program. It does things such as forking and generating a child process, communicating between processes.

Please take a look at the sample programs to experience RACK!

What is RACK?

RACK defines data structure in order for VM to behave like a "process". It wraps Nova, Neutron and other APIs. Also it provides API to control data communication and sharing. Applications have to be scripted by order of calling the APIs. The RACK also offers libraries [1] for using APIs. It is very easy to write OpenStack Native Application codes by using the libraries. The RACK may be recognized as framework for writing OpenStack Native Application.

What we call "process"

When using RACK, you might often hear a word “process”. This process is a virtual machine instance on Nova. However, this instance is created in such a way as to perform some tasks when it is started. This is similar to various execution files of Linux, which start to operate when "ls" command or "grep" command is executed. In Linux, the started execution file will become a process on memory, it handles tasks, exits and disappears from memory at the end. RACK reproduces this concept on OpenStack. It deploys the glance image (with features) on Nova, and the deployed instance performs its task and then disappears at the end. This is the reason why a virtual machine is called "process" in the RACK world.

The VM instance that is started as a process has a PID, possible to perform "Fork" and it has a parent-child relationship. Additionally, it is possible to communicate without IP address and share data among these processes (VMs) that have such relationships. The file system with the same name space is also available, and the programmer who creates this execution binary can describe the OpenStack Native Application as if programming a Linux.

Architecture

RACK has a database defined with the data structure to handle VM as a process. RACK provides Restful API to manipulate the data structure. Once it receives an order from the application, it calls OpenStack API, and then writes the attribute info of the process into its own database.

Attibutes of a process

The attribute information of the process includes the followings;

keypair, security-group, network

Each process holds above IDs as attribute information.

GID (group ID)

Each process belongs to one group. The process belongs to the same group has a benefit such as file sharing in the same name space.

PID, PPID

Each process holds its own ID (PID) and the parent's IP (PPID). These IDs represent the parent-child relationship between processes.

rack-proxy

There is a "rack-proxy" process in each group. "rack-api" prodives the public API, and the "rack-proxy" on the other hand provides an accessible API only to the particular group. File system and interprocess communication would be handled by "rack-proxy".

Deployment of rack-api and rack-proxy

"rack-api" is deployed in an independent network and it is accessible from public. "rack-proxy", which exists in every group, is booted by calling the "rack-api" API. "rack-api" and all "rack-proxy" share the same database.

Curretnly, "rack-api" and "rack-proxy" are booted as VM. Our future plan includes followings:

rack-api runs as Linux process in OpenStack controller node like nova-api.
rack-proxy runs as Linux process in OpenStack network node like nutron-ns-metadata-proxy.

Since this area is closely related to the OpenStack core project, they are deployed as VM at the moment.

Interprocess communication support

RACK has the ability to support interprocess communication and data sharing as well as data structure manipulation, booting process and shutting down as described above. They are realized by combining libraries.

File system

It provides a file system that is shared within the group. Process can access to files with the same name space just like accessing to local file system.

Shared memory

This is a function for sharing simple data between processes. It is usable when accessing the same data by multiple processes.

Pipe

It provides Unix-pipe like functionality. The result of process can be piped to the other process very easily. No need to know the peer IP address or wait until the peer boots.

Signal

It is a function that can send strings to any process only by specifying a PID. It may send the strings to multiple processes by multicast.

Use cases

You can use RACK in any ways. Anything from adding/deleting process, data sharing and data notification among processes are describable by program. Using RACK enables you to develop applications such as OpenStack Native and others easily.

As tips on how to use RACK, some sample applications are available. We advise you to try these sample applications first and experience the OpenStack Native Application.

@@ Line 1: / Line 1: @@
 = RACK (Real Application Centric Kernel) =
 == Project Resources ==
@@ Line 12: / Line 13: @@
 |-
 | Source code
-| https://github.com/stackforge/rack
+| https://github.com/openstack/rack
 |
 |-
 | Deployment guide
-| https://github.com/stackforge/rack/tree/master/tools/setup
+| https://github.com/openstack/rack/tree/master/tools/setup
-| https://github.com/stackforge/rack/blob/master/tools/setup/README_ja.md
+| https://github.com/openstack/rack/blob/master/tools/setup/README_ja.md
 |-
 | Sample applications
 |
-* https://github.com/stackforge/rack/tree/master/tools/sample-apps/montecarlo
+* https://github.com/ctc-g/rack-sample-apps/tree/master/montecarlo
-* https://github.com/stackforge/rack/blob/master/tools/sample-apps/shell-like
+* https://github.com/ctc-g/rack-sample-apps/tree/master/shell-like
 |
-* https://github.com/stackforge/rack/blob/master/tools/sample-apps/montecarlo/README_ja.md
+* https://github.com/ctc-g/rack-sample-apps/tree/master/montecarlo/README_ja.md
-* https://github.com/stackforge/rack/blob/master/tools/sample-apps/shell-like/README_ja.md
+* https://github.com/ctc-g/rack-sample-apps/tree/master/shell-like/README_ja.md
 |-
 | Python RACK client
-| https://github.com/stackforge/python-rackclient
+| https://github.com/openstack/python-rackclient
 |
 |}
-== OpenStack Native Application ==
-The present applications were designed at "before the cloud".
-As those applications are not intended for cloud, they need other tools such as Chef, Puppet, Ansible, Serf, etc to utilize the cloud power.
-Thus the systems tend to become more complex on the cloud. We must think the "after the cloud" application now.
-The “after the cloud” application we think satisfies below.
-# The application logic determines amount of required resources by itself.
-# The application logic iterates allocations/releases of resources from cloud.
-# Those allocations/releases behave like creation/deletion of process.
-We named the application with such behavior “OpenStack Native Application”.
+== The concept of RACK: "OpenStack Native Application" ==
+OpenStack Native Application is the software, which uses OpenStack resource (eg. VM or VNET) directly from application.
+Recent popular applications are designed before the cloud computing age, so affinity with cloud is not considered.
+In order to make those applications work on OpenStack, tools such as Chef, Puppet and other tools are required, and it makes systems very complex in design.
-== What is RACK? ==
+RACK provides the mechanism to create “After the Cloud” applications.
+Programmer can write codes that are scalable and migratable on OpenStack platform without cooperating with the external systems.
-RACK enables an application to control VMs like a Linux process.
+Concepts of RACK are as follows:
-It provides an application with PID, parent-child relationship, “Fork” / “Kill” VMs capabilities, interVM communication and message exchange without VM’s IP address.
+# RACK handles VM with "functions" as a single execution binary file. “Functions” here means OS, middleware and programs that are necessary for application to function. The programs here are made in such a way as to call and operate RACK API.
-Ultimately, RACK enables you to implement a large scale distributed system in a variety of programming languages on OpenStack.
+# When this execution binary is deployed onto OpenStack, the VM will behave like a Linux process and then finish its own task.
+# This process is based on the descriptions in the program. It does things such as forking and generating a child process, communicating between processes.
-== VM like a Linux process? ==
+Please take a look at the sample programs to experience RACK!
+* https://github.com/ctc-g/rack-sample-apps/tree/master/montecarlo
-Look at the picture below.
+* https://github.com/ctc-g/rack-sample-apps/tree/master/shell-like
-It indicates the image of Linux process inside RACK.
-Generally, an executable is made through the process of compile and link, and operating system manages it as process when it is loaded into memory.
-Inside RACK, VM image represents executable and it includes OS and middleware as well as an application.
-When the executable (actually VM image) is launched on OpenStack, RACK gives it a process ID (PID).
-Additionally, when the process executes "Fork", a child process is launched, and given PID and a parent PID (PPID).
 [[File:RACK-Process-Image.png]]
-== Features ==
+== What is RACK? ==
-RACK provides some features as follows.
+RACK defines data structure in order for VM to behave like a "process".
+It wraps Nova, Neutron and other APIs.
+Also it provides API to control data communication and sharing.
+Applications have to be scripted by order of calling the APIs.
+The RACK also offers libraries [https://github.com/openstack/python-rackclient|python-rackclient] for using APIs.
+It is very easy to write OpenStack Native Application codes by using the libraries.
+The RACK may be recognized as framework for writing OpenStack Native Application.
-[[File:RACK-Features.png]]
-* '''Data Structure'''<br />In order to manage a VM like a Linux process, RACK adds some additional attributes such as PID(process ID), PPID(parent's process ID) and GID(process group ID) to it.
+== What we call "process" ==
-* '''Interprocess Communication'''<br />Processes can send a message with each other without knowing each other's IP address.
+When using RACK, you might often hear a word “process”.
+This process is a virtual machine instance on Nova.
+However, this instance is created in such a way as to perform some tasks when it is started.
+This is similar to various execution files of Linux, which start to operate when "ls" command or "grep" command is executed.
+In Linux, the started execution file will become a process on memory, it handles tasks, exits and disappears from memory at the end.
+RACK reproduces this concept on OpenStack.
+It deploys the glance image (with features) on Nova, and the deployed instance performs its task and then disappears at the end.
+This is the reason why a virtual machine is called "process" in the RACK world.
-* '''Shared Memory'''<br />Processes can share the data such as the data to process and their outputs.
+The VM instance that is started as a process has a PID, possible to perform "Fork" and it has a parent-child relationship.
+Additionally, it is possible to communicate without IP address and share data among these processes (VMs) that have such relationships.
+The file system with the same name space is also available, and the programmer who creates this execution binary can describe the OpenStack Native Application as if programming a Linux.
-* '''File System'''<br />There are multiple ways to use this, storing the data file to process and the output file, and sharing the data file between some processes.
+== Architecture ==
-In near future we plan to provide following features.
+RACK has a database defined with the data structure to handle VM as a process.
+RACK provides Restful API to manipulate the data structure.
+Once it receives an order from the application, it calls OpenStack API, and then writes the attribute info of the process into its own database.
-* '''Pipeline'''<br />The feature that chains a process to other by their standard streams just like Unix pipeline.
+[[File:RACK-Architecutre-Overview.png]]
-* '''Zombie process Collector'''<br />When a process failed to be killed, it executes an endless loop processing, or its parent process is killed, it becomes a zombie process. Zombie process collector detects these processes and clean up.
+=== Attibutes of a process ===
+The attribute information of the process includes the followings;
-* '''Compiler'''<br />VM image creation automation tool. Compiler assists you to create VM image adapted RACK.
+* keypair, security-group, network
+Each process holds above IDs as attribute information.
-* '''Debugger'''<br />This is useful for developing an application. Process is typically killed soon when it finishes job, so you can't examine program state and track down the origin of the problem. Debugger supports these tasks.
+* GID (group ID)
+Each process belongs to one group.
+The process belongs to the same group has a benefit such as file sharing in the same name space.
+* PID, PPID
+Each process holds its own ID (PID) and the parent's IP (PPID).
+These IDs represent the parent-child relationship between processes.
-== Architecture Overview ==
+=== rack-proxy ===
+There is a "rack-proxy" process in each group.
+"rack-api" prodives the public API, and the "rack-proxy" on the other hand provides an accessible API only to the particular group.
+File system and interprocess communication would be handled by "rack-proxy".
-RACK provides above features as restful service.
+=== Deployment of rack-api and rack-proxy ===
-Application only communicates with RACK via useful library we provide.
+"rack-api" is deployed in an independent network and it is accessible from public.
+"rack-proxy", which exists in every group, is booted by calling the "rack-api" API.
+"rack-api" and all "rack-proxy" share the same database.
-[[File:RACK-Architecutre-Overview.png]]
+[[File:RACK-Topology.png]]
+Curretnly, "rack-api" and "rack-proxy" are booted as VM.
+Our future plan includes followings:
-== Pseudo Code ==
+* '''rack-api runs as Linux process in OpenStack controller node like nova-api.'''
+* '''rack-proxy runs as Linux process in OpenStack network node like nutron-ns-metadata-proxy.'''
-This is the image of program adapted RACK.
-You can simply develop a distributed application as below.
-You don't need to write complex and unique code.
-Especially, you don't need to write cloud-aware code, that is you don't need to know IP address.
-[[File:RACK-Pseudo-Code.png]]
+'''Since this area is closely related to the OpenStack core project, they are deployed as VM at the moment.'''
-== Use Cases ==
+=== Interprocess communication support ===
+RACK has the ability to support interprocess communication and data sharing as well as data structure manipulation, booting process and shutting down as described above.
+They are realized by combining libraries.
-You can use RACK in many cases. Followings are some examples.
+[[File:RACK-Features.png]]
-* You can implement a new architecture application. For example, you can build an application that calculates the necessary amount of computing resource(i.e. instance) depending on the data to process and launches additional instances dynamically. Then, the data will be processed very quickly since these instances work in parallel. This new architecture application is suitable for processing a large amount of data.
+* File system
+It provides a file system that is shared within the group.
+Process can access to files with the same name space just like accessing to local file system.
-* You can integrate existing system such as batch system with Hadoop and Web application using RACK. For example, RACK enables you to deploy Hadoop cluster easily and add autoscale function to your Web applications.
+* Shared memory
+This is a function for sharing simple data between processes.
+It is usable when accessing the same data by multiple processes.
-[[File:RACK-Use-Cases.png]]
+* Pipe
+It provides Unix-pipe like functionality.
+The result of process can be piped to the other process very easily.
+No need to know the peer IP address or wait until the peer boots.
+* Signal
+It is a function that can send strings to any process only by specifying a PID.
+It may send the strings to multiple processes by multicast.
-== Benchmark of OpenStack Native Application ==
-We conducted a benchmark test of OpenStack Native Application that behaves like the first example of above usecases.
+== Use cases ==
-The following graph shows the result.
-We can see one of the characteristics of OpenStack Native Application from this graph.
-This application scales worker processes out depending on the number of dataset, and these processes work in parallel, so the total execution time is held constant.
-Theoretically, the more it scales out, the shorter the execution time is.
-[[File:RACK-Benchmark.png]]
+You can use RACK in any ways.
+Anything from adding/deleting process, data sharing and data notification among processes are describable by program.
+Using RACK enables you to develop applications such as OpenStack Native and others easily.
-It's good for both customers and cloud providers.
+As tips on how to use RACK, some sample applications are available.
-For customers, it's less expensive to use a lot of low-spec instances to process data fast than a high spec instance.
+We advise you to try these sample applications first and experience the OpenStack Native Application.
-For cloud providers, a large amount of resources will come to be consumed.
+* https://github.com/ctc-g/rack-sample-apps/tree/master/montecarlo
+* https://github.com/ctc-g/rack-sample-apps/tree/master/shell-like

Difference between revisions of "RACK"