Jump to: navigation, search

Difference between revisions of "Meteos/ExampleLinear"

(Created page with "== Predict sales using Meteos == In this example, user creates a prediction model which predict sales by using Linear Regression. Linear Regression is one of the algorithms i...")
 
(→‎Predict sales using Meteos)
Line 1: Line 1:
−
== Predict sales using Meteos ==
+
== Predict a Sales Figures using Meteos ==
  
−
In this example, user creates a prediction model which predict sales by using Linear Regression.
+
In this example, you create a prediction model which predict sales by using Linear Regression.  
−
Linear Regression is one of the algorithms in supervised learning.
 
  
−
''"Supervised learning is the machine learning task of inferring a function from labeled training data. The training data consist of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal).  (From Wikipedia https://en.wikipedia.org/wiki/Supervised_learning)''
+
Linear Regression is one of the algorithms in supervised learning.  
−
 
 
−
In this example, sales is the desired output value.
 
  
 
[[File:Usecase.png]]
 
[[File:Usecase.png]]
Line 15: Line 12:
 
Experiment is a workspace of Machine Learning.
 
Experiment is a workspace of Machine Learning.
  
−
'''$ cat json/create_template.json'''
+
You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.
−
  {
+
 
 +
You can use a format located in '''python-meteosclient/sample/json/template.json'''
 +
 
 +
<pre>
 +
$ glance image-list | grep meteos
 +
| 45de4bbd-8419-40ff-8ed7-fc065c05e34f | meteos                          |
 +
</pre>
 +
<pre>
 +
$ neutron net-list | grep public
 +
| 84c13e76-ced9-4142-a885-280784f1f7a3 | public | a14de1c5-b8d4-434b-a056-9b0049b93402            |
 +
</pre>
 +
<pre>
 +
$ vim sample/json/template.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/template.json
 +
{
 
     "display_name": "example-template",
 
     "display_name": "example-template",
 
     "display_description": "This is a sample template of experiment",
 
     "display_description": "This is a sample template of experiment",
−
     "image_id" : "c48f5dba-45be-4165-9825-f4564fecebcd",
+
     "image_id" : "45de4bbd-8419-40ff-8ed7-fc065c05e34f",
 
     "master_nodes_num": 1,
 
     "master_nodes_num": 1,
 +
    "master_flavor_id": "4",
 
     "worker_nodes_num": 2,
 
     "worker_nodes_num": 2,
−
     "spark_version": "1.6"
+
    "worker_flavor_id": "2",
−
}
+
     "spark_version": "1.6.0",
−
 
+
    "floating_ip_pool": "84c13e76-ced9-4142-a885-280784f1f7a3"
−
'''$ meteos template-create --json json/create_template.json'''
+
}
 +
</pre>
 +
<pre>
 +
$ meteos template-create --json sample/json/template.json
 +
+---------------+-----------------------------------------+
 +
| Property      | Value                                  |
 +
+---------------+-----------------------------------------+
 +
| cluster_id    | None                                    |
 +
| created_at    | 2016-12-04T07:16:29.000000              |
 +
| description  | This is a sample template of experiment |
 +
| id            | 8b7b9b89-f119-4b9b-b9b0-31598f819f1a    |
 +
| master_flavor | 4                                      |
 +
| master_nodes  | 1                                      |
 +
| name          | example-template                        |
 +
| project_id    | 67401cca74c2409b939e944bc6c8fcbe        |
 +
| spark_version | 1.6.0                                  |
 +
| status        | available                              |
 +
| user_id      | 181b1caa9d5b470393ca66b9e511d5b0        |
 +
| worker_flavor | 2                                      |
 +
| worker_nodes  | 2                                      |
 +
+---------------+-----------------------------------------+
 +
</pre>
  
−
+-------------+-----------------------------------------+
 
−
| Property    | Value                                  |
 
−
+-------------+-----------------------------------------+
 
−
| cluster_id  | 0984a3e7-cd8b-4b7e-b4f0-f58dc4fc6c28    |
 
−
| created_at  | 2016-10-20T02:10:13.483231              |
 
−
| description | This is a sample template of experiment |
 
−
| id          | 26cbc33b-179e-4d22-8b1b-23c9a305f196    |
 
−
| image_id    | c48f5dba-45be-4165-9825-f4564fecebcd    |
 
−
| name        | example-template                        |
 
−
| project_id  | 5f011d076a6f4a328989c57ac7b4e501        |
 
−
| status      | available                              |
 
−
+-------------+-----------------------------------------+
 
  
 
=== 2. Create a experiment from template ===
 
=== 2. Create a experiment from template ===
  
 
Create a experiment by using template created in the above step.
 
Create a experiment by using template created in the above step.
 +
You have to confirm a neutron private network id and create keypair before creating a template.
 +
 +
You can use a format located in '''python-meteosclient/sample/json/experiment.json'''
  
−
'''$ cat json/create_experiment.json'''
+
<pre>
−
{
+
$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem
 +
</pre>
 +
<pre>
 +
$ neutron net-list | grep private
 +
| 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | private | cb58940f-859b-48c6-b92a-3861470f1fc1 20.0.0.0/26 |
 +
</pre>
 +
<pre>
 +
$ vim sample/json/experiment.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/experiment.json
 +
{
 
     "display_name": "example-experiment",
 
     "display_name": "example-experiment",
 
     "display_description": "This is a sample experiment",
 
     "display_description": "This is a sample experiment",
 
     "key_name": "key1",
 
     "key_name": "key1",
−
     "net_id": "bc85cb2a-53ad-4375-801e-ed507d416e09",
+
     "neutron_management_network": "8abc626e-2b06-4c67-9b2c-0231f0cef5b8",
−
     "template_id": "26cbc33b-179e-4d22-8b1b-23c9a305f196"
+
     "template_id": "8b7b9b89-f119-4b9b-b9b0-31598f819f1a"
−
}
+
}
−
 
+
</pre>
−
'''$ meteos experiment-create --json json/create_experiment.json'''
+
<pre>
−
 
+
$ meteos experiment-create --json sample/json/experiment.json
−
+-------------+--------------------------------------+
+
+--------------------+--------------------------------------+
−
| Property   | Value                                |
+
| Property           | Value                                |
−
+-------------+--------------------------------------+
+
+--------------------+--------------------------------------+
−
| created_at | 2016-10-20T02:11:51.778930           |
+
| created_at         | 2016-12-04T07:20:11.000000           |
−
| description | This is a sample experiment          |
+
| description       | This is a sample experiment          |
−
| id         | 0550a7db-b148-4319-bb35-16a9e4500d4a |
+
| id                 | 91504a65-01cf-428f-81aa-596be7ca8619 |
−
| name       | example-experiment                  |
+
| key_name          | key1                                |
−
| project_id | 5f011d076a6f4a328989c57ac7b4e501     |
+
| management_network | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 |
−
| status     | available                            |
+
| name               | example-experiment                  |
−
+-------------+--------------------------------------+
+
| project_id         | 67401cca74c2409b939e944bc6c8fcbe     |
 +
| status             | creating                            |
 +
| user_id            | 181b1caa9d5b470393ca66b9e511d5b0    |
 +
+--------------------+--------------------------------------+
 +
</pre>
  
 
Meteos creates a experiment using OpenStack Sahara spark plugin.
 
Meteos creates a experiment using OpenStack Sahara spark plugin.
−
User can see a sahara cluster and nova VMs created by Meteos.
 
  
−
'''$ sahara cluster-list'''
+
You can see a sahara cluster and nova VMs created by Meteos as below.
−
+------------------+--------------------------------------+--------+------------+
 
−
| name            | id                                  | status | node_count |
 
−
+------------------+--------------------------------------+--------+------------+
 
−
| cluster-0550a7db | 13b04f2e-5605-4b6b-b9fb-1ddf3dd85550 | Active | 3          |
 
−
+------------------+--------------------------------------+--------+------------+
 
  
−
'''$ nova list'''
+
<pre>
−
+--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+
+
$ openstack dataprocessing cluster list (or sahara cluster-list)
−
| ID                                  | Name                      | Status  | Task State | Power State | Networks                          |
+
+------------------+--------------------------------------+-------------+----------------+----------+
−
+--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+
+
| Name            | Id                                  | Plugin name | Plugin version | Status  |
−
| f526ec88-9000-4ddd-8f3b-ab60abbf7ca6 | cluster-0550a7db-master-0  | ACTIVE  | -         | Running    | private=10.0.0.140, 192.168.0.99  |
+
+------------------+--------------------------------------+-------------+----------------+----------+
−
| 6812aaa6-430d-4622-acd6-a558e1505e05 | cluster-0550a7db-workers-0 | ACTIVE  | -         | Running    | private=10.0.0.141, 192.168.0.92  |
+
| cluster-91504a65 | 13418fd9-5d2a-4ee6-b384-cb250b7e7714 | spark      | 1.6.0          | Spawning |
−
| 825727cf-f115-4d50-af26-99224ec94f8f | cluster-0550a7db-workers-1 | ACTIVE  | -          | Running    | private=10.0.0.139, 192.168.0.100 |
+
+------------------+--------------------------------------+-------------+----------------+----------+
−
+--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+
+
</pre>
 +
<pre>
 +
$ openstack server list (or nova list)
 +
+--------------------------------------+----------------------------+--------+----------+------------+
 +
| ID                                  | Name                      | Status | Networks | Image Name |
 +
+--------------------------------------+----------------------------+--------+----------+------------+
 +
| 58818eb5-ade7-407c-8c76-9fd9809632b4 | cluster-91504a65-workers-1 | BUILD  |          | meteos    |
 +
| a151dbd9-de51-43ca-afb8-1fdeecce2891 | cluster-91504a65-workers-0 | BUILD  |          | meteos    |
 +
| d02d85c5-0960-4b7e-880c-26b73c5dd8ad | cluster-91504a65-master-0  | BUILD  |          | meteos    |
 +
+--------------------------------------+----------------------------+--------+----------+------------+
 +
</pre>
  
 
=== 3. Upload a raw data ===
 
=== 3. Upload a raw data ===
  
−
Upload a raw data (sales data) to OpenStack Swift.
+
Upload a raw data (in this example past sales figures data) to OpenStack Swift.
−
Raw data shows "sales", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.
 
−
 
 
−
'''$ head meteos-test-data.txt'''
 
−
500000,1,10,2016,6,0,68,50
 
−
550000,2,10,2016,0,1,68,90
 
−
300000,3,10,2016,1,0,60,55
 
−
350000,4,10,2016,2,2,58,87
 
−
0,5,10,2016,3,3,58,60 # a horiday
 
−
400000,6,10,2016,4,3,60,60
 
−
330000,7,10,2016,5,2,62,87
 
−
550000,8,10,2016,6,1,66,92
 
−
600000,9,10,2016,0,1,55,93
 
−
330000,10,10,2016,1,0,57,55
 
−
 
 
−
'''$ swift upload meteos meteos-test-data.txt'''
 
−
meteos-test-data.txt
 
−
 
 
−
=== 4. Download a raw data to experiment ===
 
−
 
 
−
Download a raw data from swift to experiment.
 
−
Downloaded data is distributed in HDFS (Hadoop Distributed File System).
 
−
 
 
−
'''$ cat json/download_dataset.json'''
 
−
{
 
−
    "display_name": "sample-data",
 
−
    "display_description": "This is a sample dataset",
 
−
    "account": "demo:user01",
 
−
    "password": "0251c36e80584efd",
 
−
    "authurl": "http://192.168.0.4:5000/v2.0",
 
−
    "container_name": "meteos",
 
−
    "object_name": "meteos-test-data.txt",
 
−
    "experiment_id": "0550a7db-b148-4319-bb35-16a9e4500d4a"
 
−
}
 
−
 
 
−
'''$ meteos dataset-download --json json/download_dataset.json'''
 
−
 
 
−
+-------------+--------------------------------------+
 
−
| Property    | Value                                |
 
−
+-------------+--------------------------------------+
 
−
| created_at  | 2016-10-20T02:17:27.831814          |
 
−
| description | This is a sample dataset            |
 
−
| id          | bd134722-22cd-4247-8e20-538933e0975d |
 
−
| name        | sample-data                          |
 
−
| project_id  | 5f011d076a6f4a328989c57ac7b4e501    |
 
−
| status      | available                            |
 
−
+-------------+--------------------------------------+
 
−
 
 
−
=== 5. Parse a raw data ===
 
−
 
 
−
Parse a raw data to enable MLlib (Apache Spark's scalable machine learning library) to handle it.
 
−
Requirement format depends on machine learning algorithms.
 
−
In this example, convert to a list format using map method and remove unused fields using filter method.
 
−
 
 
−
[[File:Dataset-parse.png]]
 
−
 
 
−
'''$ cat json/parse_dataset.json'''
 
−
{
 
−
    "id": "bd134722-22cd-4247-8e20-538933e0975d",
 
−
    "method": "parse",
 
−
    "params": ["method": "map", "ars": "lambda l: l.split(',')",
 
−
              "method": "filter", "ars": "lambda l: l[0]! = '0'"],
 
−
    "experiment_id": "0550a7db-b148-4319-bb35-16a9e4500d4a"
 
−
}
 
−
 
 
−
'''$ meteos dataset-parse --json json/parse_dataset.json'''
 
−
 
 
−
=== 6. Create a prediction model ===
 
−
 
 
−
In this example, User creates a Linear Regression Model.
 
−
 
 
−
'''$ cat json/create_model.json'''
 
−
{
 
−
    "display_name": "sample-lr-model",
 
−
    "display_description": "This is a sample model",
 
−
    "dataset_id": "bd134722-22cd-4247-8e20-538933e0975d",
 
−
    "method": "LinearRegression",
 
−
    "args": "{'numIterations': 10, 'desired_output':0}",
 
−
    "experiment_id": "c08027ab-2ea5-4b57-840c-563908cfca46"
 
−
}
 
−
 
 
−
'''$ meteos model-create --json json/create_model.json'''
 
−
 
 
−
+-------------+--------------------------------------+
 
−
| Property    | Value                                |
 
−
+-------------+--------------------------------------+
 
−
| created_at  | 2016-10-20T02:17:27.831814          |
 
−
| description | This is a sample model              |
 
−
| id          | c48f5dba-45be-4165-9825-f4564fecebcd |
 
−
| name        | sample-lr-mode                      |
 
−
| project_id  | 5f011d076a6f4a328989c57ac7b4e501    |
 
−
| status      | available                            |
 
−
+-------------+--------------------------------------+
 
−
 
 
−
=== 7. Predict ===
 
−
 
 
−
Create a predict job.
 
−
 
 
−
'''$ cat json/predict_data.json'''
 
−
 
 
−
{
 
−
    "display_name": "predict job",
 
−
    "display_description": "This is a sample job",
 
−
    "model_id": "bd134722-22cd-4247-8e20-538933e0975d",
 
−
    "method": "predict",
 
−
    "args": "11,10,2016,2,0,57,58",
 
−
    "experiment_id": "c08027ab-2ea5-4b57-840c-563908cfca46"
 
−
}
 
−
 
 
−
'''$ meteos job-create --json json/predict_data.json'''
 
−
 
 
−
+-------------+--------------------------------------+
 
−
| Property    | Value                                |
 
−
+-------------+--------------------------------------+
 
−
| created_at  | 2016-10-20T02:17:37.644234          |
 
−
| description | This is a sample job                |
 
−
| id          | c38b9c22-d72c-4255-8236-7e319d351fad |
 
−
| name        | predict_job                          |
 
−
| project_id  | 5f011d076a6f4a328989c57ac7b4e501    |
 
−
| status      | executing                            |
 
−
+-------------+--------------------------------------+
 
−
 
 
−
Retrieve a predicted data as a stdout of job execution.
 
  
−
'''$ meteos job-show c38b9c22-d72c-4255-8236-7e319d351fad'''
+
You can use a sample data located in '''python-meteosclient/sample/data/linear_data.txt'''
  
−
+----------------------------+-------------------------------------------------+
+
Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.
−
| Property                  | Value                                          |
 
−
+----------------------------+-------------------------------------------------+
 
−
| stdout                    | Value : (325944.477851)                        |
 
−
| stderr                    |                                                |
 
−
+----------------------------+-------------------------------------------------+
 

Revision as of 00:39, 8 December 2016

Predict a Sales Figures using Meteos

In this example, you create a prediction model which predict sales by using Linear Regression.

Linear Regression is one of the algorithms in supervised learning.

Usecase.png

1. Create a experiment template

Create template of experiment. Experiment is a workspace of Machine Learning.

You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.

You can use a format located in python-meteosclient/sample/json/template.json 

$ glance image-list | grep meteos
| 45de4bbd-8419-40ff-8ed7-fc065c05e34f | meteos                          |

$ neutron net-list | grep public
| 84c13e76-ced9-4142-a885-280784f1f7a3 | public  | a14de1c5-b8d4-434b-a056-9b0049b93402             |

$ vim sample/json/template.json

$ cat sample/json/template.json
{
    "display_name": "example-template",
    "display_description": "This is a sample template of experiment",
    "image_id" : "45de4bbd-8419-40ff-8ed7-fc065c05e34f",
    "master_nodes_num": 1,
    "master_flavor_id": "4",
    "worker_nodes_num": 2,
    "worker_flavor_id": "2",
    "spark_version": "1.6.0",
    "floating_ip_pool": "84c13e76-ced9-4142-a885-280784f1f7a3"
}

$ meteos template-create --json sample/json/template.json
+---------------+-----------------------------------------+
| Property      | Value                                   |
+---------------+-----------------------------------------+
| cluster_id    | None                                    |
| created_at    | 2016-12-04T07:16:29.000000              |
| description   | This is a sample template of experiment |
| id            | 8b7b9b89-f119-4b9b-b9b0-31598f819f1a    |
| master_flavor | 4                                       |
| master_nodes  | 1                                       |
| name          | example-template                        |
| project_id    | 67401cca74c2409b939e944bc6c8fcbe        |
| spark_version | 1.6.0                                   |
| status        | available                               |
| user_id       | 181b1caa9d5b470393ca66b9e511d5b0        |
| worker_flavor | 2                                       |
| worker_nodes  | 2                                       |
+---------------+-----------------------------------------+


2. Create a experiment from template

Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.

You can use a format located in python-meteosclient/sample/json/experiment.json 

$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem

$ neutron net-list | grep private
| 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | private | cb58940f-859b-48c6-b92a-3861470f1fc1 20.0.0.0/26 |

$ vim sample/json/experiment.json

$ cat sample/json/experiment.json
{
    "display_name": "example-experiment",
    "display_description": "This is a sample experiment",
    "key_name": "key1",
    "neutron_management_network": "8abc626e-2b06-4c67-9b2c-0231f0cef5b8",
    "template_id": "8b7b9b89-f119-4b9b-b9b0-31598f819f1a"
}

$ meteos experiment-create --json sample/json/experiment.json
+--------------------+--------------------------------------+
| Property           | Value                                |
+--------------------+--------------------------------------+
| created_at         | 2016-12-04T07:20:11.000000           |
| description        | This is a sample experiment          |
| id                 | 91504a65-01cf-428f-81aa-596be7ca8619 |
| key_name           | key1                                 |
| management_network | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 |
| name               | example-experiment                   |
| project_id         | 67401cca74c2409b939e944bc6c8fcbe     |
| status             | creating                             |
| user_id            | 181b1caa9d5b470393ca66b9e511d5b0     |
+--------------------+--------------------------------------+

Meteos creates a experiment using OpenStack Sahara spark plugin.

You can see a sahara cluster and nova VMs created by Meteos as below. 

$ openstack dataprocessing cluster list (or sahara cluster-list)
+------------------+--------------------------------------+-------------+----------------+----------+
| Name             | Id                                   | Plugin name | Plugin version | Status   |
+------------------+--------------------------------------+-------------+----------------+----------+
| cluster-91504a65 | 13418fd9-5d2a-4ee6-b384-cb250b7e7714 | spark       | 1.6.0          | Spawning |
+------------------+--------------------------------------+-------------+----------------+----------+

$ openstack server list (or nova list)
+--------------------------------------+----------------------------+--------+----------+------------+
| ID                                   | Name                       | Status | Networks | Image Name |
+--------------------------------------+----------------------------+--------+----------+------------+
| 58818eb5-ade7-407c-8c76-9fd9809632b4 | cluster-91504a65-workers-1 | BUILD  |          | meteos     |
| a151dbd9-de51-43ca-afb8-1fdeecce2891 | cluster-91504a65-workers-0 | BUILD  |          | meteos     |
| d02d85c5-0960-4b7e-880c-26b73c5dd8ad | cluster-91504a65-master-0  | BUILD  |          | meteos     |
+--------------------------------------+----------------------------+--------+----------+------------+

3. Upload a raw data

Upload a raw data (in this example past sales figures data) to OpenStack Swift.

You can use a sample data located in python-meteosclient/sample/data/linear_data.txt

Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.

Retrieved from "https://wiki.openstack.org/w/index.php?title=Meteos/ExampleLinear&oldid=144629"