Jump to: navigation, search

Difference between revisions of "Meteos/ExampleLinear"

(→‎3. Upload a raw data)
(→‎6. Predict)
 
(17 intermediate revisions by the same user not shown)
Line 18: Line 18:
 
<pre>
 
<pre>
 
$ glance image-list | grep meteos
 
$ glance image-list | grep meteos
−
| 45de4bbd-8419-40ff-8ed7-fc065c05e34f | meteos                          |
+
| a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos                          |
 
</pre>
 
</pre>
 
<pre>
 
<pre>
 
$ neutron net-list | grep public
 
$ neutron net-list | grep public
−
| 84c13e76-ced9-4142-a885-280784f1f7a3 | public  | a14de1c5-b8d4-434b-a056-9b0049b93402             |
+
| 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public  | 1f979ae3-d6b7-4d03-ba0f-9d9112581783             |
 
</pre>
 
</pre>
 
<pre>
 
<pre>
Line 32: Line 32:
 
     "display_name": "example-template",
 
     "display_name": "example-template",
 
     "display_description": "This is a sample template of experiment",
 
     "display_description": "This is a sample template of experiment",
−
     "image_id" : "45de4bbd-8419-40ff-8ed7-fc065c05e34f",
+
     "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
 
     "master_nodes_num": 1,
 
     "master_nodes_num": 1,
 
     "master_flavor_id": "4",
 
     "master_flavor_id": "4",
Line 38: Line 38:
 
     "worker_flavor_id": "2",
 
     "worker_flavor_id": "2",
 
     "spark_version": "1.6.0",
 
     "spark_version": "1.6.0",
−
     "floating_ip_pool": "84c13e76-ced9-4142-a885-280784f1f7a3"
+
     "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
 
}
 
}
 
</pre>
 
</pre>
Line 47: Line 47:
 
+---------------+-----------------------------------------+
 
+---------------+-----------------------------------------+
 
| cluster_id    | None                                    |
 
| cluster_id    | None                                    |
−
| created_at    | 2016-12-04T07:16:29.000000              |
+
| created_at    | 2016-12-15T22:55:03.000000              |
 
| description  | This is a sample template of experiment |
 
| description  | This is a sample template of experiment |
−
| id            | 8b7b9b89-f119-4b9b-b9b0-31598f819f1a   |
+
| id            | d3180a78-74cc-432d-9e9d-68640b18abae   |
 
| master_flavor | 4                                      |
 
| master_flavor | 4                                      |
 
| master_nodes  | 1                                      |
 
| master_nodes  | 1                                      |
 
| name          | example-template                        |
 
| name          | example-template                        |
−
| project_id    | 67401cca74c2409b939e944bc6c8fcbe       |
+
| project_id    | 4034bb3cd9324776a68c140fdd18baa4       |
 
| spark_version | 1.6.0                                  |
 
| spark_version | 1.6.0                                  |
 
| status        | available                              |
 
| status        | available                              |
−
| user_id      | 181b1caa9d5b470393ca66b9e511d5b0       |
+
| user_id      | 64ad04e101df40b4b51e5f370a83412c       |
 
| worker_flavor | 2                                      |
 
| worker_flavor | 2                                      |
 
| worker_nodes  | 2                                      |
 
| worker_nodes  | 2                                      |
 
+---------------+-----------------------------------------+
 
+---------------+-----------------------------------------+
 
</pre>
 
</pre>
−
 
  
 
=== 2. Create a experiment from template ===
 
=== 2. Create a experiment from template ===
Line 75: Line 74:
 
<pre>
 
<pre>
 
$ neutron net-list | grep private
 
$ neutron net-list | grep private
−
| 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | private | cb58940f-859b-48c6-b92a-3861470f1fc1 20.0.0.0/26 |
+
| e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |
 
</pre>
 
</pre>
 
<pre>
 
<pre>
Line 86: Line 85:
 
     "display_description": "This is a sample experiment",
 
     "display_description": "This is a sample experiment",
 
     "key_name": "key1",
 
     "key_name": "key1",
−
     "neutron_management_network": "8abc626e-2b06-4c67-9b2c-0231f0cef5b8",
+
     "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9",
−
     "template_id": "8b7b9b89-f119-4b9b-b9b0-31598f819f1a"
+
     "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae"
 
}
 
}
 
</pre>
 
</pre>
Line 95: Line 94:
 
| Property          | Value                                |
 
| Property          | Value                                |
 
+--------------------+--------------------------------------+
 
+--------------------+--------------------------------------+
−
| created_at        | 2016-12-04T07:20:11.000000          |
+
| created_at        | 2016-12-15T22:56:51.000000          |
 
| description        | This is a sample experiment          |
 
| description        | This is a sample experiment          |
−
| id                | 91504a65-01cf-428f-81aa-596be7ca8619 |
+
| id                | fcc8d055-e801-4652-af8c-5aabedbf0286 |
 
| key_name          | key1                                |
 
| key_name          | key1                                |
−
| management_network | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 |
+
| management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 |
 
| name              | example-experiment                  |
 
| name              | example-experiment                  |
−
| project_id        | 67401cca74c2409b939e944bc6c8fcbe     |
+
| project_id        | 4034bb3cd9324776a68c140fdd18baa4     |
 
| status            | creating                            |
 
| status            | creating                            |
−
| user_id            | 181b1caa9d5b470393ca66b9e511d5b0     |
+
| user_id            | 64ad04e101df40b4b51e5f370a83412c     |
 
+--------------------+--------------------------------------+
 
+--------------------+--------------------------------------+
 
</pre>
 
</pre>
Line 116: Line 115:
 
| Name            | Id                                  | Plugin name | Plugin version | Status  |
 
| Name            | Id                                  | Plugin name | Plugin version | Status  |
 
+------------------+--------------------------------------+-------------+----------------+----------+
 
+------------------+--------------------------------------+-------------+----------------+----------+
−
| cluster-91504a65 | 13418fd9-5d2a-4ee6-b384-cb250b7e7714 | spark      | 1.6.0          | Spawning |
+
| cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark      | 1.6.0          | Spawning |
 
+------------------+--------------------------------------+-------------+----------------+----------+
 
+------------------+--------------------------------------+-------------+----------------+----------+
 
</pre>
 
</pre>
 
<pre>
 
<pre>
 
$ openstack server list (or nova list)
 
$ openstack server list (or nova list)
−
+--------------------------------------+----------------------------+--------+----------+------------+
+
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
−
| ID                                  | Name                      | Status | Networks | Image Name |
+
| ID                                  | Name                      | Status | Task State | Power State | Networks         |
−
+--------------------------------------+----------------------------+--------+----------+------------+
+
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
−
| 58818eb5-ade7-407c-8c76-9fd9809632b4 | cluster-91504a65-workers-1 | BUILD  |          | meteos     |
+
| 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0  | ACTIVE | -         | Running     | private=10.0.0.5 |
−
| a151dbd9-de51-43ca-afb8-1fdeecce2891 | cluster-91504a65-workers-0 | BUILD  |          | meteos     |
+
| 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | -         | Running     | private=10.0.0.7 |
−
| d02d85c5-0960-4b7e-880c-26b73c5dd8ad | cluster-91504a65-master-0  | BUILD  |          | meteos     |
+
| a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | -         | Running     | private=10.0.0.4 |
−
+--------------------------------------+----------------------------+--------+----------+------------+
+
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
 
</pre>
 
</pre>
  
Line 143: Line 142:
 
<pre>
 
<pre>
 
/sample/data$ head linear_data.txt
 
/sample/data$ head linear_data.txt
−
15000,1,10,2016,6,0,68,50
+
4500,1,1,2016,5,0,40,50
−
14000,2,10,2016,0,0,68,50
+
8000,2,1,2016,6,1,60,80
−
5500,3,10,2016,1,1,68,90
+
9500,3,1,2016,0,2,88,92
−
0,4,10,2016,2,1,68,90
+
5000,4,1,2016,1,3,90,90
−
4500,5,10,2016,3,0,60,55
+
0,5,1,2016,2,2,90,80
−
5000,6,10,2016,4,0,60,55
+
4500,6,1,2016,3,3,80,90
−
4800,7,10,2016,5,2,58,87
+
4000,7,1,2016,4,1,60,80
−
16000,8,10,2016,6,3,58,60
+
4500,8,1,2016,5,0,40,50
−
15000,9,10,2016,0,3,60,60
+
8000,9,1,2016,6,0,30,50
−
3300,10,10,2016,1,2,62,87
+
9500,10,1,2016,0,0,40,50
 
</pre>
 
</pre>
 
<pre>
 
<pre>
Line 161: Line 160:
 
=== 4. Parse a raw data ===
 
=== 4. Parse a raw data ===
  
−
Parse a raw data to enable Prediction Model to handle it.
+
Parse a raw data to eliminate exception data.
 +
 
 +
In this case, you have to eliminate holiday sales figures because exception data affect the accuracy of prediction model.
  
 
You can use a sample format located in '''python-meteosclient/sample/json/dataset_parse.json'''
 
You can use a sample format located in '''python-meteosclient/sample/json/dataset_parse.json'''
Line 171: Line 172:
 
</pre>
 
</pre>
 
<pre>
 
<pre>
−
$ cat ../python-meteosclient/sample/json/dataset_parse.json
+
$ cat sample/json/dataset_parse.json
 
{
 
{
 
     "source_dataset_url": "swift://meteos/linear_data.txt",
 
     "source_dataset_url": "swift://meteos/linear_data.txt",
 
     "display_name": "sample-data",
 
     "display_name": "sample-data",
−
     "display_description": "user skill dataset",
+
     "display_description": "This is a sample dataset",
 
     "method": "parse",
 
     "method": "parse",
 
     "params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}],
 
     "params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}],
−
     "experiment_id": "91504a65-01cf-428f-81aa-596be7ca8619",
+
     "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286",
 
     "swift_tenant": "demo",
 
     "swift_tenant": "demo",
 
     "swift_username": "demo",
 
     "swift_username": "demo",
Line 189: Line 190:
 
| Property    | Value                                |
 
| Property    | Value                                |
 
+-------------+--------------------------------------+
 
+-------------+--------------------------------------+
−
| created_at  | 2016-12-08T00:45:43.000000          |
+
| created_at  | 2016-12-15T23:04:11.000000          |
 
| description | This is a sample dataset            |
 
| description | This is a sample dataset            |
 
| head        | None                                |
 
| head        | None                                |
−
| id          | b6054cab-77b8-4b2d-b6c8-861f69859107 |
+
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
 
| name        | sample-data                          |
 
| name        | sample-data                          |
−
| project_id  | 0e3e0f01952948848d8ae438279122fe     |
+
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
 
| status      | creating                            |
 
| status      | creating                            |
 
| stderr      | None                                |
 
| stderr      | None                                |
−
| user_id    | 2c3120e8228c4d0e9768f09346fe842d     |
+
| user_id    | 64ad04e101df40b4b51e5f370a83412c     |
 
+-------------+--------------------------------------+
 
+-------------+--------------------------------------+
 
</pre>
 
</pre>
 
<pre>
 
<pre>
 
$ meteos dataset-list
 
$ meteos dataset-list
−
+--------------------------------------+-------------+--------------------------+-----------+--------------------------------+----------------------------+
+
+--------------------------------------+-------------+-----------+--------------------------------+
−
| id                                  | name        | description              | status    | source_dataset_url            | created_at                |
+
| id                                  | name        | status    | source_dataset_url            |
−
+--------------------------------------+-------------+--------------------------+-----------+--------------------------------+----------------------------+
+
+--------------------------------------+-------------+-----------+--------------------------------+
−
| b6054cab-77b8-4b2d-b6c8-861f69859107 | sample-data | This is a sample dataset | available | swift://meteos/linear_data.txt | 2016-12-08T00:45:43.000000 |
+
| 91d98f6d-a065-431b-b8e0-1b996ac85cec | sample-data | available | swift://meteos/linear_data.txt |
−
+--------------------------------------+-------------+--------------------------+-----------+--------------------------------+----------------------------+
+
+--------------------------------------+-------------+-----------+--------------------------------+
 +
</pre>
 +
You can see that holiday sales figures has been eliminated.
 +
<pre>
 +
$ meteos dataset-show 91d98f6d-a065-431b-b8e0-1b996ac85cec
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| created_at  | 2016-12-15T23:04:11.000000          |
 +
| description | This is a sample dataset            |
 +
| head        | [u'4500,1,1,2016,5,0,40,50',        |
 +
|            | u'8000,2,1,2016,6,1,60,80',          |
 +
|            | u'9500,3,1,2016,0,2,88,92',          |
 +
|            | u'5000,4,1,2016,1,3,90,90',          |
 +
|            | u'4500,6,1,2016,3,3,80,90',          |
 +
|            | u'4000,7,1,2016,4,1,60,80',          |
 +
|            | u'4500,8,1,2016,5,0,40,50',          |
 +
|            | u'8000,9,1,2016,6,0,30,50',          |
 +
|            | u'9500,10,1,2016,0,0,40,50',        |
 +
|            | u'5000,11,1,2016,1,1,60,80']        |
 +
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
| name        | sample-data                         |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | available                           |
 +
| stderr      |                                      |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
 
 +
=== 5. Create a prediction model ===
 +
 
 +
In this example, User creates a Linear Regression Model from parsed dataset.
 +
 
 +
Parsed dataset has been already distributed in hdfs of experiment environment.
 +
 
 +
So, you spefity the internal url (internal://<dataset-id>) in source_dataset_url parameter.
 +
 
 +
<pre>
 +
$ vim sample/json/model_linear.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/model_linear.json
 +
{
 +
    "display_name": "linear-model",
 +
    "display_description": "Sample LinearRegression Model",
 +
    "source_dataset_url": "internal://91d98f6d-a065-431b-b8e0-1b996ac85cec",
 +
    "model_type": "LinearRegression",
 +
    "model_params": "{'numIterations': 200}",
 +
    "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286"
 +
}
 +
</pre>
 +
<pre>
 +
$ meteos model-create --json sample/json/model_linear.json
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| created_at  | 2016-12-15T23:09:36.000000          |
 +
| description | Sample LinearRegression Model        |
 +
| id          | 3cf02c2f-f043-49e5-a0df-0dc782868312 |
 +
| name        | linear-model                        |
 +
| params      | eydudW1JdGVyYXRpb25zJzogMjAwfQ==    |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | creating                            |
 +
| stderr      | None                                |
 +
| stdout      | None                                |
 +
| type        | LinearRegression                    |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 
</pre>
 
</pre>
 
<pre>
 
<pre>
−
$ meteos dataset-show b6054cab-77b8-4b2d-b6c8-861f69859107
+
$ meteos model-list
−
+-------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
−
| Property    | Value                                                                                                                                                                                                                                                                                                    |
+
| id                                  | name        | status    | type            | source_dataset_url                              |
−
+-------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
−
| created_at  | 2016-12-08T00:45:43.000000                                                                                                                                                                                                                                                                              |
+
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | available | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
−
| description | This is a sample dataset                                                                                                                                                                                                                                                                                |
+
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
−
| head        | [u'15000,1,10,2016,6,0,68,50', u'14000,2,10,2016,0,0,68,50', u'5500,3,10,2016,1,1,68,90', u'4500,5,10,2016,3,0,60,55', u'5000,6,10,2016,4,0,60,55', u'4800,7,10,2016,5,2,58,87', u'16000,8,10,2016,6,3,58,60', u'15000,9,10,2016,0,3,60,60', u'3300,10,10,2016,1,2,62,87', u'6000,12,10,2016,3,1,55,93'] |
+
</pre>
−
|            |                                                                                                                                                                                                                                                                                                          |
+
 
−
| id          | b6054cab-77b8-4b2d-b6c8-861f69859107                                                                                                                                                                                                                                                                    |
+
=== 6. Predict ===
−
| name        | sample-data                                                                                                                                                                                                                                                                                              |
+
 
−
| project_id  | 0e3e0f01952948848d8ae438279122fe                                                                                                                                                                                                                                                                        |
+
Create a learning job predicting a Sales Figures.
−
| status      | available                                                                                                                                                                                                                                                                                                |
+
 
−
| stderr      |                                                                                                                                                                                                                                                                                                          |
+
Specify the input value as "args" parameter.
−
| user_id    | 2c3120e8228c4d0e9768f09346fe842d                                                                                                                                                                                                                                                                        |
+
 
−
+-------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
Retrieve a predicted data as a stdout of job execution.
 +
 
 +
<pre>
 +
$ vim sample/json/learning.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/learning.json
 +
{
 +
    "display_name": "example-learning-job",
 +
    "display_description": "This is a sample job",
 +
    "model_id": "3cf02c2f-f043-49e5-a0df-0dc782868312",
 +
    "method": "predict",
 +
    "args": "1,7,2016,5,0,60,80"
 +
}
 +
</pre>
 +
<pre>
 +
$ meteos learning-create --json sample/json/learning.json
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| args        | MSw3LDIwMTYsNSwwLDYwLDgw            |
 +
| created_at  | 2016-12-15T23:13:29.000000          |
 +
| description | This is a sample job                |
 +
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
 +
| method      | predict                              |
 +
| name        | example-learning-job                |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | creating                            |
 +
| stderr      | None                                |
 +
| stdout      | None                                |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
<pre>
 +
$ meteos learning-list
 +
+--------------------------------------+----------------------+-----------+--------------------+---------------+
 +
| id                                  | name                | status    | args              | stdout        |
 +
+--------------------------------------+----------------------+-----------+--------------------+---------------+
 +
| ffcd3ae6-8b53-437e-94e2-3aabde741cb0 | example-learning-job | available | 1,7,2016,5,0,60,80 | 4048.18352502 |
 +
+--------------------------------------+----------------------+-----------+--------------------+---------------+
 +
</pre>
 +
 
 +
=== 7. Online Prediction ===
 +
 
 +
You can load a Prediction Model in advance for online prediction by using "meteos-load" command.
 +
 
 +
In online prediction, user can retrieve a predicted data immediately.
 +
 
 +
''' Note: Meteos can not load multiple models at the same time. So, you have to unload a model before loading another model. '''
 +
 
 +
<pre>
 +
$ meteos model-load 3cf02c2f-f043-49e5-a0df-0dc782868312
 +
</pre>
 +
<pre>
 +
$ meteos model-list
 +
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
 +
| id                                  | name        | status    | type            | source_dataset_url                              |
 +
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
 +
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | activating | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
 +
</pre>
 +
 
 +
Status is set to "active" when loading is complete.
 +
 
 +
<pre>
 +
$ meteos model-list
 +
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
 +
| id                                  | name        | status | type            | source_dataset_url                              |
 +
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
 +
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | active | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
 +
</pre>
 +
 
 +
You can get a predicted data as a response of REST API .
 +
 
 +
<pre>
 +
$ meteos learning-create --json sample/json/learning.json
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| args        | MSw3LDIwMTYsNSwwLDYwLDgw            |
 +
| created_at  | 2016-12-15T23:13:29.000000          |
 +
| description | This is a sample job                |
 +
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
 +
| method      | predict                              |
 +
| name        | example-learning-job                |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | available                            |
 +
| stderr      | None                                |
 +
| stdout      | 4048.18352502                        |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
 
 +
<pre>
 +
$ meteos model-unload 3cf02c2f-f043-49e5-a0df-0dc782868312
 
</pre>
 
</pre>

Latest revision as of 06:45, 10 January 2017

Predict a Sales Figures using Meteos

In this example, you create a prediction model which predict sales by using Linear Regression.

Linear Regression is one of the algorithms in supervised learning.

Usecase.png

1. Create a experiment template

Create template of experiment. Experiment is a workspace of Machine Learning.

You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.

You can use a format located in python-meteosclient/sample/json/template.json 

$ glance image-list | grep meteos
| a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos                          |

$ neutron net-list | grep public
| 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public  | 1f979ae3-d6b7-4d03-ba0f-9d9112581783             |

$ vim sample/json/template.json

$ cat sample/json/template.json
{
    "display_name": "example-template",
    "display_description": "This is a sample template of experiment",
    "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
    "master_nodes_num": 1,
    "master_flavor_id": "4",
    "worker_nodes_num": 2,
    "worker_flavor_id": "2",
    "spark_version": "1.6.0",
    "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
}

$ meteos template-create --json sample/json/template.json
+---------------+-----------------------------------------+
| Property      | Value                                   |
+---------------+-----------------------------------------+
| cluster_id    | None                                    |
| created_at    | 2016-12-15T22:55:03.000000              |
| description   | This is a sample template of experiment |
| id            | d3180a78-74cc-432d-9e9d-68640b18abae    |
| master_flavor | 4                                       |
| master_nodes  | 1                                       |
| name          | example-template                        |
| project_id    | 4034bb3cd9324776a68c140fdd18baa4        |
| spark_version | 1.6.0                                   |
| status        | available                               |
| user_id       | 64ad04e101df40b4b51e5f370a83412c        |
| worker_flavor | 2                                       |
| worker_nodes  | 2                                       |
+---------------+-----------------------------------------+

2. Create a experiment from template

Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.

You can use a format located in python-meteosclient/sample/json/experiment.json 

$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem

$ neutron net-list | grep private
| e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |

$ vim sample/json/experiment.json

$ cat sample/json/experiment.json
{
    "display_name": "example-experiment",
    "display_description": "This is a sample experiment",
    "key_name": "key1",
    "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9",
    "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae"
}

$ meteos experiment-create --json sample/json/experiment.json
+--------------------+--------------------------------------+
| Property           | Value                                |
+--------------------+--------------------------------------+
| created_at         | 2016-12-15T22:56:51.000000           |
| description        | This is a sample experiment          |
| id                 | fcc8d055-e801-4652-af8c-5aabedbf0286 |
| key_name           | key1                                 |
| management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 |
| name               | example-experiment                   |
| project_id         | 4034bb3cd9324776a68c140fdd18baa4     |
| status             | creating                             |
| user_id            | 64ad04e101df40b4b51e5f370a83412c     |
+--------------------+--------------------------------------+

Meteos creates a experiment using OpenStack Sahara spark plugin.

You can see a sahara cluster and nova VMs created by Meteos as below. 

$ openstack dataprocessing cluster list (or sahara cluster-list)
+------------------+--------------------------------------+-------------+----------------+----------+
| Name             | Id                                   | Plugin name | Plugin version | Status   |
+------------------+--------------------------------------+-------------+----------------+----------+
| cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark       | 1.6.0          | Spawning |
+------------------+--------------------------------------+-------------+----------------+----------+

$ openstack server list (or nova list)
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
| ID                                   | Name                       | Status | Task State | Power State | Networks         |
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
| 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0  | ACTIVE | -          | Running     | private=10.0.0.5 |
| 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | -          | Running     | private=10.0.0.7 |
| a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | -          | Running     | private=10.0.0.4 |
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+

3. Upload a raw data

Upload a raw data (in this example past sales figures data) to OpenStack Swift.

You can use a sample data located in python-meteosclient/sample/data/linear_data.txt

Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left. 

$ cd sample/data/

/sample/data$ head linear_data.txt
4500,1,1,2016,5,0,40,50
8000,2,1,2016,6,1,60,80
9500,3,1,2016,0,2,88,92
5000,4,1,2016,1,3,90,90
0,5,1,2016,2,2,90,80
4500,6,1,2016,3,3,80,90
4000,7,1,2016,4,1,60,80
4500,8,1,2016,5,0,40,50
8000,9,1,2016,6,0,30,50
9500,10,1,2016,0,0,40,50

/sample/data$ swift upload meteos linear_data.txt
linear_data.txt

4. Parse a raw data

Parse a raw data to eliminate exception data.

In this case, you have to eliminate holiday sales figures because exception data affect the accuracy of prediction model.

You can use a sample format located in python-meteosclient/sample/json/dataset_parse.json

You can see the head data of parsed dataset by executing "meteos dataset-show <dataset-uuid>" command. 

$ vim ../python-meteosclient/sample/json/dataset_parse.json

$ cat sample/json/dataset_parse.json
{
    "source_dataset_url": "swift://meteos/linear_data.txt",
    "display_name": "sample-data",
    "display_description": "This is a sample dataset",
    "method": "parse",
    "params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}],
    "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286",
    "swift_tenant": "demo",
    "swift_username": "demo",
    "swift_password": "nova"
}

$ meteos dataset-create --json sample/json/dataset_parse.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-15T23:04:11.000000           |
| description | This is a sample dataset             |
| head        | None                                 |
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
| name        | sample-data                          |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | creating                             |
| stderr      | None                                 |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos dataset-list
+--------------------------------------+-------------+-----------+--------------------------------+
| id                                   | name        | status    | source_dataset_url             |
+--------------------------------------+-------------+-----------+--------------------------------+
| 91d98f6d-a065-431b-b8e0-1b996ac85cec | sample-data | available | swift://meteos/linear_data.txt |
+--------------------------------------+-------------+-----------+--------------------------------+

You can see that holiday sales figures has been eliminated. 

$ meteos dataset-show 91d98f6d-a065-431b-b8e0-1b996ac85cec
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-15T23:04:11.000000           |
| description | This is a sample dataset             |
| head        | [u'4500,1,1,2016,5,0,40,50',         |
|             | u'8000,2,1,2016,6,1,60,80',          |
|             | u'9500,3,1,2016,0,2,88,92',          |
|             | u'5000,4,1,2016,1,3,90,90',          |
|             | u'4500,6,1,2016,3,3,80,90',          |
|             | u'4000,7,1,2016,4,1,60,80',          |
|             | u'4500,8,1,2016,5,0,40,50',          |
|             | u'8000,9,1,2016,6,0,30,50',          |
|             | u'9500,10,1,2016,0,0,40,50',         |
|             | u'5000,11,1,2016,1,1,60,80']         |
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
| name        | sample-data                          |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | available                            |
| stderr      |                                      |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

5. Create a prediction model

In this example, User creates a Linear Regression Model from parsed dataset.

Parsed dataset has been already distributed in hdfs of experiment environment.

So, you spefity the internal url (internal://<dataset-id>) in source_dataset_url parameter. 

$ vim sample/json/model_linear.json

$ cat sample/json/model_linear.json
{
    "display_name": "linear-model",
    "display_description": "Sample LinearRegression Model",
    "source_dataset_url": "internal://91d98f6d-a065-431b-b8e0-1b996ac85cec",
    "model_type": "LinearRegression",
    "model_params": "{'numIterations': 200}",
    "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286"
}

$ meteos model-create --json sample/json/model_linear.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-15T23:09:36.000000           |
| description | Sample LinearRegression Model        |
| id          | 3cf02c2f-f043-49e5-a0df-0dc782868312 |
| name        | linear-model                         |
| params      | eydudW1JdGVyYXRpb25zJzogMjAwfQ==     |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | creating                             |
| stderr      | None                                 |
| stdout      | None                                 |
| type        | LinearRegression                     |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos model-list
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
| id                                   | name         | status    | type             | source_dataset_url                              |
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | available | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+

6. Predict

Create a learning job predicting a Sales Figures.

Specify the input value as "args" parameter.

Retrieve a predicted data as a stdout of job execution. 

$ vim sample/json/learning.json

$ cat sample/json/learning.json
{
    "display_name": "example-learning-job",
    "display_description": "This is a sample job",
    "model_id": "3cf02c2f-f043-49e5-a0df-0dc782868312",
    "method": "predict",
    "args": "1,7,2016,5,0,60,80"
}

$ meteos learning-create --json sample/json/learning.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| args        | MSw3LDIwMTYsNSwwLDYwLDgw             |
| created_at  | 2016-12-15T23:13:29.000000           |
| description | This is a sample job                 |
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
| method      | predict                              |
| name        | example-learning-job                 |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | creating                             |
| stderr      | None                                 |
| stdout      | None                                 |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos learning-list
+--------------------------------------+----------------------+-----------+--------------------+---------------+
| id                                   | name                 | status    | args               | stdout        |
+--------------------------------------+----------------------+-----------+--------------------+---------------+
| ffcd3ae6-8b53-437e-94e2-3aabde741cb0 | example-learning-job | available | 1,7,2016,5,0,60,80 | 4048.18352502 |
+--------------------------------------+----------------------+-----------+--------------------+---------------+

7. Online Prediction

You can load a Prediction Model in advance for online prediction by using "meteos-load" command.

In online prediction, user can retrieve a predicted data immediately.

Note: Meteos can not load multiple models at the same time. So, you have to unload a model before loading another model. 

$ meteos model-load 3cf02c2f-f043-49e5-a0df-0dc782868312

$ meteos model-list
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
| id                                   | name         | status     | type             | source_dataset_url                              |
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | activating | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+

Status is set to "active" when loading is complete. 

$ meteos model-list
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
| id                                   | name         | status | type             | source_dataset_url                              |
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | active | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+

You can get a predicted data as a response of REST API . 

$ meteos learning-create --json sample/json/learning.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| args        | MSw3LDIwMTYsNSwwLDYwLDgw             |
| created_at  | 2016-12-15T23:13:29.000000           |
| description | This is a sample job                 |
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
| method      | predict                              |
| name        | example-learning-job                 |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | available                            |
| stderr      | None                                 |
| stdout      | 4048.18352502                        |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos model-unload 3cf02c2f-f043-49e5-a0df-0dc782868312
Retrieved from "https://wiki.openstack.org/w/index.php?title=Meteos/ExampleLinear&oldid=149699"