Jump to: navigation, search

Difference between revisions of "Meteos/ExampleLinear"

(→‎Predict sales using Meteos)
(→‎6. Predict)
 
(18 intermediate revisions by the same user not shown)
Line 18: Line 18:
 
<pre>
 
<pre>
 
$ glance image-list | grep meteos
 
$ glance image-list | grep meteos
−
| 45de4bbd-8419-40ff-8ed7-fc065c05e34f | meteos                          |
+
| a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos                          |
 
</pre>
 
</pre>
 
<pre>
 
<pre>
 
$ neutron net-list | grep public
 
$ neutron net-list | grep public
−
| 84c13e76-ced9-4142-a885-280784f1f7a3 | public  | a14de1c5-b8d4-434b-a056-9b0049b93402             |
+
| 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public  | 1f979ae3-d6b7-4d03-ba0f-9d9112581783             |
 
</pre>
 
</pre>
 
<pre>
 
<pre>
Line 32: Line 32:
 
     "display_name": "example-template",
 
     "display_name": "example-template",
 
     "display_description": "This is a sample template of experiment",
 
     "display_description": "This is a sample template of experiment",
−
     "image_id" : "45de4bbd-8419-40ff-8ed7-fc065c05e34f",
+
     "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
 
     "master_nodes_num": 1,
 
     "master_nodes_num": 1,
 
     "master_flavor_id": "4",
 
     "master_flavor_id": "4",
Line 38: Line 38:
 
     "worker_flavor_id": "2",
 
     "worker_flavor_id": "2",
 
     "spark_version": "1.6.0",
 
     "spark_version": "1.6.0",
−
     "floating_ip_pool": "84c13e76-ced9-4142-a885-280784f1f7a3"
+
     "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
 
}
 
}
 
</pre>
 
</pre>
Line 47: Line 47:
 
+---------------+-----------------------------------------+
 
+---------------+-----------------------------------------+
 
| cluster_id    | None                                    |
 
| cluster_id    | None                                    |
−
| created_at    | 2016-12-04T07:16:29.000000              |
+
| created_at    | 2016-12-15T22:55:03.000000              |
 
| description  | This is a sample template of experiment |
 
| description  | This is a sample template of experiment |
−
| id            | 8b7b9b89-f119-4b9b-b9b0-31598f819f1a   |
+
| id            | d3180a78-74cc-432d-9e9d-68640b18abae   |
 
| master_flavor | 4                                      |
 
| master_flavor | 4                                      |
 
| master_nodes  | 1                                      |
 
| master_nodes  | 1                                      |
 
| name          | example-template                        |
 
| name          | example-template                        |
−
| project_id    | 67401cca74c2409b939e944bc6c8fcbe       |
+
| project_id    | 4034bb3cd9324776a68c140fdd18baa4       |
 
| spark_version | 1.6.0                                  |
 
| spark_version | 1.6.0                                  |
 
| status        | available                              |
 
| status        | available                              |
−
| user_id      | 181b1caa9d5b470393ca66b9e511d5b0       |
+
| user_id      | 64ad04e101df40b4b51e5f370a83412c       |
 
| worker_flavor | 2                                      |
 
| worker_flavor | 2                                      |
 
| worker_nodes  | 2                                      |
 
| worker_nodes  | 2                                      |
 
+---------------+-----------------------------------------+
 
+---------------+-----------------------------------------+
 
</pre>
 
</pre>
−
 
  
 
=== 2. Create a experiment from template ===
 
=== 2. Create a experiment from template ===
Line 75: Line 74:
 
<pre>
 
<pre>
 
$ neutron net-list | grep private
 
$ neutron net-list | grep private
−
| 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | private | cb58940f-859b-48c6-b92a-3861470f1fc1 20.0.0.0/26 |
+
| e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |
 
</pre>
 
</pre>
 
<pre>
 
<pre>
Line 86: Line 85:
 
     "display_description": "This is a sample experiment",
 
     "display_description": "This is a sample experiment",
 
     "key_name": "key1",
 
     "key_name": "key1",
−
     "neutron_management_network": "8abc626e-2b06-4c67-9b2c-0231f0cef5b8",
+
     "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9",
−
     "template_id": "8b7b9b89-f119-4b9b-b9b0-31598f819f1a"
+
     "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae"
 
}
 
}
 
</pre>
 
</pre>
Line 95: Line 94:
 
| Property          | Value                                |
 
| Property          | Value                                |
 
+--------------------+--------------------------------------+
 
+--------------------+--------------------------------------+
−
| created_at        | 2016-12-04T07:20:11.000000          |
+
| created_at        | 2016-12-15T22:56:51.000000          |
 
| description        | This is a sample experiment          |
 
| description        | This is a sample experiment          |
−
| id                | 91504a65-01cf-428f-81aa-596be7ca8619 |
+
| id                | fcc8d055-e801-4652-af8c-5aabedbf0286 |
 
| key_name          | key1                                |
 
| key_name          | key1                                |
−
| management_network | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 |
+
| management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 |
 
| name              | example-experiment                  |
 
| name              | example-experiment                  |
−
| project_id        | 67401cca74c2409b939e944bc6c8fcbe     |
+
| project_id        | 4034bb3cd9324776a68c140fdd18baa4     |
 
| status            | creating                            |
 
| status            | creating                            |
−
| user_id            | 181b1caa9d5b470393ca66b9e511d5b0     |
+
| user_id            | 64ad04e101df40b4b51e5f370a83412c     |
 
+--------------------+--------------------------------------+
 
+--------------------+--------------------------------------+
 
</pre>
 
</pre>
Line 116: Line 115:
 
| Name            | Id                                  | Plugin name | Plugin version | Status  |
 
| Name            | Id                                  | Plugin name | Plugin version | Status  |
 
+------------------+--------------------------------------+-------------+----------------+----------+
 
+------------------+--------------------------------------+-------------+----------------+----------+
−
| cluster-91504a65 | 13418fd9-5d2a-4ee6-b384-cb250b7e7714 | spark      | 1.6.0          | Spawning |
+
| cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark      | 1.6.0          | Spawning |
 
+------------------+--------------------------------------+-------------+----------------+----------+
 
+------------------+--------------------------------------+-------------+----------------+----------+
 
</pre>
 
</pre>
 
<pre>
 
<pre>
 
$ openstack server list (or nova list)
 
$ openstack server list (or nova list)
−
+--------------------------------------+----------------------------+--------+----------+------------+
+
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
−
| ID                                  | Name                      | Status | Networks | Image Name |
+
| ID                                  | Name                      | Status | Task State | Power State | Networks         |
−
+--------------------------------------+----------------------------+--------+----------+------------+
+
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
−
| 58818eb5-ade7-407c-8c76-9fd9809632b4 | cluster-91504a65-workers-1 | BUILD  |          | meteos     |
+
| 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0  | ACTIVE | -         | Running     | private=10.0.0.5 |
−
| a151dbd9-de51-43ca-afb8-1fdeecce2891 | cluster-91504a65-workers-0 | BUILD  |          | meteos     |
+
| 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | -         | Running     | private=10.0.0.7 |
−
| d02d85c5-0960-4b7e-880c-26b73c5dd8ad | cluster-91504a65-master-0  | BUILD  |          | meteos     |
+
| a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | -         | Running     | private=10.0.0.4 |
−
+--------------------------------------+----------------------------+--------+----------+------------+
+
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
 
</pre>
 
</pre>
  
Line 137: Line 136:
  
 
Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.
 
Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.
 +
 +
<pre>
 +
$ cd sample/data/
 +
</pre>
 +
<pre>
 +
/sample/data$ head linear_data.txt
 +
4500,1,1,2016,5,0,40,50
 +
8000,2,1,2016,6,1,60,80
 +
9500,3,1,2016,0,2,88,92
 +
5000,4,1,2016,1,3,90,90
 +
0,5,1,2016,2,2,90,80
 +
4500,6,1,2016,3,3,80,90
 +
4000,7,1,2016,4,1,60,80
 +
4500,8,1,2016,5,0,40,50
 +
8000,9,1,2016,6,0,30,50
 +
9500,10,1,2016,0,0,40,50
 +
</pre>
 +
<pre>
 +
/sample/data$ swift upload meteos linear_data.txt
 +
linear_data.txt
 +
</pre>
 +
 +
=== 4. Parse a raw data ===
 +
 +
Parse a raw data to eliminate exception data.
 +
 +
In this case, you have to eliminate holiday sales figures because exception data affect the accuracy of prediction model.
 +
 +
You can use a sample format located in '''python-meteosclient/sample/json/dataset_parse.json'''
 +
 +
You can see the head data of parsed dataset by executing "meteos dataset-show <dataset-uuid>" command.
 +
 +
<pre>
 +
$ vim ../python-meteosclient/sample/json/dataset_parse.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/dataset_parse.json
 +
{
 +
    "source_dataset_url": "swift://meteos/linear_data.txt",
 +
    "display_name": "sample-data",
 +
    "display_description": "This is a sample dataset",
 +
    "method": "parse",
 +
    "params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}],
 +
    "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286",
 +
    "swift_tenant": "demo",
 +
    "swift_username": "demo",
 +
    "swift_password": "nova"
 +
}
 +
</pre>
 +
<pre>
 +
$ meteos dataset-create --json sample/json/dataset_parse.json
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| created_at  | 2016-12-15T23:04:11.000000          |
 +
| description | This is a sample dataset            |
 +
| head        | None                                |
 +
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
| name        | sample-data                          |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | creating                            |
 +
| stderr      | None                                |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
<pre>
 +
$ meteos dataset-list
 +
+--------------------------------------+-------------+-----------+--------------------------------+
 +
| id                                  | name        | status    | source_dataset_url            |
 +
+--------------------------------------+-------------+-----------+--------------------------------+
 +
| 91d98f6d-a065-431b-b8e0-1b996ac85cec | sample-data | available | swift://meteos/linear_data.txt |
 +
+--------------------------------------+-------------+-----------+--------------------------------+
 +
</pre>
 +
You can see that holiday sales figures has been eliminated.
 +
<pre>
 +
$ meteos dataset-show 91d98f6d-a065-431b-b8e0-1b996ac85cec
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| created_at  | 2016-12-15T23:04:11.000000          |
 +
| description | This is a sample dataset            |
 +
| head        | [u'4500,1,1,2016,5,0,40,50',        |
 +
|            | u'8000,2,1,2016,6,1,60,80',          |
 +
|            | u'9500,3,1,2016,0,2,88,92',          |
 +
|            | u'5000,4,1,2016,1,3,90,90',          |
 +
|            | u'4500,6,1,2016,3,3,80,90',          |
 +
|            | u'4000,7,1,2016,4,1,60,80',          |
 +
|            | u'4500,8,1,2016,5,0,40,50',          |
 +
|            | u'8000,9,1,2016,6,0,30,50',          |
 +
|            | u'9500,10,1,2016,0,0,40,50',        |
 +
|            | u'5000,11,1,2016,1,1,60,80']        |
 +
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
| name        | sample-data                          |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | available                            |
 +
| stderr      |                                      |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
 +
=== 5. Create a prediction model ===
 +
 +
In this example, User creates a Linear Regression Model from parsed dataset.
 +
 +
Parsed dataset has been already distributed in hdfs of experiment environment.
 +
 +
So, you spefity the internal url (internal://<dataset-id>) in source_dataset_url parameter.
 +
 +
<pre>
 +
$ vim sample/json/model_linear.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/model_linear.json
 +
{
 +
    "display_name": "linear-model",
 +
    "display_description": "Sample LinearRegression Model",
 +
    "source_dataset_url": "internal://91d98f6d-a065-431b-b8e0-1b996ac85cec",
 +
    "model_type": "LinearRegression",
 +
    "model_params": "{'numIterations': 200}",
 +
    "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286"
 +
}
 +
</pre>
 +
<pre>
 +
$ meteos model-create --json sample/json/model_linear.json
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| created_at  | 2016-12-15T23:09:36.000000          |
 +
| description | Sample LinearRegression Model        |
 +
| id          | 3cf02c2f-f043-49e5-a0df-0dc782868312 |
 +
| name        | linear-model                        |
 +
| params      | eydudW1JdGVyYXRpb25zJzogMjAwfQ==    |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | creating                            |
 +
| stderr      | None                                |
 +
| stdout      | None                                |
 +
| type        | LinearRegression                    |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
<pre>
 +
$ meteos model-list
 +
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
 +
| id                                  | name        | status    | type            | source_dataset_url                              |
 +
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
 +
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | available | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
 +
</pre>
 +
 +
=== 6. Predict ===
 +
 +
Create a learning job predicting a Sales Figures.
 +
 +
Specify the input value as "args" parameter.
 +
 +
Retrieve a predicted data as a stdout of job execution.
 +
 +
<pre>
 +
$ vim sample/json/learning.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/learning.json
 +
{
 +
    "display_name": "example-learning-job",
 +
    "display_description": "This is a sample job",
 +
    "model_id": "3cf02c2f-f043-49e5-a0df-0dc782868312",
 +
    "method": "predict",
 +
    "args": "1,7,2016,5,0,60,80"
 +
}
 +
</pre>
 +
<pre>
 +
$ meteos learning-create --json sample/json/learning.json
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| args        | MSw3LDIwMTYsNSwwLDYwLDgw            |
 +
| created_at  | 2016-12-15T23:13:29.000000          |
 +
| description | This is a sample job                |
 +
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
 +
| method      | predict                              |
 +
| name        | example-learning-job                |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | creating                            |
 +
| stderr      | None                                |
 +
| stdout      | None                                |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
<pre>
 +
$ meteos learning-list
 +
+--------------------------------------+----------------------+-----------+--------------------+---------------+
 +
| id                                  | name                | status    | args              | stdout        |
 +
+--------------------------------------+----------------------+-----------+--------------------+---------------+
 +
| ffcd3ae6-8b53-437e-94e2-3aabde741cb0 | example-learning-job | available | 1,7,2016,5,0,60,80 | 4048.18352502 |
 +
+--------------------------------------+----------------------+-----------+--------------------+---------------+
 +
</pre>
 +
 +
=== 7. Online Prediction ===
 +
 +
You can load a Prediction Model in advance for online prediction by using "meteos-load" command.
 +
 +
In online prediction, user can retrieve a predicted data immediately.
 +
 +
''' Note: Meteos can not load multiple models at the same time. So, you have to unload a model before loading another model. '''
 +
 +
<pre>
 +
$ meteos model-load 3cf02c2f-f043-49e5-a0df-0dc782868312
 +
</pre>
 +
<pre>
 +
$ meteos model-list
 +
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
 +
| id                                  | name        | status    | type            | source_dataset_url                              |
 +
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
 +
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | activating | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
 +
</pre>
 +
 +
Status is set to "active" when loading is complete.
 +
 +
<pre>
 +
$ meteos model-list
 +
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
 +
| id                                  | name        | status | type            | source_dataset_url                              |
 +
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
 +
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | active | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
 +
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
 +
</pre>
 +
 +
You can get a predicted data as a response of REST API .
 +
 +
<pre>
 +
$ meteos learning-create --json sample/json/learning.json
 +
+-------------+--------------------------------------+
 +
| Property    | Value                                |
 +
+-------------+--------------------------------------+
 +
| args        | MSw3LDIwMTYsNSwwLDYwLDgw            |
 +
| created_at  | 2016-12-15T23:13:29.000000          |
 +
| description | This is a sample job                |
 +
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
 +
| method      | predict                              |
 +
| name        | example-learning-job                |
 +
| project_id  | 4034bb3cd9324776a68c140fdd18baa4    |
 +
| status      | available                            |
 +
| stderr      | None                                |
 +
| stdout      | 4048.18352502                        |
 +
| user_id    | 64ad04e101df40b4b51e5f370a83412c    |
 +
+-------------+--------------------------------------+
 +
</pre>
 +
 +
<pre>
 +
$ meteos model-unload 3cf02c2f-f043-49e5-a0df-0dc782868312
 +
</pre>

Latest revision as of 06:45, 10 January 2017

Predict a Sales Figures using Meteos

In this example, you create a prediction model which predict sales by using Linear Regression.

Linear Regression is one of the algorithms in supervised learning.

Usecase.png

1. Create a experiment template

Create template of experiment. Experiment is a workspace of Machine Learning.

You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.

You can use a format located in python-meteosclient/sample/json/template.json 

$ glance image-list | grep meteos
| a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos                          |

$ neutron net-list | grep public
| 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public  | 1f979ae3-d6b7-4d03-ba0f-9d9112581783             |

$ vim sample/json/template.json

$ cat sample/json/template.json
{
    "display_name": "example-template",
    "display_description": "This is a sample template of experiment",
    "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
    "master_nodes_num": 1,
    "master_flavor_id": "4",
    "worker_nodes_num": 2,
    "worker_flavor_id": "2",
    "spark_version": "1.6.0",
    "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
}

$ meteos template-create --json sample/json/template.json
+---------------+-----------------------------------------+
| Property      | Value                                   |
+---------------+-----------------------------------------+
| cluster_id    | None                                    |
| created_at    | 2016-12-15T22:55:03.000000              |
| description   | This is a sample template of experiment |
| id            | d3180a78-74cc-432d-9e9d-68640b18abae    |
| master_flavor | 4                                       |
| master_nodes  | 1                                       |
| name          | example-template                        |
| project_id    | 4034bb3cd9324776a68c140fdd18baa4        |
| spark_version | 1.6.0                                   |
| status        | available                               |
| user_id       | 64ad04e101df40b4b51e5f370a83412c        |
| worker_flavor | 2                                       |
| worker_nodes  | 2                                       |
+---------------+-----------------------------------------+

2. Create a experiment from template

Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.

You can use a format located in python-meteosclient/sample/json/experiment.json 

$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem

$ neutron net-list | grep private
| e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |

$ vim sample/json/experiment.json

$ cat sample/json/experiment.json
{
    "display_name": "example-experiment",
    "display_description": "This is a sample experiment",
    "key_name": "key1",
    "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9",
    "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae"
}

$ meteos experiment-create --json sample/json/experiment.json
+--------------------+--------------------------------------+
| Property           | Value                                |
+--------------------+--------------------------------------+
| created_at         | 2016-12-15T22:56:51.000000           |
| description        | This is a sample experiment          |
| id                 | fcc8d055-e801-4652-af8c-5aabedbf0286 |
| key_name           | key1                                 |
| management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 |
| name               | example-experiment                   |
| project_id         | 4034bb3cd9324776a68c140fdd18baa4     |
| status             | creating                             |
| user_id            | 64ad04e101df40b4b51e5f370a83412c     |
+--------------------+--------------------------------------+

Meteos creates a experiment using OpenStack Sahara spark plugin.

You can see a sahara cluster and nova VMs created by Meteos as below. 

$ openstack dataprocessing cluster list (or sahara cluster-list)
+------------------+--------------------------------------+-------------+----------------+----------+
| Name             | Id                                   | Plugin name | Plugin version | Status   |
+------------------+--------------------------------------+-------------+----------------+----------+
| cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark       | 1.6.0          | Spawning |
+------------------+--------------------------------------+-------------+----------------+----------+

$ openstack server list (or nova list)
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
| ID                                   | Name                       | Status | Task State | Power State | Networks         |
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
| 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0  | ACTIVE | -          | Running     | private=10.0.0.5 |
| 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | -          | Running     | private=10.0.0.7 |
| a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | -          | Running     | private=10.0.0.4 |
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+

3. Upload a raw data

Upload a raw data (in this example past sales figures data) to OpenStack Swift.

You can use a sample data located in python-meteosclient/sample/data/linear_data.txt

Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left. 

$ cd sample/data/

/sample/data$ head linear_data.txt
4500,1,1,2016,5,0,40,50
8000,2,1,2016,6,1,60,80
9500,3,1,2016,0,2,88,92
5000,4,1,2016,1,3,90,90
0,5,1,2016,2,2,90,80
4500,6,1,2016,3,3,80,90
4000,7,1,2016,4,1,60,80
4500,8,1,2016,5,0,40,50
8000,9,1,2016,6,0,30,50
9500,10,1,2016,0,0,40,50

/sample/data$ swift upload meteos linear_data.txt
linear_data.txt

4. Parse a raw data

Parse a raw data to eliminate exception data.

In this case, you have to eliminate holiday sales figures because exception data affect the accuracy of prediction model.

You can use a sample format located in python-meteosclient/sample/json/dataset_parse.json

You can see the head data of parsed dataset by executing "meteos dataset-show <dataset-uuid>" command. 

$ vim ../python-meteosclient/sample/json/dataset_parse.json

$ cat sample/json/dataset_parse.json
{
    "source_dataset_url": "swift://meteos/linear_data.txt",
    "display_name": "sample-data",
    "display_description": "This is a sample dataset",
    "method": "parse",
    "params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}],
    "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286",
    "swift_tenant": "demo",
    "swift_username": "demo",
    "swift_password": "nova"
}

$ meteos dataset-create --json sample/json/dataset_parse.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-15T23:04:11.000000           |
| description | This is a sample dataset             |
| head        | None                                 |
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
| name        | sample-data                          |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | creating                             |
| stderr      | None                                 |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos dataset-list
+--------------------------------------+-------------+-----------+--------------------------------+
| id                                   | name        | status    | source_dataset_url             |
+--------------------------------------+-------------+-----------+--------------------------------+
| 91d98f6d-a065-431b-b8e0-1b996ac85cec | sample-data | available | swift://meteos/linear_data.txt |
+--------------------------------------+-------------+-----------+--------------------------------+

You can see that holiday sales figures has been eliminated. 

$ meteos dataset-show 91d98f6d-a065-431b-b8e0-1b996ac85cec
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-15T23:04:11.000000           |
| description | This is a sample dataset             |
| head        | [u'4500,1,1,2016,5,0,40,50',         |
|             | u'8000,2,1,2016,6,1,60,80',          |
|             | u'9500,3,1,2016,0,2,88,92',          |
|             | u'5000,4,1,2016,1,3,90,90',          |
|             | u'4500,6,1,2016,3,3,80,90',          |
|             | u'4000,7,1,2016,4,1,60,80',          |
|             | u'4500,8,1,2016,5,0,40,50',          |
|             | u'8000,9,1,2016,6,0,30,50',          |
|             | u'9500,10,1,2016,0,0,40,50',         |
|             | u'5000,11,1,2016,1,1,60,80']         |
| id          | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
| name        | sample-data                          |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | available                            |
| stderr      |                                      |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

5. Create a prediction model

In this example, User creates a Linear Regression Model from parsed dataset.

Parsed dataset has been already distributed in hdfs of experiment environment.

So, you spefity the internal url (internal://<dataset-id>) in source_dataset_url parameter. 

$ vim sample/json/model_linear.json

$ cat sample/json/model_linear.json
{
    "display_name": "linear-model",
    "display_description": "Sample LinearRegression Model",
    "source_dataset_url": "internal://91d98f6d-a065-431b-b8e0-1b996ac85cec",
    "model_type": "LinearRegression",
    "model_params": "{'numIterations': 200}",
    "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286"
}

$ meteos model-create --json sample/json/model_linear.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-15T23:09:36.000000           |
| description | Sample LinearRegression Model        |
| id          | 3cf02c2f-f043-49e5-a0df-0dc782868312 |
| name        | linear-model                         |
| params      | eydudW1JdGVyYXRpb25zJzogMjAwfQ==     |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | creating                             |
| stderr      | None                                 |
| stdout      | None                                 |
| type        | LinearRegression                     |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos model-list
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
| id                                   | name         | status    | type             | source_dataset_url                              |
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | available | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
+--------------------------------------+--------------+-----------+------------------+-------------------------------------------------+

6. Predict

Create a learning job predicting a Sales Figures.

Specify the input value as "args" parameter.

Retrieve a predicted data as a stdout of job execution. 

$ vim sample/json/learning.json

$ cat sample/json/learning.json
{
    "display_name": "example-learning-job",
    "display_description": "This is a sample job",
    "model_id": "3cf02c2f-f043-49e5-a0df-0dc782868312",
    "method": "predict",
    "args": "1,7,2016,5,0,60,80"
}

$ meteos learning-create --json sample/json/learning.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| args        | MSw3LDIwMTYsNSwwLDYwLDgw             |
| created_at  | 2016-12-15T23:13:29.000000           |
| description | This is a sample job                 |
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
| method      | predict                              |
| name        | example-learning-job                 |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | creating                             |
| stderr      | None                                 |
| stdout      | None                                 |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos learning-list
+--------------------------------------+----------------------+-----------+--------------------+---------------+
| id                                   | name                 | status    | args               | stdout        |
+--------------------------------------+----------------------+-----------+--------------------+---------------+
| ffcd3ae6-8b53-437e-94e2-3aabde741cb0 | example-learning-job | available | 1,7,2016,5,0,60,80 | 4048.18352502 |
+--------------------------------------+----------------------+-----------+--------------------+---------------+

7. Online Prediction

You can load a Prediction Model in advance for online prediction by using "meteos-load" command.

In online prediction, user can retrieve a predicted data immediately.

Note: Meteos can not load multiple models at the same time. So, you have to unload a model before loading another model. 

$ meteos model-load 3cf02c2f-f043-49e5-a0df-0dc782868312

$ meteos model-list
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
| id                                   | name         | status     | type             | source_dataset_url                              |
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | activating | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
+--------------------------------------+--------------+------------+------------------+-------------------------------------------------+

Status is set to "active" when loading is complete. 

$ meteos model-list
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
| id                                   | name         | status | type             | source_dataset_url                              |
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+
| 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | active | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |
+--------------------------------------+--------------+--------+------------------+-------------------------------------------------+

You can get a predicted data as a response of REST API . 

$ meteos learning-create --json sample/json/learning.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| args        | MSw3LDIwMTYsNSwwLDYwLDgw             |
| created_at  | 2016-12-15T23:13:29.000000           |
| description | This is a sample job                 |
| id          | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
| method      | predict                              |
| name        | example-learning-job                 |
| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
| status      | available                            |
| stderr      | None                                 |
| stdout      | 4048.18352502                        |
| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
+-------------+--------------------------------------+

$ meteos model-unload 3cf02c2f-f043-49e5-a0df-0dc782868312
Retrieved from "https://wiki.openstack.org/w/index.php?title=Meteos/ExampleLinear&oldid=149699"