Meteos/ExampleDecisionTree - Revision history

HiroyukiEguchi: /* 6. Online Prediction */

2017-01-10T06:51:38Z

‎6. Online Prediction

HiroyukiEguchi: /* 5. Predict */

2017-01-10T06:48:58Z

‎5. Predict

HiroyukiEguchi: /* 5. Predict */

2016-12-15T23:30:07Z

‎5. Predict

HiroyukiEguchi: /* 4. Create a prediction model */

2016-12-15T23:26:10Z

‎4. Create a prediction model

HiroyukiEguchi: /* Make a Decision to buy a stock using Meteos */

2016-12-15T23:21:02Z

‎Make a Decision to buy a stock using Meteos

Show changes

HiroyukiEguchi: /* 5. Predict */

2016-12-13T08:03:40Z

‎5. Predict

HiroyukiEguchi: /* Make a Decision to buy a stock using Meteos */

2016-12-08T03:53:48Z

‎Make a Decision to buy a stock using Meteos

HiroyukiEguchi: /* 4. Create a prediction model */

2016-12-08T03:32:56Z

‎4. Create a prediction model

HiroyukiEguchi: /* 3. Upload a raw data */

2016-12-08T03:23:53Z

‎3. Upload a raw data

HiroyukiEguchi: Created page with "== Make a Decision to buy a stock using Meteos == In this example, you creates a prediction model which predict stock market by using Decision Tree Model. File:Decision_t..."

2016-12-08T03:10:58Z

Created page with "== Make a Decision to buy a stock using Meteos == In this example, you creates a prediction model which predict stock market by using Decision Tree Model. File:Decision_t..."

New page

== Make a Decision to buy a stock using Meteos ==

In this example, you creates a prediction model which predict stock market by using Decision Tree Model.

[[ File:Decision_tree.png ]]

=== 1. Create a experiment template ===

Create template of experiment.
Experiment is a workspace of Machine Learning.

You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.

You can use a format located in '''python-meteosclient/sample/json/template.json'''

<pre>
$ glance image-list | grep meteos
| 45de4bbd-8419-40ff-8ed7-fc065c05e34f | meteos |
</pre>
<pre>
$ neutron net-list | grep public
| 84c13e76-ced9-4142-a885-280784f1f7a3 | public | a14de1c5-b8d4-434b-a056-9b0049b93402 |
</pre>
<pre>
$ vim sample/json/template.json
</pre>
<pre>
$ cat sample/json/template.json
{
"display_name": "example-template",
"display_description": "This is a sample template of experiment",
"image_id" : "45de4bbd-8419-40ff-8ed7-fc065c05e34f",
"master_nodes_num": 1,
"master_flavor_id": "4",
"worker_nodes_num": 2,
"worker_flavor_id": "2",
"spark_version": "1.6.0",
"floating_ip_pool": "84c13e76-ced9-4142-a885-280784f1f7a3"
}
</pre>
<pre>
$ meteos template-create --json sample/json/template.json
+---------------+-----------------------------------------+
| Property | Value |
+---------------+-----------------------------------------+
| cluster_id | None |
| created_at | 2016-12-04T07:16:29.000000 |
| description | This is a sample template of experiment |
| id | 8b7b9b89-f119-4b9b-b9b0-31598f819f1a |
| master_flavor | 4 |
| master_nodes | 1 |
| name | example-template |
| project_id | 67401cca74c2409b939e944bc6c8fcbe |
| spark_version | 1.6.0 |
| status | available |
| user_id | 181b1caa9d5b470393ca66b9e511d5b0 |
| worker_flavor | 2 |
| worker_nodes | 2 |
+---------------+-----------------------------------------+
</pre>

=== 2. Create a experiment from template ===

Create a experiment by using template created in the above step.
You have to confirm a neutron private network id and create keypair before creating a template.

You can use a format located in '''python-meteosclient/sample/json/experiment.json'''

<pre>
$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem
</pre>
<pre>
$ neutron net-list | grep private
| 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | private | cb58940f-859b-48c6-b92a-3861470f1fc1 20.0.0.0/26 |
</pre>
<pre>
$ vim sample/json/experiment.json
</pre>
<pre>
$ cat sample/json/experiment.json
{
"display_name": "example-experiment",
"display_description": "This is a sample experiment",
"key_name": "key1",
"neutron_management_network": "8abc626e-2b06-4c67-9b2c-0231f0cef5b8",
"template_id": "8b7b9b89-f119-4b9b-b9b0-31598f819f1a"
}
</pre>
<pre>
$ meteos experiment-create --json sample/json/experiment.json
+--------------------+--------------------------------------+
| Property | Value |
+--------------------+--------------------------------------+
| created_at | 2016-12-04T07:20:11.000000 |
| description | This is a sample experiment |
| id | 91504a65-01cf-428f-81aa-596be7ca8619 |
| key_name | key1 |
| management_network | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 |
| name | example-experiment |
| project_id | 67401cca74c2409b939e944bc6c8fcbe |
| status | creating |
| user_id | 181b1caa9d5b470393ca66b9e511d5b0 |
+--------------------+--------------------------------------+
</pre>

Meteos creates a experiment using OpenStack Sahara spark plugin.

You can see a sahara cluster and nova VMs created by Meteos as below.

<pre>
$ openstack dataprocessing cluster list (or sahara cluster-list)
+------------------+--------------------------------------+-------------+----------------+----------+
| Name | Id | Plugin name | Plugin version | Status |
+------------------+--------------------------------------+-------------+----------------+----------+
| cluster-91504a65 | 13418fd9-5d2a-4ee6-b384-cb250b7e7714 | spark | 1.6.0 | Spawning |
+------------------+--------------------------------------+-------------+----------------+----------+
</pre>
<pre>
$ openstack server list (or nova list)
+--------------------------------------+----------------------------+--------+----------+------------+
| ID | Name | Status | Networks | Image Name |
+--------------------------------------+----------------------------+--------+----------+------------+
| 58818eb5-ade7-407c-8c76-9fd9809632b4 | cluster-91504a65-workers-1 | BUILD | | meteos |
| a151dbd9-de51-43ca-afb8-1fdeecce2891 | cluster-91504a65-workers-0 | BUILD | | meteos |
| d02d85c5-0960-4b7e-880c-26b73c5dd8ad | cluster-91504a65-master-0 | BUILD | | meteos |
+--------------------------------------+----------------------------+--------+----------+------------+
</pre>

=== 3. Upload a raw data ===

Upload a raw data (in this example past stock market data) to OpenStack Swift.

You can use a sample data located in '''python-meteosclient/sample/data/decision_tree_data.txt'''

Raw data shows "Flag which indicate buy or not buy", "Related Stock Code : Price Change",... from left.

<pre>
$ cd sample/data/
</pre>
<pre>
$ head decision_tree_data.txt
1 1:-5 2:3 3:1 4:-5 5:-4 6:3
0 1:2 2:-5 3:0 4:0 5:1 6:-4
1 1:-3 2:1 3:4 4:-5 5:-3 6:4
0 1:4 2:-4 3:4 4:4 5:0 6:-1
1 1:-4 2:4 3:0 4:-5 5:-4 6:0
0 1:1 2:-4 3:3 4:3 5:0 6:-4
1 1:-3 2:4 3:0 4:-4 5:-3 6:2
0 1:2 2:-5 3:2 4:2 5:4 6:-3
1 1:-1 2:0 3:1 4:-2 5:-5 6:4
0 1:0 2:-4 3:1 4:1 5:4 6:-4
</pre>
<pre>
/sample/data$ swift upload meteos decision_tree_data.txt
decision_tree_data.txt
</pre>

← Older revision		Revision as of 06:51, 10 January 2017
Line 260:		Line 260:
	In online prediction, user can retrieve a predicted data immediately.		In online prediction, user can retrieve a predicted data immediately.

−	~~Please~~ see ~~the details~~ [https://wiki.openstack.org/wiki/Meteos/ExampleLinear#7._Online_Prediction ~~below~~].	+	For more details, please see [https://wiki.openstack.org/wiki/Meteos/ExampleLinear#7._Online_Prediction here].

← Older revision		Revision as of 06:48, 10 January 2017
Line 253:		Line 253:
	+--------------------------------------+----------------------+-----------+---------------------------+---------------+		+--------------------------------------+----------------------+-----------+---------------------------+---------------+
	</pre>		</pre>
		+
		+	=== 6. Online Prediction ===
		+
		+	You can load a Prediction Model in advance for online prediction by using "meteos-load" command.
		+
		+	In online prediction, user can retrieve a predicted data immediately.
		+
		+	Please see the details [https://wiki.openstack.org/wiki/Meteos/ExampleLinear#7._Online_Prediction below].

@@ Line 219: / Line 219: @@
      "display_name": "example-learning-job",
      "display_description": "This is a sample job",
-     "model_id": "5c16c3f7-0093-4f82-a4f9-2ae421fc901f",
+     "model_id": "1e3547ff-294d-4a9c-8e94-9cc31c861288",
      "method": "predict",
      "args": "1:1 2:-4 3:1 4:2 5:2 6:-2"
@@ Line 230: / Line 230: @@
 +-------------+--------------------------------------+
 | args        | MToxIDI6LTQgMzoxIDQ6MiA1OjIgNjotMg== |
-| created_at  | 2016-12-08T03:26:25.000000           |
+| created_at  | 2016-12-15T23:27:08.000000           |
 | description | This is a sample job                 |
-| id          | bbfaa00b-0857-441a-ac12-ef766a419ae0 |
+| id          | cf2a6b1f-bb78-419e-a2fc-608fa0df52fc |
 | method      | predict                              |
 | name        | example-learning-job                 |
-| project_id  | 0e3e0f01952948848d8ae438279122fe     |
+| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
 | status      | creating                             |
 | stderr      | None                                 |
 | stdout      | None                                 |
-| user_id     | 2c3120e8228c4d0e9768f09346fe842d     |
+| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
 +-------------+--------------------------------------+
 </pre>
@@ Line 247: / Line 247: @@
 <pre>
 $ meteos learning-list
-+--------------------------------------+----------------------+----------------------+-----------+----------------------------+--------+----------------------------+
++--------------------------------------+----------------------+-----------+---------------------------+---------------+
-| id                                   | name                 | description          | status    | args                       | stdout | created_at                 |
+| id                                   | name                 | status    | args                      | stdout        |
-+--------------------------------------+----------------------+----------------------+-----------+----------------------------+--------+----------------------------+
++--------------------------------------+----------------------+-----------+---------------------------+---------------+
-| 429dc79d-bac0-4ea8-aa7d-6853066a7b26 | example-learning-job | This is a sample job | available | 1:-3 2:3 3:0 4:-3 5:-2 6:4 | 1.0    | 2016-12-08T03:30:08.000000 |
+| cf2a6b1f-bb78-419e-a2fc-608fa0df52fc | example-learning-job | available | 1:1 2:-4 3:1 4:2 5:2 6:-2 | 0.0           |
-| bbfaa00b-0857-441a-ac12-ef766a419ae0 | example-learning-job | This is a sample job | available | 1:1 2:-4 3:1 4:2 5:2 6:-2  | 0.0    | 2016-12-08T03:26:25.000000 |
++--------------------------------------+----------------------+-----------+---------------------------+---------------+
 </pre>

@@ Line 166: / Line 166: @@
 $ cat sample/json/model_decision_tree.json
+{
-     "display_name": "sample-tree-model",
+     "display_name": "tree-model",
-     "display_description": "Decision Tree Model",
+     "display_description": "Sample Decision Tree Model",
      "source_dataset_url": "swift://meteos/decision_tree_data.txt",
      "model_type": "DecisionTreeRegression",
      "model_params": "{'numIterations': 100}",
      "dataset_format": "libsvm",
-     "experiment_id": "fad1b912-8f7d-4600-b048-7db27aae1af3",
+     "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286",
      "swift_tenant": "demo",
      "swift_username": "demo",
@@ Line 183: / Line 183: @@
 | Property    | Value                                |
 +-------------+--------------------------------------+
-| created_at  | 2016-12-08T03:18:47.000000           |
+| created_at  | 2016-12-15T23:23:04.000000           |
-| description | Decision Tree Model                  |
+| description | Sample Decision Tree Model           |
-| id          | 5c16c3f7-0093-4f82-a4f9-2ae421fc901f |
+| id          | 1e3547ff-294d-4a9c-8e94-9cc31c861288 |
-| name        | sample-tree-model                    |
+| name        | tree-model                           |
 | params      | eydudW1JdGVyYXRpb25zJzogMTAwfQ==     |
-| project_id  | 0e3e0f01952948848d8ae438279122fe     |
+| project_id  | 4034bb3cd9324776a68c140fdd18baa4     |
 | status      | creating                             |
 | stderr      | None                                 |
 | stdout      | None                                 |
 | type        | DecisionTreeRegression               |
-| user_id     | 2c3120e8228c4d0e9768f09346fe842d     |
+| user_id     | 64ad04e101df40b4b51e5f370a83412c     |
 +-------------+--------------------------------------+
 </pre>
 <pre>
 $ meteos model-list
-+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+
++--------------------------------------+------------+-----------+------------------------+---------------------------------------+
-| id                                   | name              | description            | status    | type                   | source_dataset_url                      | created_at                 |
+| id                                   | name       | status    | type                   | source_dataset_url                    |
-+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+
++--------------------------------------+------------+-----------+------------------------+---------------------------------------+
-| 5c16c3f7-0093-4f82-a4f9-2ae421fc901f | sample-tree-model | Decision Tree Model    | available | DecisionTreeRegression | swift://meteos/decision_tree_data.txt   | 2016-12-08T03:18:47.000000 |
+| 1e3547ff-294d-4a9c-8e94-9cc31c861288 | tree-model | available | DecisionTreeRegression | swift://meteos/decision_tree_data.txt |
-+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+</pre>
++--------------------------------------+------------+-----------+------------------------+---------------------------------------+
 === 5. Predict ===

@@ Line 219: / Line 219: @@
      "display_name": "example-learning-job",
      "display_description": "This is a sample job",
      "model_id": "5c16c3f7-0093-4f82-a4f9-2ae421fc901f",
      "method": "predict",

@@ Line 1: / Line 1: @@
 == Make a Decision to buy a stock using Meteos ==
-In this example, you creates a prediction model which predict stock market by using Decision Tree Model.
+In this example, you creates a prediction model which make a decision to buy a stock by using Decision Tree Model.
 [[ File:Decision_tree.png ]]

← Older revision		Revision as of 03:32, 8 December 2016
Line 204:		Line 204:
	\| 5c16c3f7-0093-4f82-a4f9-2ae421fc901f \| sample-tree-model \| Decision Tree Model \| available \| DecisionTreeRegression \| swift://meteos/decision_tree_data.txt \| 2016-12-08T03:18:47.000000 \|		\| 5c16c3f7-0093-4f82-a4f9-2ae421fc901f \| sample-tree-model \| Decision Tree Model \| available \| DecisionTreeRegression \| swift://meteos/decision_tree_data.txt \| 2016-12-08T03:18:47.000000 \|
	+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+</pre>		+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+</pre>
		+
		+	=== 5. Predict ===
		+
		+	Create a learning job making a decision to buy or not to buy.
		+
		+	Specify the input value as "args" parameter.
		+
		+	<pre>
		+	$ vim sample/json/learning.json
		+	</pre>
		+	<pre>
		+	$ cat sample/json/learning.json
		+	{
		+	"display_name": "example-learning-job",
		+	"display_description": "This is a sample job",
		+	"experiment_id": "fad1b912-8f7d-4600-b048-7db27aae1af3",
		+	"model_id": "5c16c3f7-0093-4f82-a4f9-2ae421fc901f",
		+	"method": "predict",
		+	"args": "1:1 2:-4 3:1 4:2 5:2 6:-2"
		+	}
		+	</pre>
		+	<pre>
		+	$ meteos learning-create --json sample/json/learning.json
		+	+-------------+--------------------------------------+
		+	\| Property \| Value \|
		+	+-------------+--------------------------------------+
		+	\| args \| MToxIDI6LTQgMzoxIDQ6MiA1OjIgNjotMg== \|
		+	\| created_at \| 2016-12-08T03:26:25.000000 \|
		+	\| description \| This is a sample job \|
		+	\| id \| bbfaa00b-0857-441a-ac12-ef766a419ae0 \|
		+	\| method \| predict \|
		+	\| name \| example-learning-job \|
		+	\| project_id \| 0e3e0f01952948848d8ae438279122fe \|
		+	\| status \| creating \|
		+	\| stderr \| None \|
		+	\| stdout \| None \|
		+	\| user_id \| 2c3120e8228c4d0e9768f09346fe842d \|
		+	+-------------+--------------------------------------+
		+	</pre>
		+
		+	Retrieve a predicted data as a stdout of job execution.
		+
		+	<pre>
		+	$ meteos learning-list
		+	+--------------------------------------+----------------------+----------------------+-----------+----------------------------+--------+----------------------------+
		+	\| id \| name \| description \| status \| args \| stdout \| created_at \|
		+	+--------------------------------------+----------------------+----------------------+-----------+----------------------------+--------+----------------------------+
		+	\| 429dc79d-bac0-4ea8-aa7d-6853066a7b26 \| example-learning-job \| This is a sample job \| available \| 1:-3 2:3 3:0 4:-3 5:-2 6:4 \| 1.0 \| 2016-12-08T03:30:08.000000 \|
		+	\| bbfaa00b-0857-441a-ac12-ef766a419ae0 \| example-learning-job \| This is a sample job \| available \| 1:1 2:-4 3:1 4:2 5:2 6:-2 \| 0.0 \| 2016-12-08T03:26:25.000000 \|
		+	\| cff6fa79-9a8a-4452-8afa-b6b284393a8c \| example-learning-job \| This is a sample job \| available \| 1:-4 2:2 3:2 4:-1 5:-2 6:1 \| 1.0 \| 2016-12-08T03:29:42.000000 \|
		+	\| e6802bb1-24ee-4a6f-b7ff-a45e61b7e8c2 \| example-learning-job \| This is a sample job \| available \| 1:3 2:-2 3:-1 4:3 5:2 6:-4 \| 0.0 \| 2016-12-08T03:31:01.000000 \|
		+	+--------------------------------------+----------------------+----------------------+-----------+----------------------------+--------+----------------------------+
		+	</pre>

← Older revision		Revision as of 03:23, 8 December 2016
Line 156:		Line 156:
	decision_tree_data.txt		decision_tree_data.txt
	</pre>		</pre>
		+
		+	=== 4. Create a prediction model ===
		+
		+	In this example, you create a Decision Tree Model from dataset in swift directly.
		+
		+	<pre>
		+	$ vim sample/json/model_decision_tree.json
		+	</pre>
		+	<pre>
		+	$ cat sample/json/model_decision_tree.json
		+	{
		+	"display_name": "sample-tree-model",
		+	"display_description": "Decision Tree Model",
		+	"source_dataset_url": "swift://meteos/decision_tree_data.txt",
		+	"model_type": "DecisionTreeRegression",
		+	"model_params": "{'numIterations': 100}",
		+	"dataset_format": "libsvm",
		+	"experiment_id": "fad1b912-8f7d-4600-b048-7db27aae1af3",
		+	"swift_tenant": "demo",
		+	"swift_username": "demo",
		+	"swift_password": "nova"
		+	}
		+	</pre>
		+	<pre>
		+	$ meteos model-create --json sample/json/model_decision_tree.json
		+	+-------------+--------------------------------------+
		+	\| Property \| Value \|
		+	+-------------+--------------------------------------+
		+	\| created_at \| 2016-12-08T03:18:47.000000 \|
		+	\| description \| Decision Tree Model \|
		+	\| id \| 5c16c3f7-0093-4f82-a4f9-2ae421fc901f \|
		+	\| name \| sample-tree-model \|
		+	\| params \| eydudW1JdGVyYXRpb25zJzogMTAwfQ== \|
		+	\| project_id \| 0e3e0f01952948848d8ae438279122fe \|
		+	\| status \| creating \|
		+	\| stderr \| None \|
		+	\| stdout \| None \|
		+	\| type \| DecisionTreeRegression \|
		+	\| user_id \| 2c3120e8228c4d0e9768f09346fe842d \|
		+	+-------------+--------------------------------------+
		+	</pre>
		+	<pre>
		+	$ meteos model-list
		+	+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+
		+	\| id \| name \| description \| status \| type \| source_dataset_url \| created_at \|
		+	+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+
		+	\| 5c16c3f7-0093-4f82-a4f9-2ae421fc901f \| sample-tree-model \| Decision Tree Model \| available \| DecisionTreeRegression \| swift://meteos/decision_tree_data.txt \| 2016-12-08T03:18:47.000000 \|
		+	+--------------------------------------+-------------------+------------------------+-----------+------------------------+-----------------------------------------+----------------------------+</pre>