Difference between revisions of "Meteos/ExampleDecisionTree"
< Meteos
(→Make a Decision to buy a stock using Meteos) |
(→6. Online Prediction) |
||
| (5 intermediate revisions by the same user not shown) | |||
| Line 16: | Line 16: | ||
<pre> | <pre> | ||
$ glance image-list | grep meteos | $ glance image-list | grep meteos | ||
| − | | | + | | a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos | |
</pre> | </pre> | ||
<pre> | <pre> | ||
$ neutron net-list | grep public | $ neutron net-list | grep public | ||
| − | | | + | | 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public | 1f979ae3-d6b7-4d03-ba0f-9d9112581783 | |
</pre> | </pre> | ||
<pre> | <pre> | ||
| Line 30: | Line 30: | ||
"display_name": "example-template", | "display_name": "example-template", | ||
"display_description": "This is a sample template of experiment", | "display_description": "This is a sample template of experiment", | ||
| − | "image_id" : " | + | "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57", |
"master_nodes_num": 1, | "master_nodes_num": 1, | ||
"master_flavor_id": "4", | "master_flavor_id": "4", | ||
| Line 36: | Line 36: | ||
"worker_flavor_id": "2", | "worker_flavor_id": "2", | ||
"spark_version": "1.6.0", | "spark_version": "1.6.0", | ||
| − | "floating_ip_pool": " | + | "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf" |
} | } | ||
</pre> | </pre> | ||
| Line 45: | Line 45: | ||
+---------------+-----------------------------------------+ | +---------------+-----------------------------------------+ | ||
| cluster_id | None | | | cluster_id | None | | ||
| − | | created_at | 2016-12- | + | | created_at | 2016-12-15T22:55:03.000000 | |
| description | This is a sample template of experiment | | | description | This is a sample template of experiment | | ||
| − | | id | | + | | id | d3180a78-74cc-432d-9e9d-68640b18abae | |
| master_flavor | 4 | | | master_flavor | 4 | | ||
| master_nodes | 1 | | | master_nodes | 1 | | ||
| name | example-template | | | name | example-template | | ||
| − | | project_id | | + | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | |
| spark_version | 1.6.0 | | | spark_version | 1.6.0 | | ||
| status | available | | | status | available | | ||
| − | | user_id | | + | | user_id | 64ad04e101df40b4b51e5f370a83412c | |
| worker_flavor | 2 | | | worker_flavor | 2 | | ||
| worker_nodes | 2 | | | worker_nodes | 2 | | ||
+---------------+-----------------------------------------+ | +---------------+-----------------------------------------+ | ||
</pre> | </pre> | ||
| − | |||
=== 2. Create a experiment from template === | === 2. Create a experiment from template === | ||
| Line 73: | Line 72: | ||
<pre> | <pre> | ||
$ neutron net-list | grep private | $ neutron net-list | grep private | ||
| − | | | + | | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 | |
</pre> | </pre> | ||
<pre> | <pre> | ||
| Line 84: | Line 83: | ||
"display_description": "This is a sample experiment", | "display_description": "This is a sample experiment", | ||
"key_name": "key1", | "key_name": "key1", | ||
| − | "neutron_management_network": " | + | "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9", |
| − | "template_id": " | + | "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae" |
} | } | ||
</pre> | </pre> | ||
| Line 93: | Line 92: | ||
| Property | Value | | | Property | Value | | ||
+--------------------+--------------------------------------+ | +--------------------+--------------------------------------+ | ||
| − | | created_at | 2016-12- | + | | created_at | 2016-12-15T22:56:51.000000 | |
| description | This is a sample experiment | | | description | This is a sample experiment | | ||
| − | | id | | + | | id | fcc8d055-e801-4652-af8c-5aabedbf0286 | |
| key_name | key1 | | | key_name | key1 | | ||
| − | | management_network | | + | | management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | |
| name | example-experiment | | | name | example-experiment | | ||
| − | | project_id | | + | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | |
| status | creating | | | status | creating | | ||
| − | | user_id | | + | | user_id | 64ad04e101df40b4b51e5f370a83412c | |
+--------------------+--------------------------------------+ | +--------------------+--------------------------------------+ | ||
</pre> | </pre> | ||
| Line 114: | Line 113: | ||
| Name | Id | Plugin name | Plugin version | Status | | | Name | Id | Plugin name | Plugin version | Status | | ||
+------------------+--------------------------------------+-------------+----------------+----------+ | +------------------+--------------------------------------+-------------+----------------+----------+ | ||
| − | | cluster- | + | | cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark | 1.6.0 | Spawning | |
+------------------+--------------------------------------+-------------+----------------+----------+ | +------------------+--------------------------------------+-------------+----------------+----------+ | ||
</pre> | </pre> | ||
<pre> | <pre> | ||
$ openstack server list (or nova list) | $ openstack server list (or nova list) | ||
| − | +--------------------------------------+----------------------------+--------+----------+------------+ | + | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ |
| − | | ID | Name | Status | Networks | + | | ID | Name | Status | Task State | Power State | Networks | |
| − | +--------------------------------------+----------------------------+--------+----------+------------+ | + | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ |
| − | | | + | | 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0 | ACTIVE | - | Running | private=10.0.0.5 | |
| − | | | + | | 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | - | Running | private=10.0.0.7 | |
| − | | | + | | a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | - | Running | private=10.0.0.4 | |
| − | +--------------------------------------+----------------------------+--------+----------+------------+ | + | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ |
</pre> | </pre> | ||
| Line 167: | Line 166: | ||
$ cat sample/json/model_decision_tree.json | $ cat sample/json/model_decision_tree.json | ||
{ | { | ||
| − | "display_name": " | + | "display_name": "tree-model", |
| − | "display_description": "Decision Tree Model", | + | "display_description": "Sample Decision Tree Model", |
"source_dataset_url": "swift://meteos/decision_tree_data.txt", | "source_dataset_url": "swift://meteos/decision_tree_data.txt", | ||
"model_type": "DecisionTreeRegression", | "model_type": "DecisionTreeRegression", | ||
"model_params": "{'numIterations': 100}", | "model_params": "{'numIterations': 100}", | ||
"dataset_format": "libsvm", | "dataset_format": "libsvm", | ||
| − | "experiment_id": " | + | "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286", |
"swift_tenant": "demo", | "swift_tenant": "demo", | ||
"swift_username": "demo", | "swift_username": "demo", | ||
| Line 184: | Line 183: | ||
| Property | Value | | | Property | Value | | ||
+-------------+--------------------------------------+ | +-------------+--------------------------------------+ | ||
| − | | created_at | 2016-12- | + | | created_at | 2016-12-15T23:23:04.000000 | |
| − | | description | Decision Tree Model | + | | description | Sample Decision Tree Model | |
| − | | id | | + | | id | 1e3547ff-294d-4a9c-8e94-9cc31c861288 | |
| − | | name | | + | | name | tree-model | |
| params | eydudW1JdGVyYXRpb25zJzogMTAwfQ== | | | params | eydudW1JdGVyYXRpb25zJzogMTAwfQ== | | ||
| − | | project_id | | + | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | |
| status | creating | | | status | creating | | ||
| stderr | None | | | stderr | None | | ||
| stdout | None | | | stdout | None | | ||
| type | DecisionTreeRegression | | | type | DecisionTreeRegression | | ||
| − | | user_id | | + | | user_id | 64ad04e101df40b4b51e5f370a83412c | |
+-------------+--------------------------------------+ | +-------------+--------------------------------------+ | ||
</pre> | </pre> | ||
<pre> | <pre> | ||
$ meteos model-list | $ meteos model-list | ||
| − | +--------------------------------------+ | + | +--------------------------------------+------------+-----------+------------------------+---------------------------------------+ |
| − | | id | name | + | | id | name | status | type | source_dataset_url | |
| − | +--------------------------------------+ | + | +--------------------------------------+------------+-----------+------------------------+---------------------------------------+ |
| − | | | + | | 1e3547ff-294d-4a9c-8e94-9cc31c861288 | tree-model | available | DecisionTreeRegression | swift://meteos/decision_tree_data.txt | |
| − | +--------------------------------------+ | + | +--------------------------------------+------------+-----------+------------------------+---------------------------------------+ |
| + | </pre> | ||
=== 5. Predict === | === 5. Predict === | ||
| Line 219: | Line 219: | ||
"display_name": "example-learning-job", | "display_name": "example-learning-job", | ||
"display_description": "This is a sample job", | "display_description": "This is a sample job", | ||
| − | + | "model_id": "1e3547ff-294d-4a9c-8e94-9cc31c861288", | |
| − | "model_id": " | ||
"method": "predict", | "method": "predict", | ||
"args": "1:1 2:-4 3:1 4:2 5:2 6:-2" | "args": "1:1 2:-4 3:1 4:2 5:2 6:-2" | ||
| Line 231: | Line 230: | ||
+-------------+--------------------------------------+ | +-------------+--------------------------------------+ | ||
| args | MToxIDI6LTQgMzoxIDQ6MiA1OjIgNjotMg== | | | args | MToxIDI6LTQgMzoxIDQ6MiA1OjIgNjotMg== | | ||
| − | | created_at | 2016-12- | + | | created_at | 2016-12-15T23:27:08.000000 | |
| description | This is a sample job | | | description | This is a sample job | | ||
| − | | id | | + | | id | cf2a6b1f-bb78-419e-a2fc-608fa0df52fc | |
| method | predict | | | method | predict | | ||
| name | example-learning-job | | | name | example-learning-job | | ||
| − | | project_id | | + | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | |
| status | creating | | | status | creating | | ||
| stderr | None | | | stderr | None | | ||
| stdout | None | | | stdout | None | | ||
| − | | user_id | | + | | user_id | 64ad04e101df40b4b51e5f370a83412c | |
+-------------+--------------------------------------+ | +-------------+--------------------------------------+ | ||
</pre> | </pre> | ||
| Line 248: | Line 247: | ||
<pre> | <pre> | ||
$ meteos learning-list | $ meteos learning-list | ||
| − | +---------------- | + | +--------------------------------------+----------------------+-----------+---------------------------+---------------+ |
| − | | id | name | + | | id | name | status | args | stdout | |
| − | +---------------- | + | +--------------------------------------+----------------------+-----------+---------------------------+---------------+ |
| − | | | + | | cf2a6b1f-bb78-419e-a2fc-608fa0df52fc | example-learning-job | available | 1:1 2:-4 3:1 4:2 5:2 6:-2 | 0.0 | |
| − | + | +--------------------------------------+----------------------+-----------+---------------------------+---------------+ | |
| − | |||
| − | |||
| − | +--------------------------------------+----------------------+ | ||
</pre> | </pre> | ||
| + | |||
| + | === 6. Online Prediction === | ||
| + | |||
| + | You can load a Prediction Model in advance for online prediction by using "meteos-load" command. | ||
| + | |||
| + | In online prediction, user can retrieve a predicted data immediately. | ||
| + | |||
| + | For more details, please see [https://wiki.openstack.org/wiki/Meteos/ExampleLinear#7._Online_Prediction here]. | ||
Latest revision as of 06:51, 10 January 2017
Contents
Make a Decision to buy a stock using Meteos
In this example, you creates a prediction model which make a decision to buy a stock by using Decision Tree Model.
1. Create a experiment template
Create template of experiment. Experiment is a workspace of Machine Learning.
You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.
You can use a format located in python-meteosclient/sample/json/template.json
$ glance image-list | grep meteos | a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos |
$ neutron net-list | grep public | 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public | 1f979ae3-d6b7-4d03-ba0f-9d9112581783 |
$ vim sample/json/template.json
$ cat sample/json/template.json
{
"display_name": "example-template",
"display_description": "This is a sample template of experiment",
"image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
"master_nodes_num": 1,
"master_flavor_id": "4",
"worker_nodes_num": 2,
"worker_flavor_id": "2",
"spark_version": "1.6.0",
"floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
}
$ meteos template-create --json sample/json/template.json +---------------+-----------------------------------------+ | Property | Value | +---------------+-----------------------------------------+ | cluster_id | None | | created_at | 2016-12-15T22:55:03.000000 | | description | This is a sample template of experiment | | id | d3180a78-74cc-432d-9e9d-68640b18abae | | master_flavor | 4 | | master_nodes | 1 | | name | example-template | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | spark_version | 1.6.0 | | status | available | | user_id | 64ad04e101df40b4b51e5f370a83412c | | worker_flavor | 2 | | worker_nodes | 2 | +---------------+-----------------------------------------+
2. Create a experiment from template
Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.
You can use a format located in python-meteosclient/sample/json/experiment.json
$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem
$ neutron net-list | grep private | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |
$ vim sample/json/experiment.json
$ cat sample/json/experiment.json
{
"display_name": "example-experiment",
"display_description": "This is a sample experiment",
"key_name": "key1",
"neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9",
"template_id": "d3180a78-74cc-432d-9e9d-68640b18abae"
}
$ meteos experiment-create --json sample/json/experiment.json +--------------------+--------------------------------------+ | Property | Value | +--------------------+--------------------------------------+ | created_at | 2016-12-15T22:56:51.000000 | | description | This is a sample experiment | | id | fcc8d055-e801-4652-af8c-5aabedbf0286 | | key_name | key1 | | management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | | name | example-experiment | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | user_id | 64ad04e101df40b4b51e5f370a83412c | +--------------------+--------------------------------------+
Meteos creates a experiment using OpenStack Sahara spark plugin.
You can see a sahara cluster and nova VMs created by Meteos as below.
$ openstack dataprocessing cluster list (or sahara cluster-list) +------------------+--------------------------------------+-------------+----------------+----------+ | Name | Id | Plugin name | Plugin version | Status | +------------------+--------------------------------------+-------------+----------------+----------+ | cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark | 1.6.0 | Spawning | +------------------+--------------------------------------+-------------+----------------+----------+
$ openstack server list (or nova list) +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ | 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0 | ACTIVE | - | Running | private=10.0.0.5 | | 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | - | Running | private=10.0.0.7 | | a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | - | Running | private=10.0.0.4 | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+
3. Upload a raw data
Upload a raw data (in this example past stock market data) to OpenStack Swift.
You can use a sample data located in python-meteosclient/sample/data/decision_tree_data.txt
Raw data shows "Flag which indicate buy or not buy", "Related Stock Code : Price Change",... from left.
$ cd sample/data/
$ head decision_tree_data.txt 1 1:-5 2:3 3:1 4:-5 5:-4 6:3 0 1:2 2:-5 3:0 4:0 5:1 6:-4 1 1:-3 2:1 3:4 4:-5 5:-3 6:4 0 1:4 2:-4 3:4 4:4 5:0 6:-1 1 1:-4 2:4 3:0 4:-5 5:-4 6:0 0 1:1 2:-4 3:3 4:3 5:0 6:-4 1 1:-3 2:4 3:0 4:-4 5:-3 6:2 0 1:2 2:-5 3:2 4:2 5:4 6:-3 1 1:-1 2:0 3:1 4:-2 5:-5 6:4 0 1:0 2:-4 3:1 4:1 5:4 6:-4
/sample/data$ swift upload meteos decision_tree_data.txt decision_tree_data.txt
4. Create a prediction model
In this example, you create a Decision Tree Model from dataset in swift directly.
$ vim sample/json/model_decision_tree.json
$ cat sample/json/model_decision_tree.json
{
"display_name": "tree-model",
"display_description": "Sample Decision Tree Model",
"source_dataset_url": "swift://meteos/decision_tree_data.txt",
"model_type": "DecisionTreeRegression",
"model_params": "{'numIterations': 100}",
"dataset_format": "libsvm",
"experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286",
"swift_tenant": "demo",
"swift_username": "demo",
"swift_password": "nova"
}
$ meteos model-create --json sample/json/model_decision_tree.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-12-15T23:23:04.000000 | | description | Sample Decision Tree Model | | id | 1e3547ff-294d-4a9c-8e94-9cc31c861288 | | name | tree-model | | params | eydudW1JdGVyYXRpb25zJzogMTAwfQ== | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | stderr | None | | stdout | None | | type | DecisionTreeRegression | | user_id | 64ad04e101df40b4b51e5f370a83412c | +-------------+--------------------------------------+
$ meteos model-list +--------------------------------------+------------+-----------+------------------------+---------------------------------------+ | id | name | status | type | source_dataset_url | +--------------------------------------+------------+-----------+------------------------+---------------------------------------+ | 1e3547ff-294d-4a9c-8e94-9cc31c861288 | tree-model | available | DecisionTreeRegression | swift://meteos/decision_tree_data.txt | +--------------------------------------+------------+-----------+------------------------+---------------------------------------+
5. Predict
Create a learning job making a decision to buy or not to buy.
Specify the input value as "args" parameter.
$ vim sample/json/learning.json
$ cat sample/json/learning.json
{
"display_name": "example-learning-job",
"display_description": "This is a sample job",
"model_id": "1e3547ff-294d-4a9c-8e94-9cc31c861288",
"method": "predict",
"args": "1:1 2:-4 3:1 4:2 5:2 6:-2"
}
$ meteos learning-create --json sample/json/learning.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | args | MToxIDI6LTQgMzoxIDQ6MiA1OjIgNjotMg== | | created_at | 2016-12-15T23:27:08.000000 | | description | This is a sample job | | id | cf2a6b1f-bb78-419e-a2fc-608fa0df52fc | | method | predict | | name | example-learning-job | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | stderr | None | | stdout | None | | user_id | 64ad04e101df40b4b51e5f370a83412c | +-------------+--------------------------------------+
Retrieve a predicted data as a stdout of job execution.
$ meteos learning-list +--------------------------------------+----------------------+-----------+---------------------------+---------------+ | id | name | status | args | stdout | +--------------------------------------+----------------------+-----------+---------------------------+---------------+ | cf2a6b1f-bb78-419e-a2fc-608fa0df52fc | example-learning-job | available | 1:1 2:-4 3:1 4:2 5:2 6:-2 | 0.0 | +--------------------------------------+----------------------+-----------+---------------------------+---------------+
6. Online Prediction
You can load a Prediction Model in advance for online prediction by using "meteos-load" command.
In online prediction, user can retrieve a predicted data immediately.
For more details, please see here.
