Difference between revisions of "Meteos/ExampleLinear"
< Meteos
(→6. Predict) |
(→1. Create a experiment template) |
||
| Line 18: | Line 18: | ||
<pre> | <pre> | ||
$ glance image-list | grep meteos | $ glance image-list | grep meteos | ||
| − | | | + | | a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos | |
</pre> | </pre> | ||
<pre> | <pre> | ||
$ neutron net-list | grep public | $ neutron net-list | grep public | ||
| − | | | + | | 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public | 1f979ae3-d6b7-4d03-ba0f-9d9112581783 | |
</pre> | </pre> | ||
<pre> | <pre> | ||
| Line 32: | Line 32: | ||
"display_name": "example-template", | "display_name": "example-template", | ||
"display_description": "This is a sample template of experiment", | "display_description": "This is a sample template of experiment", | ||
| − | "image_id" : " | + | "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57", |
"master_nodes_num": 1, | "master_nodes_num": 1, | ||
"master_flavor_id": "4", | "master_flavor_id": "4", | ||
| Line 38: | Line 38: | ||
"worker_flavor_id": "2", | "worker_flavor_id": "2", | ||
"spark_version": "1.6.0", | "spark_version": "1.6.0", | ||
| − | "floating_ip_pool": " | + | "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf" |
} | } | ||
</pre> | </pre> | ||
| Line 47: | Line 47: | ||
+---------------+-----------------------------------------+ | +---------------+-----------------------------------------+ | ||
| cluster_id | None | | | cluster_id | None | | ||
| − | | created_at | 2016-12- | + | | created_at | 2016-12-15T22:55:03.000000 | |
| description | This is a sample template of experiment | | | description | This is a sample template of experiment | | ||
| − | | id | | + | | id | d3180a78-74cc-432d-9e9d-68640b18abae | |
| master_flavor | 4 | | | master_flavor | 4 | | ||
| master_nodes | 1 | | | master_nodes | 1 | | ||
| name | example-template | | | name | example-template | | ||
| − | | project_id | | + | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | |
| spark_version | 1.6.0 | | | spark_version | 1.6.0 | | ||
| status | available | | | status | available | | ||
| − | | user_id | | + | | user_id | 64ad04e101df40b4b51e5f370a83412c | |
| worker_flavor | 2 | | | worker_flavor | 2 | | ||
| worker_nodes | 2 | | | worker_nodes | 2 | | ||
+---------------+-----------------------------------------+ | +---------------+-----------------------------------------+ | ||
</pre> | </pre> | ||
| − | |||
=== 2. Create a experiment from template === | === 2. Create a experiment from template === | ||
Revision as of 22:55, 15 December 2016
Contents
Predict a Sales Figures using Meteos
In this example, you create a prediction model which predict sales by using Linear Regression.
Linear Regression is one of the algorithms in supervised learning.
1. Create a experiment template
Create template of experiment. Experiment is a workspace of Machine Learning.
You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.
You can use a format located in python-meteosclient/sample/json/template.json
$ glance image-list | grep meteos | a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos |
$ neutron net-list | grep public | 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public | 1f979ae3-d6b7-4d03-ba0f-9d9112581783 |
$ vim sample/json/template.json
$ cat sample/json/template.json
{
"display_name": "example-template",
"display_description": "This is a sample template of experiment",
"image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
"master_nodes_num": 1,
"master_flavor_id": "4",
"worker_nodes_num": 2,
"worker_flavor_id": "2",
"spark_version": "1.6.0",
"floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
}
$ meteos template-create --json sample/json/template.json +---------------+-----------------------------------------+ | Property | Value | +---------------+-----------------------------------------+ | cluster_id | None | | created_at | 2016-12-15T22:55:03.000000 | | description | This is a sample template of experiment | | id | d3180a78-74cc-432d-9e9d-68640b18abae | | master_flavor | 4 | | master_nodes | 1 | | name | example-template | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | spark_version | 1.6.0 | | status | available | | user_id | 64ad04e101df40b4b51e5f370a83412c | | worker_flavor | 2 | | worker_nodes | 2 | +---------------+-----------------------------------------+
2. Create a experiment from template
Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.
You can use a format located in python-meteosclient/sample/json/experiment.json
$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem
$ neutron net-list | grep private | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | private | cb58940f-859b-48c6-b92a-3861470f1fc1 20.0.0.0/26 |
$ vim sample/json/experiment.json
$ cat sample/json/experiment.json
{
"display_name": "example-experiment",
"display_description": "This is a sample experiment",
"key_name": "key1",
"neutron_management_network": "8abc626e-2b06-4c67-9b2c-0231f0cef5b8",
"template_id": "8b7b9b89-f119-4b9b-b9b0-31598f819f1a"
}
$ meteos experiment-create --json sample/json/experiment.json +--------------------+--------------------------------------+ | Property | Value | +--------------------+--------------------------------------+ | created_at | 2016-12-04T07:20:11.000000 | | description | This is a sample experiment | | id | 91504a65-01cf-428f-81aa-596be7ca8619 | | key_name | key1 | | management_network | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | | name | example-experiment | | project_id | 67401cca74c2409b939e944bc6c8fcbe | | status | creating | | user_id | 181b1caa9d5b470393ca66b9e511d5b0 | +--------------------+--------------------------------------+
Meteos creates a experiment using OpenStack Sahara spark plugin.
You can see a sahara cluster and nova VMs created by Meteos as below.
$ openstack dataprocessing cluster list (or sahara cluster-list) +------------------+--------------------------------------+-------------+----------------+----------+ | Name | Id | Plugin name | Plugin version | Status | +------------------+--------------------------------------+-------------+----------------+----------+ | cluster-91504a65 | 13418fd9-5d2a-4ee6-b384-cb250b7e7714 | spark | 1.6.0 | Spawning | +------------------+--------------------------------------+-------------+----------------+----------+
$ openstack server list (or nova list) +--------------------------------------+----------------------------+--------+----------+------------+ | ID | Name | Status | Networks | Image Name | +--------------------------------------+----------------------------+--------+----------+------------+ | 58818eb5-ade7-407c-8c76-9fd9809632b4 | cluster-91504a65-workers-1 | BUILD | | meteos | | a151dbd9-de51-43ca-afb8-1fdeecce2891 | cluster-91504a65-workers-0 | BUILD | | meteos | | d02d85c5-0960-4b7e-880c-26b73c5dd8ad | cluster-91504a65-master-0 | BUILD | | meteos | +--------------------------------------+----------------------------+--------+----------+------------+
3. Upload a raw data
Upload a raw data (in this example past sales figures data) to OpenStack Swift.
You can use a sample data located in python-meteosclient/sample/data/linear_data.txt
Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.
$ cd sample/data/
/sample/data$ head linear_data.txt 4500,1,1,2016,5,0,40,50 8000,2,1,2016,6,1,60,80 9500,3,1,2016,0,2,88,92 5000,4,1,2016,1,3,90,90 0,5,1,2016,2,2,90,80 4500,6,1,2016,3,3,80,90 4000,7,1,2016,4,1,60,80 4500,8,1,2016,5,0,40,50 8000,9,1,2016,6,0,30,50 9500,10,1,2016,0,0,40,50
/sample/data$ swift upload meteos linear_data.txt linear_data.txt
4. Parse a raw data
Parse a raw data to eliminate exception data.
In this case, you have to eliminate holiday sales figures because exception data affect the accuracy of prediction model.
You can use a sample format located in python-meteosclient/sample/json/dataset_parse.json
You can see the head data of parsed dataset by executing "meteos dataset-show <dataset-uuid>" command.
$ vim ../python-meteosclient/sample/json/dataset_parse.json
$ cat sample/json/dataset_parse.json
{
"source_dataset_url": "swift://meteos/linear_data.txt",
"display_name": "sample-data",
"display_description": "sales figures dataset",
"method": "parse",
"params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}],
"experiment_id": "fad1b912-8f7d-4600-b048-7db27aae1af3",
"swift_tenant": "demo",
"swift_username": "demo",
"swift_password": "nova"
}
$ meteos dataset-create --json sample/json/dataset_parse.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-12-08T02:31:49.000000 | | description | sales figures dataset | | head | None | | id | 4e2bfa7c-b22e-4a2c-ad6c-2ebaa91ff7de | | name | sample-data | | project_id | 0e3e0f01952948848d8ae438279122fe | | status | creating | | stderr | None | | user_id | 2c3120e8228c4d0e9768f09346fe842d | +-------------+--------------------------------------+
$ meteos dataset-list +--------------------------------------+-------------+-----------------------+-----------+--------------------------------+----------------------------+ | id | name | description | status | source_dataset_url | created_at | +--------------------------------------+-------------+-----------------------+-----------+--------------------------------+----------------------------+ | 4e2bfa7c-b22e-4a2c-ad6c-2ebaa91ff7de | sample-data | sales figures dataset | available | swift://meteos/linear_data.txt | 2016-12-08T02:31:49.000000 | +--------------------------------------+-------------+-----------------------+-----------+--------------------------------+----------------------------+
$ meteos dataset-show 4e2bfa7c-b22e-4a2c-ad6c-2ebaa91ff7de +-------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +-------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2016-12-08T02:31:49.000000 | | description | sales figures dataset | | head | [u'4500,1,1,2016,5,0,40,50', u'8000,2,1,2016,6,1,60,80', u'9500,3,1,2016,0,2,88,92', u'5000,4,1,2016,1,3,90,90', u'4500,6,1,2016,3,3,80,90', u'4000,7,1,2016,4,1,60,80', u'4500,8,1,2016,5,0,40,50', u'8000,9,1,2016,6,0,30,50', u'9500,10,1,2016,0,0,40,50', u'5000,11,1,2016,1,1,60,80'] | | | | | id | 4e2bfa7c-b22e-4a2c-ad6c-2ebaa91ff7de | | name | sample-data | | project_id | 0e3e0f01952948848d8ae438279122fe | | status | available | | stderr | | | user_id | 2c3120e8228c4d0e9768f09346fe842d | +-------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
5. Create a prediction model
In this example, User creates a Linear Regression Model from parsed dataset.
Parsed dataset has been already distributed in hdfs of experiment environment.
So, you spefity the internal url (internal://<dataset-id>) in source_dataset_url parameter.
$ vim sample/json/model_linear.json
$ cat sample/json/model_linear.json
{
"display_name": "linear-model",
"display_description": "LinearRegression Model",
"source_dataset_url": "internal://4e2bfa7c-b22e-4a2c-ad6c-2ebaa91ff7de",
"model_type": "LinearRegression",
"model_params": "{'numIterations': 200}",
"experiment_id": "fad1b912-8f7d-4600-b048-7db27aae1af3"
}
$ meteos model-create --json sample/json/model_linear.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-12-08T02:35:29.000000 | | description | LinearRegression Model | | id | 8a4e0313-3fc1-4467-8117-198320c05493 | | name | linear-model | | params | eydudW1JdGVyYXRpb25zJzogMTAwfQ== | | project_id | 0e3e0f01952948848d8ae438279122fe | | status | creating | | stderr | None | | stdout | None | | type | LinearRegression | | user_id | 2c3120e8228c4d0e9768f09346fe842d | +-------------+--------------------------------------+
$ meteos model-list +--------------------------------------+--------------+------------------------+-----------+------------------+-------------------------------------------------+----------------------------+ | id | name | description | status | type | source_dataset_url | created_at | +--------------------------------------+--------------+------------------------+-----------+------------------+-------------------------------------------------+----------------------------+ | 8a4e0313-3fc1-4467-8117-198320c05493 | linear-model | LinearRegression Model | available | LinearRegression | internal://4e2bfa7c-b22e-4a2c-ad6c-2ebaa91ff7de | 2016-12-08T02:35:29.000000 | +--------------------------------------+--------------+------------------------+-----------+------------------+-------------------------------------------------+----------------------------+
6. Predict
Create a learning job predicting a Sales Figures.
Specify the input value as "args" parameter.
Retrieve a predicted data as a stdout of job execution.
$ vim sample/json/learning.json
$ cat sample/json/learning.json
{
"display_name": "example-learning-job",
"display_description": "This is a sample job",
"model_id": "8a4e0313-3fc1-4467-8117-198320c05493",
"method": "predict",
"args": "1,7,2016,5,0,60,80"
}
$ meteos learning-create --json sample/json/learning.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | args | MSw3LDIwMTYsNSwwLDYwLDgw | | created_at | 2016-12-08T02:42:35.000000 | | description | This is a sample job | | id | 38fdc2e1-85f8-4f4d-89d3-e896e8511c89 | | method | predict | | name | example-learning-job | | project_id | 0e3e0f01952948848d8ae438279122fe | | status | creating | | stderr | None | | stdout | None | | user_id | 2c3120e8228c4d0e9768f09346fe842d | +-------------+--------------------------------------+
$ meteos learning-list +--------------------------------------+----------------------+----------------------+-----------+--------------------+---------------+----------------------------+ | id | name | description | status | args | stdout | created_at | +--------------------------------------+----------------------+----------------------+-----------+--------------------+---------------+----------------------------+ | 38fdc2e1-85f8-4f4d-89d3-e896e8511c89 | example-learning-job | This is a sample job | available | 1,7,2016,5,0,60,80 | 4044.73614882 | 2016-12-08T02:42:35.000000 | +--------------------------------------+----------------------+----------------------+-----------+--------------------+---------------+----------------------------+
