Difference between revisions of "Meteos/Usecase"
(→1. Create a template of experiment) |
(→2. Create a experiment from template) |
||
Line 42: | Line 42: | ||
=== 2. Create a experiment from template === | === 2. Create a experiment from template === | ||
− | Create a experiment using template created in the above step. | + | Create a experiment by using template created in the above step. |
'''$ cat json/create_experiment.json''' | '''$ cat json/create_experiment.json''' |
Revision as of 06:33, 20 October 2016
Contents
Predict sales using Meteos
In this example, user create a prediction model which predict sales by using Linear Regression. Linear Regression is one of the algorithms in supervised learning.
"Supervised learning is the machine learning task of inferring a function from labeled training data. The training data consist of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal). (From Wikipedia https://en.wikipedia.org/wiki/Supervised_learning)
In this example, sales is the desired output value.
1. Create a experiment template
Create template of experiment. Experiment is a workspace of Machine Learning.
$ cat json/create_template.json
{ "display_name": "example-template", "display_description": "This is a sample template of experiment", "image_id" : "c48f5dba-45be-4165-9825-f4564fecebcd", "master_nodes_num": 1, "worker_nodes_num": 2, "spark_version": "1.6" }
$ meteos template-create --json json/create_template.json
+-------------+-----------------------------------------+ | Property | Value | +-------------+-----------------------------------------+ | cluster_id | 0984a3e7-cd8b-4b7e-b4f0-f58dc4fc6c28 | | created_at | 2016-10-20T02:10:13.483231 | | description | This is a sample template of experiment | | id | 26cbc33b-179e-4d22-8b1b-23c9a305f196 | | image_id | c48f5dba-45be-4165-9825-f4564fecebcd | | name | example-template | | project_id | 5f011d076a6f4a328989c57ac7b4e501 | | status | available | +-------------+-----------------------------------------+
2. Create a experiment from template
Create a experiment by using template created in the above step.
$ cat json/create_experiment.json
{ "display_name": "example-experiment", "display_description": "This is a sample experiment", "key_name": "key1", "net_id": "bc85cb2a-53ad-4375-801e-ed507d416e09", "template_id": "26cbc33b-179e-4d22-8b1b-23c9a305f196" }
$ meteos experiment-create --json json/create_experiment.json
+-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-10-20T02:11:51.778930 | | description | This is a sample experiment | | id | 0550a7db-b148-4319-bb35-16a9e4500d4a | | name | example-experiment | | project_id | 5f011d076a6f4a328989c57ac7b4e501 | | status | available | +-------------+--------------------------------------+
Meteos creates a experiment using OpenStack Sahara spark plugin. User can see a sahara cluster and nova VMs created by Meteos.
$ sahara cluster-list
+------------------+--------------------------------------+--------+------------+ | name | id | status | node_count | +------------------+--------------------------------------+--------+------------+ | cluster-0550a7db | 13b04f2e-5605-4b6b-b9fb-1ddf3dd85550 | Active | 3 | +------------------+--------------------------------------+--------+------------+
$ nova list
+--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+ | f526ec88-9000-4ddd-8f3b-ab60abbf7ca6 | cluster-0550a7db-master-0 | ACTIVE | - | Running | private=10.0.0.140, 192.168.0.99 | | 6812aaa6-430d-4622-acd6-a558e1505e05 | cluster-0550a7db-workers-0 | ACTIVE | - | Running | private=10.0.0.141, 192.168.0.92 | | 825727cf-f115-4d50-af26-99224ec94f8f | cluster-0550a7db-workers-1 | ACTIVE | - | Running | private=10.0.0.139, 192.168.0.100 | +--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+
3. Upload a raw data
Upload a raw data (sales data) to OpenStack Swift. Raw data shows "sales", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.
$ head meteos-test-data.txt
500000,1,10,2016,6,0,68,50 550000,2,10,2016,0,1,68,90 300000,3,10,2016,1,0,60,55 350000,4,10,2016,2,2,58,87 0,5,10,2016,3,3,58,60 # a horiday 400000,6,10,2016,4,3,60,60 330000,7,10,2016,5,2,62,87 550000,8,10,2016,6,1,66,92 600000,9,10,2016,0,1,55,93 330000,10,10,2016,1,0,57,55
$ swift upload meteos meteos-test-data.txt
meteos-test-data.txt
4. Download a raw data to experiment
Download a raw data from swift to experiment. Downloaded data is distributed in HDFS (Hadoop Distributed File System).
$ cat json/download_dataset.json
{ "display_name": "sample-data", "display_description": "This is a sample dataset", "account": "demo:user01", "password": "0251c36e80584efd", "authurl": "http://192.168.0.4:5000/v2.0", "container_name": "meteos", "object_name": "meteos-test-data.txt", "experiment_id": "0550a7db-b148-4319-bb35-16a9e4500d4a" }
$ meteos dataset-download --json json/download_dataset.json
+-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-10-20T02:17:27.831814 | | description | This is a sample dataset | | id | bd134722-22cd-4247-8e20-538933e0975d | | name | sample-data | | project_id | 5f011d076a6f4a328989c57ac7b4e501 | | status | available | +-------------+--------------------------------------+
5. Parse a raw data
Parse a raw data to enable MLlib (Apache Spark's scalable machine learning library) to handle it. Requirement format depends on machine learning algorithms. In this example, convert to a list format using map method and remove unused fields using filter method.
$ cat json/parse_dataset.json
{ "id": "bd134722-22cd-4247-8e20-538933e0975d", "method": "parse", "params": ["method": "map", "ars": "lambda l: l.split(',')", "method": "filter", "ars": "lambda l: l[0]! = '0'"], "experiment_id": "0550a7db-b148-4319-bb35-16a9e4500d4a" }
$ meteos dataset-parse --json json/parse_dataset.json
6. Create a prediction model
$ cat json/create_model.json
{ "display_name": "sample-lr-model", "display_description": "This is a sample model", "dataset_id": "bd134722-22cd-4247-8e20-538933e0975d", "method": "LinearRegression", "args": "{'numIterations': 10, 'desired_output':0}", "experiment_id": "c08027ab-2ea5-4b57-840c-563908cfca46" }
$ meteos model-create --json json/create_model.json
+-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-10-20T02:17:27.831814 | | description | This is a sample model | | id | c48f5dba-45be-4165-9825-f4564fecebcd | | name | sample-lr-mode | | project_id | 5f011d076a6f4a328989c57ac7b4e501 | | status | available | +-------------+--------------------------------------+
7. Predict
Create a predict job.
$ cat json/predict_data.json
{ "display_name": "predict job", "display_description": "This is a sample job", "model_id": "bd134722-22cd-4247-8e20-538933e0975d", "method": "predict", "args": "11,10,2016,2,0,57,58", "experiment_id": "c08027ab-2ea5-4b57-840c-563908cfca46" }
$ meteos job-create --json json/predict_data.json
+-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-10-20T02:17:37.644234 | | description | This is a sample job | | id | c38b9c22-d72c-4255-8236-7e319d351fad | | name | predict_job | | project_id | 5f011d076a6f4a328989c57ac7b4e501 | | status | executing | +-------------+--------------------------------------+
Retrieve a predicted data as a stdout of job execution.
$ meteos job-show c38b9c22-d72c-4255-8236-7e319d351fad
+----------------------------+-------------------------------------------------+ | Property | Value | +----------------------------+-------------------------------------------------+ | stdout | Value : (325944.477851) | | stderr | | +----------------------------+-------------------------------------------------+