|
|
| (2 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | == Predict sales using Meteos ==
| |
| | | | |
| − | In this example, user create a prediction model which predict sales by using Linear Regression.
| |
| − | Linear Regression is one of the algorithms in supervised learning.
| |
| − |
| |
| − | ''"Supervised learning is the machine learning task of inferring a function from labeled training data. The training data consist of a set of training examples. In supervised learning, each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal). (From Wikipedia https://en.wikipedia.org/wiki/Supervised_learning)''
| |
| − |
| |
| − | In this example, sales is the desired output value.
| |
| − |
| |
| − | [[File:Usecase.png]]
| |
| − |
| |
| − | === 1. Create a experiment template ===
| |
| − |
| |
| − | Create template of experiment.
| |
| − | Experiment is a workspace of Machine Learning.
| |
| − |
| |
| − | '''$ cat json/create_template.json'''
| |
| − | {
| |
| − | "display_name": "example-template",
| |
| − | "display_description": "This is a sample template of experiment",
| |
| − | "image_id" : "c48f5dba-45be-4165-9825-f4564fecebcd",
| |
| − | "master_nodes_num": 1,
| |
| − | "worker_nodes_num": 2,
| |
| − | "spark_version": "1.6"
| |
| − | }
| |
| − |
| |
| − | '''$ meteos template-create --json json/create_template.json'''
| |
| − |
| |
| − | +-------------+-----------------------------------------+
| |
| − | | Property | Value |
| |
| − | +-------------+-----------------------------------------+
| |
| − | | cluster_id | 0984a3e7-cd8b-4b7e-b4f0-f58dc4fc6c28 |
| |
| − | | created_at | 2016-10-20T02:10:13.483231 |
| |
| − | | description | This is a sample template of experiment |
| |
| − | | id | 26cbc33b-179e-4d22-8b1b-23c9a305f196 |
| |
| − | | image_id | c48f5dba-45be-4165-9825-f4564fecebcd |
| |
| − | | name | example-template |
| |
| − | | project_id | 5f011d076a6f4a328989c57ac7b4e501 |
| |
| − | | status | available |
| |
| − | +-------------+-----------------------------------------+
| |
| − |
| |
| − | === 2. Create a experiment from template ===
| |
| − |
| |
| − | Create a experiment by using template created in the above step.
| |
| − |
| |
| − | '''$ cat json/create_experiment.json'''
| |
| − | {
| |
| − | "display_name": "example-experiment",
| |
| − | "display_description": "This is a sample experiment",
| |
| − | "key_name": "key1",
| |
| − | "net_id": "bc85cb2a-53ad-4375-801e-ed507d416e09",
| |
| − | "template_id": "26cbc33b-179e-4d22-8b1b-23c9a305f196"
| |
| − | }
| |
| − |
| |
| − | '''$ meteos experiment-create --json json/create_experiment.json'''
| |
| − |
| |
| − | +-------------+--------------------------------------+
| |
| − | | Property | Value |
| |
| − | +-------------+--------------------------------------+
| |
| − | | created_at | 2016-10-20T02:11:51.778930 |
| |
| − | | description | This is a sample experiment |
| |
| − | | id | 0550a7db-b148-4319-bb35-16a9e4500d4a |
| |
| − | | name | example-experiment |
| |
| − | | project_id | 5f011d076a6f4a328989c57ac7b4e501 |
| |
| − | | status | available |
| |
| − | +-------------+--------------------------------------+
| |
| − |
| |
| − | Meteos creates a experiment using OpenStack Sahara spark plugin.
| |
| − | User can see a sahara cluster and nova VMs created by Meteos.
| |
| − |
| |
| − | '''$ sahara cluster-list'''
| |
| − | +------------------+--------------------------------------+--------+------------+
| |
| − | | name | id | status | node_count |
| |
| − | +------------------+--------------------------------------+--------+------------+
| |
| − | | cluster-0550a7db | 13b04f2e-5605-4b6b-b9fb-1ddf3dd85550 | Active | 3 |
| |
| − | +------------------+--------------------------------------+--------+------------+
| |
| − |
| |
| − | '''$ nova list'''
| |
| − | +--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+
| |
| − | | ID | Name | Status | Task State | Power State | Networks |
| |
| − | +--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+
| |
| − | | f526ec88-9000-4ddd-8f3b-ab60abbf7ca6 | cluster-0550a7db-master-0 | ACTIVE | - | Running | private=10.0.0.140, 192.168.0.99 |
| |
| − | | 6812aaa6-430d-4622-acd6-a558e1505e05 | cluster-0550a7db-workers-0 | ACTIVE | - | Running | private=10.0.0.141, 192.168.0.92 |
| |
| − | | 825727cf-f115-4d50-af26-99224ec94f8f | cluster-0550a7db-workers-1 | ACTIVE | - | Running | private=10.0.0.139, 192.168.0.100 |
| |
| − | +--------------------------------------+----------------------------+---------+------------+-------------+-----------------------------------+
| |
| − |
| |
| − | === 3. Upload a raw data ===
| |
| − |
| |
| − | Upload a raw data (sales data) to OpenStack Swift.
| |
| − | Raw data shows "sales", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.
| |
| − |
| |
| − | '''$ head meteos-test-data.txt'''
| |
| − | 500000,1,10,2016,6,0,68,50
| |
| − | 550000,2,10,2016,0,1,68,90
| |
| − | 300000,3,10,2016,1,0,60,55
| |
| − | 350000,4,10,2016,2,2,58,87
| |
| − | 0,5,10,2016,3,3,58,60 # a horiday
| |
| − | 400000,6,10,2016,4,3,60,60
| |
| − | 330000,7,10,2016,5,2,62,87
| |
| − | 550000,8,10,2016,6,1,66,92
| |
| − | 600000,9,10,2016,0,1,55,93
| |
| − | 330000,10,10,2016,1,0,57,55
| |
| − |
| |
| − | '''$ swift upload meteos meteos-test-data.txt'''
| |
| − | meteos-test-data.txt
| |
| − |
| |
| − | === 4. Download a raw data to experiment ===
| |
| − |
| |
| − | Download a raw data from swift to experiment.
| |
| − | Downloaded data is distributed in HDFS (Hadoop Distributed File System).
| |
| − |
| |
| − | '''$ cat json/download_dataset.json'''
| |
| − | {
| |
| − | "display_name": "sample-data",
| |
| − | "display_description": "This is a sample dataset",
| |
| − | "account": "demo:user01",
| |
| − | "password": "0251c36e80584efd",
| |
| − | "authurl": "http://192.168.0.4:5000/v2.0",
| |
| − | "container_name": "meteos",
| |
| − | "object_name": "meteos-test-data.txt",
| |
| − | "experiment_id": "0550a7db-b148-4319-bb35-16a9e4500d4a"
| |
| − | }
| |
| − |
| |
| − | '''$ meteos dataset-download --json json/download_dataset.json'''
| |
| − |
| |
| − | +-------------+--------------------------------------+
| |
| − | | Property | Value |
| |
| − | +-------------+--------------------------------------+
| |
| − | | created_at | 2016-10-20T02:17:27.831814 |
| |
| − | | description | This is a sample dataset |
| |
| − | | id | bd134722-22cd-4247-8e20-538933e0975d |
| |
| − | | name | sample-data |
| |
| − | | project_id | 5f011d076a6f4a328989c57ac7b4e501 |
| |
| − | | status | available |
| |
| − | +-------------+--------------------------------------+
| |
| − |
| |
| − | === 5. Parse a raw data ===
| |
| − |
| |
| − | Parse a raw data to enable MLlib (Apache Spark's scalable machine learning library) to handle it.
| |
| − | Requirement format depends on machine learning algorithms.
| |
| − | In this example, convert to a list format using map method and remove unused fields using filter method.
| |
| − |
| |
| − | [[File:Dataset-parse.png]]
| |
| − |
| |
| − | '''$ cat json/parse_dataset.json'''
| |
| − | {
| |
| − | "id": "bd134722-22cd-4247-8e20-538933e0975d",
| |
| − | "method": "parse",
| |
| − | "params": ["method": "map", "ars": "lambda l: l.split(',')",
| |
| − | "method": "filter", "ars": "lambda l: l[0]! = '0'"],
| |
| − | "experiment_id": "0550a7db-b148-4319-bb35-16a9e4500d4a"
| |
| − | }
| |
| − |
| |
| − | '''$ meteos dataset-parse --json json/parse_dataset.json'''
| |
| − |
| |
| − | === 6. Create a prediction model ===
| |
| − |
| |
| − | '''$ cat json/create_model.json'''
| |
| − | {
| |
| − | "display_name": "sample-lr-model",
| |
| − | "display_description": "This is a sample model",
| |
| − | "dataset_id": "bd134722-22cd-4247-8e20-538933e0975d",
| |
| − | "method": "LinearRegression",
| |
| − | "args": "{'numIterations': 10, 'desired_output':0}",
| |
| − | "experiment_id": "c08027ab-2ea5-4b57-840c-563908cfca46"
| |
| − | }
| |
| − |
| |
| − | '''$ meteos model-create --json json/create_model.json'''
| |
| − |
| |
| − | +-------------+--------------------------------------+
| |
| − | | Property | Value |
| |
| − | +-------------+--------------------------------------+
| |
| − | | created_at | 2016-10-20T02:17:27.831814 |
| |
| − | | description | This is a sample model |
| |
| − | | id | c48f5dba-45be-4165-9825-f4564fecebcd |
| |
| − | | name | sample-lr-mode |
| |
| − | | project_id | 5f011d076a6f4a328989c57ac7b4e501 |
| |
| − | | status | available |
| |
| − | +-------------+--------------------------------------+
| |
| − |
| |
| − | === 7. Predict ===
| |
| − |
| |
| − | Create a predict job.
| |
| − |
| |
| − | '''$ cat json/predict_data.json'''
| |
| − |
| |
| − | {
| |
| − | "display_name": "predict job",
| |
| − | "display_description": "This is a sample job",
| |
| − | "model_id": "bd134722-22cd-4247-8e20-538933e0975d",
| |
| − | "method": "predict",
| |
| − | "args": "11,10,2016,2,0,57,58",
| |
| − | "experiment_id": "c08027ab-2ea5-4b57-840c-563908cfca46"
| |
| − | }
| |
| − |
| |
| − | '''$ meteos job-create --json json/predict_data.json'''
| |
| − |
| |
| − | +-------------+--------------------------------------+
| |
| − | | Property | Value |
| |
| − | +-------------+--------------------------------------+
| |
| − | | created_at | 2016-10-20T02:17:37.644234 |
| |
| − | | description | This is a sample job |
| |
| − | | id | c38b9c22-d72c-4255-8236-7e319d351fad |
| |
| − | | name | predict_job |
| |
| − | | project_id | 5f011d076a6f4a328989c57ac7b4e501 |
| |
| − | | status | executing |
| |
| − | +-------------+--------------------------------------+
| |
| − |
| |
| − | Retrieve a predicted data as a stdout of job execution.
| |
| − |
| |
| − | '''$ meteos job-show c38b9c22-d72c-4255-8236-7e319d351fad'''
| |
| − |
| |
| − | +----------------------------+-------------------------------------------------+
| |
| − | | Property | Value |
| |
| − | +----------------------------+-------------------------------------------------+
| |
| − | | stdout | Value : (325944.477851) |
| |
| − | | stderr | |
| |
| − | +----------------------------+-------------------------------------------------+
| |
