Meteos/ExampleLinear

Predict a Sales Figures using Meteos
In this example, you create a prediction model which predict sales by using Linear Regression.

Linear Regression is one of the algorithms in supervised learning.



1. Create a experiment template
Create template of experiment. Experiment is a workspace of Machine Learning.

You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.

You can use a format located in python-meteosclient/sample/json/template.json

$ glance image-list | grep meteos $ neutron net-list | grep public $ vim sample/json/template.json $ cat sample/json/template.json {   "display_name": "example-template", "display_description": "This is a sample template of experiment", "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57", "master_nodes_num": 1, "master_flavor_id": "4", "worker_nodes_num": 2, "worker_flavor_id": "2", "spark_version": "1.6.0", "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf" } $ meteos template-create --json sample/json/template.json +---+-+ +---+-+ +---+-+
 * a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos                         |
 * 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public | 1f979ae3-d6b7-4d03-ba0f-9d9112581783             |
 * Property     | Value                                   |
 * cluster_id   | None                                    |
 * created_at   | 2016-12-15T22:55:03.000000              |
 * description  | This is a sample template of experiment |
 * id           | d3180a78-74cc-432d-9e9d-68640b18abae    |
 * master_flavor | 4                                      |
 * master_nodes | 1                                       |
 * name         | example-template                        |
 * project_id   | 4034bb3cd9324776a68c140fdd18baa4        |
 * spark_version | 1.6.0                                  |
 * status       | available                               |
 * user_id      | 64ad04e101df40b4b51e5f370a83412c        |
 * worker_flavor | 2                                      |
 * worker_nodes | 2                                       |

2. Create a experiment from template
Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.

You can use a format located in python-meteosclient/sample/json/experiment.json

$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem $ neutron net-list | grep private $ vim sample/json/experiment.json $ cat sample/json/experiment.json {   "display_name": "example-experiment", "display_description": "This is a sample experiment", "key_name": "key1", "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9", "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae" } $ meteos experiment-create --json sample/json/experiment.json ++--+ ++--+ ++--+
 * e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |
 * Property          | Value                                |
 * created_at        | 2016-12-15T22:56:51.000000           |
 * description       | This is a sample experiment          |
 * id                | fcc8d055-e801-4652-af8c-5aabedbf0286 |
 * key_name          | key1                                 |
 * management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 |
 * name              | example-experiment                   |
 * project_id        | 4034bb3cd9324776a68c140fdd18baa4     |
 * status            | creating                             |
 * user_id           | 64ad04e101df40b4b51e5f370a83412c     |

Meteos creates a experiment using OpenStack Sahara spark plugin.

You can see a sahara cluster and nova VMs created by Meteos as below.

$ openstack dataprocessing cluster list (or sahara cluster-list) +--+--+-++--+ +--+--+-++--+ +--+--+-++--+ $ openstack server list (or nova list) +--++++-+--+ +--++++-+--+ +--++++-+--+
 * Name            | Id                                   | Plugin name | Plugin version | Status   |
 * cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark      | 1.6.0          | Spawning |
 * ID                                  | Name                       | Status | Task State | Power State | Networks         |
 * 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0 | ACTIVE | -          | Running     | private=10.0.0.5 |
 * 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | -         | Running     | private=10.0.0.7 |
 * a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | -         | Running     | private=10.0.0.4 |

3. Upload a raw data
Upload a raw data (in this example past sales figures data) to OpenStack Swift.

You can use a sample data located in python-meteosclient/sample/data/linear_data.txt

Raw data shows "sales figures", "day", "month", "year", "day of week", "parameter which indicates weather", "degree", "humidity" from left.

$ cd sample/data/ /sample/data$ head linear_data.txt 4500,1,1,2016,5,0,40,50 8000,2,1,2016,6,1,60,80 9500,3,1,2016,0,2,88,92 5000,4,1,2016,1,3,90,90 0,5,1,2016,2,2,90,80 4500,6,1,2016,3,3,80,90 4000,7,1,2016,4,1,60,80 4500,8,1,2016,5,0,40,50 8000,9,1,2016,6,0,30,50 9500,10,1,2016,0,0,40,50 /sample/data$ swift upload meteos linear_data.txt linear_data.txt

4. Parse a raw data
Parse a raw data to eliminate exception data.

In this case, you have to eliminate holiday sales figures because exception data affect the accuracy of prediction model.

You can use a sample format located in python-meteosclient/sample/json/dataset_parse.json

You can see the head data of parsed dataset by executing "meteos dataset-show " command.

$ vim ../python-meteosclient/sample/json/dataset_parse.json $ cat sample/json/dataset_parse.json {   "source_dataset_url": "swift://meteos/linear_data.txt", "display_name": "sample-data", "display_description": "This is a sample dataset", "method": "parse", "params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}], "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286", "swift_tenant": "demo", "swift_username": "demo", "swift_password": "nova" } $ meteos dataset-create --json sample/json/dataset_parse.json +-+--+ +-+--+ +-+--+ $ meteos dataset-list +--+-+---++ +--+-+---++ +--+-+---++ You can see that holiday sales figures has been eliminated. $ meteos dataset-show 91d98f6d-a065-431b-b8e0-1b996ac85cec +-+--+ +-+--+ +-+--+
 * Property   | Value                                |
 * created_at | 2016-12-15T23:04:11.000000           |
 * description | This is a sample dataset            |
 * head       | None                                 |
 * id         | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
 * name       | sample-data                          |
 * project_id | 4034bb3cd9324776a68c140fdd18baa4     |
 * status     | creating                             |
 * stderr     | None                                 |
 * user_id    | 64ad04e101df40b4b51e5f370a83412c     |
 * id                                  | name        | status    | source_dataset_url             |
 * 91d98f6d-a065-431b-b8e0-1b996ac85cec | sample-data | available | swift://meteos/linear_data.txt |
 * Property   | Value                                |
 * created_at | 2016-12-15T23:04:11.000000           |
 * description | This is a sample dataset            |
 * head       | [u'4500,1,1,2016,5,0,40,50',         |
 * | u'8000,2,1,2016,6,1,60,80',         |
 * | u'9500,3,1,2016,0,2,88,92',         |
 * | u'5000,4,1,2016,1,3,90,90',         |
 * | u'4500,6,1,2016,3,3,80,90',         |
 * | u'4000,7,1,2016,4,1,60,80',         |
 * | u'4500,8,1,2016,5,0,40,50',         |
 * | u'8000,9,1,2016,6,0,30,50',         |
 * | u'9500,10,1,2016,0,0,40,50',        |
 * | u'5000,11,1,2016,1,1,60,80']        |
 * id         | 91d98f6d-a065-431b-b8e0-1b996ac85cec |
 * name       | sample-data                          |
 * project_id | 4034bb3cd9324776a68c140fdd18baa4     |
 * status     | available                            |
 * stderr     |                                      |
 * user_id    | 64ad04e101df40b4b51e5f370a83412c     |

5. Create a prediction model
In this example, User creates a Linear Regression Model from parsed dataset.

Parsed dataset has been already distributed in hdfs of experiment environment.

So, you spefity the internal url (internal://) in source_dataset_url parameter.

$ vim sample/json/model_linear.json $ cat sample/json/model_linear.json {   "display_name": "linear-model", "display_description": "Sample LinearRegression Model", "source_dataset_url": "internal://91d98f6d-a065-431b-b8e0-1b996ac85cec", "model_type": "LinearRegression", "model_params": "{'numIterations': 200}", "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286" } $ meteos model-create --json sample/json/model_linear.json +-+--+ +-+--+ +-+--+ $ meteos model-list +--+--+---+--+-+ +--+--+---+--+-+ +--+--+---+--+-+
 * Property   | Value                                |
 * created_at | 2016-12-15T23:09:36.000000           |
 * description | Sample LinearRegression Model       |
 * id         | 3cf02c2f-f043-49e5-a0df-0dc782868312 |
 * name       | linear-model                         |
 * params     | eydudW1JdGVyYXRpb25zJzogMjAwfQ==     |
 * project_id | 4034bb3cd9324776a68c140fdd18baa4     |
 * status     | creating                             |
 * stderr     | None                                 |
 * stdout     | None                                 |
 * type       | LinearRegression                     |
 * user_id    | 64ad04e101df40b4b51e5f370a83412c     |
 * id                                  | name         | status    | type             | source_dataset_url                              |
 * 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | available | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |

6. Predict
Create a learning job predicting a Sales Figures.

Specify the input value as "args" parameter.

Retrieve a predicted data as a stdout of job execution.

$ vim sample/json/learning.json $ cat sample/json/learning.json {   "display_name": "example-learning-job", "display_description": "This is a sample job", "model_id": "3cf02c2f-f043-49e5-a0df-0dc782868312", "method": "predict", "args": "1,7,2016,5,0,60,80" } $ meteos learning-create --json sample/json/learning.json +-+--+ +-+--+ +-+--+ $ meteos learning-list +--+--+---++---+ +--+--+---++---+ +--+--+---++---+
 * Property   | Value                                |
 * args       | MSw3LDIwMTYsNSwwLDYwLDgw             |
 * created_at | 2016-12-15T23:13:29.000000           |
 * description | This is a sample job                |
 * id         | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
 * method     | predict                              |
 * name       | example-learning-job                 |
 * project_id | 4034bb3cd9324776a68c140fdd18baa4     |
 * status     | creating                             |
 * stderr     | None                                 |
 * stdout     | None                                 |
 * user_id    | 64ad04e101df40b4b51e5f370a83412c     |
 * id                                  | name                 | status    | args               | stdout        |
 * ffcd3ae6-8b53-437e-94e2-3aabde741cb0 | example-learning-job | available | 1,7,2016,5,0,60,80 | 4048.18352502 |

7. Online Prediction
You can load a Prediction Model in advance for online prediction by using "meteos-load" command.

In online prediction, user can retrieve a predicted data immediately.

''' Note: Meteos can not load multiple models at the same time. So, you have to unload a model before loading another model. '''

$ meteos model-load 3cf02c2f-f043-49e5-a0df-0dc782868312 $ meteos model-list +--+--++--+-+ +--+--++--+-+ +--+--++--+-+
 * id                                  | name         | status     | type             | source_dataset_url                              |
 * 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | activating | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |

Status is set to "active" when loading is complete.

$ meteos model-list +--+--++--+-+ +--+--++--+-+ +--+--++--+-+
 * id                                  | name         | status | type             | source_dataset_url                              |
 * 3cf02c2f-f043-49e5-a0df-0dc782868312 | linear-model | active | LinearRegression | internal://91d98f6d-a065-431b-b8e0-1b996ac85cec |

You can get a predicted data as a response of REST API.

$ meteos learning-create --json sample/json/learning.json +-+--+ +-+--+ +-+--+
 * Property   | Value                                |
 * args       | MSw3LDIwMTYsNSwwLDYwLDgw             |
 * created_at | 2016-12-15T23:13:29.000000           |
 * description | This is a sample job                |
 * id         | ffcd3ae6-8b53-437e-94e2-3aabde741cb0 |
 * method     | predict                              |
 * name       | example-learning-job                 |
 * project_id | 4034bb3cd9324776a68c140fdd18baa4     |
 * status     | available                            |
 * stderr     | None                                 |
 * stdout     | 4048.18352502                        |
 * user_id    | 64ad04e101df40b4b51e5f370a83412c     |

$ meteos model-unload 3cf02c2f-f043-49e5-a0df-0dc782868312