Jump to: navigation, search

Meteos/ExampleRecommend

< Meteos
Revision as of 23:52, 15 December 2016 by HiroyukiEguchi (talk | contribs) (→‎Recommend a Movie using Meteos)

Recommend a Movie using Meteos

In this example, you create a prediction model which recommend a movie by using Recommendation Model.

Recommendation.png

1. Create a experiment template

Create template of experiment. Experiment is a workspace of Machine Learning.

You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.

You can use a format located in python-meteosclient/sample/json/template.json 

$ glance image-list | grep meteos
| a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos                          |

$ neutron net-list | grep public
| 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public  | 1f979ae3-d6b7-4d03-ba0f-9d9112581783             |

$ vim sample/json/template.json

$ cat sample/json/template.json
{
    "display_name": "example-template",
    "display_description": "This is a sample template of experiment",
    "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
    "master_nodes_num": 1,
    "master_flavor_id": "4",
    "worker_nodes_num": 2,
    "worker_flavor_id": "2",
    "spark_version": "1.6.0",
    "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
}

$ meteos template-create --json sample/json/template.json
+---------------+-----------------------------------------+
| Property      | Value                                   |
+---------------+-----------------------------------------+
| cluster_id    | None                                    |
| created_at    | 2016-12-15T22:55:03.000000              |
| description   | This is a sample template of experiment |
| id            | d3180a78-74cc-432d-9e9d-68640b18abae    |
| master_flavor | 4                                       |
| master_nodes  | 1                                       |
| name          | example-template                        |
| project_id    | 4034bb3cd9324776a68c140fdd18baa4        |
| spark_version | 1.6.0                                   |
| status        | available                               |
| user_id       | 64ad04e101df40b4b51e5f370a83412c        |
| worker_flavor | 2                                       |
| worker_nodes  | 2                                       |
+---------------+-----------------------------------------+

2. Create a experiment from template

Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.

You can use a format located in python-meteosclient/sample/json/experiment.json 

$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem

$ neutron net-list | grep private
| e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |

$ vim sample/json/experiment.json

$ cat sample/json/experiment.json
{
    "display_name": "example-experiment",
    "display_description": "This is a sample experiment",
    "key_name": "key1",
    "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9",
    "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae"
}

$ meteos experiment-create --json sample/json/experiment.json
+--------------------+--------------------------------------+
| Property           | Value                                |
+--------------------+--------------------------------------+
| created_at         | 2016-12-15T22:56:51.000000           |
| description        | This is a sample experiment          |
| id                 | fcc8d055-e801-4652-af8c-5aabedbf0286 |
| key_name           | key1                                 |
| management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 |
| name               | example-experiment                   |
| project_id         | 4034bb3cd9324776a68c140fdd18baa4     |
| status             | creating                             |
| user_id            | 64ad04e101df40b4b51e5f370a83412c     |
+--------------------+--------------------------------------+

Meteos creates a experiment using OpenStack Sahara spark plugin.

You can see a sahara cluster and nova VMs created by Meteos as below. 

$ openstack dataprocessing cluster list (or sahara cluster-list)
+------------------+--------------------------------------+-------------+----------------+----------+
| Name             | Id                                   | Plugin name | Plugin version | Status   |
+------------------+--------------------------------------+-------------+----------------+----------+
| cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark       | 1.6.0          | Spawning |
+------------------+--------------------------------------+-------------+----------------+----------+

$ openstack server list (or nova list)
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
| ID                                   | Name                       | Status | Task State | Power State | Networks         |
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+
| 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0  | ACTIVE | -          | Running     | private=10.0.0.5 |
| 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | -          | Running     | private=10.0.0.7 |
| a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | -          | Running     | private=10.0.0.4 |
+--------------------------------------+----------------------------+--------+------------+-------------+------------------+

3. Upload a raw data

Upload a raw data (in this example movie rating data) to OpenStack Swift.

You can use a sample data located in python-meteosclient/sample/data/recommendation_data.txt

Raw data shows "UserID", "MovieID", "Rating" from left. 

$ cd sample/data/

/sample/data$ head recommendation_data.txt
1,1,4.5
1,2,1.5
1,3,5.0
1,4,2.0
2,1,5.0
2,2,1.0
2,3,4.0
2,4,1.0
3,1,1.5
3,2,4.0

/sample/data$ swift upload meteos recommendation_data.txt
recommendation_data.txt

4. Create a prediction model

In this example, User creates a Recommendation Model from swift directly. 

$ vim sample/json/model_recommendation.json

$ cat sample/json/model_recommendation.json
{
    "display_name": "recommend-movie",
    "display_description": "Sample Model",
    "source_dataset_url": "swift://meteos/recommendation_data.txt",
    "model_type": "Recommendation",
    "model_params": "{'numIterations': 10}",
    "experiment_id": "91504a65-01cf-428f-81aa-596be7ca8619",
    "swift_tenant": "demo",
    "swift_username": "demo",
    "swift_password": "nova"
}

$ meteos model-create --json sample/json/model_recommendation.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-04T07:50:10.000000           |
| description | Sample Model                         |
| id          | 6e0f8633-fac4-46d8-a0ac-e9de00ef8b4b |
| name        | recommend-movie                      |
| params      | eydudW1JdGVyYXRpb25zJzogMTB9         |
| project_id  | 67401cca74c2409b939e944bc6c8fcbe     |
| status      | creating                             |
| stderr      | None                                 |
| stdout      | None                                 |
| type        | Recommendation                       |
| user_id     | 181b1caa9d5b470393ca66b9e511d5b0     |
+-------------+--------------------------------------+

$ meteos model-list
+--------------------------------------+-----------------+--------------+-----------+----------------+----------------------------------------+----------------------------+
| id                                   | name            | description  | status    | type           | source_dataset_url                     | created_at                 |
+--------------------------------------+-----------------+--------------+-----------+----------------+----------------------------------------+----------------------------+
| 6e0f8633-fac4-46d8-a0ac-e9de00ef8b4b | recommend-movie | Sample Model | available | Recommendation | swift://meteos/recommendation_data.txt | 2016-12-04T07:50:10.000000 |
+--------------------------------------+-----------------+--------------+-----------+----------------+----------------------------------------+----------------------------+

5. Predict

Create a learning job predicting a Rating of specified user to determine weather you should recommend this movie to user or not.

Specify the input value as "args" parameter. 

$ vim sample/json/learning.json

$ cat sample/json/learning.json
{
    "display_name": "example-learning-job",
    "display_description": "This is a sample job",
    "model_id": "6e0f8633-fac4-46d8-a0ac-e9de00ef8b4b",
    "method": "predict",
    "args": "5,3"
}

$ meteos learning-create --json sample/json/learning.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| args        | NSwz                                 |
| created_at  | 2016-12-04T08:02:49.000000           |
| description | This is a sample job                 |
| id          | 8b6f17e1-a3cf-4296-9ff2-f98af04c0283 |
| method      | predict                              |
| name        | example-learning-job                 |
| project_id  | 67401cca74c2409b939e944bc6c8fcbe     |
| status      | creating                             |
| stderr      | None                                 |
| stdout      | None                                 |
| user_id     | 181b1caa9d5b470393ca66b9e511d5b0     |
+-------------+--------------------------------------+

Retrieve a predicted data as a stdout of job execution. 

$ meteos learning-list
+--------------------------------------+----------------------+----------------------+-----------+------+---------------+------------+
| id                                   | name                 | description          | status    | args | stdout        | created_at |
+--------------------------------------+----------------------+----------------------+-----------+------+---------------+------------+
| 8b6f17e1-a3cf-4296-9ff2-f98af04c0283 | example-learning-job | This is a sample job | available | 5,3  | 1.47852627868 |            |
+--------------------------------------+----------------------+----------------------+-----------+------+---------------+------------+
Retrieved from "https://wiki.openstack.org/w/index.php?title=Meteos/ExampleRecommend&oldid=145518"