Difference between revisions of "Meteos/ExampleRecommend"
(→Recommend a Movie using Meteos) |
(→4. Create a prediction model) |
||
Line 166: | Line 166: | ||
$ cat sample/json/model_recommendation.json | $ cat sample/json/model_recommendation.json | ||
{ | { | ||
− | "display_name": " | + | "display_name": "recommendation-model", |
− | "display_description": "Sample Model", | + | "display_description": "Sample Recommendation Model", |
"source_dataset_url": "swift://meteos/recommendation_data.txt", | "source_dataset_url": "swift://meteos/recommendation_data.txt", | ||
"model_type": "Recommendation", | "model_type": "Recommendation", | ||
"model_params": "{'numIterations': 10}", | "model_params": "{'numIterations': 10}", | ||
− | "experiment_id": " | + | "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286", |
"swift_tenant": "demo", | "swift_tenant": "demo", | ||
"swift_username": "demo", | "swift_username": "demo", | ||
Line 182: | Line 182: | ||
| Property | Value | | | Property | Value | | ||
+-------------+--------------------------------------+ | +-------------+--------------------------------------+ | ||
− | | created_at | 2016-12- | + | | created_at | 2016-12-15T23:56:47.000000 | |
− | | description | Sample Model | + | | description | Sample Recommendation Model | |
− | | id | | + | | id | f2c119a6-1a61-4727-9d41-87bb3971d6a3 | |
− | | name | | + | | name | recommendation-model | |
| params | eydudW1JdGVyYXRpb25zJzogMTB9 | | | params | eydudW1JdGVyYXRpb25zJzogMTB9 | | ||
− | | project_id | | + | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | |
| status | creating | | | status | creating | | ||
| stderr | None | | | stderr | None | | ||
| stdout | None | | | stdout | None | | ||
| type | Recommendation | | | type | Recommendation | | ||
− | | user_id | | + | | user_id | 64ad04e101df40b4b51e5f370a83412c | |
+-------------+--------------------------------------+ | +-------------+--------------------------------------+ | ||
</pre> | </pre> | ||
<pre> | <pre> | ||
$ meteos model-list | $ meteos model-list | ||
− | +--------------------------------------+-------- | + | +--------------------------------------+----------------------+-----------+----------------+----------------------------------------+ |
− | | id | name | + | | id | name | status | type | source_dataset_url | |
− | +--------------------------------------+-------- | + | +--------------------------------------+----------------------+-----------+----------------+----------------------------------------+ |
− | | | + | | f2c119a6-1a61-4727-9d41-87bb3971d6a3 | recommendation-model | available | Recommendation | swift://meteos/recommendation_data.txt | |
− | +--------------------------------------+-------- | + | +--------------------------------------+----------------------+-----------+----------------+----------------------------------------+ |
</pre> | </pre> | ||
Revision as of 23:58, 15 December 2016
Contents
Recommend a Movie using Meteos
In this example, you create a prediction model which recommend a movie by using Recommendation Model.
1. Create a experiment template
Create template of experiment. Experiment is a workspace of Machine Learning.
You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.
You can use a format located in python-meteosclient/sample/json/template.json
$ glance image-list | grep meteos | a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos |
$ neutron net-list | grep public | 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public | 1f979ae3-d6b7-4d03-ba0f-9d9112581783 |
$ vim sample/json/template.json
$ cat sample/json/template.json { "display_name": "example-template", "display_description": "This is a sample template of experiment", "image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57", "master_nodes_num": 1, "master_flavor_id": "4", "worker_nodes_num": 2, "worker_flavor_id": "2", "spark_version": "1.6.0", "floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf" }
$ meteos template-create --json sample/json/template.json +---------------+-----------------------------------------+ | Property | Value | +---------------+-----------------------------------------+ | cluster_id | None | | created_at | 2016-12-15T22:55:03.000000 | | description | This is a sample template of experiment | | id | d3180a78-74cc-432d-9e9d-68640b18abae | | master_flavor | 4 | | master_nodes | 1 | | name | example-template | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | spark_version | 1.6.0 | | status | available | | user_id | 64ad04e101df40b4b51e5f370a83412c | | worker_flavor | 2 | | worker_nodes | 2 | +---------------+-----------------------------------------+
2. Create a experiment from template
Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.
You can use a format located in python-meteosclient/sample/json/experiment.json
$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem
$ neutron net-list | grep private | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |
$ vim sample/json/experiment.json
$ cat sample/json/experiment.json { "display_name": "example-experiment", "display_description": "This is a sample experiment", "key_name": "key1", "neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9", "template_id": "d3180a78-74cc-432d-9e9d-68640b18abae" }
$ meteos experiment-create --json sample/json/experiment.json +--------------------+--------------------------------------+ | Property | Value | +--------------------+--------------------------------------+ | created_at | 2016-12-15T22:56:51.000000 | | description | This is a sample experiment | | id | fcc8d055-e801-4652-af8c-5aabedbf0286 | | key_name | key1 | | management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | | name | example-experiment | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | user_id | 64ad04e101df40b4b51e5f370a83412c | +--------------------+--------------------------------------+
Meteos creates a experiment using OpenStack Sahara spark plugin.
You can see a sahara cluster and nova VMs created by Meteos as below.
$ openstack dataprocessing cluster list (or sahara cluster-list) +------------------+--------------------------------------+-------------+----------------+----------+ | Name | Id | Plugin name | Plugin version | Status | +------------------+--------------------------------------+-------------+----------------+----------+ | cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark | 1.6.0 | Spawning | +------------------+--------------------------------------+-------------+----------------+----------+
$ openstack server list (or nova list) +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ | 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0 | ACTIVE | - | Running | private=10.0.0.5 | | 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | - | Running | private=10.0.0.7 | | a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | - | Running | private=10.0.0.4 | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+
3. Upload a raw data
Upload a raw data (in this example movie rating data) to OpenStack Swift.
You can use a sample data located in python-meteosclient/sample/data/recommendation_data.txt
Raw data shows "UserID", "MovieID", "Rating" from left.
$ cd sample/data/
/sample/data$ head recommendation_data.txt 1,1,4.5 1,2,1.5 1,3,5.0 1,4,2.0 2,1,5.0 2,2,1.0 2,3,4.0 2,4,1.0 3,1,1.5 3,2,4.0
/sample/data$ swift upload meteos recommendation_data.txt recommendation_data.txt
4. Create a prediction model
In this example, User creates a Recommendation Model from swift directly.
$ vim sample/json/model_recommendation.json
$ cat sample/json/model_recommendation.json { "display_name": "recommendation-model", "display_description": "Sample Recommendation Model", "source_dataset_url": "swift://meteos/recommendation_data.txt", "model_type": "Recommendation", "model_params": "{'numIterations': 10}", "experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286", "swift_tenant": "demo", "swift_username": "demo", "swift_password": "nova" }
$ meteos model-create --json sample/json/model_recommendation.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-12-15T23:56:47.000000 | | description | Sample Recommendation Model | | id | f2c119a6-1a61-4727-9d41-87bb3971d6a3 | | name | recommendation-model | | params | eydudW1JdGVyYXRpb25zJzogMTB9 | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | stderr | None | | stdout | None | | type | Recommendation | | user_id | 64ad04e101df40b4b51e5f370a83412c | +-------------+--------------------------------------+
$ meteos model-list +--------------------------------------+----------------------+-----------+----------------+----------------------------------------+ | id | name | status | type | source_dataset_url | +--------------------------------------+----------------------+-----------+----------------+----------------------------------------+ | f2c119a6-1a61-4727-9d41-87bb3971d6a3 | recommendation-model | available | Recommendation | swift://meteos/recommendation_data.txt | +--------------------------------------+----------------------+-----------+----------------+----------------------------------------+
5. Predict
Create a learning job predicting a Rating of specified user to determine weather you should recommend this movie to user or not.
Specify the input value as "args" parameter.
$ vim sample/json/learning.json
$ cat sample/json/learning.json { "display_name": "example-learning-job", "display_description": "This is a sample job", "model_id": "6e0f8633-fac4-46d8-a0ac-e9de00ef8b4b", "method": "predict", "args": "5,3" }
$ meteos learning-create --json sample/json/learning.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | args | NSwz | | created_at | 2016-12-04T08:02:49.000000 | | description | This is a sample job | | id | 8b6f17e1-a3cf-4296-9ff2-f98af04c0283 | | method | predict | | name | example-learning-job | | project_id | 67401cca74c2409b939e944bc6c8fcbe | | status | creating | | stderr | None | | stdout | None | | user_id | 181b1caa9d5b470393ca66b9e511d5b0 | +-------------+--------------------------------------+
Retrieve a predicted data as a stdout of job execution.
$ meteos learning-list +--------------------------------------+----------------------+----------------------+-----------+------+---------------+------------+ | id | name | description | status | args | stdout | created_at | +--------------------------------------+----------------------+----------------------+-----------+------+---------------+------------+ | 8b6f17e1-a3cf-4296-9ff2-f98af04c0283 | example-learning-job | This is a sample job | available | 5,3 | 1.47852627868 | | +--------------------------------------+----------------------+----------------------+-----------+------+---------------+------------+