Meteos/ExampleWord2Vec
< Meteos
Revision as of 06:53, 10 January 2017 by HiroyukiEguchi (talk | contribs) (→5. Create a learning job)
Contents
Search Synonyms by using Meteos
In this example, you create a prediction model which search synonyms by using Word2Vec Model.
1. Create a experiment template
Create template of experiment. Experiment is a workspace of Machine Learning.
You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.
You can use a format located in python-meteosclient/sample/json/template.json
$ glance image-list | grep meteos | a6b7de0b-10ff-462c-9c86-25c8a5622a57 | meteos |
$ neutron net-list | grep public | 4222b557-6d9f-405c-b1ff-0f454d2f35bf | public | 1f979ae3-d6b7-4d03-ba0f-9d9112581783 |
$ vim sample/json/template.json
$ cat sample/json/template.json
{
"display_name": "example-template",
"display_description": "This is a sample template of experiment",
"image_id" : "a6b7de0b-10ff-462c-9c86-25c8a5622a57",
"master_nodes_num": 1,
"master_flavor_id": "4",
"worker_nodes_num": 2,
"worker_flavor_id": "2",
"spark_version": "1.6.0",
"floating_ip_pool": "4222b557-6d9f-405c-b1ff-0f454d2f35bf"
}
$ meteos template-create --json sample/json/template.json +---------------+-----------------------------------------+ | Property | Value | +---------------+-----------------------------------------+ | cluster_id | None | | created_at | 2016-12-15T22:55:03.000000 | | description | This is a sample template of experiment | | id | d3180a78-74cc-432d-9e9d-68640b18abae | | master_flavor | 4 | | master_nodes | 1 | | name | example-template | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | spark_version | 1.6.0 | | status | available | | user_id | 64ad04e101df40b4b51e5f370a83412c | | worker_flavor | 2 | | worker_nodes | 2 | +---------------+-----------------------------------------+
2. Create a experiment from template
Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.
You can use a format located in python-meteosclient/sample/json/experiment.json
$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem
$ neutron net-list | grep private | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | private | e15c24a5-dfdd-4428-b27d-9827b35600c0 10.0.0.0/26 |
$ vim sample/json/experiment.json
$ cat sample/json/experiment.json
{
"display_name": "example-experiment",
"display_description": "This is a sample experiment",
"key_name": "key1",
"neutron_management_network": "e4fcc49c-48e5-48f8-9599-bb5eba1339c9",
"template_id": "d3180a78-74cc-432d-9e9d-68640b18abae"
}
$ meteos experiment-create --json sample/json/experiment.json +--------------------+--------------------------------------+ | Property | Value | +--------------------+--------------------------------------+ | created_at | 2016-12-15T22:56:51.000000 | | description | This is a sample experiment | | id | fcc8d055-e801-4652-af8c-5aabedbf0286 | | key_name | key1 | | management_network | e4fcc49c-48e5-48f8-9599-bb5eba1339c9 | | name | example-experiment | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | user_id | 64ad04e101df40b4b51e5f370a83412c | +--------------------+--------------------------------------+
Meteos creates a experiment using OpenStack Sahara spark plugin.
You can see a sahara cluster and nova VMs created by Meteos as below.
$ openstack dataprocessing cluster list (or sahara cluster-list) +------------------+--------------------------------------+-------------+----------------+----------+ | Name | Id | Plugin name | Plugin version | Status | +------------------+--------------------------------------+-------------+----------------+----------+ | cluster-fcc8d055 | 5736d157-ac7c-41de-8aca-78f7afa7e99c | spark | 1.6.0 | Spawning | +------------------+--------------------------------------+-------------+----------------+----------+
$ openstack server list (or nova list) +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+ | 48a9f429-7756-4bed-8dd6-6dc6140ef897 | cluster-fcc8d055-master-0 | ACTIVE | - | Running | private=10.0.0.5 | | 88ff2070-dfe8-45da-aa5c-02ac3e9de3b8 | cluster-fcc8d055-workers-0 | ACTIVE | - | Running | private=10.0.0.7 | | a57dfa5d-8b55-47c7-aae7-d5b3c8779787 | cluster-fcc8d055-workers-1 | ACTIVE | - | Running | private=10.0.0.4 | +--------------------------------------+----------------------------+--------+------------+-------------+------------------+
3. Upload a raw data
Upload a raw data (in this example, OpenStack Operations Guide) to OpenStack Swift.
$ sudo apt-get install poppler-utils $ wget http://docs.openstack.org/openstack-ops/openstack-ops-manual.pdf $ pdftotext openstack-ops-manual.pdf openstack-ops-manual.txt $ swift upload meteos openstack-ops-manual.txt openstack-ops-manual.txt
4. Create a prediction model
In this example, User creates a Word2Vec Model from dataset.
$ vim sample/json/model_word2vec.json
$ cat sample/json/model_word2vec.json
{
"display_name": "word2vec-model",
"display_description": "Sample Word2Vec Model",
"source_dataset_url": "swift://meteos/openstack-ops-manual.txt",
"model_type": "Word2Vec",
"model_params": "{'numIterations': 200}",
"experiment_id": "fcc8d055-e801-4652-af8c-5aabedbf0286",
"swift_tenant": "demo",
"swift_username": "demo",
"swift_password": "nova"
}
$ meteos model-create --json sample/json/model_word2vec.json +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2016-12-16T00:19:37.000000 | | description | Sample Word2Vec Model | | id | 86b7f4df-348d-4cea-88ea-dadabdadaad7 | | name | word2vec-model | | params | eydudW1JdGVyYXRpb25zJzogMjAwfQ== | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | stderr | None | | stdout | None | | type | Word2Vec | | user_id | 64ad04e101df40b4b51e5f370a83412c | +-------------+--------------------------------------+
$ meteos model-list +--------------------------------------+----------------+-----------+----------+-----------------------------------------+ | id | name | status | type | source_dataset_url | +--------------------------------------+----------------+-----------+----------+-----------------------------------------+ | 86b7f4df-348d-4cea-88ea-dadabdadaad7 | word2vec-model | available | Word2Vec | swift://meteos/openstack-ops-manual.txt | +--------------------------------------+----------------+-----------+----------+-----------------------------------------+
5. Create a learning job
Create learning job and retrieve synonyms and score.
You can specify a target word as 'word' parameter and a number of synonyms which you want to retrieve as 'num' parameter.
Retrieve a output data as a stdout of job execution.
In this example, you retrieve 'glance' and 'cinder' as a synonyms of 'keystone'.
$ vim sample/json/learning.json
$ cat sample/json/learning.json
{
"display_name": "example-learning-job",
"display_description": "This is a sample job",
"model_id": "86b7f4df-348d-4cea-88ea-dadabdadaad7",
"method": "predict",
"args": "{'word':'keystone', 'num':5}"
}
$ meteos learning-create --json sample/json/learning.json +-------------+------------------------------------------+ | Property | Value | +-------------+------------------------------------------+ | args | eyd3b3JkJzona2V5c3RvbmUnLCAnbnVtJzo1fQ== | | created_at | 2016-12-16T00:30:48.000000 | | description | This is a sample job | | id | 1bbde56c-d069-4660-ab58-d02c7c55f14a | | method | predict | | name | example-learning-job | | project_id | 4034bb3cd9324776a68c140fdd18baa4 | | status | creating | | stderr | None | | stdout | None | | user_id | 64ad04e101df40b4b51e5f370a83412c | +-------------+------------------------------------------+
$ meteos learning-list
+--------------------------------------+----------------------+-----------+------------------------------+---------------------------+
| id | name | status | args | stdout |
+--------------------------------------+----------------------+-----------+------------------------------+---------------------------+
| 1bbde56c-d069-4660-ab58-d02c7c55f14a | example-learning-job | available | {'word':'keystone', 'num':5} | -e: 2.5969709909 |
| | | | | glance: 2.35303451027 |
| | | | | glance-api: 2.34089976854 |
| | | | | screen: 2.32107789493 |
| | | | | cinder: 2.30745373084 |
+--------------------------------------+----------------------+-----------+------------------------------+---------------------------+
6. Online Prediction
You can load a Prediction Model in advance for online prediction by using "meteos-load" command.
In online prediction, user can retrieve a predicted data immediately.
For more details, please see here.
