Jump to: navigation, search

Difference between revisions of "Meteos/ExampleWord2Vec"

(→‎3. Upload a raw data)
(→‎4. Create a prediction model)
Line 167: Line 167:
 
+-------------+--------------------------------------+
 
+-------------+--------------------------------------+
 
| created_at  | 2016-12-13T07:35:50.000000          |
 
| created_at  | 2016-12-13T07:35:50.000000          |
−
| description | Word2Vec Model               |
+
| description | Word2Vec Model                       |
 
| id          | bb0effd9-a060-4a09-b11a-37806aab479c |
 
| id          | bb0effd9-a060-4a09-b11a-37806aab479c |
−
| name        | word2vec-model               |
+
| name        | word2vec-model                       |
 
| params      | eydudW1JdGVyYXRpb25zJzogMTAwfQ==    |
 
| params      | eydudW1JdGVyYXRpb25zJzogMTAwfQ==    |
 
| project_id  | b09a03e3aeb9435c854870b8582eb1ae    |
 
| project_id  | b09a03e3aeb9435c854870b8582eb1ae    |
Line 186: Line 186:
 
| bb0effd9-a060-4a09-b11a-37806aab479c | word2vec-model | Word2Vec Model | creating | Word2Vec | swift://meteos/openstack-ops-manual.txt | 2016-12-13T07:35:50.000000 |
 
| bb0effd9-a060-4a09-b11a-37806aab479c | word2vec-model | Word2Vec Model | creating | Word2Vec | swift://meteos/openstack-ops-manual.txt | 2016-12-13T07:35:50.000000 |
 
+--------------------------------------+----------------+----------------+----------+----------+-----------------------------------------+----------------------------+
 
+--------------------------------------+----------------+----------------+----------+----------+-----------------------------------------+----------------------------+
 +
</pre>
 +
 +
=== 5. Create a learning job ===
 +
 +
Create learning job and retrieve synonyms and score.
 +
 +
You can specify a target word as 'word' parameter and a number of synonyms which you want to retrieve as 'num' parameter.
 +
 +
Retrieve a output data as a stdout of job execution.
 +
 +
In this example, you retrieve 'glance' and 'cinder' as a synonyms of 'keystone'.
 +
 +
<pre>
 +
$ vim sample/json/learning.json
 +
</pre>
 +
<pre>
 +
$ cat sample/json/learning.json
 +
{
 +
    "display_name": "word2vec-job",
 +
    "display_description": "synonyms of keystone",
 +
    "model_id": "bb0effd9-a060-4a09-b11a-37806aab479c",
 +
    "method": "predict",
 +
    "args": "{'word':'keystone', 'num':5}"
 +
}
 +
</pre>
 +
<pre>
 +
$ meteos learning-create --json sample/json/learning.json
 +
+-------------+------------------------------------------+
 +
| Property    | Value                                    |
 +
+-------------+------------------------------------------+
 +
| args        | eyd3b3JkJzona2V5c3RvbmUnLCAnbnVtJzo1fQ== |
 +
| created_at  | 2016-12-13T07:43:56.000000              |
 +
| description | synonyms job                            |
 +
| id          | b0f81056-75f7-4c03-86ce-f5e2d11ec284    |
 +
| method      | predict                                  |
 +
| name        | word2vec-job                            |
 +
| project_id  | b09a03e3aeb9435c854870b8582eb1ae        |
 +
| status      | creating                                |
 +
| stderr      | None                                    |
 +
| stdout      | None                                    |
 +
| user_id    | 796f4000c6e84df9964b178ef55604f7        |
 +
+-------------+------------------------------------------+
 +
</pre>
 +
<pre>
 +
$ meteos learning-list
 +
+--------------------------------------+--------------+--------------+-----------+------------------------------+-----------------------+----------------------------+
 +
| id                                  | name        | description  | status    | args                        | stdout                | created_at                |
 +
+--------------------------------------+--------------+----------------------+-----------+------------------------------+-----------------------+----------------------------+
 +
| b0f81056-75f7-4c03-86ce-f5e2d11ec284 | word2vec-job | synonyms job | available | {'word':'keystone', 'num':5} | -e: 2.94760457709    | 2016-12-13T07:43:56.000000 |
 +
|                                      |              |              |          |                              | \: 2.52389718982      |                            |
 +
|                                      |              |              |          |                              | #: 2.40766043461      |                            |
 +
|                                      |              |              |          |                              | glance: 2.37679284381 |                            |
 +
|                                      |              |              |          |                              | cinder: 2.29982113674 |                            |
 +
+--------------------------------------+--------------+--------------+-----------+------------------------------+-----------------------+----------------------------+
 
</pre>
 
</pre>

Revision as of 08:00, 13 December 2016

Search Synonyms by using Meteos

In this example, you create a prediction model which search synonyms by using Word2Vec Model.

Word2Vec.png

1. Create a experiment template

Create template of experiment. Experiment is a workspace of Machine Learning.

You have to confirm a glance image id of meteos image, and a neutron network id before creating a template.

You can use a format located in python-meteosclient/sample/json/template.json 

$ glance image-list | grep meteos
| 45de4bbd-8419-40ff-8ed7-fc065c05e34f | meteos                          |

$ neutron net-list | grep public
| 84c13e76-ced9-4142-a885-280784f1f7a3 | public  | a14de1c5-b8d4-434b-a056-9b0049b93402             |

$ vim sample/json/template.json

$ cat sample/json/template.json
{
    "display_name": "example-template",
    "display_description": "This is a sample template of experiment",
    "image_id" : "45de4bbd-8419-40ff-8ed7-fc065c05e34f",
    "master_nodes_num": 1,
    "master_flavor_id": "4",
    "worker_nodes_num": 2,
    "worker_flavor_id": "2",
    "spark_version": "1.6.0",
    "floating_ip_pool": "84c13e76-ced9-4142-a885-280784f1f7a3"
}

$ meteos template-create --json sample/json/template.json
+---------------+-----------------------------------------+
| Property      | Value                                   |
+---------------+-----------------------------------------+
| cluster_id    | None                                    |
| created_at    | 2016-12-04T07:16:29.000000              |
| description   | This is a sample template of experiment |
| id            | 8b7b9b89-f119-4b9b-b9b0-31598f819f1a    |
| master_flavor | 4                                       |
| master_nodes  | 1                                       |
| name          | example-template                        |
| project_id    | 67401cca74c2409b939e944bc6c8fcbe        |
| spark_version | 1.6.0                                   |
| status        | available                               |
| user_id       | 181b1caa9d5b470393ca66b9e511d5b0        |
| worker_flavor | 2                                       |
| worker_nodes  | 2                                       |
+---------------+-----------------------------------------+


2. Create a experiment from template

Create a experiment by using template created in the above step. You have to confirm a neutron private network id and create keypair before creating a template.

You can use a format located in python-meteosclient/sample/json/experiment.json 

$ nova keypair-add key1 > ~/key1.pem && chmod 600 ~/key1.pem

$ neutron net-list | grep private
| 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 | private | cb58940f-859b-48c6-b92a-3861470f1fc1 20.0.0.0/26 |

$ vim sample/json/experiment.json

$ cat sample/json/experiment.json
{
    "display_name": "example-experiment",
    "display_description": "This is a sample experiment",
    "key_name": "key1",
    "neutron_management_network": "8abc626e-2b06-4c67-9b2c-0231f0cef5b8",
    "template_id": "8b7b9b89-f119-4b9b-b9b0-31598f819f1a"
}

$ meteos experiment-create --json sample/json/experiment.json
+--------------------+--------------------------------------+
| Property           | Value                                |
+--------------------+--------------------------------------+
| created_at         | 2016-12-04T07:20:11.000000           |
| description        | This is a sample experiment          |
| id                 | 91504a65-01cf-428f-81aa-596be7ca8619 |
| key_name           | key1                                 |
| management_network | 8abc626e-2b06-4c67-9b2c-0231f0cef5b8 |
| name               | example-experiment                   |
| project_id         | 67401cca74c2409b939e944bc6c8fcbe     |
| status             | creating                             |
| user_id            | 181b1caa9d5b470393ca66b9e511d5b0     |
+--------------------+--------------------------------------+

Meteos creates a experiment using OpenStack Sahara spark plugin.

You can see a sahara cluster and nova VMs created by Meteos as below. 

$ openstack dataprocessing cluster list (or sahara cluster-list)
+------------------+--------------------------------------+-------------+----------------+----------+
| Name             | Id                                   | Plugin name | Plugin version | Status   |
+------------------+--------------------------------------+-------------+----------------+----------+
| cluster-91504a65 | 13418fd9-5d2a-4ee6-b384-cb250b7e7714 | spark       | 1.6.0          | Spawning |
+------------------+--------------------------------------+-------------+----------------+----------+

$ openstack server list (or nova list)
+--------------------------------------+----------------------------+--------+----------+------------+
| ID                                   | Name                       | Status | Networks | Image Name |
+--------------------------------------+----------------------------+--------+----------+------------+
| 58818eb5-ade7-407c-8c76-9fd9809632b4 | cluster-91504a65-workers-1 | BUILD  |          | meteos     |
| a151dbd9-de51-43ca-afb8-1fdeecce2891 | cluster-91504a65-workers-0 | BUILD  |          | meteos     |
| d02d85c5-0960-4b7e-880c-26b73c5dd8ad | cluster-91504a65-master-0  | BUILD  |          | meteos     |
+--------------------------------------+----------------------------+--------+----------+------------+

3. Upload a raw data

Upload a raw data (in this example, OpenStack Operations Guide) to OpenStack Swift. 

$ sudo apt-get install poppler-utils
$ wget http://docs.openstack.org/openstack-ops/openstack-ops-manual.pdf
$ pdftotext openstack-ops-manual.pdf openstack-ops-manual.txt
$ swift upload meteos openstack-ops-manual.txt
openstack-ops-manual.txt

4. Create a prediction model

In this example, User creates a Word2Vec Model from dataset. 

$ vim sample/json/model_word2vec.json

$ cat sample/json/model_word2vec.json
{
    "display_name": "word2vec-model",
    "display_description": "Word2Vec Model",
    "source_dataset_url": "swift://meteos/openstack-ops-manual.txt",
    "model_type": "Word2Vec",
    "model_params": "{'numIterations': 100}",
    "experiment_id": "846d0f95-2e89-43b7-8d62-e61905b7af6a",
    "swift_tenant": "demo",
    "swift_username": "demo",
    "swift_password": "nova"
}

$ meteos model-create --json sample/json/model_word2vec.json
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2016-12-13T07:35:50.000000           |
| description | Word2Vec Model                       |
| id          | bb0effd9-a060-4a09-b11a-37806aab479c |
| name        | word2vec-model                       |
| params      | eydudW1JdGVyYXRpb25zJzogMTAwfQ==     |
| project_id  | b09a03e3aeb9435c854870b8582eb1ae     |
| status      | creating                             |
| stderr      | None                                 |
| stdout      | None                                 |
| type        | Word2Vec                             |
| user_id     | 796f4000c6e84df9964b178ef55604f7     |
+-------------+--------------------------------------+

$ meteos model-list
+--------------------------------------+----------------+----------------+----------+----------+-----------------------------------------+----------------------------+
| id                                   | name           | description    | status   | type     | source_dataset_url                      | created_at                 |
+--------------------------------------+----------------+----------------+----------+----------+-----------------------------------------+----------------------------+
| bb0effd9-a060-4a09-b11a-37806aab479c | word2vec-model | Word2Vec Model | creating | Word2Vec | swift://meteos/openstack-ops-manual.txt | 2016-12-13T07:35:50.000000 |
+--------------------------------------+----------------+----------------+----------+----------+-----------------------------------------+----------------------------+

5. Create a learning job

Create learning job and retrieve synonyms and score.

You can specify a target word as 'word' parameter and a number of synonyms which you want to retrieve as 'num' parameter.

Retrieve a output data as a stdout of job execution.

In this example, you retrieve 'glance' and 'cinder' as a synonyms of 'keystone'. 

$ vim sample/json/learning.json

$ cat sample/json/learning.json
{
    "display_name": "word2vec-job",
    "display_description": "synonyms of keystone",
    "model_id": "bb0effd9-a060-4a09-b11a-37806aab479c",
    "method": "predict",
    "args": "{'word':'keystone', 'num':5}"
}

$ meteos learning-create --json sample/json/learning.json
+-------------+------------------------------------------+
| Property    | Value                                    |
+-------------+------------------------------------------+
| args        | eyd3b3JkJzona2V5c3RvbmUnLCAnbnVtJzo1fQ== |
| created_at  | 2016-12-13T07:43:56.000000               |
| description | synonyms job                             |
| id          | b0f81056-75f7-4c03-86ce-f5e2d11ec284     |
| method      | predict                                  |
| name        | word2vec-job                             |
| project_id  | b09a03e3aeb9435c854870b8582eb1ae         |
| status      | creating                                 |
| stderr      | None                                     |
| stdout      | None                                     |
| user_id     | 796f4000c6e84df9964b178ef55604f7         |
+-------------+------------------------------------------+

$ meteos learning-list
+--------------------------------------+--------------+--------------+-----------+------------------------------+-----------------------+----------------------------+
| id                                   | name         | description  | status    | args                         | stdout                | created_at                 |
+--------------------------------------+--------------+----------------------+-----------+------------------------------+-----------------------+----------------------------+
| b0f81056-75f7-4c03-86ce-f5e2d11ec284 | word2vec-job | synonyms job | available | {'word':'keystone', 'num':5} | -e: 2.94760457709     | 2016-12-13T07:43:56.000000 |
|                                      |              |              |           |                              | \: 2.52389718982      |                            |
|                                      |              |              |           |                              | #: 2.40766043461      |                            |
|                                      |              |              |           |                              | glance: 2.37679284381 |                            |
|                                      |              |              |           |                              | cinder: 2.29982113674 |                            |
+--------------------------------------+--------------+--------------+-----------+------------------------------+-----------------------+----------------------------+
Retrieved from "https://wiki.openstack.org/w/index.php?title=Meteos/ExampleWord2Vec&oldid=145012"