Difference between revisions of "Meteos/DatasetsandModels"
(→DataSet Format) |
(→Meteos Prediction Model) |
||
| Line 89: | Line 89: | ||
==== Supervised learning ==== | ==== Supervised learning ==== | ||
| − | [http://spark.apache.org/docs/1.6.0/mllib | + | ===== [http://spark.apache.org/docs/1.6.0/api/python/pyspark.mllib.html#module-pyspark.mllib.Classification Classification] ===== |
| − | + | Logistic Regression Model | |
| − | [http://spark.apache.org/docs/1.6.0/mllib- | + | Naive Bayes Model |
| + | |||
| + | ===== [http://spark.apache.org/docs/1.6.0/api/python/pyspark.mllib.html#module-pyspark.mllib.regression Regression] ===== | ||
| + | |||
| + | Linear Regression Model | ||
| + | |||
| + | Ridge Regression Model | ||
| + | |||
| + | ===== [http://spark.apache.org/docs/1.6.0/api/python/pyspark.mllib.html#module-pyspark.mllib.tree Tree] ===== | ||
| + | |||
| + | Decision Tree Classification Model | ||
| + | |||
| + | Decision Tree Regression Model | ||
| + | |||
| + | Random Forest Classification Model | ||
| + | |||
| + | Random Forest Regression Model | ||
==== Unsupervised learning ==== | ==== Unsupervised learning ==== | ||
| − | [http://spark.apache.org/docs/1.6.0/mllib-clustering.html#k-means Kmeans Model] | + | ===== [http://spark.apache.org/docs/1.6.0/mllib-clustering.html#k-means Kmeans Model] ===== |
| − | [http://spark.apache.org/docs/1.6.0/mllib-collaborative-filtering.html#collaborative-filtering Recommendation Model] | + | ===== [http://spark.apache.org/docs/1.6.0/mllib-collaborative-filtering.html#collaborative-filtering Recommendation Model] ===== |
| − | [http://spark.apache.org/docs/1.6.0/ml-features.html#word2vec Word2Vec Model] | + | ===== [http://spark.apache.org/docs/1.6.0/ml-features.html#word2vec Word2Vec Model] ===== |
| − | [http://spark.apache.org/docs/1.6.0/mllib-frequent-pattern-mining.html FP-growth Model] | + | ===== [http://spark.apache.org/docs/1.6.0/mllib-frequent-pattern-mining.html FP-growth Model] ===== |
=== ML === | === ML === | ||
Not Supported. | Not Supported. | ||
Revision as of 05:39, 2 March 2017
Meteos Dataset
Dataset is a data to create a prediction model.
DataSet Format
Meteos currently supports following data format.
Supported dataset format differs depending on prediction models.
| Model/Dataset Format | CSV data format | LibSVM data format | Text data format |
|---|---|---|---|
| LinearRegression Model | YES | YES | NO |
| LogisticRegression Model | YES | YES | NO |
| DecisionTree Model | YES | YES | NO |
| Kmeans Model | YES | NO | NO |
| Recommendation Model | YES | NO | NO |
| Word2Vec Model | NO | NO | YES |
| FP-growth Model | NO | NO | YES |
Most of models support cvs dataset format, however libsvm dataset format is only supported by LinearRegression Model, LogisticRegression Model and DecisionTree Mode.
CSV data format
<label>,<value1>,<value2>, ... <valueN>
LibSVM data format
<label> <index1>:<value1> <index2>:<value2> ... <indexN>:<valueN>
Text data format is only supported Word2Vec and FP-growth Model. Any other Model can not handle text data.
Text data format
This is a text data format
DataSet URL
When creating a prediction model, user specify a "source_dataset_url" parameter which show the place where dataset is located.
A Source_dataset_url has two url types as follows:
Swift URL
Swift URL is used when creating a model from dataset in swift.
If it is no neccesary to parse a dataset, user can create a model from dataset in swift directly by specify source_data_url as below.
swift://<container_name>/<object_name>
Internal HDFS URL
Internal HDFS URL is used when creating a model from internal hdfs of meteos experiment.
"Internal" is meaning that dataset has been already downloaded or parsed by Meteos.
When creating a model from dataset in hdfs, user have to specify a url as below.
internal://<dataset_id>
Meteos Prediction Model
Currently Meteos supports these following prediction models of Apache Spark.
Apache Spark has two machine learning libraries (MLlib and Ml).
MLlib and ML has multiple prediction models by data mining and machine learning algorithms.
MLlib
Supervised learning
Classification
Logistic Regression Model
Naive Bayes Model
Regression
Linear Regression Model
Ridge Regression Model
Tree
Decision Tree Classification Model
Decision Tree Regression Model
Random Forest Classification Model
Random Forest Regression Model
Unsupervised learning
Kmeans Model
Recommendation Model
Word2Vec Model
FP-growth Model
ML
Not Supported.
