Jump to: navigation, search

Difference between revisions of "Stackalytics/HowToRun"

(Prerequistics)
m (Installation)
 
(22 intermediate revisions by 7 users not shown)
Line 4: Line 4:
 
  * Python-pip Python Package Index
 
  * Python-pip Python Package Index
 
  * libpython2.7-dev - development C headers for Python
 
  * libpython2.7-dev - development C headers for Python
  * python-lazr.restful, python-lazr.restfulclient - library required for communication with LaunchPad
+
  * libssl-dev - SSL development libraries, header files and documentation
 
  * Git - used to retrieve project repos and get their log  
 
  * Git - used to retrieve project repos and get their log  
 
  * Memcached - used to store commits
 
  * Memcached - used to store commits
* Mongo - used to store persistent data, like list of repos, engineers, configs
 
 
  * Nginx - scalable web server
 
  * Nginx - scalable web server
 
  * Uwsgi, uwsgi-plugin-python - application container for Nginx and plugin for Python
 
  * Uwsgi, uwsgi-plugin-python - application container for Nginx and plugin for Python
Line 18: Line 17:
  
 
1. Install necessary applications
 
1. Install necessary applications
  sudo apt-get install libpython2.7-dev python-pip python-lazr.restful python-lazr.restfulclient git mongodb memcached nginx uwsgi uwsgi-plugin-python
 
  
2. Create folders needed for Stackalytics
+
Particular command depends on linux distribution. Here is example for Debian based system:
 +
  sudo apt-get install libpython2.7-dev libssl-dev python-pip git memcached nginx uwsgi uwsgi-plugin-python
 +
 
 +
2. Configure memcached
 +
 
 +
Add two options to startup script:
 +
  -M Disable automatic removal of items from the cache when out of memory. Additions will not be possible until adequate space is freed up.
 +
  -m 1024 Use <num> MB memory max to use for object storage; the default is 64 megabytes.
 +
 
 +
''Note: On Debian/Ubuntu systems using the memcached package, this configuration is stored in /etc/memcached.conf''
 +
 
 +
3. Create a stackalytics user and group
 +
 
 +
sudo useradd -U stackalytics
 +
 
 +
4. Create folders needed for Stackalytics
 
   mkdir /opt/stack/
 
   mkdir /opt/stack/
   chown user.user /opt/stack
+
   chown stackalytics.stackalytics /opt/stack
 
   mkdir /var/local/stackalytics  
 
   mkdir /var/local/stackalytics  
   chown user.user /var/local/stackalytics
+
   chown stackalytics.stackalytics /var/local/stackalytics
  
The last folder is a place where all repos are checked out. user is an account that will be used for Git and Gerrit polling.
+
The last folder is a place where all repos are checked out. 'stackalytics' is an account that will be used for Git and Gerrit polling.
  
3. Check out Stackalytics code  
+
5. Check out Stackalytics code  
 
   cd /opt/stack/
 
   cd /opt/stack/
   git clone git://github.com/stackforge/stackalytics.git
+
   git clone https://github.com/openstack/stackalytics.git
  
4. Install Stackalytics:
+
6. Install Stackalytics:
 
   cd /opt/stack/stackalytics
 
   cd /opt/stack/stackalytics
   sudo python setup.py install
+
   sudo pip install .
  
 
This will install Stackalytics and all needed libraries into the system.  
 
This will install Stackalytics and all needed libraries into the system.  
  
5. Sync default data (users, repos, companies)
+
''Note: On Debian/Ubuntu systems the dependencies for package cffi require installing the distro package libffi-dev, and package cryptography requires installing distro package libssl-dev. Without these, the pip install command will fail."
  stackalytics-processor --sync-default-data
 
  
This command loads all default data into mongo.
+
7. Configure OpenStack gerrit access
  
6. Process all registered repos:
+
StackAlytics requires an account to access the Gerrit review system.  Refer to the documentation at [http://docs.openstack.org/infra/manual/developers.html the Developer's Guide] for details on setting up an account.
 +
 
 +
The private key and SSH username used to authenticate with Gerrit are configured in the file /opt/stack/stackalytics/etc/stackalytics.conf with the following parameters:
 +
 
 +
# SSH key for gerrit review system access
 +
# ssh_key_filename = /home/user/.ssh/id_rsa
 +
 +
# SSH username for gerrit review system access
 +
# ssh_username = user
 +
 
 +
8. Process all registered repos:
 
   stackalytics-processor
 
   stackalytics-processor
  
 
This command checks out all repos and parses them. Debugging may be enabled by passing --debug parameter (--verbose for error debugging)
 
This command checks out all repos and parses them. Debugging may be enabled by passing --debug parameter (--verbose for error debugging)
  
7. Run dashboard in development mode:
+
If customized configuration options are used, be sure to specify the configuration file on the command line, as in:
 +
stackalytics-processor --config-file /opt/stack/stackalytics/etc/stackalytics.conf
 +
 
 +
9. Run dashboard in development mode:
 
   stackalytics-dashboard
 
   stackalytics-dashboard
  
This command starts dashboard to listen on port 5000. The port may be changed by --listen-port parameter.
+
This command starts dashboard to listen on port 8080 of the localhost. The port may be changed by --listen-port parameter.
  
 
== Production deployment ==
 
== Production deployment ==
Line 68: Line 93:
 
</uwsgi>
 
</uwsgi>
 
</pre>
 
</pre>
 +
 +
''Note'': On Debian/Ubuntu systems using the native uwsgi package installation, this configuration should be written to /etc/uwsgi/apps-enabled/stackalytics.xml.  Then uwsgi needs to be restarted with 'sudo service uwsgi restart'
 +
  
 
nginx config:
 
nginx config:
Line 78: Line 106:
 
}
 
}
 
location  /static/ {
 
location  /static/ {
         alias  /opt/stack/stackalytics/dashboard/static/;
+
         alias  /opt/stack/stackalytics/stackalytics/dashboard/static/;
 
}
 
}
 
}
 
}
 
</pre>
 
</pre>
 +
 +
''Note'': On Debian/Ubuntu systems using the native nginx package installation, this configuration should be written to /etc/nginx/sites-available/default, or a separate config file in /etc/nginx/sites-available/  Then nginx needs to be restarted with 'sudo service nginx restart'

Latest revision as of 15:12, 5 May 2017

Prerequistics

Stackalytics depends on the following applications:

* Python-pip Python Package Index
* libpython2.7-dev - development C headers for Python
* libssl-dev - SSL development libraries, header files and documentation
* Git - used to retrieve project repos and get their log 
* Memcached - used to store commits
* Nginx - scalable web server
* Uwsgi, uwsgi-plugin-python - application container for Nginx and plugin for Python

Web dashboard is made on Flask framework. Typical deployment of Flask application uses nginx as front-end server and uwsgi as app container. For development purposes Stackalytics may be run on DevStack. The project doesn't require any libraries not listed in OpenStack requirements project.

Installation

Stackalytics deployment is verified on both Ubuntu and CentOS, however the first option is preferable since it has all necessary components as system packages.

1. Install necessary applications

Particular command depends on linux distribution. Here is example for Debian based system:

 sudo apt-get install libpython2.7-dev libssl-dev python-pip git memcached nginx uwsgi uwsgi-plugin-python

2. Configure memcached

Add two options to startup script:

 -M Disable automatic removal of items from the cache when out of memory. Additions will not be possible until adequate space is freed up.
 -m 1024 Use <num> MB memory max to use for object storage; the default is 64 megabytes.

Note: On Debian/Ubuntu systems using the memcached package, this configuration is stored in /etc/memcached.conf

3. Create a stackalytics user and group

sudo useradd -U stackalytics

4. Create folders needed for Stackalytics

 mkdir /opt/stack/
 chown stackalytics.stackalytics /opt/stack
 mkdir /var/local/stackalytics 
 chown stackalytics.stackalytics /var/local/stackalytics

The last folder is a place where all repos are checked out. 'stackalytics' is an account that will be used for Git and Gerrit polling.

5. Check out Stackalytics code

 cd /opt/stack/
 git clone https://github.com/openstack/stackalytics.git

6. Install Stackalytics:

 cd /opt/stack/stackalytics
 sudo pip install .

This will install Stackalytics and all needed libraries into the system.

Note: On Debian/Ubuntu systems the dependencies for package cffi require installing the distro package libffi-dev, and package cryptography requires installing distro package libssl-dev. Without these, the pip install command will fail."

7. Configure OpenStack gerrit access

StackAlytics requires an account to access the Gerrit review system. Refer to the documentation at the Developer's Guide for details on setting up an account.

The private key and SSH username used to authenticate with Gerrit are configured in the file /opt/stack/stackalytics/etc/stackalytics.conf with the following parameters:

# SSH key for gerrit review system access
# ssh_key_filename = /home/user/.ssh/id_rsa

# SSH username for gerrit review system access
# ssh_username = user

8. Process all registered repos:

 stackalytics-processor

This command checks out all repos and parses them. Debugging may be enabled by passing --debug parameter (--verbose for error debugging)

If customized configuration options are used, be sure to specify the configuration file on the command line, as in:

stackalytics-processor --config-file /opt/stack/stackalytics/etc/stackalytics.conf

9. Run dashboard in development mode:

 stackalytics-dashboard

This command starts dashboard to listen on port 8080 of the localhost. The port may be changed by --listen-port parameter.

Production deployment

Production deployment is done with help of uwsgi and nginx.

uwsgi config:

<uwsgi>
    <socket>/tmp/stackalytics.sock</socket>
    <pythonpath>/opt/stack/stackalytics/dashboard/</pythonpath>
    <module>web:app</module>
    <plugins>python27</plugins>
    <env>STACKALYTICS_CONF=/usr/local/etc/stackalytics/stackalytics.conf</env>
</uwsgi>

Note: On Debian/Ubuntu systems using the native uwsgi package installation, this configuration should be written to /etc/uwsgi/apps-enabled/stackalytics.xml. Then uwsgi needs to be restarted with 'sudo service uwsgi restart'


nginx config:

server {
	listen   80; 
	location / {
        	uwsgi_pass  unix:///tmp/stackalytics.sock;
        	include     uwsgi_params;
	}
	location  /static/ {
        	alias  /opt/stack/stackalytics/stackalytics/dashboard/static/;
	}
}

Note: On Debian/Ubuntu systems using the native nginx package installation, this configuration should be written to /etc/nginx/sites-available/default, or a separate config file in /etc/nginx/sites-available/ Then nginx needs to be restarted with 'sudo service nginx restart'