sds

https://docs.google.com/document/d/1E_wlNKb8ihSILPmtNbhNOZj-DCRmkr6ahntB73pRmKA/edit

Private Data Storage Configuration

Previously we learned how to use the HOT Data Store, a separate database that allows us to keep private data and merge it with OpenStreetMap data.

In this chapter we will learn how to configure our own data store on a server. This is an advanced topic, and it’s primarily aimed at those with experience creating and hosting websites. These instructions are written for use with the Ubuntu Linux operating system (version >= 11.10). If you are unable to use an Ubuntu machine, you can try setting up a virtual machine. We have provided instructions for virtual machine setup in a supplement to this guide.

Background Information

The private data store (SDS) allows users to combine public Openstreetmap data with private data in a way that allows all of the editing steps -- digitizing, editing, adding presets -- to be exactly the same as a normal editing actions. To the user, the private and public data all look the same and are handled in the same fashion; however, the private information contains a special, configurable prefix. For example, on the HOT Data Store, all tags that are prefixed with hot: are automatically saved in the private data store, keeping them separate from the public OSM server.

After public data has been loaded from the OSM server, the JOSM plug-in (SDS) queries the second (HOT operated) server. This query sends the OSM IDs of all the loaded objects to the private server and that server then replies with all of the private information associated with those objects. When the user uploads data, the plug-in separates data into public and private streams. The public, or non-prefixed data goes to OpenStreetMap, and the private data goes to the HOT server. Currently, the private data store relies on HOT infrastructure (the server, administration, etc), but in this chapter we will show how to configure you and your organization’s own private data store.

Private data is uploaded and downloaded in a separate manner than OSM public data.

The SDS Application

The private data store is a Ruby on Rails application with data stored in a PostgreSQL database. One database column keeps the OSM ID of the object. This is how the SDS knows which OSM object each database entry should connect to.

The datastore.hotosm.org web server uses Phusion Passenger as an application server for the Ruby (Rack) application and Apache as the web server. However, these applications can be changed depending on the skill and needs of your organization.

User Management

It is necessary to have an account to do anything with the application. The creators of the plugin decided it best to store the admin password in plain text form and allow the administrator to set and read user passwords (authentication is stored in a session cookie).

Once you are considered a user, you can login and access the web interface of the private data store. First, you are taken to the Landing Page, which allows you to use two different search parameters: map search and tag search. Users with administrative privileges can access the user management page and create or block users.

Installation

The data store software is open-source, and in the remainder of this chapter we will learn how to configure it on a server. We will demonstrate using Ubuntu, although the instructions will be similar for other flavors of Linux. These instructions involve setting up a web server and running a Ruby on Rails application. Although it is not extremely difficult to do, some of the steps may be confusing if you have never worked from the command line before.

Several assumptions are made in this tutorial. First, that you have Ubuntu >= 11.10 installed on your machine, and that you will be working from the command line (if not please visit our Supplement: Installing a Virtual Machine). If you already have other web applications running on your machine, you may need to adjust the instructions, but if you have a fresh install, this will work fine. In this section we will:

Install Dependencies*
Install PostgreSQL*
Install RVM, Ruby, and Rails (to run the application)
Install Passenger and Apache (the web server)
Download and configure the SDS
Configure the application on Apache
Launch the application!

*necessary for those that have not installed these before

Install Dependencies

Before we get going we need to install some packages that we will need.

 sudo apt-get update
 sudo apt-get -y install curl git-core
 sudo apt-get -y install build-essential bison openssl libreadline6 \
                         libreadline6-dev zlib1g zlib1g-dev libssl-dev \
                         libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 \
                         libxml2-dev libxslt-dev autoconf libc6-dev nodejs \
                         libcurl4-openssl-dev libpq-dev

Install PostgreSQL

We will be using PostgreSQL as a database for our application. Hopefully, you already installed this database from the Chapter 1 appendix. Run:

 sudo apt-get -y install postgresql-9.1 postgresql-9.1-postgis
                         postgresql-contrib-9.1 libpq-dev

We need to edit the PostgreSQL settings. Open the configuration file with nano by running:
```
 sudo nano /etc/postgresql/9.1/main/pg_hba.conf
```

Go to the bottom of the file. There are four lines here and we need to change the local access permissions for them to “trust”. This will allow us to access the database from the same machine without a password. The configuration will look something like this:

 local   all         postgres                          trust

 # TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

 # "local" is for Unix domain socket connections only
 local   all         all                               trust

 # IPv4 local connections:
 host    all         all         127.0.0.1/32          trust

 # IPv6 local connections:
 host    all         all         ::1/128               trust

Save the changes with Ctrl-O and Enter, end then exit nano with Ctrl-X.
Now we need to restart PostgreSQL so that the new changes take effect.
```
 sudo /etc/init.d/postgresql restart
```

Install RVM, Ruby, and Rails

The SDS application runs on Ruby on Rails, so we need to install Ruby and we need to install Rails. To do this will will also install RVM, which is a version manager for ruby.
```
 sudo su #(to make sure you are the root user)
 curl -L get.rvm.io | bash -s stable
```

When this is finished it will give some instructions on how to enable RVM. It should end with a message like this:

 Installation of RVM in /usr/local/rvm/ is almost complete:
  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with
    `umask u=rwx,g=rwx,o=rx`.
  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all
    shell windows.
 # Thank you for using RVM!
 #  I sincerely hope that RVM helps to make your life easier and more
 enjoyable!!!
 # ~Wayne

Now, we will need to add a user to RVM. Still as the root user, run the following commands. (If you are running as a different user, replace “root” with your username.)
```
 adduser root rvm
 source /etc/profile.d/rvm.sh
```

Now, while you are still running as root, install Ruby and Rails. Each of these commands will take a few minutes.

 rvm install 1.8.7 #(press q if it asks you to)
 gem install rdoc-data
 rdoc-data --install
 gem install rails

Install Passenger and Apache

Passenger makes it easy to run Ruby on Rails web applications on your web server. We will install the Passenger gem, and then we will install an apache module that makes passenger work with your web server. This assume you have already installed Apache, as covered in the Appendix of Chapter 2.
```
 gem install passenger
 sudo apt-get -y install apache2-prefork-dev libapr1-dev libaprutil1-dev
 passenger-install-apache2-module
```
Follow the instructions that the installer provides. You will need to add the following lines to the end of your /etc/apache2/apache2.conf file. These lines may be different depending on your installation, so copy the lines provided by the module installation program, rather than the lines provided here:
```
 LoadModule passenger_module /usr/local/rvm/gems/ruby-1.8.7-p371/gems/passenger-3.0.18/ext/apache2/mod_passenger.so
 PassengerRoot /usr/local/rvm/gems/ruby-1.8.7-p371/gems/passenger-3.0.18
 PassengerRuby /usr/local/rvm/wrappers/ruby-1.8.7-p371/ruby
```
Next, restart apache by running:
```
 /etc/init.d/apache2 restart
```

Now the web server is all set up, so we can install our application.

Download and Configure the SDS

Now, while still running as root, we are going to download and configure the SDS. First let’s create a database user for it:
```
 sudo su postgres -c "createuser hot\_josm" #(answer ‘y’)
```
Now, let’s create a www directory for our web applications and download the software there:
```
 cd /var/www
 wget https://github.com/hotosm/sds-server/tarball/master
 tar -zxvf master
```
Next, we will need to enter the new directory. To do this, list the folders in your www directory so that you know the exact name of your hotosm-sds-server-NUMBER
```
 ls
```
(potential result)
Next, enter into the directory. Answer ‘n’ if you are asked about the .rvmrc file.
```
 cd hotosm-sds-server-142a7d0 #(answer ‘n’)
```
HINT! When you are changing directories from the command line you can start to type the name of the directory, press TAB on your keyboard, and the rest of the directory name will be filled out for you.

This command will edit one of the settings we need to change:

 sed -i s/"config.assets.compile = false"/"config.assets.compile = true"/config/environments/production.rb

Run this command to ensure that all the necessary gems are installed:
```
 bundle install
```
Now we are ready to create our database and set it up. If you are unfamiliar with Rails, just know that it has special commands that can do this for us:
```
 bundle exec rake db:create RAILS_ENV=production
 bundle exec rake db:migrate RAILS_ENV=production
```

Lastly, we want to insert an initial user into the database so that when we open the SDS in our web browser, we are able to login. Run the following command, but replace the four values with your name, email address, and password.

 sudo su postgres -c "psql hot_josm_production -c \"insert into users
                     (firstname,lastname,email,password,active,admin) values
                     #('FIRST_NAME','LAST_NAME','EMAIL','PASSWORD',true,true);\""

You can see if everything is set up correctly by running:
```
 rails server -e production
```
If everything is right you should see something like this:

You will now be able to see the SDS running. If you are running Desktop Ubuntu open your web browser and go to http://localhost:3000. Otherwise you will be able to access the server from another computer at http://SERVER_IP_ADDRESS:3000.

Now, you are able to run the private data store from your own computer. However, running the server this way is a bit slow and bulky. We will need to use another web server to make it faster.

Configure the Application on Apache

For practice, we launched the web server that comes with rails in order to see if the application worked on our local server. Unfortunately, this webserver, WEBrick, is bulky and not useful for production. Luckily, we installed Apache, which is a powerful web server. We just need to configure a couple things to make our new application run on Apache.

Go back to your terminal, or command line. Press CTRL-C to exit WEBrick. Then create a new Apache configuration file with nano:
```
 sudo nano /etc/apache2/sites-available/sds
```

Copy the following text into the file. You will need to replace SERVER_IP_ADDRESS with the ip address of your server, and you may need to change the directory that it is pointed to if your version of the sds is different.

 <VirtualHost *:80>Re
    ServerName SERVER_IP_ADDRESS
    # !!! Be sure to point DocumentRoot to 'public'!
    DocumentRoot /var/www/hotosm-sds-server-142a7d0/public
    <Directory /var/www/hotosm-sds-server-142a7d0/public>
     # This relaxes Apache security settings.
     AllowOverride all
     # MultiViews must be turned off.
     Options -MultiViews
    </Directory>
 </VirtualHost>

Save and close nano with Ctrl-O, Enter, and Ctrl-X.
If you don’t know your localhost IP address, simply open new terminal window and type: ifconfig
Next we need to create shortcut of your sds site so you can access SDS easily at http://localhost.
```
 cd /etc/apache2/sites-enabled/
 ln -s /etc/apache2/sites-available/sds sds
 ls
```
There should be 2 files like this:
```
 000-default   sds
```
Now we need to remove 000-default:
```
 rm 000-default
```
Lastly, we need to restart Apache:
```
 /etc/init.d/apache2 restart
```
Now, you can access the SDS easily at http://localhost. Or, if you have installed on a remote server, replace localhost with the IP address.

Launch the Application!

From now on to launch your organization’s personal data store, you will simply need to give people the server address and have the server running.

Summary

This chapter covered how you can set up your own private datastore. It is useful to understand how to configure your own private data store, so that your organization can begin storing private information without the administration of HOT. This data store will provide your organization with the ability to add, edit and administer users with private data store access.

Previoussys admin Nextpostgis

Last updated 3 years ago