NDEx Server Update and Maintenance

Overview

This document provides instructions to perform NDEx server updates and maintenance. All the following instructions assume that NDEx was installed in the default directory. If you have customized your local server installation, you will need to modify these instructions to match your operating environment.

Bouncing an NDEx Server

In case it is necessary to restart (bounce) the NDEx service you can do so by following these instructions. Please note that by NDEx service we mean the Tomcat server. Instructions to bounce the SOLR Server are specified later int this document. When bouncing the Tomcat server, it is not necessary to also bounce the SOLR server.

STEP 1.

Shutdown your old NDEx service (Tomcat server):

sudo su – ndex –s /bin/bash
cd /opt/ndex/tomcat/bin
./shutdown.sh

Note: verify that the NDEx server has actually stopped before proceeding with the next step.

To verify that the Tomcat process is shutdown, use the following command:

ps –ef | grep tomcat
STEP 2.

If the Tomcat process didn't shutdown properly, kill it manually by specifying its process ID.

STEP 3.

Restart the NDEx service (Tomcat server) as user "ndex":

sudo su – ndex –s /bin/bash
cd /opt/ndex/tomcat/bin
./startup.sh


Server Update

Updating the NDEx software is a quick and straightforward operation: just follow these easy steps!In this example, let's suppose that you are running NDEx v1.3 and want to update to the newest stable release NDEx v1.3.1.

STEP 1.

Download the new updated bundle from our FTP Server.

STEP 2.

Shutdown your old NDEx service (Tomcat server):

sudo su – ndex –s /bin/bash
cd /opt/ndex/tomcat/bin
./shutdown.sh

Note: verify that the NDEx server has actually stopped before proceeding with the next step.

To verify that the Tomcat process is shutdown, use the following command:

ps –ef | grep tomcat
STEP 3.

Shutdown your SOLR server:

/opt/ndex/solr/bin/solr stop
STEP 4.

Remove the directory:

/opt/ndex/tomcat/webapps/ndexbio-rest
STEP 5.

Replace:

 /opt/ndex/tomcat/webapps/ndexbio-rest.war

with the new .war file in the downloaded bundle.

STEP 6.

Restart SOLR:

/opt/ndex/solr/bin/solr start
STEP 7.

Restart the NDEx service (Tomcat server) as user "ndex":

sudo su – ndex –s /bin/bash
cd /opt/ndex/tomcat/bin
./startup.sh


Server Maintenance

This section describes some essential maintenance procedures for the NDEx server and database. Future updates and releases will include new maintenance functions that will be explained in this document.

Database migration

The following procedure allows the migration of an NDEx database from v1.2 to v1.3. This procedure will also take care of building the SOLR index.

STEP 1.

Install the v1.3 NDEx bundle.

STEP 2.

Start the SOLR server:

/opt/ndex/solr/bin/solr start
STEP 3.

Go to the directory where NDEx web service was deployed by Tomcat using this command:

cd /opt/ndex/tomcat/webapps/ndexbio-rest/WEB-INF/
STEP 4.

Define the ndex environment variable using:

export ndexConfigurationPath=/opt/ndex/conf/ndex.properties
STEP 5.

Assuming your NDEx v1.2 database is located at /opt/ndex/orientdb/databases/ndex_1_2, you can run the following command to start the migration process:

java -classpath lib/*:../../../lib/* org.ndexbio.orientdb.migration.Migrator1_2to1_3/opt/ndex/orientdb/databases/ndex_1_2 >/tmp/output 2>/tmp/log.txt
STEP 6.

Once STEP 5 above is complete, you can start the NDEx service (Tomcat server):

sudo su – ndex –s /bin/bash
cd /opt/ndex/tomcat/bin
./startup.sh

Database Restore

NDEx creates database backups on a daily basis. All the backups are created and kept under directory:

/opt/ndex/dbbackups

Backups have the following naming convention: db_{backup_date}.export.gz, so, the 06/26/2015 backup would be: db_2015-06-26.export.gz

The NDEx server doesn't clean up the backups automatically. Server administrators should manually clean up the outdated backups periodically.

To Restore the database from a backup:

1. Make sure the NDEx server is completely shutdown (both Tomcat and SOLR).

2. Make a copy of your old database.

3. Remove the old database by navigating to the directory /opt/ndex/orientdb/databases and using the command:

rm -rf ndex

4. Start the Orientdb console as user ndex with these commands:

cd /opt/ndex/orientdb/bin
./console.sh

5. In console, re-create the database, import data from backup and rebuild all the indexes using these commands:

import database <full path of db backup file>
create index index-baseterm-id ON baseTerm (id) UNIQUE;
create index index-citation-id ON citation (id) UNIQUE;
create index index-edge-id ON edge (id) UNIQUE;
create index index-external-id ON NdexExternalObject (UUID) UNIQUE;
create index index-function-id on functionTerm (id) UNIQUE;
create index index-namespace-id on namespace (id) UNIQUE;
create index index-node-id on node (id) UNIQUE;
create index index-reifiedEdge-id on reifiedEdgeTerm (id) UNIQUE;
create index index-support-id on support (id) UNIQUE;
create index index-user-emailAddress on user (emailAddress) UNIQUE;
create index index-user-username on account(accountName) UNIQUE;
create index idx_baseterm_name on baseterm (name) FULLTEXT ENGINE LUCENE;
create index index-node-name on node (name) FULLTEXT ENGINE LUCENE;
create index idx_network-name-desc on network(name, description) FULLTEXT ENGINE LUCENE;
quit;

6. Copy the cache file to the new server after restoring the database. If you restore the db in a different server from the original one, you also need to copy the cache files over so that the ReadOnly networks can still work properly. The cache files are located at /opt/ndex/workspace/cache. You can just copy all the files under that directory to your target machine.

7. Now, you need to rebuild the SOLR index (see Rebuilding the SOLR index section).

8. Restart the SOLR server (see Server Update section).

9. Restart the NDEx (Tomcat) server (see Server Update section).


Rebuilding the SOLR index

The following procedure is used to rebuild the SOLR index. This procedure can be used if the SOLR index gets corrupted or if the indexing schema is changed. This procedure is NOT necessary after a database migration as the migration process also takes care of rebuilding the index.

STEP 1.

Shutdown the NDEx service (Tomcat server):

sudo su – ndex –s /bin/bash
cd /opt/ndex/tomcat/bin
./shutdown.sh

Note: verify that the NDEx server has actually stopped before proceeding with the next step.

To verify that the Tomcat process is shutdown, use the following command:

ps –ef | grep tomcat
STEP 2.

Shutdown the SOLR server:

/opt/ndex/solr/bin/solr stop
STEP 3.

To cleanup the old SOLR indexes, go to directory:

/opt/ndex/solr/server/solr

  1. Remove all directories that looks like a UUID (for example: "53b0602f-6194-11e5-8ac5-06603eb7f303").
  2. Remove the the directory called "index-networks".
STEP 4.

Go to the directory where NDEx web service was deployed by Tomcat using this command:

cd /opt/ndex/tomcat/webapps/ndexbio-rest/WEB-INF/
STEP 5.

Define the ndex environment variable using:

export ndexConfigurationPath=/opt/ndex/conf/ndex.properties
STEP 6.

Rebuild the SOLR index using NDEx

java -classpath lib/*:../../../lib/* org.ndexbio.common.solr.ReIndexer all >/tmp/log
STEP 7.

Once STEP 6 above is complete, you can start the NDEx service (Tomcat server):

sudo su – ndex –s /bin/bash
cd /opt/ndex/tomcat/bin
./startup.sh