General Linux Setup

While Amazon Web Services (AWS) provides the preferred hosting for SEED, running on a bare-bones Linux server follows a similar setup, replacing the AWS services with their Linux package counterparts, namely: PostgreSQL and Redis.

SEED is a Django project and Django’s documentation is an excellent place to general understanding of this project’s layout.

Prerequisites

Ubuntu server/desktop 16.04 or newer (18.04 recommended)

Install the following base packages to run SEED:

sudo add-apt-repository ppa:timescale/timescaledb-ppa
sudo apt update
sudo apt upgrade
sudo apt install libpq-dev python3-dev python3-pip libatlas-base-dev \
gfortran build-essential nodejs npm libxml2-dev libxslt1-dev git \
libssl-dev libffi-dev curl uwsgi-core uwsgi-plugin-python mercurial
sudo apt install gdal-bin postgis
sudo apt install redis-server
sudo apt install timescaledb-postgresql-10 postgresql-contrib

# For running selenium/protractor
sudo apt install default-jre

Note

postgresql >=9.3 is required to support JSON Type

Configure PostgreSQL

Replace ‘seeddb’, ‘seeduser’ with desired db/user. By default use password seedpass when prompted

$ sudo timescaledb-tune
$ sudo service postgresql restart
$ sudo su - postgres
$ createuser -P "seeduser"
$ createdb "seeddb" --owner="seeduser"
$ psql
postgres=# GRANT ALL PRIVILEGES ON DATABASE "seeddb" TO "seeduser";
postgres=# ALTER USER "seeduser" CREATEDB CREATEROLE SUPERUSER;
postgres=# \q
$ exit

Python Dependencies

clone the seed repository from github

$ git clone [email protected]:SEED-platform/seed.git

enter the repo and install the python dependencies from requirements

$ cd seed
$ pip3 install -r requirements/local.txt

JavaScript Dependencies

$ npm install

Django Database Configuration

Copy the local_untracked.py.dist file in the config/settings directory to config/settings/local_untracked.py, and add a DATABASES configuration with your database username, password, host, and port. Your database configuration can point to an AWS RDS instance or a PostgreSQL 9.4 database instance you have manually installed within your infrastructure.

# Database
DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'seeddb',
        'USER': 'seeduser',
        'PASSWORD': '<PASSWORD>',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}

Note

Other databases could be used such as MySQL, but are not supported due to the postgres-specific JSON Type

In in the above database configuration, seed is the database name, this is arbitrary and any valid name can be used as long as the database exists. Enter the database name, user, password you set above.

The database settings can be tested using the Django management command, python3 manage.py dbshell to connect to the configured database.

create the database tables and migrations:

$ python3 manage.py migrate

Cache and Message Broker

The SEED project relies on redis for both cache and message brokering, and is available as an AWS ElastiCache service or with the redis-server Linux package. (sudo apt install redis-server)

local_untracked.py should be updated with the CACHES and CELERY_BROKER_URL settings.

CELERY_BROKER_URL = 'redis://127.0.0.1:6379/1'
CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': CELERY_BROKER_URL,
    }
}

Creating the initial user

create a superuser to access the system

$ python3 manage.py create_default_user --username=[email protected] --organization=lbnl --password=badpass

Note

Of course, you need to save this user/password somewhere, since this is what you will use to login to the SEED website.

Every user must be tied to an organization, visit /app/#/profile/admin as the superuser to create parent organizations and add users to them.

Running celery the background task worker

Celery is used for background tasks (saving data, matching, data quality checks, etc.) and must be connected to the message broker queue. From the project directory, celery can be started:

DJANGO_SETTINGS_MODULE=config.settings.dev celery -A seed worker -l INFO -c 2 --max-tasks-per-child 1000 -EBS django_celery_beat.schedulers:DatabaseScheduler

Running the development web server

The Django dev server (not for production use) can be a quick and easy way to get an instance up and running. The dev server runs by default on port 8000 and can be run on any port. See Django’s runserver documentation for more options.

$ python3 manage.py runserver --settings=config.settings.dev

Running a production web server

Our recommended web server is uwsgi sitting behind nginx. The python package uwsgi is needed for this, and should install to /usr/local/bin/uwsgi We recommend using dj-static to load static files.

Note

The use of the dev settings file is production ready, and should be used for non-AWS installs with DEBUG set to False for production use.

$ pip3 install uwsgi dj-static

Generate static files:

$ python3 manage.py collectstatic --settings=config.settings.prod -i package.json -i npm-shrinkwrap.json -i node_modules/openlayers-ext/index.html

Update config/settings/local_untracked.py:

DEBUG = False
# static files
STATIC_ROOT = 'collected_static'
STATIC_URL = '/static/'

Start the web server (this also starts celery):

$ ./bin/start-seed

Warning

Note that uwsgi has port set to 80. In a production setting, a dedicated web server such as nginx would be receiving requests on port 80 and passing requests to uwsgi running on a different port, e.g 8000.

Environment Variables

The following environment variables can be set within the ~/.bashrc file to override default Django settings.

export SENTRY_DSN=https://[email protected]/123
export DEBUG=False
export ONLY_HTTPS=True

Mail Services

AWS SES Service

In the AWS setup, we can use SES to provide an email service for Django. The service is configured in the config/settings/local_untracked.py:

EMAIL_BACKEND = 'django_ses.SESBackend'

In general, the following steps are needed to configure SES:

  1. Access Amazon SES Console - Quickstart

  2. Login to Amazon SES Console. Verify which region we are using (e.g., us-east-1)

  3. Decide on email address that will be sending the emails and add them to the SES Verified Emails.

  4. Test that SES works as expected (while in the SES sandbox). Note that you will need to add the sender and recipient emails to the verified emails while in the sandbox.

  5. Update the local_untracked.py file or set the environment variables for the docker file.

  6. Once ready, move the SES instance out of the sandbox. Following instructions here

  7. (Optional) Set up Amazon Simple Notification Service (Amazon SNS) to notify you of bounced emails and other issues.

  8. (Optional) Use the AWS Management Console to set up Easy DKIM, which is a way to authenticate your emails. Amazon SES console will have the values for SPF and DKIM that you need to put into your DNS.

SMTP service

Many options for setting up your own SMTP service/server or using other SMTP third party services are available and compatible including gmail. SMTP is not configured for working within Docker at the moment.

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

local_untracked.py

# PostgreSQL DB config
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'seed',
        'USER': 'your-username',
        'PASSWORD': 'your-password',
        'HOST': 'your-host',
        'PORT': 'your-port',
    }
}

# config for local storage backend
DOMAIN_URLCONFS = {'default': 'config.urls'}

CELERY_BROKER_URL = 'redis://127.0.0.1:6379/1'
CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': CELERY_BROKER_URL,
    }
}

# SMTP config
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

# static files
STATIC_ROOT = 'collected_static'
STATIC_URL = '/static/'