Copyright 2015 OpenStack Foundation

This work is licensed under a Creative Commons Attribution 3.0
Unported License.

Migrate to infra!/story/2000158

The OpenStack Q&A site located at currently running in an instance deployed by a third-party. The main goal of this migration to make the entire deployment process repeatable, and move the project to proper infrastructure puppet repositories.

Problem Description

The site was deployed manually and contains the minimal application stack required for running the site. Currently the site is missing regular backups, security updates and deployment documentation.

Proposed Change

Create the missing puppet modules, prepare the changes in openstack-infra/system-config repo.


Leave as is.



Primary assignee:


Gerrit Topic

Use Gerrit topic “askbot-site” for all patches related to this spec.

git review -t askbot-site

Work Items

  • Lower DNS TTL to 300

  • Create puppet-askbot split-out module

  • Add vamsee-solr module and puppet-askbot to modules.env

  • Make the system-config changes, add ask.pp

  • Add SSL certificates and passwords to hiera

  • Launch new server

  • Restore database and static files from original site

  • Silent testing with /etc/hosts override

  • Backup / restore of original data

  • Update DNS entry of with the new server address

  • Redirect html traffic using nginx to new to avoid db sync issues

  • Restore DNS TTL to 3600


A new puppet-askbot repository will need to be created, along with updates to system-config to consume this module.


An will need to be created.

DNS Entries

The zone must be point to the newly created server as the last step of this migration process.


Askbot documentation need to be added to documentation.


The services will run on Ubuntu, so core operating system not requires any special attention.

The application stack have some elements that must be deployed from tar.gz or pypy instead of OS packages:

  • Apache Solr (4.7.2)

  • askbot


Askbot don’t have integration tests implemented. After instance creation and initial data migration, I suggest to do a 1-2 week long silent test of the UI and address upcoming bugs during that period.

Backup / recovery steps

Backup assets

Askbot holds all of the site data in a postgresql database, and all static files in the filesystem. In case of the existing deployment it is available at /srv/ask/dumps directory. The dumps are created by the shell script srv/ask/node2/cron/ executed daily by cron.

Invoke management commands

The management cli tool is available under /srv/askbot-sites/slot0/config, so to invoke management commands you need to enter this directory first.

$ cd /srv/askbot-sites/slot0/config
$ python <command> <args>

Stop celeryd and apache, jetty services

$ sudo service askbot-celeryd stop
$ sudo service apache2 stop
$ sudo service jetty stop

Restore database dump

Recreate an empty database:

$ sudo su - postgres -c 'dropdb askbotdb'
$ sudo su - postgres -c 'createdb askbotdb --owner=ask'

Restore from backup:

$ psql -d askbotdb -h localhost -U ask -W -f /path/to/last-dump.sql

Notice you will be prompted for the ask_db_password from hiera.

Sync db and migrate:

$ cd /srv/askbot-sites/slot0/config
$ sudo python syncdb
$ sudo python migrate

Notice this step is required to apply schema changes between askbot versions.

Start celeryd

$ sudo service askbot-celeryd start
$ sudo service apache2 start

Rebuild solr indexes

$ sudo service jetty start
$ cd /srv/askbot-sites/slot0/config
$ sudo python askbot_rebuild_index -l en
$ sudo python askbot_rebuild_index -l zh

Test the solr deployment, query string “sahara”:

$ curl ""

Notice this query must return a non-empty resultset.

Restart celeryd

$ sudo service askbot-celeryd restart
$ sudo service apache2 restart

Restore static files

Static files must be extracted into /srv/askbot-sites/slot0/upfiles directory. It mostly holds profile pictures, and site logo, so if pictures not showing up in the site those files are missing, or have a wrong file permission.

$ cd /srv/askbot-sites/slot0
$ sudo rm -rf upfiles
$ sudo tar xf /path/to/last-upfiles.tar --strip-components=2


We are using vamsee-solr module 0.0.7 from puppetforge, and it is forcing Us to use solr 4.7.2 because 4.10.x requires some extra patches to work and this upgrade also means a schema change.