Set up dev env

Quickstart using Vagrant

With Vagrant and VirtualBox installed:

vagrant up
vagrant ssh -c 'cd /vagrant && bundle exec jekyll serve --force_polling --host 0.0.0.0'

To run other interactive commands, run vagrant ssh, then cd /vagrant.

To stop Vagrant: vagrant halt

Frontend

Getting Started

1. Make sure you have the latest LTS version of NodeJS installed.
2. Make sure you have the latest version of Yarn installed.
3. Run yarn in the root folder of the repository to install all dependancies.

Jekyll-Preact Arhictecture

We are in the process of incrementally moving from a bespoke Jekyll-Preact architecture to a standard React (via Create React App) architecture.

In order to build the current Jekyll-Preact assets you can run the following in the project root folder:

yarn build

Standard React Environment

The new React environment is integrated into the existing architecture via a single JavaScript file generated by means of a Yarn monorepo workspaces (in /packages/workspace/). It also follows standard conventions for monorepos. You can run the following scripts in /packages/webapp/:

• yarn start: This is where you will spend the majority of your time, this is a library that contains (and documents) all the components/views that will be included in the built JavaScript file.
• yarn build: Builds the production JavaScript file that needs to be added to the existing Vulekamali stack.

Backend

Regenerating data files

Create a python 2 virtual environment, e.g. with virtualenv2 env

Activate it and install python dependencies

source env/bin/activate
pip install -r requirements.txt

Development

Following changes to JS or CSS

yarn build

Be sure to run Jekyll and have a quick look whether things look and work right.

Run a local server to view the site

bundle exec jekyll serve --watch

Regenerating the site can take over a minute on a typical dev machine. You can speed it up by working on a subset of the site data. We delete a bunch of the routing files and tell git to ignore those changes in your local repository using git update-index --assume-unchanged. This means only national departments and departments from Mpumalanga and the Eastern Cape will exist, bringing regeneration time down to about 10s.

git ls-files|grep provincial|egrep -v '(mpumalanga|eastern-cape)' | grep html | tr '\n' ' ' | xargs git update-index --assume-unchanged
git ls-files|grep provincial|egrep -v '(mpumalanga|eastern-cape)' | grep html | tr '\n' ' ' | xargs rm

To remove assume-unchanged from all files with that currently configured:

git ls-files -v|grep '^h'| sed 's/^h//' | xargs git update-index --no-assume-unchanged

Regenerating data files

Using Travis CI

Every branch push results in a Travis CI build, but by default it won't regenerate data files.

Add [ci] to the commit message of the latest commit in the pushed branch to opt into generating data files.

Add [staging] to that commit message to use the staging web data server instead of production. This is useful if we want to see what the data will look like for a development branch of the web data server.

If any files changed, Travis CI will add and commit the changes and push that back to the branch. That means you'll need to pull the branch and perhaps use pull --rebase if you've since committed other changes. Don't trust the changes because it was a robot - check that the site works as expected and the diff looks reasonable before merging your PR

Best practise is to place these tags as the first part of the first line of the commit message. This makes it easy to see whether staging or production was used in the pull request list of commits.

To create a commit without any code changes to trigger a build, use --allow-empty. e.g.

git commit -m "[ci][staging] trigger data regeneration for new department xyz data" --allow-empty

Locally

source env/bin/activate
python generate/from_dynamic.py

Then use git status and git diff to get an idea of what changed. If it looks sensible, add the updated files and commit and PR back into the branch where you want changes reflected.

When modifying the Dynamic Budget Portal server, you might want to point to your development server:

PORTAL_URL=http://localhost:8000/ python generate/from_dynamic.py

You can update one file at a time as follows:

curl -o _data/2016-17/national/departments/planning-monitoring-and-evaluation.yaml https://dynamicbudgetportal.openup.org.za/2016-17/national/departments/planning-monitoring-and-evaluation.yaml