Living Atlases Toolkit

• Introduction
  • How ?
  • Demo
  • How the code is organized
• Prerequisites
  • Docker
  • Docker compose
  • Data directories
• Clone or just download docker-compose.yml
• Running the la-toolkit
  • Directly using docker-compose
  • or using a docker-compose helper
  • Running the la-toolkit in an external server.
• Upgrade the toolkit
  • Notes to upgrade to 1.1.X
• Migrate your old inventories to the toolkit
• Migrate your la-toolkit to other location
• Logs and debugging
• Development
  • Using flutter web
  • Autogeneration of code
  • Backend during development
  • Flutter build
  • Ubuntu 20 vs Ubuntu 18
  • Docker image build
• Developed so far and Roadmap
• Screenshots
  • Loading:
  • Intro page:
  • Intro continuation:
  • List of created projects:
  • Project Tools:
  • Editing the project:
  • Service definition:
  • Theme selection:
  • Services in servers:
  • Servers connectivity:
  • Project tunning:
  • Project drawer with links to each service and admin interfaces:
  • SSH keys administration:
  • SSH Gateways configuration:
  • Project configuration lint warnings
  • Testing connectivity with the project servers:
  • Deployment:
  • Deployment Ansible terminal:
  • Deployment results (success):
  • Deployment results (failed):
  • History of deployments with repeat function
  • Software upgrade checks
  • Software releases selection
  • Console for the intrepids:
• License
  • Others

Introduction

This tool facilitates the installation, maintenance and monitor of Living Atlases portals.

How ?

A Living Atlas (LA) can be deployed and maintained using:

1. the Atlas of Living Australia (ALA) Free and Open Source Software, with
2. the ala-install, the official ansible code that automatically deploy and maintain a Living Atlas (LA) portal
3. some configuration that describes your LA portal that is used by ala-install

This LA Toolkit puts all these parts together with an user friendly interface, and an up-to-date environment to perform the common maintenance tasks of a LA portal.

Demo

There is a non-functional demo of this tool in:

http://toolkit-demo.l-a.site/

this demo is not functional because is only the UI frontend and does not configure any server or portal. It's just for demostration purposes. You can have a look there and create some sample project to see how this tool works.

How the code is organized

This repository is a the frontend of the LA Toolkit. It uses this repo as backend and both components are packaged together in a docker image with all the dependencies to deploy and maintain a LA Portal. It's also uses the ala-install and the LA Ansible Inventories Generator.

Prerequisites

Docker

To run the la-toolkit you need to install docker in the computer you want to use to deploy your LA Portal (like your laptop, or similar computer). Doublecheck that docker images runs in your common user (without root), for that follow this post-install docker instructions.

Docker compose

Optionally you'll need the Docker Compose.

Data directories

Your will need also some directories to store your config, logs and ssh configuration. In GNU/Linux you can use:

mkdir -p /data/la-toolkit/config/ /data/la-toolkit/logs/ /data/la-toolkit/ssh/ /data/la-toolkit/mongo /data/la-toolkit/backups
      

or similar to create it. Something like:

/data/la-toolkit
         ├── config
         ├── logs
         ├── mongo
         ├── backups
         └── ssh 

If you use a different directory, you'll have to update the docker-compose.yml files accordinly.

This directories should be writable by your user and docker.

Clone or just download docker-compose.yml

To continue you can git clone this repository:

git clone https://github.com/living-atlases/la-toolkit.git
cd la-toolkit

or just copy the docker-compose.yml and mongo-init.sh and adapt the first one to your needs (like changing the default passwords prior to start the toolkit via docker-compose) following the comments.

Running the la-toolkit

Directly using docker-compose

docker-compose up -d

This will start three containers (la-toolkit, la-toolkit-mongo and la-toolkit-watchtower). You can see it with docker ps. Verify that they start correctly.

Open the toolkit in http://localhost:2010/

or using a docker-compose helper

./dockerTask.sh compose

Note that you'll need to have installed xdg-utils (sudo apt install xgd-utils).

Run ./dockerTask.sh for more options (like how to run a development environment).

In Windows, try dockerTask.ps1 (feedback welcome).

Running the la-toolkit in an external server.

You can run the la-toolkit in another server and redirect the ports via ssh like:

 ssh -L 2010:127.0.0.1:2010 -L 2011:127.0.0.1:2011 -L 2012:127.0.0.1:2012 yourUser@yourRemoteServer -N -f

Currently we use a range of ports for the terms (2011-2100), so depending on the number of users using this instance of the la-toolkit you'll need more port redirections.

Upgrade the toolkit

Get the latest version of the la-toolkit with:

./dockerTask.sh update

or :

docker-compose kill
docker-compose rm -f
docker-compose pull 
docker-compose up -d

TODO: Add the update task to the Windows script.

Notes to upgrade to 1.1.X

• Copy the new docker-compose.yml as it includes new images and configurations
• Move your data to /data/la-toolkit and create an additional /data/la-toolkit/mongo/. If you want to use a different directories edit your docker-compose.yml volumes accordingly. You can also use symlinks.
• Change the mongo db user/passwords before start the container.
• A migration of your projects json configuration to mongo should be done at startup. Please verify that the la-toolkit start correctly. If not see the "Logs and debugging" section above.

Migrate your old inventories to the toolkit

If you were using other generated inventories, you can import it using the (+) button with some additional steps:

• Tune your imported project in the Edit and Tune tools. For instance, add your servers IPs, etc. See that there is a "Advanced" mode. There you can copy in the bottom text area your inventory local-extras.
• After enter in the Deploy Tool, some new inventories will be generated (and also a new password file). Substitute that generated local-password with your old one that you are using, to restore your passwords and not using new ones.

Migrate your la-toolkit to other location

You should:

• copy all your volumes to the new server.
• optionally, restore a recent mongo db backup. Something like mongorestore --drop -u your_la_toolkit_mongo_admin -p your_mongo_admin_pass your-backup-directory should restore it.

Logs and debugging

Startup errors can be debuged running docker-compose without -d:

docker-compose -f ./docker-compose.yml up 

Runtime server errors during the use of the la-toolkit can be debugged looking the logs with:

docker logs la-toolkit

If the la-toolkit restart continuosly durint development, it can be debugged with:

docker run -it --network=la-toolkit_default --entrypoint /bin/bash  livingatlases/la-toolkit:latest -s

In some cases the browser devtools console can show some info about browser code errors.

Please fill an issue with this information if you encounter some problem.

Development

This frontend is developed using Flutter Web.

A few resources to get you started if your are new to Flutter:

• Lab: Write your first Flutter app
• Cookbook: Useful Flutter samples
• Flutter Web support

Using flutter web

https://flutter.dev/docs/get-started/web

$ flutter channel stable
$ flutter upgrade

Autogeneration of code

There are some code (like the json serialization) that should be generated when some model changes. This is done with:

flutter pub run build_runner watch --delete-conflicting-outputs

Backend during development

During development you'll need to have running the backend and also a docker container of the la-toolkit (with this name).

Flutter build

We need to have the frontend build prior to build the docker image:

flutter test && flutter build web

Ubuntu 20 vs Ubuntu 18

Right now there are two images based in Ubuntu 20.04 and 18.04 respectively. We have tested more u18 as deployment environment, but we are tryng to start using u20. So if you have some ansible issue during deploying (like python2/ptython3 issues), try 18.04 instead.

Docker image build

You will need to build the flutter web as described below prior to build a la-toolkit image.

docker build . -f ./docker/u18/Dockerfile -t la-toolkit # for ubuntu 18.04
docker build . -f ./docker/u20/Dockerfile -t la-toolkit/u20 # for ubuntu 20.04 (testing right now)

Developed so far and Roadmap

• Basic configuration of LA Portals
• Tunning of LA configurations with advanced mode
• Software compatibility and other project checks and recommendations
• Branding theme selection
• Helper for configuration of ssh
• Connectivity and other servers checks
• Generation and update of inventories
• CAS additional tasks (like keys generation)
• Map helper to configure collections, spatial and regions homepages
• Ansible deployment of portals
• Inline help and descriptive commands that are executed for educational purposes
• Terminal with deployment environment
• Ansible use stats (number of tasks executed with success or failed,...)
• Import of previous inventories
• Template projects of existing LA Portals
• Logs store and replay previous deploy tasks
• Ansible task errors summary
• Software dependencies release checking and notification of available upgrades
• Pre-deploy tasks (wip)
• Portal status tool
• Branding deployment
• Post-deploy tasks
• Support additional hubs configuration
• Services redundancy
• Better ALA software versions control
• LA pipelines support
• Concurrent user support
• SSL support via letsencrypt (work in progress)
• Toolkit users management (work in progress)
• Improve ssh keys management

Screenshots

Loading:

Intro page:

Intro continuation:

List of created projects:

Project Tools:

Editing the project:

Service definition:

Theme selection:

Services in servers:

Servers connectivity:

Project tunning:

Project drawer with links to each service and admin interfaces:

SSH keys administration:

SSH Gateways configuration:

Project configuration lint warnings

Testing connectivity with the project servers:

Deployment:

Deployment Ansible terminal:

Deployment results (success):

Deployment results (failed):

History of deployments with repeat function

Software upgrade checks

Software releases selection

Console for the intrepids:

License

MPL © Living Atlases

Others

• check_by_ssh from https://github.com/nagios-plugins/nagios-plugins/ under GNU License