GNIP 48 Uploader Refactor

GNIP 48 - Uploader Refactor

Overview

Refactor and enhance the uploader architecture to simplify code, improve testability, and support new features such as fetching remote data, queueing, and notification.

Proposed By

Ian Schneider, Simone Dalmasso, Ariel Núñez

Assigned to Release

TBD

State

Planning

Motivation

The current uploader is very complex and prone to breakage. Testing, fixing bugs and adding new features is very difficult.

Proposal

1. Refactor Architecture

   1. remove use of any session state, move to models only

   2. break upload into multiple steps

      1. file transfer (or remote job configuration)
      2. resolving destination app if needed (e.g. provided tiff is actually a document)

      3. configuration

         1. for layers, what datastore, geogit, import transforms, etc.
         2. for documents, link to resource
         3. resolve potential naming issues (invalid layer/attribute names, duplicate resource name, etc.)

   3. queue for processing (celery being considered for implementation)
   4. provide means to 'automate' configuration steps (fire and forget) as per the current approach
   5. provide means to resume when failure occurs in (C)
   6. asynchronously notify users of success/failure by email, notifications, etc.
   7. provide and document API

2. Refactor UI

   1. build out multi-page user interface to match steps in upload

      1. upload files page

      2. manage uploaded files page

         1. resolve ambiguities (e.g. layer or document)
         2. configure ingest to database or geogit
         3. configure publishing to geoserver

      3. queued task status page

3. Enhance UI

   1. add upload progress indicators

   2. optional pre-upload server-side verification

      1. check file names/extensions to verify ability to process
      2. check projection information

4. Support for remote data

   1. allow specification of a URL to get files/data from
   2. configuration of ingest/publishing process (if layer)
   3. configuration of check interval (for periodic ingest)
   4. queue tasks
   5. on completion of transfer, proceed with automation or manual steps as per 'normal' upload

See Stories for rationale.

Issues

As of the time of creation:

#1686

#1674

#1425

#1319

#1271

#1195

#917

#362

#145

#135

Testing

Alternatives

Do nothing.

Stories

Epic: As an authenticated user, I can easily publish my spatial data and documents to GeoNode.

Random Data Stories:

1. I can upload any data through it's web interface and configure permissions.
2. If my data is not recognized as a 'layer', I can specify that it is.

Document Data Stories:

1. After uploading my data, I can specify that it has a link to a Layer or Map

Generic Layer Data Stories:

1. After uploading my spatial data through the web interface, I can configure how it will be ingested specifying any relevant data transformations
2. I don't have to wait for ingestion to complete and can be notified asynchronously
3. After my data is ingested, I can configure how my data will be published in GeoServer

Remote Data:

1. I can specify a link to a remote resource and GeoNode will download this a let me know when it's complete or if there were errors.
2. I can configure how often a remote resource is fetched and how it will be processed (big story!)

Notifications/Status:

1. I can tell GeoNode to email me when it is done doing some asynchronous task
2. I can obtain a list of all of my pending tasks and discover their status or progress

Usability Stories:

1. Before uploading my data, I can get feedback on how it will be supported in GeoNode - can it be used as a layer or not?
2. Before uploading my data, I can find out if the projection is supported
3. Before ingesting my data, I can modify any names that don't conform with valid XML or just let the system handle this
4. Before automatically renaming my data to avoid conflicts, I can specify my own name
5. If something goes wrong ingesting my data, I can change parameters and try again
6. I can specify WMS dimension configuration after ingestion if I make a mistake
7. If I provided an incomplete upload, I can append files to it
8. If I have a spotty connection and/or for some reason javascript doesn't load, I can still get my work done

Administrator Stories:

1. As an administrator, I'd like to be able to prevent unrecognized data from being uploaded.
2. As an administrator, I'd like to be able to schedule periodic maintenance of incomplete uploads (data transferred but not assigned to an 'app')

API:

1. As a power user/integrator, I can use GeoNode's API to perform the same tasks as a web interface user can.

Feedback

Voting