Skip to content

Jupyter Notebook-based workflows for programmatically accessing, processing, and visualizing 3D Elevation Program (3DEP) lidar data

License

Notifications You must be signed in to change notification settings

OpenTopography/OT_3DEP_Workflows

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

NSF-1948997 NSF-1948994 NSF-1948857

Reproducible scientific workflows for accessing, processing, and visualizing USGS 3DEP lidar data

The U.S. Geological Survey 3D Elevation Program (3DEP) is acquiring high-quality light detecting and ranging (lidar) data over the conterminous United States, Hawaii, and US Territories to meet the growing need for high-resolution 3-D representations of Earth's surface, vegetation, and other constructed features.

This repository contains a suite of customizable and reproducible workflows implemented as Jupyter Notebooks for programmatically accessing, processing, and visualizing USGS 3DEP lidar point cloud data and creating a range of derivative products (DEMs, topographic difference maps, canopy height models, and colorized point clouds). Each notebook is designed to address a specific use case/geospatial application and leverages open source geospatial software (e.g. PDAL,GDAL) along with detailed documentation that lower the barrier of entry for users with minimal programming experience.

workflow_examples

Installation

The Jupyter Notebooks can be run locally or on the Google Colaboratory cloud platform (Requires Google account with Google Drive access).

Dependencies

  • gdal
  • geopandas
  • ipyleaflet
  • matplotlib
  • pdal
  • pyproj
  • requests
  • rioxarray

Option 1 (Recommended): Google Colaboratory Installation

For quickly getting started and ease-of-use, it is recommended to launch and execute these notebooks on the Google Colaboratory ("Colab") cloud platform. Dependencies will be installed on a virtual environment on Google's cloud servers and the code will be executed in the browser. A major benefit is direct access to Google high-end CPU/GPUs and avoiding software dependency issues. All products will be saved to your personal Google Drive. Google Colab users are allocated 12GB of RAM (which can be upgraded to 25GB with Colab Pro). User's are advised that accessing and processing large point clouds (>100,000,000 points) may consume this RAM allocation and cause the runtime to fail. Those seeking to process large quantities of data may benefit from running the notebooks locally (Option 2).

To experiment and run one of the below Jupyter Notebooks on Google Colab click the Open In Colab badge beneath the corresponding Jupyter Notebook in the USGS 3DEP Jupyter Notebooks section, below.

Option 2: Local Installation

In order to run the Jupyter Notebook on your local machine, follow these steps:

Make a new directory on your local file system where the 3DEP Jupyter Notebooks (and all 3DEP data, if desired) will be saved. In this case, the directory will be called 3DEP. Change into the new directory and git clone the Github repository containing the Jupyter Notebooks and other relevant files to your local file system.

mkdir 3DEP
cd 3DEP
git clone https://github.com/OpenTopography/OT_3DEP_Workflows

Anaconda is recommended for Python package installation and management. Package versions in Anaconda are managed by the package management system conda. Anaconda installers for MacOS/Linux/Windows can be downloaded from https://docs.anaconda.com/anaconda/install/. Follow the instructions to install the appropriate version of Anaconda for your operating system.

After installing Anaconda, create a conda virtual environment with the required dependencies (contained in environment.yml). Note: Exectuting the following command will automatically create the conda environment with name 3dep and all of the required dependencies installed. If you would prefer a different name, replace 3dep with another name in the following command:

cd OT_3DEP_Workflows
conda env create -n 3dep --file environment.yml

Activate the conda environment with all of the necessary dependencies installed. (If different, substitute '3dep' for the name of your new conda environment.)

conda activate 3dep

Now, launch the chosen Jupyter Notebook. If unsure how to launch a Notebook, refer to this guide (https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html).

USGS 3DEP Jupyter Notebooks

  1. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for user-defined area of interestOpen In Colab

  2. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for USGS 7.5’ QuadranglesOpen In Colab

  3. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for USGS Hydrologic UnitsOpen In Colab

  4. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for user-defined corridorsOpen In Colab

  5. Generate Canopy Height Model (CHM) using USGS 3D Elevation Program (3DEP) lidar data for user-defined area of interestOpen In Colab

  6. Topographic Differencing using USGS 3D Elevation Program (3DEP) lidar data for user-defined area of interestOpen In Colab

  7. Generate colorized (RGB) point clouds using USGS 3D Elevation Program (3DEP) lidar data and National Agriculture Imagery Program (NAIP) ImageryOpen In Colab

Issues and Feature Improvements

Reporting issues and/or providing suggestions for improving these workflows or developing new ones is highly encouranged. Issues or suggested feature improvements may be submitted as an issue or by email to OpenTopography Support.

Funding

Funding for the creation and distribution of these Jupyter Notebook-based workflows was provided as by the USGS Community for Data Integration (CDI) through the funded grant Enhancing usability of 3DEP data and web services with Jupyter Notebooks.

OpenTopography is supported by the National Science Foundation (NSF) under Award Numbers 1948997, 1948994 & 1948857.

Additional Resources

About

Jupyter Notebook-based workflows for programmatically accessing, processing, and visualizing 3D Elevation Program (3DEP) lidar data

Topics

Resources

License

Stars

Watchers

Forks

Languages

  • Jupyter Notebook 100.0%