Skip to content

Python package providing read/write support of the WOUDC Extended CSV format.

License

Notifications You must be signed in to change notification settings

woudc/woudc-extcsv

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build Status Build status Downloads this month on PyPI Latest release License

WOUDC Extended CSV library

Python package providing read/write support for files of the WOUDC Extended CSV format.

Installation

Requirements

woudc-extcsv requires Python 3 or above. See requirements.txt.

Development

# install dev requirements
pip install -r requirements-dev.txt

Installing the Package

# via pip
pip install woudc-extcsv
# via easy_install
easy_install woudc-extcsv

Usage

Reader Objects

from woudc_extcsv import Reader
# read from file
with open('file.csv', 'r') as ff:
    ecsv = Reader(ff.read())
# read from string
ecsv = Reader(my_ecsv_string)

Writer Objects

import woudc_extcsv
# create new writer object with common/metadata tables and fields available
ecsv = woudc_extcsv.Writer(template=True)

# Add file comments
ecsv.add_comment('This is a file level comment.')
ecsv.add_comment('This is another file level comment.')

# Add metadata
ecsv.add_data('CONTENT', 
              'WOUDC,Spectral,1.0,1')
ecsv.add_data('DATA_GENERATION',
              '2002-05-29,JMA,1.0')
ecsv.add_data('PLATFORM',
              'STN,7,Kagoshima,JPN,47827')
ecsv.add_data('INSTRUMENT',
              'Brewer,MKII,059')
ecsv.add_data('LOCATION',
              '31.63,130.6,283')

# Add new table
ecsv.add_table('TIMESTAMP')
# Add fields
ecsv.add_field('TIMESTAMP', 'UTCOffset,Date,Time')
# Add data
ecsv.add_data('TIMESTAMP', '+08:38:47,1991-01-01,06:38:47')

# Add new table, fields, and data at the same time
ecsv.add_data('GLOBAL_SUMMARY',
              '06:38:47,7.117E-04,8.980E-03,94.12,99.99,114.64,001000,999',
              field='Time,IntACGIH,IntCIE,ZenAngle,MuValue,AzimAngle,Flag,TempC')
ecsv.add_data('GLOBAL',
              '290.0,0.000E+00',
              field='Wavelength,S-Irradiance,Time')
ecsv.add_data('GLOBAL',
              '290.5,0.000E+00')
ecsv.add_data('GLOBAL',
              '291.0,0.000E+00')
ecsv.add_table_comment('GLOBAL', 'This is a table level comment', index=1)
# Add table for new groupings
ecsv.add_data('TIMESTAMP',
              '+08:38:46,1991-01-01,07:38:46',
              field='UTCOffset,Date,Time',
              index=2)

ecsv.add_data('GLOBAL_SUMMARY',
              '07:38:46,2.376E-02,3.984E-01,82.92,6.75,122.69,100000,999',
              field='Time,IntACGIH,IntCIE,ZenAngle,MuValue,AzimAngle,Flag,TempC',
              index=2, table_comment='This is a table level comment.')
ecsv.add_table_comment('GLOBAL_SUMMARY', 'This is another table level comment', index=2)
# Write to string
ecsvs = woudc_extcsv.dumps(ecsv)
                
# Write to file
# validate (check if all common tables and their fields are present), if so dump to file
# if not, print violations
woudc_extcsv.dump(ecsv, 'spectral-sample.csv')

Convenience Functions

import woudc_extcsv
# load from file into Reader object
ecsv = woudc_extcsv.load('file.csv')
# load from string into Reader object
ecsv = woudc_extcsv.loads(my_ecsv_string)
# dump to file from Writer object
ecsv = woudc_extcsv.dump(ecsv_writer, 'file.csv')
# dump to string from Writer object
ecsv = woudc_extcsv.dumps(ecsv_writer)

ExtendedCSV Objects

The ExtendedCSV class is a parser class used in the Reader and Writer classes, and can be used to both read and write to an Extended CSV object.

from woudc_extcsv import ExtendedCSV
# read from file
with open('file.csv', 'r') as ff:
    ecsv = ExtendedCSV(ff.read())
# read from string
ecsv = Extended(my_ecsv_string)

# Add a file-level comment
ecsv.add_comment('This is a file level comment')
# Add new table to object
ecsv.init_table('GLOBAL', ['Wavelength', 'S-Irradiance'], line_num)
# Add another field to the new table
ecsv.add_field_to_table('GLOBAL', ['Time'])
# Add data to table
ecsv.add_values_to_table('GLOBAL', ['290.5', '0.000E+00', ''], line_num)
# Add a table comment
ecsv.add_table_comment('GLOBAL', 'This is a table level comment')
# Remove a table
ecsv.remove_table('GLOBAL')
# Validate Extended CSV and collimate the tables
# Check metadata tables
ecsv.validate_metadata_tables()
# Check dataset-specific tables
ecsv.validate_dataset_tables()

Error Handling

from woudc_extcsv import ExtendedCSV, NonStandardDataError, MetadataValidationError

try:
    ecsv = ExtendedCSV('bad content!')
except (NonStandardDataError, MetadataValidationError) as err:
    print(err.message)
    for error in err.errors:
         print(error)

Development

For development environments, install in a Python virtualenv:

virtualenv foo
cd foo
. bin/activate
# fork master
# fork https://github.com/woudc/woudc-extcsv on GitHub
# clone your fork to create a branch
git clone https://github.com/{your GitHub username}/woudc-extcsv.git
cd woudc-extcsv
# install dev packages
pip install -r requirements-dev.txt
# create upstream remote
git remote add upstream https://github.com/woudc/woudc-extcsv.git
git pull upstream master
git branch my-cool-feature
git checkout my-cool-feature
# start dev
git commit -m 'implement cool feature'
# push to your fork
git push origin my-cool-feature
# issue Pull Request on GitHub
git checkout master
# cleanup/update once your branch is merged on GitHub
# remove branch
git branch -D my-cool-feature
# update your fork
git pull upstream master
git push origin master

Running Tests

# via distutils
python setup.py test
# manually
python run_tests.py
# report test coverage
coverage run --source woudc_extcsv setup.py test
coverage report -m

Code Conventions

woudc_extcsv code conventions are as per PEP8.

# code should always pass the following
find -type f -name "*.py" | xargs flake8

Issues

All bugs, enhancements and issues are managed on GitHub.

History

The roots of woudc-extcsv originate within the WOUDC backend processing system in support of processing data submissions. woudc-extcsv was refactored into a standalone library providing read/write support of the data centre's core ingest format.

In 2015 woudc-extcsv was made publically available in support of the Treasury Board [Policy on Acceptable Network and Device Use] (http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=27122).

Contact