Parsers heterogeneity

[marblestation]

Most parsers have a parse(...) method that accept a raw string to be parsed such as:

adsabs-pyingest/pyingest/parsers/arxiv.py

Line 52 in 15ae056

def parse(self, fp, **kwargs):

But others, act very differently. For instance:

• For ATels, an URL is passed and the raw data is fetched from there:

adsabs-pyingest/pyingest/parsers/atel.py

Lines 55 to 61 in 15ae056

	def parse(self, url, **kwargs):
	atel_recs = [{}]
	headers = {
	'Content-type': 'text/xml',
	'Accept': 'text/html,application/xhtml+xml,application/xml',
	'User-agent': 'Mozilla/5.0'}
	data = self.get_records(url, headers=headers, **kwargs)

It would be convenient to separate the download action from the parse action.

• For proquest, there are no arguments, so it is unclear what it is parsing:

adsabs-pyingest/pyingest/parsers/proquest.py

Line 62 in 15ae056

def parse(self):

Unless we read the code and understand that the input is a file name (not a raw string) passed in the constructor and not in the parse method:

adsabs-pyingest/pyingest/parsers/proquest.py

Lines 23 to 27 in 15ae056

	def __init__(self, filename):
	marc_input_file = config.PROQUEST_BASE_PATH + filename
	oa_input_file = marc_input_file.replace('.UNX', '_OpenAccessTitles.csv')
	self.records = open(marc_input_file).read().strip().split('\n')
	oa_input_data = open(oa_input_file).read().strip().split('\n')

• For Affiliations and GCNCirc, there are no arguments again:

adsabs-pyingest/pyingest/parsers/affils.py

Line 67 in 15ae056

def parse(self):

adsabs-pyingest/pyingest/parsers/gcncirc.py

Line 83 in 15ae056

def parse(self):

It is again in the constructor that the input is passed, but this time is not a file but a raw string:

adsabs-pyingest/pyingest/parsers/affils.py

Lines 12 to 13 in 15ae056

	def __init__(self, input_string):
	self.original_string = input_string

adsabs-pyingest/pyingest/parsers/gcncirc.py

Lines 16 to 22 in 15ae056

	def __init__(self, data):
	# econv = EntityConverter()
	# econv.input_text = data
	# econv.convert()
	# self.raw = econv.output_text
	self.raw = data
	self.data_dict = dict()

The package would benefit from some homogenization so that all parser have some common methods (e.g., parse) with similar arguments / requirements, behavior / expectations and return values. For instance, sometimes we return the parsed result:

adsabs-pyingest/pyingest/parsers/arxiv.py

Line 125 in 15ae056

return result

Sometimes we do not return but we store it in the object:

adsabs-pyingest/pyingest/parsers/proquest.py

Lines 232 to 234 in 15ae056

	self.results.append(output_metadata)
	return