Rewrite headers when write mode is append?

[xizhihui]

Hi, it seems that headers will be append many times if I append contents to an existing sam when these contents are not appended in just one time(open and close multiple times). Or in other words, does simplesam only support for write not append?

[mdshw5]

Or in other words, does simplesam only support for write not append?

Thanks for asking this question. I did only consider w mode file objects, and did not do any check for file modes in the source. If you could describe your use case a bit maybe we can support this. One issue I can see is that appending to an existing SAM file would mean that the existing SAM headers would be immutable, since there is no way we can update just the header portion of an existing file.

[xizhihui]

Thanks for your reply!
I just wanna split reads into single cells identified by cell barcodes in tag "CR" or "CB" (I found it is hard to do that, maybe there are some other convenient ways.) . So it is not a common usage ~~

[mdshw5]

Ah, so you would need to either keep ~10,000 file handles open or open and append to an existing file. I think that is a reasonable use case, and I'll take a look to see how that could be implemented here.

[xizhihui]

so you would need to either keep ~10,000 file handles open or open and append to an existing file

Yeah, it troubles me hours 😅.
I just find some codes in bamtools issue, herrinca firstly sort the bam to avoid this. Hope it will help.

[mdshw5]

@xizhihui I've added some initial support for appending to existing SAM files using simplesam.Writer. If you pass an open file with mode="a" then it the Writer will only add a header if we determine that the SAM file is empty (new). In all other cases we just ignore the header and continue appending records to the existing file.

simplesam/simplesam.py

Lines 223 to 242 in 8e914ce

	class Writer(object):
	""" Write SAM/BAM format file from :class:`.Sam` objects. """
	def __init__(self, f, header=None):
	try:
	_, ext = os.path.splitext(f.name)
	if ext == '.bam':
	# Why not just pipe to samtools?
	raise NotImplementedError("Bam writing support is not implemented.\n")
	except AttributeError: # pipe?
	pass
	self.file = f
	if self.file.mode == 'a' and self.file.tell() == 0:
	# We're appending to an empty file. Assume we need a header.
	self._merge_header(header)
	self._header_dict_write()
	elif self.file.mode == 'a' and self.file.tell() > 0 and header:
	raise NotImplementedError("Updating headers on existing SAM files is not supported.\n")
	else:
	self._merge_header(header)
	self._header_dict_write()

[mdshw5]

If you want to test this out, you can install from the develop branch using pip install -e "git+https://github.com/mdshw5/simplesam.git@develop#egg=simplesam".

[xizhihui]

Thanks for you updating.

I've tested it. In this check code, if mode is append and self.file.tell() >0, and provide no new header, it goes to the else branch which add a None type header.
I modified it and worked well:

        if self.file.mode == 'a' and self.file.tell() == 0:
            # We're appending to an empty file. Assume we need a header.
            self._merge_header(header)
            self._header_dict_write()
        elif self.file.mode == 'a' and self.file.tell() > 0:
            if header:
                raise NotImplementedError("Updating headers on existing SAM files is not supported.\n")
        else:
            self._merge_header(header)
            self._header_dict_write()

[mdshw5]

Thanks @xizhihui. I've incorporated that change and will push this as a new point release.

[mdshw5]

Closing this as the change is now in v0.1.4.0. Please let me know if there's anything else I can do to make your use case work better!