Help: Deleting specific DICOM tags in a nested tree with pydicom

[Ede1994]

I want to delete specific DICOM tags in a DICOM file, which are contained in a nested tree and which is repeated over and over again, here is an excerpt:

(5200, 9229)  Shared Functional Groups Sequence  1 item(s) ---- 
   (0021, 0010) Private Creator                     LO: 'SIEMENS MR SDS 01'
   (0021, 10fe)  Private tag data  1 item(s) ---- 
      (0021, 0010) Private Creator                     LO: 'SIEMENS MR SDS 01'
      (0021, 1019) Private tag data                    OB: Array of 146770 elements
      (0021, 1026) Private tag data                    IS: Array of 38 elements

I've tried several approaches before, but most recently my idea was to do it via a callback function and then use walk(callback[, recursive]) from pydicom. So that I don't have this tree structure any longer I wanted to use iterall()→ Iterator[DataElement], so to say flatten the dataset.

def _private_callback(dataset, data_element):
    tags_to_remove = [
        (0x0021, 0x1019),
        (0x0021, 0x1026),
        (0x0021, 0x1171)
        ]

    for data_element in dataset.iterall():
        if data_element.tag in tags_to_remove:
            print(data_element)
            # del dataset[data_element.tag]

dcm.walk(_private_callback)

This way all tags from the tags_to_remove list will really be found. Now I want to delete them all, but as soon as I use
del dataset[data_element.tag] I get the error:

(0021, 1019) Private tag data                    OB: Array of 146770 elements
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/.conda/envs/qa_pipeline/lib/python3.9/site-packages/pydicom/tag.py:28, in tag_in_exception(tag)
     27 try:
---> 28     yield
     29 except Exception as exc:

File ~/.conda/envs/qa_pipeline/lib/python3.9/site-packages/pydicom/dataset.py:2390, in Dataset.walk(self, callback, recursive)
   2389 data_element = self[tag]
-> 2390 callback(self, data_element)  # self = this Dataset
   2391 # 'tag in self' below needed in case callback deleted
   2392 # data_element

Cell In[26], line 14, in curves_callback(dataset, data_element)
     13 print(data_element)
---> 14 del dataset[data_element.tag]

File ~/.conda/envs/qa_pipeline/lib/python3.9/site-packages/pydicom/dataset.py:636, in Dataset.__delitem__(self, key)
    635 elif isinstance(key, BaseTag):
--> 636     del self._dict[key]
    637     if self._private_blocks and key.is_private_creator:

KeyError: (0021, 1019)

The above exception was the direct cause of the following exception:
...
     30 stack_trace = traceback.format_exc()
     31 msg = f"With tag {tag} got exception: {str(exc)}\n{stack_trace}"
---> 32 raise type(exc)(msg) from exc

KeyError: 'With tag (0008, 0005) got exception: (0021, 1019)\nTraceback (most recent call last):\n  File "/home/einspaennere/.conda/envs/qa_pipeline/lib/python3.9/site-packages/pydicom/tag.py"

This is an extremely large DICOM header with about 7000 lines.

[darcymason]

First problem is that your callback function iterates over the entire dataset. Walk itself already provides that functionality. The other problem is that it is tricky to delete things that you are iterating over.

You need to pick one or the other -- I suggest using walk, and copy/modify the source code for Dataset.remove_private_tags -- it does very similar to what you want, just replace if elem.tag.is_private: with if elem.tag in tags_to_remove:.

[Ede1994]

Thanks for your help. I have tried exactly the same before and it dowsn't work for all tags in tags_to_remove except the one in my 1st post. This then remains anyway, which is why I had then tried other approaches, like the one I posted.

def _xa50_private_callback(dataset, data_element):
    r'''
    Callback function to find sensible private tags corresponding for
    software version XA50, and remove all.
    '''
    tags_to_remove = [
        (0x0021, 0x1019),
        (0x0021, 0x1026),
        (0x0021, 0x1171)
        ]

    if data_element.tag in tags_to_remove:
        del dataset[data_element.tag]

ds.walk(_xa50_private_callback)

Or have I misunderstood something?
For example, the tag (0021, 1171) is not removed from the header at all.

[darcymason]

Several new thoughts:

• you've mentioned it is a large file - using walk is inefficient in this case as it visits all data elements at whatever nesting depth. Perhaps the approach could be more finely tuned if you know, for example, that they are all under the sequence in your first post. Regardless, there is still another problem to figure out with the current approach, so ...
• Private tag data 1 item(s) ---- -- it would be good to know if that is an actual 'SQ' (in an explicit VR file) or if pydicom has just assumed it is because of finding sequence markers. It may be that it is not iterated correctly. Can you do:

  from pydicom import config
  config.debug()
  ds = dcmread("path/to/file.dcm")

  and copy the parts around the example in the first post?
• the problematic (0021, 1171) you mention is not shown in your posts -- is it at the same level as the others, or nested differently? (please also include it in the same dump from bullet point above).

[Ede1994]

Sorry for my late reply, but I was on annual leave. The following output results when I use the debug config:

• (0021, 1019):

00000af6: 48 65 61 64 b'Head'    
00000afa: 21 00 19 10 4f 42 00 00 c0 42 02 00 (0021, 1019) OB Length: 148160
00000b06: 3c 58 50 72 6f 74 6f 63 6f 6c 3e 20 0a 7b 0a 20 20 3c 4e 61 ... b'<XProtocol> \n{\n <Na' ...
00024dc6: 21 00 1b 10 44 53 02 00 (0021, 101b) DS Length: 2

• (0021, 1026):

00024e14: 21 00 25 10 53 4c 0c 00                (0021, 1025) SL Length: 12
00024e1c: 00 00 00 00 00 00 00 00 b1 fb ff ff     b'\x00\x00\x00\x00\x00\x00\x00\x00\xb1\xfb\xff\xff'    
00024e28: 21 00 26 10 49 53 4c 00                (0021, 1026) IS Length: 76
00024e30: 30 5c 30 5c 30 5c 30 5c 30 5c 30 5c 30 5c 30 5c 30 5c 30 5c ... b'0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\' ...
00024e7c: 21 00 27 10 55 53 02 00                (0021, 1027) US Length: 2

• and (0021, 1171):

00025190: 21 00 5e 11 49 53 02 00                (0021, 115e) IS Length: 2
00025198: 30 20                                  b'0 '    
0002519a: 21 00 71 11 55 54 00 00 40 00 00 00    (0021, 1171) UT Length: 64
000251a6: 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 ... b'XXXXXXXXXXXXXXXXXXXX' ...
000251e6: 21 00 75 11 43 53 18 00                (0021, 1175) CS Length: 24

Here is the the full ouput as txt-file: dicom_debug.txt

By the way, in the meantime the above function works as intended, the reason was that I had to restart the kernel again. well, so far so good. Only it is quite slow, which as I have understood is due to the inefficient use of the walk function.

[darcymason]

Thanks for the outputs, however I don't see anything obvious from them. However, the original SQ (0021, 10fe) doesn't appear to be in there.

in the meantime the above function works as intended, the reason was that I had to restart the kernel again. well, so far so good

I'm not entirely clear -- is it (consistently) working okay now? If not then perhaps if you could post the relevant code in your latest use of walk, that might be helpful.

[Ede1994]

Yes, so it's working consistently now across multiple studies with no issues. It was just me ;)

So I understood that walk in the context is disadvantageous in terms of performance, which I also understand. I have now understood it so that I specify the tags directly, so for example:

ds[(0x5200, 0x9230)][0][(0x0021, 0x11fe)][0][(0x0021, 0x1171)]

And then delete the tags, right?

I use an external config.py file for my function, in which I have stored all tags software specific. Here I would like to store the nested tags as well. How would one do that then?

configurations = {
    "PhoenixZIPReport": {
        "blank_values": [
            (0x0070, 0x0084), # content creator's name
            (0x300E, 0x0008), # reviewer name
            (0x0032, 0x1060), # requested procedure description
            (0x0032, 0x1064), # requested procedure code sequence
            (0x0021, 0x1011)  # private tag (array)
        ],
        "dummy_values": [
            (0x0040, 0xA075), # Verifying Observer Name
            (0x0040, 0xA088)  # Verifying Observer Identification Code Sequence
        ]
    },
    "syngo MR XA50": {
        "blank_values": [
            (0x0010, 0x0032), # patient's birth time
            (0x0070, 0x0084), # content creator's name
            (0x300E, 0x0008), # reviewer name
            (0x0032, 0x1060), # requested procedure description
            (0x0032, 0x1064)  # requested procedure code sequence
        ],
        "dummy_values": [
            (0x0040, 0xA075), # Verifying Observer Name
            (0x0040, 0xA088)  # Verifying Observer Identification Code Sequence
        ]
    },
    "default": {
        "blank_values": [
            (0x0070, 0x0084), # content creator's name
            (0x300E, 0x0008), # reviewer name
            (0x0032, 0x1060), # requested procedure description
            (0x0032, 0x1064) # requested procedure code sequence
        ],
        "dummy_values": [
            (0x0040, 0xA075), # Verifying Observer Name
            (0x0040, 0xA088)  # Verifying Observer Identification Code Sequence
        ]
    }
}

[darcymason]

Here I would like to store the nested tags as well. How would one do that then?

Well, that depends on the depths, etc. It's really a Python question rather than a pydicom question. If there is a always a consistent structure (e.g. delete certain tags within specific sequences) then you could use lists or dicts. If it is variable (sometimes it is tags within a sequence within another sequence) then you would have to do a kind of recursion thing, e.g. with first level (sequence keyword or tag) being a dict key, then lists within that, one item of which could be a dict signalling another sequence to recurse into.

Another thought, if there are only certain top-level sequences that could contain keys to remove, then you could use walk on the items of those sequences (sequence items are Datasets, so can use walk on them) rather than over the whole dataset. Thus you get ease-of-use but with some increase in performance.

for seq_tag in list_of_seq_tags:
   seq = ds[seq_tag].value
   for seq_item in seq:
       seq_item.walk(callback_function)

Another thought, if just deleting you could also look at deid which can do this kind of thing for you with defined rules.

EDIT: Added line to get actual Sequence instance. Note code shown is untested.