BUG: ValueError in npyio.py loading oxdna topology and trajectory

[alphataubio]

Describe the bug
topology and trajectory created with recent version of oxDNA 3.6.1 does not load in MN

To Reproduce

[http://tacoxdna.sissa.it/XYZ_oxDNA]

• generate test.oxdna using tacoxdna

[https://lorenzo-rovigatti.github.io/oxDNA/relaxation.html]

• First stage: Monte Carlo relaxation
• Second stage: molecular dynamics relaxation

Blender:

• topology: test.oxdna
• trajectory: test.traj
• "LOAD"

Error Codes

Traceback (most recent call last):
  File "/Users/mitch/Library/Application Support/Blender/4.1/scripts/addons/molecularnodes/io/dna.py", line 312, in execute
    load(
  File "/Users/mitch/Library/Application Support/Blender/4.1/scripts/addons/molecularnodes/io/dna.py", line 256, in load
    topology = read_topology_old(top)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mitch/Library/Application Support/Blender/4.1/scripts/addons/molecularnodes/io/dna.py", line 129, in read_topology_old
    array_str = np.loadtxt(lines[1:], dtype=str)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/Blender.app/Contents/Resources/4.1/python/lib/python3.11/site-packages/numpy/lib/npyio.py", line 1356, in loadtxt
    arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Applications/Blender.app/Contents/Resources/4.1/python/lib/python3.11/site-packages/numpy/lib/npyio.py", line 1026, in _read
    next_arr = _load_from_filelike(
               ^^^^^^^^^^^^^^^^^^^^
ValueError: the number of columns changed from 5 to 15 at row 3; use `usecols` to select a subset and avoid this error
Error: Python: Traceback (most recent call last):
  File "/Applications/Blender.app/Contents/Resources/4.1/python/lib/python3.11/site-packages/numpy/lib/npyio.py", line 1026, in _read
    next_arr = _load_from_filelike(
               ^^^^^^^^^^^^^^^^^^^^
ValueError: the number of columns changed from 5 to 15 at row 3; use `usecols` to select a subset and avoid this error

Desktop (please complete the following information):
OS: MacOS 13.2.1
Hardware: Mac mini M1
Blender Version: 4.1
MolecularNodes Version: v4.1.0 for Blender 4.1

Additional context
test.oxdna and test.traj too big to attach, available on demand

[BradyAJohnston]

Hi @alphataubio thanks for the bug report! I'm away hiking for another two weeks so won't be able to look into it until then, but if possible if you could share a minimum topology and trajectory for testing that would be great. From the error it should be straight forward enough I think to fix and will require some minor tweaking, but again won't be able to fix for a couple of weeks.

[alphataubio]

Hi @alphataubio thanks for the bug report! I'm away hiking for another two weeks so won't be able to look into it until then, but if possible if you could share a minimum topology and trajectory for testing that would be great. From the error it should be straight forward enough I think to fix and will require some minor tweaking, but again won't be able to fix for a couple of weeks.

no rush at all, you need to disconnect and enjoy your vacation.

[alphataubio]

Hi @alphataubio thanks for the bug report! I'm away hiking for another two weeks so won't be able to look into it until then, but if possible if you could share a minimum topology and trajectory for testing that would be great. From the error it should be straight forward enough I think to fix and will require some minor tweaking, but again won't be able to fix for a couple of weeks.

oxDNA_performance benchmarks from @ErikPoppleton is a good source of minimal oxdna test examples.

the bug i reported a few weeks ago is no longer reproducible and sorry i don't know why. the only issue now is if the oxdna trajectory does not include momenta ( [trajectory_print_momenta = false] option ) then i get the following warnings but the trajectory loads fine.

N8.zip

this option is quite useful to reduce the size of the trajectory file by almost half, and the individual particle momenta are not needed to render a video.

Unable to get velocity attribute from coordinates. Error: index 9 is out of bounds for axis 1 with size 9
Unable to get angular_velocity attribute from coordinates. Error: index 12 is out of bounds for axis 1 with size 9
Unable to get velocity attribute from coordinates. Error: index 9 is out of bounds for axis 1 with size 9
Unable to get angular_velocity attribute from coordinates. Error: index 12 is out of bounds for axis 1 with size 9
Unable to get velocity attribute from coordinates. Error: index 9 is out of bounds for axis 1 with size 9
Unable to get angular_velocity attribute from coordinates. Error: index 12 is out of bounds for axis 1 with size 9
appending material

however, when i try to load my original ~500k dsdna top and traj (~500 mb for 10 frames), then Blender bloats up to >20gb memory and becomes unresponsive. Either a memory leak, or a new feature to stream the oxdna trajectory frames from disk instead of loading everything into memory is needed.

i cant attach these big files here and i cant find your email, so if you go to my website alphataubio.com, you can see my handwritten email in a png at the bottom. i can send you a google drive link if you want to test. no rush however, i wont be using MN until at least august or september 2024 at the earliest.

focus on finishing your phd first ...

[ErikPoppleton]

Thanks for the tag. Sound like Molecular Nodes is loading the whole trajectory into memory instead of splitting it up. One thing you can do is use oxDNA Analysis Tools' minify script with the -d flag which reduces the precision on a trajectory file (1-2 is probably sufficient for visualization purposes). This can get you another 50% or more reduction in file size.

For both oxDNA Analysis Tools and oxView we implemented random-access trajectory readers which first scan the trajectory files for all t characters and stores their byte offsets as an array, they then only load a couple configurations into memory at a time and can jump to any specific configuration by checking the array. I'm not sure how the internals of Molecular Nodes look, but something like this might be something useful for dealing with large trajectories in general.

[BradyAJohnston]

thanks @ErikPoppleton for the details. Currently yes every frame is read into memory. Since I implemented this we have gained an MDAnalysis backend for MD trajectories that stream from the disk on similar to how you describe. I had a quick play around with it yesterday and was able to write an MDA trajectory reader for oxDNA that then allows streaming from the disk as well - so the issue can hopefully be addressed!