Memory usage with Detector1 pipeline

[stscijgbot-jp]

Issue JP-3610 was created on JIRA by Maria Pena-Guerrero:

Several Help Desk tickets have either directly stated or it has been found  through investigation that the Detector1 pipeline uses too much memory. 

Probably both repos the jwst and stdatamodels would have to be modified. In the stdatamodels repository, the culprit seems to be the open function, which when given a datamodel to open creates a copy or "clone" but the original is still referenced so the previous object cannot be garbage collected. Hence incrementing the number of copies and memory used in the different steps of detector1 on the jwst repo. For example, calling model2 = RampModel(model) (as is done at the beginning of detector1, and during charge_migration, etc) will create a new model ({}model2 is not model{}) however it will reference the same data ({}model2.data is model.data{}). Which links the primary memory load for both model and probably more importantly the underlying asdf objects are the same ({}model2._asdf is model._asdf{}). As the _asdf tree references effectively everything in the model their lifetimes are linked which may contribute to the growing memory load.

Here are some of the tickets in question (more may be added as they show up):

• INC0198744
  https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D467a901f47340a10ec5b9448436d43fe%26sysparm_stack%3Dincident_list.do%3Fsysparm_query%3Dactive%3Dtrue%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0198484 
  https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D58e176349734829023a2fe971153afb3%26sysparm_stack%3Dincident_list.do%3Fsysparm_query%3Dactive%3Dtrue%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0189965
  https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D190aab8b97b2ad5023a2fe971153afad%26sysparm_record_list%3Dassignment_group%3D8dbea402dbf2620033b55dd5ce9619ba%5Estate%3D2%5EORstate%3D3%5EORstate%3D1%5Ecompany%3D%5EORwatch_listCONTAINSa89f418ddb8f620042685434ce961990%5EORcompany%3Dc3478b5edb013e0042685434ce961951%5EORcaller_id%3Da89f418ddb8f620042685434ce961990%5EORopened_by%3Da89f418ddb8f620042685434ce961990%5EORDERBYopened_at%26sysparm_record_row%3D10%26sysparm_record_rows%3D52%26sysparm_record_target%3Dincident%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0198744 
  https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D467a901f47340a10ec5b9448436d43fe%26sysparm_record_list%3Dassignment_group%3D8dbea402dbf2620033b55dd5ce9619ba%5Estate%3D2%5EORstate%3D3%5EORstate%3D1%5Ecompany%3D%5EORwatch_listCONTAINSa89f418ddb8f620042685434ce961990%5EORcompany%3Dc3478b5edb013e0042685434ce961951%5EORcaller_id%3Da89f418ddb8f620042685434ce961990%5EORopened_by%3Da89f418ddb8f620042685434ce961990%5EORDERBYopened_at%26sysparm_record_row%3D38%26sysparm_record_rows%3D49%26sysparm_record_target%3Dincident%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0187520
  https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3De7137d954719e510ec5b9448436d4315%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
•