Parca for vivarium-ecoli [brainstorm thread]

[1fish2]

[I'm starting this thread with notes from the team meeting on March 19, 2021. Feel free to add ideas to it.]

The vivarium-ecoli project uses a snapshot of the Parca's output rather than replicating that code. However:

• We don't have a good way to tell when vivarium-ecoli needs an updated snapshot.
• The Parca's output was only intended to use with the contemporaneous wcEcoli code, while ultimately we'd like to generate suitable parameter data each time we reconfigure the vivarium-ecoli processes. Maybe some parts of the data should get recalculated via parameter sweeps using the vivarium-ecoli processes.

Ideas:

• Review and trim down the network of objects that get saved when pickling a sim_data to make it less dependent on the current wcEcoli code.
  • Move some instance variables to class variables for values that apply uniformly to all instances and are pretty stable, e.g. self._compartment_tag = re.compile(r'\[[a-z]]').
  • Implement pickle protocol methods for the relevant classes so pickling a sim_data saves only the defining state; no derived objects such as caches.
  • Maybe include less of raw_data contents (which is saved in rawData.cPickle).
• Evolve the sim_data output format to more explicitly-designed and longer-lived, probably on a pickle, JSON, or sqlite file format. Do this incrementally by writing suitable pickle methods for the relevant classes.
• Add a format version ID so the reader can detect when a saved sim_data needs replacing, assuming we update the format version ID at the right times. Some kind of automatic compatibility checksum would be nice but how to compute that?