In-Allocation Management (Scheduled Conductor)

[FrankD412]

I think in order to make this design happen there's a couple vectors of discussion that need to happen.

1. Short term enablement -- what will it take to enable this in Maestro's current design?
2. Redesign for long-term support -- what changes now will make overall support better later?

Short Term Enablement -- The shortest term solution is the introduction of a new adapter class that takes in the batch settings as all adapters do, except it has a method to schedule a conductor (or arbitrary command). The adapter that's then used would be a resource enabled local adapter that uses the specified adapter to run the particular scheduler's MPI substitution. @jwhite242 -- you and I have discussed this one at length; basically, if Maestro's front end detects the use of this ResourceAdapter or ResourceManager, then it will use the specified type to scheduler the conductor call with a defined set of resources.

Long Term Enablement -- This suggestion might be a result of my current mental love affair with mix-ins... but I think the future enablement of this functionality is best handled by making a number of the data structures compositions of derived classes. @jwhite242's current closed test of the "local parallel" concept modified the ExecutionGraph to handle checks on available resources. In theory, this could be handled by constructing a specialized class from mix-ins making the API consistent and change the functionality without strictly inherited classes.

[FrankD412]

@jwhite242 -- I just thought about this feature a bit more, and I think we could actually go straight for a mix-in as a starting solution. We can make an allocation execution mix-in that overrides the execute_ready_steps method of the ExecutionGraph with your beta of the local parallel execution. Thoughts?

[jwhite242]

Given that mixins are generally stateless, can we really use them for all of it, or you thinking just a mix of inheritance and mixins. Meaning all the resource adapters would have to inherit the same resource data structures or whatever that the execution graphs' check resource mixin would have to be aware of in order to do the checking? Or am I thinking of something wholly different from what your mixin design would look like?

[FrankD412]

Sorry -- I was unclear. Yeah, that's not how I was thinking of it. I think the eventual solution will involve a mixin system for the MPI prefix (working on that now for pyaestro) and a mixin derived class that uses the local Executor. That would move the execution tracking away from the ExecutionGraph itself, and make it more consistent for the various use cases.

For a shorter term solution: If I'm remembering correctly, you implemented the resource check in the ExecutionGraph and had it use the throttle. So the mixin in this case would be used to make a derived class that uses your version of execute_ready_steps. We could also use some other mixins to tweak the method parameters if needed.

[FrankD412]

Referencing the comment from #345 :

@jwhite242 -- Some considerations based on why we were discussing earlier:

* I was hoping to eliminate the checks for local execution in the `ExecutionGraph._StepRecord`. This would bring all execution in line with scheduler-based adapters. You had mentioned some benefits to keeping the distinction around. What are your thoughts?

* We need to conceptualize how we would handle job packing in an Executor for pyaestro and then do something similar here. What have you found in terms of packing and the like?

So I was thinking this could be a useful distinction based on the two possible run modes: standalone batch jobs (the current scheduler adapters) and running maestro inside of an allocation manually for job packing (or spawned by another tool built on top of maestro). Doesn't necessarily need to use local as the distinction, but do we need some hook to enable the latter job packing mode?
As for the job packing, this is something where i think plugins would be really useful, or some way for users to write these things like with pgen. The number of scheduling behaviors and the optimization algorithms for implementing them is a pretty large space that doesn't necessarily need to be hard wired into maestro. A few simple/interesting ones would be good of course.

I was thinking about this and the case of running Maestro in an allocation would be where you have an "allocation adapter" class that comes in handy. That class would take in the global set of resources requested and schedule the conductor call. That's actually where I wanted to split out the MPI related functionality from the SchedulerScriptAdapter since you could either monkey patch the method in the LocalPoolAdapter to generate the right MPI or just have it in the batch settings to use a particular MPI (usual factory pattern there to get the class). We could mock that up in a discussion thread.

For this PR, I was thinking if I can get the local pool as the main local adapter (it would still return that it's local, but in the ExecutionGraph it'd be treated just like everything else since it's a process pool. The records would still return local otherwise.

I'm starting to wonder if this is a good time to introduce an orchestrate sub-command to handle the allocation clear so it's more transparent to the user? -- we should make a discussion on this haha

Hopefully this is making sense.