Is it possible to identify a Rule (or Cmd) in a "textual" way?

[saurabhnanda]

When a lot of commands are executed in parallel, the STDOUT of the entire process becomes an unintelligible mess. I"m trying to setup a central way of organizing the output of all the build steps.

I found shakeCommandOptions that could possibly help in this scenario (by passing [FileStdout fstdout, FileStderr fstderr], but it's going to end-up using the same two files for ALL commands.

It might be possible to have the conceptual equivalent of shakeCommandOptions :: Rules () -> [CmdOption], so that one could change the files based on the rule being evaluated, but that leads to three questions:

• is such a change acceptable? is there any other way in which the same result can be achieved?
• if we are talking about a change on these lines, how does one write a function that goes from Rules() -> FilePath (ref: this issue's title - how to identify a Rule in a "textual" way)?
• is parallelism implemented at the Rule level or the Cmd level? If the latter, this could still result in interleaved output; and then the question would change to "how to identify a Cmd in a textual way"

PS: What exactly is the "unit of external computation" that Shake uses internally? how does that one associate it with the a Rule, or Cmd, or output file (whatever is appropriate)?

[saurabhnanda]

As a continuation of the original question, I discovered shakeTrace and traced. Is there any reason why:

1. shakeTrace doesn't also include another argument to indicate whether the action being traced was successful, or not?
2. And in case it threw an exception, also communicate the exception to shakeTrace?
3. Provide an extra argument (apart from "key" and filename) to identify the "rule"

As a continuation to point 3 above, I'm realizing that there is some impedance mismatch between my mental model and how Shake has been built (and this could be related to the problems that I've described in #758):

• My mental model is similar to gulp (from the JS world), or capistrano (from the Rails world), where you have a bunch of build steps with dependencies between them. You specify which top-level build step you want, and the build-system executes each dependent step in the appropriate order.
  • This would allow the output of each build step to be recorded & reported separately -- even it it involved multiple different commands.
  • This would also allow the success/failure of each build step to be reported separately
  • Usually the number of build steps would be few (say, less than twenty)
• It seems that shake's model is completely different. You specify which file you want to build / generate along with the file's dependencies. Then, based on the rules you provide, shake figures out the rest.
  • Does this mean that a rule like "*.css" %> \out -> -- compile SASS file here, is a single rule? Can it be given a name such that it approximately corresponds to a build step (from the gulp/capistrano world), called compile all SASS files? Or, internally, is it treated as N different rules, where N = number of CSS files that *.css matches?
    • From the screenshots in Progress tracking interface could be better? #771 it seems that each matching file is treated as a separate rule. If that's the case, what entity represents "*.css" %> \out -> ... independent of the files it matches?
  • The same rule may end-up being triggered during the build process of completely different top-level targets. Is the internal book-keeping (a) separate for each rule, (b) separate for each command ("traced action"), (c) separate for each top-level target, or (d) some combination of the above?

[ndmitchell]

Have you seen EchoStdout - that would be a way of stopping everything going to stdout.

As to your question, Rule () are a set of rules, so they don't have any unique designation, and don't correspond to just one rule in many cases. Functions such as %> take an Action () and produce a Rule (). At their most primitive, these correspond to addUserRule, which doesn't take any naming information.

Usually parallelism operates at the Rule level, but within an Action, you can use the parallel function.

Each Rule is identified as a Key. So shakeTrace seems to do exactly what you want there. I agree that passing failure information or exceptions would be possible - but currently shakeTrace is not bracketing - if there is an exception it doesn't get called on the way out. To do so would require setting up stack frames, more stack usage, more time spent.

You might want to read through https://hackage.haskell.org/package/shake-0.19.1/docs/Development-Shake-Rule.html, which gives a fairly low-level view on what a Shake rule is.

I think the Shake model is mostly like you describe in gulp, but with the oddity that in many cases build steps are specified by files. But they equally don't have to be. There is a single rule type (at the addBuiltinRule level) which describes files, one for oracles, one for does-file-exist etc. Then there are user rules (with addUserRule) that can be matched on by a builtin rule to see how it should be customised. So "*.css" %> is a user rule, but as far as Shake is concerned, its not a thing that Shake deals with - builtin rules may decide to use them if they want (and the file builtin rule definitely does). Shake is concerned with keys, which might be "File foo.css", and that's the key that exists as a dependency in Shake.

[saurabhnanda]

Thanks for the question, and sorry it's taken so long to reply - personal stuff cropped up.

I'm already using that, but how do I capture the output of every individual command in a separate handle/pipe/file?

Let me rephrase my question to be more actionable, rather than being theoretical ==> In a parallel build (multiple rules/action/commands being executed in parallel), one, or more, commands fail. How does one display that particular rule/action/command's stdout & stderr in a fashion that makes sense to the user and allows him/her to debug things sensibly?

Each Rule is identified as a Key. So shakeTrace seems to do exactly what you want there. I agree that passing failure information or exceptions would be possible - but currently shakeTrace is not bracketing - if there is an exception it doesn't get called on the way out. To do so would require setting up stack frames, more stack usage, more time spent.

This was just an alternative that came to my mind because Shake doesn't seem to allow capturing the stdout/stderr of a rule/action/command at a granular level. What do you think of changing FileStdout FilePath to FileStdout (key -> FilePath) and similarly for FileStderr?

[ndmitchell]

How does one display that particular rule/action/command's stdout & stderr in a fashion that makes sense to the user and allows him/her to debug things sensibly?

Is the information you are looking for in the exception message? Is being in the exception message sufficient?

(Regarding the key -> FilePath, there are a lot of variations on that, so happy to discuss those, but probably best to focus on one thread of solution, and then circle back if its not enough)

[saurabhnanda]

Is the information you are looking for in the exception message? Is being in the exception message sufficient?

for my current use-case the information in the exception may be sufficient. But I'm also thinking about my (ambitious) project - https://github.com/saurabhnanda/shake-ui. For a general terminal-UI for shake, I'm envisioning a viewing-pane for build errors which display the complete stdout+stderr from the specific rule that failed without interspersing it with other outputs.

Sometimes an exception makes sense only with a log of all the actions/commands that led up that that exception.

[ndmitchell]

I guess if you did had the key -> FilePath, would you put them in a temp dir, and have shakeTraced scan that directory to find them? I guess what's the plan to go from that to the more general UI?

[saurabhnanda]

I guess if you did had the key -> FilePath, would you put them in a temp dir, and have shakeTraced scan that directory to find them? I guess what's the plan to go from that to the more general UI?

Yes, provided that shakeTrace is also called for failures, which is why I said:

shakeTrace doesn't also include another argument to indicate whether the action being traced was successful, or not?
And in case it threw an exception, also communicate the exception to shakeTrace?

[saurabhnanda]

@ndmitchell Here's how I'm thinking of the general UI:

1. Change CmdOption (possibly in a backward compatible manner):

type RuleKey = String

data CmdOption = FileStdoutKey (RuleKey -> FilePath)
               | FileStdErr (RuleKey -> FilePath)
               | ...

2. Use these newly created CmdOtptions and also piggy-back on shakeTrace:

runBuild = do
  let myShakeOptions = shakeOptions
      { shakeThreads = 0
      , shakeCommandOptions = [FileStdoutKey (ruleLogFile "out"), FileStderrKey (ruleLogFile "err")]
      , shakeTrace = ruleTrace
      }
  shakeArgs myShakeOptions $ do ...

ruleLogFile :: String -> RuleKey -> FilePath
ruleLogFile ext key = "build-outputs" </> ext </> key

ruleTrace :: RuleKey -> String -> Bool -> IO ()
ruleTrace key c isStart = appendFile (ruleLogFile "out") $ "[ " <> (if isStart then "START" else "STOP") <> " ] " <> c <> "\n"

Concerns:

• Should this be FileStdoutKey (RuleKey -> FilePath) or FileStdoutKey (RuleKey -> Handle)? Is using appendFile performant, or should ruleTrace use the underlying Handle directly?
• Will this infra automatically log exceptions and errors?

What do you think? Should I attempt a PR for this?

[saurabhnanda]

@ndmitchell bump

[ndmitchell]

I'm still a little concerned about putting the rule key into the CmdOption in that way - it feels wrong (I'll try and elucidate more when I've thought about it a little more). The concrete ask is that you want to produce a UI such that you can see which commands were run by which actions, and for any command see the stdout/stderr?

[saurabhnanda]

The concrete ask is that you want to produce a UI such that you can see which commands were run by which actions, and for any command see the stdout/stderr?

At a UI level the "grouping" of stdout/stderr will either be at a cmd-level, or at a rule-level. I need to determine that based on practical.usefulness.

[omnibs]

I'm running into a similar need to @saurabhnanda's, I think, tho I haven't done as much homework as he did.

We moved from a less efficient system full of shell scripts and a large Jenkinsfile that made Jenkins' Blue Ocean look like:

To a more efficient one based on shake, that looks like:

The problem is those 20+ build steps output to stdout/stderr all at the same time, making debugging unapproachable for folks not intimately familiar with our shake code.

My needs differ a bit from his in how I want to use the data, I guess, but the data itself would be the same: either a hierarchical grouping of stdout/stderr by Rule->cmd, or at flat one by Rule.

[ndmitchell]

The problem is those 20+ build steps output to stdout/stderr all at the same time, making debugging unapproachable for folks not intimately familiar with our shake code.

Could that be solved by simply buffering the stdout/stderr? That's a much more localised change.

At a UI level the "grouping" of stdout/stderr will either be at a cmd-level, or at a rule-level. I need to determine that based on practical.usefulness.

In most systems there tends to be at most one cmd per rule, which makes me think the rule level might be sufficient. Although grouping command information with commands is much easier to do, since its more localised.

[omnibs]

Could that be solved by simply buffering the stdout/stderr? That's a much more localised change.

I'd definitely help, but I'd still look for ways to improve visibility over our pipeline.

In most systems there tends to be at most one cmd per rule

We might be using shake in an unorthodox manner then I guess, and we're using it a lot (14k LoC of shake or shake-adjacent code so far). One example is we run aws cli calls for setting up auth context, crane for pushing and tagging images, helm for deploying, plus a few kubectl calls, all in a single rule, to do blue/green deploys in kubernetes.

In this scenario, it'd still be difficult to understand what's going on if we only buffer per cmd. We deploy a little over half a dozen services at the same time and outputs would still get mixed up between different services.

I guess my needs would be totally satisfied if we had a view with rules -> cmds -> logs of all types, in shake's report.html, but it might too ambitious as a first step towards solving this :S