Dependencies are unchanged, yet rule is rerun

[gergoerdi]

(Migrated from https://stackoverflow.com/q/69222477/477476 since my interpretation of Neil's comment is that he expects this to just be an output issue -- but it's not!)

I have a complicated setup where I generate a Vivado project file, and then run Vivado's synthesis tool to consume said project file and produce a bitfile. During this last step, Vivado updates some fields in the project file.

Below is a self-contained model of the situation:

import Development.Shake
import Development.Shake.FilePath
import Data.Time.Clock
import Control.Concurrent

outDir = "_build"

vivadoXPR :: IO ()
vivadoXPR = do
    putStrLn "vivadoXPR"
    threadDelay $ 1 * 1000 * 1000
    writeFile (outDir </> "xpr") "xpr"

vivadoBitfile :: IO ()
vivadoBitfile = do
    putStrLn "vivadoBitfile"
    threadDelay $ 1 * 1000 * 1000
    time <- getCurrentTime
    writeFile (outDir </> "bitfile") (show time)
    writeFile (outDir </> "xpr") (show time)

main = shakeArgs shakeOptions{ shakeFiles = outDir {-, shakeChange = ChangeModtimeAndDigestInput -} } $ do
    outDir </> "manifest" %> \out -> do
        alwaysRerun
        writeFileChanged out "manifest"

    outDir </> "tcl" %> \out -> do
        need [outDir </> "manifest"]
        writeFileChanged out "tcl"

    outDir </> "xpr" %> \out -> do
        need [outDir </> "tcl"]
        liftIO vivadoXPR

    outDir </> "bitfile" %> \out -> do
        need [outDir </> "xpr"]
        liftIO vivadoBitfile

What I would like to avoid is re-building xpr after a full build, since doing that invalidates all of Vivado's internal state, triggering a full re-synthesis. Note that none of xpr's dependencies change.

Building _build/bitfile twice with --trace shows the following:

$ rm -rf _build && \
  stack exec -- shake --trace --trace _build/bitfile && \
  stack exec -- shake --trace --trace _build/bitfile
% Starting run
% Number of actions = 1
% Number of builtin rules = 9 [FilesQ,DoesDirectoryExistQ,GetDirectoryContentsQ,AlwaysRerunQ,GetDirectoryDirsQ,DoesFileExistQ,GetDirectoryFilesQ,GetEnvQ,FileQ]
% Number of user rule types = 1
% Number of user rules = 4
% Before usingLockFile on _build/.shake.lock
% After usingLockFile
% Missing -> Running, _build/bitfile
# _build/bitfile
% Missing -> Running, _build/xpr
# _build/xpr
% Missing -> Running, _build/tcl
# _build/tcl
% Missing -> Running, _build/manifest
# _build/manifest
% Missing -> Running, alwaysRerun
% Running -> Ready, alwaysRerun
    = ((),"") (changed)
% Running -> Ready, _build/manifest
    = ((Just File {mod=0x540BBC70,size=0x8,digest=NEQ},"")) (changed)
% Running -> Ready, _build/tcl
    = ((Just File {mod=0x540BBC70,size=0x3,digest=NEQ},"")) (changed)
vivadoXPR
% Running -> Ready, _build/xpr
    = ((Just File {mod=0x178FA5A0,size=0x3,digest=NEQ},"")) (changed)
vivadoBitfile
% Running -> Ready, _build/bitfile
    = ((Just File {mod=0xECA7F588,size=0x21,digest=NEQ},"")) (changed)
Build completed in 2.01s

% Starting run
% Number of actions = 1
% Number of builtin rules = 9 [FilesQ,DoesDirectoryExistQ,GetDirectoryContentsQ,AlwaysRerunQ,GetDirectoryDirsQ,DoesFileExistQ,GetDirectoryFilesQ,GetEnvQ,FileQ]
% Number of user rule types = 1
% Number of user rules = 4
% Before usingLockFile on _build/.shake.lock
% After usingLockFile
% Chunk 0 [len 34] 01000100000000000000040000000100000001000000010000000000000000000000 Id 1 = (StepKey (),Loaded (Result {result = "\SOH\NUL\NUL\NUL", built = Step 1, changed = Step 1, depends = [], execution = 0.0, traces = []}))
% Chunk 1 [len 30] 0a000600000000000000000000000100000001000000cd02cc3700000000 Id 6 = (alwaysRerun,Loaded (Result {result = "", built = Step 1, changed = Step 1, depends = [], execution = 2.432e-5, traces = []}))
% Chunk 2 [len 73] 0800050000000f0000005f6275696c642f6d616e696665737414000000000000000000000072bc0b540a000000010000000100000001000000d6ea4439080000000400000006000000 Id 5 = (_build/manifest,Loaded (Result {result = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NULr\188\vT\n\NUL\NUL\NUL\SOH\NUL\NUL\NUL", built = Step 1, changed = Step 1, depends = [[Id 6]], execution = 1.87795e-4, traces = []}))
% Chunk 3 [len 68] 0800040000000a0000005f6275696c642f74636c14000000000000000000000072bc0b5405000000010000000100000001000000b1ef8d39080000000400000005000000 Id 4 = (_build/tcl,Loaded (Result {result = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NULr\188\vT\ENQ\NUL\NUL\NUL\SOH\NUL\NUL\NUL", built = Step 1, changed = Step 1, depends = [[Id 5]], execution = 2.70722e-4, traces = []}))
% Chunk 4 [len 68] 0800030000000a0000005f6275696c642f787072140000000000000000000000a2a58f17050000000100000001000000010000002c32803f080000000400000004000000 Id 3 = (_build/xpr,Loaded (Result {result = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\162\165\143\ETB\ENQ\NUL\NUL\NUL\SOH\NUL\NUL\NUL", built = Step 1, changed = Step 1, depends = [[Id 4]], execution = 1.0015311, traces = []}))
% Chunk 5 [len 72] 0800020000000e0000005f6275696c642f62697466696c651400000000000000000000008af5a7ec23000000010000000100000001000000263a803f080000000400000003000000 Id 2 = (_build/bitfile,Loaded (Result {result = "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\NUL\138\245\167\236#\NUL\NUL\NUL\SOH\NUL\NUL\NUL", built = Step 1, changed = Step 1, depends = [[Id 3]], execution = 1.0017745, traces = []}))
% Chunk 6 [len 50] 0000070000000000000000000000010000000100000000000000080000000400000002000000080000001e4500401e450040 Id 7 = (Root,Loaded (Result {result = "", built = Step 1, changed = Step 1, depends = [[Id 2]], execution = 0.0, traces = [Trace {traceMessage = "", traceStart = 2.0042186, traceEnd = 2.0042186}]}))
% Read 7 chunks, plus 0 slop
% Found at most 8 distinct entries out of 7
% Loaded -> Running, _build/bitfile
% Loaded -> Running, _build/xpr
% Loaded -> Running, _build/tcl
% Loaded -> Running, _build/manifest
% Loaded -> Running, alwaysRerun
% Running -> Ready, alwaysRerun
    = ((),"") (changed)
# _build/manifest
% Running -> Ready, _build/manifest
    = ((Just File {mod=0x540BBC70,size=0x8,digest=NEQ},"")) (unchanged)
% Running -> Ready, _build/tcl
    = ((Just File {mod=0x540BBC70,size=0x3,digest=NEQ},"")) (changed)
# _build/xpr
vivadoXPR
% Running -> Ready, _build/xpr
    = ((Just File {mod=0xDC183BD8,size=0x3,digest=NEQ},"")) (changed)
# _build/bitfile
vivadoBitfile
% Running -> Ready, _build/bitfile
    = ((Just File {mod=0x9F9C2120,size=0x21,digest=NEQ},"")) (changed)
Build completed in 2.01s

Note that this is not just a mis-reporting of the state of build/xpr! We can see from the output line vivadoXPR that the rule to produce build/xpr really DOES get re-run!

[ndmitchell]

If you run with lint enabled, it reports:

Lint checking error - value has changed since being depended upon:
  Key:  _build/xpr
  Old:  (Just File {mod=0x73B021ED,size=0x3,digest=NEQ},"")
  New:  File {mod=0x744BBF69,size=0x1F,digest=NEQ}

In particular, the file _build/xpr is written to by two different rules. That's not allowed. Are the fields that Vivado updates essential? Why doesn't those fields changing require things to be rerun? Should you perhaps be making a copy of the project file, which you pass to Vivado, so it doesn't corrupt the original?

[gergoerdi]

In particular, the file _build/xpr is written to by two different rules. That's not allowed. Are the fields that Vivado updates essential? Why doesn't those fields changing require things to be rerun? Should you perhaps be making a copy of the project file, which you pass to Vivado, so it doesn't corrupt the original?

So the setting here is that Vivado is a shitty proprietary vendor product and all its file formats are shitty as well. My real build system is trying to do pretty much the same as this model:

1. My main entry point to the toolchain is by calling a black box that takes a Tcl file and produces an initial project file foo.xpr. This step is not particularly expensive so it's OK if we end up with a solution that needs to re-do it.
2. Then, the actual build is done by another black box that operates on that project file. "Operates on" as in, it doesn't just read it and then produce a bunch of output files; ohhhoho no, that would make too much sense! No, what it does is it reads the project file, decides what to do based on the state stored in that file, then produces a bunch of output files while updating the project file...
3. The output of this whole process that I am interested in is the FPGA configuration bitfile. Pretty much everything between the Tcl file and the bitfile is incidental as far as I'm concerned.

My goal would be to be able to run actions further downstream (that depend only on the bitfile as input) without redoing the very slow second step.

One subtlety is that the location (directory and file name) of the project file is baked into the project file itself during the generation from Tcl. So it's not even easy to generate a "pristine" project file without messing up the current state of an existing project file, when there is one.

If writing to the project file from step 2 is not allowed without messing up Shake, what do you recommend for this situation?