Use file-watching mechanism

[mitchellwrosen]

I managed to get a shake watch working with this:

import Control.Concurrent (newChan, readChan)
import Control.Exception (SomeAsyncException(..), fromException, throwIO, try)
import Data.Functor (void)
import Development.Shake
import System.Directory (getCurrentDirectory)
import System.Environment (getArgs, getEnvironment, withArgs)
import System.FSNotify (eventPath, watchTreeChan, withManager)

import qualified Data.Set as Set

main :: IO ()
main =
  -- Get the command line args before shake sees them. This is not ideal because
  -- you must run e.g.
  --
  --   shake watch -j2
  --
  -- as opposed to
  --
  --   shake -j2 watch
  --
  getArgs >>= \case
    "watch":args ->
      -- Strip off the "watch" arg for the real shake script.
      withArgs args $ do
        -- Run the real main under 'try', ignoring synchronous exceptions
        -- because we don't care if the build fails - we want to watch & rebuild
        -- regardless. Re-throw async exceptions.
        try realMain >>= \case
          Left (fromException -> Just (SomeAsyncException ex)) -> throwIO ex
          _ -> pure ()

        -- Get the list of files that shake considers to be alive. This assumes
        -- we've set
        --
        --   shakeLiveFiles = [".shake/live"]
        --
        cwd <- getCurrentDirectory
        files <- Set.fromList . map (cwd </>) . lines <$> readFile ".shake/live"

        -- Start watching the filesystem, and rebuild once any of these files
        -- changes.
        withManager $ \manager -> do
          chan <- newChan
          void (watchTreeChan manager "." ((`elem` files) . eventPath) chan)
          void (readChan chan) -- We block here

        -- Loop. Here my compiled shakefile is itself a build target, so I exec
        -- it rather than loop here, to get the latest & greatest shake
        -- executable.
        env <- getEnvironment
        executeFile "bin/Shakefile" False ("watch":args) (Just env)

    _ ->
      realMain

It works okay, but I really don't want to keep writing this code :)

Should/could something like this live inside shake? shake --watch?

[ndmitchell]

Yes/no/maybe... Yes, I'd love to include something like that. No, I don't want to incur a file watch dependency because it complicates efforts such as the GHC Hadrian work. Maybe, because if we do, I want it to be a lot smarter and "know" what files Shake really cares about. Thoughts about how we solve the "No"?

[mitchellwrosen]

Oh, interesting. Sounds like this feature belongs in some higher-level package built on shake that could be more liberal with its dependency footprint. Boilerplate language-specific rules could go in there, too (for example, to build Haskell projects, I have been using //*.hs for finding relevant source files, but this could be made a lot more accurate by parsing the .cabal file to identify the modules that are actually part of the package).

And re: "maybe" - do you mean that shakeLiveFiles isn't quite sufficient for this use-case? If so, I think I agree. For example, my snippet above won't trigger a re-build if a new file that matches some existing glob is added.

[ndmitchell]

Yep, seems like a higher-level package is the way to go. It could also provide profiling without having the source files available, the TemplateHaskell snippet in https://hackage.haskell.org/package/shake-0.16.3/docs/Development-Shake.html#v:getHashedShakeVersion etc.

I think shakeLiveFiles is a crude approximation, and it gets written to a file, so I think you want an API entry point rather than a batch mode as we have now.

[mgajda]

Any progress on that? I would love to use watch mode in my builds.

[ndmitchell]

Progress is being made. There is Development.Shake.Database a few weeks ago with shakeLiveFilesDatabase, so you now probably have enough that you can write something robust on the outside. If someone does write shake-watch or similar I'll certainly add a link to it.

[ndmitchell]

Merging this issue with #580. Possibly makes sense to provide integration with Watchman as the file watching mechanism: https://facebook.github.io/watchman/

[srid]

Possibly makes sense to provide integration with Watchman as the file watching mechanism: https://facebook.github.io/watchman/

Why watchman instead of plain ol' fsnotify? Is the latter not fast enough for ghc code tree?

[ndmitchell]

@srid I'm not fussed either way. I suspect whoever does the work will end up setting the direction here. I suspect both are workable solutions.

[chrismwendt]

I'm implementing file watching in #839