Question: dev workflow?

[vkz]

Hey Bogdan.

As I've been playing with koyo I found myself at odds with the usual way I'd write Racket code. What's the dev workflow story for koyo powered apps? I suppose this is a "function" or "compiler" vs long-running process or service development question, for which Racket approach of always restarting execution afresh feels kinda icky. If you are using Emacs with racket-mode then perhaps you find yourself in the same boat and maybe settled on a nice enough workflow?

Koyo of course takes it further than even DrRacket or racket-mode - it watches the filesystem and recompiles and restarts everything. That makes REPL pretty much redundant. Delays introduced by such recompilations and restarts are significant enough to make it annoying - computers are after all quite fast but perhaps Racket expander is still not fast enough. Where does REPL powered dev cycle fit here? How do you approach programming new "components" or adding or changing functionality of the existing ones? Adding new modules etc.

Thank you

P.S.: Also briefly touched that in greghendershott/racket-mode#472 (comment)

[Bogdanp]

I don't have a great answer for this, unfortunately.

My workflow for modifying components and pages is I work on those modules in isolation in the racket-mode REPL, reloading them (C-c C-k) after every change. When I'm satisfied with those localized changes, I switch to the browser where the automatic recompilation and reloading kicks in.

You could try adding --disable-recompile to turn off recompilation, which may help reload things faster in your case, but my experience has been that recompiling things explicitly amortizes the cost over time: change five modules and you only pay the compilation cost for each of the five on the first reload rather than on every subsequent one.

The code reloading parts of koyo initially worked by leveraging dynamic-rerequire. When a file changed, its dependency tree was traced and all of those modules would get re-required. Using that approach, reloading was fast, but I ran into a number of show-stopper issues like IO operations sometimes blocking for tens of seconds after (and only after) a reload and problems with changes to structs. While the former could probably be fixed by tracking down the IO issue in the web server or in Racket itself, the latter is by design because structs are generational.

[Bogdanp]

It looks like I misremembered how the reloading used to work. It didn't build a dependency tree of the module being reloaded. It built a tree of modules that depended on the module that changed, meaning that the struct thing might not have been a problem after all and it was only the IO issue that was giving me grief.

For posterity, here's the relevant code since it was never a part of this repo:

(define (build-dependents-tree mod)
  (define (local? mpi)
    (define-values (name _)
      (module-path-index-split mpi))

    (string? name))

  (define (find-dependencies mod)
    (for*/fold ([dependencies null])
               ([phase (module->imports mod)]
                [dependency (cdr phase)]
                #:when (local? dependency))
      (cons (resolved-module-path-name (module-path-index-resolve dependency)) dependencies)))

  (let loop ([dependents (hash)]
             [mods (list mod)]
             [seen (set)])
    (match mods
      [(list)
       dependents]

      [(list (? (curry set-member? seen)) mods ...)
       (loop dependents mods seen)]

      [(list mod mods ...)
       (parameterize ([current-load-relative-directory (simplify-path (build-path mod 'up))])
         (define dependencies (find-dependencies mod))
         (define dependents*
           (for/fold ([dependents dependents])
                     ([dependency dependencies])
             (hash-update dependents dependency (curry cons mod) null)))

         (loop dependents* (append mods dependencies) (set-add seen mod)))])))

(define (find-dependents root mod)
  (define dependents-tree
    (build-dependents-tree (simplify-path root)))

  (let loop ([dependents null]
             [modules (list (simplify-path mod))])
    (match modules
      [(list)
       (set->list (list->set dependents))]

      [(list mod mods ...)
       (define dependents* (hash-ref dependents-tree mod null))
       (loop (append dependents dependents*)
             (append dependents* mods))])))

(define (touch-dependents root mod)
  (for ([path (find-dependents root mod)])
    (file-or-directory-modify-seconds path (current-seconds))))

Every time a mod changed, it used to call (touch-dependents dynamic-module-path mod) (horrible, I know) followed by (dynamic-rerequire dynamic-module-path).

It may be worth experimenting with this at some point again.

[Bogdanp]

After racket/racket@14f0f86 and racket/racket@60ac39b, the dynamic-rerequire approach now seems viable and doesn't require the hacky touch-dependents stuff above (or it won't, beginning with Racket 8.3). Testing the approach on matchacha yields some pretty huge improvements.

[Bogdanp]

It's been over a year, but I think 0.9 addresses the two problems you brought up:

• reloading after a change is now significantly faster and
• it's now possible to start a system within a REPL via start-console-here or (require koyo/console/dev).

The latter is intended as an extension of the workflow I described in my first comment on this issue. You can work on a module in isolation using the racket-mode REPL, but also bring in any instantiated dependencies that it might need by calling start-console-here.

It's still not the load-then-redefine model of Clojure and CL, but I think Racket isn't suited to that model and with these changes in place, I think the turnaround time is quick enough that I'm happy with it.

Here's a quick demo of the two changes: https://www.youtube.com/watch?v=wWj7OPvXGgA

[AiziChen]

After koyo version 0.9:
While the code occurred errors because of I make code mistakes, and I have been fixed it now. koyo will recompile rather than reload.

[Bogdanp]

@AiziChen yes, that is normal. If there is an error during reload, it'll stop the application process, wait for changes and then restart it, which involves an initial compilation to amortize reload times later. If that turns out to be too annoying, it can also be improved.

[AiziChen]

@AiziChen yes, that is normal. If there is an error during reload, it'll stop the application process, wait for changes and then restart it, which involves an initial compilation to amortize reload times later. If that turns out to be too annoying, it can also be improved.

Is there a way to reload but no stop the application and recompile after a code error be correted?

[Bogdanp]

@AiziChen it's not possible right now. In general, there's a complication here, too, which is that the app needs to be recompiled in certain cases in order to work with reloading (reloading does not support module constants) and that's why koyo defaults to recompiling in those cases. Can you say more about why you would like to avoid recompilation in that case? Is it just too slow for you? Or is it something else?

[AiziChen]

@AiziChen it's not possible right now. In general, there's a complication here, too, which is that the app needs to be recompiled in certain cases in order to work with reloading (reloading does not support module constants) and that's why koyo defaults to recompiling in those cases. Can you say more about why you would like to avoid recompilation in that case? Is it just too slow for you? Or is it something else?

no, I just seen this feature can work on npm via 'npm run dev' , so I think it maybe support for koyo, too. I don't really need it. thanks for your answer!