Saturday, June 23, 2012

ClojureCLR 1.4 released with code gen redone

I just pushed version 1.4.0 of ClojureCLR.

This version matches all fixes and enhancements in the same version of Clojure/JVM/TheMotherShip.  It also  sports, at long last, the new non-DLR-based code generation phase for the compiler.  My earlier post titled "Code gen redo preview".  The benefits outlined in that post--smaller assemblies, faster startup, some speedup--are now available in this release.

In the preview post, I mentioned attempting to re-implement 'light compilation' in the new compiler.  The previous  (DLR-based) version of the compiler has a special compilation mode used during evaluation (but not during AOT-compilation) for any function that does not need to have its own class defined.   It used DynamicMethods, a lighter weight version of IL generation that can be significantly faster than methods generated to assemblies.

The way the DLR generated DynamicMethods allowed live constants to be embedded in a way that full compilation did not; as a result, there were some constructs that would load evaluated but would not AOT-compile.  It never happened to me, but I know of one person who got bit by that (and who asked me to leave light compilation out of this version).  That person need not worry -- I have not yet finished coding light compilation and it is not active in this release.

Does it matter?  Most likely not.  You could see some speedup loading very large files, such as clojure/core.clj.  However, I'm pretty sure light compilation was massively speeding up running the Clojure test suite.  For some reason, with the current version, if you start up ClojureCLR under the debugger and run the full test suite, msvsmon.exe, the Visual Studio Remote Debugging Monitor, goes absolutely nuts allocating memory.  It appears to max at about 6GB on my PC with 8GB.  One sees some serious thrashing at that point.  I have no idea why.  I have no problem compiling the entire clj source bootstrap code, which starts off evaluating core.clj and friends first and then going back to compile it, with not a hiccup from msvsmon.

So, at present, I run the test suite from the command line.  If I need to debug on the tests, I just load the test file in question and run the tests for it only.

If you see this kind of memory-hogging behavior in mvsmon.exe, let me know.

The near-term roadmap for ClojureCLR development, in no particular order, is:

  1. Catch up with Clojure 1.5.x-alpha changes, particularly the new work on reducers.
  2. Examine the perf of the test suite and see what's going on there.
  3. Work on a single assembly distribution of ClojureCLR (using nested assembly loading), following the trail blazed by aaronc and Ralph Moritz.
  4. Work on running ClojureCLR on Mono.  (Some very positive reports from Robert Johnson on his experiments).
  5. (Maybe) finish implementing light compile, for my own sense of fulfillment if nothing else.
The work on reducers on the JVM side has introduced version-specific implementation (using Fork/Join in JVM 7 or falling back to other means on earlier versions).   I suspect a similar 3.5/4.0 .Net distinction could be made to take advantage of the Task Parallel Library (or whatever it is called these days).  If you've played with TPL under ClojureCLR, I'd love to hear about it.

Cheers.






7 comments:

  1. During the June meeting of the Amsterdam Clojure group we looked at reducers, also under the covers. Java's Fork/Join and the Task Parallel Library seem to do similar things indeed.

    ReplyDelete
  2. Is there an available list of bugs reported against ClojureCLR? I am interested in using ClojureCLR for COM Interop; currently this works for a subset of interface data types.

    My specific interest is in using ClojureCLR as a front-end to computation engines but I note that there are a great number of applications that wrap COM objects, which might be alternatively wrapped by ClojureCLR scripts.

    ReplyDelete
    Replies
    1. You can find lists at http://dev.clojure.org/jira/browse/CLJCLR and at https://github.com/richhickey/clojure-clr/issues?direction=desc&page=1&sort=created&state=open

      I'm happy to deal with COM interop issues as you find them. I've not had a need and so haven't spent time making this work smoothly. There are probably some tricky issues on reflection to COM interfaces, judging by some of the special cases I've seen in the DLR code.

      Delete
  3. Hi, sorry in advance for what I'm sure is kind of a silly question but ... I've done some development with Clojure for the JVM where I use my primary development environment: lein + emacs. Is it possible to do the same with ClojureCLR? I realize lein is for JVM only but will clojure-mode, nrepl, etc., still work with ClojureCLR?

    ReplyDelete
    Replies
    1. I comfortably use emacs + inferior-lisp with ClojureCLR. It's a great combination. You basically just need to have some repl script (a batch file for instance) which sets up CLOJURE_LOAD_PATH for your project before the actual repl is started. We are working on a fork of nrepl https://github.com/aaronc/tools.nrepl/tree/clr) but it might be a little while before it's ready.

      Delete
    2. @aaronc, I would add:
      - set CLOJURE_LOAD_PATH in Windows system control
      - for inferior-lisp set in .emacs file:
      (setq inferior-lisp-program "Clojure.Main.exe")
      (add-to-list 'exec-path "C:\\path\\to\\Clojure.Main.exe")

      Delete
  4. Lein does not know how to work with ClojureCLR. Perhaps a plugin could be developed. However, there is a new lein-clr being developed. It might help. https://github.com/kumarshantanu/lein-clr

    You can use emacs, of course. If you are doing Clojure-only dev, that will probably work fine. If you are trying to combine C# and Clojure development in a single project -- good luck with that. There is a vsClojure plug-in. Reports are mixed on how well this is working at the moment.

    ReplyDelete