To start, who are you and what is your background?
I’m a 50 year old programmer. I studied at Ecole Normale Supérieure in Paris where I obtained a doctorate in Physics. I also have an engineering degree from Ecole Nationale Supérieure des Télécommunications in Paris.
Before Sage, I worked for two startups. The first one was called Neuron Data. I participated to the development of the company’s first product, an Expert System Shell called Nexpert Object, in the late 80s. I was at the origin of two of the company’s later products: a cross platform UI toolkit called Open Interface and a business rules engine called Blaze Advisor. I left the company in 1999 and joined a small startup called Ubiquis where I developed an e-commerce application closely integrated with Sage’s accounting products. We sold the company to Sage in 2000.
My programming experience is mostly with C, Java and C# but I’ve been exposed to all sorts of O-O languages: Smalltalk, Objective-C, C++, Eiffel, etc. I was fluent in emacs-lisp at some point and I also wrote stuff in esoteric AI languages like OPS5. I worked on various systems: Unix and Windows of course but I also did a lot of development on VMS and some on Macintosh.
What does Sage do and what is your role there?
Sage develops business software for small and medium size businesses. We have many products because Sage has grown by acquiring local software vendors. About 18 months ago I took the lead on the architecture of our global ERP product, Sage ERP X3.
Sage is a very decentralized company and product architects have had a lot of autonomy in their technology choices in the past. Things are starting to change though as we are more and more faced with global challenges. Sage is not considering SSJS as a global platform at this time as it is too immature for the type of applications we are building. But, despite its conservative image, Sage is open to innovation and is supportive of the choice that I made to use SSJS for a project around Sage ERP X3. If this project is successful (and I think it will be), SSJS may be considered for other projects.
The benefit is not just in using the same code, it is also in having the same language, even for code that is not shared. This way we can share expertise, best practices, etc.
And the problem exists in the browser too, as we start to develop more complex applications and put thicker business logic in the browser to support offline scenarios.
How does streamline.js solve this problem?
Streamline.js solves the problem with weaker mechanisms that don’t let threads creep in. Streamline.js solves the problem with a preprocessor that transforms the code. Basically, streamline lets you write your code as if the APIs were synchronous and gives you a special placeholder parameter (an underscore) that you pass everywhere a callback is expected.
You have the choice of running the preprocessor before executing your program, in which case your program won’t need to embed the streamline transformation engine, or you can embed the engine and let the node.js “require” infrastructure invoke the transformation as it loads your modules.
From a more theoretical standpoint, streamline performs a CPS (Continuation Passing Style) transform. I am not a specialist of CPS transforms but I think that two things differentiate this CPS transform from other CPS transforms. The first one is that the transformation is partial: only the code flows that contain asynchronous calls are transformed. The second one is that it is based on an algebraic application of patterns (the patterns being the patterns that I discovered progressively when I was still writing the callbacks by hand). These two characteristics explain why streamline.js somehow “writes the callbacks for you” instead of turning the entire code upside down like some other CPS-based tools do.
Also, streamline comes with some goodies to facilitate asynchronous programming. For example it provides “futures” to let you parallelize I/O operations.
This underscore argument, is that just there for technical reason (to detect which functions should be called asynchronously) or does it serve another purpose as well?
The first version that I published on GitHub did not have the underscore argument. Instead, I had used a different marker: an underscore at the end of the function name. I introduced the underscore argument shortly after, when I did the CoffeeScript adaptation. As CoffeeScript only generates anonymous functions, and as I did not want to get dragged into hacking compilers, I changed the syntax and introduced the underscore parameter/argument.
And this was a lucky move! First, this syntax translated a key property of asynchronous functions, i.e. the fact that asynchronism is “contagious”, into a simple scoping rule: if a function calls an asynchronous function, it becomes asynchronous itself (unless you don’t care about completion of the sub function). This translates into the fact that you may only pass the underscore argument from a scope where the underscore is defined (as a parameter of the current function). The only violation of this rule is for top level calls in a script and there is another gotcha: the underscore parameter must be in the current function, not one of its ancestors in the scope.
But the syntax also had some practical benefits: it allows streamline.js code to call node.js functions in which the callback is not the last parameter; it also makes it easy to design functions that have optional parameters, by putting the callback in first rather than last position. And, it also allows allowed me to introduce a special syntax (__wrapXxx(_)) for wrapper functions that adapt the callback for APIs that do not adhere to the standard node convention (for example callbacks that don’t take an error parameter).
And more recently, I found a really cute way to leverage this underscore parameter. I was investigating ways to initiate several asynchronous operations in parallel and being able to join them later, without introducing much extra syntax or heavy libraries. I knew about “promises” and “futures” and I had the idea that asynchronous functions could return a “future” when called without a callback. So if
foo(arg1, _) is an asynchronous function, calling
f = foo(arg1) would return (synchronously) a future, which could be used later as
f(_) to retrieve the result. In CS jargon, this translates into: “futures are obtained by currying the callback away”. So, and this is were I was really lucky with the underscore parameter design, futures came almost for free: “if you omit the underscore argument you get a future”.
Yes, it could be seen as a leaky abstraction, but actually, is this really worse than working with an asynchronous helper library? If you use a helper library and something goes wrong, you often end up stepping into the library, and then you have to understand the internals of the library. With streamline.js, you step through code in which you easily recognize the code that you have written yourself and the callback “decorations” that streamline has added around it. So, stepping through may actually be easier because you are not jumping between your code and a library, you have everything under your eyes in a single file.
Also, I would reformulate your statement a bit: to debug streamline code, you do not have to understand “what the streamline preprocessor does exactly”, you have to understand “the code that it generates”. This is much easier.
I’d be careful with the term “leaky abstraction” though. Joel gives a subtle definition but a lot of forum posters are abusing it to easily disqualify things that they dont’t like. There is an obvious association with memory leaks, leaky engines, etc. In that sense, I do not consider streamline.js to be leaky. It may have emerged from a somewhat pragmatic investigation but it is based on an algebraic application of patterns and I think that it would be possible to write a formal proof that this transformation does not distort semantics (some of the limitations that I mentioned in the wiki, like the fact that the order of subexpression evaluation is not always preserved, are not too difficult to lift but lifting them would increase the size of the generated code and I chose not to do it).
I haven’t investigated if debuggers could be modified to operate on streamline source rather than on the generated code. Will Conant has contributed a feature that maps the output lines to the source lines, which might be a first step. But there is probably hard work to get to a really transparent debugging experience.
How do streamline.js’s futures and “goodies” compare to the libraries to facilitate async programming already out there?
There aren’t that many goodies actually.
The first set is just an async version of the ECMAScript 5 array methods (forEach, map, filter, every, some, reduce, reduceRight). They are probably equivalent to what you can find in other libraries. The only special thing is that the callback is passed as first argument rather than last, which works better in this case because the functions have optional arguments.
Then, there is a special function that I really like and that I called “funnel”. It allows you to control the number of concurrent executions over one or several code blocks. It can be used to limit the level of parallelism and avoid exhaustion of system resources (for example you quickly run out of file descriptors if you blindly parallelize recursive traversal of directories). It can also be used to set up “critical sections” by setting the limit to 1. I’m also thinking of introducing a variant that would handle exclusion between one writer and multiple readers.
The “futures” feature compares to futures found in promise/future libraries (Krys Zip and Kris Kowal are experts here). But there is a big difference because streamline only provides futures and it does not provides them as classes with methods. Instead, every function that you write with streamline will return a future if you call it without providing a callback; and the future itself is just an asynchronous function that returns the result via a callback. So, there is no special API to learn.
I just introduced another set of goodies to wrap node.js stream objects. The idea behind these wrappers it to let the consumer of the stream “pull” the data by calling a “read” method instead of having the stream “push” the data by emiting events. There are libraries that ease the work with node’s streams but I haven’t seen any that takes the radical approach of “inverting the flow” completely and exposing “read” methods instead of events. I think that this API style blends really well with streamline.
One thing that streamline does not provide is libraries to chain asynchronous calls. There are a lot of them around but they simply don’t apply to streamline because the chaining problem is solved differently, by a CPS transform.
Overall, I try to avoid introducing large APIs. I try to keep the APIs minimalist and very thin. Also, I have an unfair advantage over people who implement other helper libraries because I write them for streamline source. So I don’t have to constantly fight against callbacks; the CPS transform moves them out of the way which removes a big thorn. For example it becomes very easy to design functions that chain with each other.
There are a few fans who praise it and some who have contributed but I also got a lot of negative feedback. For example, the first response I got when I announced it on the node forum was something like “callbacks are just fine, we don’t need any extravagant framework like this one” and it was followed by a series of “+1” posts. On one hand, this was a bit of a surprise to me because I was expecting that people were actually having a hard time with callbacks and that they would be happy to have an alternative but this was not the case in general. On the other hand, I’m an old programmer and I’ve seen many ugly religious debates. Programmers will always be programmers!
I’m nevertheless a bit concerned about this negative feedback that I got from the node community. I’m not concerned because of streamline itself but rather because of node.js. I really think that node.js is a great technology: simple, based on simple innovative principles, and very fast, but I think that the entry ticket is way too high today. There is a lot of buzz around node.js today but I would not be too surprised to hear some discordant voices soon. What concerns me is that node.js has the ambition of being the next PHP or the next RoR but it lacks the “basic spirit” of the former and the “structuring nature” of the latter. If we want this to happen, we need to work on the “entry ticket” problem. So, to me, people who stick to the “programmers just have to learn how to program with callbacks to deserve node.js” mantra do a great disservice to node.js. Node.js cannot become a mainstream platform without lowering its “entry ticket”. Maybe streamline.js is not the answer but maybe there are some good ideas to pick from it.
* [streamline.js on GitHub](https://github.com/Sage/streamlinejs)
* [Bruno’s weblog](http://bjouhier.wordpress.com/)
* [Bruno on twitter](https://twitter.com/bjouhier)