Hello Again, Linux

Wed 24 September 2014

It’s about 8 years ago that I decided to buy my first Mac. At the time I was using PCs with Windows and Linux. My first Mac was an iBook G4, a 1Ghz PowerPC. I loved it. Everything was so fresh and new. Since then I’ve always used Mac laptops.

October 1st I’m starting a new job at a Python company called STX Next here in Poznań. For the past half a year, I’ve been working on Zed close to full-time. I enjoyed it, but I decided I wanted a change: I wanted to be surrounded by colleagues rather than sitting in an office by myself with my nearest colleague 1000km away. Over the past three years I never experienced that longer than a few weeks — I always worked remote. That’s going to change now.

STX Next is not a Mac company. They do a bit of iOS development, but not much. Everybody’s using Linux, essentially. I decided I wanted to fit in, and was ready for a change anyway, so decided to buy a PC laptop.

I forgot what it was like to decide on a PC to buy. The amount of choice is insane, and the trade-offs you have to make are very tricky. This one has a faster processor, that one has more memory, that one has a better screen, and the other one has a better trackpad. Also, will everything work under Linux? In the Apple world things are simple. If you decide on a laptop you have roughly two choices with some minor configuration possible. In the PC laptop world… you have a dozen brands each with seemingly dozens of models with unpronouncable names.

After a lot of research I decided on a Lenovo Thinkpad t440s (catchy name, huh?). Tada:

I’ve been using it full-time for the past two weeks, running Ubuntu GNOME 14.04 and I’m pretty happy with it. A lot of stuff I expected to cause trouble just worked. Hardware-wise basically every part of my Thinkpad is supported, although the fingerprint scanner software crashes a lot. When I plug in my iPhone it is automatically detected and I can access music and import pictures.

Over the past years I haven’t used Linux much for anything else other than servers. I hadn’t really kept up with how desktop Linux evolved at all. Similar to the hardware story, there’s an insane amount of choice in the Linux world: there are many distribution, and within a Linux distribution you have to choose a desktop environment to use. After using Ubuntu’s default “Unity” for a while, I switched to GNOME 3, and a week and a half in — I’m pretty happy.

My theory, as a long-time smug Mac user was that Linux had always been about copying and design-by-comittee (or design-by-community), which seemed inherent to the open source development model with lots of parties involved. Implying: much in the Linux space is a giant compromise with little original thought, especially on the desktop. And while in many ways that’s true, there are some exceptions. GNOME 3 is surprisingly daring and different. It borrows many elements from existing environments (including OS X), but it feels pretty different.

With some tweaking — but not that much — here’s what I’m looking at now:

gnome-3-1

(Indeed, winter is coming.)

What I like is that the amount of space used by the environment itself is pretty minimal. By default it’s just the bar along the top, I enabled the window list at the bottom myself (although I’m still not sure I’ll keep it). So how do you launch applications? You hit the Windows key (yep — I got one of those now). When you do that, this happens:

gnome-3-2

This is a mix of OS X’s Expose, the dock (on the left) as well as a quick launcher. To launch an application, simply start typing its name. Want to activate another open window? Click it. Launch one of your favorite applications? Click it on the dock. All behind the Windows key. Pretty good idea, IMHO. Did I mention that you can set Alt-Tab to switch between all windows, not just applications? Finally.

Desktop application-wise Linux doesn’t compare very favorably with Mac and Windows. The choice is much more limited, desktop apps are often (but not always) of poor quality, more difficult to use and ugly. Big vendor applications typically lag behind their Windows and Mac counterparts (e.g. Skype and Spotify). My solution is to use web and Chrome apps for mostly everything.

Is this ideal? No, but it’s workable. Also, it means that a move to ChromeOS wouldn’t be a giant leap.

Although this may appear risky — as a file system I opted for btrfs. Btrfs is Linux’ next-generation file system, on par in many ways with ZFS. The big advantage I get from it is an efficient way to do back-ups. I have an external 2TB hard drive also with btrfs, and use a simple bash script running as a daily cronjob, that takes snapshots of the filesystem, and only transfers the diffs to my backup drive. This is way more efficient than using backup software that scans your whole drive for changes and transfers changed fles. Of course there’s the risk of using btrfs on two places, I’m thinking how to do offsite backups safely too.

So far so good.

The March Towards Go

Fri 04 July 2014

marching

Today, TJ Holowaychuk, one of node.js’ most prolific module writers (express, jade, mocha, koa) has announced he’s leaving node.js for Go:

I’ve been fighting with Node.js long enough in production now that I don’t enjoy working with it anymore unfortunately, so at least for now this my formal farewell!

[…]

Node does some things well, but ultimately it’s not the right tool for the type of software I’m interested in these days. I still plan on using Node for web sites.

[…]

The more I’ve been working with distributed systems, the more I’m frustrated by Node’s direction, which favors performance over usability and robustness. In the past week I’ve rewritten a relatively large distributed system in Go, and it’s robust, performs better, it’s easier to maintain, and has better test coverage since synchronous code is generally nicer and simpler to work with.

TJ is not the first high-profile “noder” to leave for Go. A year and a half ago, Felix Geisendoerfer (author of node-mysql and node-formidable, among other well-known npm modules) tweeted:

bye #nodejs, hello #golang

and

#golang is technically better in almost every conceivable aspect

Some visible node.js-based start-ups are also making the switch, such as Koding (which provides a web-based development environment):

We’re not trying to say nodejs is bad in general. for some use cases it may be awesome. we didn’t feel any of its shortcomings when our codebase was small, when our userbase was smaller than 1000 users. it became bad for us when we had hundred thousand users, and we moved away as quickly as we could.

Go is amazingly stable and awesome. I can’t begin to list why everything about it is just great.

[…]

I wrote my first server with it in 2010, when it didn’t even have a proper database library. I wrote Koding’s first command execution server using it’s OS library, it processed at least a billion system commands much better than anything else that was out there.

And PopTip:

PopTip’s problems with Node started about two years ago. Falter said she knew how to write JavaScript and thought she could use that background to learn Node. The speed of development made it easy for the company to get started with Node.

[…]

PopTip helps companies provide a statistical breakdown on public consensus. Its clients include CNN, L’Oreal, Spotify, and a host of others. The size of these customers required PopTip to scale its code. Its service depends on machine learning and streaming data. As the company scaled, the task of administration became increasingly complex, due to “call-back soup”, a common complaint with Node.js.

[…]

The switch came when one of PopTip’s engineers spent a weekend replicating the core processing structure with Go. Text processing increased 64 percent just by moving from Node to Go. PopTip formally evaluated Go,and it passed the scorecard test: it had the runtime speed, the cost to administer was minimal, the talent pool was excellent, and competing frameworks were not vying for the attention of the developers.

It’s not just node.js users making the switch. Others are too.

One of Python’s most visible users, Dropbox — who also employs Python’s creator Guido van Rossum — recently announced it has migrated major parts of its back-end infrastructure from Python to Go:

Dropbox owes a large share of its success to Python, a language that enabled us to iterate and develop quickly. However, as our infrastructure matures to support our ever growing user base, we started exploring ways to scale our systems in a more efficient manner. About a year ago, we decided to migrate our performance-critical backends from Python to Go to leverage better concurrency support and faster execution speed. This was a massive effort–around 200,000 lines of Go code–undertaken by a small team of engineers. At this point, we have successfully moved major parts of our infrastructure to Go.

And, less surprisingly, Google is also switching projects over:

  • dl.google.com serves Googles downloads [Chrome, Android SDK, Earth, etc.]
  • Was written in C++
  • Now in Go
  • Now much better [more maintainable]

Naturally, it’s not just existing projects making the switch.

New high-profile projects are also started using Go. Most well-known is probably Docker. Many project using Docker under the hood are also written in Go, such as Flynn, a Docker-based PaaS.

Go 1.0 was only released 2 years ago. Clearly there’s a lot of momentum to use it for building high-performance web servers now. It’s fast, easy to distribute (because it produces a single statically linked binary) and has good async features without “callback hell.”

Not looking at Go yet? It may be a good time to do so now — everybody else is.

Discussion on Reddit and Hacker News.

The Wolfram Language Is Here — Now What?

Wed 25 June 2014

Wolfram just launched Wolfram Cloud, a development and deployment environment for building applications using the Wolfram Language. The Wolfram Language introduction video (that I recommend you watch before reading on) went viral in programmer circles — people got really excited about this.

As a programmer watching the demo video of the Wolfram Language you are immediately overwhelmed with the sense that this is the future. This is the programming language we’ve all been waiting for. One language to rule them all. Finally. Mission accomplished.

Now that the Wolfram Cloud is live and you can actually play with it (for free in limited form) — I can confirm, indeed, it’s very impressive. Following the tutorials and examples you combine all kinds of data sources with each other easily, and do super complicated computations quickly and present them in impressive ways.

But then you think step back and think…

How can I use this in my daily work or life?

And then you draw a blank.

I had a similar situation with Wolfram Alpha, Wolfram’s “search engine” launched a few years ago. It’s an impressive service that allows you to search impressive quantities of data with natural language. But after asking what the temperature was of your day of birth in your birthplace (22C for me, if you were wondering) divided by the number of earthquakes in Sudan between 1950 and 1990 — you’re kind of done. What else do you want to know?

The Wolfram Language has the same problem. I can imagine you’re a data scientist or mathematician and you have all kinds of uses for this. But I’m not. So after trying a few examples, I’m done playing.

As I’m a web developer, possibly I could use the Wolfram language to build a cool web app. It seems to have tools to build interactive UIs. Turning formulas into web services is built-in and also easy to do. But is that a good idea? The Wolfram Language — beside being cool, and, well, “the future of programming” is also the world’s most proprietary language.

You can deploy services to the Wolfram Cloud for free in limited capacity (you get some free computation time), but beyond that you have to start coughing up dough. You can host your own Wolfram Cloud, but that isn’t exactly free either. Once you’re committed to building stuff in the Wolfram Language you’re completely locked in. There’s no way out. No migration path, no open source implementation — and there will likely never be. The Wolfram Language isn’t something you build an alternative compiler for, because really, the language is just a relatively tiny aspect — the real investment is the gajillion petabyte of data that Wolfram has gathered over the past decades — and they’re not going to upload that to Github.

Nevertheless, there’s a lot of inspiration that can be drawn from the Wolfram Language and its development environment. Of particular interest to me is its “Notebook” style of development, which is really a beefed up REPL.

Many languages offer a REPL (Read-Eval-Print Loop) that allows you to quickly test out a few lines of code. However, in practice “real” applications are developed in a text editor editing text files, not inside a REPL. In the Wolfram Language, your Notebook (REPL session) is your application. After you define a function, it persists between sessions. You can do everything from a notebook, even deploy services to the Wolfram Cloud.

1-kdJIWzAbGrJpwRihkkmw4w

I’m fascinated by this idea — the idea of making whatever you do in the REPL persistent, and the idea of deploying your application from there as well with a single function call — talk about an Integrated Development Environment.

However, if a REPL is going to be your development environment then you need a UI that’s more sophisticated than your average terminal-based Python, Ruby or Clojure REPL.

In a sense, the LogicBlox REPL that we built is one step in this direction. Rather than giving you a “dumb” input line you enter commands in an advanced ACE-based editor — which means you get features like syntax highlighting, code completion and potentially errors and warnings while you type. While the effects of commands you enter don’t persist between sessions (because we create an empty workspace every time) you can download your work, and upload it later allowing you to continue. However, like with most software development platforms — its purpose is primarily to play and test things, not build real apps.

Apple has recently also adopted the idea of a REPL with Playgrounds for its new Swift language. However, it, too, is like most REPLs in use today: a tool for playing and exploration, not development. That’s a valuable thing, don’t get me wrong, but I think the idea of using the REPL (or Notebook) as a your main development environment for applications is something worth exploring. Something worth exploring outside of an extremely proprietary environment.

nthings

Mon 23 June 2014

Paul Graham wrote an essay back in 2009 entitled “The List of N Things”:

I bet you the current issue of Cosmopolitan has an article whose title begins with a number. “7 Things He Won’t Tell You about Sex,” or something like that. Some popular magazines feature articles of this type on the cover of every issue. That can’t be happening by accident. Editors must know they attract readers.
Why do readers like the list of n things so much? Mainly because it’s easier to read than a regular article. Structurally, the list of n things is a degenerate case of essay. An essay can go anywhere the writer wants. In a list of n things the writer agrees to constrain himself to a collection of points of roughly equal importance, and he tells the reader explicitly what they are.

After reading this essay I distinctly remember getting an idea for a web app immediately. Although only now I realize it has been 5 years since that essay came out, it was only last week that I decided to use the spare 3 days left in the week after a vacation to actually build it. I challenged myself to build the whole from scratch app during those three workdays (altogether 20 effective hours).

I did. I had fun. And I’m pretty pleased with the result.

Introducing n things

So, what is it? nthings.net is a simple web app that makes it easy to collaboratively create and edit these types of “n things about x”-style articles, sometimes referred to as listicles.

Anybody can create an account, create a new list with a title along the lines “n things you didn’t know about lemons,” and start adding things to it. Other registered users can extend the list with their own things, and vote on things already on the list that you didn’t author yourself. Items with more points end up higher in the list, resulting in lists (that can potentially get very long) with the most interesting stuff at the top. A kind of reddit for thing lists, if you will. If things you add to a list get voted up or down, you receive or lose points. There’s a ranked user list, topic pages (as subdomains), search etc.

The funny thing is that I have had this idea for many years, and only 5 minutes ago bothered to check if anybody else built something like this. “If I’ve never heard of it, it must not exist” — I thought. Turns out there are many of them. None of them seem particularly successful, though (or look in the least attractive). Oh well, we’ll see how my implementation of this concept will work out.

“Fascinating stuff, Zef — just tell me about the tech already.”

Usually, I take any side project as an opportunity to use some hyper new stack. So what did I use? Clojure with MongoDB? React.js with node.js? Nope.

Prepare to be disappointed.

This time I decided to take my own advice: to pick my battles. I had three days, did I really want to waste it on new tech with little documentation or with difficult to reliably deploy dependencies?

No. This time the challenge was to build something in a reasonably polished state quickly, without taking much additional risk.

So, I decided on the a stack I had worked with a few years ago (or some variant thereof):

  1. Google App Engine running Python: this is the oldest and still best-supported language App Engine offers. Any error message you get and weird thing you want to try is quickly Google’able. In addition, using a PaaS like this solves any trouble in deploying and reliably keeping up the site. The last thing that I want to do is operate servers and fail over databases reliably. App Engine offers all this out of the box: scalable datastore, memcache, reliable email sending, full-text search, image manipulation, data blob storage, one-command deployment, auto scaling. Perfect for this project.
  2. Python Flask: while not the most main-stream choice on App Engine (the SDK comes with the webapp2 framework), I had heard good things about flask, so wanted to try it. Documentation was thorough, lots of questions and answers on StackOverflow too. Seemed like a safe choice.
  3. Jinja2 templates: this came with Flask, so I used it.
  4. jQuery: that’s right, how more conservative does it get? Frankly, I didn’t intend to do that much JavaScript work in this project. So something basic was enough.

That’s it!

Some tidbits from the implementation:

  • The project consists of ~1200 lines of custom Python code, ~220 lines of JavaScript, and ~170 lines of CSS.
  • There are no images used at all. You can upload an image as part of a “thing,” but the site lay-out uses none.
  • Practically all content is generated statically on the server. AJAX is only used in a few places, e.g. to validate the registration form, load comments dynamically and to search while typing in the UI to create a new list. All AJAX calls send back pre-rendered HTML, so no templating happens in the browser at all.
  • The about page is a n things list itself. So, that was easy to implement.
  • A lot of caching using memcache is used. App Engine’s ndb database module does part of this automatically already, but I had to add some query caching manually, e.g. for recent list, related lists etc.
  • With a simple header-tag I made the site look reasonable on mobile devices (at least on my iPhone and iPad).

I found the App Engine with Python combination a very gratifying rapid development environment. I had used it a few years ago, so I roughly remembered the restrictions and features of the App Engine data store. I also had used Python before, so that came natural. The development experience was productive: I wrote all my code in Zed, obviously (and improved the Python mode a bit, too), reloading the page picked up changes in code automatically and errors were reported clearly.

I set specific goals for each day and always ended up doing significantly more than I planned, which why the end result is more polished than anticipated (there are even some moderation features, and the site doesn’t look terrible even though I have no design talent whatsoever).

I found the experience of building a full app from scratch in such a short time frame very gratifying. If you have a few days to set aside at some point I suggest you give it a try. Of course, the challenge is to come up with a “scalable” project — on where it’s possible to get to something usable very quickly, and extend from there.

So, there you have it: nthings.net. Who knows, you’ll may see some nthings.net lists appear on the Twitters soon. I already created one on iOS 8 features and Apple’s new Swift language.

Now, let’s get back to “real” work.

Page 1 / 344 »