A More Clearly-Stated Version of My ArgumentPosted: October 6, 2008
I was perhaps a little poorly stated and a little (intentionally) combative in my last post, so I figure I should clarify a few points. Thanks to everyone who commented or otherwise pointed out errors in my reasoning, over-reaching statements, or other ways in which I’m wrong. I think this is a subject upon which highly-intelligent, reasonable people will disagree, so this is really just my take on the direction things will go in the future (and not necessarily which direction they should go), based on my (sometimes more poorly-informed than I’d like) opinions about things like the ease of mastering particular languages, irregardless of the actual utility of those languages.
My real targets were the arguments I’ve heard time and again lately, the first of which is basically that the coming increase in multicore processors and the basic halt to clockspeed improvements will change programming languages fundamentally, finally pushing high-concurrency languages like Erlang or functional programming in general into the mainstream while leading to a slow abandonment of existing languages that don’t have that level of concurrency support. The second argument I was targeting is the general subcurrent of “But language X is so much more powerful, so everyone should use it” that always seems to exist in the programming community.
So to make my first argument a bit more explicit, it goes something like this:
- Small-scale parallelism (i.e. parallelizing what was previously a single thread of execution across mulitple cores on a local box) and large-scale parallelism (i.e. scaling an application across multiple cores on multiple boxes to handle large user/data volumes) are different problems
- Small-scale parallelism isn’t really helped too much by just using a functional language. While in theory a pure functional language can allow for parallelism of certain operations without the programmer having to do anything, in reality sustained usage of multiple cores requires more explicit parallelism on the part of the programmer, where the algorithm is broken down into explicitly independent pieces. In other words, if you want your HTML template rendering system or your video-encoding program to use multiple cores, you’ll have to design the algorithms with that goal in mind. Using a functional language might help with the implementation, or it might help with the thought process, but mere use of the language won’t be any kind of a magic bullet, and the same algorithmic approach is generally translatable to an imperative language as well. The algorithm is the important part if you really want to scale on that level, not the language (though, again, programming in a functional language might help get you thinking the right way).
- In general, with more software moving off of individual desktops, the need for small-scale parallelism is even more minimized, as per-user/per-request parallelism makes it easy to saturate an N-core box. If you really care about efficiency at that level, you’ll be more worried about absolute language/framework performance rather than concurrency support.
- Large-scale parallelism is helped by using functional techniques to parallelize and distribute work, but doesn’t necessarily require a functional language, as either per-request parallelism or explicit work-queue-like models can often allow horizontal scaling up to fairly large volumes regardless of the language.
- More advanced techniques might be required above a certain performance threshold, but the vast majority of applications never make it there and those techniques can impose a huge tax on development, so it’s more important for most developers to focus on getting the application to work rather than getting it to scale infinitely.
- Therefore, cloud computing/multicore processors aren’t going to be the “killer app” that switches people over to more concurrency-friendly languages. General developers will switch to those languages or not based on their merits as a problem-solving language and not due to their concurrency support. There’s a place for high-concurrency languages, but support for concurrency won’t be enough to propel a language into the mainstream.
So will functional languages go mainstream on their own merits as programming languages? I honestly don’t think so, though that’s an even-more-contentious argument. My reasoning there was essentially:
- Functional programming techniques don’t come naturally to most people; in my opinion, the human brain is designed for imperative algorithms to solve problems sequentially, since people are really only capable of doing one thing at once. Some people are probably wired a bit differently and have an easier time with functional programming, but most people naturally think in imperative terms, which will always make functional languages more difficult for most people to grasp.
- Being harder to learn and master means that the languages are also restricted to a subset of the current development community. There are a large number of people out there that can be reasonably competant in Java or Ruby that would flail if they were asked to learn Haskell or Scheme or OCaml. More “advanced” languages tend to be magnifiers; while they can make really-talented developers more productive, they can make less talented developers far less productive.
- Network effects, community size, talent pool, and barriers to entry matter for a lot of projects. Most software companies have a significant turnover or growth rate among their staff, making the ability to recruit people and bring them up to speed important. The best companies will hire for general ability rather than particular skills, but out-sourcing and contract work generally isn’t done that way, and ramp-up time still matters even in the optimal case. In addition, most companies can’t restrict themselves to the top 5% or 10% of development talent and can’t afford to limit their talent pool by choosing a language many developers won’t ever be able to master. As a result, the languages that are hardest to learn will inherently be a bit marginalized, since fewer people will already know them, the prospective talent pool will be smaller, and the ramp-up times will be longer.
- The scalability of certain languages to large, long-lived codebases with large development teams is suspect due to a small sample size. Most developers and project leads would rather choose a proven technique that they’ve seen work and that’s been used by hundreds or thousands of other teams, or that at least is similar enough, rather than trying to push the envelope with a radically different approach. There will always be outliers, but most developers are going to choose a well-trod path that they know can work rather than a less-clear path that might lead to increased developer productivity.
- Mainstream languages will continue to pull in more functional concepts (like closures), eroding some of the advantages that functional languages might otherwise have, meaning that the functional concepts become more mainstream but the languages that they originated in won’t.
So again, my arguments aren’t around what should happen or which languages are better in any sense, but rather they’re observations about what I think is happening and what will continue to happen in the future. It also is part of the reasoning that informs the direction we’ve gone with GScript; we’ve tried to emphasize ease of use, readability, speed of development, and suitability for building tools and frameworks, and we’ve avoided adding in features that we feel like will complicate the language but which might allow for better concurrency support, more flexible syntax, avoidance of side-effects, or more extensible syntax.