Java Shop Politics _ Michael O | Scala (Programming Language ...

Michael O. Church Rants, essays, and diatribes. APRIL 13, 2012 BY MICHAELOCHURCH

Java Shop Politics Once, I was at a company that was considering (and eventually did so) moving its infrastructure over to Java, and there was a discussion about the danger of “Java Shop Politics”. It would seem strange to any non-programmer that a company’s choice of programming language would alter the political environment– these languages are just tools, right? Well, no. In this case, almost all of us knew exactly what was being talked about. Most software engineers have direct experience with Java Shop Politics, and it has a distinct and unpleasant flavor. When Unix and C were developed, they were designed by people who had already experienced firsthand the evils of Big Software, or monolithic systems comprised of hundreds of thousands of lines of code without attention paid to modularity, that had often swelled to the point where no one understood the whole system. (In the 1970s, these monoliths were often written in assembly language, in which macroscopically incomprehensible code is not hard to create.) The idea behind the Unix environment, as a reaction to this, was to encourage people to write small programs and build larger systems using simple communication structures like pipes and files. Although C is a strictly compiled language and no Lisp-style REPL existed for it, C programs were intended to be small enough that the Unix operating system was an acceptable REPL. This was not so far from the functional programming vision, and far more practical in its time. The idea behind both is to write small programs (functional “building blocks”) that are easy to reason about, and build more complex systems out of them, while retaining the ability to piecewise debug simple components in event of failure. This style of development works extremely well, because it encourages people to build tools for general use, rather than massive projects that, if they fail, render

almost all of the effort put into the project useless. As more code is written, this leads to the growth of generally useful libraries and executable utilities. In the big-program model of development, for a contrast, it leads to increasing complexity within one program, which can easily make whole-program comprehension impossible, and make decay (in the absence of forward refactoring) inevitable, regardless of programming language or engineer talent. I adhere to this small-program mentality. I’m not saying “don’t be ambitious”. Be ambitious, but build large systems while keeping individual modules small. If a file is too big to read (for full comprehension) in one sitting, break it up. If “a program” is too big to read in a week, then it should be respected (even if it runs as one executable) as a system, and systems are harder to manage than single modules. While it is harder, up front, to develop in a modular style, the quality of the product is substantially higher, and this saves time in the long run. Many software managers, unfortunately, like Big Projects. They like huge systems with names like cool-sounding names like “Rain Man” and “Dexter” that swell to a hundred thousand lines of code and provide features no one asked for. They like having the name of something Big, something their bosses might have heard of, in their story. Big Projects also provide a lot more in the way of managerial control. A manager can’t control the “chaotic”, often as-needed, growth of small-program development, whereas directing a Big Project is relatively straightforward. Make it Object-Oriented, use this set of analytic tools, complete at least 20 of the 34 feature-request tickets in the queue (it doesn’t matter which 20, just do 20) and have it done by Friday. Beginning around 1990 was a pernicious attempt by software managers to commoditize programming talent. Enter C++, designed with unrelated intent, but a language that made big-program development seem palatable for C. C’s not high-level enough for most application development in 2012, but C++ is not the solution (except in very specific cases from numeric programming where templates allow veryfast code to be written once for a range of numeric types, with no runtime performance overhead) either. For over 90 percent of the applications that use it, it’s the wrong language. Here’s the metaphor I like to use: assembly coding is infantry: as fine-grained as one could want to be, but slow (to write), lumbering along at 3.1 miles per hour. C is a tank. It’s robust, it’s powerful, and it’s extremely impressive and well-designed, but it moves at ground speeds: about 50 miles per hour. That’s what it’s designed to do. Languages like Python and Ocaml are airplanes– very fast, from a development perspective, but not fine-grained at all. C++ exists because someone had a fever dream in which these two classes of vehicles got mixed up and thought, “I’m going to put wings on a fucking tank”. The drag and awkwardness imposed by the wings made it terrible as a tank, but it doesn’t fly well either. Java was invented after a few horrible tankplane crashes, the realization being that the things are too powerful and fly too fast. It’s a similarly ridiculous “tank-icopter”. It’s not as fast as the tank-plane, and few people enjoy flying them, but it’s less likely to kill people. It’s not that C++ or Java, as languages, are evil. They’re not. Languages are just tools. Java was designed to run in embedded systems like automatic coffee pots and cable-TV “set top boxes”, so closures were cut for time in the first release, because these use cases don’t require high-level programming features. I haven’t map-reduced a toaster cluster for years. Contrary to popular history, Java wasn’t designed

with an ideological, “enterprise” distrust of the programmer. That trend, in the Java community, came later, when the 1990s attempt to commoditize programming talent (a dismal FailureFactory) co-opted the language. Java and C++ became languages in which “object-oriented programming” was the norm. The problem is that what is currently called OOP is nothing like Alan Kay’s vision. His inspiration was the cell, which hides (encapsulates) immense mechanical complexity behind a simpler interface of chemical and electrical signals. The idea was that, when one needs complexity, simpler interfaces are invaluable, and that complex systems generally should have comprehensible interfaces. Kay was not saying, “go out there and create giant objects” or “use object-oriented programming everywhere”. He was attempting to provide tools for dealing with complexity when it becomes inevitable. Unfortunately, a generation of software managers took “object-oriented” magic and immodularity as virtues. This is similar to the “waterfall” software methodology, named by a person clearly stating it was the worst idea, and yet taken by suits as having been “recommended by some smart guy” once it was given a name. What’s wrong with Big Project development? First, it encourages reliance on internal vaporware. Important work can be delayed “until Magneto is done”. When Magneto is done, that will solve all our problems. This works for managers seeking to allocate blame for slow progress– that fucking Magneto team can never get their shit done on time– but it’s a really bad way to structure software. In the small-program model, useful software is being continually released, and even an initiative that fails will provide useful tools. In the big-program arena, the project is either delivered in toto or not at all. What if half the Magneto team quits? What if the project fails for other reasons? Then Magneto est perdu. It’s a huge technical risk. [ETA: when I wrote this essay, I was unaware that a company named Magneto existed. “Magneto”, in this essay, was a potential name for a large project in a hypothetical software company. There is absolutely no connection between these project names used here and real companies. Consider names like “Cindarella” and “Visigoth” to be gensyms.] Second, under a big-program regime, people are trackable, because most programmers are “on” a single program. This is also something that managerial dinosaurs love, because it provides implicit time-tracking. Mark is on Cindarella, which has a headcount of 3. Sally is on Visigoth, which has a headcount of 5. Alan is on the project that used to be called 4:20 until Corporate said that name wasn’t okay and that now can’t decide if it wants to be called 4:21 or BikeShed. What this kills, however, is extra-hierarchical collaboration– the lifeblood of a decent company, despite managerial objections. In a smallprogram software environment, people frequently help out other teams and contribute to a number of efforts. A person might be involved in more than 20 programs, and take ownership of quite a few, in a year. Those programs end up having general company-wide use, and that creates a lot of cross-hierarchical relationships. MBA dinosaurs, alas, hate cross-hierarchical, unmetered, work. That kind of work is impossible to measure, and a tightly-connected company makes it hard to fire people. On the other hand, if a programmer only works on one Big Project at a time, and has no other interaction with other teams, it’s much easier to “reduce headcount”.

The third problem with big-program methodology is that it’s inefficient. Six months can be spent “on-boarding” an engineer into the complexities of the massive Big Project he’s been assigned to work on. In light of the average job lasting two to three years, that’s just intolerable. The on-boarding problem also restricts internal mobility. If someone is a bad fit for his first Big Project, moving him to another means he could spend up to a year just on-boarding, accomplishing zilch. Transfers become unacceptable, from a business perspective, so people who don’t fit well with their first projects are Just Fucked. Why is the on-boarding problem so severe? Big Projects, like large Java programs in general, tend to turn into shitty, ad-hoc domainspecific languages (DSLs). Greenspun’s Tenth Rule sets in, because programmers tend to compensate for underpowered tools by adding power, but in hasty ways, to the ones they have. They end up developing a terminology that no one outside of them understands. They become systems where to understand any of it requires understanding all of it. This means that people hired to modify or expand these Big Projects have to spend months understanding the existing system, whose intricacies are only known to a few people in the company, before they can accomplish anything. The fourth problem with the big-program methodology is the titular Java Shop Politics. In a small-program development environment, engineers write programs. Plural. Several. An engineer can be judged based on whether he or she writes high-quality software that is useful to other people in the organization, and this knowledge (talent discovery) is redundant throughout the organization because the engineer is continually writing good code for a wide array of people. What this means is that technology companies can have the lightweight political environment to which they claim to aspire, in which a person’s clout is a product of (visible) contribution. On the other hand, in a big-program shop, an engineer only works on one Project, and that project is often a full-time effort for many people. Most people in the company– especially not managers– have no idea whether an individual engineer is contributing appropriately. If John is chugging away at 5 LoC per day on Lorax, is that because he sucks, because the team failed to on-board him, or because Lorax is a badly structured project? In a small-program environment where John could establish himself, such a question could be answered. The good programmers and bad projects (and vice versa) could be objectively discovered. In a big-program world, none of that will ever be known. The person with the most clout on Lorax, usually the technical lead, gets to make that assessment. Naturally, he’s going to choose the theory that benefits him, not John. He’ll never admit that Lorax is badly designed or, worse yet, was a mistake in the first place. So under the bus John goes. In general, it is this fog of war that creates “office politics”. When no one knows who the good and bad contributors are, there are major incentives toward social manipulation. Eventually, these manipulations take more of peoples’ emotional energy than the actual work, and the quality of the latter declines. This is not limited to technology; it’s the norm in white-collar environments. Small-program development is an antidote. Large-program development accelerates (but does not necessarily cause) the poison.

The solution to this is simple: Don’t become a Java Shop. I’m not saying that Java or the Java Virtual Machine (JVM) is evil. Far from it, I think Clojure and Scala (which run on the JVM) are excellent languages, and the JVM itself is a great piece of software. Writing an occasional (small) Java or C++ program won’t destroy a company, obviously. On the other hand, fully becoming a Java Shop (where the vast majority of development is done on large Java programs, and where success relies on understanding defective “design patterns” and flawed “best practices” instead of being able to code) will. There is no avoidance of this; politically speaking, Java Shops go straight down. The quality of engineering, predictably, follows. The company I discussed earlier, which was one of this country’s most promising startups before this happened, did become a Java Shop, despite furious protest from existing talent. Within weeks, the politics of the organization became toxic. There was an “old team” that adhered to the Unix philosophy, and a “new team” that was all-Java. (No Scala or Clojure; Scala was flirted with and had serious managerial support at first, but they killed their Scala efforts for being “too close” to the old team.) This old/new cleavage ruptured the company, and led to an epic talent bleed– a small company, it lost several engineers in a month. Java Shop Politics had arrived, and there was no turning back. Here’s what that looks like. First, it’s not that Java (or C++) code is inherently evil. Any decent software engineer will have a passing competency in half a dozen languages (or more) by my age, and these languages are worth knowing. Some Clojure and Scala programs require classes to be written in Java for performance. That’s fine. Where a company starts to slide is when it becomes clear that “the real code” is to be written in Java (or, as at Google, C++) and when, around the same time, big-program methodologies become the norm. This makes the company “feel” managerially simpler, because headcount and project efforts can be tracked, but it also means that the company has lost touch with the ability to assess or direct individual contribution. Worse yet, because big-program methodologies and immodular projects are usually defective, the usual result is that people with bad tastes thrive. Then the company becomes less like a software enterprise and more like a typical corporation, in which all the important decisions are made by the wrong people and decline becomes inevitable. This entry was posted in Programming and tagged C++, java, java shop, java shop politics, office politics, software engineering, unix. Bookmark the permalink.

36 thoughts on “Java Shop Politics”

johnwcowan | April 13, 2012 at 10:48 pm Brilliant. This is your best post since you started posting again, and some of the others were no slouches either. Reply michaelochurch | April 14, 2012 at 8:35 am Thanks! I’m glad you enjoyed it. Thanks for your excellent comments (in general) as well. I’m nowhere close to being an expert on either (a) programming languages (I’ve read SICP and CTM, but I haven’t finished Pierce yet) or (b) how to run a technology company (i.e. “CTO stuff”) but I think my experience (Jane Street kicking ass with OCaml, a few startups of varying quality and success, and Google in its mid-2011 free-fall) has made me one of the 100 leading experts on how language policies affect a technical organization. It can be pretty subtle. I wouldn’t have guessed, even 2 years ago, that PL would have such a powerful impact on the way programmers work together and evaluate each other. Where I may have reached is in my inclusion of C++. It’s clear that Google has fallen into the big-program morass (even though the line-by-line code quality of Google’s C++ is very high). But C++ is such a huge language, and I know it so poorly, that I can’t necessarily say that it ends up “naturally” falling into big-program development, or if that’s something that just sometimes happens. I have a lot more confidence in that claim over Java. I would like to see where Scala ends up landing on this continuum. Odersky designed the language to allow pretty much every major paradigm, so it can be used with big- and small-program methodologies. I’m curious to see, 10 years from now, if the leading Scala projects are using the big- or small-program methodologies. I think Scala’s long-term success favors the latter, but it’s shortterm adoption curve favors the former. People who don’t understand small-program development tend to think that small programs are “toys” and therefore languages that don’t favor their big-program preferences are “not serious”. What I wish I had included is that another major advantage of small-program development is that companies don’t have to be “an [X] shop”. It’s when large-program mentalities set in that people start having language wars about what gets on the white-list and what doesn’t. The small-program attitude is, “If I have to read a 200-line C program now and then, who cares?” The large-program attitude is, “don’t you dare put Haskell on the white-list because we’ll never be able to hire Haskell talent if the original architect leaves FUD FUD FUD FUD”. Reply

bibac | April 15, 2012 at 2:55 am I understand your point and completely agree. Big monolithic software is evil. But why make this about Java? And why is Scala et al better? I can write a big ball of mud in any programming language. And I can take java and write small, modular apps. I guess “Java Shops” and the thinking behind it attracts just the wrong kind of people who don’t know how to write software. Reply rahmu | April 24, 2012 at 9:27 am Very good remark, I was thinking the same thing while reading the article. If the author himself admits that the language is not bad and the JVM is robust, they why Java? I think the answer lies in a better definition of the term Java. I believe it was jwz that tried to separate Java in 4 distinct definitons: – A programming language – A virtual machine – A vast collection of libraries – A fourth one I cannot remember. The vast collection of libraries (which has since been dubbed “JEE” or “Java Enterprise Edition” is the culprit here. If you look at it, it is completely built from the ground-up with the “Big project” mentality. With servlets sitting inside servlet containers sitting inside web servers; with object-relational mapping with Hibernate with persistance; with Beans; with JBoss; and don’t forget the endless XML… Using Java (the language) or any language running on Java (the machine) won’t necessarily turn you into a “Java Shop”. But it does increase the chance of your manager to stumble across an ad in a magazine (or a consultant with an MBA) and get tempted by the managerial allure of “Big project”. On the other hand, do a string of 5 successful “small projects” using Scheme or Lua, and the “big project” temptation is less likely to happen. And just to be perfectly clear, I am not arguing that software companies should use exotic languages (not that scheme or lua are really “exotic”), but that the choice of language will influence decisions taken by your manager (which ultimately is the underlying argument of this article).

Reply Bernardo Bombaim | May 10, 2012 at 7:55 pm because Java is too mainstream … Reply Hermann Schmidt (@alpengeist_de) | April 15, 2012 at 8:17 am I very much enjoyed reading your insights into the mechanics of software shops. At many points I actually felt pain because it is so familiar to me. I am a Java guy in my job, mostly, and I think I know what your finger is pointing at, and I can relate to your rage. However, I’d like to share a different view. Java is currently the save haven, the mainstream. You will not be beaten when you choose Java as most others do the same. That alone attracts all the risk averse and non-innovative personalities. Enterprise (big) software always has the suck factor, no matter which language is used. It is the people who work (I almost wrote enjoy) in environments where processes and rules are most important, where any innovation must have a business case with numbers (lies) before it has even started to form. I can write perfectly small programs in Java – like in the old days if you will. I think that every new language first lives in the nonenterprise niche, where programming is fun and less for profit. With growing maturity a language has to prove if it supports systems with a lifecycle of 5, 10 or more years and that is where it becomes enterprisey and gets in the hands of the aforementioned organisations and people – if it makes it this far. There are other examples (OO, agile, Scrum) where the “mainstream furnace”, as I call it, burns a thing down until it fits narrowminded personalities. Reply Jeppe Cramon | April 15, 2012 at 12:04 pm I like your analysis of patterns & problems that occur when (big) companies are doing big software development. Your circular conclusion that states since these companies use Java, then Java shops are like this and then if you use Java you’re bound to end up like them; is IMO way too simplistic and that’s where you loose me. Too bad you decided to keep ridin’ the “Java’s the problem” horse when discussing the solution. The two most productive companies I

know of both use Java, so what does that tell us? Nothing about Java, but more about how they master their domain, skills and development process. IMO it’s much more a people issue than a technology/language issue, and you could as well have written .NET which is another mainstream language that’s used in big projects…. Reply Benoit "tsuna" Sigoure | April 16, 2012 at 3:28 am I totally agree with you, although despite my hatred for Java, I have to concede that the problem you’re talking about is more about the way big companies tend to manage Big Software projects than about the language. I’d like to follow up on what you said about Google and C++. I worked 2 years at Google (2008/2009) and the conclusion I came to as to the reason it wound up in the “Big Software” camp is that it’s because Googlers worship complexity. Don’t get me wrong, I had a blast at Google, and to this day I still believe Google is 3 to 5 years ahead of the rest of the industry in virtually everything they do that’s technical (and not social). But unfortunately Google has been continuously rewarding hugely complex Big Software projects. If you KISS at Google, you’re unlikely to get promoted. And I’m not saying Google has no justification for complex systems. They do, because they built the best and most advanced infrastructure and distributed systems on the planet, by far. But the problem is that it’s the only thing they have: complex systems strung together with other complex systems, that together form huge conglomerates of mindbogglingly complicated super-systems. It’s virtually impossible to decompose each system into smaller logical units. So Google suffers heavily from the onboarding problem and also from retention problems (people who leave take away with them a lot of precious, hard earned knowledge about the systems they built or worked on). I think the reason you’re picking on Java is because everything in Java-land lends itself better for Big Software. The JDK is a prime example of how to make simple things unnecessarily complicated, object-oriented, abstracted, and AbstractFactoryInterfaceBuilderfriendly. Except for a few select packages, the code of the JDK itself is full of WTF moments, with tons of unnecessary layers of indirection and shit. The other issue with Java Shop Politics in particular is that it’s a problem of culture and education. Java culture is something the majority of programmers are picking up at school nowadays. See, software managers aren’t the only ones who want to commoditize programming “talent”. CS programs are also given ever increasing quotas to meet the never ending demand of the industry. Virtually all grad CS programs around the world now include a lot of Java stuff and very little C/UNIX stuff. Don’t get me wrong, this isn’t about the language so much as about the mentality. For the last 10 years we’ve been producing an entire generation of programmers that are primed for Java Shop Politics because that’s how they’ve been taught software engineering works. From the beginning they’ve

been shown how to put together their projects using massive Java frameworks instead of writing small incremental functional modules and binding them together in a sound fashion. So the majority of new grads, once they become “Sr. Software Engineer” by virtue of sitting on a chair in some tech company for 2 years, will find it natural that Big Software is the way to go, because, well, how else could it be? Reply Pingback: Functional programming is a ghetto « Michael O.Church

vpatryshev | April 26, 2012 at 7:00 pm Unless you also add something about Haskell/OCaml/Closure/Scala, the whole text gives an impression of a dinosaur rant. Oi-vei, the times have changed. They always change, brace up. No, I’m not a Java fan or a J2EE evil warrior. I believe Java is a pile of garbage. But it’s way ahead of c coding thinking level. Reply vpatryshev | April 26, 2012 at 7:03 pm Oh, oops, OCaml enters the picture in the next message. Then I concur. Unnecessary complexity kills creativity; whether it kills productivity, depends on how you define productivity. Reply Vlad Patryshev | August 12, 2014 at 10:14 pm And my apologies; my opinions have changed a lot since then. Reply Pingback: Technology And Software » Java Shop Politics « Michael O.Church

Yuhong Bao | May 29, 2012 at 8:27 pm

“Kay was not saying, “go out there and create giant objects” or “use object-oriented programming everywhere”.” And BTW, with C++, you don’t have to. I have almost always named my files as .cpp, but still don’t use classes often. Reply Hugo | June 1, 2012 at 8:16 am Have you seen this one? I can’t say that either you or him are wrong… there is never a final decision on things like this… http://magicscalingsprinkles.wordpress.com/2010/02/08/why-i-love-everything-you-hate-about-java/ Reply Pingback: Functional programming is a ghetto « Jelastic — Rock-Solid Java in the Cloud, Java Server Hosting, Java Cloud Computing

mokus | July 24, 2012 at 7:52 am Interesting metaphor with the tank-planes. In this case, though, I think reality beat you – “tank-plane” is a fair description of the AC130, which has a pretty impressive service record. I think C++ got the better end of that comparison Reply Pingback: Six languages to master. « Michael O.Church Pingback: Computer Programming: What advice would you give a non-programmer with a CS degree willing to invest 2 years? Quora

John | September 8, 2012 at 8:57 pm Thanks for publishing your essay. I have been employed for a handful of years and you’ve crystallised why I feel I haven’t gotten much intellectually out of the experience. Namely, lack of extra-hierarchical collaboration, and no internal marketplace for “Programs. Plural.” I have a mind to “start over” my career, because although I can presently bluff being a somewhat-experienced programmer, I know that if I enter a culture that is actually healthy, I’ll be thoroughly green.

Reply Pingback: What’s wrong with Enterprise Software? « thoughts shared, rarely

orubel | December 11, 2012 at 8:58 am I came from monolithic PHP/LAMP shops and switched to Groovy/Grails (java’s convention over configuration MVC). This defies everything you just said; I am able to develop twice as fast, have better tools, better security, more stability, more vertical AND horizontal scalability, better devops support, etc etc etc. So how is this monolithic? It is a web environment which promotes convention over configuration. You say ‘don’t use Java’ like a blanket statement ignoring the changing development wave in the Java community. Their is ‘old school’ java that you talk about and the ‘new wave’ java like Scala and Groovy/Grails which little to do with what you just said. You should really research this a bit more before making blanket statements like that. Reply Pingback: The unbearable B-ness of software « Michael O.Church Pingback: Quora Pingback: IDE Culture vs. Unix philosophy « Michael O.Church Pingback: Gervais / MacLeod 23: The shodan programmer | Michael O. Church

Allen | April 28, 2013 at 1:27 pm Hi! This post couldn’t be written any better! Reading this post reminds me of my good old room mate! He always kept talking about this. I will forward this post to him. Fairly certain he will have a good read. Many thanks for sharing! Reply georgerogers42 | May 16, 2013 at 2:09 pm

Reblogged this on 42 and commented: X shop politics Reply Pingback: Why Clojure will win | Michael O. Church Pingback: Technology’s Loser Problem | Michael O. Church

Vlad Patryshev | August 12, 2014 at 4:49 pm Thanks a lot. Great reading. Took me some time; it’s so important. Sheds light on the issues I’ve been pondering for years. Reply AK | September 13, 2014 at 9:52 am C++ is multi-paradigm. Lately it is distancing itself more and more from classic OOAD and becoming more and more enjoyable. Reply michaelochurch | September 13, 2014 at 2:05 pm I absolutely hated C++ when I was at Google but, 3 years on, I think my hatred had more to do with the shitty legacy code itself, not the language. Reply Pingback: Are Haskell engineers second-rate? | Michael O. Church Pingback: It might be time for software engineers, especially in Silicon Valley, to unionize. | Michael O. Church

翻訳】Haskellのエンジニアは二流なのか？　（答えはノーである） | POSTD

Pingback: 【

Create a free website or blog at WordPress.com. | The Misty Lake Theme.