Software and complexity

Over the past few weeks, I have been conducting interviews for the position of a software developer. I must have conducted more than a score of interviews of people with about 1 to 4 years of experience in the industry. Many can’t answer what the size of an integer is or what the largest representable integer value with a given number of bits is. One person said he couldn’t recall the formula. Of those who get past that, not one person so far has been able to write a 10 line function with confidence. And this is after some initial filters applied by the HR agency that supplies us with resumes – a minimum salary and some development experience. These people have salaries in the range of 3 to 6 lacs. Somehow the industry is able to extract a commensurate amount of value from them.

The product that we develop has over 200,000 lines of code. How will someone who finds it difficult to write a 10 line function work on a codebase of this size? And yet, 200,000 lines of code would be classified as a small to medium project. There are so many software solutions that have millions of lines of code. Presumably these solutions are being maintained by people with the same levels of skills/abilities as the ones I have been interviewing.

The only way I can make sense of this is to hypothesize that the people who wrote those solutions with multi-million lines of code designed them to be maintained by people far below their own levels of ability. That is an aspect of programming that I have never paid sufficient attention to. I have always worried about how to make my code flexible in terms of the requirements it can support and scalable in the data it can handle. My consideration has always been how to minimize my effort when requirements change and load increases. Often this has meant an increase in complexity. I have never worried too much about it confident in amy ability to handle the complexity.

As I look to delegate more and more of my work, do I need to consider this additional factor – how do I write code that can be maintained by someone who barely knows programming? I am beginning to believe that the answer to that is yes.

Complexity in Software

Software entities are more complex for their size than perhaps any other human construct because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into a subroutine — open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound.

— Frederick P. Brooks

Encountered in a book (warning: large pdf file – 53 MB) found at Mentofacturing

Specialization – Applied Philosophy – 3

In an essay titled “Why Nerds are Unpopular?”, Paul Graham writes that life in elementary school is warped and savage because it is isolated from reality and identifies specialization as the reason for the isolation.

“Teenage kids used to have a more active role in society. In pre-industrial times, they were all apprentices of one sort or another, whether in shops or on farms or even on warships. They weren’t left to create their own societies. They were junior members of adult societies.

Now adults have no immediate use for teenagers…
The cause of this problem is the same as the cause of so many present ills: specialization. As jobs become more specialized, we have to train longer for them…”
(Emphasis added)

Specialization and trade are the primary mechanisms of human progress. Todays industrial societies and the incredibly complex global economy would be impossible without specialization – without men who spend most of their lives working in a narrow field. Specialization has given us the sophisticated gadgets we use in our daily lives, the means to communicate with people across the globe, the opportunity to excel in our chosen careers. Specialization has given artists the time needed to create works of art and others the opportunity of enjoying them. Specialization has given sportsmen the time needed to perfect their skills and others the opportunity of being inspired by human perfection. In short, specialization has given us most of the things that we value in life.

Specialization has also caused innumerable problems. Specialization has made it very difficult for young people to make an informed choice of career or to change a choice of career once made. Specialization has made it difficult for people to adjust to economic changes. Specialization has created complex chains of dependencies among people. Specialization has made it difficult for people to understand fields other than their own. Specialization has made it difficult for anyone to understand the broader picture – the workings of the world. Specialization is atleast partly responsible for the large number of fallacious beliefs held by mostly reasonable people – particularly in economics and politics. Specialization is partly responsible for today’s rampant pragmatism – the lack of respect for abstract ideas and philosophy.

Perhaps the single biggest problem caused by specialization is the problem of knowing what to believe outside of one’s chosen field. Men have evolved a number of mechanisms to solve this problem – peer reviewed journals and techincal associations in science, the concepts of degrees and certifications in education, the concept of branding in advertising, independent rating agencies in industry, efforts like wikipedia, government regulatory bodies for everything, etc. While some mechanisms work better than others, it is clear that there can be no complete solution. The body of human knowledge is so vast and varied that it is impossible for anyone to establish trusted authorities in every field. The mixed success achieved in solving this problem is an important reason for the general lack of respect for abstract ideas and general principles. It also raises (well founded) questions about whether the entire system can sustain itself without directed effort. But the questions cannot be answered without abstract ideas and general principles, i.e without philosophy. Contrary to popular belief these are not merely questions of economics. They cannot be answered without an understanding of the nature of man, the function of his reason, the nature and structure of his knowledge and the reasons for his motives.

Specialization is a natural phenomenon. As long as men deal with each other, they have to trade. And as long as they trade, they will choose to spend their time on that which they are best equipped to do. In the absence of a catastophic disruption, a society will continue to grow in complexity. A system that constantly increases in complexity cannot be sustained without directed effort. Without that effort or with wrong efforts a catastrophic disruption is inevitable. Anyone who believes that the economy will continue to prosper irrespective of the social and political system is deeply mistaken. As the level of specialization continues to accelerate, the need for the right philosophy becomes ever more crucial.

%d bloggers like this: