PHP for Java Developers

After years (and years) of working solely with Java, I've started learning some basic PHP. Coming from a strongly-typed, object-oriented background, it has been quite an interesting adventure. First off, I'll say that with an IDE like Eclipse, the programming language is almost irrelevant, once you understand the basic syntax for control structures and declarations. Once over that hurdle, it really becomes a matter of finding the right functions in the right libraries to do what you need to do.

I don't intend to get into which language is better. It's like arguing whether a hammer is a better tool than a saw. They fill different niches, and the bottom line is it's always better to have more tools in the drawer. But there are a few things that will catch a Java developer, and those are what I want to mention.

Global Scope and Declaration
PHP lets you do something like this:

$textlocation = $linelocation + 5;

Simple, right? Well, the gotcha is if you forgot to declare $linelocation, it will try to figure out what you meant. In this case, $linelocation will be an integer with value 0. In Java, that wouldn't compile, since there's no type associated with $linelocation, and no value assigned to it.

On the same topic, global variables have to be declared to be global.
$var = 5;
fuction foo()
{
echo "var: $var\n";
}

Would not output 5 as you might expect from
public class Bar()
private int var = 5;
public void foo()
{
System.out.println("var: " + var);
}

The reason is again, the dynamic variables. unless $var is declared as "global $var;" inside of foo, PHP will assume you don't want the global variable.

Now, the good side of dynamic variables. PHP will also allow you to do things like
$data = array ("date" => "12-29-2008", "name" => "Winter Park",
"min temp" => 3);
all in the same array. First element is a date, second element is a string, and third element is an integer. Now you can argue that Java deliberately doesn't allow that type of array, and the Generics changes in JDK 1.5 further restrict that kind of action, but there are instances where it is very handy and very quick.

OK, quick complaint: PHP doesn't have a println function. You have to remember to print "\n" after every echo or print command. There, I said it. I guess I can make a
function println($var)
{
echo "$var\n";
}

but that seems like overkill (even for a Java developer).

Database Access
Unlike Java, PHP has a much more streamlined database access system.
$connection = mysql_connect("localhost:3306", "user", "pass");
$result = mysql_query("select * from schemaname.table");
while(($row = mysql_fetch_row($result)) != null)
{
var_dump($row);
}
mysql_close($connection);
Opens, queries, prints the returned rows, and closes the database connection. 7 lines (counting braces). And since $row is an array, you can access it like any other array $row[0] is the first column, etc. Which leads into the next point:

Exception Handling
PHP does not require any exceptions to be caught, even though they can be thrown. It will handle migrating exceptions up the stack for you until it finds a handler or the top of the stack. So you can effectively ignore errors without having ... throws OneException, TwoException on every method like Java.

Arrays
Maybe I've written one too many classes using StringTokenizer, but being able to turn
$var = "0 1 2 2";

into an array of string using
$vararray = explode(" ", $var);

is really handy.
As is being able to interchange arrays, lists, and tables on the fly. So you can do things like this:

$vararray[0] = "zero";
$vararray["one"] = 1;
$vararray[] = "two"; // add to end of list

more to come!

Reformed Adblocker Speaks Out

Like most internet users, I am annoyed by ads. So I use Adblock Plus in Firefox to eliminate ads. Seems like 28,742,796 people agree with me, since that's the number of downloads Adblock Plus has had, according to Mozilla.com (as I write this). That's a lot of people saying one thing: "Stop bugging me with ads." But is it throwing out the baby with the bathwater?

Many (most?) web site survive on a mixture of ad revenue and sales. One of which is minute (ad revenue hangs around $0.005 per view, so 1,000 views will pay $5.00), the other is relatively large (buying a shirt at $30 or so). Used correctly, the two can be used to create sustainable income. Sustainable for the site bills, not necessarily for the owner's, anyway. But the problem comes from logic that "if a little is good, more will be better," in this case, a little advertising generates X dollars, then more, intrusive, annoying advertising will generate X+Y dollars (I'm not even going to bother to make up numbers).

Those of you who have been around long enough remember the progression. First, a banner ad at the top of the site. Then, an X10 pop-up (remember those?), then an inter-site ad between pages. Now, all of those above, plus the textual-context pop-up and whatever else they can think of. At some point the line was crossed and Adblock became a lifesaver. People said "Enough blinking, flashing punching of monkeys!" and blocked it all.

Then along came Google, with simple, targeted text ads. It's like the English butler of web ads. "Excuse me sir, if I may, I have some products that may be related to what you are looking at. They're over here if you'd like." Compared to the used-car salesman method used before, Google single-handedly overhauled the web advertising model. But, how many people can see them, assuming they are blocking all ads with Adblock? Some may argue that they will look for products when they look for products and content when they look for content and don't want the two to mix. That's fine, those people are not the ones I'm worried about.

I think a reasonable middle-ground can be reached. Use Adblock, but with a whitelist to approve of sites that you enjoy/aren't annoying with their ads. Start with this:

@@/pagead2.googlesyndication.com/*$script,subdocument

to unblock Google ads. Then remember to right-click on the Adblock stop sign icon and click "Disable on site" so they can get some of their revenue back.

I think a more permanent solution can be found with something like OpenID. With it, you can have an account and log into any participating site. Using it, a site (or series of sites) can track how much a user has contributed and show/hide ads appropriately. If a user makes a donation of $10, they don't see ads for a year. Like a subscription, but applied to all the sites under the umbrella. Recognizing that:

• people want something other than being bombarded by ads
• web sites won't survive without revenue

will create a new model that benefits everyone.

Where Does The Time Go?

When you start a project, don't you marvel at how efficient you are? How much you can get done in so little time? Why can't we always be that efficient? There are a number of reasons why efficiency drops off. Producing documentation (oh, you want someone else to help you, or you want someone else to use your project?), meetings (you need to work with these other people?), and bug fixes are the primary detractors from producing new functionality.

I've said before that Line of Code count is a horrible metric for measuring software development. It's like measuring hammer strokes in building a house, or the number of licks to get to the center of a Tootsie Pop. So I will be using percentage of time as a virtual metric. What percentage of time is spent in which task is a much more useful estimation than counting the number of nails used in a wall.

When you start a project, obviously most of your time is spent building new features. There may be some note-taking along the way to help you remember what you're doing, but really you're just trying to produce something that works, to solve your original problem. As you move along, there will be a equilibrium between new code, bug fixes, documentation, and meetings. You will wind up with a graph that looks like this:

Good amount of new growth, reasonable amount of bugs being fixed, not too much other stuff. Enough documentation to be useful, but not so much as to be a mountain. A few meetings to discuss what's going on and how it's going.

Compare to these two horror cases:













or













Given that there are only 24 hours in a day (please don't let my boss find that out...), for every additional chunk of time taken doing something else, that's time that can't be spent building software. So how do we maximize the time spent building software, and find the sustainable balance between too much documentation and too little? That's a topic for another day.

Making a List, Checking it Twice

I woke up this morning and thought to myself "How does Santa Claus keep himself motivated? He has one big deadline once a year. For 364 days of the year, he really doesn't have much to do."

OK, seriously, like most people, I've been having a hard time getting out of bed, especially now that it's dark out when the alarm goes off. And the first thing that goes through my head is "Ugh, another day, just like the last one." But today I made a quick list of things I needed/wanted to get done for the day, and I actually felt motivated to get out of bed and go to work. Of course, I'm now working on my blog, but I've still got an agenda for the day.

The list for the day is pretty simple (hey, I though it up at 5:45 this morning, cut me some slack):

• Configure CruiseControl to do something useful
• Find my best three job postings and send resumes to them
• Write up a couple business ideas (things to do if I get tired of programming)

It definitely makes it easy to get out of bed when you've got something to do, something new to learn, something new to try. Hope it helps you as well.

Is Your Company In Trouble?

Obviously, I'm going to look at this from a software development perspective, but it will include some general business observations (10 years of work experience gives you some business sense). Without espousing a particular methodology (although I'm sure I've made it clear what I support), ask yourself the following questions:

• Does this Company offer a challenge that's innovative and interesting?
• Do they offer easy access to innovative and interesting challenges (how easy is it to work on something that interests me)?
• How would a new idea be received? With questions like "How do we budget for it?" or "How can we use that to save money?"
• Is the Company worried about meeting estimates, rather than providing functionality?
• What's the Meeting/Time Quotient? Do you have to constantly attend meetings and provide status, or can you get your work done with few interruptions?
• How is the office environment? Are you provided with just what you need to get the job done, or do you have an office that you'd like to show people?
• Is their business model currently or potentially threatened by other Companies? Are they doing anything to become/remain the industry leader? How open are they to new ideas to help the company?
  
• What is the company attitude about new software? Can you try it out and see if it helps your situation, or do you need approval to try it?
• How long does it take to "do something"? How many steps in how many systems involving how many people does it take to report a bug/checkout/fix/build/deploy/test/check in/close the bug? If any of those numbers is large, what is preventing the system from being optimized?
  

Are You Wasting Time?

No, I don't mean reading my blog or surfing the internet or watching youtube videos. I mean even when you are working, are you spending time doing repeatable tasks that could be automated, and are you doing tasks that do not contribute to the development of the product?

The first question is the easy one to ask, but can be a hard one to fix. For example, does your build require you to "do something" to go from the built baseline to a running executable to test or deploy? If not, what's holding it up? Is the build simply not setup to do that, or is some portion of your system not configured to do it? If it's the second case, that's a bigger problem, but it can still be overcome by working with your system administrator and addressing the holdup.

The second question is the bigger task, and will usually require buyoff from your program Higher-Ups. I see this problem as the corollary to the Agile Manifesto, which states (among other things) "Working software over comprehensive documentation." I like to sum that statement up simply "Does this help me do my job? Would this help someone else do my job?" If the answer is "No" don't do it. You can quickly see where that would be a problem with the Higher-Ups, and the "That's the Way We Do It" crowd.

Another corollary to the Agile Manifesto is "It's OK to Screw Up", meaning, do something, give it to the users, and they'll tell you what they like and don't like about it. But do it fast enough to have enough time to fix the problems, and add the new features to it. This can't be done in a rigid environment, where it takes weeks just to get a fix to the users, never mind the amount of time to develop a new feature. To do it, and do it fast, and get it right, actually takes less overhead than you'd think.

Building Blocks or Jigsaw Puzzle?

Is your system building blocks for a larger system, or jigsaw puzzle pieces to build the same system? A puzzle piece fits into one and only one spot. A building block will fit into one spot, but can fit into other spots as well. I also like to think of it as a "spiky vs. smooth" interface.

What make good building blocks? The obvious buzz-words "extensible, re-usable, decoupled, cohesive modules" apply, but it's easier too look at. How hard is it to add new data to the system, and how hard is it to make the system do something else? If the answer to any of those is "not easy", then you're building jigsaw puzzle pieces. Look at your method signatures. Do they perform specific functions on specific types of data? Those are puzzle pieces. Do they perform specific functions on generic data, or (better still) generic functions on generic data? Those are building blocks.

So what makes reusable, extensible data objects? Obvious answers are object inheritance and using a syntax like XML. But no matter how you do it, you need to identify similarities between data and only handle unique data in special cases. Abstraction of data into generic objects for modeling and design helps describe the mappings between similar data.

An example of what I'm talking about. How many times have you seen "The Bank Example?" But imagine that example built to handle one very specific type of currency and a couple specific use cases:
depositDollar
and
withdrawDollar.

Simple, straightforward, and almost totally non-reusable (without refactoring). The use cases would be diligently mapped to methods named
depositDollar(Dollar dollar)
and
Dollar withdrawDollar()

in class AccountAccess. Dollar, being a good data class, would contain all the useful metadata about your dollar: serial number, creation year, number of wrinkles, that sort of thing. On the server, the business logic would need to store the Dollar and withdraw the Dollar from the Account, and do all the associated checking that goes along with it. So now, how do we add a new data type or another operation? Yup, create a new use case and a new method. Repeat ad infinitum (or at least ad naseum).

Now we've got a system doing very similar things on very similar data, but it's hard to refactor because you're used to thinking about the specific types of data, that a Dollar is somehow different than a Quarter, or a Euro. The only thing that can pull us out of this mess is to re-evaluate the data, and more importantly, what the users want to do. They want to put money into their account and withdraw money from their account. Now, generically, we can use methods
deposit(Money amount)
and
Money withdraw(Money amount)

where Money is, of course, an interface that Dollar, Pound, Euro, etc. all implement.

Now, I realize that there will be have to be some "business logic" in the system, otherwise the system doesn't do anything unique. But the amount of uniqueness should be low, and decrease, not increase, as the system expands.

Branches, Refactoring, and Deprecation

One of the biggest hurdles in refactoring is answering the question "How do I merge changes into something that's no longer there?" I will attempt to lay out a useful strategy for addressing that problem.

In our example, we've got a class that has a number of methods. The number of methods continues to grow (and the amount of logic in those methods grows as well), to the point that the class is unmanageable. It's no longer clearly defining its task (if it ever did), and has wandered off into areas that are outside its scope. But, because of its importance to many areas of the system, and because of its poorly defined task, it has a lot of bugs in it. The bugs may be from improper implementation, or incomplete analysis. Either way, there's a lot going on in here.

The obvious answer is "Refactor it." However, that's not going to be easy, since there are still bugs that need to be fixed during and after refactoring. How do we deal with this? With a cunning use of deprecation and branching.

Step one is to create a branch for the refactoring. The syntax and details of doing this are different for every version control system out there, so I won't get into it. Now you've got an area set aside just for refactoring without any outside influence and are able to verify that you haven't broken anything, through automated or manual regression testing.

Step two is to deprecate the methods or classes that you've modified during refactoring. But don't remove them! That will keep a merge point for you to pick up an additional changes that happend during refactoring and testing, and inform other developers that the class is about to change and the current method is no longer supported. In addition, a beneficial practice when deprecating (I'll use Java as an example, since I'm most familiar with it) is to include the @see tag to point the other developers at the right method.

Step three is merge in any changes into your branch. This will be a two step process: merge the changes into the original spot, then apply them into the refactored locations. This may seem time consuming, but is really the only way to ensure that your branch is up to date before merging into the main line. Re-run any tests for the changes to make sure that you merged the changes correctly (at this point, the benefit of automated testing over manual testing should be more obvious). Now you can merge your branch into its parent.

Step four is only necessary if other people are working on their own branches as well. After such time has passed that all the branches have picked up the refactored class, you may delete the deprecated methods or classes. This additional wait allows the other branches to migrate to the new organization at their own pace. They will have to do the same step you did before merging: apply any changes they've made to the original method to the refactored one.

Development by Telephone Game

Does your development process have a lot in common with the Telephone Game (also called Chinese Whispers or Russian Scandal)? I mean, how many steps, and how many people, are involved in getting what the users want into their hands?

Do the users tell their management, who tell the prime contractor's systems engineers, who tell your system engineers, who tell you what to build? Then, do you give your build off to the integration team, who gives it to the maintenance team to give to the users? Let's see, that's seven steps between the users and the end product.

That's one key area where open-source does it right, because they have to. The user submits a bug or new feature to the developers. The developers fix it or build it and give it back. That's one step (OK, there's probably a couple internal steps for reviews, and testing, but I overlooked those in the first example, so I'm overlooking them in this example too). And funny enough, it creates a stable product that delivers the functionality the user wanted faster than "Doing it the right way."

Are you naive enough to think "But think of how good the product would be if they did it our way?" The reason open source works (and work fast) is that there aren't multiple steps between the user and his product. The best tools are used BY the developers (see GCC for example) during development. If there's uncertainty, the developer talks to the user. There's no "pushing issues up the chain" and waiting for resolution. There's just a need, and the fulfillment of that need.

So call it Agile, call it XP, call it whatever you want. But if you base your development around creating user functionality instead of data functionality or system functionality, you will produce more user-usable functionality faster, simply because everything you build addresses some user need.

The Cost of Big Servers

I decided to compare the cost/benefit of a blade arrangement versus a "big server" setup. For the comparison I've configured a Dell PowerEdge M600 as my blade and a Sun Fire X4600 M2 as my "big server" setup.
Price:
Dell = $2,605
Sun = $15,995.00

CPU:
Dell = 2xQuad Core Intel® Xeon® E5405, 2x6MB Cache, 2.0GHz, 1333MHz FSB
Sun = 4 Dual-Core AMD Opteron - Model 8222, 3.0 GHz

RAM:
Dell = 8GB 667MHz (8x1GB), Dual Ranked DIMMs
Sun = 16 GB (8 x 2 GB DIMMs)

Disk:
Dell = 2x73GB 10K RPM Serial-Attach SCSI 3Gbps 2.5-in HotPlug Hard Drive
Sun = 292 GB (2 x 146 GB) 10000 rpm SAS Disks

OS:
Dell = RHEL 4.5ES (although I would put some other Linux variant on if it were up to me).
Sun = Solaris 10 (shocker, I know)

Roughly, the Dell is 6x cheaper than the Sun, which means I can buy 6 of them for the price of the Sun (duh). Now, partitioning that system could create two database servers (those would need more disk, obviously) and four application/web servers. So for the price of one "big server," I've already got a cluster of four app servers and a cluster of two database servers. Arrange as needed for your system.

Throwing disk arrays into the mix continues this thought. Comparing a Dell PowerVault NF600 to a Sun StorageTek 5320 NAS Appliance again shows the case against "one big server", even though we're comparing Network disk servers

Price:
Dell = $4,926
Sun = $ 48,164.00

Disk capacity:
Dell = 4x400GB 10K RPM Serial-Attach SCSI
Sun = 16 x 500 GB 7200 rpm SATA Disk Drives

CPU:
Dell = Dual Core Intel® Xeon® 5110, 1.60GHz, 4MB Cache
Sun = 1 x 2.6 GHz (assuming AMD)

Now we're in the ballpark of 10x price difference. So I could have 10 Dell NAS drives serving my 6 servers (again, I know there's no reason to have 10 drives for 6 hypothetical servers, but just to prove a point), vs one big drive serving one big server.

The price of adding "commodity-level" hardware is so (relatively) low, that there's almost no reason not to do it. Planning to build on this type of architecture quickly leads to partitioned applications, so that a new application can get its own host, or any service that needs more power gets more power, rather than improving the performance of the service. Figure $100/hour per developer. If it takes more than half a man/week to profile, re-code, retest and redeploy the service, you'd be better off throwing hardware at the problem. And as the net load on the system grows, it's much easier to build out horizontally when the increment size is small.

Now I know that you can run multiple virtual servers on one "big server", but the point I'm trying to get across is that the bang-for-the-buck of the "little" servers is much much greater, and acutally building around small servers creates a more modular, scalable software architecture.

Agile Architecture

Agile (or eXtreme Programming) development usually focuses on the development of the software itself, but it can (and should) be applied to all parts of the system. If the architecture of the system is determined before the first iteration and not open for refactoring or other redesign principles, then it falls into the category of BigDesignUpFront. To put the problem another way, how can the domain of the system be determined if what the system will do is not determined? And, if the system functionality is already determined, then there's no room to DoTheSimplestThingThatCouldPossiblyWork.

One argument against DoTheSimplestThingThatCouldPossiblyWork is that nothing complex (like a compiler) would be built. I disagree, simply because DTSTTCPW should only be applied to the domain as it is understood. So TheSimplestThing when you are getting started should be different than TheSimplestThing as your system becomes operational. TheSimplestThing when you are working in isolation on your personal computer becomes massively different from TheSimplestThing when you are providing a service on the Internet. As a ridiculous example, think about the problem of searching for a given word in your filesystem. Would you build Google to do it? No (or I sincerely hope not). You'd use find or perl or some other script language. Now, if you want to let anyone search for any word in any word on the Internet, now you've got to build Google, and all the system availability and performance benchmarks that go along with it.

The great thing about Web technologies is that they scale well, and don't require replacing to service greater demand or performance objectives. If one webserver can handle your own requests, then a cluster can handle thousands of them. But how do you refactor an architecture? Most people think of architecture as an inflexible part of the design. Nothing could be further from the truth. Writing POJOs (Plain Old Java Objects) leads quickly into EJBs (Enterprise Java Beans) in a J2EE container, or web services. Same goes for any web-based scripting language. Command line PHP can be easily hosted by Apache and serve the same data over the web.

The key point is that just as your software should be designed iteratively, so should the archtiecture. As the system requirements grow and change, the architecture should grow and change with it. Will there be additional software changes to handle the new architecture? Probably. Will the overall system be better suited to handle the system parameters? Yes. Otherwise, you will be trying to put a now square peg into a hole that's still round. It can be done, but is sure isn't TheSimplestThing.

The Network Is The Problem

The first goal of any architecture should address two questions: how does the user interact with the system, and what does the system do when the user can't connect. Obviously, the goal of the system is to provide as much connectivity as possible, but there will always be potential problems between the client and the first network connection. The system will need to address how to handle data that does not make it to the server (like, blogger, for example).

How the user interacts with the system will affect how the system handles the data when the network is down. Of primary importance is what to do with any data that would have been transmitted from client or server, and how to recover once connectivity is restored. For a time-critical system, an additional important decision must be addressed: what to do if the request isn't received in time. If the request can't be fulfilled, does it still have value?

By focusing on the user input, you can keep your project in scope. If you start looking at data, then you can start looking at all the places that data can go, and all the systems that need to interact to deal with the data. By focusing on what the user needs, rather than what the system requires, you can iteratively add more and more features, and rebuild the system as new capabilities are required for those features.

A trivial example: a user needs to know what his work assignments are. So he 1) logs into the system (define the scope of the user environment, address network connectivity - see, I told you it would be important), 2) requests his work (pre-defined data, manual query?) 3) displays the data (what kind of display is available?)

Now, once those decisions are addressed, they don't need to be addressed for the user to create a new work assignment. Rather than attempting to figure out all of the system up front, the system can provide a needed functionality quickly. Once a decision is made, it can be applied to all other functionality in the system.

Agility 101: Refactor Your Build

One of the problems with using an adjective like "agile" or "extreme" (ugh) is that it implies a degree. As in, "A mouse is more agile than an elephant." However, you could say the elephant is more agile than, say, an aircraft carrier. So, off the bat, agile seems open for interpretation, whereas Rational Unified Process or Waterfall have well defined steps that you either are doing or aren't doing. This is wrong. Agile has a very well defined set of processes, but since the term seems to lend itself to statements like "We need to add some agility," it is working at a disadvantage.

The primary goal of agile development is continuous, sustainable development. If this is not possible in your current environment, ask yourself why not? A primary area for improvement is the build/deploy/test environment cycle. If it takes hours to build, deploy, and coordinate testing, then you've reduced the amount of time available to do anything useful. Some important points to consider:

How many steps does it take to go from build to running system? If it's very many, re-examine all the functionality that Ant provides.

Do multiple versions of the system potentially interfere with each other? If so, figure out a way to run multiple versions in isolation. As an additional point, can multiple versions of the same software co-exist on the same server or in the same environment? If I need to care what you are doing, that's going to take time to rectify.

Can I build and test multiple branches without interfering with a previous build? To reiterate the previous point, if it takes time to reconfigure and set up to test a fix or enhancement, that's time that can't be used for coding.


Given the capabilities of a tool like Ant, there's no reason that a complete, executing system cannot be produced out of every build.

Improve Product, Not Process

Believing that process improvement will improve the product is like believing spending more time looking at the map before taking a trip will prevent you from running into a traffic jam along the way. Process improvement creates paint-by-numbers, it creates fast food. Which is fine if that's your goal, but don't expect anything other than paint-by-numbers or fast food. Otherwise, we're treading into Einstein's definition of insanity "Doing the same thing over and over and expecting different results."

If you feel you must improve your process, plan your process for change. There are too many uncontrollable variables to have any expectation of a successful delivery with a single waterfall method. So plan your process for more frequent changes, more frequent reviews, and more frequent releases. Hmm, this sounds like agile (without a capital A).

Just Because You Can, Doesn't Mean You Should

A note on the Agile adage "Do the Simplest Thing That Could Possibly Work. (DTSTTCPW)" DTSTTCPW does not mean "Do the Easiest Thing," or put another way "Just Because You Can, Doesn't Mean You Should." (Hey look, a title!) There's a file line between those two ideas, because the Simplest Thing to one person isn't necessarily the Simplest Thing for everyone else involved. Reducto ad absurdum example: suppose someone keeps the requirements in a Word doc on their desktop. Anyone who wants to know what they have to do has to go look at that one person's desktop. Now that's surely the Simplest Thing in this case, but that is definitely not the Simplest Thing for everyone.

So, as usual, we have a problem. How do we figure out the Simplest Thing when it comes to organization and communication? Here's some guidelines:

• if you have to search for it, it's not in the right place
• if you have to use different tools, you don't have the right tool
• if you have to ask, it's not documented well enough
• if you have to change the process to get something done, the process needs to change

The first point will stop the practice of "Put it on the data server" in a unnavigable directory structure that combines software, business development, schedule, test, birthday lists, and everything else under the sun. Do you have a number of bookmarks and shortcuts to help you navigate your data drive? So do I. This problem can be traced back to the problem that the project is not focused on the Product, but more on everything else that goes into the Product. If the data is not related to the person who has to wade through it, get rid of it (or at least put it somewhere else). Sourceforge once again comes up as the model. All the documentation related to the Product is stored there. Want to use it? Read the documentation. Want to see the bugs? Look in the tracker. Want to help? Download the source for Subversion or CVS. It's a very simple idea that's very easy to overlook: The Product is the Project (hopefully Scott McNealy will forgive me).

A documentation system based on a file system just isn't effective anymore. Sure, it works. Much like any outdated technology still works, there's just a much better way to do it.

The Simpsons as a Software Model

Like most things in life, The Simpsons show us the way. Or, rather in this case, the way NOT to do something. In "Oh Brother, Where Art Thou?" Homer gets to request all the features he wants in his dream car. In effect, he gets to design the car himself. You can see the result on the left. That's like what happens if our customers get all the features they ask for.

Our end product ends up looking like that. Lots of features "stuck on" with very little concern for the overall design. Sure, it will work, and come out as a finished product, but the true art of software comes from taking all those features and still producing a cohesive, solid design. But, like most designs that are "customer-driven," it is way over budget (at least it's not behind schedule, as well). Now, if you're in a market where you can continue to charge the customer to make up the budget, that's a feature, and probably a goal. However, in the real world (and the Simpson's world as well), an ugly, poorly designed product won't sell.

So how do we get from Homer's vision (which, incidentally, he was quite pleased with) to a mass-market friendly one including all the features the customer requested? One simple step that seems to be largely overlooked is "Saying No." I can't believe I've even got to put this down, but if the customer makes an outrageous request, saying no (and explaining what we should do instead) will cut the largest potential budget/schedule over-runners. For example, I'm currently supporting a custom data synchronization engine for my current project. Now, we are using a database that supports replication, but it is not available. If we'd just said "No" (and for the record I said "No" but my managers didn't care, because they didn't want to say "No" to our customer), we would be able to use the built-in replication. But we didn't. C'est la vie.

A Software Fable

Gather 'round, kids. It's story time. There's so few fables written nowadays, and few of them would apply to software (if you read between the lines). So here goes.

Once upon a time, there were two groups of engineers out for a walk. They came upon a creek, and, being engineers, decided to dam the creek. The first group set to work immediately measuring the depth and width of the creek, calculating flow rates and how high the water would rise once dammed. The second group looked at the creek and started throwing rocks into it. The first group, having taken all their measurements, set off to carve a rock big enough to block the creek. They left the second group still throwing rocks into the creek.

Time passed, and the second group came back with their huge rock. But unfortunately for them, the rain had turned the creek into a raging river and their rock was still not big enough! Or it would have been a raging river if it was flowing. To their surprise, they saw a dam built out of the rocks that the second group had been throwing! The first group was amazed that something so big had been build by so many little rocks. They went and asked the second group how they did it. The second group said "Some of us looked for more rocks to put in a pile, and the rest of us threw the rocks from the pile onto the dam. So while you were measuring and calculating, we were already solving the problem. If anything goes wrong with the dam, we just need to throw more rocks. If it rains more, we just need to throw more rocks. You need to carve a whole new rock."

Many morals apply to this story: do the simplest thing that could possibly work, don't over-engineer the solution. But the one I like is this: if you leave engineers in the woods for too long, this kind of thing is bound to happen.

Ghost Town

No, not the blog. I know I missed a week due to The Big Release deadline on Friday. I do this as a hobby, and comment about my real job. So if I miss a week, my loyal readers (of which as far as I know there aren't any), will just have go without.

That business aside, I started wondering about the New Developer Problem, or more specifically, the worst-case version of it. "What if everyone here quit/disappeared/died/otherwise didn't work here any more?" Maybe I'm on too much of an Indiana Jones train of thought lately, but I see it like an archeology expedition. Some guys in fedoras pry open the door to our dusty office and see a bunch of dusty cubes. They sit down and fire up the boxes. Would they

• be able to figure out what we were working on?
• be able to continue on with what we were doing?

The first task is possible. Assuming they can get user logins (Dr. Jones would probably Post-It surf around to find user names and passwords), they can probably click enough desktop files and folders to get an idea of what was going on. But does that combination of information contained in SDFs, schedules, budget figures, requirements, etc. actually describe any of what was going on? If you found a document in Egypt that said "June 1, 2500 B.C - build ground floor," would you be able to ascertain that you were looking at requirements for the Pyramids? Because that's what our artifacts tell the outside observer: not much.

The second task is much more daunting. Even if Dr. Jones was able to figure out what we were making, the first question would be "Where is it?" followed by "How do I make it work?" and "How do I fix it if it breaks?" Can he find the answers to those questions buried in all the process-mandated documentation? Maybe it's possible, but in my experience, it's unlikely. Most of my projects have been treated as one big project, rather than a large number of small projects. So all the documentation was (is) created with respect to the whole project, and didn't reflect the individual pieces of the system. That's like trying to understand the internet by reading the HTML and HTTP specs. Yes, you'll understand what it does, but how it does it is totally ignored.

Now imagine Dr. Jones's expedition into a open-source development model software facility (adverb chain ahoy!). He walks in, clicks the browser icon on the desktop (hopefully the only icon there), and the browser goes to the project home page. Now he can see the project roadmap, open bugs, read documentation, and download code and start making changes and fixes. With that kind of organization based on the product, you don't need to be an Indiana Jones. Even Marcus Brody could find his way around that project, and he got lost in his own museum (supposedly).

Most of this problem comes from each group involved doing the easiest thing for them. For testers, they create Word docs. For systems engineers, they create Rational UML. For software engineers, they create code. But all those things add up to create documentation focused on whatever each group finds important. The documentation needs to be correlated to the end product, and should be stored in a common place that can show the correlation of the documentation to the product, i.e. a project website. Sure, software can be built in many ways, but you are at risk if you project depends on "project experts" or "domain specialists." If there are areas that aren't immediately understandable, that's an area for improvement of documentation or refactoring.

Setting the Bar Low

I work in a company that is primarily concerned with doing just enough to get the next contract. That is a demotivation for any employee who wants to keep up with technology, or branch out into a new area of development. The problem with that mindset is that it is difficult to convince anyone that any change is a good idea. You will be met with answers such as "It worked fine (we delivered something on time), why change?" and more simply "That's the way we do it."

That's a perfectly fine mindset if you are working on an assembly line. But, as I've stated repeatedly, treating software as simply production is to do disservice to it. More importantly, the company will lose their top developers who would quickly tire of doing the same thing over and over. But this is not change just for the sake of change. Rather, it is change for product improvement. Doing it better, and/or faster. Take the old vi argument. Sure, you can develop software in vi, but I can do it much faster with Netbeans or Eclipse. But the metrics are already skewed toward the development rate with vi, so I've now got free time on my hands. What do to with it? I'd like to work on something else, investigate something new, or otherwise improve the product in development, deployment, organization, or maintenance. But I'm told I can't, because "There's no budget" or "You're not authorized." So we keep on happily cranking out buggy whips (no pun intended).

Einstein's old adage "Insanity is doing the same thing over and over and expecting different results" is well understood in a large company. The side of it that's not noticed is the side that wants to keep talented developers, but doesn't realize that we're really just doing the same thing over and over. What's really needed is to totally throw out the existing schedule and really challenge the developers. Don't give me three months to do a task, give me three weeks, or three days. You will get a radically different solution, simply by being prevented from solving the problem the same old way.

K.I.S.S Applies to Process As Well

K.I.S.S. (Keep It Simple, Stupid, not the 70's band) is a tenet of agile software development. But it should also apply to all facets of development and not be limited to just writing code. If you've got to go to one server to get requirements, another one to view design documents, a third to build your code, and a fourth to view problem reports, odds are one or more of those systems will be out of date. "One server to rule them all" is a much more maintainable approach and ultimately a more useful approach for everyone involved.

There are a number of great sites organized like this, but SourceForge is probably the largest. One project page contains all the necessary elements for the lifecycle of the project: code, documentation, support forums, bug/feature tracking, and release downloads. A user, a developer, and a manager would all access the same site. This is, as always, by necessity. In a distributed development environment, every developer has to have all the necessary information available to them.

The Problem with the Office

No, not Microsoft Office. Not the TV show. I mean the place where you work. You know, your cube. The problem isn't exactly with the office, but rather with having office-mates around. Specifically, the problem is the availability of those people. You know, Bob the database guy, Phil the systems guy, or Jill the tester. If you have a question, you go ask them. If they have a question, they come ask you. What's the problem with that, you ask? The problem is that the resolution to that question is only known to you two. The problem is that if anyone else wants to know what the answer is, they have to ask one of you. That's job security, to some people.

When an open-source developer has a question, or has to make a decision, there's usually no one there to ask. So the developer has to go the project mailing list, or forum, or website, and ask the question to the group. Then anyone else can see the answer. If anyone else thinks the question is very important, it can be added to the documentation, FAQ, wiki, or whatever. That way, the documentation represents what anyone really needs to know about the project.

This problem is reflected in the "learning curve" for business projects. Some companies try to overcome this shortcoming by mentoring, or shadowing, or other methods for effectively "picking someone's brain." But all they are doing in ingraining the mindset that some person knows the answer and will give it to you if you ask. This is a dangerous process, because it creates many single points of failure. What if Bob the database guy gets hit by a bus? Who else knows how to run the database? If he'd written it down, then someone else will be able to take over. Overcoming the job security mentality is a tough task. But, assuming that no one will work at the same job for their whole career, a repository of information is absolutely necessary.

Your coworkers may be great people, but stick to talking to them about non-work issues. If it's about the project, create a record of it. If you really like process, create a process to review all the wiki entries, problem reports, and mailing list entries for review for possible inclusion into the formal documentation. But, above all else, write it down!

Government Software, An Oxymoron

Or "Software by Earl Scheib: I Can Build that System for $49.99"
I can't wait for Google to start a Federal Systems group. Until they (or someone else with their deep pockets and development methodology) get involved, the rest of us will continue with status quo. It is unfortunately not in our company's best interests to improve the way we develop software. In fact, poor software process is encouraged because it allows budget and schedule over-runs, otherwise known as Cost Plus and Follow-On Work.

Both sides are at fault for this, but ultimately the blame has to fall on the developer side. The client is only asking for what they want, as they always do. The problem lies with the development staff not explaining that they are making unreasonable requests. Furthermore, allowing the client to determine the technology to create the system will most likely not utilize the most optimal architecture for the problem. Which leads to more Follow-On Work. Perfect: the client gets what they want: a convoluted, inefficient system, and we get more money to fix our mistakes later!

Honestly, I can't wait for Google's first meeting with govvies where they say "You'll get your software when it's ready" and watch multiple General's heads explode. I don't have any idea if Google is crazy enough to enter this arena, but if they do, they will dominate it. Why? Because the existing competition is some combination of lazy and inept. Or, we are happy doing whatever crazy job the customer thinks up next.

I can't believe that the Form-A-Committee-to-Investigate-A-Commission world of the government works as closely as they do with the Long-Haired-Free-Thought world of software development. I know they've had in-house staffs before outsourcing the work, and I think that's pretty good evidence that they don't understand how to do it. But rather than say "Build me my system and give it to me when it's ready" they still want to have control over the process, but have success at the end. Remember Einstein's definition on insanity?

So why Google? They've got the money to make it work, but more importantly they've got the influence to break the model. Get government's hand off the software except deciding what they want and making sure it works, and you get a better product. Come to think of it, "Get government's hands off... and you get a better product" is a more succinct explanation anyway.

Keep It Small

One of the largest (no pun intended) problems with building government software is too much knowledge at project inception. For the most part, we are rebuilding or adding onto an existing system. Therefore, we think we know where we want the end product will be, and think we know all the problems we will encounter along the way. As Murphy's adage states, that is a recipe for disaster, or at least schedule slip. By not building the system as a progression, but rather taking it as a monolithic task, the environment for innovation is decreased (to put it as nicely as possible). Like a used car, if the system works perfectly now, why are we rebuilding it?

The fallacy that "We built this before, so we know what it does" should never be applied to a system redesign. The problem with it is that it does not address all the other things done under the covers to create the end result. The existing system should not be used as a model for the new one, but rather the functionality the existing system attempts to provide should be recreated in the new one. Or more simply put: look at what it does, not how it does it.

This problem results from the XP concept of Big Design Up Front. Simply put, it's taking too big of a bite that results in a system that's too complex to be understood, tested or extended. Yes, it sounds like the paradox "Can God Create a Rock So Big He Can't Move It?" and, when applied to software, quickly becomes "Can People Create a System So Big They Can't Understand It?" I'm sure you'll agree the answer is yes, and very quickly. In fact, the greater task is preventing that from happening.

So how do we resolve the problem of Big Design Up Front? Stop thinking about it. Take the known pieces of the system, design them such that they can produce query-able, re-usable output (XML Web Services or EJBs in this era), and build it. Define the architecture and the data flows, but keep your hands off the system components. There's enough to do to define or redefine what the users are doing, or better, what they want to do.

I propose the Rule of One: if the problem can't be defined, described, and agreed upon in one hour, it is too complex. Most people would groan at a meeting that is scheduled for more than one hour. Therefore, one hour is quite enough time to organize an approach and discuss the issues with that approach. That quickly leads into The Simplest Thing That Could Possibly Work . Additional issues can be addressed in other iterations, but to get the ball rolling requires a quick, basic action.

Happy Birthday, Open Source Software

Bruce Perens wrote an article noting that today is the 10th anniversary of the Open Source Initiative. As discussed, on slashdot, this doesn't necessarily mean ten years of open, collaborative software development, but rather ten years of promoting open software as a viable option to closed, proprietary software.

Today's post is not to discuss the pros and cons of open versus closed software development (Lord knows there's enough of that without me getting involved. See slashdot again. I'm much more interested in open source development as a model for general software development, and what it offers that closed development does not. Specifically, how can the open-source model create so much (MySQL, Firefox, OpenOffice, Apache, etc. etc.) with so little (development staff, management, time, money, etc. etc)?

Knowing that the development staff won't be available forces open source to make a number of decisions about project organization. Because there's no full-time staff, all the assignments have to be given out in a way that is verifiable, discreet, and in isolation. Every developer knows where their piece fits into the application, and all the application information is available and up-to-date. The project has to be self-contained, easy to set-up and document progress. See Mozilla.org's development site for a great model of how to keep the staff informed simply.

Knowing that the staff won't be available requires the use of a software version control system that allows disconnected operation. While not crucial, this model allows the developer to work in isolation until they are ready to submit the changes for approval. This allows the product to be more stable for longer periods of time while development is ongoing, and requires the use of a branching/merging policy to handle bugfixes and new feature development.

Because the development staff may change at any time, code reviews are given a higher priority. There's little room for the attitude that "He did good work last time, so it can slide." Everything has to be reviewed, documented and tested long before hitting the main baseline. Again, this keeps the stable release as stable as possible and removes many opportunities for error.

Open software development follow the model "Give them the tools they need and get out of the way." Innovation and creativity can flow freely in an environment like this, at lower cost with higher productivity. But corporate development is stuck in a Catch-22: managers won't go for a "radical" change like this, since it would put them out of a job. So even though us lowly developers know there is a better way, we won't be allowed to use it. And that's very sad.

Making Programming Boring

Back to software stuff, a day late again.

My company is working very hard to make programming boring. Maybe boring is too strong of a word, but certainly mundane, simple, and straightforward. But, in the usual double-whammy of bad idea and impossibility, this is a move in the wrong direction.

The problem lies in identifying and defining the task. If the task is so tightly defined that no innovation can take place, then the project is bound to stagnate. While there are circumstances where a tight definition of requirements is both desired and necessary, most situations would dictate that you development staff knows more than you do about the domain of the problem to be solved. Joy's law states that "Most intelligent people work somewhere else", which is really a corollary to "50% of the people are below average." That simple fact can be used to prove that both the designer and programmer are average at best. But it also says that there may be someone in the staff who does "think outside the box," which is only possible if the box is not closed on all sides.

Programming is not brick-laying, or assembly-line work. People may mistake the "Cathedral and Bazaar" analogy as indicating that the work must either be done by skilled craftsmen or by "workers." Software is neither a blank canvas nor a paint-by-number. It is somewhere in between, and each absolute end is used very infrequently. The innovative work done by a company like Google was either designed or created by a few, very smart people and then passed to less-smart (again, invoking Joy's Law) people to implement or use and expand upon.

Innovation is what created the software technology we have today, and to try to over-engineer the design and implementation to the point of stifling innovation will lead to a loss of talent who don't want to work where their ideas aren't recognized and ultimately to a loss of market share to companies that can innovate and create software more quickly, for lower cost, and taking better advantage of the existing technologies. If all you know how to make is widgets, and all your people know about is making widgets, then you would be oblivious to anything that solves your problem better than a widget.

There's No Such Thing as Bottom-Up Design

Yes, I'm stealing "There Ain't No Such Thing As A Free Lunch," from Robert Heinein as the title this week. It's catchier than "Bottom-Up Design is really Stupid and Hard to do Right." I'm not saying that it is impossible, but only because I believe that nothing is impossible given enough time, talent and organization. But Bottom-Up design is much more difficult to manage than Top-Down design, for a number of reasons:

• Management of many incomplete services is much more time-consuming than management of a few complete services. The ramp-up for staffing to begin defining interfaces and interactions of those interfaces is much steeper than working slowly and getting the first part working, correctly, before beginning step two. This ramp-up smacks right into the Mythical Man Month formula that group communication equals n(n − 1) / 2, where n = the number of developers. But by working slowly, the group can reach consensus one one point much quicker than making a decision on many unknowns. As a result, much more time than is necessary is spend monitoring the progress of many, partially functioning services.
  
• Bottom-Up implies Top-Down, but not vice-versa. This one should be obvious enough to be the nail in the coffin of Bottom-Up design on its own. Any class/function needed in a Bottom-Up design should only be created to fulfill a need. But what defines the need? That's right, some sort of Top-Down analysis. But then, why perform a perfunctory analysis and stop there? If you need to declare the situation in which you need the function, it makes sense to continue defining the situation to an implementable solution, rather than defining ALL possible situations with little effort given to the interactions of those situations.
• Bottom-up design fails another Brooks truism. Create a working prototype and "grow" the software iteratively. By trying to define everything that needs to be done at once, the developer may miss out on opportunities to refactor the design. Creating a working prototype allows the developers to focus on what IS known, rather than spending time and effort trying to define everything that isn't know yet.

Bottom-Up design fails to design and build the system that you need now, but is more focused on the total system that you will need. Thinking that way, the system fails to take into account the ripple effect of design changes into the defined interfaces. Any flaws in a Bottom-Up design take more time to repair due to the cascading effect of changes across many, partially defined services.

LOC is a Pointless Metric

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.

—Bill Gates

If the head of a major company holds that opinion, what could that imply? To me, it means that you are tracking something that has no direct bearing on the end product. It's related to the product, no doubt about it, but it's a function of other factors, most notably functions or use cases, depending on if you are developing bottom-up or top-down.

The pointlessness of LOC as a metric is directly related to its nebulous nature. What makes a line depends on how you measure it. Is a line truly one line in the source, or is it more like a statement? But what defines a statement? A semicolon in Java? Anything between braces? Something else? When building in assembly, each line of code actually did something to the computer. But how do you count an external library function call? It's an over-simplification to think that a line takes x amout of time to create and test, since each line is almost totally unique from the one preceding it.

A LOC ain't what it used to be. How do you count lines when some of those lines are generated? And then, when your application gets refactored (you do refactor software, don't you), and your LOC goes down, then what? But there's time spent learning the tool to perform the code generation, time spent learning the tool syntax, and that time can't be accounted for until the tool is chosen and integrated.

LOC is considered useful for the reasons that make it useless. You are tracking and managing one of the details of the project as if it has a bearing on the progress of the project. Imagine if my productivity was measured by the feet I walked per day. Less feet = more time at my desk = more productivity. Simple, measurable, but totally inaccurate. Simply because a LOC cannot be defined other than by convention, any use of it as a metric is debatable at best.

So what? Simply, bid per high level requirement. Each requirement costs x dollars to implement, test and maintain. It's not a perfect system, but it would save time and money during analysis by not pretending to perform an estimated analysis on anything other than the functionality that has been requested.