I’m going to assume that you’ve read my original post, but in one line, the reason many large organizations still use Internet Explorer 6 is application compatibility. My employer (a large government department) has recently experienced this first-hand, having had to postpone deployment of Internet Explorer 8 after discovering compatibility issues (mid-deployment) with some important business applications. I have no idea how come they only discovered the issues after starting to deploy. Either someone didn’t do due diligence and test before-hand, or worse, they were completely clueless about the risks of incompatibilities. Either way, as a fellow IT employee in the same organization, it’s down right embarrassing.

One of the most frequent comments I receive as feedback on my original post is that folks get the compatibility issues, but they believe that we wouldn’t be in this situation if these incompatible applications had been written using web standards. It’s the fault of the application developers that their applications only work properly with Internet Explorer 6.

It is definitely true that Internet Explorer 6 isn’t very standards compliant… well, not by 2010 standards anyways. Thing is, these incompatible apps weren’t written last week, or last year. The state of web standards was very different when many of these applications were written.

Time for a little web history lesson. Let’s go back in time, about 13 years or so to late 1997.

HTML 3.2 is an W3C recommendation (HTML 4.0 will be a recommendation by the end of 1997, but will be updated and re-published again in April 1998). CSS level 1 is also a W3C recommendation. In the world of browsers, the world is basically using Netscape 4.0 or Internet Explorer 4.0 (I’m ignoring Opera, cause lets face it, most other folks did too). Both of the 4.0 browsers support HTML 3.2 and some CSS 1, but both browsers have serious bugs in their CSS implementations. At this point, other than HTML 3.2, neither browser is really “standards compliant”. Opera was closer, but web site designer/developers weren’t about to implement standards compliant sites if it would only work in Opera.

Fast forward a couple years to 1999. HTML 4.0 has been published (twice) as a W3C recommendation, as has CSS level 2. Internet Explorer is now at version 5.0 and Netscape has… essentially vanished from the scene. The Netscape story is a long and interesting one, but in a nutshell, they open sourced a large part of the code, started building version 5 on top of that, but discovered that the code was a complete disaster. So they tossed it out and started from scratch. Netscape 5.0 would never see the light of day. Internet Explorer 5 on the other hand included improved CSS and HTML standards support. In fact, Internet Explorer 5.0 for Mac was the first browser to achieve full support of CSS 1 (it had even better standards support than Opera). Internet Explorer 5.0 was also the first browser to include XMLHttpRequest, the basis for AJAX, the technology on which almost every major web site today depends (there is no “#NewTwitter” without AJAX & XMLHttpRequest). XMLHttpRequest in IE5.0 wasn’t script accessible, but it was there behind the scenes.

Move forward another year to 2000. Nothing has changed in the world of W3C web standards (they went off and focussed on XML, XSLT, etc. for the next few years, I think) Netscape 6 is re-born, with a largely re-written codebase. But the browser is bloated, and generally doesn’t work well on any PC that isn’t relatively new. By the time a couple of minor versions (6.1 & 6.2) come along to improve the performance situation, the world has been introduced to Internet Explorer 6.0. Shortly thereafter, the world is also introduced to Windows XP, with which comes a bundled IE6. This is about where I (and a huge chunk of the rest of the world) made the switch to Internet Explorer as my personally preferred web browser. Both of the IE and Netscape 6.x browsers support the current W3C standards, but the realm of CSS 2 in particular there is a huge difference in the implementation. Both products still contains major bugs and in many cases each product has interpreted some aspects of CSS 2 differently (so-called CSS 2 errata, clarified in CSS 2.1). In fact, the next revision of CSS, version 2.1 will drop parts that nobody implemented correctly, and will change other parts to match how the browsers of the day actually worked. The browsers drove CSS 2.1, rather than the other way around. By this time, it is important to note, Internet Explorer 6.0 accounts for almost +90% of the browser market share. Internet Explorer 6.0 is for all intents and purposes, the standard. It has bugs in its implementation of CSS (most notably in the box model). But in the first few years of the 21st century, if you build a site to 100% W3C standards, it’s going to be broken in +90% of the worlds browsers.

This is the era in which these applications which are incompatible today, in 2010, were built. To the Internet Explorer 6 de facto web standard.

Over the next 4 years, the situation doesn’t change much. HTML stays at 4, CSS stays at 2. Netscape and Opera (and several other smaller browsers) continue to release updates with improved implementations of W3C standards, but can’t seem to gain (or re-gain) any significant market share. In fact, Internet Explorer remains at version 6 (with a couple of service packs added) and peaks at +95% market share. Of course Microsoft is also been investigated for anti-trust issues related to their bundling of Internet Explorer with their operating systems. All this to say that frankly they haven’t got much motivation to release newer browser versions. The only other significant thing to come out of these years is XHTML, which frankly was a misguided detour which thankfully will likely be abandoned as a result of the adoption of HTML 5. The reasons why deserve their own post, but in a nutshell, there isn’t a single released version of Internet Explorer available today which supports XHTML. At least not when served by a web server using the correct MIME-type (application/xhtml+xml). So most web servers deliver XHTML with a MIME-type of text/html which most browsers parse as HTML (not XML or XHTML). Even if your site is XHTML compliant, if you serve it with the correct MIME-type to a supported browser, it won’t look the way you’d expect. Only a tiny percentage of sites are XHTML coded and served, on purpose and by design, using the correct MIME-type. And most of those are owned by computer science academics and grad students. But I digress.

Things begin to change at the end of 2004, when a little browser known as Firefox begins to actually take a chunk out of Internet Explorer’s market share. What made Firefox different from the dozens of other browsers that had tried before? I honestly couldn’t tell you. Probably just a good example of right place at the right time. I know I jumped on Firefox at about 1.0 and haven’t looked back. But Firefox re-ignited the so-called browser-wars of the late 1990’s, but in the 2000’s one of the things that earns you points is standards compliance. It also re-ignited the actual development of web standards: CSS 2.1 (2005 & 2007), CSS 3 (2005) and HTML 5 (2008 draft).

The thing I want you to take away from this post is that these critical business applications which today are holding us back from moving forward with more standards compliant browsers were in fact developed to the standard that existed in their day. Namely, Microsoft Internet Explorer 6’s (buggy) interpretation of HTML 4 and CSS 1&2. With +90% market share, for better or for worse, Internet Explorer 6’s version of the standard was THE standard.

Well, responses to that included “don’t punish users for your deployment issues” and “are you afraid of losing your job if I upgrade my own browser”? Oh, and the term “visionless IT geeks” was tossed around. My response to this was a flurry of tweets quoting other folks rhyming off reasons why large organizations all over still use Internet Explorer 6. I then signed off Twitter and did not log back in for two days.

I essentially had a hissy-fit.

Screw you guys… I’m going home! – Eric Cartman

Not particularly mature, but what can I say. I’m passionate about IT and took things a bit more personally than I should have. But I’ve spent several days since thinking about it and have decided there is an opportunity here to share some knowledge of how the IT group in a large organization views a web browser upgrade. As folks who know me can attest, I am not one to miss an opportunity to share (or receive) knowledge, so here goes.

First, a little about me for the sake of context.

• I am an IT guy. My entire professional career has been IT, all within the same department (the name has changed more times than I can count, but it’s still the same department). I am NOT a LAN support/Helpdesk person. I am NOT an IT standards person. I am NOT an IT Security person.

  In other words I personally have NO control what so ever about what gets installed or not installed on your GoC workstation (nor do I have anything to do with what Internet sites are blocked, by the way). I am simply a programmer, and like everyone else, I have to deal with the consequences of what “IT” deems suitable.

• While my entire career has been IT, I spent the first several years of it OUTSIDE the IT group, in what would be labelled a shadow IT group. A shadow IT group is basically folks that the business people deal with to get things done. They get stuff done independently, and regardless of what the official IT group says. We had our own servers, our own un-filtered Internet connection and our own, separate network and workstations (we ran Linux on our desktops). To be clear though, our stuff was completely separate from the corporate stuff. We were a much smaller threat than the average shadow IT shop (I don’t mean threat to IT jobs, but threat to the security/stability of the department).

  So I know first hand the “I/We can do it better than IT” mentality that exists in large organizations, outside of the IT group. Been there, done that.

• I also side-line as a web developer/designer. I own a company that started life as a web design firm. I am VERY well aware of the limitations of Internet Explorer 6. Access to my own stuff is broken when I am at work.

So, why one standard browser?

Once upon a time there was a large government department with some 24,000 employees and two Internet browsers, Netscape 4 and Internet Explorer 4. This was in the height of the so-called browser wars and things that worked in one browser didn’t necessarily work in the other. Applications were increasingly being developed as web-based applications, because it was simpler: they can be centrally managed and there is no need to worry about incompatibilities between products on workstations. Think about it: standardize on one web browser, make your web application work for that web browser, and your application will just work for everyone! Bliss!

And that’s exactly what happened. Internet Explorer was likely chosen because it is basically built right in to the Windows operating system. You can’t really uninstall Internet Explorer, like you can Netscape. So Internet Explorer it is. I suspect the same story played out in other departments, and in fact in other corporations around the world.

One browser also has the advantage that it means only one product to monitor for vulnerabilities and patches/security updates. Internet Explorer updates, additionally, can be deployed using the same infrastructure that is publishing Windows operating system updates, Office Suite updates, etc.

Time marched on, Netscape all but disappeared from the scene and for awhile Internet Explorer was just about the only game in town. During this time, internal applications continued to be developed targeting Internet Explorer 6. Third-party companies developing software for large organizations could also pretty much guarantee that those organizations were using Internet Explorer 6 so they to targeted their software at that platform.

Fast forward to today and that 24,000 employee department has 100s (if not 1000s) of applications that were built to target Internet Explorer 6. Some just work in Internet Explorer 7, others have some minor things that don’t, and some just don’t work at all. And the story is even worse when we contemplate a switch to a completely difference browser, such as Firefox, Opera, Safari or Chrome.

But Internet Explorer 7 is free. There is no cost to upgrade.

Okay, let’s talk about where that reasoning breaks down. Say that in our 24,000 employee organization, 500 use Product X, sold to us 7 years ago by Company ABC. Product X works with Internet Explorer 6, but doesn’t work at all with Internet Explorer 7. Product X is an essential tool for those 500 people to do their jobs.  We contact Company ABC for an update for IE 7, but it turns out Company ABC was bought 2 years ago by its competitor, Company 123. Company 123 has no update, but instead has their Product Z with works with IE 7. And it will only cost $10 million for licenses. And $1 million for a consultant to help migrate data from Product X to Product Z. And $1 million to train the users on Product Z.

But you know what? I don’t think we can’t legally just upgrade from one company’s product to another. We have to go through a Request For Proposal process, to allow other company’s to also compete on the contract. Optimistically that’s a 2 year process, just to acquire the new software. The process also requires the business area’s time, because it’s their software. They have to make sure that the replacement is capable of handling everything they need it too. Oh, and the business area has to pony up the money too.

Think about that from the business side’s point of view. You’ve got 500 employees working each day, happy doing their jobs. Then IT comes to you and says “We’re going to upgrade to IE 7. But first we need 2 years of you time (for the RFP alone, plus deployment, migration and training) and $12 million of your dollars (doesn’t include salary dollars). I know what my response would be, but I am too polite to actually publish it here.

And that conversation would replay 100s of times across the organization.

Internet Explorer 7 is nowhere near free. Not even close.

Internally, or in-house, developed applications are a similar story. They need to be upgraded, but the resources that would be used for the upgrade have to be prioritized. Government, business/program and IT priorities all have to be balanced. So instead of upgrading existing applications for Internet Explorer 7, IT is implementing Common Look and Feel 2.0 before the Treasury Board deadline passes, or they’re re-themeing existing web sites because the old red and white theme was too “Liberal” for the current government’s tastes (seriously, this is what you tax dollars get spent on).

Oh, by the way. Internet Explorer 7 requires Microsoft Windows XP or better. For a 24,000 employee organization, an on-going migration from Windows 2000 to Windows XP is an expensive and time consuming task, which suffers from its own set of compatibility issues (they are applications in use that don’t work in Windows XP).

Upgrading a web browser on your PC at home is an easy task. In a large government department however, it’s a huge tangled web of issues.

I was a little worried about working from home. I wasn’t sure how productive I’d be. But as it turns out, I am more productive at home than I am in the office. Here is a quick pro and con list of my experience working from home so far:

Pros:

• I control the work environment (temperature, lighting, etc.)
• Quiet! I am all by myself all day (heaven for an introvert like me)
• More screen real-estate. At the office I have dual 17″ monitors (both @ 1280×1024). At home I am using a laptop (@ 1600×1050), plus the laptop is hooked into my 2nd monitor via the KVM (@ 1280×1024) PLUS I have my 24″ monitor (@ 1920×1200) connected to my home PC, which I can use for web browsing, media, etc). That’s over 5 million pixels in all!
• Quiet! I can get focused and stay focused (again, for an introvert, getting focused, or getting back into focus, can be difficult)
• Lunch and snacks are closer, and cheaper.
• I have full access to the office network (via VPN) and my office PC’s hard drive.
• Better chair than the one in my office.
• I can play my music as loud as I want!

Cons (both relatively easy to resolve):

• The laptop only has 1 GB of RAM!
• I haven’t been able to get the laptop to share my mouse and keyboard (the mouse works, but it’s laggy. No wonder though: USB Mouse -> USB to PS/2 adapter -> KVM -> KVM Cable -> PS/2 to USB adapter -> laptop). Update: Resolved via laptop docking station borrowed from the office.

Here’s my home office setup:

It’s a basic .Net web forms application (in C#). Their issue was that they had one page that was taking upwards of 75 seconds to load! Seriously. 75 seconds! They said they were using NHibernate and they had tracked the issue down to one method that was taking up the majority of those 75 seconds. They also said they had tried some direct database (Oracle) queries that returned MUCH faster than the NHibernate.

We suggested that they look at caching the results of the database queries and that they confirm exactly what queries NHibernate is making on their behalf (I think they assumed it would execute the same query they had tested directly, which may or may not be the case).

Now I am a fan of NHibernate, and I have done some things with it that resulted in less than stellar performance, but I’ve always found tweak things to near direct database performance levels without much effort by changing/correcting some basic assumptions I had about the data and model. So when I left the meeting I felt that NHibernate had unfairly been labelled as the source of the performance issues, so I decided to have a look at the code for myself.

After grabbing a local copy of the code from their source control repository, I ran the code in debug mode and quickly came to the conclusion that the method they had identified as taking a long time to execute was in fact really taking a long time to execute. So I had a closer look at that method, and here (essentially) is what I saw:

/* ... */
IQuery query = CreateQuery("SELECT ...");

for (i = 0; i < query.List().Count; i++)
{
    if ((!((object[])(new ArrayList(query.List())[i]))[0] == null)
       && (!((object[])(new ArrayList(query.List())[i]))[2] == null))
    {
        String value = (((object[])(new ArrayList(query.List())))[2]).ToString();
    }
}
/* ... */

I am “paraphrasing” a bit (the real code was worse).

The query.List() call returns an IList instance containing all the results of the SQL query. I don’t know if each call to query.List() actually executes the database query again (there may be some caching involved somewhere) but each call to query.List() did involve some database activity (according to my network sniffer). You notice that query.List is being called 3 times per loop (in the real code it was more like 6 times per loop)!

Changing this code to make one call to query.List, storing the results in a local IList variable and subsequently re-using that local variable shaved nearly 50 seconds of the response time.

On top of that there would also be improvements by removing the code that creates 3 ArrayLists per loop (again in the real code, 6 ArrayLists per loop), copies the IList of database results into each ArrayList, uses each of those ArrayLists to access only one single item and them discards the ArrayLists (only to recreate them again on the next loop).

NHibernate wasn’t the source of their problem…

Here is a simplified version of three of the classes I was given to work with: Address, CorrectedAddress and FormattedAddress. An Address is what you would provide to the verification software and the other two classes would be returned from either the address correction or address formatting functionality.

Public Class Address
    Public AddressLine As String
    Public City As String
    Public Province As String
    Public PostalCode As String
    Public Country As String
End Class

Public Class CorrectedAddress : Inherits Address
    Public Status As CorrectedAddress.Status
    Public Message As String

    Public Enum Status
        Valid
        Invalid
        Corrected
        [Error]
    End Enum
End Class

Public Class FormattedAddress
    Public AddressLineOne As String
    Public AddressLineTwo As String
    Public AddressLineThree As String

    Public Status As FormattedAddress.Status
    Public Message As String

    Public Enum Status
        Valid
        Invalid
        [Error]
    End Enum
End Class

Note that CorrectedAddress inherits from Address, but FormattedAddress does not. Both CorrectedAddress and FormattedAddress have a Status and Message properties (implemented as public fields above only for brevity). The Status properties both return similar, but not identical enumerations. In the actual code there were six classes like these two, but I am focusing only on these two for simplicity.

The whole thing didn’t seem quite right to me, and I struggled a bit to put my finger on how this should have been implemented. But then it hit me. CorrectedAddress isn’t really an Address. It’s a result of an address correction (that happens to have an address). “Is A” versus “Has A”. Remember the oft quoted Object-Oriented design principle:

Favour Composition over Inheritance

So I pulled the Address class out of the inheritance hierarchy, renamed CorrectedAddress and FormattedAddress to CorrectedResult and FormattedResult respectively and even threw in some Generics stuff to deal with the similar but different Status enumerations.

The result (the Address class was left as-is):

Public MustInherit Class Result(Of T As Structure)
    Public Status As T
    Public Message As String
End Class

Public Class CorrectedResult : Inherits Result(Of CorrectedResultStatus)
    Public Address As Address

    Public Enum CorrectedResultStatus
        Valid
        Invalid
        Corrected
        [Error]
    End Enum
End Class

Public Class FormattedResult : Inherits Result(Of FormattedResultStatus
    Public AddressLineOne As String
    Public AddressLineTwo As String
    Public AddressLineThree As String

    Public Enum FormattedResultStatus
        Valid
        Invalid
        [Error]
    End Enum
End Class

Much better!