Yuriy Zubarev

Cynical Agile and Scrum Dictionary

Tue, 24 Apr 2012 00:00:00 -0700

As a reader with a sense of humor,
I want to find down-to-earth definitions,
So that I can smile, frown, and reflect.

Agile

a faith
is whatever you want it to be to achieve what you need to achieve

Agile Certification

a pilot license that both an eagle and an elephant can easily obtain

Agile Manifesto

a document that tipped the balance from analysis paralysis to refactor distractor
a proof that principles are often mistaken for rules
a license for avoiding documentation, and not having a plan

Backlog

a land of forgotten dreams

BDD

an ironic example of a development community building tools for non developers, and ending up being the only ones using them

Burn Down

a condition of a Development Team at the end of a Sprint

Code Review

the most common example of Technical Debt
a process of curing root rot by trimming leaves

Continuous Delivery

a blockbuster sequel to Continuous Integration

Continuous Integration

a good idea turned into a semi-automated Continuous Disaster

Daily Scrum or Daily Stand-up

weekly two-hour long meeting
the three forbidden questions we never speak of
a license not to talk to your team members in between stand-ups

Development Team

is the same as Development Team in waterfall but with an added illusion of empowerment

DevOps

a new breed of professionals who can write “Hello World” and deploy it to Heroku

DDD

a methodology so advanced that it could only be used once to produce the venerable TimeAndMoney
Deadline Driven Development

Emergent Design

a process in which parents who read Tarzan abandon their infant in a forest only to see it grow into Bigfoot

Enterprise Agile

an oxymoron

Evolutionary Architecture

a process equally capable of producing hummingbirds as well as vampire bats

Extreme Programming (XP)

a methodology that was so extreme that it’s not practiced anymore

Feedback

a rare and valuable currency of Agile: everybody wants to receive it, nobody wants to give it away

Impediment

a manifestation of organizational culture

Kanban

a paradox where the Software Development industry tries very hard to differentiate itself from traditional manufacturing systems by borrowing a method originated in those very systems
an example that anything sounds more legitimate if it’s in Japanese

Pair Programming

an unsolved puzzle for economists, managers, and compensation specialists

Planning Poker

a parody on a real poker: everybody bluffs that they don’t know the hands and the next moves, and pretend to have fun

Product Owner

a guardian to protect Stakeholders from an unwashed Development Team

Refactoring

a word one says before one sins
a source to bail out the global economy if Development Teams had one dollar for every promise to refactor

Scrum

a religion
a commercialization of Agile

Scrum/Agile Project Management Software

a kick in the stomach to the first principle of Agile Manifesto

Scrum Master

a person who removes stumbling blocks and in so-doing eventually self destructs
a person with a stopwatch, a squeaky toy, and a whistle

Software Craftsmanship

an effort to turn one’s hobby into a full time job and have one’s employer to pay for it

Spike

a gas pedal for a Velocity of a Sprint

Sprint

an enchanted quest to collect magical Story Points

Sprint Planning

a meeting where bells and whistles are prioritized over nuts and bolts

Sprint Retrospective

a mystical meeting that is always skipped to give the next Sprint more time

Sprint Review

a meeting ignored by Stakeholders
a ceremony where a production crew is watching their own movie and is handing themselves the Oscar statuette

Stakeholders

people who secretly hate Agile

Story Points

a relative measure of absolute inability to estimate

TDD

an approach to build a car with hundred air bags and no engine

Technical Debt

a measurement of a sentiment of a Development Team
a plague ship that always sinks to the bottom of a Backlog

User Story

an oversimplification of requirements to the level where it’s embarrassing to read it, so nobody does

Velocity

the next best thing after real results

xDD Family of Methodologies

a choir made up of prophets each singing their own hymns

Screaming architecture and the (one and only) architecture diagram

Tue, 06 Dec 2011 00:00:00 -0800

It’s a wild wild west in the Software Architecture land! Uncle Bob and Simon Brown are not deferring to draw guns and fire bullets wrapped in carefully chosen words. The argument is centered around what is considered to be architecturally significant elements, and how long to defer “annoying details”: http://www.infoq.com/news/2011/11/Debate-The-Annoying-Detail. This is almost as raw and exciting as it gets in our land. We should probably get a 30 minute spot on Spike TV.

I want to go to the start of the argument – “Screaming Architecture” article: http://blog.8thlight.com/uncle-bob/2011/09/30/Screaming-Architecture.html. I understand what Uncle Bob was trying to say, and I believe he had only best intentions in his mind, but I’m weary of the language of absolutes and concocted examples. Mostly important, the original article perpetuates the notion of “the one and only architecture diagram” which is rather harmful.

The gist of the argument could be extracted from the first couple of paragraphs:

Imagine that you are looking at the blueprints of a building. This document, prepared by an architect, tells you the plans for the building. What do these plans tell you?

… As you looked at those plans, there’d be no question that you were looking at a house. The architecture would scream: house.

So what does the architecture of your application scream? When you look at the top level directory structure, and the source files in the highest level package; do they scream: health care system, or accounting system, or inventory management system? Or do they scream: rails, or spring/hibernate, or asp?

Ahh, the building blueprints! Those big, fancy and surprisingly easy to understand technical drawings that we see in spy and heist movies. Here is an entrance, here is a corridor, here is a switch to turn off the alarm, and finally here is a door and a safe! What we don’t see are real life blueprints of heating, ventilation, cooling, electrical, water supply and sanitation systems. To an untrained eye those blueprints will scream absolutely nothing. Are they unimportant annoying details? Sure, if you don’t mind spending a winter in Winnipeg with no heating, electricity, toilet, hot water and inadequate wind protection.

When you look at the top level directory structure, and the source files in the highest level package; do they scream: health care system, or accounting system, or inventory management system?

Judging architecture of a software system by top level directory structure is like judging building architecture by a construction site. Construction sites don’t scream library or school or a city hall. They scream: raw materials. Yes, of course, it would be nice if the top level directory would be called “amazon” and the sub-directories – “books”, “magazines”, “electronics”, etc. We would know right away what we’re dealing with. The difference between construction industry and software industry is that we invent and deal with levels of abstraction. For a non trivial software system those abstractions could be multiple levels removed from something holistic that would scream back at us. At the level of the source code the abstractions are the raw materials.

If your architecture is based on frameworks, then it cannot be based on your use cases.

Yes it can, if a framework is based on my use cases.

Enough of nitpicking. The bigger issue here is a pernicious trend similar to magnum opus. While alchemists were trying to discover a philosopher’s stone, we have a tendency to contrive The One And Only Architecture Diagram. As such, it should scream every answer to any question.

There are many questions to be asked about a software system. What is it for? Who is it for? What does it do? How does it do it (that’s a loaded one)? What does it consist of? How does it get deployed? How does it handle concurrency? How does it handle outages in dependent services? How does it map to the source code? Let’s stop trying to pretend we can answer all questions with one architecture diagram. Better yet, let’s stop avoiding those questions because we cannot address them with one diagram and let’s educate ourselves.

Obscure Inheritance or Homomorphism

Tue, 22 Nov 2011 00:00:00 -0800

Here is a rather simple class diagram:

At the first glance it makes sense and it’s rather self explanatory. If you’re an experienced developer you can actually scream “bloody murder” right away, but let’s take it slow. Here is a hint that something evil is happening here. The rest of a system has an explicit dependency on Displayable interface, which is a good intent, but the details are very ominous. Every implementation block always has the same structure:

if ("list".equals(displayble.getName())) then {
    ...
} else if ("paragraph".equals(displayble.getName())) then {
    ...
} else if ("section".equals(displayble.getName())) then {
    ...
}

This is a complete fiasco in abstraction, plain and simple. The code above might as well look like:

if (displayable instanceof DisplayableList) {
    ...
} else if (displayable instanceof DisplayableParagraph) then {
    ...
} else if (displayable instanceof DisplayableSectionName) then {
    ...
}

The only thing that the base interface exposes is getName() method which is akin to Object.getClass() method. It doesn’t encapsulate any useful behavior, it just provides a hook to query object’s implementation structure.

Depending on the original intent, this could be an example of either Obscure Inheritance or Homomorphism or both.

Inheritance is a way to reuse code of existing objects, establish a subtype from an existing object, or both, depending upon programming language support.

In this situation there is zero code reuse. Sub-typing is rather superficial: it offers no more than a fact that the classes have getName() method. The method itself is not what consumers care about, but a hook to make further decisions based on essentially internal structure of a class. This is the example of Obscure Inheritance.

In computer science, polymorphism is a programming language feature that allows values of different data types to be handled using a uniform interface.

The word derives from the Greek “πολυμορφισμός” meaning “having multiple forms”.

This case presents the exact opposite: different data types are handled without a uniformed interface. The presence of “if” statement per data type is the culprit. This is an example of Homomorphism.

Alternative

This family of classes deal with tree-like structures to display lists, headings, paragraphs, etc. There is a software design pattern that deals with this exact concern: composite pattern. The key benefit, as applied to this case, is “… lets clients treat individual objects and compositions uniformly”.

TDD aha moment

Wed, 09 Nov 2011 00:00:00 -0800

Here is a new tag line for TDD: “spend more time with your family”! TDD is an approach to a software development that ultimately saves time during a construction and maintenance cycles. Want to take your kids to an early game of their favorite team, or disappear for a couple of days with your significant other? If you do, then you cannot afford to waste hours per day restarting your applications and navigating to certain locations just to check a presence of a link. You cannot allow yourself to wait for servers to start up, and do a manual testing. On a more global scale, you cannot be interrupted by a call during a date because of an embarrassing bug in production. If you could shave off 20 minutes from one hour by test automation (not atypical), it runs up to saving about 3 hours during one day, and 2 days in a course of one week! Here is another tag line: “TDD – go home early”. Emergency calls at night make you drive to the office before the sun-rise? “TDD – stay home late”. Ok, ok, emergency calls at night cannot be completely eliminated, but the number can be reduced.

The aha moment came to me one day when I made a tiny silly mistake. I wanted to fix it right away to regain some self-respect. It was one of those errors that you think even a human with a half brain cannot make, so writing a test for such a situation is below you. The fix would be so simple and straightforward and it would take me just 10 seconds to do it. Then I started thinking about starting a server, selecting a set of data that would trigger the exceptional situation and making a “visual sign-off” to confirm the correct behavior. This cycle would have taken me about 10 minutes. My enthusiasm started to fade away little by little. Aha! It would take me only 5 minutes to write a failing test, fix the code and re-run the test.

I went home 5 minutes earlier that day.

PS: TDD is a serious subject. There are excellent arguments out there why TDD is not just a fad, but an essential software engineering practice. For example, in an article “What Is Software Design?” (1), Jack W. Reeves suggests among other things that “… it is cheaper and simpler to just build the design and test it than to do anything else”. It’s a fascinating read where Jack convinces you that engineering rigor in software is testing.

I should also recommend “Neal Ford on Agile Engineering Practices” (2) from O’Reilly Media to learn more about TDD.

(1) http://www.developerdotstar.com/mag/articles/reeves_design.html
(2) http://shop.oreilly.com/product/0636920020271.do

PSS: TDD is not all or none proposition. I believe that a goal of 100% test coverage might be harmful if it’s just a goal by itself without a broader context and a consideration of trade-offs. But that’s a topic for a separate discussion.

Set-up for Testing Solr backed Scalatra app with SBT and ScalaTest

Sat, 20 Aug 2011 00:00:00 -0700

We are in a process of rolling out Scalatra application that will be powering our new inventory Web Services at Dominion Marine Media. Everything about the new stack has been new to me, and it took me some time to get the project in a state where an automated unit/functional testing is set up in a decent way. Here I’m going to share the project set-up and hopefully it will help somebody to save hours or days. I fully aknowledge that there might be a better way of doing the same thing, so you’re warned.

Technology stack:

Scala 2.9
Solr 3.3
SBT 0.10
Scalatra 2.0.0.M4
ScalaTest 1.6

Project layout and the key files:

my-project\
    |- build.sbt
    |- project\
        |- build.properties
        |- plugins\
            |- build.sbt
    |- src\
        |- main\
            |- scala\
                |- MyResource.scala
                |- SolrServerProvider.scala
            |- webapp\
        |- test\
            |- scala\
                |- EmbeddedSolr.scala
                |- MyResourceTest.scala
                |- ScalatraSolrSuite.scala
            |- resources\
                |- solr\
                    |- solr.xml
                    |- conf\
                        |- schema.xml
                        |- solrconfig.xml

Project files:

my-project\build.sbt:

...

scalaVersion := "2.9.0"

resolvers ++= Seq(
  "Sonatype Nexus Releases" at "https://oss.sonatype.org/content/repositories/releases"
)

libraryDependencies ++= Seq(
  "org.scalatra" %% "scalatra" % "2.0.0.M4",
  "org.scalatra" %% "scalatra-scalatest" % "2.0.0.M4",
  "org.mortbay.jetty" % "jetty" % "6.1.22" % "test",
  "org.mortbay.jetty" % "servlet-api" % "2.5-20081211" % "provided",
  "org.apache.solr" % "solr-solrj" % "3.3.0",
  "org.apache.solr" % "solr-core" % "3.3.0" % "test",
  "org.apache.solr" % "solr-test-framework" % "3.3.0" % "test",
  "commons-codec" % "commons-codec" % "1.5"
)

seq(webSettings :_*)

my-project\project\build.properties:

sbt.version=0.10.1

my-project\project\plugins\build.sbt:

resolvers += "Web plugin repo" at "http://siasia.github.com/maven2"    
libraryDependencies <+= sbtVersion(v => "com.github.siasia" %% "xsbt-web-plugin" % ("0.1.1-"+v))

my-project\src\main\scala\SolrServerProvider.scala:

import org.apache.solr.client.solrj.SolrServer

trait SolrServerProvider {

    def solrServer(): SolrServer

}

my-project\src\main\scala\MyResource.scala:

import org.scalatra._
import scala.collection.mutable.{ArrayBuffer, Map}
import org.apache.solr.client.solrj.{SolrServer, SolrQuery}
import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
import org.apache.solr.client.solrj.response.QueryResponse
import org.apache.solr.common.SolrDocumentList
import collection.JavaConversions._

class MyResource extends ScalatraServlet with SolrServerProvider {
    
    get("/owner/:partyid/boats") {
        val query = new SolrQuery()
        query.setQuery("my query").addField("myfiled")
        val rsp: QueryResponse = solrServer().query(query)
        val docs: SolrDocumentList = rsp.getResults()

        // generate appropriate output
    }
    
    override def solrServer(): SolrServer = {
        new CommonsHttpSolrServer("URL to your Solr instance")
    }
    
}

my-project\src\test\scala\EmbeddedSolr.scala:

import org.apache.solr.client.solrj.SolrServer
import org.apache.solr.util.TestHarness
import org.apache.solr.core.SolrConfig
import org.apache.solr.SolrTestCaseJ4
import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer
import org.apache.solr.common.SolrInputDocument

trait EmbeddedSolr extends SolrServerProvider {

    override def solrServer(): SolrServer = {
        return InstantiatedOnceEmbeddedSolr.solrServer()
    }

    def addToSolr(solr: SolrServer, fields: Map[String, String]) {
        val document = new SolrInputDocument()
        
        fields.foreach { case(key, value) => document.addField(key, value) }
        
        solr.add(document)
        solr.commit()
    }
    
}

object InstantiatedOnceEmbeddedSolr {

    private var solr: SolrServer = _

    def solrServer(): SolrServer = {
        if (solr == null) {
            System.setProperty("solr.directoryFactory","solr.RAMDirectoryFactory")
            val dataDir = new java.io.File(System.getProperty("java.io.tmpdir"), getClass().getName() + "-" + System.currentTimeMillis())
            dataDir.mkdirs()
            val configFile = getSolrConfigFile()
            System.setProperty("solr.solr.home", SolrTestCaseJ4.TEST_HOME())
            val solrConfig: SolrConfig = TestHarness.createConfig(getSolrConfigFile());
            val h: TestHarness = new TestHarness( dataDir.getAbsolutePath(), solrConfig, getSchemaFile())
            val lrf = h.getRequestFactory("standard", 0, 20, "version", "2.2")
            solr = new EmbeddedSolrServer(h.getCoreContainer(), h.getCore().getName())
        }
        solr
    }

    def getSolrConfigFile(): String = {
        """solrconfig.xml"""
    }
  
    def getSchemaFile(): String = {
        """schema.xml"""
    }
    
}

my-project\src\test\scala\ScalatraSolrSuite.scala:

import org.scalatra.test.scalatest._
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.BeforeAndAfterEach

trait ScalatraSolrSuite extends ScalatraFunSuite with BeforeAndAfterEach with ShouldMatchers with EmbeddedSolr {

    val solr = solrServer()
    
    override def beforeEach() {
        solr.deleteByQuery("*:*")
    }
    
    def addToSolr(fields: Map[String, String]) {
        addToSolr(solr, fields)
    }
    
}

my-project\src\test\scala\MyResourceTest.scala:

class MyResourceTest extends ScalatraSolrSuite {
    addServlet(new MyResource() with EmbeddedSolr, "/*")
    
    test("get two documents for the owner") {
        addToSolr(Map[String, String] ("DocumentID" -> "11", "OwnerPartyID" -> "100" ))
        addToSolr(Map[String, String] ("DocumentID" -> "21", "OwnerPartyID" -> "100" ))
        addToSolr(Map[String, String] ("DocumentID" -> "31", "OwnerPartyID" -> "101" ))
        
        get("/owner/100/boats") {
            status should equal (200)
            body.vrCount should equal (2) // "vrCount" is our implicit 
        }
    }
  
}

And finally the coveted test results:

$ sbt test

[info] ...
[info] MyResourceTest:
[info] - get two documents for the owner
[info] ... many other tests ...
[info] Passed: : Total 42, Failed 0, Errors 0, Passed 39, Skipped 3
[success] Total time: 24 s, completed Aug 20, 2011 10:28:48 AM

Airplanes are Obsolete

Mon, 11 Jul 2011 00:00:00 -0700

NASA declared airplanes obsolete and not capable of meeting demands of orbital and interplanetary space flights.

“We’re not just talking about propeller aircrafts anymore. Even the latest airliners like Airbus A380 and Boeing 747 fall short of modest expectations of simple sub-orbital space flights.” – commented Joanna Igobins, the official NASA spokesperson. “How is one supposed to reach an altitude of a mere 100 km if those birds already want to nose dive at 15 km? And even if one could, it would take a heck of a long time doing so at 1,000 km/h” – she added forcing an awkward smile.

Chief NASA technologist John Binogis was even more critical: “Space exploration is not on a fringe anymore. Today you may need to cover a range of only 15,000 km, and the next week you might be thinking of traveling distances of 100 and 1000 times longer. Turbines got you here but they are not going to thrust you to the Mars. It was a good technology for its days and those days are over. The days of Airbus and Boeing are over as well. They are trying to sell us their outdated technologies at the time when everybody knows that rocket-powered spacecrafts can handle bigger payloads at significantly longer distances.”

“You can trick your grandma but you ain’t tricking me. You cannot call yourself a pilot if you’re still flying those dinosources. It ain’t no cool, it ain’t no hardcore.” – stated Johny Gobi, a young astronaut with over-developed sense of self-importance.

Jouahn Gisbinos, a representative from one of NASA fulfilment partners observed that: “… we haven’t received a single order for atmospheric crafts in the last 3 years. We are recommending to all of our customers to start looking past conventional airplanes if they don’t want to be left behind”.

Sounds silly and far-fetched? Not if you’re in IT and software industry. It ain’t no hardcore if you’re not running NoSQL on the cloud. After all, that what Facebook might be doing, and clearly, we all have the same scalability concerns as they do.

Macleans - Our Biggest News Issue Ever

Thu, 12 May 2011 00:00:00 -0700

It’s a funny coincidence. Last week I unsubscribed from Macleans magazine and earlier this week I received yet another confirmation supporting my decision. Macleans proudly stated on the front: “Our Biggest News Issue Ever”.

The magazine flirted with increasing of its size for awhile already and that’s exactly what got me thinking of cancelling my subscription. Every time I received a bit fatter issue bragging about gaining extra pounds, I wanted to throw it back at the publisher. If you don’t want me as your customer, just tell it directly, I won’t be insulted. In this day and age when information overload is every day reality, increasing a dose seems to be rather counter-productive. I have blogs, RSS, twits, wall posts, emails, podcasts, books and e-books to follow, read and provide feedback. Did I mention video casts and presentations? Do you think I have extra time or energy to digest your extra pages? Or even leaf through them? Heck no. I do appreciate reading analysis, investigations and different takes on popular issues but not at an expense of having to spend more time on it when there is so much medium competing for my eyes and ears.

Here is an idea, Macleans. I caught myself thinking couple of times that I would be willing to pay more for thinner issues…

Time Travel

Thu, 12 May 2011 00:00:00 -0700

Competent Communication. Project 2.

I always had a very narrow vision of a time travel. Having seen Hollywood movies on this subject, I had a picture of being strapped to a chair and moving convulsively, closing my eyes, letting out a wild scream of pain and then, all of a sudden, feeling woken up hundreds of years in the past. With this vivid imagination there was also a realization of unlikely plausibility of the whole enterprise. I hoped for it, but deep inside I didn’t believe in time travel. This all changed last year… when I travelled to Iceland.

I don’t know exactly how many years back I went but the time shift was felt everywhere. It was evident in the place itself and in the people who lived there.

First of all there was this nature as I never saw it before. At the time I was already spoiled by our magnificent West coast. Only more treacherous mountains and more thicker and taller trees could evoke a sense of awe in me. This is what I thought before being completely awestruck by a unique beauty of lava fields, stone deserts and… moss. Moss! I never thought one could be so amazed by moss. I always pictured it as wet, nasty and bacteria infested stuff leaching to a rock in some cold and dark place. Iceland moss was nothing less but a luxurious carpet with intricate tapestry and bright colors stretching miles and miles in every direction. Countless ship were peacefully grazing all around while being completely oblivious to strong winds, frequent rains and how naturally they fit into this silent picture of perfection.

Nature surprises didn’t stop there. Next came a lifeless sight of a rock desert. It was cooked by mixing a cooled magma with geothermal chemistry, and produced Moon landscapes right here, on Earth. Every picture of a Moon surface I ever saw, manifested right in front of my eyes, indistinguishable from the real thing, as much as I could judge. It was very clear to me then why NASA sent Apollo astronauts to train into this very desert.

Couple of days into my travel around the island I noticed a troubling trend. I saw so much beauty that I was getting de-sensitised by it. Looking at dozens waterfalls on each side of a road made me speechless just yesterday, but today these densely packed sites of amazement didn’t inspire but irritated me. “Oh, please, another breath-taking waterfall! Is it even real?!”. I managed to quell seeds of modern cynicism and went on to be further inspired.

The highlight of my trip was one of the natural wonders of Iceland – Jökulsárlón, also known as Glacier Lagoon. It presented a picturesque parade of ghostly and luminous blue ice-bergs. Ever since I was a child, I wanted to see and touch an ice-berg. I don’t remember the root of this desire, but I always wanted to stand next to a still and cold giant and touch it with an open palm. There is something majestic about them. Describing them as chunks of frozen water is so simplistic and unimaginative. The sheer size, unlimited combination of forms and ever so intricate reflections of light, make those boulders look like pedestals for all the drama and triumph unfolded during tens of thousands of years as nature shaped itself. I took about 500 pictures of the lagoon and I still cannot reduce them to a manageable number – no two pictures are alike when you are admiring something as harmonious as a lake full of ice-bergs.

No travel story could be complete without mentioning people you meat on your journey.

One evening, after a regretful whale watching trip where I disgorged plenty, I was sitting in a restaurant in a northern-most harbour, together with other tourists. We met just earlier that day and a conversation was warm and engaging. What struck me the most, is that the island that is relatively young in geological terms, and therefore somewhat situated back in time, drew people who wanted to go back in time themselves. I remember this young woman from Germany earnestly trying to understand the appeal of Face-book but not able to put her finger on it. In her opinion, it just didn’t have the same punch as sending a friend a physical postcard from a remote place, or picking up a phone and having a conversation. She then went on to describe advantages of a traditional film camera. Limited number of available shots makes you think twice about a composition and it is the best antidote from being a shutter-happy. Next she completely won me over with her description of a time when she waits for pictures to be developed in a photo studio. Not knowing how pictures are going to come out and discovering them one by one – is a joy, she explained, that can only be matched by a joy of taking these pictures in the first place.

I will always remember native Icelanders by two aspects. First, their language seemed to be made up entirely form consonants; and second, their faces were a living example of natural purity. I am in a danger of over generalizing here or making observations on a relatively small sample, but I cannot forget the admiration I experienced while looking into their eyes. There was a complete absence of anxiety and a pointless haste. It was so refreshing. All I saw was kindness and serenity… And how difficult is it to remain calm when red signals on many traffic lights have a heart shape?! I dare you to have a road rage when you look at a heart. How difficult is it to be placid if you can leave your bike outside of your apartment building during night, unsecured and fully assembled, and find it in exactly same conditions the next morning?

I don’t if whether it’s a place that makes people who they are or the other way around, but there is something magical about Iceland. It draws me back, and I hope, that may be in 10 years, I will re-discover it again – the secret of a time travel.

Reach Your Escape Velocity

Tue, 10 May 2011 00:00:00 -0700

Source: http://ecorner.stanford.edu/authorMaterialInfo.html?mid=2725

This notion of the problem of we’re stuck and it just — I have to say it’s virtually universal. People kind of get, look, we have a successful franchise but you can’t stand still intact between technology and globalization. Things are really, really moving. Now, that’s great news for opportunity but it’s also for people that stand still, it’s a threat. As our friends at Boarders, for example, discovered this year and numerous other companies. So everybody gets that they’ve got to engage growth. It’s not like, hey, I’ve got an idea for you. They got it. What they don’t like talking about which they universally acknowledge is there is massive internal resistance to moving resources away from established activities into new activities; massive.

So the problem is the new stuff, it’s not that there’s not new stuff, there’s tons of new stuff but it doesn’t ever reach materiality. Meaning, in never reaches like being 10 or 20% of the total company’s revenue. It gets announced, it gets started and then somewhere along the line it dies from the vine. And so that’s where we came with this notion. These are questions that basically our clients ask us. That’s where the title came from. How can we achieve a escape velocity? How do we free ourselves from the pull of the past? So just to make clear, this is not a trivial problem nor is it trivial companies that have struggled with this problem.

So the question then becomes, what the heck is going on?

So what’s the mistake we keep making over and over again? So it turns out the way that the global management system works, and largely this is a management system that was developed in the United States, but it’s become completely global. It’s a performance-oriented management system. That’s how people get compensated in large corporations.

And we’re good at managing it, but here is the second sense, power fuels performance and performance consumes power. That is not visible in a large corporation.

Well, consuming power in most compensation systems, in most corporations is free. And further more, creating power, say around a new strategy, is not compensated. So I get compensated for consuming power for my performance. I do not get compensated nor do am I held accountable except in the most kind of qualitative ways for generating power. So why would you be surprised that large organizations systematically consume their power?

And so what the — we created a model called the hierarchy of powers. It’s kind of a framework of frameworks. And the intention of this hierarchy is to let people sort out the power equation in a set of levels that make sense. So the hierarchy of power is just five powers but you talk about each power separately from the other four and you talk about them in sequence from top to bottom.

… category power has to do with how fast are the categories that you are in growing? Because if you’re in a category that’s growing 100%, even if you’re kind of a doofus, you’re probably going to do pretty well this year. And investors aren’t stupid. By the way, vice versa, if you’re in a category that’s growing at 4%, you have enormous pressure on you to make sure you make every penny of your earnings. And it’s very, very hard for you to do much better than that because the category is giving you no lift whatsoever. You can take share in a category that’s not growing but not forever. Eventually, you run into a wall. So investors at the margin would prefer your company to be in high-growth categories, not low growth. So a company like General Electric would routinely sell off low-growth businesses in order to get capital to buy high-growth businesses and they would kind of migrate the footprint of the corporation going forward. That’s in order to have more category or power. You can hold your company accountable to what category is you’re in. The board of directors can hold the management team accountable to that.

The next one is company power. So in the categories you’re in, how powerful? Do you control your own destiny or somebody else is calling the tune and you’re just trying to keep step and keep pace with them? This is market share. This is Jack Welch saying, you want to be number one or number two or you don’t want to play because you wouldn’t have enough company power to get the return on being in that category. You ought to get out of that category and get into some categories where you can be number one or number two. So it’s about power.

The third one is market power that has to do with, are you in the markets that are the fastest growing segments or the more strategic segments? Now, category and market sometimes get confused but category is defined by a group of competitors and markets are defined by a group of customers.

So a market is about getting a group of customers and the idea here is a little bit like winning primaries for a presidential nomination. You want to win in New Hampshire primary. There are not very many delegates in New Hampshire but is a very strategic primary to win. And in markets, there are always segments that are early bellwether segments that are different for different markets, different technology, different categories, well, there’s always some. And so winning those early or maybe there’s a segment that’s in transition where the segment like this particular part of the economy is completely being disrupted by the Internet this year so whatever they’re going to buy, they’re going to buy this year. So get in there and be successful if it’s the kind of stuff that you sell. That’s what market power is about; winning segment shares in the critical segments.

Offer power is based on how compelling is your offer compared to the rest. We’re talking about differentiation. How different is it, but then also how comparable is it to the — are you living up to the norms of your category?

And then execution power is, when you guys set yourself out to do something, when you’re going to enter a new category, when you’re going to do one of these transformational initiatives, can you actually get it done or do you find yourself kind of redoing it?

… One of the things Steve is about is he cares about power and he assumed performance will take care of itself. Jeff Bezos is the same way to Amazon, that’s why periodically, he does stuff that just drive the shareholders crazy because he gives away performance in the short-term because he says, I want the power.

OK, great, great, great. What do you do? How do we start?

You can start at any point in this hierarchy. You can start with the category power level which tends to be we need to look at portfolio. We tend to have way too much money in aging categories with the Boston Consulting Group calls, cash cows and we don’t have enough rising stars. So that could be the problem. Or we can say, no, it’s a company power problem and the problem is we can never — first of all, we can kill nothing and second of all, we can never really make any big bets. Every big bet we hedge and after a while we’re like the person at the roulette table who put the chip on every single square. So we win every time. Except, we just lose money. So on the market focus, we can actually go after markets in a directive way. It turns out it’s extremely hard for large corporations to go after niche markets. But it’s critical. It’s critical because otherwise, you’ll end up coming in second or third in every primary in the entire election and you don’t get nominated, right?

There are ways to organize inside a large company to create — carve out an entrepreneurial space not just to incubate innovation because all of them do that but to actually go from an incubated innovation to a material business. That’s when they all get killed and so there’s a whole thing around execution power and organization for dealing with that.

In general, by the way, venture investing is about power and public exchange investing is about performance if you think about it. Because in a venture-backed company, there’s not enough revenue to really say the performance is the game changer. It’s power is the game changer in venture.

I am now — what is amazing about tech by the way, is that product managers — the unit of wealth creation in tech all of my life has been the product. Now increasing the service as a sort of invisible product. The product and product management is largely conducted by people in the first 10 years of employment. So all of a sudden, relatively early in a career, unlike most professions, most other industries, you actually have your hand on the tiller that changes that fate of your company. So it’s an amazing privilege to work in a high tech company because the amount of power you get short — particularly if you can get into this role of product marketing and product management.

And whenever you charter a single team in a single project stream to do two or more, you’re screwing up. And I would say that 99% of all the project streams I see are combinations of those three things which is a fairly high ratio for screwing up.

… Pretty obvious and pretty exciting and by the way, every engineer in the world wakes up thinking that’s what I do for a living. I’m the smartest person on the planet and I will be able to demonstrate it. Neutralize, catch up with the competition, nobody likes to do this. Nobody likes to do this. But they get — but it’s part of being an adult.

So differentiation. First of all, well, it’s different but the thing is if you’ve ever actually noticed the world, everything is different. There’s no two things that are the same, right? So different isn’t a big deal, OK?

This differentiation along a vector value, OK, so how much differentiation do we have and can we go beyond the limits of the competitive set?

So you can’t just say I’m different, you have to say I’m also meeting the norms of the category in the stuff that I’m not trying to differentiate…

So the net differentiation is what — as an offer manager, product manger, service manger, that’s what I want you to think about. Every year my power increased because my net differentiation increased relative to my competitive set in my category.

There’s a sentence in that thing, the new CEO of Nokia, he writes a memo that’s a public memo to the world which says how is it possible that Apple came out with an iPhone in 2007 and Nokia does not have response? Would you want to answer that question? Gee, Steven, I wasn’t the product manager.

But the point was — so ask yourself why. Why do you think? I don’t know but my bet is because the engineers at Nokia were too proud to neutralize because they wanted to differentiate. They wanted to over leap the over leaper. And when you do that, you leave year after year after year of no effective response in the market which is absolutely fatal.

Google apps versus Microsoft Office, Google is not trying to beat Microsoft Office, it’s just trying to annoy them. It’s just trying to keep them busy and to kind of keep something going. Yahoo on the other hand, was more like Nokia. Yahoo wanted to overtake Google Search. It was kind of like Captain Ahab and the white whale and it was a quest. And it was a fatal quest. And so what Carol had to do is get in and sell off the business because she had to free her company’s future from the pull of the past. That was a really, really important thing for her to do.

Apple and Kindle, so what did Apple do with Kindle? It co-opted it. Kindle — most people — most read their Kindle books on iPads because they co-opted it. Whereas the borders and their thing…

… This is what Microsoft does better than any company in the world by about an order of magnitude and maybe not so much anymore as they used to. But this is the thing, this is why Microsoft could routinely let anybody innovate anything and say I’m going to take it away from you. I’m going to take it away. I’m going to be good enough very quickly and then eventually I’m going to assimilate it either Windows or Office. One of the two. All life will be part of either Windows or Office.

OK, so this leads to a concept we call one playbook per project. Now, the way you fund things in the world probably you have one budget that has to be split among those three things. All I’m asking you to do is don’t tie the differentiation train to the neutralization train or either of them to the optimization train. Have three trains manage them separately. You manage it one for speed, one for distance and one for fuel.

… Or optimization — optimization projects that don’t really go after the sacred cows. So you optimize everything except sales and engineering. So you just beat the — out of the people that are in the cafeteria but if it’s an engineer or sales person, oh, wow, I can’t touch those two. Well, 85% of your head count it’s 100% of the things that are making the difference, so it’s like crazy enough to optimize important things.

Developing Products that Save Lives

Sat, 07 May 2011 00:00:00 -0700

Source: http://ecorner.stanford.edu/authorMaterialInfo.html?mid=2716

I have to say I give the company a lot of credit because, basically, I think it has paid off for them. But it took me two years before I had any idea what I was even doing in business. I mean, I knew about science but it really was a steep learning curve but not something that happens overnight.

One of the huge differences is, I found that in business, you actually get feedback on how you’re doing.

Well, gee, people say I’m aloof, dismissive and arrogant and I can understand this. I mean, my parents said that. My wife says that, my colleges at Stanford say that. You say that. I cannot figure out how you’re all wrong in the same way.

So, I would say, from the concept of “maybe this molecule would work in this disease” to actually marketing the drug, well, it depends. If it went incredibly fast it could be ten years. It’s more like 15 years and the average cost is about $1.5 billion to get it to the market. Now the $1.5 billion includes the failures. So that’s taken into account. So how many make it? That varies from company to company. I would say probably on the average 10 percent make it. For us, it’s probably more like 25 percent. But I think the industry is going to do much, much better in the next decade. And the reason is that there’s been an explosion in knowledge about biology.

I voted for Obama that is charging Genentech $550 million this year to pay for Obama care. Now, where do we get $550 million? We fired $550 million worth of people. Where else are we going to get the money? So now they don’t have jobs or health care.

What are the things do you wished you have learned when you were in school? What would have been the most valuable things that would have helped you really hit the ground running when you were walking into a really important leadership position?

People ask that and I think any time that I would have spent getting an MBA, for example, wouldn’t have been worth it. I really think that the reason I was hired is because I was a really, really good scientist and taking-you notice the was-any time away from that would have frankly been a mistake, I think. And the only reason our company will be successful in the future is if the right technical decisions are made. And people don’t get that. It all depends on the science. If we pick the wrong target, if we make the wrong molecule, if we choose the wrong disease, we try and develop it and it doesn’t or run over and over again, the company is finished. It’s all driven by the scientific decisions that are made at the early stages and through development. You can have all… I mean we need the commercial folks. So they’re terrific, I love going to the sales meeting and the manufacturing people and the finance folks, that’s all absolutely critical. But if the science is wrong, there’s no foundation, there’s no basis for the company. So I think just becoming the absolute best scientist that I could be in being just so focused on that was actually the best use of my time to be the best head of R&D. So, nothing. Nothing different.

How much science is needed between your job versus the management and leadership skills?

You can go really far in life with a little common sense. The kinds of leadership issues that come up, I just sort of resort back to my roots and my beliefs and why I’m at Genentech. If you keep patients in mind, and if you’re always doing the best thing for patients in your decisions, usually, it’s just pretty clear what to do. So it’s sort of hard for me to say much more than that. And by the way, I have a small lab. So I have every Tuesday at noon, a lab meeting. And most of the higher level management people at Genentech have a lab. I think that’s really, really important. Otherwise, it gets too easy. You go sit and some big rooms and when one makes a presentation, you say, “Do this, spend some money here, do that,” You forget how hard it is to actually make an experiment work unless you’re sitting with your lab group. And then when you see how “Jeez, it didn’t work again. What are we going to do?” It keeps you grounded in reality. So I think that’s extremely important and I’m extremely proud of the fact that the high level scientist at Genentech still have labs and white papers and so on.

Performance Testing - Starting Out

Thu, 17 Mar 2011 00:00:00 -0700

We recently started implementing performance testing procedures for our products in my company. The trigger wasn’t an execution of some thought-out plan but a desperate measure following a service disruption unfortunately. I always thought that we had more time. I have to admit, it was very naive and irresponsible thinking. Regardless of the precursor, here are some initial observations after I started looking for performance testing practices.

First of all, in comparison to all resources dedicated to scalability and NoSQL, the web felt semi abandoned when it came to performance testing. The reason for this comparison is not arbitrary. My line of thinking was that before jumping on scalability initiatives, and before moving from relational databases to document ones to address performance concerns, for example, people would first need to show that those are, in fact, concerns. I thought I wouldn’t have any problems locating first hand accounts on how teams approached performance testing and found pressing needs to tackle scalability and try out NoSQL. No such luck. I hope folks choose NoSQL because it’s a better tool for certain jobs, not because it solves imaginary performance issues.

Search results for performance testing tools was a list of well known and old suspects. Some of them weren’t updated in years. By itself it’s not such a bad thing. The tools could have matured and reached a level where frequent updates weren’t necessary any longer. The biggest surprise was a discovery of almost perfect vacuum in place of how-tos and tutorials on integration performance testing as a part of an application lifecycle management. No set-ups shared, no words of wisdom, no proverbial best practices.

There are two reasons that I can think of to explain this:

1) Performance testing is bordering art and magic, and it’s very dependent on the application or system being tested. There may be hundreds of companies in my city only, that do this every day, but their set-ups are so custom that they don’t feel sharing any details would benefit anybody else.

2) Performance testing is not really common. Applications go down generating wake up calls to development and operation teams, who them scramble in haste to fix the problem.

What is your experience?

Frantic pace of technology

Fri, 23 Jul 2010 00:00:00 -0700

The latest issue of “Computer” has an informative article “Simplicity as a Driver for Agile Innovation” with the following gem:

… The frantic pace of technology provides its own chaos: before a certain technology reaches maturity and can repay the enormous investments for its development and production, a newer option attracts attention with novelty and fresh promises.

This quote succinctly describes the environment of IT and underlines my own 10+ years of experience. At every point and with almost every decision we have to balance out maturity of incumbent products and promises of new ones.

Ominous hashCode() method

Fri, 05 Mar 2010 00:00:00 -0800

I just spent couple of hours debugging a subtle problem only to pinpoint the root cause to an erroneous implementation of hashCode() method. It was a nice refresher in Java basics. Here is the culprit:

  @Override
  public int hashCode() {
    return super.hashCode() + (this.code == null ? 0 : this.code.hashCode());
  }

Any guesses?

CAS - Non Sharable Credentials

Fri, 26 Feb 2010 00:00:00 -0800

We have a requirement that a certain type of users cannot use the same credentials to have two or more simultaneous active sessions. I called this feature as “Non Sharable Credentials”. Probably there is a better name for it, but this one will do.

I first posted a question to cas-user mailing list and got a rather encouraging response. Even though CAS doesn’t provide such functionality right out of the box, it seems possible to search ticket registries to find existing ticket for a given principal. Luckily we have JPA-backed registry and at the time I didn’t quite understand why Scott Battaglia cautioned against loading the entire table of tickets. Couldn’t I just craft an appropriate JPA query?

Upon further inspection of database table-backed ticket registries it was quickly discovered that a principal for a ticket doesn’t have a dedicated column. Instead, this information is a small part of a serialized object stored in a BLOB field. I couldn’t query for it directly, so I had no choice but load records into memory, de-serialize the BLOB field and then check for my principal until the match is found. This approach could seriously slow down authentication process and therefore was quickly ruled out.

I really needed an efficient way to search for tickets belonging to a specified principal. I didn’t want to modify CAS code and change structure of database tables for the ticket registries. There is always a new version of CAS and an upgrade should be as easy as possible. Instead, I create another simple table:

CREATE TABLE `principalticket` (
  `ticket_id` varchar(255) NOT NULL,
  `principal` varchar(255) default NULL,
  PRIMARY KEY  (`ticket_id`)
)

The purpose for this table is self-evident. The real question is how to populate it every time a new ticket is issued. CAS is a perfect example of well written code thoroughly utilizing Spring framework. I decided to use Spring AOP and have an aspect that would be run as a ticket gets persisted. A snippet from deployerConfigContext.xml:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
       http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">

  <aop:aspectj-autoproxy/>

  <aop:config>
    <aop:aspect id="ticketAddAspect" ref="nscBean">
      <aop:pointcut id="interceptingAddTicket" expression="bean(ticketRegistry) and execution(public * add*(..))"/>
      <aop:around pointcut-ref="interceptingAddTicket" method="addTicket"/>
    </aop:aspect>
  </aop:config>

  <bean id="nscBean" class="com...NSCBean">
    <constructor-arg index="0" ref="entityManagerFactory" />
  </bean>

  ...

I wanted to use the same approach for deleteTicket pointcut on ticketRegistry bean but I kept running into AOP related exceptions when a scheduled quartz job was trying to clean up ticket registries. I was not AOP guru and I had to finish the task in 2 days and I opted for a less glamorous solution of removing records from principalticket table. If I get back to this task and refactor the approach then I will update this post as well.

NSCBean (NonSharableCredentialsBean) is rather simple:

package ...;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import org.aspectj.lang.JoinPoint;
import org.jasig.cas.ticket.TicketGrantingTicket;
import org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint;
import org.springframework.orm.jpa.JpaTemplate;

public class NSCBean {

  private JpaTemplate jpaTemplate;

  public NSCBean(final EntityManagerFactory factory) {
    this.jpaTemplate = new JpaTemplate(factory);
  }

  public void addTicket(JoinPoint point) throws Throwable {
    if (point.getArgs() != null && point.getArgs().length > 0 && point.getArgs()[0] instanceof TicketGrantingTicket) {
      TicketGrantingTicket ticket = (TicketGrantingTicket) point.getArgs()[0];
      if (ticket.getAuthentication() != null && ticket.getAuthentication().getPrincipal() != null) {
        String principal = ticket.getAuthentication().getPrincipal().getId();
        EntityManager em = this.jpaTemplate.getEntityManagerFactory().createEntityManager();
        if (em.createNativeQuery("select * from principalticket where ticket_id = '" + ticket.getId() + "'").getResultList().size() == 0) {
          em.getTransaction().begin();
          em.createNativeQuery("insert into principalticket (ticket_id, principal) values ('" + ticket.getId() + "', '" + principal + "')").executeUpdate();
          em.getTransaction().commit();
        } else {
          // ticket/principal is already recorded
        }
      }
    } 

    if (point instanceof MethodInvocationProceedingJoinPoint) {
        ((MethodInvocationProceedingJoinPoint) p).proceed(p.getArgs());
    } 
  }

}

Records from principalticket table are cleaned up in a rather contentious way. During authentication process, if a ticket doesn’t exist in ticketgrantingticket but exists in principalticket then it gets deleted from the latter. Authentication itself, checking for non sharable credentials and cleaning up the new table is all part of a custom AuthenticationHandler. There is no point providing its code here as it deals with lots of logic particular to my company.

This post explains one way of achieving non-sharable credentials with CAS. At the heart of the approach is a usage of Spring AOP. AOP is usually advocated while dealing with cross-cutting concerns but it also comes very handy when you want to extend functionality of an existent product beyond its predefined extension hooks.

CAS "branded" Authentication Filter

Mon, 15 Feb 2010 00:00:00 -0800

Seemingly complex tasks can have surprisingly easy resolutions. All is needed is not trying to make things more complex than they are.

We have two domain names for two distinct categories of customers: www.example1.com and www.example2.com. These domains point to the very same Web application – Super Widgets 2.0. The application has to redirect new visitors to CAS log-in page. Super Widgets 2.0 provides “service” (or “serverName”, please read on) parameter to CAS so that CAS can redirect users back to Super Widgets 2.0 after a successful authentication. “service” parameter comes from web.xml of Super Widgets 2.0 and it’s a constant value as far as CAS Client is concerned. So the “service” value can either be http://www.example1.com or http://www.example2.com. When it comes to CAS, it knows where to redirect users after the authentication but it doesn’t know the original URL users tried to access (CAS Client doesn’t pass Referer HTTP header).

On CAS log-in page we would have to know the original URL so that we could greet users with either “Welcome to Example 1” or “Welcome to Example 2” messages (can be easily achieved with a simple JSP scriplet).

In the nutshell the solution (suggested by my colleague Alexey Malyshev) is to initialize and call org.jasig.cas.client.authentication.AuthenticationFilter from within our own custom filter. As we initialize Jasig’s AuthenticationFilter we calculate “service” value dynamically based on the value of request.getServerName(). I’m not really sure what design pattern should be selected form the list to describe this approach.

First, we need a custom but simple implementation of FilterConfig:

[sourcecode lang=“java”]
package com.superwidgets;

…
import javax.servlet.FilterConfig;
import javax.servlet.ServletContext;

public class BrandingFilterConfig implements FilterConfig {

private FilterConfig parent; private Map<String, String> params = new HashMap<String, String>(); public BrandingFilterConfig(FilterConfig parent) { this.parent = parent; } public String getFilterName() { return “Super Widgets Branded Filter”; } public ServletContext getServletContext() { return this.parent.getServletContext(); } public String getInitParameter(String s) { if (this.params.containsKey(s)) { return this.params.get(s); } else { return this.parent.getInitParameter(s); } } public Enumeration getInitParameterNames() { return this.parent.getInitParameterNames(); } public void setParam(String key, String value) { params.put(key, value); } public void removeParam(String key) { params.remove(key); }

}
[/sourcecode]

Second, we need to implement our own Authentication Filter:

[sourcecode lang=“java”]
package com.superwidgets;

import javax.servlet.Filter;
…

public class AuthenticationFilter implements Filter {

protected final Log log = LogFactory.getLog(getClass()); private String serviceMapping; private Properties serviceProperties = new Properties(); private String serverNameMapping; private Properties serverNameProperties = new Properties(); private BrandingFilterConfig filterConfig; public String getServiceMapping() { return serviceMapping; } public void setServiceMapping(String serviceMapping) { this.serviceMapping = serviceMapping; } public void init(FilterConfig fc) throws ServletException { this.filterConfig = new BrandingFilterConfig(fc); // service this.serviceMapping = fc.getInitParameter(“serviceMapping”); log.info("serviceMapping = " + this.serviceMapping); try { if (this.serviceMapping != null) { this.serviceProperties.load(new ByteArrayInputStream(this.serviceMapping.getBytes())); } } catch (Exception ex) { log.error("Cannot load serviceMapping property: " + ex.getMessage()); } log.info("serviceProperties = " + this.serviceProperties); // serverName this.serverNameMapping = fc.getInitParameter(“serverNameMapping”); log.info("serverNameMapping = " + this.serverNameMapping); try { if (this.serverNameMapping != null) { this.serverNameProperties.load(new ByteArrayInputStream(this.serverNameMapping.getBytes())); } } catch (Exception ex) { log.error("Cannot load serverNameMapping property: " + ex.getMessage()); } log.info("serverNameProperties = " + this.serverNameProperties); } public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { org.jasig.cas.client.authentication.AuthenticationFilter realFilter = new org.jasig.cas.client.authentication.AuthenticationFilter(); String hostKey = request.getServerName(); log.debug("hostKey = " + hostKey); if (serviceMapping != null) { if (serviceProperties.containsKey(hostKey)) { this.filterConfig.setParam(“service”, serviceProperties.getProperty(hostKey)); } else { this.filterConfig.removeParam(“service”); } } if (serverNameMapping != null) { if (serverNameProperties.containsKey(hostKey)) { this.filterConfig.setParam(“serverName”, serverNameProperties.getProperty(hostKey)); } else { this.filterConfig.removeParam(“serverName”); } } realFilter.init(this.filterConfig); realFilter.doFilter(request, response, chain); realFilter.destroy(); } public void destroy() { // nothing to do } public Properties getServiceProperties() { return serviceProperties; } public Properties getServerNameProperties() { return serverNameProperties; }

}
[/sourcecode]

Now we have to modify web.xml and everything falls into places:

[sourcecode lang=“xml”]
-

CAS Authentication Filter
com.superwidgets.AuthenticationFilter

casServerLoginUrl
https://cas.superwidgets.com/login

service

http://www.example1.com

serviceMapping

www.example2.com=http://www.example2.com www.example3.com=http://www.example3.com www.example4.com=http://www.example4.com

[/sourcecode]

“serviceMapping” is a secret sauce here. The value is loaded in an instance of java.util.Properties class and then it’s used to dynamically calculate “service” value for Jasig’s AuthenticationFilter.

If AuthenticationFilter is using “serverName” parameter instead of “service”, then instead of “serviceMapping” parameter, “serverNameMapping” should be used.

A little summary. AuthenticationFilter has a dynamic behavior based on the original URL a user was trying to access. To achieve this we wrapped Jasig’s filter inside our own filter and we didn’t have to change Jasig’s filter internals in any way. Jasig’s AuthenticationFilter filter is initialized on every request and this has a light and acceptable (for us) performance implication.

New York ITARC 2009, Final Thoughts

Sun, 18 Oct 2009 00:00:00 -0700

During the Architect Boot-camp Paul ran through perspectives people have about architects and architecture and one of them was Cult of Personality. The idea is that some people respond on challenges, uncertainties and arguments in a somewhat predictable way: “Grady Booch already figured it out”, “Eric Evans solved that problem already”, etc. After 3 days of lectures, presentations and discussions I think I found an antidote for the Personality Cult.

All you have to do is attend a similar industry event like New Your ITARC. The biggest discovery for me was how individually smart but collectively distant though leaders were. Everybody was paying a due respect for each other but was also quick to note that they disagreed with many points as well.

As a side observer I didn’t think any of the statements of any of thought leaders were particularly wrong. In my opinion it’s all about context. Give me a pattern and anti-pattern and I can turn them upside down by changing the context within which they were applied. The same is with thought leaders. They worked for decades on a certain slice of a huge IT Architecture pie and they were influenced and constrained by a corresponding context. They figured out patterns and anti-patterns but they were not universal. So when I saw, for example, Rogers Sessions disagreeing with Grady Booch, I was witnessing a clash of different opinions stemming from different contexts and nothing more than that.

Next time I develop a Cult of Personality, I only wish it’s going to be within the right context.

Day 1
Day 2
Day 3

New York ITARC 2009, Day 3

Sun, 18 Oct 2009 00:00:00 -0700

The third day started with probably the best keynote presentation of the whole conference. Of course I’m talking about Grady Booch and his “Thoughts on Architecture”.

If I had the whole day I would tell you all the things that I don’t understand about architecture. But I have 1 hour only, so I will tell you what I reasonably understand.

I took a picture of almost every slide from this presentation but I probably shouldn’t post them here. Even if the legal aspects don’t prohibit me from doing so, the bare slides wouldn’t make a justice to Grady’s talk. You have to be there to hear how intelligently he weaves the slides together and produces a though-provoking story on architecture. Here are just some notes I found particularly memorable:

One of the greatest challenges is to connect Enterprise Architecture to Software Architecture
Google is not search but an advertising companies. Interesting to see where they end up.
Innovation cannot be outsourced. Grady gave GM a really hard time here, calling their decision to outsource all software development the stupidest thing.
Grady was surprised and pleased to see how far UML went but he never intended it to be a programming language (shot at MDD)
It was interesting to learn that Grady was a very spiritual person. He is on the board of Iliff School of Theology in Denver.
Presence of software intensive systems changes the context in which they run – it offers additional capabilities.
Grady disagreed with previous comparison of architecture to a road, made by William H. Inmon on the first day. He said the road was just a tactical decision. Grady then spent some time going through his famous “Architecture is like river” work.
You cannot move the river. You can only move banks. The river will do what it will do.

There are many forces in the river ecosystem. Architecture is to balance those forces.

The industry has responsibility for legal, ethical and moral aspects of the architecture profession. This is one of the biggest challenges for the future.

Third edition of “Documenting Software Architecture” is coming!

The reason why we have so many EA frameworks is because we don’t have the right answer yet.

The reason why we have fewer number of technical frameworks is because we are closer to the right answer here. Example: 4+1

Models should be predictable and elicit questions

IRS has an assembly program comprised of 5000 lines they cannot replace. Only one person knew how to run tax rebates promised by government last year. Ouch.

Good documentation tip: if you can express it in programming language – don’t put it in UML

Google and Facebook also have architectural problems. (but he didn’t elaborate)

eBay spends 10% of their budget on refactoring

Grady’s presentation was unique. He has the best grasp of English language I’ve ever heard. Each slide was like a poem: deep and beautiful. There was something profound in every statement he made. He’s a very humble person and his eyes are paragon of sadness. I think they reveal the true love for what he does, acceptance of never ending challenges and a refusal to give up. He is an excellent ambassador for IT architects and it was my sincere privilege to see and hear him speak.

The next session was by Roger Sessions and it was called “Simple Iterative Partitioning”. Behind the cryptic name there was a straightforward message: complexity is the root of all IT troubles and we should relentlessly attack and eliminate complexity.

Roger introduced Glass’s Law and its application for IT. He used it to devise a concept of SCU – Standard Complexity Unit. He then used this combination to calculate a number that gave a certain design a definite numeric value for its complexity. This is an exciting development as far as I’m concerned. As a saying goes: “you can’t manage what you can’t measure”.

Here are some highlights:

Grady said we have to have simple systems. But how do we know the system is simple? Grady gave metrics but they are not enough. Sometimes they can only be applied to the system once it’s completely built.
Code re-use can sometimes create more complex systems.
There is no simplicity, there is a relative absence of complexity just like there is cold, there is an absence of heat.
Yesterday Len Bass presented us with six key non-functional architectural requirements, but where is simplicity among those six?
By measuring complexity of solutions and using SIP, we can part ways with “decibel based decision making”
If you want high availability, you build simplicity, not redundancy

There parting thoughts:

Complexity is the enemy
It’s simple to make things complex. It’s complex to make things simple, thus the paradox.
Only the simple will survive

The conference ended with a thought leaders panel. All the questions but one was asked by Paul Preiss. Here are some most memorable excerpts:

Q: what’s in store for architecture in 5 years?
- Scott Anderson: cloud computing will mean more changes to service companies than to architects themselves
- Lean Bass: a) we will see a bigger split between state of art and state of practice in the architecture; b) the trend is that we’re casting ourselves as dictators on what should be done
- Paul: questions of ethics and integrity will move closer toward central stage
- Angela: a) I would like to see us more as a community; b) architects in large companies will get a seat at stakeholders table
- Roger Sessions: problem of complexity will be the most critical one
- Grady Booch: a) software intensive systems will keep increasing in their complexity and we have to counter-balance it by increasing transparency; b) we will still be arguing what architecture means
Q: What would our commercial be?
- Scott: “Architects – we make order out of IT chaos”
- Paul: “Technology working for you and making you more human”
- Angela: “First call you make is for architects. Architects – foundation of your business”
- Roger: “Simplifying world one IT project at a time”
- Booch: “Join IASA and improve your sex live”. (did I miss something about this conference?)
Q: what would you take with you on a deserted island?
- Grady: “How to build a boat” book

New York ITARC 2009, Day 2

Sun, 18 Oct 2009 00:00:00 -0700

Today I had a notepad with me, so be ready for a long read. Additional observations about New York: mountains of garbage bags are along sides of streets ready for collections. I think it’s because there are no back alleys in Manhattan. So much different from Vancouver. I also tried and beat couple of cars by crossing on the red line. I got couple of honor honks. I felt completely assimilated.

The day was opened by Paul Preiss who talked about “Next Generation Activities”. After having gone to Architect Boot-camp, most of Paul’s talk sounded already familiar but there was new information as well. Paul said that he started in the industry as a programmer but he didn’t feel completely at home doing that. He noted that it wasn’t like he didn’t like programming, it just didn’t make sense for him. He didn’t like writing code for the sake of writing code. This sounded familiar to me. Sometimes I felt a void inside me, some dissatisfaction about my role and my contributions. I went on crazy programming spikes, creating solid and sometimes elegant code, and wholeheartedly enjoying the process of discovery and creation. I finished those spikes only to discover the void inside me grew even bigger. The code did solve some problem, but the amount of effort and sophistication was disproportionally big compared to the actual problem being solved. In many cases I was writing code and refactoring it only for the sake of writing code and refactoring.

Paul then went on to deliver one of his, in my opinion, main statements. He argued that we’re the only group that separates us from the rest of the company – “the business”. We’re just IT and we’re here to server “the business”. Finance doesn’t “serve the business”. They do what they are good at and they are part of the business. This caught me by surprise. I was completely content with being a second class citizen to “the business”. That is what was engraved in my head. Well, it’s a good practice to question everything and this was an excellent example.

This central thought led him to suggest that IASA was to make architecture actionable. He was surprised that there were no solid requirements for entry for profession that impacts so many people and influences budget of billions dollars. He joked that his hair-dresser had to meet more state requirements before they can practice their trade than an IT Architects had. Comparisons with professions in medicine was even more grimmer.

Paul continued and underlined that IASA chapter was not meant to be a user group. The chapter is there to support us and our career growth, give us jobs and make sure they’re paid well. I see where Paul is going and what he want IASA to become. What’s different about IASA that it’s not run by geeks. Paul is a business savvy and a shrew person and he wants to carve out a bigger piece of the corporate pie for IT Architects. He will have lots of challenges and resistance from other groups along the way. But this is where we, as members, here to support and help. He emphasized that he’s practicing architecture only 20% of his time and he’s selling it 80% of his time. This is what chapter leadership also has to expect.

He then shared some numbers, problems and priorities for IASA. The association has more than 8000 members. He acknowledged the IASA web site sucked, they knew about it and kept working. That’s nice to know. About 6 months ago I sent email to Paul to confirm IASA is still alive since the web site didn’t give me any signal of pulse at that time. IASA problems included being to big, underfunded and unfocused. The problem of size would be solved by giving local regions more responsibilities. Focus for the next year was to create filters for bad architects and recognize good ones through education, certification and accreditation.

Paul then introduced various aspects of body of knowledge for architects. He mentioned, among other things, rigorous taxonomy of skills and that the foundational skills for architects was Business Technology Strategy. He cut to the chase by stating that developers were there to write code and architect to get rid of code. Code has to make money. Architects should concentrate on delivering the most value for the least number of lines written. I tend to agree. This could be also a reason why some good coders fail to become good architects. They always stay coders in their hearts and cannot have no code written either by themselves or developers in their companies.

Paul also clarified IASA position that Enterprise Architect is not a specialization but a role. The comparison was with a chief of surgery in hospitals. I had a benefit of learning this statement from Max Poliashenko, digesting it and having agreed with it, but I wonder what Rogers Sessions would say. Paul then outlined four levels of Architect certification. This is the quick summary:

Aspiring architect – knowledge test
Associate architect – experience + knowledge test
Professional architect – experience + board exam
Master architect – thought leaders, speakers, writers and notable contributors

Local chapters would run certification board as local values matter for every profession.

Paul concluded that IASA’s mission statement is to empower the profession and, as an example, added that cloud wasn’t future. Cloud is just another tool. Future are people standing up and speaking for themselves and IASA is there help and facilitate.

I liked Paul’s speech (I can give Toastmaster evaluation as usual, if he interested). I think IASA has a good leadership. Paul possesses the right amount of technical and business knowledge, energy and charisma.

The next key note “Principles of Software Architecture Design” was delivered by Len Bass. Lean is a legend and I personally read couple of books he co-authored. Seeing him and listening him was a great opportunity. He started with outlining types of requirements and noting that requirements are good things. Requirements are constraints and the more we have the less decisions we have to make.

Len then delved into non-functional requirements and ways of their elicitation. He explained differences between categories and taxonomies and suggested to stick with categories. The essence is that in taxonomies, a thing can only fall under one taxonomy element. His example with “denial of service” proved to be a challenge for taxonomies. During denial of service, many non-functional requirements are impacted: availability, performance, security, usability. He offered then a common for for quality attributes:

Len followed with a really deep dive into concepts of ASP – Architecturally Significant Requirements. He started describing ATAH – Architecture Trade-off Analysis Method, utility trees and designing paths. One interesting thought he offered was that in many cases architecture and design testing are thought experiments. So much for xUnit frameworks. At some point I caught myself losing my attention. What was supposed to be a keynote speech turned into highly technical lecture by a university professor. His topic was much better suited for one of the tracks rather than a keynote address.

The day was coming to the end closing keynote speech was presented by Angela Yochem. It was titled “Architecture is the business”. This was a logical continuation of Pauls’s message and I liked this connection through a common theme. Paul introduced Angela with humor: “Somebody told me that enterprise architects are like unicorns. Yes, they would be pretty if they existed. But they do exists!”, and he gave her a stage.

Angela started with a message that already had a chance to be absorbed by the audience: we’re just as business as finance, HR, purchasing, legal, etc. We’re all partners. She then compared architects to a really bad salesmen: they annoy people by asking them for couple of hours of undivided attention so that they could learn about our problems and then offer us products and services to solve them. I recognized here myself on a couple of occasions…

Angela threw another observation that caught me by surprise: there was no reasons architects couldn’t apply for funding for some future benefits. She saw it done successfully. Hm… Here is a thought. Now I just have to come up with a value proposition.

For all us, wondering where to start, she suggested creating a functional view of our business or a segment to gain a perspective and to spot inefficiencies.

Angela concluded with a bitter pill. She paraphrased John Zachman: if you cannot describe your value to your company then may be you’re not bringing that much value. I’m definitely bringing value as a developer, designer and occasional operational support but hardly any value as an architect. Well, I honestly have to thank Angela for this perspective.

Brief Q&A session then followed:

Q: what organizational metrics could be used in architecture?
A: develop architecture scorecards. Track delivery of components and enablers by all company units that contribute to enterprise architecture.

Q: how to sell architecture initiatives?
A: Understand the value you deliver. Formulate the value as clear as possible.

Q: One thing you wish you knew looking back (asked by Grady Booch)
A: Waited many years to be noticed as a decision maker. The reason was not taking ownership of problems.

The day ended with a productive Open Space session. I heard about this concept from Java Posse but never participated before. I threw 3 topics in the mix:

1) Documenting software architecture
2) CTO vs. Architect role (since architects own IT strategy)
3) Company’s revenue level to afford dedicated architects

The first topic received a critical mass of votes and was included in the main session. Hedging the bets paid off. When the discussion started, my corner got fewer people than votes – it’s hard to compete with Grady Booch’s topic.

My discussion was interesting but not in the best way – everybody had questions but hardly anyone could offer answers. The group ended up sharing what worked for them and what didn’t – that the main point of Open Space. One person recommended Software Systems Architecture: Working With Stakeholders Using Viewpoints and Perspectives which I should definitely investigate. There was a consultant from Sparx System among us who advocated Enterprise Architect and a certain framework he developed. I also own a copy of this package. Almost everything he was suggesting wasn’t applicable to small and medium companies unfortunately.

For the second round of discussions I joined “Web Frameworks Components” discussions. I pitched in by describing our experience with Open Social API and Shindig in particular and contrasted it with Java Portlet API. Other people described their experiences with Flex, Google Gears, etc. After the session, one gentleman approached me with some questions about Shindig. We had a good conversation and found that we have similar set-ups in our companies. The last discussion I jumped to was revolving around a question of a single IT architect in a company. This situation was common among attendees. It looked like if a company had only one IT Architect, then what they really had was a senior developer and a team lead who happened to be called an architect for whatever reason. Oh, that sounds very familiar. Some people were seriously upset by not being able to escape from the chains of development, testing, deployment, etc. I offered a thought that “pure” IT architects are not needed for all companies. Not all companies have a dedicated lawyer. This is actually why we need a profession and this is why I wanted to discuss “Company’s revenue level to afford dedicated architects” topic. We also agreed that everybody in this position should do what Angela prescribed – articulate our value proposition as architects, and if we cannot – well, keep writing more code pal.

New York ITARC 2009, Day 1

Sun, 18 Oct 2009 00:00:00 -0700

I’m a lucky attendant of “”http://www.iasahome.org/web/itarc/2009/NYC">IT Architect Regional Conference" held in New York City from October 12-14. I signed up for the conference mainly because of the impressive line-up of speakers, many of which I admire professionally, and I also wanted to learn more about IASA, its president, its mission and other people behind it.

This was my first time being in Manhattan, New York. Some quick observations and impressions: it’s big, it’s insanely busy, seems like half of the traffic are cabs and people always try to cross streets on the red light. Being from Vancouver, the last point feels especially foreign. Oh yes, and on my first day walking to a conference, I saw a celebrity: Elizabeth Hurley.

The opening ceremony was started by Timothy Leonard who joined IASA less than an year ago as a president. He explained that he came on board with the sole purpose of adding more clarity to IASA’s mission and to grow the organization. The growth would be quantitative and qualitative. Members would now see the clear benefits of joying the association: body of knowledge, white papers, variety of on-line courses, certification and accreditation. Tim then invited VP of Education to expand on the last two benefits. Certification path IASA is taking is drawing from examples of other, more established, disciplines (medicine, construction architecture, etc) and it looks well thought-out and beneficial for the members and potential employers. Board certification is the most respectful aspect of the whole package. I hope IASA will make it tough (but relevant and fare) and will never back down from it. I have one architecture certification, which I will not name here, that wasn’t a piece of cake to obtain. It took me some time and effort to do an assignment and write two exams. Still, I would be so much more satisfied and proud had a board certification was a part of that process.

Everybody was treated by a keynote speech by John A. Zachman, the famous creator of Zachman Framework. John was a very passionate, animated and authentic speaker. It seemed like he wasn’t given all the time he would love to for his presentation, so he simply flew through his slides using an antiquated overhead projector. It’s amazing how much he managed to deliver though: the history of his studies, parallels between his framework and works of ancient philosophers and knowledge seekers, and conceptual similarities between his schema and periodic table of elements. Seemingly dry topic was sprinkled by his own funny stories which made everybody constantly engaged and the time went really fast.

Next there was an Architect Boot-camp – pretty much the main reason I came here for. The boot-camp was held by Paul Preiss, President and Founder of IASA. Paul is one of those people who “have you at hello”. His energy, passion and love for what he does and what he stands for is second to none. Being a member of Toastmasters club, I cannot help it but to evaluate each in every speaker in my head. Let me tell you that Paul has it all figured out. He did have quite a few “ums” and “ahs” and his body language wasn’t always sharp, but it didn’t matter in this case. He was prepared, he sounded authoritative, knowledgeable and personal. Vocal variety was rich and appropriate and his eye contact made me feel welcomed. OK, enough about Toastmasters stuff.

The main point about Architect Boot-camp is in its content. I was simply blown away. Five years that Paul and his associates spent on making IASA what it is and working on all educational material manifested itself right there and then. I will not reproduce content of his slides here, I just want to point out three things. First, I got my main question answered. I now know the value proposition of IASA and what differentiates it from other IT architecture associations and institutions. In the nutshell, IASA strives to make “IT Architect” a profession in its fullest meaning of the word. This is exciting. I cannot help but feel sometimes as a hobbyist and amateur compared to people from law, medical and traditional engineering fields. This doesn’t have to be this way. Second, the breath and depth of software architecture is staggering and it seems like IASA has all the bases covered. As I continue my studies I will test this assumption, but so far I’m impressed. Third, many things Paul mentioned really resonated with me and I want to be a more integral part of IASA.

The boot-camp was going at a tremendous pace, delivering lots of value, but then Paul had to leave and Cliff Berg stepped up to deliver the next chapter. This is where the fairy tale ended. The reason why this session was so useful and full with insight and knowledge was only in part because of content. The delivery of the content was an essential catalyst. Cliff is a seasoned IT professional and a smart man but he wasn’t ready to lead the boot-camp. I don’t mean any disrespect here, may be there was a miss-communication, lack of time, or any other objective reason. The end result was that we covered only about 20% of what was intended. Paul came back couple of hours later and finished the course as best as he could.

The day ended with another treat for all attendees: thought leadership panel with Angela Yochem, Grady Booch, Eric Evans, John Zachman, Roger Sessions and William H. Inmon. This was absolutely worth it. John predicted a new role of VP of Change in the future of organizations and William compared architecture with a road to his house. If I was making notes, I would have remembered much more… One interesting thing that I noticed is that in more than couple occasions, Grady was shaking his head in disagreement when John was answering questions. I wish the host would have caught it all those times and let Grady offer his side. John was only present at the first day of the conference, so that was the only chance for him and Grady to exchange opinions.

Looking forward for the day two now!

Reboot

Thu, 30 Jul 2009 00:00:00 -0700

It’s my first inspirational 2 minutes speech for Seymore Speakers club of Toastmasters International. I was extremely nervous earlier today and my voice trembled a bit during the presentation. Anyway, here it goes:

Reboot. It’s a very simple and short procedure that could yield very long lasting benefits. You reboot your computer when its performance becomes sluggish. It sometimes reboots itself when you try to stress it beyond its capacity. After a clean start, it is ready to take your orders again, and it generally performs faster and feels snappier, at least for some time. Even cell phones have become so much more sophisticated nowadays that they also demand an occasional reboot from their picky users.

As advanced as the modern gadgets are, they are still like awkward, wooden toys compared to a complex nature of human body and mind. We granted reboot capability to these toys but we often fail to apply it to ourselves. It’s time we learn back from our own creations.

We live in a fast paced world. We try to multitask but fall short because only few people can truly multitask. The rest of us simply switch-tasking. Still, we want to stay competitive and we try even harder. We want to be nice to people who surround us and we make genuine promises that we find it harder and harder to keep. We get caught in a vortex of resolutions , deadlines, promises, goals and numerous to-do lists. We often burn ourselves, we get stressed and
sluggish. It is time to reboot.

Meet with a person who you made the most promises to but weren’t able to keep them. Apologize and ask to reboot your relationship. Make one promise and keep it.

Throw away all your to-do lists, give yourself a week or few just to enjoy life without any guilt or stress. Compose then a new list with some goals that are really important to you and pursue them with a rekindled eagerness and enthusiasm.

Sooner or later we may find ourselves exactly where we started: tired and frustrated, but we will know what to do. Reboot, and start all over again.

97 Things Every Software Architect Should Know

Sat, 11 Jul 2009 00:00:00 -0700

This is a Twitter of a book: 97 short but precise and relevant advices for software architects. Many advices may not address all of your concerns on particular topics and leave you only half-satisfied. Please don’t fall into this trap! This book is not meant to be the main architecture book in your bookshelf with answers on all the questions. For me personally, reading “97 Things” was like looking into a mirror. I recollected doing some things right and other things wrong in the past; I saw myself making a good progress in one areas and luck of sufficient advancements in another ones. The book helped me to do a quick check of my skills and classify my experiences that I almost forgot. What more, it added more oil to the fire of wanting to keep learning and practicing software architecture.

Overall, I would definitely recommend the book but only as a desert to a main course of classical books on software architecture. I would also like to provide some suggestions for improvements for the format.

Advices from giants of software architecture are noticeably missing. If you had to collect 97 pearls of wisdom, wouldn’t you ask what Grady Booch, Philippe Kruchten and John A. Zachman, for example, have to say?
Many advices convey the same point using a slightly different language. Because all the advices are not grouped in any way, the repetition doesn’t seem to accentuate the point but rather appears as a filler. Offending topics are
- Architect is a coder as well
- Data is a king
- Don’t over-engineer
- Don’t leave your soft skills undeveloped
- Architect is a leader

Communications of the ACM 02/09 vol. 52 no. 02

Sun, 22 Mar 2009 00:00:00 -0700

Photography’s Bright Future

“There is tremendous enthusiasm for computational photography,” says Shree Nayar, a professor of computer science at Columbia University. “The game becomes interesting when you think about optics and computation within the same framework, designing optics for the computations and designing computations to support a certain type of optics.”

One of the most visually striking examples of computational photography is high dynamic range (HDR) imaging, a technique that involves using photoediting software to stitch together multiple images taken at different exposures.

“We’d like to do the same thing with cameras,” Levoy says. “We’d like to allow a researcher to program a camera in a high-level language, like C++ or Python. I think we should have something in a year or two.”

This is pretty interesting. Cameras are following the footsteps of mobile phones. Soon, taken YachtWorld.com as an example, we will be writing Python scripts to automatically upload photos of yachts our brokers are taking. Those photos will be automatically cropped inside the camera itself and then uploaded to our Inventory Management Tool providing right credentials. Future does look interesting.

The Extent of Globalization of Software Innovation

Has innovative activity in software become global? The question attracts interest because software has become a global business. For example, exports of business services and computer and information services grew at an average annual rate
of 27% in India (1995–2003) and at a rate of 46% in Ireland (1995–2004), with similar rapid growth in Brazil, China, and Israel. Yet, the question remains open. While software production takes place in many countries, innovative software is not created everywhere.

In many cases innovation cannot be defined. For that matter, sometimes software cannot be defined.

One measure of innovation appears in patent data. To be sure, not all inventions are patented and not all patents are important. … In 2007, 12,692 U.S. software patents were issued to inventors in the U.S.—a larger number of patents than all other areas of the world combined (6,397).

A key determinant of the location of development activities in software is the location of the user. This is particularly true with business software, which is often bundled with a set of business rules and assumptions about business processes.

We see the same pattern today. Conditions for development of innovative new software products (and software firms) are propitious when they occur in proximity to potential lead users. A good example is Israel’s longstanding strength in security software, which arose in part due to the advanced needs of the Israeli defense forces.

… The firms we interviewed told us that supplies of fresh engineering graduates were plentiful in India, but it was much more difficult to find developers with project management experience and more difficult still to find those with critical business knowledge.

What will the future bring? First, it does appear software product development and testing activities have become increasingly global. …

Many of these entry-level jobs are now going overseas. A declining demand today for entry-level programming jobs in the U.S. and increasing demand elsewhere could make it more difficult for U.S. workers—and relatively easier for those from other countries—to perform complex software design activity in the future. The implications of this would be a more globally dispersed pattern of innovation in software than we see today.

This makes a lot of sense. If I take only a technical knowledge from my 10 year IT experience, it wouldn’t be valued that much on its own. What gives it an extra value is a context of business problems that I had to understand first before coming up with a technical solution. I’ve been fortunate to participate in many sessions with subject matter experts and acquire knowledge from many different areas: HR, telecommunications, billing, sales, advertising and management. Having technical skills and understanding of a business is exactly what gives me, and many others like me, an advantage in coming up with innovative solutions. But it all started with me, as a junior programmer listening to “mambo-jumbo” of our director of HR. If today’s junior developers cannot get jobs because those jobs are mostly technical and it’s cheaper to outsource them, then developers from other countries will slowly start building up business knowledge and centers of innovations will be indeed dispersed. The lesson here is to be more then just a programmer to be in demand – you have to have people, organizational and leadership skills, and business knowledge from couple of areas.

Parallel Programming with Transactional Memory

Good article that goes through theory, problems, potential solution and current state of Transactional Memory. It was interesting to see that databases were used to illustrate how transactions help with the problem of consistency. Everything old is new again. I already saw couple of projects using Transactional memory approach popping up on the horizon. I think I will wait a bit longer before given them a try:

TM promises to make parallel programming much easier. The concept of transaction is already present in many programs (from business programs to dynamic Web applications) and it proved to be reasonably easy to grasp for programmers. We can see fi rst implementations coming out now, but all are far from ready for prime time. Much research remains to be done.

Improving Performance on the Internet

The article is written by co-founder of Akamai Technologies who serves as chief scientist there. The article introduces the concept of “middle mile”, highlights the challenges it presents and offers a look at the approaches to overcome these challenges.

When it comes to achieving performance, reliability, and scalability for commercial-grade Web applications, where is the biggest bottleneck? In many cases today, we see that the limiting bottleneck is the middle mile, or the time data spends traveling back and forth across the Internet, between origin server and end user.

On the other hand, the Internet’s middle mile—made up of the peering and transit points where networks trade traffic—is literally a no man’s land. Here, economically, there is very little incentive to build out capacity. If anything, networks want to minimize
traffic coming into their networks that they don’t get paid for. As a result, peering points are often overburdened, causing packet loss and service degradation.

It turns out that because of the way the underlying network protocols work, latency and throughput are directly coupled. TCP, for example, allows only small amounts of data to be sent at a time (that is, the TCP window) before having to pause and wait for acknowledgments from the receiving end. This means that throughput is effectively throttled by network round-trip time (latency), which can become the bottleneck for file download speeds and video viewing quality.

Four Approaches to Content Delivery

Centralized Hosting. Traditionally architected Web sites use one or a small number of collocation sites to host content.

“Big Data Center” CDNs. Content delivery networks offer improved scalability by offloading the delivery of cacheable content from the origin server onto a larger, shared network. It may seem counterintuitive that having a presence in a couple dozen major
backbones isn’t enough to achieve commercial-grade performance. In fact, even the largest of those networks controls very little end-user access traffic. For example, the top 30 networks combined deliver only 50% of end-user traffic, and it drops off quickly from there, with a very long tail distribution over the Internet’s 13,000 networks. Even with connectivity to all the biggest
backbones, data must travel through the morass of the middle mile to reach most of the Internet’s 1.4 billion users.

Highly Distributed CDNs. Another approach to content delivery is to leverage a very highly distributed network—one with servers in thousands of networks, rather than dozens. On the surface, this architecture may appear quite similar to the “big data center” CDN. In reality, however, it is a fundamentally different approach to content-server placement, with a difference of two orders of magnitude in the degree of distribution.

Peer-to-Peer Networks. P2P can be thought of as taking the distributed architecture to its logical extreme, theoretically providing nearly infi nite scalability. Moreover, P2P offers attractive economics under current network pricing structures. In reality, however, P2P faces some serious limitations, most notably because the total download capacity of a P2P network is throttled by its total uplink capacity.

… but many optimizations are made possible by using a highly distributed infrastructure:

Optimization 1: Reduce transportlayer overhead.
Optimization 2: Find better routes.
Optimization 3: Prefetch embedded content.
Optimization 4: Assemble pages at the edge.
Optimization 5: Use compression and delta encoding.
Optimization 6: Offload computations to the edge.

Designing a scalable system that works under these conditions means embracing the failures as natural and expected events. The network should continue to work seamlessly despite these occurrences. We have identified some practical design principles that result from this philosophy, which we share here:

Principle 1: Ensure significant redundancy in all systems to facilitate failover.
Principle 2: Use software logic to provide message reliability.
Principle 3: Use distributed control for coordination.
Principle 4: Fail cleanly and restart.
Principle 5: Phase software releases.
Principle 6: Notice and proactively quarantine faults.

IEEE Computer January 2009

Mon, 02 Feb 2009 00:00:00 -0800

Is Cloud Computing Really Ready for Prime Time?

Proponents tout the technology’s advantages, including cost savings, high availability, and easy scalability.

… IT departments are still wary of it because they don’t control the cloud-computing platform.

… So far, venture capitalists have not invested a lot of money in cloud-computing providers.

… key risks include reliability, security, the additional cost of the necessary network bandwidth, and getting locked into specific cloud-computing vendor.

Types of cloud services
Services. Some products offer internet-based services – such as storage, middleware, collaboration, and database capabilities – directly to users.
IaaS. Infrastructure-as-a-service products deliver a full computer infrastructure via the Internet.
PaaS. Platform-as-a-service products offer a full a partial application development envirnment that users can access and utilize online, even in collaboration with others.
SaaS. Software-as-a-service products provide a complete, turnkey applications – including complex programs such as those for CRM or enterprise-resource management via the Internet.

A recent survey of CIOs and IT executives by IDC rated security as their main cloud-computing concern. Almost 75 percent of respondents said they were worried about security.

… data stored in the cloud night be used anywhere in the world and thus might be subject to state or national data-storage laws related to privacy or record keeping.

“This has caused Amazon and other companies to develop offerings using storage facilities located in the EU.”"

Companies cannot pass audits of their capabilities by prospective clients if they can’t demonstrate who has access to their data and how they keep unauthorized personnel from retrieving information.

Cloud-computing vendors are addressing this concern by having third parties audit their systems in advance and by documenting procedures designed to address customers’ data-security needs.

Cloud computing hasn’t always provided round-the-clock reliability.

For example, Salesforce.com left customers without service for six hours on 12 February 2008.

And Amazon’s S3 and EC2 services suffered a three-hour outage three days later.

The changing paradigm of data-intensive computing

I get easily mesmerized by big scales, and the cover feature of the issue delivered on this premise. Words like Tbyte, Pbyte and “1000-fold increase” are common occurrences throughout this issue and this article in particular.

I’m also fortunate to work at YachtWorld.com. This is not the first time when the main theme of the current issue of a magazine strikes an accord with the progress and challenges we experience at YachtWorld. No, we don’t process Pbytes of raw data (yet) but we recently hit a bar with Tbyte of raw data to be crunched and data-intensive computing is now a part of our world as well.

How can a mid-size company can be constantly on the cutting edge of so many facets of IT? I think it’s because we’re a niche company. Our niche is a marine industry and we develop and support software solutions for nearly all facets of that industry: portals, SEO, web services, CMS, data warehousing, data exchange, inventory managements, ad management, e-commerce, etc.

The continued exponential growth of computational power, data-generation sources, and communication technologies is giving rise to a new era in information processing: data-intensive computing.

… Ability to tame a tidal wave of information will distinguish the most successful scientific, commercial, and national-security endeavors.

… found that "datasets being produced by experiments and simulations are rapidly outstripping our ability to explore and understand them…

The North American electric power grid operations generate 15 terabytes of raw data per year, and estimates for analytic results from control, market, maintenance, and business operations exceed 45 Tbytes/day. As developers add new high-resolution sensors to the grid, this data volume is increasing rapidly while the time available to make control decision remains constant.

I think the highlighted portion of the quote above is the universal challenge for all data-intensive computing.

High-energy physics remains a leading generator of raw data. For example, the Atlas experiment for the Large Hadron Collider … will generate raw data at a rate of 2 Pbytes per second beginning in 2008 and store about 10 Pbytes per year of processed data.

This is simply unbelievable! I fish these guys did a presentation on how they handle data processing at such an enormous rate.

In contrast to compute-intensive tasks where available processing power is the rate-limiting factor, data-intensive computing could be qualitatively defined as “any computational task where data availability is the rate-limiting factor to producing time-critical solutions.”

Data-intensive computing styles

Data-processing pipelines

Data warehouses. … Wal-Mart’s, has grown over a decade to store more than a petabyte, fueled by daily data from 800 million transactions generated by its 30 million customers.

Data centers

Underlying the MapReduce programming model is the Google File System, along with its open source counterpart, the Hadoop Distributed File System. While these systems have much in common with traditional distributed file systems, they differ in that they are built based on the assumption that terabyte datasets will be distributed across thousands of disks attached to commodity compute nodes. In such environments, hardware failure occurs regularly. Hence, data redundancy, fault detection, and computation recovery are core facilities that the file system provides transparently to applications.

The speed of light ultimately limits latency; this problem is at the heart of many data-intensive analyses.

The 24-hour knowledge factory: can it replace the graveyard shift?

Discovery of the issue! I’m already planning to talk to my boss about feasibility of this approach.

Collaborating centers in time zones six to eight hours apart can transfer work so that every center is working during the daytime. Although this concept avoids the hazards of night work, it requires careful planning and a way to automatically capture evolving knowledge.

The key issues are how to hand over work in progress, decompose tasks into relatively autonomous units, and manage operations effectively in such a decentralized environments.

… Work transfer is really like work transformation, since each worker adds and reshapes knowledge before passing it on. Transformed tasks, in turn, lead to redefined job descriptions, which require expertise to break down and define.

One obvious advantage was less time for tasks resolution. The gradual one-to-one pairing between onshore and offshore members made is easier to transfer tasks and knowledge. In one case, the US and Indian developers working in an interlaced mode (US sending the code for testing in the evening and getting programmer feedback from India in the morning) resolved a bug in half the time they would have taken in a conventional structure.

A by-product of constant communication is superior documentation and knowledge dissemination.

Geographic distance also means considerable handoff overhead. Work is transitioned every eight hours, and time and effort are involved in each handoff activity. The factory paradigm also involves considerable upfront effort in task decomposition. Someone must decide how to break down tasks in the various disciplines.

Management and quality control become more burdensome.

Professional and ethical dilemmas in software engineering

The authors identify, categorize, and name nine specific ethical and professional dilemmas in software engineering, placing them in the context of the IEEE code of conduct…

Mission imposible
Mea culpa
Rush job
Not my problem
Red lies
Fictionware versus vaporware
Nondiligence
Canceled vacation
Sweep it under the rug

Web 3.0 emerging

With Web 3.0, on the other hand, the explosion of data on the Web has emerged as a new problem space, and the new game-changing applications of this next generation of technology have yet to be developed.

In one article I read about an year ago, it was suggested that Web 3.0 is going to be a contest between 3 enablers: semantic web, rich multimedia and personalization/suggestion technologies. Please don’t quote me on this but that’s what I remember. This article declares a winner and it’s Semantic Web (RDFS, OWL) and Linked Data (RDF, SPARQL). The previous article also outlined a challenge for semantic web adoption which was an unclear business model for profitability. I have to agree here. Let’s take YachtWorld.com where we could expose boat information in a more semantic way. The first thing this would enable is for our competitors to scrape our data more easily than ever and present this data as their own. The current article is rather silent about this point but provides lots of success stories for non-commercial application of the technology. I’m still very interested about a potential business model for semantic web applications but so far I failed to see it.

IEEE Computer December 2008

Mon, 26 Jan 2009 00:00:00 -0800

Third-Generation RFID Is on It’s Way

Early tags were passive, using signals from RFID readers to power up. Newer technologies use active tags, which have their own battery power. The added power increases their communication range.

The third generation of RFID – called wireless-sensor-network (WSN) technology – works with active tags. They gather information and then send it from tag to tag over a mesh network until it reaches the location at which the data will be processed.

Communications of the ACM 01/09 vol. 52 no. 01

Mon, 26 Jan 2009 00:00:00 -0800

Students flock to game design

As governments and universities worldwide develop strategies to reverse a dizzying downturn in computer science students, a hot field of study is getting even hotter and helping to rekindle interest in computer science. Game design has become a popular new major at more than 200 schools across the US… Because game creation crosses several disciplines, the diversity of programs that offer such courses is stunning: Fine arts colleges, engineering schools, film schools, music schools, and even drama programs… The jobs vary from inventing characters, to writing dialogs, composing music, creating digital scenes, and writing software that rules the fantasy worlds.

Calculating the future

The article is about the scope and challenges of computer power in climate research. It’s amazing how many input variables researchers have to account for considering all the possible permutations and a complex effect of one variable on others. Multiply this on the number of 3D points the calculations have to be applied on and you get a clear picture of how process intensive those calculations are. The scope and complexity also results in a trade-off:

Is it better to compute existing models in finer details, or to make the models bigger by adding more scientific content?

If the limiting factor in running a climate model on a multiprocessor system is inefficient communication of information within the program, then the amount of processing power dedicated to solving equations falls and the model fails to take advantage of the raw processing power available. The programming model we use now is not viable anymore in the next couple of generations of computers…

As in many other areas, the software is slow to take advantage of hardware evolution.

The universe in Your Computer

The article describes Microsoft’s WorldWide Telescope (WWT) and Google sky. As a former teenager astronomer-enthusiast I found it fascinating that future generations of kids won’t even have to have a telescope to study sky. All they would need is a computer and the Internet connection. I don’t think, though, telescopes would ever go out of fashion. There is something magical in exploring the sky with your own eyes and catching a glimpse of a fallen star.

Interesting excerpt in regards to bringing new forms of education into classrooms:

Christin believes the key to Google Sky’s success will be innovative educators. “I’m coming across the problem that the kids are way ahead of teachers,” he says. “We need to get teachers… to understand that the student of the future and the worker of the future needs to be facile with finding information, analyzing it, applying critical thinking, making decisions, and finding the data they need to answer a question.”

Payment Via Wallet Phone

There is also the major obstacle of working out new business models as the lines blur between banks, financial institutions, and cellphone companies.

Get Smart

The article is about context aware capabilities of future generations of smartphones.

The problem with today’s smartphones is that there’s nothing terribly smart about them. Even the most souped-up devices amount to little more than stripped-down PCs, with tiny screens and maddening keyboards.

The next generation of context-aware smartphones will take advantage of the growing availability of built-in physical sensors and better data exchange capabilities to support new applications that not only keep track of your personal data, but can also track your behavior and – this is where truly smart part will finally come into play – anticipate your intentions.

Researches at PARC, based in Palo Alto, CA, proposed four levels of system awareness: basic context awareness (including location, time, and other details of physical environment); behavior awareness (typing, walking, standing, or clicking a button); activity awareness (shopping, dining, or traveling); and intent awareness (predicting the future).

… To bring contextual computing to the masses, phone manufacturers and software vendors will have to overcome some major hurdles. Not the least of these is the lack of open standards for exchanging context data between applications.

This seems to be the rule of the game. When any software or hardware technology advances high enough to create a niche market, one of the biggest hurdles to bring the technology to the masses is the lack of open standards.

Law and Technology. The End of the Generative Internet

Jonathan Zittrain, the author of The Future of the Internet—And How to Stop It, shares his thoughts on platforms market and its future implications. This is a pretty interesting analysis of likes of Apple and Facebook.

The book’d thesis is that the mainstream computing environment we’ve experienced for the past 30-plus years … is an anomaly… The basic building blocks of modern IT are PCs that anyone can reprogram, connected to an Internet that unquestioningly routes bits between two arbitrary points. This has led to a generative revolution where novel and disruptive technologies have come from obscure backwaters – and conquered. While incumbents bet on gated-community networks like CompuServe, Prodigy, and AOL, or maker of “smart appliances” such as dedicated word processors and video-game consoles, dark-hourse candidates like the Internet and the PC emerged unexpectedly and triumphed, helped along by commercial forces that belatedly hpped on their bandwagons.

So why anomalous?

Interesting thesis and even more interesting analysis.

Imagine if Windows had adjusted Windows to act the way the iPhone and Facebook aoos platforms do.

The iPhone apps model is powerful, and it is serving some useful purpose in shielding people, prospectively and retroactively against bad code.

… but not against bad apps. Couldn’t refuse to add. Anyway this is a serious article with many provocative points:

… We likely wouldn’t accept this situation in PC architecture, and yet it is commonplace in the ecosystem that will soon replace it. … but even if one vendor doesn’t capture the mobile phone or social networking spaces, the choices among them are shaping up to be choices among gates communities: equivalent of old AOL vs. Prodigy vs. CompuServe… This is one reason why Google’s Android project is so fascinating: and attempt to bring the generativity of the PC to the mobile phone space. Without a security model better than the PC’s security provisions, however, Android is a tough proposition. How long will users tolerate a phone for which clicking on the wrong link can disable it?

Good question especially if you take into consideration the previous article from the same issue: “Payment Via Wallet Phone”. If you phone is more then just a phone then will you be comfortable without a centralized body of control?

The way forward – for both PCs and smartphones – lies in a new security architecture that lets users make better-informed decisions about whether to run new software… There are many details to work out, but without ways of managing our generative platforms without a central gatekeeper, chances seem strong that most people will accept – even demand – outside control.

From the Front Lines. Ground Control to Architect Tom…

Humorous article with an awesome summary:

The ability to create an illusion of knowledge among people who don’t know any better is really a trait of a shrewd salesperson, not a software architect.

I’m heavily involved in hiring processes of technical talent at Boats.com/YachtWorld.com and I can attest that there are few shrewd and really slick people who are pretty good at creating illusion of knowledge.

The Business of Software. The Ontology of Paper.

Interesting article about a diminishing role of paper in designing complex systems. The author still pays respect to paper and gives examples what it’s good for. One aspect of the article that drew my attention is a reference to steam engines. Now, as a software engineer, I know exactly when to start thinking about a new career.

It’s a common cliche that we are currently going through an “Information Revolution” that may be more profound in its consequences than the development of steam engines was in the Industrial Revolution. This is undoubtedly true, but it is missing one important point: the Industrial Revolution did not occur when we built steam engines, it occurred when we used steam engines to build steam engines.

The true information and computing revolution will not occur until we use software to build software; until we really use executable knowledge to create executable knowledge.

Eventually Consistent

If you’re looking at this issue in a store then this article is enough to swing your vote and get the magazine. I follow religiously the blog of Werner Vogels where he published the original post served as a basis for this article. That blog post was the most interesting in 2007 in my books. When I first read it I felt enlightened because so much new knowledge opened before my eyes, and I also felt insignificant and dwarfed before challenges some people were facing. I think every IT professional should read this article.

The Long Road To 64 Bits

Discovery of the issue. I ventured to read this article thinking that I would drop it soon after because it would cater primary to hardware folks. I was wrong. I finished it in one gulp and found it extremely interesting for software people as well. The article masterfully tells a story of a painful progress to 64 bits and shares invaluable lessons:

Standards often get created in nonstandard ways, and often, de facto standards long precede oficial ones.

Reasonable people can disagree, especially when looking at different sets of data.

Sometimes one must work with competitors to make anything reasonable happen.

Programmers take advantage of extra bits or ambiquity of specification…

Code can be recompiled, but once data gets written somewhere, any new code must still be able to describe it cleanly. Current software is rarely done from scratch but has to exist inside a large ecosystem.

Hardware decisions last a long time, but software decisions may well last longer. If you’re a practicing programmer, take pity on those who end up maintaining your code, and spend sometime thinking ahead. Allow for natural expansion of hardware, hide low-level details, and use the highest-level language you can. C’s ability to make efficient use of hardware can be both a blessing and a curse.

Computational Challenges in E-Commerce

In this article, we prove an overview of four areas of computation in which incentives play a crucial role: resource allocation, knowledge integration, peer production and interaction, and the security and privacy.

Allocating scarce resources – from bread to bytes – is a fundamental process that permeates economics and, indeed, society…

Advertising is a business based on allocating attention, one of the scarcest and most valuable of resources…

Voyagers and Voyeurs: Supporting Asynchronous Collaborative Visualization

This article describes mechanisms for asynchronous collaboration in the context of information visualization, recasting visualization as not just analytic tool, but social spaces. We contribute the design and implementation of sense.us…

My sentiment from this issue

Mobile computing is rivaling PCs.