Recently, I was working with a fairly progressive client that was spending a lot of time working its internal architecture into an SOA solution. I was brought on in January as a consultant to build a greenfield application of modest complexity, but probably more complex than the apps they'd built internally to date. They placed several strict requirements on certain technological aspects of development, which I could easily handle. However, they did declare one that gave me some fits.
A few months before I arrived, their internal staff had started to develop a web service to centralize contact information for the entire agency. During the initial phases of my project, I was told that my app was to integrate with this new contact web service when in comes online in March. (Those of you with inter-team experience already know where I'm going.) March came and went; no contact web service. I kept hearing rumors of re-work from members of the team, which made me very nervous. My project manager (also a consultant) and I began to think of contingencies as this was probably going to bite us and delay our release. We were then told May; the web service would be up and running by May. May came and went. Our thoughts eventually turned entirely to "It's never going to be delivered. We're going to have to implement something ourselves to make our release date."
Our app was due to make it's initial production release the first of July. We had to have that contact piece to go live. But we had nothing. At the end of May, we decided to write our own and merge it with theirs later. I wrote our solution for contacts the following week and we moved into production the first week of July. Whew!
Sometime that month, they finally pushed their contact web service to production and I was able to begin integration with my app on July 30th. They were using an ASMX web service, so I figured that it'd be easy enough to create a proxy and adapt it to our solution. I silently thought to myself, "I bet I can have this integrated in just a couple days."
I was in for shock though. When I dug into the WSDL, I was both disgusted and astonished. The schema was flat. They were using SOAP to usher strings (and only strings) back and forth to the web service. The schema had no depth at all anywhere - everything was a string, a string that had to be parsed manually. If an operation returned person information, it returned a string which contained the information in XML form, but still a string that'd have to be parsed or accessed via XPath. There other issues that come about by having such a poorly defined schema. For example, if everything is a string, I can send anything I want to the service including the current version of Webster's dictionary or the New York City phone book. And oh yeah! I was delayed for this? It took them months and months to build this? Less than twenty-five operations, no schema... What were they doing all that time?
It's as if they intended to build a house, took as long as building a sky scraper and wound up with a tool shed. They decided to leverage a very powerful tool, an air nailer, but used it as a hammer. When they went to drive a nail, they turned the air nailer sideways and drove it with the side of the air nailer. In this way, not only did they negate the benefits of the air nailer, but they also endured a loss of productivity.
I had no doubt; this was going to be painful. To add some extra salt to the wound, some features my application needed were simply not implemented. So, on top of having to break it to my users that some of their favorite contact functions were going away, now I have to write custom parsing logic to extract information from the operations, rather than let .NET's out-of-the-box SOAP digestion handle if for me. Which is going to delay the whole process by days. I cringe when I think of parsing XML myself for any reason; especially when so many good people worked hard to write frameworks for SOAP and various other specifications so I wouldn't have to.
Thus, I plead to the masses, "Embrace Your Technology". It seems more and more commonplace that my clients not only fail to understand the technologies they choose, but choose technologies they don't understand. From ASP.NET to Oracle to Visual Studio to web services, they don't understand what these choices provide and end up misusing them negating the benefit. They stand up and claim "We're using SOAP web services!", but have no idea what that the specification does for them. They miss the benefits a specification provides by allowing them to leverage pre-established frameworks to forgo the normal rigmarole that other web service specifications impose on them, allowing them the ability to move on with more important development without writing custom XML parsing and operation security for invalid data.
I will contend that too many folks are content to acquire a technology without acquiring a book on the subject. Heaven forbid someone even try reading the book before acquiring the technology. This is so unfortunate as there are typically some really great minds on these technologies that share their expertise through writing books and whitepapers.
I'll take it a step further and sideways; I will contend that too many folks that barely need a firecracker acquire a nuclear bomb. Some acquire Oracle when a text file will suffice for the moment; even when an RDBMS is necessary, MySQL will do the trick before you bring in the heavy hitter. Remember, your real need is persistence of data, not the acquisition of Oracle. A good friend watched his startup fail because his colleagues spent their capital on Oracle rather than a prototype of their idea - a simple case of confusing needs.
So to all you hard-working folk out there, embrace the technology you choose and choose the technology you need. Perhaps, you'll deliver a functioning house on time.
Showing posts with label SOA. Show all posts
Showing posts with label SOA. Show all posts
Tuesday, August 4, 2009
Friday, January 18, 2008
Dataundermining
Suppose Bob is your investment agent. Bob uses many tools and algorithms to determine the best investment for your money; he always seems to make good decisions and you profit from his skill and judgment. Bob's investing ability is surpassed only by his technological ineptitude. He knows nothing of computers, yet remains meticulous with his record-keeping. After every day of investing, he places a paper log of his investment activities in the second drawer of his desk.
Suppose further that Sue is your accountant. She, too, is great at what she does, as long as she is provided with complete and accurate information. The trouble resides with you and your inability to provide her with the information she requires on the part of your investments (since Bob takes care of that for you). Luckily, Bob is a trusting soul, and he has provided you with a key to his office in case of emergencies. You know about the log in his desk drawer and realize how much easier your life would be if Sue could obtain your investment activity information straight from the source. In a shady move, you give a copy of the key to Sue and inform her about the information in this log. Periodically, Sue lets herself into Bob's office and searches the log for investment activity he's done on your part. She uses this information to keep accurate tabs on your investments without having to badger you.
A while later, Bob completes a computer course at the local college. He now recognizes the productivity benefits of electronic record keeping and transfers the records from the log to his PC. Bob realizes that maintaining a physical copy of his records has no merit and only takes time away from doing his job; he shreds the log, and against his nutritionist's better judgment, fills the desk drawer with soda and snacks.
Later that evening, Sue comes to inspect the log. She's alarmed to find the log has been replaced with soda, candy bars and other goodies. You get a phone call from an upset accountant, demanding that you have Bob continue to put the daily log in his desk drawer, "... or we're back where we started", she says.
This is an example of dataundermining; Bob and his record-keeping system have been undermined, and by my count, you have roughly four options:
Option 1 requires Bob to either abandon his path of evolution or do double the work by keeping both his electronic records and his physical records up-to-date. Neither suit your needs from an investment point of view. You want Bob to be able to grow and evolve with the rest of the world; after all, he's making your bank. You also want Bob to spend the maximum amount of time and resources furthering your investment success; maintaining two forms of records detracts from his ability to do that.
Option 2 requires extra effort on your part. Your keeping Sue updated on your investment activities is just not in the cards. The speed and accuracy with which you could deliver information to Sue severely handicaps her ability to keep your financial records straight. After all, let's face it, you're no expert on your investments, that's why you have Bob.
Option 3 is what I call classic. You recognize that you have a broken system, but too much effort is involved in keeping everybody happy, so you just sit on your hands and accept the fact that one piece of the equation is just not going to be broken.
Clearly, I saved the appropriate option for last.
To me, option 4 is the option that should have been chosen in the first place. When Bob and Sue were hired, they should have been informed of the need for your financial information to be shared and communicated to other entities (including each other) upon request by one of those entities. In short, they need to open themselves to integration.
The preceding is an analogy for a problem with which I've wrestled countless times. A shop creates an application to solve a particular business problem. This application requires a database for persisting data relating to the problem domain, and consequently one is created along-side the application's development. Sooner or later, this application is deployed, beginning its life in the wild. We'll refer to this application as "Application 1".
Later, another application is developed to solve one of the many other problems the business faces. Nothing is really special about this application, which we will call "Application 2" for lack of a better name, except that in order to fulfill its use-cases efficiently, it requires information created and managed by Application 1. No problem, we'll just hook in to Application 1's database and retrieve some of this data. Herein lies our problem. We've just undermined Application 1 in the same way that Sue undermined Bob in our analogy. We don't yet notice any adverse effects, and Application 2 is completed and released to the wild.
Sometime later, the business has acquired a list of new features and use-cases for Application 1, along with some obsolete use-cases that can be removed. After inspection of the requirements, a design for the new version of Application 1 has been formulated. This design calls for a few additions to existing structures and a few changes in how existing data is structured, greatly simplifying the application development work.
Suppose the development team quickly realizes that Application 2 depends on the data structures of Application 1's database. Dataundermining at its finest. Updating Application 1 is not going to be as easy as the design suggested, and they have a very tough choice to make. Do they update Application 2 as a part of the Application 1 upgrade? Sounds like scope-creep to me, and the business may not have budgeted for multiple applications to come under the knife. Do they re-think their approach and make updates to Application 1's database and use-cases to "work-around" the dependency that's been created? This would keep the data structures that are depended upon by Application 2 intact, but could potentially create other illogical data structures and increase the overhead & complexity for the interaction of Application 1 and its database. It would also make further evolution of Application 1 more difficult. I hope its clear to see this scenario resembles option 1 of our analogy.
Suppose again the development team realizes the dependency situation on Application 1's database. Another solution consists of an attempt to remove the integration piece from Application 2 and for the user to consult Application 1 prior to use, but this brings about a greater potential for error and a loss of productivity. In this case, we're closely matching option 2 of our analogy.
In a different light, suppose the development team is oblivious to processes outside of Application 1. Development ensues, the structures are modified, the application update is complete and Application 1 v2.0 is released. Now, the effects of dataundermining are manifest after the fact. Application 2 crashes. What happened? It's rather simple; Application 2 is looking in Application 1's database for a data structure shaped like v1.0 and got a data structure shaped like v2.0. It follows naturally that issues should arise from a scenario like this, a scenario very closely mirroring option 3 from our analogy.
The illustration above exemplifies how dataundermining becomes a problem. Application 2 has accessed Application 1's database to fulfill its use-cases, inhibiting Application 1's ability to evolve. The most robust solution uses techniques of SOA to allow each application to communicate without acquiring illegitimate access to the other's data store. Whether that be through text-based communication, like Web Services, or binary-based communication like CORBA, DCOM or RMI, the best scenario for all involved requires letting each application be solely responsible for its own data, and building into each app the communication abilities for access to its data.
Suppose further that Sue is your accountant. She, too, is great at what she does, as long as she is provided with complete and accurate information. The trouble resides with you and your inability to provide her with the information she requires on the part of your investments (since Bob takes care of that for you). Luckily, Bob is a trusting soul, and he has provided you with a key to his office in case of emergencies. You know about the log in his desk drawer and realize how much easier your life would be if Sue could obtain your investment activity information straight from the source. In a shady move, you give a copy of the key to Sue and inform her about the information in this log. Periodically, Sue lets herself into Bob's office and searches the log for investment activity he's done on your part. She uses this information to keep accurate tabs on your investments without having to badger you.
A while later, Bob completes a computer course at the local college. He now recognizes the productivity benefits of electronic record keeping and transfers the records from the log to his PC. Bob realizes that maintaining a physical copy of his records has no merit and only takes time away from doing his job; he shreds the log, and against his nutritionist's better judgment, fills the desk drawer with soda and snacks.
Later that evening, Sue comes to inspect the log. She's alarmed to find the log has been replaced with soda, candy bars and other goodies. You get a phone call from an upset accountant, demanding that you have Bob continue to put the daily log in his desk drawer, "... or we're back where we started", she says.
This is an example of dataundermining; Bob and his record-keeping system have been undermined, and by my count, you have roughly four options:
- Demand Bob continue with his paper logging practices so as not to keep Sue from efficiently keeping track of your investment activity. (This would also lead to a confession about your giving Sue the proverbial key to the kingdom and her sneaking data out from under Bob.)
- Revert back to your having to keep Sue manually apprised of Bob's investment activities on your behalf.
- Do nothing, letting Bob continue on oblivious to the situation and forcing Sue to attempt her job with incomplete information.
- Force Bob and Sue to communicate with each other.
Option 1 requires Bob to either abandon his path of evolution or do double the work by keeping both his electronic records and his physical records up-to-date. Neither suit your needs from an investment point of view. You want Bob to be able to grow and evolve with the rest of the world; after all, he's making your bank. You also want Bob to spend the maximum amount of time and resources furthering your investment success; maintaining two forms of records detracts from his ability to do that.
Option 2 requires extra effort on your part. Your keeping Sue updated on your investment activities is just not in the cards. The speed and accuracy with which you could deliver information to Sue severely handicaps her ability to keep your financial records straight. After all, let's face it, you're no expert on your investments, that's why you have Bob.
Option 3 is what I call classic. You recognize that you have a broken system, but too much effort is involved in keeping everybody happy, so you just sit on your hands and accept the fact that one piece of the equation is just not going to be broken.
Clearly, I saved the appropriate option for last.
To me, option 4 is the option that should have been chosen in the first place. When Bob and Sue were hired, they should have been informed of the need for your financial information to be shared and communicated to other entities (including each other) upon request by one of those entities. In short, they need to open themselves to integration.
The preceding is an analogy for a problem with which I've wrestled countless times. A shop creates an application to solve a particular business problem. This application requires a database for persisting data relating to the problem domain, and consequently one is created along-side the application's development. Sooner or later, this application is deployed, beginning its life in the wild. We'll refer to this application as "Application 1".
Later, another application is developed to solve one of the many other problems the business faces. Nothing is really special about this application, which we will call "Application 2" for lack of a better name, except that in order to fulfill its use-cases efficiently, it requires information created and managed by Application 1. No problem, we'll just hook in to Application 1's database and retrieve some of this data. Herein lies our problem. We've just undermined Application 1 in the same way that Sue undermined Bob in our analogy. We don't yet notice any adverse effects, and Application 2 is completed and released to the wild.
Sometime later, the business has acquired a list of new features and use-cases for Application 1, along with some obsolete use-cases that can be removed. After inspection of the requirements, a design for the new version of Application 1 has been formulated. This design calls for a few additions to existing structures and a few changes in how existing data is structured, greatly simplifying the application development work.
Suppose the development team quickly realizes that Application 2 depends on the data structures of Application 1's database. Dataundermining at its finest. Updating Application 1 is not going to be as easy as the design suggested, and they have a very tough choice to make. Do they update Application 2 as a part of the Application 1 upgrade? Sounds like scope-creep to me, and the business may not have budgeted for multiple applications to come under the knife. Do they re-think their approach and make updates to Application 1's database and use-cases to "work-around" the dependency that's been created? This would keep the data structures that are depended upon by Application 2 intact, but could potentially create other illogical data structures and increase the overhead & complexity for the interaction of Application 1 and its database. It would also make further evolution of Application 1 more difficult. I hope its clear to see this scenario resembles option 1 of our analogy.
Suppose again the development team realizes the dependency situation on Application 1's database. Another solution consists of an attempt to remove the integration piece from Application 2 and for the user to consult Application 1 prior to use, but this brings about a greater potential for error and a loss of productivity. In this case, we're closely matching option 2 of our analogy.
In a different light, suppose the development team is oblivious to processes outside of Application 1. Development ensues, the structures are modified, the application update is complete and Application 1 v2.0 is released. Now, the effects of dataundermining are manifest after the fact. Application 2 crashes. What happened? It's rather simple; Application 2 is looking in Application 1's database for a data structure shaped like v1.0 and got a data structure shaped like v2.0. It follows naturally that issues should arise from a scenario like this, a scenario very closely mirroring option 3 from our analogy.
The illustration above exemplifies how dataundermining becomes a problem. Application 2 has accessed Application 1's database to fulfill its use-cases, inhibiting Application 1's ability to evolve. The most robust solution uses techniques of SOA to allow each application to communicate without acquiring illegitimate access to the other's data store. Whether that be through text-based communication, like Web Services, or binary-based communication like CORBA, DCOM or RMI, the best scenario for all involved requires letting each application be solely responsible for its own data, and building into each app the communication abilities for access to its data.
Subscribe to:
Posts (Atom)