Monday, April 11, 2005

An efficient way to Debug Domino Web Agents

by Tony Austin
Principal,
Asia/Pacific Computer Services


The Intractability of Domino Web Agents
I've been programming since the mid 1960s, and never cease to be surprised how tricky debugging can be on some platforms, or in some environments on a particular platform. One such environment is the debugging of Web agents running on a Domino server. I know this is a sore point for most Domino developers.

Debugging of LotusScript for the Notes Client is pretty easy, since there's a nice debugger available (although the same can't be said for the Formula Language and Java environments, which is a shame.)

You put lots of loving care into developing your slick Web agent, then confidently launch it for the first time, and what do you see? Maybe nothing at all seems to happen. If you're lucky, you get a message or two in the Domino console log, but typically is so obscure as to be of no real use to you or on the surface seems to have no relationship to what your agent was trying to achieve!

I was in this situation a year or two ago, when extending NotesTracker, a database usage reporting toolkit (described at http://notestracker.com or http://asiapac.com.au ) from its originhal Notes-Client-only environment to embrace the Web browser environment.

The NotesTracker LotusScript code was quite intricate, and I had a few challenges to modify parts of it to get it working via WebQueryOpen and WebQuerySave agents. Additionally, this was some years ago, under Notes R4 and R5 so could only use the tools available at that time. The Remote Debugger capability didn't come along until Domino 6 (and even when it did become available, I found ND6 remote debugging too fiddly for my liking, not all that easy to set up and use).

Naturally enough, not wanting to "reinvent the wheel" I did the right thing by carrying out exhaustive research to see how others had gone about debugging Domino Web agents. I searched articles, forums and knowledge bases on the Lotus Web site, as well as quite a range of non-Lotus sites (nearly all of which latter you'll find conveniently listed at http://notestracker.com/Links/NotesDomino.htm ).

Firstly, refer to the following IBM developerWorks articles for some excellent general guidance about debugging Domino agents:

Prior to ND6, typical debugging methods suggested were to Include strategically-placed Print or MessageBox statements to display debugging information, or the more elegant use of the AgentLog class to write out the debug info into a log database.

But I was seeking a more direct method. So here's what I came up with. (I generally use it insteaad of the other techniques like inserting Print statements).

A brief outline of my technique follows. I suggest that you also watch the screencam demonstration at http://notestracker.com/tips/debug_web_agent.htm that I hope to make available soon.

Note that while I'm talking LotusScript in this article, the technique should also work for Domino Java Web agents and maybe even for Formula Language Web agents (although I haven't tested either of these). In fact, it might work in just about any release of any programming language on any platform (a wild claim, but we Aussies are not averse to making such assertions).


The Debugging Technique
Key steps in locating the coding problem in your Web agent:

  1. Develop a single, simple LotusScript statement that will cause the agent to throw a specific, easily-recognized error message at the Domino console. (This, as usual, will incorporate your Web agent’s name, which will distinguish your agent from all the other console message sources.)

    It helps if you have handy access to the Domino console. In fact I find it most convenient if you can run the Domino server, Domino Designer, Web browser (and Notes Client, for that matter) all on a single machine -- nothing could be handier than that.

  2. Place the debug statement at a strategic point in the agent’s code stream, then trigger the agent and
    watch for a specific termination error message at the console. You now know that the Web agent has successfully reached exactly that particular point in the code stream before strinking your deliberate error trap.

  3. Now cut-and-paste (not copy-and-paste) the debug statement to another strategic point further on
    in the agent’s code stream. Then go back to Step 2, running the Web agent again.


  4. Iterate through the above two steps until eventually a point is reached in your agent’s code stream where either
    (a) there is an error message sent to the Domino console that is caused by something in the agent's code stream that is other than your deliberate debug statement, or
    (b) the agent terminates without that error message. In the latter case you know that the faulty code must lie between that statement and the end of the code stream.

What do I mean by “strategic points” in the code stream? Each such point is simply a statement that represents a major logical step in either the agent's mainstream or in one of its underlying subroutines and functions.

I strongly suggest that you utilise a binary search approach to decide where such points are in the agent's code stream. (I don't want to teach you to "suck eggs" -- but if you're not familiar with this approach you can find out about it at a site like http://www.nist.gov/dads/HTML/binarySearch.html or by doing a Google search on "binary search".)

Focus first on the agent's mainline because the error might lie in the mainline itself. If you don't locate the error in the mainline, place the debug statement one-at-a-time at the very start of each successive subroutine (or function), and move on in the “binary search” fashion within the subroutine until you’re sure that the fault does not lie in
that subroutine, then move on to the next subroutine. This methodical approach is far more efficient than randomly picking points to place the statement.

My choice was to deliberately force a Zerodivide error to generate the Domino Console errors.

Naturally, in a mathematical operation, there cannot be an attempt to divide by zero, and the LotusScript compiler is smart enough to disallow statements that it is able to predict will cause division by zero, such as:
z% = 1 / 0 or z% = 1 / ( 1 – 1 )

To get around this issue, I use a statement of this general form:
zerodivide% = 1 / ( zerodivide% - zerodivide% )

The % character forces the field to be an integer, Leave it off, then the field becomes a LotusScript variant and the zerodivide technique fails.

In practise, I use this terse form:
zd% = 1 / (zd% - zd% )

or simply: z% = 1 / ( z% - z% )

And in all but simple agents I use the following variation:
z% = 1 / ( z% - z% ) ‘ ###########################

where the string of hash symbols (#) is added to be an "eye-catcher". This helps to make your debug statement stand out in your agent's code stream, enabling you to locate the statement much more easily when you want to cut-and-paste it elsewhere. Likewise it also helps you not to overlook it (and so leave it in the agent's code stream) when you've finished debugging and need to remove it entirely.

As mentioned earlier, I try to run the Domino test server on the same workstation that I'm using for the Domino Designer and Web browser (nice, but not always achievable). This way, I can see the Zerodivide error message the very instant that it appears on the Domino Console, a fraction of a second after opening the affected page in the Web browser. (You definitely should view the screencam to see what I mean by this.) The next best thing is
to run a remote Domino Console on your workstation, or to have the Domino server system very close by. This arrangement can significantly improve your debugging turnaround time, compared with not having to use a remote Domino test server.



Summing Up

I'm not pretending that the above technique is "rocket science" but that was precisely my objective: to describe a straightforward and painless technique which enables you to quickly hone in on the whereabouts of that elusive coding problem in your Domino Web agent. Try it out and see if it works well for you too. Otherwise, if you have an even simpler approach, send us your feedback.


Technorati tag:

Notes-a-wocky

By Tony Austin
Principal, Asia/Pacific Computer Services

with Apologies to Lewis Carroll


Ed Brillig, and the slithy toves
Did gyre and portal in the wabe;
All mimsy were the Bloggergroves,
And Lotusphere outgrabe.

"Beware the WebSpherewock, my son!
The jaws that bite, the claws that catch!
Beware the DB2 bird,
and shunThe frumious Javasnatch!"

He took his portal sword in hand:
Long time the Exchange foe he sought--
So rested he by the Workplace tree,
And stood awhile in thought.

And, as in URLish thought he stood,
The dotNETwock, with eyes of flame,
Came whiffling through the Spam-filled wood,
And Outlooked as it came!

One two! One two! Pride of Big Blue,
The Domino blade went snicker-snack!
He left it dead, and with its head
He went a-Replicating back.

"And hast thou slain the dotNETwock?
Come to my arms, my Lotus boy!
O Eclipse day! Callooh! Callay!"
He chortled in his NSF joy.

Ed Brillig, and the slithy toves
Did gyre and portal in the wabe;
All mimsy were the Bloggergroves,
And Lotusphere outgrabe.


-------
For the Lotus Notes/Domino community.

Inspired by a recent visit to Jabberwocky Variations -- read the original poem there (and other variations), then see if you too can go one better than Lewis Carroll.

Note Item Not Found

by Tony Austin
Principal,
Asia/Pacific Computer Services

Q: Why did the Error Message cross the road?
A: To get the point across!

Or, putting another spin on it ...
Q: When is an Error Message not an Error Message?
A: When it doesn't convey readily useful information.

I was introduced to usability design in the mid-1970s, fairly early in my career at IBM Australia. This gave me a keen ongoing resolution to keep usability, efficiency, practicality and ease of use high on my list of priorities -- for all the products that I use personally and those that I deliver to my clients.

During recent years I've been on the lookout for of usability-related resources on the Web, and just in the last few weeks I came across some articles that got me pondering about error messages and error handling. Here are just a few resource sites relevant to the point I'm making software usability, error handling and error messages:

  • Human Factors International and Good Experience and This Is Broken
  • Software Testing Shouldn't Be Rocket Science - If you think that YOU are having a tough time debugging Notes/Domino software, read this to find out what they had to do to debug and fix radio problems with the Cassini spacecraft, launched way back in October 1997. ... "Good testing is about attitude, where a developer takes pride not just in the elegance or volume of his or her code, but in whether it meets the user's requirements and performs reliably in its first incarnation."
  • Make your error messages meaningful - Many applications treat users as if they were programmers. Messages that report errors are often cryptic, contain meaningless codes, and provide no help regarding what to do next. While the developers who wrote the application can use those messages, most users are left with one option: call the help desk. This article describes a more appropriate kind of error message for users: one that includes description, cause, and recovery steps. (I worked extensively with IBM's System/36-System/38-AS/400, precursors to the current iSeries range, and their Help and error message support functions pretty much like this. In contrast, I've found this side of Notes -- and quite a few other PC-based products for that matter -- quite deficient and counter-productive.)
  • Joel on Software (by Joel Spolsky) -- a rich feast, including Lord Palmerston on Programming with its concept of "programming worlds, each of which requires a tremendous amount of knowledge for real proficiency"
  • Annoyances.org
  • Dr Dobb's Journal and BYTE and SD Magazine
You'll find these links and many more like them on my web site, at http://asiapac.com.au/Links/Programming.htm (or its mirror http://notestracker.com/Links/Programming.htm) as well as http://asiapac.com.au/Links/Design.htm (or its mirror page http://notestracker.com/Links/Design.htm).

Seasoned IT practitioners will know that requirements analysis, design and programming is very hard work (at least it is if it's done conscientiously, thoroughly and professionally). No non-trivial piece of software is perfect -- not mine, not yours, not anybody else's. However even the best products have their blemishes and limitations.

The larger and more complex that a piece of software is, the more likely that it will contain minor or even major flaws. Incomplete analysis, wrong design assumptions, inferior coding implementations, unforeseen situations, the overstepping of functional limits, server and network performance limitations, general sloppiness and carelessness, all can lead to errors of one sort or another.

All the same, it's pleasantly surprising that things work as well as they do -- most of the time! Yet I never cease to be amazed how some rough edges and annoying behaviors seem never to be reported or publicly discussed, lingering on in what are otherwise outstanding products.


Is it just me who has noticed them? What about the product developers, testers, the marketers -- didn't they notice them? Or maybe they did, but the "powers that be" consider the deficiencies too insignificant (or costly, or awkward) to fix. I'm sure that many flaws survive because there's no easy and painless mechanism to report the deficiencies (bugs, usability issues, incorrect or obsolete or missing documentation, performance problems, and so on)?

It's far easier to be a destructive critic than to be a designer and builder! (And it certainly can be a humbling experience to have your own deliverables validly criticized.)

Don't get me wrong, I remain a dedicated Notes/Domino enthusiast, and it's in a vein of positive criticism I've submitted this article.

All the more so with the "buzz" coming out of last month's Lotusphere 2005 conference indicating that there's new IBM impetus behind Notes/Domino. The doomsayers stand confounded, So surely it's not too late in the product cycle for it to be worth fixing matters like the ones discussed below.

Some things that aren't all that Notes-worthy
Let me now change into "squeaky wheel" mode and illustrate my point via some of my pet peeves, niggling issues and bad experiences over the years with Lotus Notes/Domino and their companion products -- up to and including the very latest Beta3 (late January 2005) release of Notes Domino 7. Naturally I'd like to see any remaining Notes/Domino issues fixed before ND7 ships.

Below are a few things that I find particularly annoying and time-wasting.


Vague or Misleading Error Messages
What do you think is being indicated by the following Domino console log entry, which came up as soon as I started testing a new web agent (which came up as soon as I clicked on a URL for a Notes document in a Customer database)?

11/10/2003 07:06:49 PM HTTP Web Server: Lotus Notes Exception - Entry not found in index [/NotesTracker_Customer.NSF/27fb8884f5997b4dca256bba004168de/ba19bbf87ed08706ca256c6300242527?OpenDocument]

That's a real gem, isn't it? A model of clarity. It must be that a document is missing from a view index, since it says "Entry not found in index". What are those two IDs? Maybe the first is a view ID and the second is a document ID. Can I somehow trace which document is missing from the view? Where do I start? ... Help! Aidez moi! Aiuto!

In this case it turned out that it had nothing at all to do with any user documents in regular views. I had fallen into the common trap of incorrectly spelling the name of the web agent in the form's WebQueryOpen event. The index being referred was some sort of internal structure used by web agents, not some view index that I had any control over.

The error message should have told me something meaningful. such as: "Entry not found in index [NotesTrackerWebQueryOpen]" or better still "Agent not found [NotesTrackerWebQueryOpen]". Maybe the vague nature of the message derives from the original Domino Go Webserver code -- coming from a non-Domino source, IBM's Internet Connection Server (ICS), that was never tailored to fit Notes/Domino constructs such as web agents -- but to me that's "a reason, not an excuse".

Totally Obscure Error Messages
Now to explain the title for this article. It was way back in 1994 (Release 3 days, only my second year working with Notes). I was asked to enhance a database by secure some information a form with a signed field. I entered the necessary modifications to the form's design and confidently saved them. Then all hell broke loose - or that's how it felt to me at the time -- I had been in IT for some 25 years, but it was only my second year of my "programming world" with Notes).

Every time that I now attempted merely to open a document using the modified form design, I got nothing but a dialog box telling me "Note Item Not Found". Nothing else, just those four words. (This is but one example of Notes error messages that aren't documented anywhere at all. The same holds for quite a few other products. They all should be comprehensively documented. If you're lucky, you might find them described in some support knowledge base or other, but you're not always lucky.)

At the time, to me this was a totally meaningless and inscrutable message. In fact, it still is!
I asked myself questions like: What on earth does "Note Item not found" mean? What is meant by a "Note item"? (At that point in time I had never investigated the Notes C API, and not until a year or two later when LotusScript arrived on the scene with R4 did I learn what was meant by a Notes "Item" It certainly was never mentioned in any of the standard Notes classes conducted by Lotus Education.)
Why couldn't the message have told me which particular item was not found, and what was supposed to be happening during the opening of the document when the absence of the item caused the failure?

And what should I do to rectify the situation? Nothing, as it eventuated. There was nothing at all wrong with the form design. I learned a few weeks later later that the error was caused by a bug in Notes R3 with the handling of signed fields, and only a bug fix make the unwanted message go away.
I expected better, because of my IBM background working for years with the superb IBM System/38 and AS/400 systems, where the error messages are much more specific (and many of the messages advise you ins ome detail where to look and what to try in order to recover from the situation).

Inconsistent Behavior
"Little things are important," they say, and this is one of them. The inconsistent and off-putting behaviour that I'm discussing here first appeared, as far as I can recall, with Domino Designer R5 (but it might have been as far back as R4).

A developer often needs to have four or more windows open simultaneously (Notes client, Domino Designer, Web browser, Domino console, Help documentation, maybe a Java IDE, etc). Therefore screen real estate is at a premium, and she/he doesn't need any more windows open. Irritation aside, because of this inconsistency precious screen real estate is chewed up and your productivity is diminished.

The issue is this. When you're using Domino Designer against a database, opening some but not all of the various types of design element (forms, views, etc) automatically forces the Properties box to open, whether or not you want it open -- and often you want the Properties box to remain closed because it really gets in the way during some design activities (such as moving view columns around).
Unfortunately this errant behaviour persists in ND7 Beta 3, as follows:
  • The Properties box quite properly remains closed when you edit a Frameset, Page, Form, Outline, Subform, Script Library, Navigator, Database Icon, Help About and Help Using documents, Database Script.
  • It inappropriately opens when you edit a View or Folder, Agent, Web Service, Shared Field, Shared Column, Shared Action, Image (maybe), File Resource (maybe), Applet (maybe), Style Sheet (maybe), and Data Connection (maybe). Here, the "maybes" mean that it may be appropriate for the box to open when you click on the item, since there is not underlying form to open for each of these.
If you want the Properties box to remain closed, it definitely should not open against your wishes. There's no excuse for this unsystematic behaviour!

Time for a Break?
This one isn't to do with error messages and error handling as such, but it is "in the next ballpark" so to speak, and is certainly one of my pet peeves.

It's this: hitting the CTRL+BREAK keyboard shortcut is supposed to be "Stop operation in progress". Sometimes it does do that almost instantaneously.

However,sometimes it doesn't. I get extremely frustrated when the CTRL+BREAK key sequence has no immediate effect whatsoever, and Notes continues on its merry way for tens of seconds or even minutes. This tends to happen most often when you attempt to open a database that Notes has to hunt for and connot immediately locate. Notes then can "get the bit in its teeth" and start searching on goodness knows how many Domino servers across your network.
You may want to kill the search once you realize that clicking the Open button was a mistake, so you want CTRL+BREAK to immediately cancel the search for that database. But the keyboard locks up for what seems an eternity nd no amount of pounding on CTRL+BREAK has any effect. I sometimes have despaired and resorted to killing the entire Notes Client -- not at all an elegant solution, but at least I then get back to where I meant to be in ten or twenty seconds rather than minutes.

So another of my expectations is for the CTRL+BREAK key sequence to rapidly and consistently stop the current operation, whatever it is. (Just like CTRL+ALT+DEL nearly always does for Windows.) Get with it, Notes!


Have Your Say
I hope this article prompts you to think more keenly about usability, better error messages, and related matters. Am I being reasonable? Why do I sometimes feel a bit like the Tom hanks character, cast away on a remote island, and conversing with a volleyball to remain sane!

If you share similar concerns, here's a chance for you too to get them off your chest. Let me know how you feel about the above and any of your own problems and peeves with Notes Domino (and their related products).