TheLadders.com Developer Blog

The initial idea for creating a QA Toolbar came to me while I was ruing the monotony my good friend and QA Manager Sharon Fowler had to endure on a daily basis. I was concerned with the redundant manual effort she employed while she was reporting bugs: remembering the precise steps she took to produce a bug, accurately expressing the context in which the bug manifested, and then finally, communicating and clarifying that information with the developer(s) responsible for fixing the bug. It seemed wholly unnecessary that multiple humans be responsible for interpreting and re-interpreting data which is readily available at runtime. Also, since the developers are the ones responsible for defining variable names and their values, it seemed logical to include the information available at runtime since it is already presented in developers’ terms. These factors made the QA Toolbar seem like an obvious win for both the QA and Development teams.

Given the obvious benefits, there were only a few problems to solve, and most of them had very simple solutions.

The first problem was retrieving all the data for the QA Toolbar without affecting any existing functionality, or creating extra dependencies in the production environment.

The second problem was injecting HTML into every page which would not affect the layout of the page being tested.

So...How could I make the QA Toolbar appear on each page without affecting production, the page layout, or any other javascript functionality?

The enabling factor was an omni-present JSP include. I added some javascript into the JSP include which would show the QA Toolbar so long as the environment and the client met certain criteria.

Namely, we don’t want the QA Toolbar to show up for you all in production. As a precaution, we also only enable the QA Toolbar for machines inside our office. To address the production issue, I leveraged a variable set by the build and deployment script which indicates whether the application is running in “test mode”. The “test mode” environment variable and an IP filter are used to enable the QA Toolbar in all non-production environments.

Here’s how I injected the QA Toolbar into the page without affecting any other elements:

  
  var qaToolbarDiv = document.getElementById("qaToolbar");
  // make sure the qaToolbarDiv exists in the page
  if(qaToolbarDiv != null)
  {
    // set the qa toolbar div as the first element in the body
    var body = document.getElementsByTagName("body")[0];
    body.insertBefore(qaToolbarDiv, body.childNodes[0]);
  }
  else
  {
    // This lets the QA Team know that the tracking header (and thus the QA Toolbar) was not included in the page…we’d better fix this right away!
    window.alert('Unable to find QA Toolbar UI Elements.  Tracking footer not included on ' + window.location.href);
  }

Here is a screenshot of the QA Toolbar when you first load a page (before clicking the tab to expose the QA data):

What information did we need to see?

I figured I’d start with the basics. Which controller was invoked? Which logical view name was chosen? Which JSP is showing (after any filters have been invoked, the JSP may be different than the one originally specified by the controller)? Which user is logged in? What is the user’s email address, user id, and account status? I also figured it’d be nice to have a link directly to that user in our administrative application. Other information proved useful in testing marketing attribution, such as the user’s origin, the placement within that origin, as well as the particular creative the user clicked on. Finally, I added information describing which multivariate tests and which test cells had been selected for the current page view. This wound up being very helpful in debugging multivariate tests. Here’s a screenshot of the QA Toolbar for a basic user:

Yay! I was able to display all this information on the page. I was feeling pretty good after only one day of hacking, but I still hadn’t mitigated the bane of QA’s existence: having to copy and paste information from the page into the bug tracker. The killer feature would be a single click which would create a bug in our issue tracking system, JIRA, and include all this juice in the issue description. Luckily, I was already familiar with the JIRA source code, having extended it to add inline editing of issues as well as some other conveniences. This enabled the implementation of the “Create Jira Issue” button.

Here’s the code which assembles the information from the QA Toolbar for submission to JIRA:

  function populateJiraFields()
  {
    var includePageInformation = document.getElementById("includePageInformation").checked;
    var includeUserInformation = document.getElementById("includeUserInformation").checked;
    var includeABInformation = document.getElementById("includeABInformation").checked;
    var includeTrackingInformation = document.getElementById("includeTrackingInformation").checked;
    var includeSessionInformation = document.getElementById("includeSessionInformation").checked;
 
    var description =
            '\n\n------------------------------------\n' +
            'URL: ' + document.location.href + '\n' + 
            'Browser: ' + navigator.userAgent + '\n' +
            (includeUserInformation ? document.getElementById("userInformation").value : "") +
            (includePageInformation ? document.getElementById("pageInformation").value : "") +
            (includeABInformation ? document.getElementById("ABInformation").value : "") +
            (includeTrackingInformation ? document.getElementById("trackingInformation").value : "") +
            (includeSessionInformation ? document.getElementById("sessionInformation").value : "") +
            '\n\n---\nPowered by QA Toolbar';
 
    document.getElementById("jiraDescription").value = description;
    return true;
  }

To see how the runtime information is stored in the HTML page, here’s a glimpse of the rendered source:

<textarea id="userInformation" style="display:none;">

Quick Login: http://www.josh.laddersoffice.com:8080/login?LOGIN_USER=jlevinexxxxxx@theladders.com&LOGIN_PASS=xxxxxxx

 

 

User: 

  Email: jlevinexxxxxx@theladders.com

  

  Password: xxxxxxxx

  Subscriber Id: 13xxxxxx

  Name: Joshua Levine

  Account Status: Basic

 

  Attribution

    Basket: dl

    Carton: 00

    Egg: 01

 

  Subscriptions

    Product Name: Basic subscription

    Issue Quantity: 0

    Ladder: SalesLadder

    ---

  

</textarea>

 

<textarea id="ABInformation" style="display:none;">

AB Selections:

 

  Module xx - Live Chat On/Off Aug06

  Cell 3xx- Live Chat On/Off - Test

  

</textarea>

 

<textarea id="pageInformation" style="display:none;">

Page:

  

  Web Page ID: 11xxx

  Name: SEO Guest Job Search Job view

  Controller: com.theladders.web.controllers.xxxxxxxxxxxxxxxxxx

  JSP: xxxxxxxxxxxxxx.jsp

  View: indexView

</textarea>

 

<textarea id="trackingInformation" style="display:none;">

User Tracking:

 

  192.168.2.42 - Thu Nov 30 12:25:06 EST 2006 - http://sales.josh.laddersoffice.com:8080/job-search - 

  192.168.2.42 - Thu Nov 30 12:24:53 EST 2006 - http://sales.josh.laddersoffice.com:8080/Resub - 

</textarea>

 

<textarea id="sessionInformation" style="display:none;">

Session Variables:

 

  - __saved_searches = com.theladders.commons.util.CLASS

  - sessionConfig = com.theladders.commons.CLASS

  - __profile_stats = com.theladders.commons.util.CLASS

  - __preferences = com.theladders.commons.web.CLASS

  - qaToolbar = true

  - loginRoadblocks = com.theladders.web.controllers.CLASS

  - recruiterAuthentication = null

  - site = {}

  - accesslog.clientAddr = 192.168.2.42

  - tracking = com.theladders.commons.web.CLASS

  - authentication = User[13xxxxx,jlevinexxxxxx@theladders.com]

  - accesslog.user = jlevinexxxxxx@theladders.com

 

</textarea>

I’m hoping all this inside information will help some of you to better your QA team’s lives forever!

Happy hacking!

- Josh

Special thanks to Joe Ruglio for the QA Toolbar logo and for the infamous QA Toolbar t-shirts.

Posted by Josh Levine at 11:10 AM | Permalink | Comments (0) | TrackBacks (0)

Consider the following situation:

"
While testing the web site, QA goes to the "Join Now" page by clicking on a Job Listing from another website. After filling in the required information and submitting the form, they come to the Homepage (incorrect) rather than the View Job (correct) page. Why did that happen? The tester then opens up the issue tracking system and submits a new defect. They describe the steps they took and what they saw and submit the issue to a developer.

Now the developer has to reproduce the error. Which controller (an MVC architecture) was responsible for choosing the homepage for display? Which user was logged in? What was the state of the session at the time? What was the state of any other dynamic configuration on the site? What pages did they visit before coming to the page with the problem?

A lot of time is spent trying to rediscover information that was available at the time of the bug incident.
"

If you've ever been in this situation, say "Aye!"

Josh Levine, one of our developers had a good idea to address this situation. He wanted to create a client-side QA toolbar. That is, some AJAX code that runs in the browser, which is automatically attached to every page in our QA environment. The toolbar would expose server side state information and enable shortcuts for common testing and debugging needs.

The initial implementation had three functions:

1. Expose more information for QA to validate (not a replacement for functional testing, but a good shortcut)

2. Automatic issue submission (in our case, integrated with Atlassian JIRA)

3. Provide debugging information for the developers (session variables, click history, etc).

After some experience, we uncovered an additional use:

4. As a hook for other "session" based controls in the live environment. For example, choose a test cell (In our AB Testing framework), adjust error logging levels, quick-logout (usually a multi-step process), set and clear other flags and variables for testing and debuggnig purposes.

With all this information available, the situation plays out very differently. Now, when an error happens, QA can submit it with one click. And the submission can include controller name, username, JSP name, session state, click history, dynamic configuration data, etc. The QA person just has to add a few comments about what it is that was broken.

The model has proven to be extremely flexible and is responsible for substantial productivity gains for both QA and development.

The core functionality was completed and implemented in 4 days.

Look for Josh's post next, with details.

Posted by Marc Limotte at 10:38 AM | Permalink | Comments (1) | TrackBacks (0)

There's some great material in 37signal's new book "Getting Real". I just want to comment on one, "There's Nothing Functional about a Functional Spec". Lot's of spot-on points here. These are good points on how a functional specification can be used incorrectly. But there can be legitimate reasons to create and use one, if done properly.

Admittedly, the approach described in their book is typical and suffers from the problems they discuss. Instead, I recommend a different approach as described here in response to 37signal's Functional Spec point-by-point criticisms.

"Functional specs are fantasies" "They don't reflect reality" [of an application]

This is true, but they can reflect what the business thinks they want, as a first cut. As a development group, we want to be involved early on in the formative stages of product design. The functional spec gives an introduction into the current thinking of the business. We can digest it and discuss what we think it means and suggest alternatives.

"Functional specs are about appeasement"

Often the case, but if the functional spec is followed by real, open and honest dialog, people actually will be involved, rather than just "feel[ing] involved".

"Functional specs only lead to an illusion of agreement"

Don't bother getting sign-off. What are they signing off on? It's not detailed enough to mean anything and certainly doesn't represent any mutually understood agreement of either "what is being asked for" or "what will be built".

"Functional specs force you to make the most important decisions when you have the least information"

Definitely a danger. Decisions at this point should be non-binding. They should be decisions about the direction to take next, not the final direction.

"Functional specs lead to feature overload"

You can push back during the spec phase. Ideally, the spec will primarily contain core functionality. And the functional spec, in particular, should talk about functions, not features.

"Functional specs don't let you evolve, change, and reassess"

If you follow the advice above, this won't be the case. The functional spec represents the state of the current thinking, not the final product.

An additional benefit of the functional spec:

The functional spec and subsequent conversations can inform decisions about upcoming hiring needs for technology. While I don't expect a real, binding scope estimate to come from this exercise, you can at least get a ball park of the minimum effort. And that might indicate that you need more people. The work effort implied by the functional can be considered a minimum... I've never seen a project shrink in scope from what was in the functional.

Posted by Marc Limotte at 05:02 PM | Permalink | Comments (0) | TrackBacks (0)