Fervent Coder

Coding Towards Utopia...by Rob Reynolds
posts - 278 , comments - 431 , trackbacks - 0

My Links


Rob Reynolds

Subscribe to Fervent Coder RSS
Subscribe to Fervent Coder by Email

About Me

I manage several open source projects. Need...
   ...package management for Windows?
   ...automated builds?
   ...database change management (migrations)?
   ...your application to check email?
   ...a monitoring utility?

I also write for

Like what you are reading? Want to buy me a cup of coffee?
PayPal - The safer, easier way to pay online!


Post Categories


End2End Automated Testing: Second Day & More Thoughts on Why Automation

How about doing a full regression of the system under development every ten minutes?

Second Day Automation Testing

One of the things I neglected to mention (on purpose) in my End2End post the other day was that you also test for the second day.  In this test you are sending in new items and updating some existing items. You also leave some of the items from the first day completely alone to see what happens to them.

This is done in the same way as the first, except you look for a second input file (Input_Update) to move to the application and a second results file (Results_Update) to compare the second day to.  This is the equivalent of finding out how your application will behave when presented both new information and information it receives of already existing items.  Does it update all rows or only certain rows as defined by your business rules?  What about existing rows with no updates? Do they get deleted or not? Does the system work correctly in this aspect?

Why Automated End2End Testing?

The interesting thing to note here is that this testing is no different than QAs (quality assurance) / testers do anyway.  This kind of testing is nothing new.  Every self-respecting tester worth his/her weight does all of this.  MANUALLY.  BUT, and this is big, but we have so much information to verify sometimes that the cost/benefit of verifying everything is not there. We are presented with so much information and scenarios that it would take WAAAAAAAYYYY too much time to verify everything, so instead we sample certain areas and tend to miss things. Actually "tend to" is not correct. We miss things sometimes.  It just happens. We just can't look at it all manually and justify the cost of it and the sheer amount of time it would take.

Now can you imagine having to do a regression test of the system before a release? Think of one module with 400 scenarios and another module with 250 scenarios.  How long would that take someone if they were doing a full regression?  That's right, a LONG time.  I would go so far as to say because of the costs and time, people in software development either don't do them or only do them very infrequently because they absorb weeks of your time (usually right before a release).

How about doing a full regression of the system under development every ten minutes?

Up until now this concept for us has been unheard of. If we automate the process with verifying the results against expected results given particular inputs, we black box the system in the same way that a tester would.  But now the expected results are not in the QAs head and on the test scripts which have to be manually verified and prone to human error. The results are in files and every row, every column is checked for correctness. Every Row. Every Column. All checked and verified. Not just a sample. 1 item, 20 items, 10,000 items. Every single piece of information verified for correctness. Now imagine the possibilities of doing a regression whenever you feel like it or multiple times a day!!  Imagine saying this as a tester:

I just regression tested over 4,000 different scenarios in ten minutes and only had two places that we need to look at. 

Minutes! Not weeks! Minutes!!!! The testers I know salivate at this concept.  End2End is not a system for developers, it is a system for our testers to help us ensure we have better software in the end. To fully realize the benefits of what this could provide your testers, share these articles with them.  Find out how they feel about it.

Isn't This Like Fit/FitNesse?

Well, End2End is kind of like Fit, but not really.  The ideas behind Fit are the same as the ideas behind End2End. From Wikipedia:

It integrates the work of customers, analysts, testers, and developers.

From C2:

Fit is a tool for enhancing collaboration in software development. It's an invaluable way to collaborate on complicated problems--and get them right--early in development.

Fit allows customers, testers, and programmers to learn what their software should do and what it does do. It automatically compares customers' expectations to actual results.

End2End provides the idea of the same great collaboration between users, testers, and programmers with the same core concepts as Fit. Fit/FitNesse are different because they can cut sections of the system and test there.  They don't automate the actions a user or tester would take to run the system. The benefit of this is that they give testers the ability to white box test the system a little more.

I believe there is a place for white box tests, and that is with unit (logical) and integration testing done through frameworks such as NUnit, MbUnit, MSTest, etc. As good developers who are Test Driven, we have already completed testing many of the scenarios that Fit/FitNesse fit into.  It is much easier to maintain tests in these other test frameworks, and if we are already doing these kinds of tests with unit and integration testing, then why do we need to repeat the process with Fit/FitNesse (and violate DRY)?  If we have good reporting with MbUnit's framework, and soon with Gallio's Universal Testing Platform, the results of these tests can be provided as part of the documentation for testing.

From what I have noticed across the industry, Fit/FitNesse is not something people stick with after adoption, for one reason or another.  Perhaps it is maintenance.  Perhaps people are trying to do things with it that it was never meant to do. We tried FitNesse for awhile but decided it wouldn't work for us due to some of the reasons cited here in addition to other reasons.


Back To Basics

With End2End we are testing the system as a whole without cutting into any piece of it.  We get back to the basics of what QA testers do, which is mostly black box testing.  And then we automate it so they can focus on what they really want to focus on.  Breaking our systems.

Print | posted on Wednesday, June 25, 2008 10:38 AM | Filed Under [ Code ]


Comments are closed.
Comments have been closed on this topic.

Powered by: