Fervent Coder

Coding Towards Utopia...by Rob Reynolds
posts - 278 , comments - 431 , trackbacks - 0

My Links

News


Rob Reynolds

Subscribe to Fervent Coder RSS
Subscribe to Fervent Coder by Email

About Me

I manage several open source projects. Need...
   ...package management for Windows?
   ...automated builds?
   ...database change management (migrations)?
   ...your application to check email?
   ...a monitoring utility?

I also write for



Like what you are reading? Want to buy me a cup of coffee?
PayPal - The safer, easier way to pay online!

Article Categories

Archives

Post Categories

Image Galleries

Sites

End2End Automated Testing

We have started an initiative this year. We call it End2End Automated Testing. The idea is based on a couple of factors. Many times in the past we have not had an easy way to verify our configuration, security and deployment to an environment in an automated way.  The second is that regression testing is a very arduous process for our QA staff when it must be completed.  How do we give time back to our testers and verify everything in a given environment is correct? With automation of course!

So what is End2End Testing? It is a way of taking what we expect the system to do based on our inputs and verifying that against the actual results in an automated fashion.  We influence the inputs into a system so we know what we expect the outputs are going to be.  Then we check those outputs, or actual results, against some expected outputs, or expected results files to see if we have matches. This helps free up the Quality Assurance staff to focus more on figuring out what should happen when a given scenario is passed into the system. 

We make it easy enough for them to focus on "If I pass this into the system, this is what I expect to get out of the system or to happen with the system."  The real benefit of this is an actual automated regression suite with a focus on what if for the QA staff! They get to concentrate more on actually breaking, err, testing the system to ensure it works correctly and meets a level of acceptable quality for the ultimate end users.

End2End testing is not a replacement for unit tests.  You need that quick feedback cycle that you get from those.  End2End is also not a replacement for your integration tests.  Although a little slower than unit (logical) tests, integration tests give you feedback much faster than End2End will be able to.  End2End is not a replacement for your testers. They must really focus on specifications to put good scenarios through the system to accurately tests all aspects of what might happen.

End2End is a strong collaboration between developers and testers.  In incorporating End2End, you will work very closely with each other. End2End is unforgiving. When it runs, it only passes if everything is 100% correct against specifications.  It takes awhile to get expected results built to verify actual system results against.  Once you do, the benefit is knowing the application you are creating is correct up to that point.

Principles of Automated End2End Testing

  • End2End must be able to affect the inputs to the system.
  • End2End must be able to verify results (possibly in a database).
  • End2End will couple to those two points only, but will be tightly coupled to those points.
    • This is really no different than a QA (tester) testing a system under development would be coupled.
  • End2End is not actually part of your project at all. It is in itself a mini-project.
    • That means it deserves it's own unit tests to ensure it's functionality.
  • End2End should start from a known point for both inputs and outputs (preferably empty starting outputs).
  • End2End should be able to be completely encapsulated on a LOCAL developer machine as well as on some remote environment.
    • That means you may start some more processes or do a couple more setup options locally than you would in other environments.
  • End2End should provide a Reset Test to clean up everything for your users or testers so they can start the process over.
  • End2End should include a readme file for both your developers and testers.
  • End2End should be helpful in identifying what exactly is wrong for every point that is wrong and possibly provide pointers on how to correct the issues.

 

How To Do the End2End Test

  1. Setup Step1: Automatically set up the inputs.  This could be putting a file somewhere or setting up database tables with scripts.
  2. Setup Step2: Optional - Set up network permissions on the remote directory.
  3. Setup Step3: Call the remote application, have it execute in the remote directory. Optionally, wait for exit.
  4. Setup Step4: Optional - Remove network permissions to the directory.
  5. Setup Step5: Figure out the trigger to know when your application is ready. You could put a file watcher or a database table watcher to know when ready. Or possibly, waiting for the application to exit is good enough for your needs.
  6. Setup Step6: Make a data table from your expected results.
  7. Setup Step7: Make a data table from your actual results.
  8. Setup Step8: Remove the columns that you have decided not to verify against.  We call these the ignored Fields.
  9. Test1: Verify the total rows of expected results versus the actual results.
    1. Print the information. "There are _ rows in the expected results and _ rows in the actual results."
    2. Set the testSuccess variable to false if different.
    3. Continue the test suite.
  10. Test2: Verify the column count of the expected results versus the actual results.
    1. Print the information. "There are _ columns in the expected results and _ columns in the actual results."
    2. Set the testSuccess variable to false if different.
    3. Continue the test suite.
  11. Test3: Verify the columns are named the same in expected results versus actual results.
    1. Set the testSuccess variable to false if different.
    2. Print the information if fails. "I had column1, column2 in my expected results but could not find them in the actual results.  I had column3 in my actual results but could not find it in the expected results."
    3. Continue the test suite.
  12. Test4: Verify the actual data in each row versus the expected result.
    1. Do not expect the results of both to be in order. Filter down and find the specific row.
    2. Print both the expected result row and the actual result row. Include headers if possible.
    3. Set the testSuccess variable to false if different.
    4. Print the information if it fails per column. "I expected a value of _ in column1 but actually had _."
    5. Continue the test suite for every row.
  13. Print how many rows passed and how many failed.
  14. Verify whether the End2End test failed or not. Assert.IsTrue(testSuccess)

Print | posted on Monday, June 23, 2008 1:26 AM | Filed Under [ Code ]

Feedback

Gravatar

# re: End2End Automated Testing

Follow up post: http://ferventcoder.com/archive/2008/06/23/end2end-automated-testing-setting-network-permissions-automatically.aspx
6/25/2008 10:41 AM | Robz
Gravatar

# re: End2End Automated Testing

Follow up post: http://ferventcoder.com/archive/2008/06/23/end2end-automated-testing-calling-applications-remotely.aspx
6/25/2008 10:42 AM | Robz
Gravatar

# re: End2End Automated Testing

Follow up post: http://ferventcoder.com/archive/2008/06/25/end2end-automated-testing-second-day-amp-more-thoughts-on-why.aspx
6/25/2008 10:43 AM | Robz
Gravatar

# re: End2End Automated Testing

Setup Step6: Make a data table from your expected results.

This is a CODE DataTable. You could set up something else if that works better...
7/8/2009 3:50 PM | Robz
Comments have been closed on this topic.

Powered by: