Insanity is defined as
"doing the same things
over and over and
expecting a different
result". As a
tester, it is easy to
get into a rut of doing
the same thing over and
over again but never
really improving your
test process or
identifying things that
can aid in bringing real
success to your job.
To improve, we need to
understand our goals and
measure our progress
towards them. This
article discusses how
to develop metrics that
aid you in achieving
specific goals.
What are
Powerful Metrics?
Metrics are simply a way
to measure specific
things you are doing,
but you could create
hundreds of irrelevant
metrics that really
don't move you any
closer to solving your
everyday problems.
Powerful metrics
are a small set of
indicators that are
targeted to help you
accomplish a specific
set of goals.
Metrics Driven
by Specific Goals
Before defining your
metrics, ask yourself
"What goal(s) am I trying
to accomplish?", then
write your goals down.
For example, typical
goals for a tester might
be to:
-
Ensure that each new
feature in the
release is fully
tested
-
Ensure that our release date is realistic
-
Ensure that existing features don't break with the new release
Now that we have our
goals defined, let's
figure out how we can
create a set of metrics
to help us accomplish
them.
“Ensure that
each new feature in the
release is fully tested”
To do
this, we must have
assurance that we have
enough positive and
negative test cases to
fully test each new
feature. This can be
done by creating a
traceability matrix that
counts the number of
total test cases, number
of positive test cases
and number of negative
test cases for each
requirement:
Requirement |
# Test
Cases |
#
Positive |
#
Negative |
Comments |
Visual Studio
Integration |
104 |
64 |
40 |
Good Coverage |
Perforce
Integration |
0 |
0 |
0 |
No Coverage |
Enhanced Contact
Management
Reporting |
45 |
45 |
0 |
No Negative Test
Case Coverage |
Contact
Management
Pipeline Graph |
10 |
8 |
2 |
Too few test
cases |
“Ensure that
our release date is
realistic”
To do
this, we must have daily
running totals that show
us test progress (how
many test cases have
been run, how many has
failed, how many has
passed, how many are
blocked). By trending
this out, day-by-day, we
can determine if we are
progressing towards the
release date.
Software Planner
provides graphs that
make this easy:
Another way to do it is
to create a Burn down
chart that spans the
testing days. On the
chart, it will decrement
each day by the number
of outstanding test
cases still to be run
(awaiting run, failed or
blocked) and we can
compare those totals to
the daily expected
number to determine if
the release date is
realistic. Let's
assume we have a 2 week
QA cycle, we can define
the number of test cases
to run and daily we can
keep track of how many
are still left to
complete versus how many
should be left to
complete.
In the example above, we
can see that we are not
getting through the test
cases quickly enough.
On day 5, we are behind
schedule by 15 test
cases and we can see
this clearly on the Burn
down graph -- the red
area (actual test cases
left) is taller than the
blue area (expected test
cases left).
“Ensure that
existing features don’t
break with the new
release”
To do
this, we must run
regression tests (could
be manual or automated)
that tests the old
functionality. Note:
This assumes we have
enough test cases to
fully test the old
functionality, if we
don’t have enough
coverage here, we have
to create additional
test cases. With
regression tests, you
can normally stick to
“positive test cases”
with this group, unless
you know of specific
"negative test cases"
that would be helpful to
prevent common issues
once in production.
To ensure you have
enough of these test
cases, it is a good idea
to group the number of
regression test cases by
functional area.
Let's assume we were
developing a Contact
Management system and we
wanted to be sure we had
regression test cases
assigned to each
functional area of that
system:
Functional Area |
# Test
Cases |
Comments |
Contact
Management
Maintenance |
25 |
Good Coverage |
Contact
Management
Reporting |
0 |
No Coverage |
Contact
Management
Import Process |
10 |
Good Coverage |
Contact
Management
Export Process |
5 |
Coverage may be
too light |
To determine if release
dates are reasonable for
your regression testing
effort, use the same
Burn down approach
illustrated above.
Summary
As the new year begins,
dedicate yourself to
improving your job by
identifying your goals
and tracking metrics
that help you determine
how you are trending
towards your goals.
The metrics listed above
work fine for my team
but I would like to hear
what metrics you find
are helpful in your
organization. To
communicate that to me,
please fill out this
survey:
http://www.surveymonkey.com/s/DW656JZ.
If we great feedback
from this, we will
publish the results in
our next month's article
so that we can all learn
from each other.
| 
