I hate having to reply to my own emails. I just saw the "developer-facing"
comments as I was re-reading this.
Maybe this really is a terminology issue.
You and I may not be that far off. If you're talking quadrant 3, then I agree
-- it's very hard to automate those tests.
http://blog.jonasbandi.net/2010/02/agile-testing-quadrants.html
However, Quadrants 1,2,and 4, are highly automatable. Quadrant 2(Q2) is still
customer facing(or business facing in Janet/Lisa's parlance), and those tests
are highly automatable.
I would also like to point out whether these code level tests are produced via
TDD or whether they are unit tests, story tests, component tests, service tests,
etc is pretty irrelevant IMO. Code level tests can be in Q1 and Q2, and they
are generally highly automatable, and I believe there to be huge value in
automating them. (I also believe you can overdo it too -- anything can be done
wrong.) I do think the best Q1/Q2 tests are produced via TDD and ATDD, but that
is not required for them to be highly valuable. Sometimes you write tests for
previous code after you've learned what TDD was. :-) There's still value there.
I would propose that code level tests(Q1 and often Q2,and maybe even Q3) tests
are the bees knees as well. Code level tests produced via TDD are probably the
best kind, but they need not be originated from TDD to be highly valuable. They
also need not be unit tests to be highly valuable. I know many people think
"Unit Test" equals roughly "Anything written by the developers in a harness like
JUnit", but that's not accurate. JUnit can run all sorts of tests: unit,
component, integration, service, UI, and even GUI tests. The most credible Unit
Test definition is roughly:
http://www.artima.com/weblogs/viewpost.jsp?thread=126923. For all but the most
trivial of classes, Unit Tests must create test doubles for their dependencies
(Mocks, Fakes, Dummies, Stubs). There is a lot of value in Unit Testing, but
those aren't the only code level tests that are highly valuable and highly
automatable.
So, in summary, I think Q2 customer facing tests are highly automatable, but I
also believe Q3 customer facing tests are generally hard to automate, if not
downright impossible. Q1 tests are highly automatable as well.
And Full Circle...
I believe you do need automated tests to "succeed with Agile." Technically, you
don't need to do automated tests to be Agile (as defined by the Agile Manifesto
and associated Agile Principles), but I contend, for projects of non-trivial
size or duration, that you cannot "succeed with Agile"without automated tests.
I looked at Allistair's project page, but that didn't have the test automation
details I'd need to see where some project "succeeded with Agile" without test
automation(indeed, many of the projects looked non-Agile). If someone has a
real life case study, I'd sure like to hear about it. I'm not saying I'm
totally right, I'm just saying that I am, as of yet, highly unconvinced. Maybe
there are projects out there that have a ginormous budget to hire a whole city
of manual testers, but I'm not aware of a "successful with Agile" one.
-------
Charles Bradley, CSM, PSM I
Experienced Scrum Coach
My blog: http://scrumcrazy.wordpress.com/
________________________________
From: Charles Bradley - Scrum Coach CSM PSM I <chuck-lists2-***@public.gmane.org>
To: agile-testing-***@public.gmane.org
Sent: Wed, March 9, 2011 8:41:55 PM
Subject: Re: [agile-testing] Re: Agile Tester Certification?
Do we have a terminology conflict here?
Post by MatthewYes, if (customer-facing) testing were a straightforward, definable process,
sure, we could automate it.
It turns out, not so much.
So, while, again, I do believe (developer-facing) TDD is the bees knees and I'm
happy to work with developers who do strong unit testing, it turns out to not be
sufficient. (I would go so far as to say, in some cases, not
necessary. Foolish to ignore, maybe, but not absolutely necessary for certain
types of projects of certain durations under certain constraints.)
Are you saying that "customer-facing" tests are not terribly automatable, but
code level tests are the bees knees?
-------
Charles Bradley, CSM, PSM I
Experienced Scrum Coach
My blog: http://scrumcrazy.wordpress.com/
________________________________
From: Matthew <matt.heusser-***@public.gmane.org>
To: agile-testing-***@public.gmane.org
Sent: Wed, March 9, 2011 3:04:58 PM
Subject: [agile-testing] Re: Agile Tester Certification?
Post by MatthewI sense that you guys have had this debate before on the list
Yes. And every year a new crop of CS students - smart, fresh-faced from MIT,
uCal Berkley, and Carnegie Mellon Graduates. Young people with IQ's in the
130 to 150 range.
They are CS students. What they /*do*/ is automate business processes.
So they look at software testing and say "Oh, I see, a simple straightforward,
repeatable business problem. Why, we should automate it!"
And we have this conversation again.
Yes, if (customer-facing) testing were a straightforward, definable process,
sure, we could automate it.
It turns out, not so much.
So, while, again, I do believe (developer-facing) TDD is the bees knees and I'm
happy to work with developers who do strong unit testing, it turns out to not be
sufficient. (I would go so far as to say, in some cases, not necessary. Foolish
to ignore, maybe, but not absolutely necessary for certain types of projects of
certain durations under certain constraints.)
Post by Matthew"You don't need automated tests to succeed with (capital A)Agile."
Is there a good resource for documented case studies for non trivial projects
where this was true?
Alistair was a full-time methodologist working at IBM doing research when he
came to this conclusion; he interviewed about 200 people and two dozen teams.
You can see the short list of projects in this article:
http://alistair.cockburn.us/Characterizing+people+as+non-linear%2c+first-order+components+in+software+development
He doesn't get into how those projects felt about test automation, but he told
me that was his conclusion. (I am certain there are more detailed break downs
of those projects on his site /somewhere/).
regards,
--
Matthew Heusser,
Personal Blog: http://xndev.blogspot.com/
Test Community Blog: http://softwaretestpro.com/blog/
Twitter: mheusser