[Cucumber] Do you live in a magic kingdom?

Discussion:

Matt Wynne

2012-10-26 14:59:31 UTC

A few weeks ago, a well-kown open-source contributor wrote a blog post[1] discussing some of his predictably controversial attitudes to test-driven development.

Dont use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if youre there!)

Of course he's missed the point slightly: in my magic kingdom non-programmers don't write tests, they *collaborate with* programmers to write tests. And then they read them, safe in the knowledge that they're accurate. But I digress.

The magic kingdom does indeed exist, and I love to spend time there. I bet you do to. I want the world to know how wonderful our magic kingdom is, and I need your help. I'd like some real quotes from real people who have experienced the value of using business-readable Cucumber tests as part of their development workflow.

Can you tell me a story about your magic kingdom? Do non-programmers REALLY collaborate with programmers to write tests? GOSH! Can you tell me a little bit about what that's like?

Thanks in advance.

(Cross-posted on https://groups.google.com/forum/?fromgroups=#!topic/behaviordrivendevelopment/DNvULZP0VUg - sorry if you've got this twice)

[1] http://37signals.com/svn/posts/3159-testing-like-the-tsa

cheers,
Matt

--
Freelance programmer & coach
Author, http://pragprog.com/book/hwcuc/the-cucumber-book
Teacher, http://bddkickstart.com
Founder, http://www.relishapp.com/
Twitter, https://twitter.com/mattwynne

--
-- Rules --

1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.

You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cukes-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org To unsubscribe from this group, send email to cukes+***@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en

Rob Crawford

2012-10-26 15:06:52 UTC

Permalink

Not *direct* "collaboration" in the sense of sitting next to each other and
building them, but our customer handed us a dozen scenarios they'd seen and
described the behavior they expected from the system. I built Cucumber
tests from those scenarios, and when we found one that disagreed with the
signed-off requirements, shared the Cucumber scenario with them. The test
was clear enough they could see what was happening, and how their
expectation differed from the requirements.

Even better -- demonstrating the fix (which we literally just did!), we
were able to replay the scenarios, and show them the behavior they wanted.

Even if you don't have the users spending their time on writing the tests,
Cucumber lets you create tests that the users can comprehend *and are
executable*. The combination is unbeatable, IMHO.

Post by Matt Wynne
A few weeks ago, a well-kown open-source contributor wrote a blog post[1]
discussing some of his predictably controversial attitudes to test-driven
development.

Dont use Cucumber unless you live in the magic kingdom of

non-programmers-writing-tests (and send me a bottle of fairy dust if
youre there!)
Of course he's missed the point slightly: in my magic kingdom
non-programmers don't write tests, they *collaborate with* programmers to
write tests. And then they read them, safe in the knowledge that they're
accurate. But I digress.
The magic kingdom does indeed exist, and I love to spend time there. I bet
you do to. I want the world to know how wonderful our magic kingdom is, and
I need your help. I'd like some real quotes from real people who have
experienced the value of using business-readable Cucumber tests as part of
their development workflow.
Can you tell me a story about your magic kingdom? Do non-programmers
REALLY collaborate with programmers to write tests? GOSH! Can you tell me a
little bit about what that's like?
Thanks in advance.
(Cross-posted on
https://groups.google.com/forum/?fromgroups=#!topic/behaviordrivendevelopment/DNvULZP0VUg -
sorry if you've got this twice)
[1] http://37signals.com/svn/posts/3159-testing-like-the-tsa
cheers,
Matt
--
Freelance programmer & coach
Author, http://pragprog.com/book/hwcuc/the-cucumber-book
Teacher, http://bddkickstart.com
Founder, http://www.relishapp.com/
Twitter, https://twitter.com/mattwynne
--
-- Rules --
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers
http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
You received this message because you are subscribed to the Google Groups
To unsubscribe from this group, send email to
https://groups.google.com/d/forum/cukes?hl=en

George Dinwiddie

2012-10-27 03:05:30 UTC

Permalink

Matt,

Post by Matt Wynne
A few weeks ago, a well-kown open-source contributor wrote a blog
post[1] discussing some of his predictably controversial attitudes to
test-driven development.

Don’t use Cucumber unless you live in the magic kingdom of

non-programmers-writing-tests (and send me a bottle of fairy dust if
you’re there!)
Of course he's missed the point slightly: in my magic kingdom
non-programmers don't write tests, they *collaborate with* programmers
to write tests. And then they read them, safe in the knowledge that
they're accurate. But I digress.
The magic kingdom does indeed exist, and I love to spend time there. I
bet you do to. I want the world to know how wonderful our magic kingdom
is, and I need your help. I'd like some real quotes from real people who
have experienced the value of using business-readable Cucumber tests as
part of their development workflow.
Can you tell me a story about your magic kingdom? Do non-programmers
REALLY collaborate with programmers to write tests? GOSH! Can you tell
me a little bit about what that's like?

Sorry, Matt, but as a consultant and coach, I don't get to live in any
kingdom.

Post by Matt Wynne
Thanks in advance.
(Cross-posted on
https://groups.google.com/forum/?fromgroups=#!topic/behaviordrivendevelopment/DNvULZP0VUg -
sorry if you've got this twice)

I suggest also asking on agile-testing
(http://groups.yahoo.com/group/agile-testing).

Post by Matt Wynne
[1] http://37signals.com/svn/posts/3159-testing-like-the-tsa

Matt Wynne

2012-10-28 20:48:48 UTC

Permalink

George,

Matt,

Post by Matt Wynne
A few weeks ago, a well-kown open-source contributor wrote a blog
post[1] discussing some of his predictably controversial attitudes to
test-driven development.

Dont use Cucumber unless you live in the magic kingdom of

Sorry, Matt, but as a consultant and coach, I don't get to live in any kingdom.

Come on now George, I'm sure you have a tale or two to tell from a visit there though?

Post by Matt Wynne
Thanks in advance.
(Cross-posted on
https://groups.google.com/forum/?fromgroups=#!topic/behaviordrivendevelopment/DNvULZP0VUg -
sorry if you've got this twice)

I suggest also asking on agile-testing (http://groups.yahoo.com/group/agile-testing).

Thanks!

Post by Matt Wynne
[1] http://37signals.com/svn/posts/3159-testing-like-the-tsa

- George
--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------
--
-- Rules --
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.

cheers,
Matt

--
Freelance programmer & coach
Author, http://pragprog.com/book/hwcuc/the-cucumber-book
Teacher, http://bddkickstart.com
Founder, http://www.relishapp.com/
Twitter, https://twitter.com/mattwynne

Joel Byrnes

2012-10-29 02:00:57 UTC

Permalink

I've been bringing Cucumber testing to my team for about a year, as a
software engineer - turned - technical tester. I started out writing
features and then implementing the steps myself, but I knew something was
missing when I did that: the knowledge and the expectations of a business
analyst. I knew they would have different ideas about how we should expect
the system to work, and express that differently, and that would likely
drive a conversation which would enlighten both of us. Most importantly,
just because they would do things differently didn't mean I could also do
it my way by adding extra scenarios.

So I recently got the BAs involved in writing Cucumber scenarios. They used
to write fairly loose acceptance criteria, sometimes organised into a table
of "When I do this/This happens" where the first column was really a
disguised Given/When. Their first attempts at writing features ended up out
of order, very verbose (they didn't know how to use tables, and just
copy/pasted), and as a result were very difficult to read and separate out
what the difference was. This required rework by myself and the developers,
to put them in proper Given/When/Then order, which meant overall it had
taken more time to specify and understand the requirements. And then I had
to distill those, make them more concise Example tables, and implement them
as steps.

So the first attempt didn't go too well, but we learned from that. We
learned for one thing that Given/When/Then is actually not the best or most
concise way to specify requirements in all cases, that the old table might
be better. And the BAs are getting better at writing what is actually
needed.

But more importantly, in one instance, the developers added a test which
they thought was intended behaviour. The BA read this and disagreed. So
there was a discussion and we determined what was actually intended, and
wrote that as a set of scenarios. This required a code change, which
already had tests which the BA could read and agree to. Had this been
written as a Java test, the BA would not have noticed, and it would have
been signed off and released that way, and it would work differently from
how the BA thought it did or intended it to.

Of course, the BAs don't write perfect tests, because they want to describe
how things should work when they do work. They don't always think about
error conditions, what it should do when it fails, or edge cases like empty
or bad input. That's what I and/or the developers are here for. There are
still unit and integration tests under the hood, but we can also add error
conditions to our human-readable behaviour tests, which can test that a
certain job says "FAILED" when it should, and that shows up in this report,
which BAs and customers read. If a customer is going to be seeing the
output, they should be able to read and validate the specification, not
just the rough description which they submitted. Or at least, the BA can do
so on their behalf, and clarify it with them if required.

It would be worth revisiting this in 6 months to see how things are going.
By then we might have progressed to the developers also helping implement
steps as part of the story, making the loop tighter and avoiding test
duplication. I would also like to hear from other teams who have integrated
cucumber into their whole process, not just as a developer tool.

Hope this helps.
Joel

Post by Matt Wynne
A few weeks ago, a well-kown open-source contributor wrote a blog post[1]
discussing some of his predictably controversial attitudes to test-driven
development.

Dont use Cucumber unless you live in the magic kingdom of

Peter R-S

2012-10-29 09:28:24 UTC

Permalink

The last batch of Cucumber tests I worked on were added by the BA to the
feature file, basing the Gherkin on what was already there. I only had to
do two minor changes to the Gherkin, add one or two more step definitions,
and the tests ran.

That the BAs and testers can write the test scenarios has really helped
spread the acceptance of Cucumber in my work place beyond being a cool tool
for developers.

So from that point I do live in a magic kingdom.
Regards,

Peter.

rorygibson

2012-10-29 09:58:46 UTC

Permalink

I work at a UK digital agency - http://www.technophobia.com - and we use
BDD on the majority of our projects now, including large scale public
sector projects with traditionally Waterfall customers (though we're having
a lot of success introducing agile methods)

We work with cross-functional teams of devs, UX, designers, IT, QA and BAs.

- BAs (generally working ahead with the UX guys) write specifications in
Gherkin
- Devs use the specifications, and tend to flesh them out with a couple
more happy-path examples per specification while implementing.
- During the iteration, QA specialists are writing a bunch more examples
to test edge cases, negative paths and so on, running them on local desktop
workstations and QA-controlled, hot-deployed environments

Everything runs in the CI build.

Generally, the specifications being written by BAs and devs go in a
specifications/ folder, and the QA guys read these and write something more
at the level of 'functional tests'; they're stored in a separate
functional-tests/ folder, and they're a little less business focussed and
more "when I click here I see this text".

We find that this gives us a short, high-level set of specifications that
we can (and do) use to communicate with the customer - we use the specs as
acceptance criteria for stories, and link and track everything through JIRA
and FishEye.
And the extra level of detail needed to build a full regression pack (we
work on a lot of big, local- and public-sector projects) lives in the
more-QA-focussed specs.

As an aside, which is probably not pertinent to the DHH-raised questions;
we don't use Cuke itself any more.
We've developed SubSteps [1], a 99% Gherkin-compatible BDD framework that
we believe makes it a bit easier for (particularly) QA to write tests, by
reducing the overhead of step implementation development.
It does this by introducing a second level of parseable Gherkin-ish files
(*.substeps) that provide a flexible DSL that's closer to the browser, and
an abstract Java DSL below that for developing new custom step
implementations.

SubSteps is Free Software (LGPL) and we're working on supporting
infrastructure (we already have a whizzy reporting format, Maven plugin,
ANT task, and a rough cut of an Eclipse plugin).

[1] https://github.com/technophobia/substeps

Matt Wynne

2012-11-08 12:57:00 UTC

Permalink

I work at a UK digital agency - http://www.technophobia.com - and we use BDD on the majority of our projects now, including large scale public sector projects with traditionally Waterfall customers (though we're having a lot of success introducing agile methods)
We work with cross-functional teams of devs, UX, designers, IT, QA and BAs.
- BAs (generally working ahead with the UX guys) write specifications in Gherkin
- Devs use the specifications, and tend to flesh them out with a couple more happy-path examples per specification while implementing.
- During the iteration, QA specialists are writing a bunch more examples to test edge cases, negative paths and so on, running them on local desktop workstations and QA-controlled, hot-deployed environments
Everything runs in the CI build.
Generally, the specifications being written by BAs and devs go in a specifications/ folder, and the QA guys read these and write something more at the level of 'functional tests'; they're stored in a separate functional-tests/ folder, and they're a little less business focussed and more "when I click here I see this text".
We find that this gives us a short, high-level set of specifications that we can (and do) use to communicate with the customer - we use the specs as acceptance criteria for stories, and link and track everything through JIRA and FishEye.
And the extra level of detail needed to build a full regression pack (we work on a lot of big, local- and public-sector projects) lives in the more-QA-focussed specs.
As an aside, which is probably not pertinent to the DHH-raised questions; we don't use Cuke itself any more.
We've developed SubSteps [1], a 99% Gherkin-compatible BDD framework that we believe makes it a bit easier for (particularly) QA to write tests, by reducing the overhead of step implementation development.
It does this by introducing a second level of parseable Gherkin-ish files (*.substeps) that provide a flexible DSL that's closer to the browser, and an abstract Java DSL below that for developing new custom step implementations.
SubSteps is Free Software (LGPL) and we're working on supporting infrastructure (we already have a whizzy reporting format, Maven plugin, ANT task, and a rough cut of an Eclipse plugin).
[1] https://github.com/technophobia/substeps

Thanks everyone for your wonderful replies to this thread.

I've blogged about it here: http://blog.mattwynne.net/2012/11/02/is-cucumber-just-a-scam/

Shall we send DHH a bottle of fairy-dust?

cheers,
Matt

--
Freelance programmer & coach
Author, http://pragprog.com/book/hwcuc/the-cucumber-book
Teacher, http://bddkickstart.com
Founder, http://www.relishapp.com/
Twitter, https://twitter.com/mattwynne

David Kowis

2012-11-08 14:49:40 UTC

Permalink

Received: by 10.50.169.103 with SMTP id ad7mr5220609igc.2.1352386184213;
Thu, 08 Nov 2012 06:49:44 -0800 (PST)
X-BeenThere: cukes-/***@public.gmane.org
Received: by 10.50.194.196 with SMTP id hy4ls17393217igc.0.canary; Thu, 08 Nov
2012 06:49:43 -0800 (PST)
Received: by 10.42.202.196 with SMTP id ff4mr1986403icb.32.1352386183451;
Thu, 08 Nov 2012 06:49:43 -0800 (PST)
Received: by 10.42.202.196 with SMTP id ff4mr1986401icb.32.1352386183438;
Thu, 08 Nov 2012 06:49:43 -0800 (PST)
Received: from dkowis.cymry.org (dkowis.cymry.org. [72.29.99.92])
by gmr-mx.google.com with ESMTP id s9si769736igw.0.2012.11.08.06.49.43;
Thu, 08 Nov 2012 06:49:43 -0800 (PST)
Received-SPF: neutral (google.com: 72.29.99.92 is neither permitted nor denied by best guess record for domain of dkowis-***@public.gmane.org) client-ipr.29.99.92;
Received: from mail.shlrm.org (75-1-82-255.lightspeed.snantx.sbcglobal.net [75.1.82.255])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(Client CN "mail.shlrm.org", Issuer "shlrm.org CA" (verified OK))
by dkowis.cymry.org (Postfix) with ESMTPS id 6A1184C702
for <cukes-/***@public.gmane.org>; Thu, 8 Nov 2012 08:49:42 -0600 (CST)
Received: from localhost (localhost [127.0.0.1])
by mail.shlrm.org (Postfix) with ESMTP id 60F26BD6B
for <cukes-/***@public.gmane.org>; Thu, 8 Nov 2012 08:50:54 -0600 (CST)
X-Virus-Scanned: amavisd-new at shlrm.org
Received: from mail.shlrm.org ([127.0.0.1])
by localhost (kain.shlrm.org [127.0.0.1]) (amavisd-new, port 10024)
with LMTP id 6YxEKnTa2Ba6 for <cukes-/***@public.gmane.org>;
Thu, 8 Nov 2012 08:50:54 -0600 (CST)
Received: from raziel.shlrm.org (unknown [69.20.3.135])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(Client did not present a certificate)
by mail.shlrm.org (Postfix) with ESMTPSA id 02839BC9F
for <cukes-/***@public.gmane.org>; Thu, 8 Nov 2012 08:50:53 -0600 (CST)
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121029 Thunderbird/16.0.2
In-Reply-To: <485FF44C-9D9E-4AC4-9495-5E559F848176-***@public.gmane.org>
X-Enigmail-Version: 1.4.5
X-Original-Sender: dkowis-***@public.gmane.org
X-Original-Authentication-Results: gmr-mx.google.com; spf=neutral (google.com:
72.29.99.92 is neither permitted nor denied by best guess record for domain
of dkowis-***@public.gmane.org) smtp.mail=dkowis-***@public.gmane.org
Precedence: list
Mailing-list: list cukes-/***@public.gmane.org; contact cukes+owners-/***@public.gmane.org
List-ID: <cukes.googlegroups.com>
X-Google-Group-Id: 178380748255
List-Post: <http://groups.google.com/group/cukes/post?hl=en>, <mailto:cukes-/***@public.gmane.org>
List-Help: <http://groups.google.com/support/?hl=en>, <mailto:cukes+help-/***@public.gmane.org>
List-Archive: <http://groups.google.com/group/cukes?hl=en>
Sender: cukes-/***@public.gmane.org
List-Subscribe: <http://groups.google.com/group/cukes/subscribe?hl=en>, <mailto:cukes+subscribe-/***@public.gmane.org>
List-Unsubscribe: <http://groups.google.com/group/cukes/subscribe?hl=en>, <mailto:googlegroups-manage+178380748255+unsubscribe-/***@public.gmane.org>
Archived-At: <http://permalink.gmane.org/gmane.comp.programming.tools.cucumber/12113>

Post by Matt Wynne
Thanks everyone for your wonderful replies to this thread.
I've blogged about it
here: http://blog.mattwynne.net/2012/11/02/is-cucumber-just-a-scam/
Shall we send DHH a bottle of fairy-dust?

ooooh, can we?!? That'd be pretty awesome :)

--
David

worldofchris

2012-11-18 21:49:54 UTC

Permalink

Post by Matt Wynne
Can you tell me a story about your magic kingdom? Do non-programmers

REALLY collaborate with programmers to write tests? GOSH! Can you tell me a
little bit about what that's like?

A couple of years ago, when I was working on what became the YouView Set
Top Box[1] I went along to the BDD Exchange at Skills Matter and saw Gojko
Adzic talk about Specification by Example. I ordered 'Bridging the
Communication Gap' as soon as I got home and thought that the Specification
Workshops described therein would really help me out with the specification
for possibly the most complex part of the box UI, the Digital Video
Recorder (DVR).

I arranged a Spec Workshop with the product development team, members of
BBC R&D, UX designers, developers from all tiers of the stack and testers.
I naively thought we would cover all the ground we needed in two or three
workshops but as we worked through the features and scenarios, capturing
them in Gherkin we discovered a host of knowledge and experience within the
team which revealed the true complexity of what we were trying to do.
Without those Spec Workshops I think we would have really struggled to get
the DVR right.

We did the same for another complex part of the UI, Software Upgrade. In
every workshop we did we would discover either something which was known to
one part of the group and not another or something which contradicted
hitherto held assumptions.

The scenarios that came out of those workshops guided the development of
the software and provided the tests for its QA.

In my new role at Base79[2] I'm again using Specification Workshops, this
time working with our Operations Team to define the behaviour of software
our team is building for them to use with content discovery metadata and
rights ownership management.

[1] http://www.youview.com/
[2] http://www.base79.com/

Roberto Lo Giacco

2012-11-20 17:48:57 UTC

Permalink

Post by Matt Wynne
A few weeks ago, a well-kown open-source contributor wrote a blog post[1]
discussing some of his predictably controversial attitudes to test-driven
development.

Dont use Cucumber unless you live in the magic kingdom of

non-programmers-writing-tests (and send me a bottle of fairy dust if
youre there!)

The funny part is that a few days ago I had a conversation with a colleague
that was stating the same thing to support another tool to write automation
tests, mainly saying the tool he was suggesting was better because it was
for developers, while Cucumber is intended for business people, but "in our
company business people will never write tests!".

Obviously this guy did not understand the value Cucumber provides and does
not understand the important, fundamental, concept Cucumber is not about
writing automation tests but describing acceptance tests, and as such
describing the application itself. I spent two hours without being able to
let him understand the difference and the real value of all that.

Post by Matt Wynne
Of course he's missed the point slightly: in my magic kingdom
non-programmers don't write tests, they *collaborate with* programmers to
write tests. And then they read them, safe in the knowledge that they're
accurate. But I digress.

In my magic world the BAs write down the acceptance criteria and they are
most of the times just a literal transposition from a Word document into a
Gherkin file: fram that point on developers take care of the step
definitions implementation and eventually discover a feature cannot be
delivered the way the business wants to :-D

Post by Matt Wynne
The magic kingdom does indeed exist, and I love to spend time there. I bet
you do to. I want the world to know how wonderful our magic kingdom is, and
I need your help. I'd like some real quotes from real people who have
experienced the value of using business-readable Cucumber tests as part of
their development workflow.
Can you tell me a story about your magic kingdom? Do non-programmers
REALLY collaborate with programmers to write tests? GOSH! Can you tell me a
little bit about what that's like?

The Gherkin part of the work is not done by people with no programming
capabilities in my magic kingdom, not the business partners though, but
they are amazed at the fact they could really change a small piece in a
text file and get a different result! They obviously don't do that because
they should be pulling and pushing into the VCS which is something they do
not understand, but I think that in a not distant time they will came back
with some suggestions just pointing at the feature files we publish for
them :-D

The magic kingdom is the one you build, not the one that is given to you:
the latter is always rubbish, otherwise why do they need you? :D

Post by Matt Wynne
Thanks in advance.
(Cross-posted on
https://groups.google.com/forum/?fromgroups=#!topic/behaviordrivendevelopment/DNvULZP0VUg -
sorry if you've got this twice)
[1] http://37signals.com/svn/posts/3159-testing-like-the-tsa
cheers,
Matt
--
Freelance programmer & coach
Author, http://pragprog.com/book/hwcuc/the-cucumber-book
Teacher, http://bddkickstart.com
Founder, http://www.relishapp.com/
Twitter, https://twitter.com/mattwynne