[Cucumber] Do Not Skip Next Steps On Step Failure in Cucumber

[e***@decathlon.com]

Hello,

First of all, thanks for this wonderful framework.

Step execution results can have two main states: PASSED or FAILED (if we
ignore PENDING, SKIPPED, UNDEFINED, etc.).
I'd like steps to have another middle state: WARN.
This state would threat the step as FAILED but will not skip the execution
of the following steps.

Here is why:
I have such a scenario:

When I go to the category page "CATEGORY ONE" of my website
Then The product "VISIBLE 1" is "Visible"
And The product "VISIBLE 2" is "Visible"
And The product "NOT VISIBLE 1" is "Not visible"
And The product "NOT VISIBLE 2" is "Not visible"

This is one scenario for speed reasons:
All products are (or are not) on the page: we load the category page once.
If we divide the scenario in 4 (with eg. an "Examples:" section), we would
load the category page 4 times (we cannot have a Background section because
the .feature file has more scenarios that don't share that background).

And this scenario is quite simple, but in real life we get more complex
scenarios that navigate through several pages and have additional OPTIONAL
assertions on each page: such assertion failures should be reported as
errors, but the test must continue in order to spot all assertion failures,
and not stop at the first failure and hide other potential failures...

Is it possible for a future version of Cucumber to add such FAILED step
that continues to execute the next steps?
So that an optional assert failure will not hide other failures by not
executing their steps.

This could be done with an annotation like that:

@WarningInsteadOfFailure
@Then("^The product " + PLACEHOLDER_STRING + " is " +
PLACEHOLDER_STRING + "$")
public void the_product_is(String productName, Visibility visibility) {
assertThat(page(CategoryPage.class).getProduct(productName) !=
null).isEqualTo(visibility.isVisible());
}

Or this could be more fine-grained:
Instead of a @WarningInsteadOfFailure annotation, we could request only
specific exceptions to be treated as warnings:

@WarningInsteadOfFailure({AssertionError.class,
NoSuchElementException.class})

What do you think about this proposal?

Have a nice day,
Sébastien Laoût.

[George Dinwiddie]

Sébastien,

Post by e***@decathlon.com
Hello,
First of all, thanks for this wonderful framework.
Step execution results can have two main states: PASSED or FAILED (if we
ignore PENDING, SKIPPED, UNDEFINED, etc.).
I'd like steps to have another middle state: WARN.
This state would threat the step as FAILED but will not skip the
execution of the following steps.
When I go to the category page "CATEGORY ONE" of my website
Then The product "VISIBLE 1" is "Visible"
And The product "VISIBLE 2" is "Visible"
And The product "NOT VISIBLE 1" is "Not visible"
And The product "NOT VISIBLE 2" is "Not visible"
All products are (or are not) on the page: we load the category page once.
If we divide the scenario in 4 (with eg. an "Examples:" section), we
would load the category page 4 times (we cannot have a Background
section because the .feature file has more scenarios that don't share
that background).

Note that Background would execute for each scenario, also. Does it take
a long time for the page to load? Have you considered testing at an API
below the GUI?

Have you considered a data table?

When I go to the category page "CATEGORY ONE" of my website
Then the page reflects the following:
| VISIBLE 1 | Visible |
| VISIBLE 2 | Visible |
| NOT VISIBLE 1 | Not visible |
| NOT VISIBLE 2 | Not visible |

Post by e***@decathlon.com
And this scenario is quite simple, but in real life we get more complex
scenarios that navigate through several pages and have additional
OPTIONAL assertions on each page: such assertion failures should be
reported as errors, but the test must continue in order to spot all
assertion failures, and not stop at the first failure and hide other
potential failures...

Automated verification works better with separate, little scenarios.
It's very different from manual verification where you're trying to save
human time.

Post by e***@decathlon.com
Is it possible for a future version of Cucumber to add such FAILED step
that continues to execute the next steps?
So that an optional assert failure will not hide other failures by not
executing their steps.
@WarningInsteadOfFailure
@Then("^The product " + PLACEHOLDER_STRING + " is " +
PLACEHOLDER_STRING + "$")
public void the_product_is(String productName, Visibility visibility) {
assertThat(page(CategoryPage.class).getProduct(productName) !=
null).isEqualTo(visibility.isVisible());
}
@WarningInsteadOfFailure({AssertionError.class,
NoSuchElementException.class})
What do you think about this proposal?

I wonder if you're using Cucumber as a testing tool. What is the process
that results in the scenarios?

- George
--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[e***@decathlon.com]

Hello,

I agree this scenario is perhapse not the best one to show my need.
In real life, I have several such scenarios, and they are a little bit more
complicated.
But I wanted to keep the sample short here, to not submerge you with more
or less relevant details.
And yes, I have some tests that use scenario-outlines but also need to
check several assertions at once.

In our company, we have another team who is using Cucumber.
Theire solution to the problem is to currentScenario.write("WARNING"), mark
the step as PASSED, and interpret this in their custom report builder to
show a warning.
But in JUnit (or any runner that don't use their report builder),
everything is green and good: I don't want such a wobbly solution: JUnit is
of a good help when developing and debugging tests in Eclipse.

Yes, we are also testing underlying APIs.
Our company is commited to have several types of tests:
- Human test explorations
- Automated service integration tests: test APIs with FitNess
- Automated regression tests: test the user interface with Selenium, like a
real user would see it
We have a big database of human tests already there in a tool.
We pinpointed a subset of critical "Sanity Check" tests that we need to
automate.
The goal is to achieve continuous integration and run the tests several
times per day, ideally after each commit on Git.

So, execution need to be fast in order for the system to be adopted by
developers team.
And the tool needs to map well to existing sanity check tests to be adopted
by the QA team...
We adapted some tests to better map to Cucumber spirit.
But the "several checks needs to be performed at once" is a recurrent
problem we currently face and can't adapt to Cucumber spirit without a big
multiplication of the number of scenarios, and then a big duplication of
them, and then a big execution time increase...

I'm aware a lot of other people already asked you to tread failures as
temporar successes or such bizarre things.
But, IMHO, I think the proposal of a
@WarningInsteadOfFailure({AssertionError.class}) annotation (in
cucumber-JVM, at least, I don't know how to do it in Ruby or...) would keep
the spirit of Cucumber intact (again, it's only my opinion, I don't have
the big picture in minde like you): failures are failures.
It's about the same working as Hibernate's @Transactional(noRollbackFor =
{MailException.class})...

Or an alternative to the annotation would be the step definition to throw a
WarningException... It would work weel with Ruby, I suppose.
Or a better name could be OptionalAssertFailureException: as the step would
be FAILED (and not WARNING) but this failure being optional means it can
continue the next steps.

I think the annotation or WarningException proposal would somewhat reply to
what the following users wanted (not exactly what they wanted, but in a
more correct way, or at least a less hacky way):

1)
http://stackoverflow.com/questions/6841467/how-can-i-make-cucumber-run-all-the-steps-not-skip-them-even-if-one-of-them-fa
=> Their solution is wobbly (catch exception, log it but the step won't
throw any exception):
Quote: "This makes the overall scenario fail, but it masks the step failure
from Cucumber, so all the step results are green, including the ones that
aren't right."

2)
http://stackoverflow.com/questions/15298521/continue-running-cucumber-steps-after-a-failure
=> Theire solution is also wobbly (log the exception in steps, don't throw
any exception in steps, but throw it in @After):
Quote: "The only issue would be that, in the report, all the steps would
look green but the test case looks failed." That's exactly why you don't
want to do this.

3) I've also read another thread on that forum, the guy asked to mark some
errors as PASSED because they are known and will be fixed in next release
(but I don't find back this thread right now).
If we use @WarningInsteadOfFailure AND we add another state (WARNING), this
guy could use that mecanism to achieve somewhat what he wanted, in a more
strict way.

From what I see in the Cucumber-JVM code source, the
Runtime.runStep(String, Step, Reporter, I18n) method sets skipNextStep to
true here:
try {
match.runStep(i18n);
} catch (Throwable t) {
error = t;
status = isPending(t) ? "pending" : Result.FAILED;
addError(t);
skipNextStep = true; // <= HERE!
}
I think it could work if it's set to true only if the exception is not part
of the @WarningInsteadOfFailure annotation (or is not
WarningException.class).
I have not tested that assumption, tough.

Kind regards,
Sébastien Laoût.

[George Dinwiddie]

Sébastien

Post by e***@decathlon.com
Hello,
I agree this scenario is perhapse not the best one to show my need.
In real life, I have several such scenarios, and they are a little bit
more complicated.
But I wanted to keep the sample short here, to not submerge you with
more or less relevant details.
And yes, I have some tests that use scenario-outlines but also need to
check several assertions at once.

Perhaps you didn't read my post carefully. I suggested using a data
table and letting the step definition check all of the assertions in one
go. It can report which of them pass and which of them fail, and then
fail the test if any of them fail.

Post by e***@decathlon.com
In our company, we have another team who is using Cucumber.
Theire solution to the problem is to currentScenario.write("WARNING"),
mark the step as PASSED, and interpret this in their custom report
builder to show a warning.
But in JUnit (or any runner that don't use their report builder),
everything is green and good: I don't want such a wobbly solution: JUnit
is of a good help when developing and debugging tests in Eclipse.

I would never want to return success for a failure and depend on human
eyes to notice the difference.

Post by e***@decathlon.com
Yes, we are also testing underlying APIs.
- Human test explorations
- Automated service integration tests: test APIs with FitNess
- Automated regression tests: test the user interface with Selenium,
like a real user would see it
We have a big database of human tests already there in a tool.
We pinpointed a subset of critical "Sanity Check" tests that we need to
automate.
The goal is to achieve continuous integration and run the tests several
times per day, ideally after each commit on Git.
So, execution need to be fast in order for the system to be adopted by
developers team.
And the tool needs to map well to existing sanity check tests to be
adopted by the QA team...
We adapted some tests to better map to Cucumber spirit.
But the "several checks needs to be performed at once" is a recurrent
problem we currently face and can't adapt to Cucumber spirit without a
big multiplication of the number of scenarios, and then a big
duplication of them, and then a big execution time increase...

To me, it's easy enough to specify all the things you want to see and
not see on a page in a data table. I'm not completely convinced of the
value of such, but sometimes it's helpful.

Proceeding on to subsequent steps sounds like a bad idea to me.

- George

Post by e***@decathlon.com
I'm aware a lot of other people already asked you to tread failures as
temporar successes or such bizarre things.
But, IMHO, I think the proposal of a
@WarningInsteadOfFailure({AssertionError.class}) annotation (in
cucumber-JVM, at least, I don't know how to do it in Ruby or...) would
keep the spirit of Cucumber intact (again, it's only my opinion, I don't
have the big picture in minde like you): failures are failures.
= {MailException.class})...
Or an alternative to the annotation would be the step definition to
throw a WarningException... It would work weel with Ruby, I suppose.
Or a better name could be OptionalAssertFailureException: as the step
would be FAILED (and not WARNING) but this failure being optional means
it can continue the next steps.
I think the annotation or WarningException proposal would somewhat reply
to what the following users wanted (not exactly what they wanted, but in
1)
http://stackoverflow.com/questions/6841467/how-can-i-make-cucumber-run-all-the-steps-not-skip-them-even-if-one-of-them-fa
=> Their solution is wobbly (catch exception, log it but the step won't
Quote: "This makes the overall scenario fail, but it masks the step
failure from Cucumber, so all the step results are green, including the
ones that aren't right."
2)
http://stackoverflow.com/questions/15298521/continue-running-cucumber-steps-after-a-failure
=> Theire solution is also wobbly (log the exception in steps, don't
Quote: "The only issue would be that, in the report, all the steps would
look green but the test case looks failed." That's exactly why you don't
want to do this.
3) I've also read another thread on that forum, the guy asked to mark
some errors as PASSED because they are known and will be fixed in next
release (but I don't find back this thread right now).
this guy could use that mecanism to achieve somewhat what he wanted, in
a more strict way.
From what I see in the Cucumber-JVM code source, the
Runtime.runStep(String, Step, Reporter, I18n) method sets skipNextStep
try {
match.runStep(i18n);
} catch (Throwable t) {
error = t;
status = isPending(t) ? "pending" : Result.FAILED;
addError(t);
skipNextStep = true; // <= HERE!
}
I think it could work if it's set to true only if the exception is not
WarningException.class).
I have not tested that assumption, tough.
Kind regards,
Sébastien Laoût.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[Andrew Premdas]

Post by e***@decathlon.com
Hello,
First of all, thanks for this wonderful framework.
Step execution results can have two main states: PASSED or FAILED (if we
ignore PENDING, SKIPPED, UNDEFINED, etc.).
I'd like steps to have another middle state: WARN.
This state would threat the step as FAILED but will not skip the execution
of the following steps.
When I go to the category page "CATEGORY ONE" of my website
Then The product "VISIBLE 1" is "Visible"
And The product "VISIBLE 2" is "Visible"
And The product "NOT VISIBLE 1" is "Not visible"
And The product "NOT VISIBLE 2" is "Not visible"

For speed, simplicity, and maintenance I would write this scenario with a
single Then e.g.

Then the correct products should be displayed

And then implement that in a way that eloquently expresses the idea that
some products should be there and some shouldn't e.g

Then "the hidden products should hidden and visible products should be
visible" do
should_be_visible @visible_products
should_be_hidden @hidden_products
end

I'd further improve the above by extracting into a helper method.

Cucumber does not need extra programming abilities for features. I'd argue
it has far to many of these already. Perhaps this contributes too so many
people trying to program in Gherkin.

Use language and abstraction to write scenarios that are elegant, concise
and not littered with detail.

All best

Andrew

Post by e***@decathlon.com
All products are (or are not) on the page: we load the category page once.
If we divide the scenario in 4 (with eg. an "Examples:" section), we would
load the category page 4 times (we cannot have a Background section because
the .feature file has more scenarios that don't share that background).
And this scenario is quite simple, but in real life we get more complex
scenarios that navigate through several pages and have additional OPTIONAL
assertions on each page: such assertion failures should be reported as
errors, but the test must continue in order to spot all assertion failures,
and not stop at the first failure and hide other potential failures...
Is it possible for a future version of Cucumber to add such FAILED step
that continues to execute the next steps?
So that an optional assert failure will not hide other failures by not
executing their steps.
@WarningInsteadOfFailure
@Then("^The product " + PLACEHOLDER_STRING + " is " +
PLACEHOLDER_STRING + "$")
public void the_product_is(String productName, Visibility visibility) {
assertThat(page(CategoryPage.class).getProduct(productName) !=
null).isEqualTo(visibility.isVisible());
}
@WarningInsteadOfFailure({AssertionError.class,
NoSuchElementException.class})
What do you think about this proposal?
Have a nice day,
Sébastien Laoût.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
------------------------
Andrew Premdas
blog.andrew.premdas.org

[Andrew Premdas]

sry typos in above , corrections follow

Then the correct products should be displayed AND

Then "the hidden products should hidden and visible products should be
visible"

SHOULD BE

Then hidden products are hidden and visible products are visible

Post by Andrew Premdas

Post by e***@decathlon.com
Hello,
First of all, thanks for this wonderful framework.
Step execution results can have two main states: PASSED or FAILED (if we
ignore PENDING, SKIPPED, UNDEFINED, etc.).
I'd like steps to have another middle state: WARN.
This state would threat the step as FAILED but will not skip the
execution of the following steps.
When I go to the category page "CATEGORY ONE" of my website
Then The product "VISIBLE 1" is "Visible"
And The product "VISIBLE 2" is "Visible"
And The product "NOT VISIBLE 1" is "Not visible"
And The product "NOT VISIBLE 2" is "Not visible"

For speed, simplicity, and maintenance I would write this scenario with a
single Then e.g.
Then the correct products should be displayed

Then hidden products are hidden and visible products are visible

Post by Andrew Premdas
And then implement that in a way that eloquently expresses the idea that
some products should be there and some shouldn't e.g
Then "the hidden products should hidden and visible products should be
visible" do

Then "hidden products are hidden and visible products are visible" do

Post by Andrew Premdas
end
I'd further improve the above by extracting into a helper method.
Cucumber does not need extra programming abilities for features. I'd argue
it has far to many of these already. Perhaps this contributes too so many
people trying to program in Gherkin.
Use language and abstraction to write scenarios that are elegant, concise
and not littered with detail.
All best
Andrew

Post by e***@decathlon.com
All products are (or are not) on the page: we load the category page once.
If we divide the scenario in 4 (with eg. an "Examples:" section), we
would load the category page 4 times (we cannot have a Background section
because the .feature file has more scenarios that don't share that
background).
And this scenario is quite simple, but in real life we get more complex
scenarios that navigate through several pages and have additional OPTIONAL
assertions on each page: such assertion failures should be reported as
errors, but the test must continue in order to spot all assertion failures,
and not stop at the first failure and hide other potential failures...
Is it possible for a future version of Cucumber to add such FAILED step
that continues to execute the next steps?
So that an optional assert failure will not hide other failures by not
executing their steps.
@WarningInsteadOfFailure
@Then("^The product " + PLACEHOLDER_STRING + " is " +
PLACEHOLDER_STRING + "$")
public void the_product_is(String productName, Visibility visibility) {
assertThat(page(CategoryPage.class).getProduct(productName) !=
null).isEqualTo(visibility.isVisible());
}
@WarningInsteadOfFailure({AssertionError.class,
NoSuchElementException.class})
What do you think about this proposal?
Have a nice day,
Sébastien Laoût.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
------------------------
Andrew Premdas
blog.andrew.premdas.org

--
------------------------
Andrew Premdas
blog.andrew.premdas.org

[Matt Wynne]

I understand why you need this.

I can imagine that, rather than adding a "WARN" status, we simply allow the
user to choose (either per scenario or per run) whether we should skip
remaining steps or carry on when a step fails. I can see how we could
implement this, at least in the Ruby version.

We'd still fail the scenario if any of the steps failed, it's just that
we'd give you the chance to see whether any of the other steps failed too.
I'm not against this.

On Thursday, January 28, 2016 at 7:12:59 AM UTC-5,

Post by e***@decathlon.com
Hello,
First of all, thanks for this wonderful framework.
Step execution results can have two main states: PASSED or FAILED (if we
ignore PENDING, SKIPPED, UNDEFINED, etc.).
I'd like steps to have another middle state: WARN.
This state would threat the step as FAILED but will not skip the execution
of the following steps.
When I go to the category page "CATEGORY ONE" of my website
Then The product "VISIBLE 1" is "Visible"
And The product "VISIBLE 2" is "Visible"
And The product "NOT VISIBLE 1" is "Not visible"
And The product "NOT VISIBLE 2" is "Not visible"
All products are (or are not) on the page: we load the category page once.
If we divide the scenario in 4 (with eg. an "Examples:" section), we would
load the category page 4 times (we cannot have a Background section because
the .feature file has more scenarios that don't share that background).
And this scenario is quite simple, but in real life we get more complex
scenarios that navigate through several pages and have additional OPTIONAL
assertions on each page: such assertion failures should be reported as
errors, but the test must continue in order to spot all assertion failures,
and not stop at the first failure and hide other potential failures...
Is it possible for a future version of Cucumber to add such FAILED step
that continues to execute the next steps?
So that an optional assert failure will not hide other failures by not
executing their steps.
@WarningInsteadOfFailure
@Then("^The product " + PLACEHOLDER_STRING + " is " +
PLACEHOLDER_STRING + "$")
public void the_product_is(String productName, Visibility visibility) {
assertThat(page(CategoryPage.class).getProduct(productName) !=
null).isEqualTo(visibility.isVisible());
}
@WarningInsteadOfFailure({AssertionError.class,
NoSuchElementException.class})
What do you think about this proposal?
Have a nice day,
Sébastien Laoût.

--
The Cucumber logo is the intellectual property of Cucumber Ltd, a limited
company registered in Scotland, number 456793.

UK Headquarters: Cucumber Ltd, Drumsyniebeg, Lochgoilhead, Cairndow,
Argyll, PA24 8AN UK.

CONFIDENTIALITY NOTICE: The information in this e-mail is confidential and
privileged; it is intended for use solely by the individual or entity named
as the recipient hereof. Disclosure, copying, distribution, or use of the
contents of this e-mail by persons other than the intended recipient is
strictly prohibited and may violate applicable laws. If you have received
this e-mail in error, please delete the original message and notify us by
email immediately. Thank you. Cucumber Ltd.

[e***@decathlon.com]

@Matt Wynne:
Oh, I should have refreshed the page before replying.

Yes, that's exactly what I mean.

You said it in a (lot) more succinct way ;-)

[e***@decathlon.com]

@apremdas:
Ah, that's a good idea.
Do you mean that:

Given These products should be hidden:
| NOT VISIBLE 1 |
| NOT VISIBLE 2 |
And These products should be visible:
| VISIBLE 1 |
| VISIBLE 2 |
When I go to the category page "CATEGORY ONE" of my website
Then Then hidden products are hidden and visible products are visible

That solve one of my use cases.
Thanks.

Though, this needs some work in step definitions in order to create a human
readable message aggregating all failures: "The products {1} are visible
but should be hidden and the products {2} are hidden visible but should be
visible".
With one step per product, there is the elegant solution that each failed
step appears red at the first time, and we do not need to construct a
unique exception message aggregating all failed steps (here, all failed
product visibilities).

That solve one of my use cases, but the problem remains for other use cases
where, for instance, a lot of pages need to be accessed on the same
scenario without being able to break it into several scenarios.
For instance, on an e-commerce website, adding a product to cart, going to
the checkout process, and paying: we cannot break this scenario into 3
scenarios, and each one needs "optional" checks that don't impede other
steps ability to succeed, but imply the scenario as a whole would fail.

@Matt Wynne
Abandoning the WARNING state, the annotation could rather be named
@ContinueNextStepsFor(Throwable[] t)
Or the exception to be thrown by step definitions could be named
ContinueNextStepsException(Throwable t)

[Andrew Premdas]

Post by e***@decathlon.com
Ah, that's a good idea.
| NOT VISIBLE 1 |
| NOT VISIBLE 2 |
| VISIBLE 1 |
| VISIBLE 2 |
When I go to the category page "CATEGORY ONE" of my website
Then Then hidden products are hidden and visible products are visible

Yes, though I would make it even simpler. This is the essence of writing
good scenarios, refining the language until you have the just enough of the
essence of the problem without all the froth of detail. So perhaps:

Given a category with hidden and visible products
When I view the category
Then I should see the visible products
And I should not see the hidden products

would be an improvement.

Post by e***@decathlon.com
That solve one of my use cases.
Thanks.
Though, this needs some work in step definitions in order to create a
human readable message aggregating all failures: "The products {1} are
visible but should be hidden and the products {2} are hidden visible but
should be visible".
With one step per product, there is the elegant solution that each failed
step appears red at the first time, and we do not need to construct a
unique exception message aggregating all failed steps (here, all failed
product visibilities).

Yes thats the point. The place to do this sort of work is in the step
definitions (or better yet in support code that the step definitions use).
If you use cucumber in this way you can do just about anything, because the
step definitions and below are using a programming language. Another
benefit of this approach is that your scenarios become much simpler to
write and maintain.

Post by e***@decathlon.com
That solve one of my use cases, but the problem remains for other use
cases where, for instance, a lot of pages need to be accessed on the same
scenario without being able to break it into several scenarios.
For instance, on an e-commerce website, adding a product to cart, going to
the checkout process, and paying: we cannot break this scenario into 3
scenarios, and each one needs "optional" checks that don't impede other
steps ability to succeed, but imply the scenario as a whole would fail.

Actually you can break any process into seperate scenarios very easily.
With your ecommerce example we would have

Scenario: Adding a product to the cart
Given I am viewing a product
When I add the product to the cart
Then the cart should contain the product

Scenario: Going to the checkout
Given I have added a product to the cart
When I go to the checkout
Then I should see the product
...

Scenario: Paying
Given i have a checkout with products
When I pay
...

Each scenario build on what is done previously. This isn't seperate in the
sense that paying doesn't include product selection and adding to the cart,
but it is seperate in the sense that you can add the product to the cart
without having to have the payment code. Cucumber is designed to drive
development (rather than follow development), and when you use it this way
its alot easier to break scenarios down into smaller components. In your
case this is a harder thing to do, but its not impossible, you just have to
examine whats there and find the boundaries between things (in this case
viewing, choosing, and paying - or perhaps product, cart, payment)

All best

Andrew

Post by e***@decathlon.com
@Matt Wynne
Abandoning the WARNING state, the annotation could rather be named
@ContinueNextStepsFor(Throwable[] t)
Or the exception to be thrown by step definitions could be named
ContinueNextStepsException(Throwable t)
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
------------------------
Andrew Premdas
blog.andrew.premdas.org

[e***@decathlon.com]

Hello,

After a lot of discussion in our team, we reworked some scenarios, but not
all of them were reworkable.
We decided we really needed the @ContinueNextStepsFor annotation.

So I've made a fork of Cucumber, and we're now using it.
The fork is available here:
https://github.com/slaout/cucumber-jvm/tree/continue-next-steps-for-exceptions-1.2.4
Its whole purpose is to offer the @ContinueNextStepsFor(Class<? extends
Throwable>[]) annotation.

The README.md explains how to use it (published on Open Source Project
Repository Hosting).
In this README, I also added a lot of links to explain why we've forked.
And the first links point to your explanations on how we're supposed to use
Cucumber, to warn people who want to use the fork, so they use it if really
needed.

@Matt Wynne: you were not totally opposed to this: what do you think about
it?
I've implemented it only in Java, for our own purpose.
I don't know how to do it in other languages like Ruby...
In addition to the annotation, step definitions could ALSO throw a
Cucumber's ContinueNextStepsException that would implicitely have the same
effect as the annotation.

Have a nice day,
Sébastien Laoût.

[e***@decathlon.com]

Based on our current use of the fork in our project, I added a section
"Recommendations for Using the Annotation" in the README.md.
It aggregates some best practices we found by using this "super-power".
It's only a few days old, so this will evolve.
This will allow you to get our feedback for using the feature in the wild,
so you can decide it's definitely not going into Cucumber, or it could be
integrated in a very light mode that ensure all the best practices are
applied.

[dhru]

On Wednesday, 10 February 2016 02:02:17 UTC-8, External Sébastien LAOUT

Post by e***@decathlon.com
Hello,
After a lot of discussion in our team, we reworked some scenarios, but not
all of them were reworkable.
So I've made a fork of Cucumber, and we're now using it.
https://github.com/slaout/cucumber-jvm/tree/continue-next-steps-for-exceptions-1.2.4
Throwable>[]) annotation.
The README.md explains how to use it (published on Open Source Project
Repository Hosting).
In this README, I also added a lot of links to explain why we've forked.
And the first links point to your explanations on how we're supposed to
use Cucumber, to warn people who want to use the fork, so they use it if
really needed.
@Matt Wynne: you were not totally opposed to this: what do you think about
it?
I've implemented it only in Java, for our own purpose.
I don't know how to do it in other languages like Ruby...
In addition to the annotation, step definitions could ALSO throw a
Cucumber's ContinueNextStepsException that would implicitely have the same
effect as the annotation.
Have a nice day,
Sébastien Laoût.

Hello sebastien, I have tried this project and
@ContinueNextStepsFor(AssertionError.class) this it is not working!!!! I
have followed the instructions by adding dependencies listed and I have
added ***@ContinueNextStepsFor(AssertionError.class) and my tests are being
skipped...as there is no error I am not able to troubleshoot...Please
help...

[e***@decathlon.com]

With this:

Given a category with hidden and visible products

Post by Andrew Premdas
When I view the category
Then I should see the visible products
And I should not see the hidden products

I see how Cucumber should be used.

But:

- All details are hidden in the Java code: they are so abstract that
business people don't understand all what's going on: what are the visible
and hidden products that are checks? They can't tell if they don't have
access to the Java code. And as they are not developer, we do not give them
access to the Java code. We use Cucumber as a discussion layer between
business people and technical people: it's the missing link :-) So we don't
want to hide too much data deep into Java, so discussion can happen
without misunderstanding.

- In fact, a few dozen existing test cases are based on Excel files:
datasets have carefully be prepared by QA team (not developer people) to
test all cases. When automating these Excel-based tests, I use scenario
outline (or repeated steps if it helps), and I can easily transform their
Excel file into Examples section (or suite of steps). When a "line" (or
step) fails, I can easily tell QA people what happened, and where, by
annotating their Excel file, or giving them the Cucumber results they can
easily map to their original Excel file (the Excel file thus can be deleted
and Cucumber scenarios become the reference).

- Step definition sentences are short and without a lot of context: it's
easier to create duplicate sentences: "I view the category" can mean a lot
of different things or a lot of different categories in all scenarios.

- Your two "Then" and "And" last sentences imply that both are executed
even if the first fails, so I sense you also secretly dream of a a
@ContinueNextStepsFor(Throwable) annotation ;-) Come on, it's only a few
lines of code to add to Cucumber :-) OK... and a few days to test and adapt
to other languages...

As for breaking processes into separate scenarios, that's a great idea.
I'll think about it for our own project, to see if that's possible.
I perhaps missed something in the Cucumber documentation, but all these
tips could be a good documentation topic on how Cucumber is opinionated and
how to use it in the best way, with a variety of samples like the ones in
that thread, and the ones in other threads of this forum.

[Justin C]

@MattWynne

If my team were to embark on a PR to accomplish this for cucumber-ruby,
would there be love for it on the cucumber org side? I quite like the
implementation route that was chosen for cucumber-js by Sébastien Laoût,
this is similar to how Thoughtworks solved the problem for their 'Gauge'
framework.

On Thursday, January 28, 2016 at 7:12:59 AM UTC-5,

Post by e***@decathlon.com
Hello,
First of all, thanks for this wonderful framework.
Step execution results can have two main states: PASSED or FAILED (if we
ignore PENDING, SKIPPED, UNDEFINED, etc.).
I'd like steps to have another middle state: WARN.
This state would threat the step as FAILED but will not skip the execution
of the following steps.
When I go to the category page "CATEGORY ONE" of my website
Then The product "VISIBLE 1" is "Visible"
And The product "VISIBLE 2" is "Visible"
And The product "NOT VISIBLE 1" is "Not visible"
And The product "NOT VISIBLE 2" is "Not visible"
All products are (or are not) on the page: we load the category page once.
If we divide the scenario in 4 (with eg. an "Examples:" section), we would
load the category page 4 times (we cannot have a Background section because
the .feature file has more scenarios that don't share that background).
And this scenario is quite simple, but in real life we get more complex
scenarios that navigate through several pages and have additional OPTIONAL
assertions on each page: such assertion failures should be reported as
errors, but the test must continue in order to spot all assertion failures,
and not stop at the first failure and hide other potential failures...
Is it possible for a future version of Cucumber to add such FAILED step
that continues to execute the next steps?
So that an optional assert failure will not hide other failures by not
executing their steps.
@WarningInsteadOfFailure
@Then("^The product " + PLACEHOLDER_STRING + " is " +
PLACEHOLDER_STRING + "$")
public void the_product_is(String productName, Visibility visibility) {
assertThat(page(CategoryPage.class).getProduct(productName) !=
null).isEqualTo(visibility.isVisible());
}
@WarningInsteadOfFailure({AssertionError.class,
NoSuchElementException.class})
What do you think about this proposal?
Have a nice day,
Sébastien Laoût.

[Matt Wynne]

Hi Justin,

Sorry for the lag!

Yes, I’d be happy to support someone exploring this in a PR, I can understand why you might need it sometimes.

In the Ruby version, changes would need to be concentrated on the state machine in https://github.com/cucumber/cucumber-ruby-core/blob/master/lib/cucumber/core/test/runner.rb <https://github.com/cucumber/cucumber-ruby-core/blob/master/lib/cucumber/core/test/runner.rb>

Ping me on bitter if you need any more support with it.

Post by Justin C
@MattWynne
If my team were to embark on a PR to accomplish this for cucumber-ruby, would there be love for it on the cucumber org side? I quite like the implementation route that was chosen for cucumber-js by Sébastien Laoût, this is similar to how Thoughtworks solved the problem for their 'Gauge' framework.
Hello,
First of all, thanks for this wonderful framework.
Step execution results can have two main states: PASSED or FAILED (if we ignore PENDING, SKIPPED, UNDEFINED, etc.).
I'd like steps to have another middle state: WARN.
This state would threat the step as FAILED but will not skip the execution of the following steps.
When I go to the category page "CATEGORY ONE" of my website
Then The product "VISIBLE 1" is "Visible"
And The product "VISIBLE 2" is "Visible"
And The product "NOT VISIBLE 1" is "Not visible"
And The product "NOT VISIBLE 2" is "Not visible"
All products are (or are not) on the page: we load the category page once.
If we divide the scenario in 4 (with eg. an "Examples:" section), we would load the category page 4 times (we cannot have a Background section because the .feature file has more scenarios that don't share that background).
And this scenario is quite simple, but in real life we get more complex scenarios that navigate through several pages and have additional OPTIONAL assertions on each page: such assertion failures should be reported as errors, but the test must continue in order to spot all assertion failures, and not stop at the first failure and hide other potential failures...
Is it possible for a future version of Cucumber to add such FAILED step that continues to execute the next steps?
So that an optional assert failure will not hide other failures by not executing their steps.
@WarningInsteadOfFailure
@Then("^The product " + PLACEHOLDER_STRING + " is " + PLACEHOLDER_STRING + "$")
public void the_product_is(String productName, Visibility visibility) {
assertThat(page(CategoryPage.class).getProduct(productName) != null).isEqualTo(visibility.isVisible());
}
@WarningInsteadOfFailure({AssertionError.class, NoSuchElementException.class})
What do you think about this proposal?
Have a nice day,
Sébastien Laoût.
--
Posting rules: http://cukes.info/posting-rules.html <http://cukes.info/posting-rules.html>
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/xTqSyR1qvSc/unsubscribe <https://groups.google.com/d/topic/cukes/xTqSyR1qvSc/unsubscribe>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

cheers,
Matt

—
[phone]
***@cucumber.io
https://cucumber.io
https://twitter.com/mattwynne
--
The Cucumber logo is the intellectual property of Cucumber Ltd, a limited
company registered in Scotland, number 456793.

UK Headquarters: Cucumber Ltd, Drumsyniebeg, Lochgoilhead, Cairndow,
Argyll, PA24 8AN UK.

CONFIDENTIALITY NOTICE: The information in this e-mail is confidential and
privileged; it is intended for use solely by the individual or entity named
as the recipient hereof. Disclosure, copying, distribution, or use of the
contents of this e-mail by persons other than the intended recipient is
strictly prohibited and may violate applicable laws. If you have received
this e-mail in error, please delete the original message and notify us by
email immediately. Thank you. Cucumber Ltd.

[Aslak Hellesøy]

I'm open to this feature request, but let's lift the discussion out of cucumber-ruby into the parent project. Please open an issue on github.com/cucumber/cucumber where we can discuss this.

I'm in charge of making sure Cucumber implementations in different languages converge towards similar behaviour. We should avoid adding new functionality to only one Cucumber implementation.

Aslak

[e***@decathlon.com]

Thanks very much.

I've opened this issue: https://github.com/cucumber/cucumber/issues/79

[e***@decathlon.com]

Hello,

Thanks Justin.
I didn't know about Gauge.
After some research, this project looks promising.

I found what you told us about:
http://getgauge.io/documentation/user/current/advanced_readings/continue_on_failure.html
It's a simple @ContinueOnFailure, without exception class parameters.
I'm OK with it too.

I realize my last message was "It's only a few days old, so this will
evolve.".

Now, it's a few months old, and in fact, it didn't evolve.
We still use this annotation extensively, and only use it with these two
exception classes: @ContinueNextStepsFor({AssertionError.class,
NoSuchElementException.class })
It's simpler for developers to be able to copy/paste the same thing without
thinking.
And when another exception is thrown, it's practically always because of a
critical bug that needs fixing.

Best regards,
Sébastien Laoût.

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

Hi dhru,

I put this:

<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-core</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-java</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-spring</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-junit</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>

Does it work with these dependencies?

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

What do you mean by your tests are being skipped?
The scenarios do not execute?
An exception on the given steps make the scenario stop execute?

Le lundi 27 mars 2017 15:54:52 UTC+2, External Sébastien LAOUT -BTWIN

Post by External Sébastien LAOUT -BTWIN VILLAGE LILLE-
Hi dhru,
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-core</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-java</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-spring</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-junit</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
Does it work with these dependencies?

[Swathi Reddy]

Hi,

Is the spring depedency required for the implementaion to work.
When i added spring depedency it needed other spring librabries to compile.

And without the spring depedency , my tests on Assetion error with this
@ContinueNextSteps({Assertion.class}) still stop execution
and the rest of the tests are skipped.

Its not working as intended.Can you please help?


On Monday, March 27, 2017 at 7:26:33 PM UTC+5:30, External Sébastien LAOUT

Post by External Sébastien LAOUT -BTWIN VILLAGE LILLE-
What do you mean by your tests are being skipped?
The scenarios do not execute?
An exception on the given steps make the scenario stop execute?
Le lundi 27 mars 2017 15:54:52 UTC+2, External Sébastien LAOUT -BTWIN

Post by External Sébastien LAOUT -BTWIN VILLAGE LILLE-
Hi dhru,
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-core</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-java</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-spring</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-junit</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
Does it work with these dependencies?

[Mahesh babu]

Hi,

This solution is not working as expected.

@Sample
Scenario Outline: Test Cucumber Next step on failure functionality
Given I have two numbers "<Number1>" and "<Number2>"
Then Sum will be "<Sum>"
And Difference will be "<Difference>"
And Multiplication will be "<Multiply>"
Examples:
|Number1|Number2|Sum|Difference|Multiply|
|7 |3 |10 |2 |21 |


@ContinueNextStepsFor(AssertionError.class)
@Then("^Sum will be \"(.*?)\"$")
public void sum_will_be(String arg1) throws Throwable {
// Write code here that turns the phrase above into concrete actions
sum = Integer.parseInt(arg1);
if(sum!=a+b)
throw new AssertionError("Addition is incorrect");
System.out.println("Sum is correct");
}

@ContinueNextStepsFor({AssertionError.class})
@Then("^Difference will be \"(.*?)\"$")
public void difference_will_be(String arg1) throws Throwable {
// Write code here that turns the phrase above into concrete actions
diff = Integer.parseInt(arg1);
if(diff!=a-b)
throw new AssertionError("Difference is incorrect");
System.out.println("Difference is correct");
}

@ContinueNextStepsFor(Throwable.class)
@Then("^Multiplication will be \"(.*?)\"$")
public void multiplication_will_be(String arg1) throws Throwable {
// Write code here that turns the phrase above into concrete actions
mul = Integer.parseInt(arg1);
if(mul!=a*b)
throw new AssertionError("Multiplication is incorrect");
System.out.println("Multiplication is correct");
}



*With the given data, I expect Multiplication should not skip (Difference step is fail),but still the step is skipped*

*Need help.*





On Monday, 27 March 2017 21:56:33 UTC+8, External Sébastien LAOUT -BTWIN

Post by External Sébastien LAOUT -BTWIN VILLAGE LILLE-
What do you mean by your tests are being skipped?
The scenarios do not execute?
An exception on the given steps make the scenario stop execute?
Le lundi 27 mars 2017 15:54:52 UTC+2, External Sébastien LAOUT -BTWIN

Post by External Sébastien LAOUT -BTWIN VILLAGE LILLE-
Hi dhru,
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-core</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-java</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-spring</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.github.slaout.fork.info.cukesthreads</groupId>
<artifactId>cucumber-junit</artifactId>
<version>1.2.4-SNAPSHOT</version>
</dependency>
Does it work with these dependencies?

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

@Swathi Reddy Yes, I've created a new project with only the Cucumber fork,
and I had to include these two other dependencies:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>4.0.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>4.0.2.RELEASE</version>
</dependency>

I'm sorry.
I will update the doc, and try to find a way to make it optional.

[External Sébastien LAOUT -BTWIN VILLAGE LILLE-]

@Mahesh babu : I don't understand: you added
@ContinueNextStepsFor(Throwable.class) on top of @Then("^Multiplication
will be \"(.*?)\"$")

So you expect the step to make the scenario continue even if the step is

failed.

If the step fails, it is not "marked as success", if that's what you meant
by "skipped".
It would be counter-productive to hide errors.
The purpose of the fork is to be able to spot ALL errors, even those hidden
after (an)other error(s); it is not meant to reduce the number of errors by
ignoring some.