Logo: Relish

  1. Sign up
  2. Sign in

Project: RSpec Core 3.7

Bisect

  • @with-clean-spec-opts

RSpec's --order random and --seed options help surface flickering examples that only fail when one or more other examples are executed first. It can be very difficult to isolate the exact combination of examples that triggers the failure. The --bisect flag helps solve that problem.

Pass the --bisect option (in addition to --seed and any other options) and RSpec will repeatedly run subsets of your suite in order to isolate the minimal set of examples that reproduce the same failures.

At any point during the bisect run, you can hit ctrl-c to abort and it will provide you with the most minimal reproduction command it has discovered so far.

To get more detailed output (particularly useful if you want to report a bug with bisect), use --bisect=verbose.

Background
Given
a file named "lib/calculator.rb" with:
class Calculator
  def self.add(x, y)
    x + y
  end
end
And
a file named "spec/calculator_1_spec.rb" with:
require 'calculator'

RSpec.describe "Calculator" do
  it 'adds numbers' do
    expect(Calculator.add(1, 2)).to eq(3)
  end
end
And
files "spec/calculator_2_spec.rb" through "spec/calculator_9_spec.rb" with an unrelated passing spec in each file
And
a file named "spec/calculator_10_spec.rb" with:
require 'calculator'

RSpec.describe "Monkey patched Calculator" do
  it 'does screwy math' do
    # monkey patching Calculator affects examples that are
    # executed after this one!
    def Calculator.add(x, y)
      x - y
    end

    expect(Calculator.add(5, 10)).to eq(-5)
  end
end
Scenarios
Use `--bisect` flag to create a minimal repro case for the ordering dependency
When
I run rspec --seed 1234
Then
the output should contain "10 examples, 1 failure"
When
I run rspec --seed 1234 --bisect
Then
bisect should succeed with output like:
Bisect started using options: "--seed 1234"
Running suite to find failures... (0.16755 seconds)
Starting bisect with 1 failing example and 9 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent

Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9 (0.30166 seconds)
Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5 (0.30306 seconds)
Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.33292 seconds)
Round 4: bisecting over non-failing examples 1-2 . ignoring example 1 (0.16476 seconds)
Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.26 seconds.

The minimal reproduction command is:
  rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234
When
I run rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234
Then
the output should contain "2 examples, 1 failure"
Ctrl-C can be used to abort the bisect early and get the most minimal command it has discovered so far
When
I run rspec --seed 1234 --bisect and abort in the middle with ctrl-c
Then
bisect should fail with output like:
Bisect started using options: "--seed 1234"
Running suite to find failures... (0.17102 seconds)
Starting bisect with 1 failing example and 9 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent

Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9 (0.32943 seconds)
Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5 (0.3154 seconds)
Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.2175 seconds)

Bisect aborted!

The most minimal reproduction command discovered so far is:
  rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234
When
I run rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234
Then
the output should contain "3 examples, 1 failure"
Use `--bisect=verbose` to enable verbose debug mode for more detail
When
I run rspec --seed 1234 --bisect=verbose
Then
bisect should succeed with output like:
Bisect started using options: "--seed 1234"
Running suite to find failures... (0.16528 seconds)
 - Failing examples (1):
    - ./spec/calculator_1_spec.rb[1:1]
 - Non-failing examples (9):
    - ./spec/calculator_10_spec.rb[1:1]
    - ./spec/calculator_2_spec.rb[1:1]
    - ./spec/calculator_3_spec.rb[1:1]
    - ./spec/calculator_4_spec.rb[1:1]
    - ./spec/calculator_5_spec.rb[1:1]
    - ./spec/calculator_6_spec.rb[1:1]
    - ./spec/calculator_7_spec.rb[1:1]
    - ./spec/calculator_8_spec.rb[1:1]
    - ./spec/calculator_9_spec.rb[1:1]
Checking that failure(s) are order-dependent..
 - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (n.nnnn seconds)
 - Failure appears to be order-dependent
Round 1: bisecting over non-failing examples 1-9
 - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 (0.15302 seconds)
 - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.19708 seconds)
 - Examples we can safely ignore (4):
    - ./spec/calculator_6_spec.rb[1:1]
    - ./spec/calculator_7_spec.rb[1:1]
    - ./spec/calculator_8_spec.rb[1:1]
    - ./spec/calculator_9_spec.rb[1:1]
 - Remaining non-failing examples (5):
    - ./spec/calculator_10_spec.rb[1:1]
    - ./spec/calculator_2_spec.rb[1:1]
    - ./spec/calculator_3_spec.rb[1:1]
    - ./spec/calculator_4_spec.rb[1:1]
    - ./spec/calculator_5_spec.rb[1:1]
Round 2: bisecting over non-failing examples 1-5
 - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.15836 seconds)
 - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.19065 seconds)
 - Examples we can safely ignore (2):
    - ./spec/calculator_4_spec.rb[1:1]
    - ./spec/calculator_5_spec.rb[1:1]
 - Remaining non-failing examples (3):
    - ./spec/calculator_10_spec.rb[1:1]
    - ./spec/calculator_2_spec.rb[1:1]
    - ./spec/calculator_3_spec.rb[1:1]
Round 3: bisecting over non-failing examples 1-3
 - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 (0.21028 seconds)
 - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.1975 seconds)
 - Examples we can safely ignore (1):
    - ./spec/calculator_2_spec.rb[1:1]
 - Remaining non-failing examples (2):
    - ./spec/calculator_10_spec.rb[1:1]
    - ./spec/calculator_3_spec.rb[1:1]
Round 4: bisecting over non-failing examples 1-2
 - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.17173 seconds)
 - Examples we can safely ignore (1):
    - ./spec/calculator_3_spec.rb[1:1]
 - Remaining non-failing examples (1):
    - ./spec/calculator_10_spec.rb[1:1]
Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.47 seconds.

The minimal reproduction command is:
  rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234
When
I run rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234
Then
the output should contain "2 examples, 1 failure"

Last published 27 days ago by myronmarston.