How to hook after example has execution result and status with :aggregated_failures flag

Question

I'm maintaining a standalone test automation suite written in Rspec & Capybara and SitePrism (No Rails).

Recently I started integrating it with Testrail for reporting - I used rspec-testrail gem for that but I had to modify it a bit, because I also wanted to send Slack notifications with test progress and report.

Anyway, the integration works smoothly, except the fact that example processing is relying on example having an exception and lack of exception causes setting the test status as Passed on Testrail. As it appears, after :each nor after :example in Rspec.configure doesn't guarantee that the example has finished running. I also tried around :example and around :each as described here, but to no avail.

I inspected contents of example.metadata and it looks that example.metadata[:execution_result] has only started_at variable, but a finished example would have also finished_at and status variables, according to the docs

My suspicion (after reading the relish docs) is that :aggregated_failures is the cause of different metadata structure and multiple expects running in threads that are later merged into one backtrace.

Do you know how can I wait for the example to finish or how to hook into the state where it's finished? Or maybe I should create a custom formatter where I would hook after example notifications printed to the console (I would like to keep the stacktrace there).

My code is as follows:

Test (both assertions are failing):

require 'spec_helper'

feature 'Sign in' do
  let(:login_page) { LoginPage.new }
  let(:user) { { email: ENV['ADMIN_EMAIL'], password: ENV['ADMIN_PASSWORD'] } }

  scenario 'is successful and user is redirected to dashboard for user with correct credentials', testrail_id: 5694 do
    login_page.load
    login_page.form.email.set(user[:email])
    login_page.form.password.set(user[:password])
    login_page.form.submit_btn.click
    expect(login_page.sidebar).to have_jobs(text: "some_nonexistenttext")
    login_page.logout
    expect(current_url).to have_content "google.com"
  end
end

Console output from the above test:

Failures:

  1) Sign in is successful and user is redirected to dashboard for user with correct credentials
     Got 2 failures:

     1.1) Failure/Error: expect(login_page.sidebar).to have_jobs(text: "blala")
            expected #has_jobs?({:text=>"some_nonexistenttext"}) to return true, got false
          # ./spec/auth/login_spec.rb:13:in `block (3 levels) in <top (required)>'

     1.2) Failure/Error: expect(current_url).to have_content "google.com"
            expected to find text "google.com" in "https://example.com/"
          # ./spec/auth/login_spec.rb:15:in `block (3 levels) in <top (required)>'

Finished in 53.91 seconds (files took 1.45 seconds to load)
4 examples, 1 failure

Failed examples:

rspec ./spec/auth/login_spec.rb:8 # Sign in is successful and user is redirected to dashboard for user with correct credentials

Spec helper:

require 'rubygems'
require 'capybara/rspec'
require 'selenium-webdriver'
require 'site_prism'
require 'slack-notifier'

require_relative '../config'
require_relative '../lib/testrail'

...

RSpec.configure do |config|
  config.define_derived_metadata do |meta|
    meta[:aggregate_failures] = true
  end
  config.example_status_persistence_file_path = 'examples.txt'
  config.before :all do
    testrail_initialize_test_run!
  end

  config.after :example, testrail_id: proc { |value| !value.nil? } do |example|
    RSpec::Testrail.process(example)
  end
end

processing method (slightly modified from original)

  def process(example)
    if example.exception
      status = 5
      message = example.exception.message
      slack_notifier.publish(message_text "Failed")
    elsif example.skipped? || example.pending?
      puts 'Example skipped or pending'
      status = 10
      message = 'Pending or not implemented'
    else
      status = 1
      message = ''
    end
    client.send_post("add_result_for_case/#{testrun['id']}/#{example.metadata[:testrail_id]}",
                      status_id: status,
                      comment: message)
  end

Answer 1

So basically all I had to was to use a reporter listener and process notifications inside it :)

config.reporter.register_listener RSpec::Testrail::Listener.new, :start, :example_failed, :example_passed, :example_pending, :stop