Getting simplecov to work with minitest's rake task

[zenspider]

I've always hated how rake runs tests... it's SO broken. I've been running my own test task via hoe for a very long time and recentlyish ported it over to minitest (see the file here)... it runs in a different manner and as such, the recipe in the simplecov readme doesn't work...

I'm left wondering if this is related to / a variant of #1023 or not...

Here's what I've figured out so far:

with the following recipe at the top of my test:

if ENV['COV'] then
  require 'simplecov'
  SimpleCov.start
end

a manual run works fine:

%  ruby -I... -w test/test_debride.rb

but what the minitest rake task is doing does not:

% ruby -I... -w -e 'require "minitest/autorun"; require "test/test_debride.rb"' -- 

Instead, it runs the report BEFORE the tests start.

If I remove the recipe at the top of the file and manually modify the rake task command:

% ruby -I... -w -e 'require "simplecov"; SimpleCov.start; require "minitest/autorun"; require "test/test_debride.rb"' -- 

then it works again.

AFAICT... this has something to do with the way you're checking for minitest (the guess methods?) or hooking at_exit ... having the minitest/autorun inclusion might be messing it up. We can either try to figure out how to make simplecov happy no matter how tests are run (minitest/autorun before or after), or I can chalk this up to my rake task doing too much too soon and build in support for simplecov to the task itself.

What do you think?

[copiousfreetime]

Finding this in 2024 and the following seems to be working for me now, not sure what update I did with which gem, but I had 99% test coverage and then it went to 40% without any of my code changes, just gem dependencies.

Ended up with this as my task:

  Minitest::TestTask.create(:coverage) do |t|
    t.test_prelude = 'require "simplecov"; SimpleCov.start;'
    t.libs << "spec"
    t.warning = true
    t.test_globs = "{test,spec}/**/{test_*,*_spec}.rb"
  end

Resulting rake coverage:cmd

ruby -Ilib:test:.:spec -w -e 'require "simplecov"; SimpleCov.start;; require "minitest/autorun"; require "spec/paths_spec.rb"; require "spec/timed_metric_spec.rb"; require "spec/mutex_stats_spec.rb"; require "spec/metric_spec.rb"; require "spec/value_metric_spec.rb"; require "spec/interval_spec.rb"; require "spec/stats_spec.rb"; require "spec/initialize_spec.rb"; require "spec/hitimes_spec.rb"; require "spec/timed_value_metric_spec.rb"; require "spec/version_spec.rb"' --

[zenspider]

@copiousfreetime: Agreed... This is what hoe does these days w/ the test_prelude, but you might want to change yours a bit to possibly fix the test coverage numbers:

t.test_prelude = %(require "simplecov"; SimpleCov.start { add_filter %p }) % [cov_filter]

where cov_filter defaults to %w[tmp test].

I've given up on the maintainers tending to their issues so I'm closing this to get it off MY radar.

[PragTob]

Hi @zenspider,

I hope you can appreciate that maintaining OSS projects is sometimes hard (as you maintain a lot of projects) and we might have things in our lifes that prevent us from working on OSS projects. I don't appreciate when it's put as just one of simplecov's issues, yes we're mostly the once trying to fix it but it's quite the task to keep up with a whole eco system of testing tools and whatever they do with processes, evolving them and expecting that simplecov will work.

We have tried to solve our problems integrating with minitest via this plugin: https://github.com/simplecov-ruby/simplecov/blob/main/lib/minitest/simplecov_plugin.rb

Simplecov needs to be required and started before everything else (as that's how the coverage library works for tracking files via require) and then needs to run an at_exit hook (which we need to work around for minitest, see plugin) to essentially show the results when tests have finished.

If you have an idea how to fix that for minitest/minitest with autorun generally from our side without adjusting rake tasks etc. then I'm all ears. I don't use minitest, and so have only cursory knowledge of it, so would appreciate input on this.

[zenspider]

This is probably going to be a semi-randomly ordered set of observations... I might come back and clean it up but ... dunno.

---

Minitest (usually) runs via at_exit.

Simplecov (always?) runs via at_exit.

The two are intertwined, and need to cooperate in order to both work properly.

Multiple calls to at_exit act as a stack, not a queue, so the order in which things are loaded makes a big difference here. We generally want simplecov to be required FIRST so that its at_exit is run LAST.

Simplecov has a default at_exit block defined at lib/simplecov/defaults.rb:30 and this block escapes early if SimpleCov.external_at_exit? but otherwise generates the report by running SimpleCov.at_exit_behavior.

If simplecov is loaded via a test_helper.rb file, then it needs to be before requiring minitest/autorun AND you're not going to get coverage generated on that file (I think this is a global limitation of simplecov). [ETA: generally you're not wanting to measure coverage on test_helper, so this is NBD]

When running a test directly (eg ruby test/test_blah.rb), with a test_helper that loads/starts simplecov first, everything works as expected with the caveat mentioned above.

When running tests via minitest's test task, the way that it runs tests is essentially:

ruby -e 'test_prelude; require "minitest/autorun"; require "test/files"' --

With no test_prelude, this requires minitest/autorun BEFORE simplecov! This changes the ordering of at_exit and that default block runs BEFORE the tests get run. This is bad.

With a test prelude of:

t.test_prelude = %(require "simplecov"; SimpleCov.external_at_exit = true)

then the default at_exit block is defined before minitest's at_exit, the simplecov plugin defines an Minitest.after_run which calls SimpleCov.at_exit_behavior. This is good, but the above caveat still applies for coverage on test_helper.

With a test prelude of:

t.test_prelude = %(require "simplecov"; SimpleCov.start)

then simplecov is started before minitest is loaded and is done so before the loading of test_helper. This is good.

Hoe provides this via an extra cov task:

    desc "Run tests and analyze code coverage"
    task :cov => :isolate do
      test_task.test_prelude =
        %(require "simplecov"; SimpleCov.start { add_filter %p }) % [cov_filter]

      Rake::Task[:test].invoke
    end

but I think the most normal way for someone to set up new Rakefile would be like:

Minitest::TestTask.create :test

Minitest::TestTask.create :coverage do |t|
  t.test_prelude = %(require "simplecov"; SimpleCov.start)
end

Is there anything more to do at this point?

[adam12]

I had an opportunity to review this, and the use of test_prelude when paired with the new Minitest::TestTask does seem like a great approach. I've tested it and it works great.