How to force pytest to run every test (stop skipping tests not marked skip)?

Question

Pytest, when run from the command line, keep skipping some of my parameterized tests even though I have not told it to do so. Strangely, it does not skip the tests when run using the command palette in VSCode.

I've tried running the tests and test files individually and tweaking the command line options, all to no avail. Presumably, there's some subtlety of configuration I'm missing. Can you help?

Example test

@pytest.mark.parametrize(
    "inputs, expec",
    helpers.get_samples('inouts/kmp'),
    ids=helpers.get_ids('inouts/kmp'))
def test_kmp(capsys, inputs, expec):
    """Sample test
    """
    with patch('kmp.sys.stdin', inputs):
        kmp.main()
    captured = capsys.readouterr()
    print(captured.err, file=sys.stderr)
    assert captured.out == expec.getvalue()

Helper functions (support parametrization)

INGLOB = '*in*'
OUTGLOB = '*out*'

def _get_globs(path):
    """Collect input/output filename pairs
    """
    if not path.endswith("/"):
        path = path + "/"
    infiles = sorted(glob.glob(path + INGLOB))
    outfiles = sorted(glob.glob(path + OUTGLOB))
    assert [i.split('.')[0]
            for i in infiles] == [o.split('.')[0] for o in outfiles]
    return zip(infiles, outfiles)


def get_samples(path):
    """Reads sample inputs/outputs into StringIO memory buffers
    """
    files = _get_globs(path)
    inputs_outputs = []
    for infile, outfile in files:
        with open(infile, 'r') as f1:
            inputs = StringIO(f1.read())
        with open(outfile, 'r') as f2:
            outputs = StringIO(f2.read())
        inputs_outputs.append(tuple([inputs, outputs]))
    return inputs_outputs


def get_ids(path):
    """Returns list of filenames as test ids
    """
    return [f for f, _ in _get_globs(path)]

Running this project in VSCode from the command palette produces:

platform darwin -- Python 3.7.2, pytest-4.1.0, py-1.7.0, pluggy-0.8.1
rootdir: ... , inifile:
plugins: cov-2.6.1
collected 79 items

test_1.py ........................................         [ 50%]
test_2.py ...................................              [ 94%]
test_3.py ....                                             [100%]

generated xml file: 
/var/folders/pn/y4jjr_t531g3x3v0s0snqzf40000gn/T/tmp-...
========================== 79 passed in 0.55 seconds ===========================

But running the same tests from the command line produces:

=============================== test session starts ===============================
platform darwin -- Python 3.7.2, pytest-4.1.0, py-1.7.0, pluggy-0.8.1
rootdir: ... , inifile:
plugins: cov-2.6.1
collected 59 items

test_1.py ssss...................s.....                           [ 49%]
test_2.py s.......................ss.s                            [ 96%]
test_3.py s.                                                      [100%]

====================== 49 passed, 10 skipped in 0.42 seconds ======================

How do I get pytest to collect and run the full 76 items? I'm not using any special options in VSCode. I don't understand why pytest is skipping tests without being told to do so.

Thanks!

Answer 1

pytest will automatically "skip" tests which are parametrized but have no entries.

The most trivial example:

@pytest.mark.parametrize('a', ())
def test(a): pass

and output

$ pytest -v t.py 

...

t.py::test[a0] SKIPPED                                                    [100%]

=========================== 1 skipped in 0.02 seconds ===========================

your issue here is likely the two execution environments, either current working directory or some such is causing your data collection to return zero results in your terminal but actual results when run from vscode. I'd check your current working directory and the virtualenv you have activated first and debug from there. Maybe even put a breakpoint inside of get_samples / get_ids.