Codecov broken tests

[cpu82]

Hi,

It seems that there are some test cases that fail to pass.

Tested on FreeBSD by invoking the following command:

/usr/local/bin/python2.7 -m pytest -v tests/test.py

http://sprunge.us/hIEC

Is it possible to skip the tests explicitly when invoking the test suite?

[cpu82]

I noticed that if the .git directory is present all test cases run fine, except 3 tests skipped.

% python2.7 -m pytest -v tests/test.py
============================= test session starts ==============================
platform freebsd12 -- Python 2.7.13, pytest-3.1.0, py-1.4.33, pluggy-0.4.0 -- /usr/local/bin/python2.7
cachedir: .cache
rootdir: /usr/home/cpm/work/gh/codecov-python, inifile:
collected 68 items 

tests/test.py::TestUploader::test_bowerrc PASSED
tests/test.py::TestUploader::test_bowerrc_none PASSED
tests/test.py::TestUploader::test_ci_appveyor PASSED
tests/test.py::TestUploader::test_ci_buildkite PASSED
tests/test.py::TestUploader::test_ci_circleci PASSED
tests/test.py::TestUploader::test_ci_codeship PASSED
tests/test.py::TestUploader::test_ci_drone PASSED
tests/test.py::TestUploader::test_ci_gitlab PASSED
tests/test.py::TestUploader::test_ci_jenkins PASSED
tests/test.py::TestUploader::test_ci_jenkins_blue_ocean PASSED
tests/test.py::TestUploader::test_ci_jenkins_env PASSED
tests/test.py::TestUploader::test_ci_magnum PASSED
tests/test.py::TestUploader::test_ci_none PASSED
tests/test.py::TestUploader::test_ci_semaphore PASSED
tests/test.py::TestUploader::test_ci_shippable PASSED
tests/test.py::TestUploader::test_ci_snap PASSED
tests/test.py::TestUploader::test_ci_travis PASSED
tests/test.py::TestUploader::test_ci_wercker PASSED
tests/test.py::TestUploader::test_command PASSED
tests/test.py::TestUploader::test_disable_detect PASSED
tests/test.py::TestUploader::test_disable_gcov SKIPPED
tests/test.py::TestUploader::test_disable_search PASSED
tests/test.py::TestUploader::test_discovers PASSED
tests/test.py::TestUploader::test_exits_0 PASSED
tests/test.py::TestUploader::test_exits_1 PASSED
tests/test.py::TestUploader::test_gcov SKIPPED
tests/test.py::TestUploader::test_ignore_report_1__coverage_worker10 PASSED
tests/test.py::TestUploader::test_ignore_report_2_coverage_jade PASSED
tests/test.py::TestUploader::test_ignore_report_3_include_lst PASSED
tests/test.py::TestUploader::test_ignore_report_4_inputFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_5_createdFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_6_scoverage_measurements_blackandwhite_xml PASSED
tests/test.py::TestUploader::test_ignore_report_7_test_hello_coverage_txt PASSED
tests/test.py::TestUploader::test_ignore_report_8_conftest_blackwhite_c_gcov PASSED
tests/test.py::TestUploader::test_ignored_path_10__tox PASSED
tests/test.py::TestUploader::test_ignored_path_11_venv PASSED
tests/test.py::TestUploader::test_ignored_path_12__venv_python_2_7 PASSED
tests/test.py::TestUploader::test_ignored_path_1_vendor PASSED
tests/test.py::TestUploader::test_ignored_path_2_node_modules PASSED
tests/test.py::TestUploader::test_ignored_path_3_js_generated_coverage PASSED
tests/test.py::TestUploader::test_ignored_path_4___pycache__ PASSED
tests/test.py::TestUploader::test_ignored_path_5_coverage_instrumented PASSED
tests/test.py::TestUploader::test_ignored_path_6_build_lib PASSED
tests/test.py::TestUploader::test_ignored_path_7_htmlcov PASSED
tests/test.py::TestUploader::test_ignored_path_8__egg_info PASSED
tests/test.py::TestUploader::test_ignored_path_9__git PASSED
tests/test.py::TestUploader::test_include_env PASSED
tests/test.py::TestUploader::test_is_report_10_luacov_report_out PASSED
tests/test.py::TestUploader::test_is_report_11_gcov_info PASSED
tests/test.py::TestUploader::test_is_report_12_nosetests_xml PASSED
tests/test.py::TestUploader::test_is_report_1_coverage_xml PASSED
tests/test.py::TestUploader::test_is_report_2_jacoco_xml PASSED
tests/test.py::TestUploader::test_is_report_3_jacocoTestResults_xml PASSED
tests/test.py::TestUploader::test_is_report_4_coverage_txt PASSED
tests/test.py::TestUploader::test_is_report_5_gcov_lst PASSED
tests/test.py::TestUploader::test_is_report_6_cov_gcov PASSED
tests/test.py::TestUploader::test_is_report_7_info_lcov PASSED
tests/test.py::TestUploader::test_is_report_8_clover_xml PASSED
tests/test.py::TestUploader::test_is_report_9_cobertura_xml PASSED
tests/test.py::TestUploader::test_none_found PASSED
tests/test.py::TestUploader::test_not_jacoco PASSED
tests/test.py::TestUploader::test_read_token_file PASSED
tests/test.py::TestUploader::test_require_branch_1 PASSED
tests/test.py::TestUploader::test_returns_none PASSED
tests/test.py::TestUploader::test_run_coverage SKIPPED
tests/test.py::TestUploader::test_run_coverage_fails PASSED
tests/test.py::TestUploader::test_send PASSED
tests/test.py::TestUploader::test_send_error PASSED

===================== 65 passed, 3 skipped in 2.20 seconds =====================

[stevepeak]

We welcome a pull request if you feel inclined to adjust the code base for this use case.
Speaking of which: why is the use case for your teste suite?

[cpu82]

According to the FreeBSD python port policy [1]: python ports must include a do-test target if the package includes tests.

For now, I have removed the failed tests using a simple patch [2].

Therefore, it would be great that all test cases work properly.

[1] https://wiki.freebsd.org/Python/PortsPolicy#Tests
[2] https://reviews.freebsd.org/D10873#1609c58f

[stevepeak]

@naglalakk can you help out here. Thanks

[stevepeak]

I'm cleaning up issues. What is the status of this one?

[cpu82]

Hi,

Today I updated codecov to 2.0.13 but it still failing with a couple of AppVeyor CI tests.

As a workaround I skipped both tests at the moment.

--- tests/test.py.orig  2018-01-08 10:57:15 UTC
+++ tests/test.py
@@ -134,7 +134,7 @@ class TestUploader(unittest.TestCase):
                 self.assertEqual(codecov.main(), None)
                 assert post.called and put.called
 
-    @unittest.skipIf(os.getenv('CI') == "True" and os.getenv('APPVEYOR') == 'True', 'Skip AppVeyor CI test')
+    @unittest.skipIf(sys.platform.startswith('freebsd'),'Skip test')
     def test_send(self):
         with patch('requests.post') as post:
             with patch('requests.put') as put:
@@ -249,7 +249,7 @@ class TestUploader(unittest.TestCase):
         else:
             raise Exception("Did not raise AssertionError")
 
-    @unittest.skipIf(os.getenv('CI') == "True" and os.getenv('APPVEYOR') == 'True', 'Skip AppVeyor CI test')
+    @unittest.skipIf(sys.platform.startswith('freebsd'),'Skip test')
     def test_bowerrc_none(self):
         with open(self.bowerrc, 'w+') as f:
             f.write('{"other_key": "tests"}')

Here is the make test output

$ make test
===>  License APACHE20 accepted by the user
===>   py27-codecov-2.0.13 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by py27-codecov-2.0.13 for building
===>  Extracting for py27-codecov-2.0.13
=> SHA256 Checksum OK for codecov-codecov-python-v2.0.13_GH0.tar.gz.
===>  Patching for py27-codecov-2.0.13
===>  Applying FreeBSD patches for py27-codecov-2.0.13
===>   py27-codecov-2.0.13 depends on package: py27-setuptools>0 - found
===>   py27-codecov-2.0.13 depends on file: /usr/local/bin/python2.7 - found
===>  Configuring for py27-codecov-2.0.13
running config
===>  Building for py27-codecov-2.0.13
running build
running build_py
creating build
creating build/lib
creating build/lib/codecov
copying codecov/__init__.py -> build/lib/codecov
copying codecov/__main__.py -> build/lib/codecov
running egg_info
creating codecov.egg-info
writing requirements to codecov.egg-info/requires.txt
writing codecov.egg-info/PKG-INFO
writing top-level names to codecov.egg-info/top_level.txt
writing dependency_links to codecov.egg-info/dependency_links.txt
writing entry points to codecov.egg-info/entry_points.txt
writing manifest file 'codecov.egg-info/SOURCES.txt'
reading manifest file 'codecov.egg-info/SOURCES.txt'
writing manifest file 'codecov.egg-info/SOURCES.txt'
===>  Staging for py27-codecov-2.0.13
===>   py27-codecov-2.0.13 depends on package: py27-coverage>0 - found
===>   py27-codecov-2.0.13 depends on package: py27-requests>=2.7.9 - found
===>   py27-codecov-2.0.13 depends on package: py27-setuptools>0 - found
===>   py27-codecov-2.0.13 depends on file: /usr/local/bin/python2.7 - found
===>   Generating temporary packing list
running install
running build
running build_py
running egg_info
writing requirements to codecov.egg-info/requires.txt
writing codecov.egg-info/PKG-INFO
writing top-level names to codecov.egg-info/top_level.txt
writing dependency_links to codecov.egg-info/dependency_links.txt
writing entry points to codecov.egg-info/entry_points.txt
reading manifest file 'codecov.egg-info/SOURCES.txt'
writing manifest file 'codecov.egg-info/SOURCES.txt'
running install_lib
creating /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7
creating /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages
creating /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov
copying build/lib/codecov/__init__.py -> /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov
copying build/lib/codecov/__main__.py -> /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov
byte-compiling /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov/__init__.py to __init__.pyc
byte-compiling /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov/__main__.py to __main__.pyc
writing byte-compilation script '/tmp/tmptHoyRL.py'
/usr/local/bin/python2.7 -O /tmp/tmptHoyRL.py
removing /tmp/tmptHoyRL.py
running install_egg_info
Copying codecov.egg-info to /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov-2.0.13-py2.7.egg-info
running install_scripts
Installing codecov script to /usr/ports/devel/py-codecov/work-py27/stage/usr/local/bin
writing list of installed files to '/usr/ports/devel/py-codecov/work-py27/.PLIST.pymodtmp'
===> Creating unique files: Move MAN files needing SUFFIX
===> Creating unique files: Move files needing SUFFIX
Move: bin/codecov --> bin/codecov-2.7
Link: @bin/codecov --> bin/codecov-2.7
====> Compressing man pages (compress-man)
====> Running Q/A tests (stage-qa)
===>  Testing for py27-codecov-2.0.13
===>   py27-codecov-2.0.13 depends on package: py27-ddt>0 - found
===>   py27-codecov-2.0.13 depends on package: py27-mock>0 - found
===>   py27-codecov-2.0.13 depends on package: py27-pytest>0 - found
===>   py27-codecov-2.0.13 depends on file: /usr/local/bin/python2.7 - found
============================= test session starts ==============================
platform freebsd10 -- Python 2.7.14, pytest-3.2.5, py-1.5.2, pluggy-0.4.0 -- /usr/local/bin/python2.7
cachedir: .cache
rootdir: /usr/ports/devel/py-codecov/work-py27/codecov-python-2.0.13, inifile:
plugins: xdist-1.13.1
collected 67 items                                                              

tests/test.py::TestUploader::test_bowerrc PASSED
tests/test.py::TestUploader::test_bowerrc_none <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_appveyor <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_buildkite <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_circleci <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_codeship <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_drone <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_gitlab <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_jenkins <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_jenkins_blue_ocean <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_jenkins_env <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_magnum <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_none <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_semaphore <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_shippable <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_travis <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_wercker <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_command PASSED
tests/test.py::TestUploader::test_disable_detect PASSED
tests/test.py::TestUploader::test_disable_gcov SKIPPED
tests/test.py::TestUploader::test_disable_search PASSED
tests/test.py::TestUploader::test_discovers PASSED
tests/test.py::TestUploader::test_exits_0 PASSED
tests/test.py::TestUploader::test_exits_1 PASSED
tests/test.py::TestUploader::test_gcov SKIPPED
tests/test.py::TestUploader::test_ignore_report_1__coverage_worker10 PASSED
tests/test.py::TestUploader::test_ignore_report_2_coverage_jade PASSED
tests/test.py::TestUploader::test_ignore_report_3_include_lst PASSED
tests/test.py::TestUploader::test_ignore_report_4_inputFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_5_createdFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_6_scoverage_measurements_blackandwhite_xml PASSED
tests/test.py::TestUploader::test_ignore_report_7_test_hello_coverage_txt PASSED
tests/test.py::TestUploader::test_ignore_report_8_conftest_blackwhite_c_gcov PASSED
tests/test.py::TestUploader::test_ignored_path_10__tox PASSED
tests/test.py::TestUploader::test_ignored_path_11_venv PASSED
tests/test.py::TestUploader::test_ignored_path_12__venv_python_2_7 PASSED
tests/test.py::TestUploader::test_ignored_path_1_vendor PASSED
tests/test.py::TestUploader::test_ignored_path_2_node_modules PASSED
tests/test.py::TestUploader::test_ignored_path_3_js_generated_coverage PASSED
tests/test.py::TestUploader::test_ignored_path_4___pycache__ PASSED
tests/test.py::TestUploader::test_ignored_path_5_coverage_instrumented PASSED
tests/test.py::TestUploader::test_ignored_path_6_build_lib PASSED
tests/test.py::TestUploader::test_ignored_path_7_htmlcov PASSED
tests/test.py::TestUploader::test_ignored_path_8__egg_info PASSED
tests/test.py::TestUploader::test_ignored_path_9__git PASSED
tests/test.py::TestUploader::test_include_env PASSED
tests/test.py::TestUploader::test_is_report_10_luacov_report_out PASSED
tests/test.py::TestUploader::test_is_report_11_gcov_info PASSED
tests/test.py::TestUploader::test_is_report_12_nosetests_xml PASSED
tests/test.py::TestUploader::test_is_report_1_coverage_xml PASSED
tests/test.py::TestUploader::test_is_report_2_jacoco_xml PASSED
tests/test.py::TestUploader::test_is_report_3_jacocoTestResults_xml PASSED
tests/test.py::TestUploader::test_is_report_4_coverage_txt PASSED
tests/test.py::TestUploader::test_is_report_5_gcov_lst PASSED
tests/test.py::TestUploader::test_is_report_6_cov_gcov PASSED
tests/test.py::TestUploader::test_is_report_7_info_lcov PASSED
tests/test.py::TestUploader::test_is_report_8_clover_xml PASSED
tests/test.py::TestUploader::test_is_report_9_cobertura_xml PASSED
tests/test.py::TestUploader::test_none_found PASSED
tests/test.py::TestUploader::test_not_jacoco PASSED
tests/test.py::TestUploader::test_read_token_file PASSED
tests/test.py::TestUploader::test_require_branch_1 PASSED
tests/test.py::TestUploader::test_returns_none PASSED
tests/test.py::TestUploader::test_run_coverage SKIPPED
tests/test.py::TestUploader::test_run_coverage_fails PASSED
tests/test.py::TestUploader::test_send <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_send_error PASSED

==================== 47 passed, 20 skipped in 1.72 seconds =====================

[thomasrockhu]

Closing this out as tests should be passing now.

[yzgyyang]

Closing this out as tests should be passing now.

This is not true - OP was talking about that, by simply removing the .git directory, tests will begin to fail (I got "12 failed, 38 passed, 20 skipped in 3.19s"). This happens if one downloads a release (which doesn't include .git) and try to run the test suite.