Skip to content

Infrastructure: Upgrading pythontest service #122795

Closed
Closed
Infrastructure: Upgrading pythontest service#122795
Assignees
JacobCoffee

Description

@JacobCoffee
JacobCoffee
opened on Aug 7, 2024

I'm commencing OS upgrades for the pythontest service to the latest Ubuntu LTS - which is used for some network tests in the CPython test suite (see lines referencing pythontest.net.)

No changes should be necessary on anyones end, but if anyone happens to run their tests with internet access enabled some may fail during this upgrade.

### Tasks
- [x] Host provisioned
- [x] Data migrated
- [x] Test suite tested against current `pythontest` service
- [x] Test suite tested against new `pythontest` service
- [x] Test suite tested against bogus endpoint to ensure fails indeed... fail
- [x] Old production host shut down
- [x] New production host swapped out
- [x] Test suite tested against replaced `pythontest` service
- [x] DNS records swapped in Route53

Activity

  1. JacobCoffee
    self-assigned this
    on Aug 7, 2024

  2. JacobCoffee commented on Aug 7, 2024

    @JacobCoffee
    JacobCoffee
    on Aug 7, 2024
    MemberAuthor

    TTL is 7200 for the DNS entries so they will take awhile to roll over.

  3. JacobCoffee commented on Aug 7, 2024

    @JacobCoffee
    JacobCoffee
    on Aug 7, 2024
    MemberAuthor

    DNS has propagated and tests are now passing

    cpython [⇣1] via C v15.0.0-clang via  pyenv took 5s
➜ ./python.exe -m test -v test_urllib2net -u all -j16
== CPython 3.14.0a0 (heads/main:5b8a6c5186, Aug 6 2024, 16:03:59) [Clang 15.0.0 (clan
== macOS-14.5-arm64-arm-64bit-Mach-O little-endian
== Python build: debug
== cwd: /Users/coffee/git/internal/python/cpython/build/test_python_worker_20375æ
== CPU count: 16
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 681115882
0:00:00 load avg: 2.32 Run 1 test in parallel using 1 worker process
0:00:05 load avg: 2.21 [1/1] test_urllib2net passed
test_close (test.test_urllib2net.CloseSocketTest.test_close) ... ok
test_custom_headers (test.test_urllib2net.OtherNetworkTests.test_custom_headers) ... ok
test_file (test.test_urllib2net.OtherNetworkTests.test_file) ... ok
test_ftp (test.test_urllib2net.OtherNetworkTests.test_ftp) ... ok
test_redirect_url_withfrag (test.test_urllib2net.OtherNetworkTests.test_redirect_url_withfrag) ... ok
test_sites_no_connection_close (test.test_urllib2net.OtherNetworkTests.test_sites_no_connection_close) ... skipped 'XXX: http://www.imdb.com is gone'
test_urlwithfrag (test.test_urllib2net.OtherNetworkTests.test_urlwithfrag) ... ok
test_ftp_basic (test.test_urllib2net.TimeoutTest.test_ftp_basic) ... ok
test_ftp_default_timeout (test.test_urllib2net.TimeoutTest.test_ftp_default_timeout) ... ok
test_ftp_no_timeout (test.test_urllib2net.TimeoutTest.test_ftp_no_timeout) ... ok
test_ftp_timeout (test.test_urllib2net.TimeoutTest.test_ftp_timeout) ... ok
test_http_basic (test.test_urllib2net.TimeoutTest.test_http_basic) ... ok
test_http_default_timeout (test.test_urllib2net.TimeoutTest.test_http_default_timeout) ... ok
test_http_no_timeout (test.test_urllib2net.TimeoutTest.test_http_no_timeout) ... ok
test_http_timeout (test.test_urllib2net.TimeoutTest.test_http_timeout) ... ok

----------------------------------------------------------------------
Ran 15 tests in 5.375s

OK (skipped=1)

== Tests result: SUCCESS ==

1 test OK.

Total duration: 5.6 sec
Total tests: run=15 skipped=1
Total test files: run=1/1
Result: SUCCESS
  4. JacobCoffee
    closed this as completedon Aug 7, 2024

  5. hugovk commented on Aug 7, 2024

    @hugovk
    hugovk
    on Aug 7, 2024
    Member

    Confirmed PRs are now passing, thank you for keeping the infra up-to-date!

  6. ned-deily commented on Aug 14, 2024

    @ned-deily
    ned-deily
    on Aug 14, 2024
    Member

    We are seeing some ftp tests failing from macOS and iOS buildbots / CI with pythontest.net vsftpd apparently returning 500 OOPS: vsf_sysutil_bind errors. I think these are new errors but can't swear to it. @freakboy3742 noticed the failures with the iOS buildbots. Sounds like it might have to do with not enough passive ports or something similar. It could be that connections are taking a while to time out from some buildbot or CI instances. This is one of those things that just worked for years so perhaps the old OS configuration had tweaked something?

    Here are two examples from today:

    https://github.com/python/cpython/actions/runs/10380012215/job/28739153641
    https://buildbot.python.org/#/builders/1380/builds/1068/steps/10/logs/stdio

  7. freakboy3742 commented on Aug 14, 2024

    @freakboy3742
    freakboy3742
    on Aug 14, 2024
    Contributor

    FWIW - I've also seen the failure on Linux as well (see 1, 2).

    There's possibly an interwoven issue with socket_helper. transient_internet; these errors all appear to be 500 service errors from an FTP service, and the helper is choking because that particular error doesn't set reason on the error, so what would be normally be marked as a resource skip is raising as an error.

  8. ewdurbin commented on Aug 14, 2024

    @ewdurbin
    ewdurbin
    on Aug 14, 2024
    Member

    Looks like we are only opening 10 passive ports, which is the same as past hosts. Seems reasonable to just extend that further to ~100 ports.

    vsftpd.conf: https://github.com/python/psf-salt/blob/f348b8a70632f5bc238bba6fd826d5fb4f8cf6f6/salt/pythontest/config/vsftpd.conf#L35-L42
    firewall config: https://github.com/python/psf-salt/blob/f348b8a70632f5bc238bba6fd826d5fb4f8cf6f6/pillar/base/firewall/ftp.sls#L7

  9. JacobCoffee commented on Aug 15, 2024

    @JacobCoffee
    JacobCoffee
    on Aug 15, 2024
    MemberAuthor

    This is deployed so if you continue to see errors let me know

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions