Skip to content

CI failures: 20180925 #17

Closed
Closed
CI failures: 20180925#17

Description

@joyeecheung
joyeecheung
opened on Sep 25, 2018

Failures in node-test-pull-request/17334 to node-test-pull-request/17430 that failed more than 2 PRs
(Generated using ncu-ci walk pr --stats --copy)

JSTest Failure

Reason parallel/test-bootstrap-modules
Type JS_TEST_FAILURE
Failed PR 7 (nodejs/node#22845, nodejs/node#22951, nodejs/node#23014, nodejs/node#23010, nodejs/node#23042, nodejs/node#23044, nodejs/node#22998)
Appeared undefined
First CI https://ci.nodejs.org/job/node-test-pull-request/17339/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17399/
Example 
not ok 154 parallel/test-bootstrap-modules
  ---
  duration_ms: 0.336
  severity: fail
  exitcode: 1
  stack: |-
    
    events.js:167
          throw er; // Unhandled 'error' event
          ^
    AssertionError [ERR_ASSERTION]: Internal Binding contextify,Internal Binding worker,NativeModule events,Internal Binding trace_events,NativeModule internal/safe_globals,NativeModule internal/async_hooks,NativeModule internal/errors,Internal Binding uv,Binding buffer,Internal Binding async_wrap,Binding config,Binding icu,NativeModule util,NativeModule internal/validators,NativeModule internal/encoding,NativeModule internal/util,Binding constants,Internal Binding util,NativeModule internal/util/types,Internal Binding types,NativeModule buffer,NativeModule internal/buffer,NativeModule internal/process/per_thread,NativeModule internal/process/worker_thread_only,NativeModule internal/process/stdio,NativeModule internal/worker,NativeModule assert,NativeModule internal/assert,NativeModule fs,NativeModule path,NativeModul...
Reason parallel/test-gc-net-timeout
Type JS_TEST_FAILURE
Failed PR 7 (nodejs/node#22331, nodejs/node#23058, nodejs/node#22846, nodejs/node#22974, nodejs/node#22947, nodejs/node#23056, nodejs/node#23047)
Appeared test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-rackspace-win2012r2-x64-5, test-joyent-smartos16-x64-1, test-osuosl-aix61-ppc64_be-1, test-digitalocean-freebsd11-x64-2, test-digitalocean-ubuntu1804_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/17402/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17428/
Example 
not ok 1366 parallel/test-gc-net-timeout
  ---
  duration_ms: 120.88
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    We should do 500 requests
    Done: 10/500
    Collected: 0/22
    Done: 33/500
    Collected: 10/44
    Done: 55/500
    Collected: 33/66
    Done: 64/500
    Collected: 55/77
    Done: 77/500
    Collected: 64/99
    Done: 110/500
    Collected: 77/121
    Done: 132/500
    Collected: 110/143
    Done: 154/500
    Collected: 132/165
    Done: 187/500
    Collected: 154/187
    Done: 211/500
    Collected: 187/231
    Done: 231/500
    Collected: 211/253
    Done: 253/500
    Collected: 231/275
    Done: 275/500
    Collected: 253/286
    Done: 289/500
    Collected: 275/308
    Done: 321/500
    Collected: 289/341
    Done: 352/500
    Collected: 321/363
    Done: 375/500
    Collected: 352/396
    Done: 407/500
    Collected: 375/418
    Done: 431/500
    Collected: 407/440
    Done: 451/500
    Collected: 431/462
    Done: 455/500
    Collected: 451/484
    Done: 484/500
    Co...
Reason sequential/test-http2-session-timeout
Type JS_TEST_FAILURE
Failed PR 7 (nodejs/node#22953, nodejs/node#22951, nodejs/node#23010, nodejs/node#23048, nodejs/node#22331, nodejs/node#22413, nodejs/node#22914)
Appeared test-requireio_mhdawson-debian9-armv6l_pi1p-1, test-requireio_chrislea-debian9-armv6l_pi1p-1, test-requireio_securogroup-debian9-armv6l_pi1p-1
First CI https://ci.nodejs.org/job/node-test-pull-request/17379/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17409/
Example 
not ok 322 sequential/test-http2-session-timeout
  ---
  duration_ms: 6.310
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:126
      throw err;
      ^
    
    AssertionError [ERR_ASSERTION]: function should not have been called at /home/iojs/build/workspace/node-test-binary-arm/test/sequential/test-http2-session-timeout.js:9
        at Http2Server.mustNotCall (/home/iojs/build/workspace/node-test-binary-arm/test/common/index.js:407:12)
        at Http2Server.emit (events.js:182:13)
        at ServerHttp2Session.sessionOnTimeout (internal/http2/core.js:2631:15)
        at Object.onceWrapper (events.js:273:13)
        at ServerHttp2Session.emit (events.js:182:13)
        at ServerHttp2Session._onTimeout (internal/http2/core.js:1306:10)
        at ontimeout (timers.js:454:11)
        at tryOnTimeout (timers.js:326:5)
        at listOnTimeout (timers.js:300:5)
        at processTimers (timers.js:257:5)
  ...
Reason parallel/test-gc-http-client-connaborted
Type JS_TEST_FAILURE
Failed PR 5 (nodejs/node#22846, nodejs/node#23016, nodejs/node#23075, nodejs/node#23048, nodejs/node#23056)
Appeared test-rackspace-win2012r2-x64-5, test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-digitalocean-alpine37_container-x64-2, test-rackspace-win2008r2-x64-6
First CI https://ci.nodejs.org/job/node-test-pull-request/17407/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17430/
Example 
not ok 184 parallel/test-gc-http-client-connaborted
  ---
  duration_ms: 120.66
  severity: fail
  exitcode: 1
  stack: |-
    timeout
    We should do 500 requests
    Done: 44/500
    Collected: 0/99
    Done: 131/500
    Collected: 44/176
    Done: 219/500
    Collected: 131/264
    Done: 271/500
    Collected: 219/407
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
    Done: 271/500
    Collected: 271/500
   ...
Reason parallel/test-gc-http-client-onerror
Type JS_TEST_FAILURE
Failed PR 4 (nodejs/node#22846, nodejs/node#23016, nodejs/node#23075, nodejs/node#23056)
Appeared undefined, test-packetnet-centos7-arm64-1, test-digitalocean-alpine37_container-x64-2, test-digitalocean-alpine37_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/17407/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17426/
Example 
not ok 1395 parallel/test-gc-http-client-onerror
  ---
  duration_ms: 120.26
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    We should do 500 requests
    Done: 0/500
    Collected: 0/40
    Done: 50/500
    Collected: 0/90
    Done: 90/500
    Collected: 40/130
    Done: 140/500
    Collected: 80/180
    Done: 190/500
    Collected: 130/230
    Done: 243/500
    Collected: 180/290
    Done: 290/500
    Collected: 240/330
    Done: 335/500
    Collected: 280/380
    Done: 390/500
    Collected: 330/430
    Done: 417/500
    Collected: 380/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 417/500
    Collected: 417/500
    Done: 4...
Reason parallel/test-gc-http-client-timeout
Type JS_TEST_FAILURE
Failed PR 4 (nodejs/node#23058, nodejs/node#22906, nodejs/node#22947, nodejs/node#23075)
Appeared test-digitalocean-alpine37_container-x64-2, test-osuosl-aix61-ppc64_be-2, test-digitalocean-ubuntu1804_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/17403/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17420/
Example 
not ok 1186 parallel/test-gc-http-client-timeout
  ---
  duration_ms: 120.91
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    We should do 550 requests
    Done: 0/550
    Collected: 0/33
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    Done: 0/550
    Collected: 0/44
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    timeout (expected)
    ti...
Reason parallel/test-gc-http-client
Type JS_TEST_FAILURE
Failed PR 3 (nodejs/node#23072, nodejs/node#23048, nodejs/node#23056)
Appeared test-rackspace-win2012r2-x64-4, test-digitalocean-alpine37_container-x64-2, test-osuosl-aix61-ppc64_be-2
First CI https://ci.nodejs.org/job/node-test-pull-request/17417/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17425/
Example 
not ok 184 parallel/test-gc-http-client
  ---
  duration_ms: 120.81
  severity: fail
  exitcode: 1
  stack: |-
    timeout
    We should do 300 requests
    Done: 29/300
    Collected: 0/90
    Done: 59/300
    Collected: 29/120
    Done: 90/300
    Collected: 59/150
    Done: 104/300
    Collected: 90/180
    Done: 104/300
    Collected: 90/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
    Collected: 104/300
    Done: 104/300
  ...
Reason parallel/test-crypto-keygen
Type JS_TEST_FAILURE
Failed PR 2 (nodejs/node#22845, nodejs/node#22969)
Appeared test-packetnet-centos7-arm64-1, test-linuxonecc-rhel72-s390x-3, test-azure_msft-win10-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/17341/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17343/
Example 
not ok 2388 parallel/test-crypto-keygen
  ---
  duration_ms: 240.71
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    assert.js:662
        throw actual;
        ^
    
    Error: error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long
        at Sign.sign (internal/crypto/sig.js:83:27)
        at testSignVerify (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:43:42)
        at assert.throws (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:171:7)
        at getActual (assert.js:574:5)
        at Function.throws (assert.js:692:24)
        at AsyncWrap.common.mustCall (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:170:12)
        at AsyncWrap.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/common/index.js:350:15)
        at AsyncWrap.wrap.ondone (internal/crypto/...
Reason parallel/test-net-connect-options-port
Type JS_TEST_FAILURE
Failed PR 2 (nodejs/node#22331, nodejs/node#22846)
Appeared test-packetnet-ubuntu1604-arm64-2, test-digitalocean-alpine37_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17407/
Example 
not ok 2361 parallel/test-net-connect-options-port
  ---
  duration_ms: 240.140
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...

Build Failure

Reason sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory
Type BUILD_FAILURE
Failed PR 3 (nodejs/node#22891, nodejs/node#22969, nodejs/node#20713)
Appeared test-linuxonecc-rhel72-s390x-1
First CI https://ci.nodejs.org/job/node-test-pull-request/17334/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17336/
Example 
sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory
Reason ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
Type BUILD_FAILURE
Failed PR 2 (nodejs/node#22331, nodejs/node#22914)
Appeared test-requireio_davglass-debian9-arm64_pi3-1, test-requireio_securogroup-debian9-arm64_pi3-2, test-requireio_pivotalagency-debian9-arm64_pi3-2, test-requireio_williamkapke-debian9-arm64_pi3-3, test-requireio_kahwee-debian9-arm64_pi3-1, test-requireio_notthetup_sayanee-debian9-arm64_pi3-1, test-requireio_securogroup-debian9-armv6l_pi1p-2, test-requireio_securogroup-debian9-arm64_pi3-1
First CI https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17427/
Example 
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, /home/iojs/build/workspace/node-test-binary-arm/out/junit/test.xml is 29 min old

Notifying upstream projects of job completion
Finished: FAILURE
Reason error: pathspec ... did not match any file(s) known to git.
Type BUILD_FAILURE
Failed PR 2 (nodejs/node#22969, nodejs/node#22418)
Appeared test-azure_msft-win10-x64-3, test-azure_msft-win10-x64-4, test-rackspace-win2008r2-x64-2, test-rackspace-win2008r2-x64-1, test-azure_msft-win2008r2-x64-1, test-azure_msft-win2008r2-x64-2, test-rackspace-win2012r2-x64-9, test-azure_msft-win2012r2-x64-2, test-rackspace-win2012r2-x64-4, test-rackspace-win2012r2-x64-2, test-rackspace-win2012r2-x64-5, test-azure_msft-win2012r2-x64-1, test-rackspace-win2012r2-x64-3, test-rackspace-win2012r2-x64-8, test-azure_msft-win2016-x64-2, test-azure_msft-win2016-x64-5, test-azure_msft-win2016-x64-3, test-azure_msft-win2016-x64-4, test-azure_msft-win10-x64-2, test-rackspace-win2008r2-x64-5, test-rackspace-win2012r2-x64-7, test-rackspace-win2012r2-x64-6
First CI https://ci.nodejs.org/job/node-test-pull-request/17347/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17359/
Example 
error: pathspec 'refs/remotes/jenkins_tmp/jenkins-node-test-commit-windows-fanned-94402b043f54b82c706b86f02d05b5290e48c872-bin-win-vcbt2015' did not match any file(s) known to git.

c:\workspace\node-test-binary-windows>if 1 NEQ 0 exit /b 1 
Build step 'Execute Windows batch command' marked build as failure
Run condition [Always] enabling perform for step [[Execute Windows batch command, Execute shell]]
[node-test-binary-windows] $ cmd /c call C:\Users\ADMINI~1\AppData\Local\Temp\jenkins984036347200566261.bat
Reason fatal: Not a git repository (or any of the parent directories): .git
Type BUILD_FAILURE
Failed PR 2 (nodejs/node#22950, nodejs/node#23004)
Appeared test-rackspace-win2008r2-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/17357/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17358/
Example 
fatal: Not a git repository (or any of the parent directories): .git
Reason sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
Type BUILD_FAILURE
Failed PR 2 (nodejs/node#22331, nodejs/node#22914)
Appeared test-requireio_chrislea-debian9-armv6l_pi1p-1, test-requireio_svincent-debian9-armv7l_pi2-1, test-requireio_mininodes-debian9-armv6l_pi1p-1, test-requireio_mcollina-debian9-armv7l_pi2-1, test-requireio_securogroup-debian9-armv6l_pi1p-1, test-requireio_rvagg-debian9-armv7l_pi2-1, test-requireio_bengl-debian9-armv6l_pi1p-2, test-requireio_joeyvandijk-debian9-armv7l_pi2-2, test-requireio_indieisaconcept-debian9-armv6l_pi1p-1, test-requireio_joeyvandijk-debian9-armv7l_pi2-1, test-requireio_mhdawson-debian9-armv6l_pi1p-1, test-requireio_jasnell-debian9-armv7l_pi2-1, test-requireio_ceejbot-debian9-armv7l_pi2-1
First CI https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17427/
Example 
sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory

Git Failure

Jenkins Failure

Reason java.io.IOException: Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 2 (nodejs/node#22997, nodejs/node#22998)
Appeared test-macstadium-macos10.11-x64-2, test-rackspace-win2008r2-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/17356/
Last CI https://ci.nodejs.org/job/node-test-pull-request/17399/
Example 
java.io.IOException: Backing channel 'JNLP4-connect connection from 207.254.58.162/207.254.58.162:62697' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
	at com.sun.proxy.$Proxy85.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129)

Activity

  1. joyeecheung commented on Sep 25, 2018

    @joyeecheung
    joyeecheung
    on Sep 25, 2018
    MemberAuthor

    Progress

    • java.io.IOException: Backing channel 'JNLP4-connect connection from ... is disconnected.
    • parallel/test-net-connect-options-port

    In progress

    Fixed

    Resume failure

    Tracking in nodejs/build#1496

    • fatal: Not a git repository (or any of the parent directories): .git
    • error: pathspec ... did not match any file(s) known to git.
    • ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
    • sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory

  2. richardlau commented on Sep 25, 2018

    @richardlau
    richardlau
    on Sep 25, 2018
    Member

    sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory was addressed by nodejs/build#1501.

  3. refack commented on Sep 25, 2018

    @refack
    refack
    on Sep 25, 2018

    parallel/test-gc-http-client - nodejs/node#22336
    parallel/test-gc-http-client-timeout - nodejs/node#23066
    parallel/test-gc-net-timeout - nodejs/node#23067

    @joyeecheung could you update the pattern matching to ignore
    sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
    and
    ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
    Those are errors that don't cause build FAILURE. In those cases the error is probably something else. In the case of node-test-binary-arm it usually means that a resume build failed because the temporary branch passing the binaries was deleted during the original build.

  4. joyeecheung commented on Sep 25, 2018

    @joyeecheung
    joyeecheung
    on Sep 25, 2018
    MemberAuthor

    @refack I can add an ad-hoc pattern for now (maybe replace/append that line with (possible resume failures)), yes, but I am not sure how long that association will hold? (Maybe I can just add a TODO in the tool and lift it when there is a new association)

  5. refack commented on Sep 25, 2018

    @refack
    refack
    on Sep 25, 2018

    @joyeecheung What I meant is that in the current configuration of CI
    sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
    and
    ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
    should not cause the build to fail.

    You are right about associating them with a real cause, "failed resume" is just a guess.

  6. joyeecheung commented on Sep 26, 2018

    @joyeecheung
    joyeecheung
    on Sep 26, 2018
    MemberAuthor

    @refack I see. The reasons here are not accurate reasons, they are more like keys to group the errors by.
    Just think of another solution: I can check if the PR build comes from another PR build (i.e. a resumed build) and check if the failed job is a fanned one. If so I'll tentatively mark them as RESUME_FAILURE (a new category). That's how my brain currently works anyway.

  7. refack commented on Sep 26, 2018

    @refack
    refack
    on Sep 26, 2018

    From a big picture POV, I would like to tip my 🎩 to @joyeecheung @Trott @joaocgreis and rest of @nodejs/build that (AFAIR for the first time ) made it so infra failures do not dominate this list. One small step...

  8. Trott commented on Sep 26, 2018

    @Trott
    Trott
    on Sep 26, 2018
    Member

    @refack Right back at you times 💯

  9. joyeecheung
    closed this as completedon Oct 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions