Skip to content

The diagram exporter 404s on the dev box: download/current/diagram was deleted #230

Closed

Description

@adamjohnwright

GET /ContentService/exporter/diagram/<id>.{png,svg,jpeg} returns 404 on the dev
box for every pathway, while production returns 200. It is not an EHLD or
illustration problem, and not an application bug — the Java ContentService reads
these from disk and the files are gone.

The diagnosis, which took three steps

The error names its own path, which is the useful part:

Could not read diagram json for pathway
  /usr/local/reactomes/Reactome/production/Website/static/download/current/diagram/R-HSA-109606.json
  • download/current is a symlink to release 97, which is what this box serves
  • 97/ still holds ehld/ and the 528 MB reactome.graphdb.dump
  • 97/diagram/ had been deleted to reclaim space

Caching the .json files moved the error rather than clearing it, because each
diagram needs two files
:

R-HSA-109606.json         286,603 bytes
R-HSA-109606.graph.json    ~46,000 bytes

With both present the exporter answers 200 immediately.

What was done on the box, and what it is not

Cached the 11 pathways the e2e suite uses — 1,262 KB of .json plus 506 KB of
.graph.json — from https://download.reactome.org/97/diagram/, which still
publishes release 97. The full diagram/ directory for a release is gigabytes
and this host has 11 GB free, so restoring all of it is not an option.

This is a local workaround. It lives on one machine and disappears the next
time the directory is cleaned. It is recorded here so the next person does not
rediscover .graph.json from a 404.

Why it matters before curator testing

Downloading an illustration as SVG, PNG or JPEG is broken on beta right now
and works on production. A curator will find that and report it as a website bug.
Ordinary diagram downloads are unaffected because the app exports those
client-side from cytoscape — an EHLD has no cytoscape graph, so it falls through
to this endpoint.

Worth a line in CURATOR-REPORT.md either way, since the cache covers only the
pathways our tests touch.

The actual fix

Serve these from S3 rather than from a local directory, with a local cache for
what is used. That fits the existing plan for node fronting the exporters, and it
removes a class of failure where the site is fine and the backend quietly is not.

Activity

  1. adamjohnwright commented on Sep 20, 2026

    @adamjohnwright
    ContributorAuthor

    Re-measured today. Three things have changed since this was written, one of them making it worse.

    ehld/ is gone too. The issue records 97/ as still holding ehld/. It is now empty — 0 files. diagram/ holds 11 of the 9,559 pathways that have one, and those 11 are the ids our own tests use, which is a trap worth naming: R-HSA-109606 is one of them, so the first spot check anyone runs returns 200 and the exporter looks healthy.

    The image formats no longer matter, because nothing asks the exporter for them. Diagram downloads go to the render service, and illustrated pathways are now made in the page (#271). The detail-diagram fallback still points here and is inoperative on this box; its comment now says so rather than claiming the exporter "still draws something for every pathway".

    SBGN is the part that is still broken, and it 500s rather than 404s. That is a download button on the detail bar and in the download tab, and it tracks the missing files exactly:

    R-HSA-73894    sbgn 500   (diagram file on disk: no)
    R-HSA-1640170  sbgn 500   (diagram file on disk: no)
    R-HSA-168256   sbgn 500   (diagram file on disk: no)
    R-HSA-69306    sbgn 200   (diagram file on disk: yes)
    R-HSA-109581   sbgn 200   (diagram file on disk: yes)
    

    SBML and PDF are fine without the files — SBML answers in 27s, which is its own question. The site itself is unaffected throughout, because the app fetches diagram JSON from download.reactome.org directly rather than from this disk; only the Java exporter reads locally.

    What restoring would cost. Sampled 30 pathways spread across all 9,559 (mean 56 KB per file, two files each): 1.02 GB. The box has 5.6 GB free, so it fits, but not comfortably — and another deploy here is already waiting on 6 GB. Removing the one superseded docker image reclaimed nothing, its layers being shared with the current one.

    So the decision is a resource one rather than a technical one: 1 GB to make SBGN work for 9,548 pathways it currently fails for, on a disk that is the binding constraint. Per the port plan SBGN stays with Java, so it needs these files — the alternative is teaching that exporter to read from the download host, which is a change in a different repository.

  2. adamjohnwright commented on Sep 20, 2026

    @adamjohnwright
    ContributorAuthor

    Correcting my earlier comment, and answering why the exporter needs these files at all.

    The empty ehld/ is not a fault. I listed it as making this worse; it does not. Illustrations live on download.reactome.org and the app fetches them from there. The only things that read a local ehld/ were the Java image exporters — exporter/diagram/<id>.{png,svg,jpeg,pptx} — and nothing asks for those any more: diagram figures come from the render service and illustrations are made in the page (#271). An empty ehld/ on this box is correct.

    Why SBGN needs the diagram JSON on the fly, from ExportManager.getDiagram():

    if (event instanceof Pathway) {
        File aux = new File(diagramJsonFolder + "/" + diagram + ".json");
        return DiagramFactory.getDiagram(IOUtils.toString(new FileInputStream(aux), UTF_8));
    } catch (IOException | DeserializationException e) {
        errorLogger.error(e.getMessage(), e); return null;
    } else {
        ReactionLikeEvent rle = (ReactionLikeEvent) event;
        return reactionExporter.getReactionDiagram(reactionExporter.getReactionLayout(rle));
    }

    SBGN carries layout, and a pathway's layout is a curated artefact that exists only as diagram JSON — it cannot be derived from the graph. A reaction's layout is computed from the graph, by the else branch, which is why reaction SBGN works here and pathway SBGN does not.

    The 500 is not a "file not found" path either. A missing file makes getDiagram return null, and that null goes straight into new SbgnConverter(null).

    Production is fast because it is not generating anything. getCachedFile reads ContentService/exporter/<version>/sbgn/, and its comment says that folder "will be created during release phase". Production answers in 0.15s. Here the folder held one file from 3 Sep — the two beside it now are from 04:47 today, written by my own test requests, so the cache did not pre-exist and the local diagram JSON really is what decides.

    Sizing, measured rather than extrapolated: 9,559 pathways have their own diagram — 1,382 human and 8,152 computationally inferred into other species. All of them: 1.02 GB. Human only: 252 MB, mean 94 KB per file over a 25-pathway sample.

    ~/restore-diagram-files.mjs fetches the missing ones into the download directory, atomically and resumably. Since nothing readable on the box says which directory the exporter is configured to read — service.properties is tomcat:tomcat 0640 — it takes --only <stId>, so one pathway can prove the path for 90 KB before committing to the rest.

  3. adamjohnwright commented on Sep 20, 2026

    @adamjohnwright
    ContributorAuthor

    Fixed. The files are restored and SBGN works again.

    19,092 files fetched from the release bucket into download/current/diagram/, 0.96 GB, bringing it to 19,118 — two per pathway for all 9,559. Checked across 14 pathways spanning 13 species, human and computationally inferred, including an illustrated one:

    200      1861b  R-BTA-109581   Bos taurus (inferred)
    200     88597b  R-DME-5654736  Drosophila melanogaster (inferred)
    200    137826b  R-RNO-1483249  Rattus norvegicus (inferred)
    200       627b  R-HSA-9687139  Homo sapiens [ehld]
    ...
    14 of 14 return real SBGN
    

    What this was, in the end. Not the image exporters — nothing asks those for anything now. It was /ContentService/exporter/event/<id>.sbgn, a button on the download panel and the detail bar, returning 500 for 9,548 of 9,559 pathways. A pathway's SBGN carries layout, that layout is a curated artefact stored as diagram json, and ExportManager.getDiagram returns null when the file is missing, which then goes into new SbgnConverter(null). A reaction needs no file, which is why reaction SBGN never broke.

    Why it went unnoticed for weeks, which is the part worth keeping. The eleven pathways whose files survived are the ids our own fixtures use. Every spot check anyone ran — including my first one here, on R-HSA-109606 — came back healthy. e2e/release/sbgn-export.spec.ts now asserts on Cell Cycle, chosen precisely because it is not a fixture, and asserts the pathway and reaction cases separately so a future failure says which half is wrong.

    And the ehld/ directory is fine empty, as noted above — illustrations live on the download host, and the only things that read a local copy were the image exporters we no longer use.

    ~/restore-diagram-files.sh is on the box if this happens again after a release; it fetches only what is absent, so it is safe to re-run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions