Skip to content

Gradle cache increases by 400 MB every time updating Gradle wrapper version #269

Description

@DevCharly

Description:
When updating the Gradle wrapper to a new version (changed gradle-wrapper.properties), then

  1. setup-java@v2 first restores the gradle cache from previous run (why??)
  2. new Gradle version is downloaded and build
  3. then setup-java@v2 creates a new cache file (~800 MB) which is about 400 MB larger than before because it contains two Gradle versions

Changing Gradle wrapper version again increases the cache file size by 400 MB to 1200 MB, and so on...

Here is a run where it happened: https://github.com/JFormDesigner/FlatLaf/runs/4705134891

In "Setup Java 17" the outdated cache is restored (https://github.com/JFormDesigner/FlatLaf/runs/4705134891#step:4:35):

Cache Size: ~425 MB (445413982 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/19b74bcf-0264-4065-b1ef-448c93c2d4fc/cache.tzst -P -C /home/runner/work/FlatLaf/FlatLaf
Cache restored successfully
Cache restored from key: setup-java-Linux-gradle-707ed6f8250ea8ec2ee660640962308e354fe354d36338715ef1bcab4fcc837f

In "Post Setup Java 17" a new cache with ~836 MB is created (https://github.com/JFormDesigner/FlatLaf/runs/4705134891#step:11:5):

Cache Size: ~836 MB (876885346 B)
Cache saved successfully
Cache saved with the key: setup-java-Linux-gradle-af7e537f7a03362010edc6596ab7fba05bdfb901982028cbd2d21e5291cd932d

Note the different hash keys in the cache file names!

Task version:
actions/checkout@v2

Platform:

  • Ubuntu
  • macOS
  • Windows

Runner type:

  • Hosted
  • Self-hosted

Repro steps:
See above description.

Expected behavior:
When changing gradle-wrapper.properties (or build.gradle), then the cache should be not restored.

Actual behavior:
Cache is restored even if gradle-wrapper.properties (or build.gradle) changed, which increases the size of the cache file by aboud 400 MB every time a new Gradle wrapper version is used.

Activity

  1. dmitry-shibanov commented on Jan 6, 2022

    @dmitry-shibanov
    Contributor

    Hello @DevCharly. Thank you for your report. We'll investigate the issue.

  2. schuenadel commented on Jan 13, 2022

    @schuenadel
    Contributor

    Expected behavior:
    When changing gradle-wrapper.properties (or build.gradle), then the cache should be not restored.

    I think so too, especially if there is not other cache eviction happening than this, which would mean it grows "forever" (if you access it once a week).
    My suggestion would be to remove the restoreKeys (3rd) parameter in here, so that whenever the cache key changes you start with a completely clean cache. But on the other hand that means for every change all(!) dependencies are downloaded again.

  3. DevCharly commented on Jan 13, 2022

    @DevCharly
    Author

    Shouldn't the cache ID calculated from the (gradle) files in working directory, which was previously checked out from git?

    Currently it seems that the cache ID from the previous run is used,
    which always restores cache from previous run.
    Seems to be useless because the cache never becomes dirty...

    But on the other hand that means for every change all(!) dependencies are downloaded again.

    Isn't this the expected behavior?
    The cache does not know what is still needed and what can be removed
    when you change the version of a dependency or remove a dependency.

  4. schuenadel commented on Jan 14, 2022

    @schuenadel
    Contributor

    Shouldn't the cache ID calculated from the (gradle) files in working directory, which was previously checked out from git?

    It is, but when no cache with that id is found, like when you run the first time with a new dependency configuration, it falls back to a cache from a previous run. That fallback is defined with the restoreKeys parameter, which I suggested above to remove.

    But on the other hand that means for every change all(!) dependencies are downloaded again.

    Isn't this the expected behavior?

    One could argue that when you change only one small dependency out of many others it may be more efficient to keep 2 versions of that in the cache instead of downloading all again. But as you said, when the cache never becomes dirty I would also prefer to start over with a completely empty cache whenever something changes.

  5. schuenadel commented on Jan 25, 2022

    @schuenadel
    Contributor

    I put my suggestion into a PR here to make it more clear what I mean to prevent this problem.

  6. devminded commented on Jan 29, 2022

    @devminded

    After a quick scan through the code it looks like it creates a single cache-key and includes both build/dependency caches and wrapper cache in said key.

    Would it not make more sense to have two separate caches/keys with different restore policies? In addition to that, when saving the build/dependency caches we should be able to exclude files older than a configurable number of hours/days. That way old dependencies will be evicted over time and we can have the best of both worlds.

    Same issue will need to be solved within #245 .

  7. devminded commented on Feb 14, 2022

    @devminded

    What is the policy for bumping?

    This is a bump!

  8. mandrachek commented on Apr 14, 2022

    @mandrachek

    Is this why my cache is now 3GB in size, actions/setup-java@v3 takes ~12 minutes to run (downloading and restoring the cache), and why my post run setup-java action is failing?

    Is there some way to purge the cache and get a clean slate at least for the time being?

  9. fl250144 commented on Apr 28, 2022

    @fl250144

    Is this why my cache is now 3GB in size, actions/setup-java@v3 takes ~12 minutes to run (downloading and restoring the cache), and why my post run setup-java action is failing?

    Is there some way to purge the cache and get a clean slate at least for the time being?

    i have the same question...

  10. schuenadel commented on Apr 29, 2022

    @schuenadel
    Contributor

    After a quick scan through the code it looks like it creates a single cache-key and includes both build/dependency caches and wrapper cache in said key.

    Yes, that is true, but that did not change in the PR.

    Would it not make more sense to have two separate caches/keys with different restore policies?

    I also think that could make sense. But I would see that as additional improvement, which could be addressed in a separate PR. I think your issue actions/cache#788 is already going in that direction.

  11. schuenadel commented on Apr 29, 2022

    @schuenadel
    Contributor

    @mandrachek and @fl250144

    Is this why my cache is now 3GB in size, actions/setup-java@v3 takes ~12 minutes to run

    I don't know your setup, but the cache filling up over time sounds like you have this issue.

    Is there some way to purge the cache and get a clean slate at least for the time being?

    One thing you could do for now is to not use the built-in cache of actions/setup-java but instead use actions/cache directly. There you can define cache keys to fit your needs. Maybe something time-related, like week number to make sure the cache is dropped at least once per week. Of course this is not ideal, but maybe helps you for now.

  12. mandrachek commented on Apr 29, 2022

    @mandrachek

    @schuenadel - that's exactly the approach I wound up taking.

    @fl250144 - here's what I used instead of the built-in cache:

      # This works around an issue in setup-java which allows the dependency cache to grow each time the gradle wrapper is upgraded 
     ​      - ​name​: ​Cache Gradle Wrapper and Dependencies 
     ​        ​uses​: ​actions/cache@v3 
     ​        ​with​: 
     ​          ​path​: ​| 
     ​            ~/.gradle/caches 
     ​            ~/.gradle/wrapper 
     ​          ​key​: ​${{ runner.os }}-gradle-cache-${{ hashFiles('**/*.gradle', '**/*.gradle.kts', '**/gradle/wrapper/gradle-wrapper.properties') }}

    It adds the hash of gradle-wrapper.properties, so when the gradle version is updated, the cache is rebuilt.

    My cache is now back down under a gig.

  13. 3 remaining items

  14. bishal-pdMSFT commented on Jun 30, 2022

    @bishal-pdMSFT

    I think so too, especially if there is not other cache eviction happening than this, which would mean it grows "forever" (if you access it once a week).

    There is a nuance. If there are more recently used caches in a repo and the total size is more than 10 GB even a weekly accessed cache will get evicted. Essentially the eviction logic find caches which need to be purged to bring total cache usage size to 10GB and it purges the ones which are oldest till the size goal is met.

  15. dsame commented on Jul 11, 2022

    @dsame
    Contributor

    Hello @DevCharly

    It seems i found the root cause and can suggest at least the workaround which can be called a solution as well.

    The origin of the problem is the logic of actions/cache restore: it DO restores the most recent cache even if the cache-key was not found.

    As a result, despite gradle/wrapper/gradle-wrapper.properties has been changed to the new version of the gradle and cash-key has changed the old cache with the previous gradle still restored and new gradle jar with the dependencies files does not replace the old one, but is added to the .gradle/cache folder.

    Thus, to resolve the problem it is necessary either 1) changes the logic of actions/setup or 2) add a cleaning step to actions/java-setup

    My opinion is: neither of above can be applied because of 1) is breaking change and 2) is not flexible enough

    I'd suggest to add the project specific step to the pipeline, something like:

            - uses: actions/setup-java@v3
              id: setup-java
              with:
                java-version: '8'
                distribution: 'zulu'
    
            - name: Force clear gradle cache
              if: steps.setup-java.outputs.cache-hit != 'true'
              run: rm -r ~/.gradle/caches
    

    This way completely solves the issue but leaves an ability to fine tuning the specific build.
    Any thoughts?

  16. dsame commented on Jul 16, 2022

    @dsame
    Contributor

    I put my suggestion into a PR here to make it more clear what I mean to prevent this problem.

    The PR is merged now, and the workaround i've suggested in the reply will become unnecessary on next release.

  17. dsame commented on Sep 12, 2022

    @dsame
    Contributor

    The issue is to be closed due to the merged PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions