Skip to content

Support packages with sources not in src #16

Closed

Description

@f-f

For any package published on Bower right now it is possible to assume that one would get their source files with the following glob: src/**/*.purs

In purescript/spago#288 we explored what it would take to lift that limitation (it's a good read to get some context on what follows here)

One of the main blockers for that was the assumption coming from Bower registry that "one package = one repo". Since here we're moving away from the git-based model and towards packaging sources in immutable archives, I thought we could take this occasion to allow publishing packages with sources in arbitrary paths.
This would allow things like publishing several package from the same monorepo.

There has been some discussion in #4 about this, and we're spinning off this issue to keep track of a concern expressed there:

The issue I'm worried about with sources is basically for setups which don't use Spago; I want to avoid making life difficult for people who are still using other things like psc-package or pulp/bower. Yes, it's probably not too much effort to switch a build script of just a few lines in most cases, especially with spago init, but it's harder if the assumption is built into more complex tooling, such as, say, editor plugins. This assumption is built into purs publish right now too. It becomes significantly more complicated if these build scripts or tooling can't/don't want to assume the presence of Spago on the machine in use, or don't want to add Dhall as a dependency.

I would be happy if we could have the curator require that the sources for the lib target for any published package are just ["src/**/*.purs"], at least to start with; I could see us relaxing that later once most of the ecosystem is using this registry.

Activity

  1. hdgarrood commented on Apr 26, 2020

    @hdgarrood
    Contributor

    This would allow things like publishing several package from the same monorepo.

    This ought to be possible anyway with the new design even if we require source files to be in src, right? If you have multiple packages in a monorepo, I would expect that they would each be in separate directories with separate package manifest files, and the sources key in the package manifest file would be interpreted relative to the directory in which the manifest file lives.

  2. added this to the v1 milestone on May 20, 2020
  3. f-f commented on Sep 27, 2020

    @f-f
    MemberAuthor

    There is now (as of #76) support for "multiple manifests in the same repo" via the subdir key in the Repo type.

    However, I think the concern expressed of "we should limit to one package only until the ecosystem is using this" is still valid, and more specifically I'd like to have support for this in Spago before we allow the publishing of packages that specify a subdir. Makes sense?

  4. hdgarrood commented on Sep 27, 2020

    @hdgarrood
    Contributor

    I still don't understand what we gain by making source paths configurable by a package author, over having the rule be "the visible source files are always those which are located in the src folder relative to the package manifest file, and only those". That still permits managing a number of packages together in a monorepo, and as far as I'm aware that is the only concern which has been brought up in the context of this issue.

  5. f-f commented on Sep 27, 2020

    @f-f
    MemberAuthor

    How do we find out the locations of several package manifests?

  6. hdgarrood commented on Sep 27, 2020

    @hdgarrood
    Contributor

    I think the subdir key in Repo is good; I agree that we should allow the author to put a package manifest file in an arbitrary subdirectory of a repo. I just think that package authors should not have freedom to put source files wherever they want beyond that; I think source files should always be placed in a src/ directory next to the package manifest file.

  7. f-f commented on Sep 27, 2020

    @f-f
    MemberAuthor

    Oh I see, this makes sense. Since the "official package source" is going to be contained in a Target called lib (or whatever name we want to pick for that) should we just enforce that the sources for that Target are exactly ["src/**/*.purs"]?

  8. hdgarrood commented on Sep 27, 2020

    @hdgarrood
    Contributor

    Yeah, exactly.

  9. f-f commented on Sep 28, 2020

    @f-f
    MemberAuthor

    Agreed 👏

    So I'd say the actionable points for this issue are:

    1. add a CI check to ensure that a package being added has sources: ["src/**/*.purs"] in its lib Target
    2. add a CI check to ensure that no subdir is present to new packages until we are confident that downstream tooling can deal with it

    Makes sense?

  10. hdgarrood commented on Sep 28, 2020

    @hdgarrood
    Contributor

    Sounds good!

  11. added
    help wantedExtra attention is needed
    and removed on Sep 28, 2020
  12. hdgarrood commented on Sep 28, 2020

    @hdgarrood
    Contributor

    Oh also, how should tagging work for libraries in monorepos? Right now we require tags to be of the form vX.Y.Z in package-sets, right? But if there are different libraries together in one repo, that won't work (unless the libraries' versions are all kept in sync).

  13. 1 remaining item

  14. hdgarrood commented on Oct 24, 2020

    @hdgarrood
    Contributor

    That sounds good to me 👍

  15. f-f commented on Oct 26, 2020

    @f-f
    MemberAuthor

    @hdgarrood cool, I'll update the spec to use commit hashes instead of tags

    I just realized this feature would enable us to possibly keep all the core libraries in a monorepo, making cross-library updates (such as in the case of whole-ecosystem-breakage) much easier, as we could compile all the libraries together in the same build - cc @JordanMartinez

  16. JordanMartinez commented on Oct 26, 2020

    @JordanMartinez
    Contributor

    @f-f While it might make it possible to build things together via a monorepo, the same can't be said for issue trackers, right? Meaning, there isn't a corresponding way to distinguish issues within the issue tracker to repo-specific issues (or at least in a good way).

  17. f-f commented on Oct 26, 2020

    @f-f
    MemberAuthor

    All the monorepos I've seen use labels to classify tickets. In practice that seems to work well, and personally I'd be more concerned about notification spam, i.e. one might not want to be notified for all the things happening across all the core libraries.

    In any case I'm not proposing anything, mine was just a note admiring how this feature could enable some interesting workflows that we don't have available right now 🙂

  18. JordanMartinez commented on Oct 27, 2020

    @JordanMartinez
    Contributor

    I think git (or perhaps GitHub) needs to have first-class support for mono-repos.

    Regardless, yeah, there's a lot of things this can enable, such as running all tests on packages in the package sets. It'll be interesting to explore.

  19. added
    document meImprovements or additions to documentation
    and removed
    help wantedExtra attention is needed
    on Nov 11, 2020
  20. self-assigned this
    on Nov 11, 2020
  21. f-f commented on Dec 15, 2020

    @f-f
    MemberAuthor

    Update:

    I'll update the spec to use commit hashes instead of tags

    This means also updating the version field here to be called ref

  22. f-f commented on Dec 19, 2020

    @f-f
    MemberAuthor

    I updated the spec in #107, so the actionable points for this issue remain:

    1. add a CI check to ensure that a package being added has sources: ["src/**/*.purs"] in its lib Target
    2. add a CI check to ensure that no subdir is present to new packages until we are confident that downstream tooling can deal with it
  23. removed
    document meImprovements or additions to documentation
    on Dec 19, 2020
  24. removed their assignment
    on Dec 19, 2020
  25. added a commit that references this issue on Jan 24, 2021
    971f498
  26. wclr commented on May 8, 2021

    @wclr

    It is good to find those issues, which I didn't consider prior to that.

    I believe a package is basically a container for just two things: package manifest file and src directory (which should be the standard restricted name), what else is needed for a PS package to be a package (huh there is also need to be some standard stuff like readme/licence)? Until it is not supposed to be something more complex and bloated where you can place whatever data you want, but I believe it is not the case for PureScript packages.

    I just realized this feature would enable us to possibly keep all the core libraries in a monorepo, making cross-library updates (such as in the case of whole-ecosystem-breakage) much easier, as we could compile all the libraries together in the same build -

    I'm glad that there is an understanding or at least a suspicion that a monorepo based model may be a good thing for maintaining PS libraries. That is what I was talking about in the proposal that assumes that some of the existing packages have to be decomposed and split apart. This wouldn't lead to a drastic increase in the number of packages as a whole because it really refers only to a limited number of the bloated packages, but certainly, if we consider the model package == repo, this would be almost ridiculous to have a separate repo for each package.

    I also believe this presumption (that package == repo) as well leads to a desire to combine and merge packages, instead of decomposing them, and when you still have to decompose, it causes additional pain of setting up repos with the infrastructure for each, etc. With the monorepo approach, such things become much more obvious and easy (even though it requires some effort to setup).

    Oh also, how should tagging work for libraries in monorepos? Right now we require tags to be of the form vX.Y.Z in package-sets, right? But if there are different libraries together in one repo, that won't work (unless the libraries' versions are all kept in sync).

    A simple tagging strategy surely doesn't play well with monorepos. Though as you can have multiple tags it could be quite ok if it would be really needed. But for packages that are supposed to be published in the registry, there is no reason to be strongly coupled with the repository that holds the source.

    and personally I'd be more concerned about notification spam, i.e. one might not want to be notified for all the things happening across all the core libraries.

    I think in general for big repos people don't want to have all the notifications enabled (only the most cautious/responsible ones). If there is a need to draw the attention of a particular person it is achieved using the name mention, and more accurate automation and targeting can be accomplished with custom Github bots.

    I think git (or perhaps GitHub) needs to have first-class support for mono-repos.

    I don't think that something meaningful in this direction is going to happen soon, just because actually support for mono repo of git itself is good enough, all the other stuff is achieved by external tooling (as well as mentioned GitHub bots), which should not be complex, and usually still crafted for a particular use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions