Metadata merge precedence, and setting default metadata for custom formats

[davidskalinder]

Description

I'm trying to write an article in a single source document and prepare it for submission to several different academic journals with different formatting guidelines. As far as I can tell (though someone please correct me if I'm wrong!), the best way to do this in Quarto is to install or create, within the directory containing the document file, one custom format extension for each set of journal guidelines.

Of course, some journals are silent on how to format certain elements of an article, and so I'd like to have a way to set default options for journals where that is the case. For example, if I want to prepare for the possibility of having to send an article to five journals, then I might want to use custom Citation Style Language file for the two of those five journals that provide them, but to use a standard CMOS CSL file otherwise.

I had hoped that I could do this by carefully choosing the placement of the default option in the document source file. However, it seems that when Quarto adds YAML from custom formats, it always merges in the custom-format YAML first. As a result, any options set in the main document clobber the options set in the custom format, so they can't be used as a fallback.

The example below shows this using an arbitrary metadata key test-meta that takes sequence values to show the merge order, and another key test-metaB that takes string values to show how (as expected) the last value in the merge order clobbers the earlier setting.

Main document file: /test.qmd:

---
test-meta: [MAINDOCTOPSETTING]
test-metaB: MAINDOCTOPSETTING
format:
  myext-html:
    test-meta: [MAINFORMATSETTING]
    test-metaB: MAINFORMATSETTING
---

Show the merge order:

{{< meta test-meta.1 >}}

{{< meta test-meta.2 >}}

{{< meta test-meta.3 >}}

Show how non-sequence metadata gets clobbered:

{{< meta test-metaB >}}

YAML file for the custom format, at _extenstions/myext/_extension.yml:

contributes:
  formats:
    html:
      test-meta: [EXTFORMATSETTING]
      test-metaB: EXTFORMATSETTING

This produces an HTML file with the following:

Show the merge order:

EXTFORMATSETTING

MAINDOCTOPSETTING

MAINFORMATSETTING

Show how non-sequence metadata gets clobbered:

MAINFORMATSETTING

This task seems like a core piece of Quarto's intended function, so I'm surprised that I haven't been able to find a way to do this. If anyone could enlighten me, or point me toward a resource that I missed, I'd appreciate it!

[mcanouil]

There are many ways:

• use profile
• use _metadata.yml and different directories
• use conditional content
• use Lua filter to handle metadata the way you want
• ...

---

Side note: coming from academics, you should not spend time making your article work for five journals because you cannot submit to one until you've been rejected from the previous one.
If you predict with certainty that you'll be rejected from four, then there is possibly another issue with your article.

[davidskalinder]

Thanks for the extremely quick response!

• use profile
• use _metadata.yml and different directories

I haven't looked carefully at setting up a Quarto project here yet since as far as I can tell this task (with one source file and several outputs) doesn't fall into any of the listed use cases of Quarto projects. However it sounds like it's worth me diving into that toolbox and seeing what I can find.

• use conditional content

It's not clear to me how this would work since as far as I've been able to discover metadata options can't be set conditionally (without resorting to, e.g., R code)? And so I don't think I would be able to specify one CSL file for one format and a different one for another? But it's entirely possible that I'm missing something about how this works.

• use Lua filter to handle metadata the way you want

Of course this is always an option! But quite a difficult and error-prone option, especially for what seems like it'd be a very common task?

Side note: coming from academics, you should not spend time making your article work for five journals because you cannot submit to one until you've been rejected from the previous one. If you predict with certainty that you'll be rejected from four, then there is possibly another issue with your article.

Thank you for your advice on my career decisions! I am well aware of how the journal submission process works. Of course I can't predict anything with certainty, but in my field multiple rejections (not to mention the need for other formats for preprints, circulated drafts, dissertation documents, etc.) are very common, and so the core promise of Quarto as a write-once-publish-anywhere tool for use cases like this is quite valuable indeed.

Anyway it sounds like housing the single source doc within a Quarto project is the most promising approach here, so I'll look into that. Thanks again!

[mcanouil]

I'm pretty sure "default" is well suited, especially since that's the default and what is used (ish) for single documents.

About metadata, your example does not seem reproducible or at least I cannot reproduce any behaviour like the one you showed.
I also don't understand why you are using the meta shortcode using indices while your keys are atomic.

If you set something globally, it overwrites/sets defaults the same way in Quarto and extensions.
Not sure why you would assume it wouldn't in extensions.
If you set it under the format, it takes precedence.

edit: I did't not see with your nearly equal values in all caps, that the metadata was an array.
With arrays, indeed, it's merged.

[mcanouil]

Here is a minimal an simple reproducible example, it does need extensions, only arrays and different scopes:

---
meta:
  - "C"
format:
  html:
    meta:
      - "B"
---

{{< meta meta.1 >}}

{{< meta meta.2 >}}

To be clear, this merging behaviour is by design and allows flexible composition of metadata from project defaults, directory-level configuration, and document-specific settings.
The array concatenation ensures that all specified values are preserved in the final metadata.

[davidskalinder]

About metadata, your example does not seem reproducible or at least I cannot reproduce any behaviour like the one you showed. I also don't understand why you are using the meta shortcode using indices while your keys are atomic.

edit: I did't not see with your nearly equal values in all caps, that the metadata was an array. With arrays, indeed, it's merged.

Yes, I did that with the arrays just to figure out and show the merge order -- feel free to ignore it (since you presumably already know the merge order very well!). What I think really matters is the scalar values, for which (as expected) options higher up in the merge chain seem to be clobbered by the later ones.

Here is a repo branch with the reprex; it renders fine for me with Quarto 1.8.27 (within Rstudio).

If you set something globally, it overwrites/sets defaults the same way in Quarto and extensions. Not sure why you would assume it wouldn't in extensions. If you set it under the format, it takes precedence.

Yes, but the issue (I think?) is the precedence of what's set in the main document vs. in the extension -- that is, the between-file precedence rather than the within-file precedence. Format extensions, as I understand it, should contain specific information about how to format an output for a particular recipient; and so it seems like the format extensions should be able to override customizable defaults that are set elsewhere. I can't find a way to do this with only a source file and an extension, though perhaps as you say above something like profiles can do this. (And if so, I should go learn how those work!)

I'm pretty sure "default" is well suited, especially since that's the default and what is used (ish) for single documents.

Do you mean that the Quarto defaults should be suitable as fallback options for all presentation elements for all formats? If so, then I think that would mean (perhaps sensibly) that the formatting instructions specified in format extensions cannot be layered onto any set of formatting instructions other than the Quarto defaults? I could not, for example, decide that my house typeface is Palatino, and that I would like my PDFs to use Palatino unless a custom format overrides my house typeface. Instead, I would have to either specify Palatino in the main source doc and have it clobber the typeface specified in all extensions, or I'd have to make sure that every extension specifies a typeface (writing Palatino into any extensions where it was previously unspecified).

Maybe I'm thinking about this entirely the wrong way, but it seems like there should be some way to say "Use the format extension's instruction; if it's not there, use my house style's instruction; if it's not there, use what the Quarto devs set as the default"?

As I mentioned above though, it's possible that Quarto projects (or their components like profiles or directory separation with metadata files) can indeed do this in a way that won't be clear to me until I go learn more about those.

[mcanouil]

Let me try to explain another way:

• Quarto has default values for options.
• Extensions can also provide default values for options that takes precedence over Quarto's default.

Now, when you set anything explicitly in YAML, it takes precedence over any default.
If you want your default values to be "active", then you should not set global options (top-level) options.

Let's say you have defined "myext-html" format with "meta: A".

Scenario 1

You don't set meta anywhere in your document or YAML (project/directory), then myext-html still has meta: A, but the other formats (variants) do not have any meta.

format:
  myext-html: default
  html+A: default
  html+B: default

Scenario 2

You set meta globally, thus meta is set for everything regardless of the formats having or not default, because you've set meta and that meta are merged as explained before.
All three formats now has meta: "C" which overrides whatever was set.

meta: "C"
format:
  myext-html: default
  html+A: default
  html+B: default

Scenario 3

You set meta globally, but you set meta also under one HTML variant, then myext-html and html+A have meta: C, but html+B has meta: B

meta: "C"
format:
  myext-html: default
  html+A: default
  html+B:
    meta: "B"

Scenario 4

You don't set meta globally, but you set it for one variant.
myext-html has meta: A, html+A has no meta, and html+B has meta: B.

format:
  myext-html: default
  html+A: default
  html+B:
    meta: "B"

---

With arrays merging, you can see the same pattern in the order in which you find the values.
The purpose of setting options in YAML or documents is not to provide a "default".
It's to set an actual value to everything.
If you want a "fallback", then you need to use Lua in the current metadata system and to create your logic which is "if a metadata is set in extension and is set globally in YAML, then the metadata from extension should be used". As you can probably see, this means you no longer have the ability to override globally, instead you have to specify the everything under the format.

Hope this helps clarify.

[davidskalinder]

Thanks @mcanouil, I think that confirms my intuition about how the merge order was working for Quarto-set defaults, format extensions, top-level YAML in .qmd docs, and format-level YAML in .qmd docs.

Given the suggestions above about projects and profiles, I was curious to know the precedence of other files where metadata can be set. I've tried to find all the places I could where arbitrary metadata options (that is, options that don't need to be set in predetermined places, like contributes.revealjs-plugins) can be set. (It's very possible I missed some -- if so please post and let me know!)

I've updated my reprex repo to include all of these possibilities and to render to a file that shows the order in which Quarto merges array settings into a single metadata array.

As discussed above, I would expect that for scalar metadata, later entries in this list will override earlier entries (though given the length of the list I have not tested this).

Here is the output showing the order of locations in the merged metadata:

1. Extension’s format key
2. _quarto.yml’s top-level YAML
3. Top-level YAML of profile named first in profile.group.2
4. Top-level YAML of profile named first in profile.group.1
5. Top-level YAML of profile named in profile.default
6. _quarto.yml’s format key
7. format key of profile named first in profile.group.2
8. format key of profile named first in profile.group.1
9. format key of profile named in profile.default
10. _metadata.yml’s top-level YAML
11. _metadata.yml’s format key
12. Main .qmd doc’s top-level YAML
13. Included metadata file’s top-level YAML
14. Main doc’s format key
15. Included metadata file’s format key

This ordering seems to be invariant to within-file specification order.

This obviously provides many layers at which an option (like, say, csl) might be specified! And because each location can overwrite earlier locations (all the way back to the Quarto defaults), it seems like there are therefore many opportunities to set fallback options: for example, I could easily do something like "use the file's setting; if it's not there, use the directory's; if it's not there, use the project's", and so on.

However, with the format extension fixed at the start of the list, there's nowhere to fall back to from there except to the Quarto defaults. I'll have to think more about whether it makes sense to put my layers of format preferences elsewhere in that list.

I'm surprised that I couldn't find documentation of that full merge order, especially given that metadata is so important in Quarto. (The only thing that comes close is a section in the Projects documentation, which leaves out many of the often-surprising details.) I suppose I could file this as a documentation bug report if it would be useful?

[davidskalinder]

There were a few other things that surprised me while I was investigating the metadata merge order but that are hard to show in a single merged array like the one above. They seem important to know about though, and they might affect my goal of having a fallback from recipients' preferences, so I'll list them here (even if only for my own reference).

• When any profiles are activated via quarto render --profile in the command line:
  • Settings from group-default profiles (that is, profiles that are listed first in a profile.group metadata entry) remain in the merged metadata and appear above the CLI-specified profile(s)
  • Settings from the global default profile (specified in profile.default) are ignored!
  • Multiple profiles activated in this way are added to the merged metadata in reverse order of how they are specified (so quarto render --profile p1,p2 will put p1's options after p2's)
  • If those same profiles are also specified in profile.group metadata, the order they are named on the CLI will affect their merge order and the way they are specified in profile.group metadata will not
• When a .qmd is rendered to a custom format (e.g., myext-html), any metadata settings applied to the base format (e.g., html) are ignored
• The documented temporary restriction on metadata extensions seems to mean that only the small number of keys that are legal subkeys of project are allowed (like type, render, output-dir, etc., though I can't find a list of these); therefore arbitrary metadata (e.g., csl) cannot currently be set in metadata extensions

I couldn't find any documentation about these points, though of course it's possible I just missed it.

If I find more stuff like this I'll try to come back here to edit this list.