Rendering content from custom collections in posts

[groundh0g]

• I believe this to be a bug, not a question about using Jekyll.

• I updated to the latest Jekyll (or) if on GitHub Pages to the latest github-pages

• I ran jekyll doctor to check my configuration

• I read the CONTRIBUTION file at https://jekyllrb.com/docs/contributing/

• I am on (or have tested on) macOS 10+

• I had an error on GitHub Pages, and I have reproduced it locally.

My Reproduction Steps

This is not a build error, it's a rendering quirk. There are no logs to provide.

I'm extending Jekyll using only Liquid and JavaScript, so that things work on GitHub Pages (no plugins). This issue uses no JavaScript, though. I believe it's a rendering quirk in how Liquid is interpreted from within collections.

I have a collection for _sidebars and _footers. They're pulled into the templates via {% include ... %} tags that run Liquid to iterate over the collection files.

• For pages, everything works as expected.
• For collections, {{ x }} and {% x %} tags are rendered literally (rather than expanded).
• For posts, which are technically in the _posts collection, the behavior is the same.

The Output I Wanted

I would love it if {{ x }} and {% x%} tags were expanded when:

• rendered within the collection's markdown (normally, this is just fine -- it works)
• when they are included from within another collection (in the simplest case, _posts)

There is no log for this issue. It's a question surrounding the rendering of Liquid variables within nested collections. (In my use case, it's not obvious that the collections are nested, since _posts is implemented as a collection, itself.)

/cc @jekyll/build

[groundh0g]

FYI. I tried capturing the rendered text and applying the markdownify filter on it. No luck.

I did add a hack to the _footer collection that does a string replace on the literal with the variable value. Of course, that only supports the variables that I've listed in the replace filter (in this case {{ site.baseurl }}.

    {% for file in my_collection %}
    <div class="col-md-3 footer-item">
      {% assign fileContent = file.content | replace: "{{ site.baseurl }}", site.baseurl %}
      {{ fileContent | markdownify }}
    </div>
    {% endfor %}

When the Liquid tags are expanded (on pages), there's nothing to replace. When the literal is injected (on posts), the replace cleans it up. This isn't a real solution, though.

[ashmaroli]

There may be an intersection of this "quirk" with another reported in #6209
Can you comment out L#81 of renderer.rb (or equivalent if not v3.5.0) in your local Jekyll gem and report back if it solves your issue..? Thanks.

[groundh0g]

Commented out the line, but no change. v3.4.3. I see the same behavior on GitHub Pages.

      output = document.content

      if document.render_with_liquid?
        Jekyll.logger.debug "Rendering Liquid:", document.relative_path
        output = render_liquid(output, payload, info, document.path)
      end

      Jekyll.logger.debug "Rendering Markup:", document.relative_path
      output = convert(output)
#      document.content = output

      if document.place_in_layout?
        Jekyll.logger.debug "Rendering Layout:", document.relative_path
        place_in_layouts(
          output,
          payload,
          info
        )
      else
        output
      end

[groundh0g]

Note: I'm not a Ruby dev, and I think all I did was change the source. I added gibberish and jekyll ran as usual, so .... how does one rebuild the gem from source? Google keeps pointing me to "building" your site (aka. jekyll usage).

[ashmaroli]

and I think all I did was change the source

source? by that do you mean the repo cloned from GitHub?

how does one rebuild the gem from source?

An average user need not know about this, So you're better off not breaking Jekyll unintentionally.
You may un-comment that line.. I was just checking if the two tickets had a common solution.
Thanks

[groundh0g]

I took some time to examine the collection values (from the default.html template), with the interesting bit being:

{% assign foo = site.collections | where: "label", "content_sidebars" %}
{{ foo.first.docs.first.output | inspect }}

Everywhere except for posts, I get the rendered HTML. Even the {{ x }} and {% x %} tags are expanded, as hoped.

For posts, however, output is always null, and the content is the un-markdownified text, and nothing I try will expand the {{ x }} and {% x %} tags.

Hope that helps.

[ashmaroli]

Everywhere except for posts, I get the rendered HTML

What do you mean by that? Is it assigning the foo to 'posts' like

{% assign foo = site.collections | where: "label", "posts" %}

??
The exact use case is still not clear to me..

[groundh0g]

I have a collection where users can put 0 or more markdown files for the footer (_collection_footer), and the same thing for the sidebar (_collection_sidebars).

Those items are included in the default.html (if they exist) and it works beautifully. Even on custom layouts. Everywhere that is, except for when the page is a layout: post, coming from the _posts folder.

In the case of a post, the output of the collection docs is null, where the documentation says to expect the rendered HTML. And for content other than posts, that's exactly what I see.

For example, the blog list is an index.md that renders the sidebar and footer just fine...

But, as soon as I view an actual post (eg. from the _posts folder), the {{ variable }} and {% include xyz.abc %} fail to render, and they're injected as string literals.

[ashmaroli]

Interesting.. can you provide a link to the exact "template" that includes the code for the sidebar? I was not able to find it.

[pathawks]

Hey @groundh0g! I worked with you on a few PRs to plusjade/jekyll-bootstrap way back when 👋

I think this is a really cool idea and I see the use case you are going for. I see how this could make things easier for users who are new to Jekyll.

Could you provide just a simple repository with a really basic "This works, this doesn't" example? I think it would help if we were all looking at the same code.

[groundh0g]

Sure thing. Thanks!

Starting a new job today, but will try to get a simple repro case posted on a branch and post a link here as soon as possible.

[groundh0g]

Hi, @pathawks.

Here's a simple example to reproduce the issue. Just clone locally, then jekyll server to experience it in all its glory!

Navigate between Home / About / Post to see that the includes expand just fine in pages, but not in posts.

I believe it's because posts are implemented and rendered as collections themselves, but that's just wild speculation on my part.

Thanks for taking a peek.

https://github.com/groundh0g/jekyll-collection-render-issue

[ashmaroli]

@groundh0g can you configure GitHub Pages to build the dummy-repo above? That'd help (me) inspect without cloning locally.. Thanks..

[groundh0g]

Sure thing. Try this:

https://groundh0g.github.io/jekyll-collection-render-issue/index.html

[ashmaroli]

Ah yes.. I see it now.. 😃 👍

[ashmaroli]

Got a chance to look into this.
The reason you're getting raw liquid is because your requesting for file.content instead of file.output But having said that I came across the bug this ticket was indirectly reporting: {{ file.output }} returns null for Documents.

I even tried accessing default collection docs hash (Posts) to rule out other factors:

{% for file in site.posts %}
  <h3>{{ file.title }}</h3>
  <pre>{{ file | inspect }}</pre>
  <hr/>
{% endfor %}

[groundh0g]

Is there a work-around, or is this a wait until GitHub Pages refreshes after a fix thing?

[groundh0g]

While not a great workaround, and I'm not terribly pleased with the solution, I thought I would document a hack that seems to meet my needs. Maybe this will help fill the gap for those who google to this issue, this comment.

So, the terrible hack is to move the collection content to their own pages, segregated by folder. That way, markdown files (with YAML headers) in the same path are in the same "collection".

My main objection to this is performance. I know that sounds strange for a preprocessed, static blog, but I use Jekyll for almost all my sites -- and that extends beyond standard blogs. I've had GitHub Pages time out on some of my earlier designs (when you get to 1,000+ posts), and had to concern myself with build times.

This means iterating over all the pages in the site whenever I want to zero in on one of my groupings. The where_exp filter is my friend, but I secretly worry that it's internally iterating over every page too.

It's not very elegant, but I get my precious collections again.

[pathawks]

So, the terrible hack is to move the collection content to their own pages, segregated by folder. That way, markdown files (with YAML headers) in the same path are in the same "collection".

I'm not sure I understand. Can you provide an example directory listing, or something?

[groundh0g]

If I had a collection defined in a folder named _my_collection, the equivalent would be to just put that same content in an (always-rendered) folder - ./my_folder.

To get to that markdown's content, I would have to write a {% for file in site.pages %} loop and check for {% if file.path contains "my_folder/" %}, or something similar with a where_exp filter.

[pathawks]

Thinking out loud:

Not all documents are rendered at the same time. If posts are rendered before collections, it would make sense for the content of collections documents to appear unrendered.

[groundh0g]

Makes sense to me. Especially if _posts are just another collection. Explains the content being populated, but the output being null, right?

[pathawks]

Explains the content being populated, but the output being null, right?

Right.

Not sure there's a great solution here.

[groundh0g]

So, the options are likely these, then? (Just spitballing for the record.)

1. Jekyll treats posts as first-class pages (not collections)
2. Jekyll processes other collections first, then _posts (2-pass render)
3. Jekyll ensures {{ x }} and {% include ... %} can be processed within collections, referenced from _posts some other way
4. Jekyll remains the same and we update the docs to warn folks

NOTE: if you only ever want the collection's metadata and pre-processed markdown, there's no problem at all. It's just output that's causing headaches. Maybe that's a more typical use case?

In any case, I assume that I'll need to devise another scheme for my scenario in the interim for GitHub Pages.

Thanks for the help guys. I hope this thread leads to a verified root cause, and that it's something that's easier to fix than an overhaul of the render pipeline.

[ashmaroli]

the options are likely these, then?

1. Jekyll treats posts as documents in a collection with label="_posts". There're ways to verify this if you want. but not readily possible via Liquid Add a line {{ file.collection }} in your test sample to debug
2. Nope, nothing of that sorts.. (from my understanding of the codebase). Try creating another collection say _movies/jumanji.md. You'd encounter the same bug when you try to access movies/jumanji.html
3. Dunno about this
4. Better fix than warn IMO

[MattSturgeon]

I think I may have just run into this (or very similar) bug.

I have a collection assets in which I keep my js and css files:

_assets
`- css
  `- style.scss
`- js
  `- main.js
  `- header.coffee
  `- setup.coffee

main.js includes header.coffee and setup.coffee like so:

---
---
{% include_relative setup.coffee %}
{% include_relative header.coffee %}

Which works fine, other than the fact that the included files are not rendered in time for the include. If I inspect the files afterwards, both header.js and setup.js have been rendered by coffeescript, but main.js has the unrendered text of each.

---

I also noticed that I couldn't use liquid tags withing a coffeescript file, if I rename main.js to main.coffee I get

Conversion error: Jekyll::Converters::CoffeeScript encountered an error while converting '_assets/js/main.coffee':
                    SyntaxError: [stdin]:5:2: unexpected %

[RNCTX]

Guys, I have the same bug here, and it began with a markdown parser change. Have you changed parsers, @groundh0g ? The below works okay for me with the default parsers, but after changing to one called via a plugin, I get the same issue.

collections:
  panels:
   output: true

{% for panel in site.panels limit: 17 %}
     {% for category in panel.categories %}
          {% if category == "post" %}
             <article id="{{ panel.title | slugify | downcase }}">
             <h2>{{ panel.title }}</h2><br />{{ panel.output }}
             </article>
          {% endif %}
      {% endfor %}
{% endfor %}

I get proper processing of everything (files, frontmatter) right up until panel.output, and then I get nada.

<article id="test-post-9">
<h2>Test Post 9</h2><br />
</article>

test-post-9.md contains...

---
date: 2017-07-09 00:00:00 CDT
title: Test Post 9
category: post
tags: maintenance
facebook:
twitter:
instagram:
---
![image](images/pic01.jpg)
This is a test post. The quick brown fox jumped over the lazy dog.

Commenting the line from the renderer that @ashmaroli mentioned above had no effect.

[groundh0g]

Nope. No change in parser from the default, @RNCTX. Also, no added plugins.

[RNCTX]

1. Jekyll treats posts as first-class pages (not collections)
2. Jekyll processes other collections first, then _posts (2-pass render)
3. Jekyll ensures {{ x }} and {% include ... %} can be processed within collections, referenced from _posts some other way
4. Jekyll remains the same and we update the docs to warn folks

Based on your post above I think the answer, considering I can reproduce the problem by changing markdown parsers (via plugin, and thus likely changing the order in which things are processed), I would bet on 2 or a variation thereof.

My plugin-called (global) parser was converting content, which I just tried changing to output but it didn't have any effect.

So I changed the default parser back to kramdown, and instead of using my third-party parser globally I called it via a pipe to the plugin like... {{ post.content | markdown-it }} in the template, and everything works fine other than output and content being the same, which is back to the other bug mentioned previously.

In short, @ashmaroli 's linked fix is a fix that needs applied regardless, otherwise there is no way to get discrete data from {{output}} and {{content}}.

Is it possible to call a second renderer as a standalone plugin? If so that would fix my issue, as I could pipe collection.content | to-my-plugin.

[MattSturgeon]

Why can't Jekyll render a file as part of the include process? Isn't that the expected behavior of include and include_relative: to trigger a render and then include the result?

[RNCTX]

@MattSturgeon this is all specific to collections. There is a different level of template control with a collection, I don't think this affects default posts and pages or there would surely be more bug reports. This seems to arise from calling content into/out of a collection from a file not in the collection. In my case I have a single-page site template that calls a 'recent post' list to populate hidden divs in the index file at the site root. My links are show/hide actions on those divs. I call the content from the collection folder into the index file at the site root.

There are two bugs at play here, as ashmaroli eluded to.

One, the gem package as distributed renders content and output from a collection the same but the docs say that one should be processed and the other should be raw. Commenting out the line linked above fixes that. I'm using v3.1.6 so that bug goes back at least that far.

Two, there is something in the order in which these files are processed that causes both output and content to return blank strings for collection page content under certain circumstances. I can trigger this bug by changing the global markdown parser to a third-party one in _config. Not sure about all of the circumstances that groundh0g has reproduced, but at least one other because he says he hasn't touched the default markdown parser.

[MattSturgeon]

Sorry, didn't mean pages only, I was just saying that whenever you include anything you expect whatever you include to be rendered. I'll edit for clarity.

If it's already been rendered by a previous render pass, fine, use the result of that. Otherwise it should be rendered as part of the include process.

[RNCTX]

Yes, I suspect that's what we're getting to, that there is a problem in the order in which these processes happen related only to collections. There is no apparent difference in the rendering process from two startup logs of the bundled webserver for me, one with kramdown and one with Markdown-it via a plugin.

Only difference is one has post text and the other one doesn't.

[i-give-up]

After studying this for a bit, I don't think this is a bug except for maybe the document's content attribute (rendered or unrendered depending on when you access it) whose behavior seems to contradict the documentation which says that content contains the unrendered content.

Attempt at tl;dr

• You are using content of documents from a collection content_sidebar for sidebar content (via {{ file.content }}).
• A document's content is unformatted before that document is rendered.
• The posts collection always get rendered before the content_sidebar collection so {{ file.content }} and by extension the sidebar content is still unformatted at that time.
• pages always get rendered after content_sidebar so {{ file.content }} and by extension the sidebar content is already formatted by then.

I guess mine is a more elaborate version of this comment by @pathawks, though I have to point out that posts is itself a collection.

Long version

If you look at code for Site class

• here it shows that documents (i.e. files in collections; posts is also a collection, as mentioned before) are processed before pages.
• here it shows that the code iterates over collections which is a hash and Ruby hashes are ordered.

If you write a generator plugin in _plugins/test.rb

module Test
    class Generator < Jekyll::Generator
        def generate(site)
           p site.collections
        end
    end
end

then examine the output. You can see that the order in collections is posts first, followed by other collections specified in /_config.yml by order of appearance. To summarize, the order of rendering goes like this from earliest to last

• posts collection
• other collections
• pages

These are my guesses about liquid rendering

• Once a liquid variable ({{ liquid_var }}) is processed, there is no second pass. So even if liquid_var contains something like {{ 1 | plus: 2 }}, that won't get processed anymore.
• Liquid's include tag is processed before other liquid constructs. So if include added something like {{ 1 | plus: 2 }}, that will get processed.

As I established earlier, posts is the first element of collections, so the files there gets rendered first (i.e. earlier than documents in content_sidebar collection which you use for the sidebar content). This means that when posts are getting added to your customized layouts which includes /_includes/sidebar.liquid, the {{ file.content }} in there (where file is a document in content_sidebar collection) still contains the unformatted content. Since liquid variables only gets processed once, the unformatted liquid markup remains unformatted and appears so in the generated posts.

On the other hand, when pages get processed, content_sidebar collection is already processed (because collections/documents are processed before pages) so the file.content in sidebar.liquid is the content that has been processed by Liquid. This is why they show up fine in the generated pages.

You can sort of see this in action if you create another collection col with layout: default in front matter. In your /_config.yml, if you add col (with output: true) before content_sidebar in collections variable, the sidebar will appear unformatted in col's documents because col is rendered before content_sidebar. Whereas if you add col after content_sidebar, the sidebar will appear formatted just fine.

Apologies for the (possibly bordering on inarticulate) wall of text. There's quite a lot going on and I don't know how to make it more concise.

By the way, I tested with @groundh0g's example site which uses the default Kramdown converter and doesn't have any coffeescript files so I don't know if they can cause this issue.

[groundh0g]

@i-give-up, thanks for the follow up. Curious about one thing, though. How would rendering the posts collection last affect the render pipeline? Would that solve anything?

If this is just a byproduct of collections, and there's no simple fix, what's the next step? Should we document the scenario(s) / pitfalls and add it to the user docs?

[i-give-up]

How would rendering the posts collection last affect the render pipeline? Would that solve anything?

Haven't tested but in that case posts should be no different from a collection specified after content_sidebar in _config.yml's collections variable, i.e. it should get the rendered sidebar content.

If this is just a byproduct of collections, and there's no simple fix, what's the next step? Should we document the scenario(s) / pitfalls and add it to the user docs?

I don't think it's common to use collections this way i.e. as snippets to be included in a layout file. It seems to be something that include files are supposed to do. As such, in my opinion a more sensible (structurally correct?) approach is to modify certain aspects of layout/include files

• make it such that if an include file has markdown extension (e.g. md, markdown), it will get converted by the markdown parser.
• make include files accessible through a Liquid variable - maybe something like site.include_files - so that you can filter/sort them (instead of having to put files in a Liquid-accessible structure like collections just to be able to sort them) and include them programmatically.

[chitinlink]

would be wonderful if I could simply use a plugin to force a second render. ie.
{{ doc.content | render_liquid }}

This issue is nearly two years old, has anything changed?

[rhyslg]

Banging my head against a wall with this as well. Any update at all?

[rvansa]

I believe this is related: #6465

[iagox86]

I'm guessing this isn't fixed and no workaround? Just spent my whole morning troubleshooting a bug that ended up being this

[chitinlink]

This issue is nearly two years old, has anything changed?

And just like that, 4 years...

[ashmaroli]

Hello Everyone, this issue is still "unfixed" as of Jekyll 4.3.
Unfortunately, I still don't see a good, compatible way to fix this even though I know the reasons behind it.. ☹️