Custom Page Generator Not Working

Related to: Accessing Custom Attributes in Liquid

So I have tried to abstract the logic in the related post by creating a custom Generator and Page classes. The main content of the page using it doesn’t load and the pages don’t render correctly for categories or tags using the same layout template.

The custom Generator:

module Generators
            class RandomSectionGenerator <  Jekyll::Generator
                safe true
                priority :lowest 

                def generate(site)
                    #homepage
                    site.pages << SectionedPage.new(site,"/","Home","default.html",site.posts)
                    
                    #iterate through the categories and their posts
                    category_path = (!site.config['categories'] || !site.config['categories']['basedir'] ? "/" : site.config['categories']['basedir']) 
                    category_layout = (!site.config['categories'] || !site.config['categories']['layout'] ? "default.html" : site.config['categories']['layout']) 
                    
                    site.categories.each do |category, posts| 
                        site.pages << SectionedPage.new(site,File.join(category_path,Jekyll::Utils.slugify(category)),category,category_layout,posts)
                    end

                    #iterate through the tags and their posts
                    tag_path = (!site.config['tags'] || !site.config['tags']['basedir'] ? "/" : site.config['tags']['basedir']) 
                    tag_layout = (!site.config['tags'] || !site.config['tags']['layout'] ? "default.html" : site.config['tags']['layout']) 
                    
                    site.tags.each do |tag, posts| 
                        site.pages << SectionedPage.new(site,File.join(tag_path,Jekyll::Utils.slugify(tag)),tag,tag_layout,posts)
                    end
                end
            end
        end

The custom Page:

class SectionedPage < Jekyll::Page
            attr_accessor :collection, :sections

            ATTRIBUTES_FOR_LIQUID = %w(
                collection
                sections
                content
                dir
                name
                path
                url
            )

            def initialize(site,path,collection,layout,posts)
                @site = site
                @base = site.source
                @dir = path
                @collection = collection

                @basename = "index"
                @ext = ".html"
                @name = "index.html"

                @data = {
                    "linked_docs" => posts
                }

                #initialize layout defaulting to 'default'
                @layout = File.join("_layouts",layout)
                @layout = File.exist?(File.join(@base,@layout)) ? @layout : File.join("_layouts","default.html")
                
                #Execute super "constructor"
                super(@site,@base,"",@layout)
                
                #create sections data
                config_sections 

                #final initialization
                self.read_yaml(@base, @layout)
                self.data.merge!('title' => @collection)
            end

            #set up posts data in random layout
            def config_sections
                #initialize 
                @boxes = !@site.config['sections'] || !@site.config['sections']['boxes'] ? {'single' => 1 } : site.config['sections']['boxes']    
                @placements = !@site.config['sections'] || !@site.config['sections']['placement'] ? ['left','center','right'] : site.config['sections']['placement']
                
                @section = nil
                @placement_idx = 0
                
                @sections = []
                
                #initialize posts for the page
                @posts = []

                #copy posts to another Hash for popping
                @docs = (@data["linked_docs"].nil? || @data["linked_docs"].empty?) ? {} : @data["linked_docs"]
                @docs.each { |p| @posts << p.dup }

                #do till there are no posts left
                while @posts.size > 0
                    Jekyll.logger.debug "#{@posts.size} posts are left for page #{@dir}"

                    #are all the placements in a section filled
                    if @placement_idx > 2 
                        @placement_idx = 0 
                    end

                    if @placement_idx == 0
                        #if there was already a section object, put into site configuration
                        if @section != nil
                            Jekyll.logger.debug "Completed Section: #{@section} for #{@dir}"
                            @sections.push(@section)
                        end

                        #create a new section object
                        @section = Section.new()
                    end

                    #add posts to the randomly selected placement
                    @tmp = @boxes.to_a.sample
                    @section.add_posts(@placements[@placement_idx], @tmp[0], @posts.pop(@tmp[1]))

                    #increment the placement counter
                    @placement_idx += 1
                end

                #add to page sections data in the site
                @data["sections"] = @sections
            end

            #alias to get the page's sections
            def sections
                @data["sections"]
            end
        end

And the custom Section class

class Section
            attr_reader :data, :collection
            
            def initialize
                @data = {}
            end

            def add_posts(place, template, posts)
                @data[place] = {"template" => template, "posts" => posts}
            end

            def get_placement(place) 
                @data[place] ? @data[place] : {}
            end

            def to_liquid
                @data
            end
        end

If we focus on a portion of the custom Generator generate code:

#iterate through the categories and their posts
category_path = (!site.config['categories'] || !site.config['categories']['basedir'] ? "/" : site.config['categories']['basedir']) 
category_layout = (!site.config['categories'] || !site.config['categories']['layout'] ? "default.html" : site.config['categories']['layout']) 
                    
site.categories.each do |category, posts| 
  site.pages << SectionedPage.new(site,File.join(category_path,Jekyll::Utils.slugify(category)),category,category_layout,posts)
end

This code is creating a Page for every site categories and using a template _layouts/collection.html which is below

---
layout: default
permalink: /:collection/
---

{% for section in sections %}
<section class="section">
    <div class="container-fuild">
      <div class="masonry-blog clearfix">
        {% for placement in section %}
          {% assign include_file = placement[1].template | prepend: "posts/" | append: ".html" %}
          {% assign include_posts = placement[1].posts %}
          <div class="{{ placement[0] }}-side">
            {% include {{ include_file }} posts=include_posts %}
          </div>
        {% endfor %}
      </div>
    </div>
</section>
{% endfor %}

In the Jekyll logging output, I see the following once for each category

Layout source: site
Rendering: _layouts/collection.html
Pre-Render Hooks: _layouts/collection.html
Rendering Liquid: _layouts/collection.html
Rendering Markup: _layouts/collection.html
Rendering Layout: _layouts/collection.html

Writing: /srv/jekyll/_site/:collection/index.html

When I run Jekyll, these pages don’t get generated exactly. A file in _site/:collection/index.html is generated. It is rendered in the default layout, but there is no main content (a.k.a. where the posts should render). So my two questions are:

  1. Why isn’t this generating a separate file for each category as an index.html (e.g. _site/process/index.html)?
  2. How do I make sure that the layout see the data my custom generator put into the Page.data[“sections”] when rendering and get that template to render to the expected unique URL (in question 1)?

Thanks in advance!!!

You should not be using the permalink front-matter variable in a layout.

Regarding your first question, I suggest two things :

  • Since what you are trying to achieve looks close enough to what jekyll-archive does, you can take a look at its source code if you haven’t already.
  • Re-read Jekyll’s documentation on permalinks, especially the part about using placeholders and the section about collections. Then, try achieving this by setting the permalink in your _config.yml

In order to answer your second question, it would help a lot if you could share the content of one of your partials (one of the include_file from {% include {{ include_file }} posts=include_posts %}) and the content of the resulting _site/:collection/index.html file.

Thank you @pcouy! Very helpful. Seems like so many different implementations to create these virtual pages. I am almost there with one thing that is still confusing me. Using the methods above (altered with your suggestions), my homepage renders. My category pages are created, but they are empty. I am running jekyll build/serve from the root of the site, but I am noticing some errors/warnings in the process.

Error reading file /srv/jekyll/big-data/index.html: No such file or directory @ rb_sysopen - /srv/jekyll/big-data/index.html 
Error reading file /srv/jekyll/_layouts/collection.html: No such file or directory @ rb_sysopen - /srv/jekyll/big-data/index.html

The path to the layout is it trying to read is correct, but the second one isn’t; that isn’t a real page
and during the rendering portion:

Build Warning: Layout '_layouts/collection.html' requested in personal/index.html does not exist. 

Seems like it is looking for the layout inside the page, but the page doesn’t exist as a file (because I am making it up. My homepage does have a page on the filesystem though as one difference.

It would be a lot easier to help you if you included a minimal reproducible example of your issue, or at least the relevant source code from your project.

Apologies. Here it is…

_plugins/Customizers.rb

    module JekyllCustomizers
        class Section
            attr_reader :data
            
            def initialize
                @data = {}
            end

            def add_posts(place, template, posts)
                @data[place] = {"template" => template, "posts" => posts}
            end

            def get_placement(place) 
                @data[place] ? @data[place] : {}
            end

            def to_liquid
                @data
            end
        end
        
        
        class SectionedPage < Jekyll::Page
            attr_accessor :sections

            ATTRIBUTES_FOR_LIQUID = %w(
                sections
                content
                dir
                name
                path
                url
            )

            def initialize(site:,posts:,dir: "/",title: "",layout: "")
                @site = site
                @base = site.source
                @dir = dir
                @name = "index.html"
                @ext = File.extname(@name)
                @content = ""
                @data = {}
                @title = title

                #Execute super "constructor"
                super(@site,@base,@dir,@name)
                
                #process page name
                self.process @name

                #merge into data
                @data.merge!('linked_docs' => posts)
                
                if layout != "" 
                    self.read_yaml(@base,layout)
                    @data.merge!('layout' => layout)
                end
                
                if @title != ""
                    @data.merge!('title' => @title)
                end

                #create sections data
                generate_section_data
            end

            def inspect 
                "#<JekyllCustomizers:SectionedPage @url=#{url} @relative_path=#{relative_path} @sections=#{sections} @data=#{data.inspect}"
            end

            #set up posts data in random layout
            def generate_section_data
                #initialize 
                boxes = !@site.config['sections'] || !@site.config['sections']['boxes'] ? {'single' => 1 } : site.config['sections']['boxes']    
                placements = !@site.config['sections'] || !@site.config['sections']['placement'] ? ['left','center','right'] : site.config['sections']['placement']
                sections = []
                section = nil
                placement_idx = 0
                posts = []

                #copy posts to another Hash for popping
                docs = (data["linked_docs"].nil? || data["linked_docs"].docs.empty?) ? [] : data["linked_docs"].docs
                docs.each { |p| posts << p.dup }

                #do till there are no posts left
                while posts.size > 0
                    #are all the placements in a section filled
                    if placement_idx > 2 
                        placement_idx = 0 
                    end

                    if placement_idx == 0
                        #if there was already a section object, put into site configuration
                        if !section.nil?
                            sections.push(section)
                        end

                        #create a new section object
                        section = Section.new()
                    end

                    #add posts to the randomly selected placement
                    tmp = boxes.to_a.sample
                    section.add_posts(placements[placement_idx], tmp[0], posts.pop(tmp[1]))

                    #increment the placement counter
                    placement_idx += 1
                end

                #add to page sections data in the site
                if !sections.nil? and sections.size > 0
                    data.merge!('sections' => sections)
                end
            end

            #alias to get the page's sections
            def sections
                data["sections"] || []
            end
        end
    
        module Generators
            class RandomSectionGenerator <  Jekyll::Generator
                safe true
                priority :lowest 

                def generate(site)
                    #iterate through the categories and their posts
                    site.categories.each do |category, posts| 
                        coll = Jekyll::Collection.new(site,"category")
                        coll.docs = posts

                        pg = SectionedPage.new(
                            site: site,
                            dir: Jekyll::Utils.slugify(category),
                            title: category,
                            layout: File.join("_layouts","collection.html"),
                            posts: coll)

                        site.pages << pg
                    end
                 end
            end
        end
    end

_config.yml section

sections:
  placement:
  - left
  - center
  - right
  boxes: 
    single: 1
    doublebottom: 3
    doubletop: 3
  tag_colors: [green,aqua,red,yellow,grey,pink]

  #Collections
  collections:
    category:
      output: true
      permalink: /:title/
    tag:
      output: true
      permalink: /tags/:title/

_layouts/collection.html

---
layout: default
---

{% for section in page.sections %}
<section class="section">
    <div class="container-fuild">
      <div class="masonry-blog clearfix">
        {% for placement in section %}
          {% assign include_file = placement[1].template | prepend: "posts/" | append: ".html" %}
          {% assign include_posts = placement[1].posts %}
          <div class="{{ placement[0] }}-side">
            {% include {{ include_file }} posts=include_posts %}
          </div>
        {% endfor %}
      </div>
    </div>
</section>
{% endfor %}

It seems like we’re still missing an essential part of your project to figure what went wrong. Do you have a link to a github repo where we can browse your whole project and reproduce your issue ourselves ? If not, can you share a zip archive of a jekyll project that produces the same error ?

Here is a link to the archive of the site. I run the site using a docker container jekyll/jekyll:4

I have gotten to the point where seems like my only issue is that the rendering process is looking for the layout I specified in the page path versus from site source. This is the warning I get:

Build Warning: Layout ‘/srv/jekyll/_layouts/collection.html’ requested in team/index.html does not exist.

If I inspect the page via the pre_render hook

#<JekyllCustomizers:SectionedPage @url=/team/ @relative_path=team/index.html @sections=[#<SevenElephants::JekyllCustomizers::Section:0x00000040080620c8 @data={“left”=>{“template”=>“doublebottom”, “posts”=>[#<Jekyll::Document _posts/2021-11-14-everything-large-then-what-point.md collection=posts>, #<Jekyll::Document _posts/2021-11-07-consistency-is-your-greatest-trait.md collection=posts>, #<Jekyll::Document _posts/2021-10-30-transformation-is-painful.md collection=posts>]}, “center”=>{“template”=>“single”, “posts”=>[#<Jekyll::Document _posts/2021-11-23-i-will-do-that.md collection=posts>]}, “right”=>{“template”=>“doubletop”, “posts”=>[#<Jekyll::Document _posts/2022-01-15-pre-pre-pre-planning.md collection=posts>, #<Jekyll::Document _posts/2022-01-02-who-is-flawless-execution.md collection=posts>, #<Jekyll::Document _posts/2021-12-27-forks-in-the-road.md collection=posts>]}}>, #<SevenElephants::JekyllCustomizers::Section:0x0000004008060ac0 @data={“left”=>{“template”=>“doublebottom”, “posts”=>[#<Jekyll::Document _posts/2022-03-14-pressure-teaching-team.md collection=posts>, #<Jekyll::Document _posts/2022-01-21-technical-debt-not-real.md collection=posts>, #<Jekyll::Document _posts/2022-01-17-kinds-of-thinkers.md collection=posts>]}, “center”=>{“template”=>“single”, “posts”=>[#<Jekyll::Document _posts/2022-04-02-caring-about-how.md collection=posts>]}, “right”=>{“template”=>“single”, “posts”=>[#<Jekyll::Document _posts/2022-07-19-cost-of-better.md collection=posts>]}}>, #<SevenElephants::JekyllCustomizers::Section:0x00000040080339d0 @data={“left”=>{“template”=>“doubletop”, “posts”=>[#<Jekyll::Document _posts/2023-02-06-direct-conversations.md collection=posts>, #<Jekyll::Document _posts/2023-01-28-need-to-drop.md collection=posts>, #<Jekyll::Document _posts/2022-10-31-lead-without-fear.md collection=posts>]}, “center”=>{“template”=>“doubletop”, “posts”=>[#<Jekyll::Document _posts/2023-02-27-trust-starts-here.md collection=posts>, #<Jekyll::Document _posts/2023-02-14-agile-is-rigorous.md collection=posts>]}}>] @data={“layout”=>“/srv/jekyll/_layouts/collection.html”, “title”=>“team”}

Again, looking for the layout to be rendered and using the information in in @sections to iterate through. This all works for the homepage of the site which I have a physical file representation for, but doesn’t work for these categories.

How do I ensure that Jekyll is looking for the layout specified in the right place?

Thanks.

I was able to figure out the issue with a little more help from https://github.com/field-theory/jekyll-category-pages’s code.

Specifically in initializing the page to “swap” the page itself (dir) and the layout (*layout")

def initialize(site:,dir:,title:,layout:)
                @site = site
                @base = site.source
                @dir = dir
                @content = ""
                @title = title
                
                #check if the layout exists [via theme]
                if ! File.exist?(File.join(@base, layout)) && 
                    ( site.theme && File.exist?(File.join(site.theme.root, layout)) )
                      @base = site.theme.root
                end
                
                if layout == "" 
                    super(@site,@base,@dir,"index.html")
                else
                    #Execute super "constructor"
                    super(@site,@base,"",layout)
                
                    @dir = dir

                    self.process "index.html"
                    self.read_yaml(@base,layout)
                end
                
                if @title != ""
                    self.data.merge!('title' => @title)
                end
            end

Following this flow has all my pages rendering in the proper layouts.