I can't underestand the behaviour of --incremental flag

#1

The first time I tried to build my website with --incremental flag, it took about 1000 seconds. Then immediately I tried to build again and it took 14 seconds and it is really a good result. However I restarted my pc and then tried rebuilding again. This time it took about 400 seconds to build while nothing was changed in my website. Although its still better than the initial 1000 second to build the website from scratch but its still too much while nothing is changed. I cant understand this behaviour.

0 Likes

Is there a way to not copy unchanged static files to build folder?
#2

I added --verbose and --profile flags to find the bottleneck however I couldnt find anything

bundle exec jekyll build -d public --incremental  --profile --verbose
  Logging at level: debug
    Jekyll Version: 4.0.0.pre.alpha1
Configuration file: /home/hosna/Storage/repositories/zoom/_config.yml
         Requiring: /home/hosna/Storage/repositories/zoom/_plugins/metareader.rb
         Requiring: /home/hosna/Storage/repositories/zoom/_plugins/ext.rb
         Requiring: /home/hosna/Storage/repositories/zoom/_plugins/jdate.rb
         Requiring: /home/hosna/Storage/repositories/zoom/_plugins/include_absolute.rb
         Requiring: /home/hosna/Storage/repositories/zoom/_plugins/jekyll-postfiles.rb
         Requiring: /home/hosna/Storage/repositories/zoom/_plugins/file_exists.rb
         Requiring: /home/hosna/Storage/repositories/zoom/_plugins/onebox_filter.rb
         Requiring: jekyll-sitemap
         Requiring: jekyll-paginate-v2
         Requiring: jekyll-redirect-from
            Source: /home/hosna/Storage/repositories/zoom
       Destination: /home/hosna/Storage/repositories/zoom/public
 Incremental build: enabled
      Generating... 
       EntryFilter: excluded /Gemfile
       EntryFilter: excluded /Gemfile.lock
       EntryFilter: excluded /commit.sh
           Reading: _posts/time/2018-03-15-time.markdown
       EntryFilter: excluded /common/commit.sh
           Reading: _daddys-home-collection/index.md
           Reading: _daddys-home-collection/daddys-home-i/index.md
           Reading: _daddys-home-collection/daddys-home-i/with-music/the-co-dads-battle-ii/index.md
           Reading: _daddys-home-collection/daddys-home-i/with-music/index.md
           Reading: _daddys-home-collection/daddys-home-i/with-music/co-dads-make-peace/index.md
            ////......deleted some bunch of reading log here 
           Reading: _thrive/part-02/index.md
        Generating: ReadMetaFiles::Generator finished in 9.736257167 seconds.
        Generating: Jekyll::PostFileGenerator finished in 0.002440468 seconds.
        Generating: JekyllRedirectFrom::Generator finished in 2.191940422 seconds.
        Generating: Jekyll::JekyllSitemap finished in 0.013961898 seconds.
        Pagination: Starting
        Pagination: Is enabled, but I couldn't find any pagination page. Skipping pagination. Pages must have 'paginate: enabled: true' in their front-matter for pagination to work.
        Generating: Jekyll::PaginateV2::Generator::PaginationGenerator finished in 0.020414395 seconds.
         Rendering: redirect.html
  Pre-Render Hooks: redirect.html
  Rendering Markup: redirect.html
  Rendering Layout: redirect.html
     Layout source: site
         Rendering: redirect.html
  Pre-Render Hooks: redirect.html
  Rendering Markup: redirect.html
  Rendering Layout: redirect.html
     Layout source: site
         Rendering: redirect.html
  Pre-Render Hooks: redirect.html
  Rendering Markup: redirect.html
  Rendering Layout: redirect.html
     Layout source: site
         Rendering: redirect.html
  Pre-Render Hooks: redirect.html
  Rendering Markup: redirect.html
  Rendering Layout: redirect.html
     Layout source: site
         Rendering: redirect.html
  Pre-Render Hooks: redirect.html
  Rendering Markup: redirect.html
  Rendering Layout: redirect.html
     Layout source: site
         Rendering: redirect.html
  ///.... deleted some bunch of rendering logs here
  Pre-Render Hooks: robots.txt
  Rendering Liquid: robots.txt
  Rendering Markup: robots.txt
  Rendering Layout: robots.txt
           Writing: /home/hosna/Storage/repositories/zoom/public/p/part/2066/index.html
           Writing: /home/hosna/Storage/repositories/zoom/public/p/part/2059/index.html
           ///....deleted some bunch of other writing here
           Writing: /home/hosna/Storage/repositories/zoom/public/p/part/2076/index.html
           Writing: /home/hosna/Storage/repositories/zoom/public/redirects.json
           Writing: /home/hosna/Storage/repositories/zoom/public/sitemap.xml
           Writing: /home/hosna/Storage/repositories/zoom/public/robots.txt
  Writing Metadata: .jekyll-metadata

Filename               | Count |    Bytes |  Time
-----------------------+-------+----------+------
sitemap.xml            |     1 |  310.97K | 0.849
_layouts/redirect.html |  2095 | 1555.54K | 0.066
robots.txt             |     1 |    0.04K | 0.000

 
                    done in 471.909 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

I just relized that there is a great time gap between the line

EntryFilter: excluded /common/commit.sh

and

Reading: _daddys-home-collection/index.md

I dont know what is being done there.

0 Likes

#3

In my experience, the first run (e.g. “cold start”) of Jekyll after a reboot is always slower.

I’ve noticed this with tools like Gulp as well. I think it has more to do with the environment getting setup and run that first time, than anything related to Jekyll or it’s incremental mode.

0 Likes

#4

I don’t think it would make sense to be slower as much as 400 seconds.

0 Likes

#5

You’d be surprised.

Every time I start up Jekyll after my computer was restarted, it takes awhile for it to get going. Each build after is about 50% faster, and I don’t use Jekyll’s incremental mode.

I don’t know the internals of Ruby, but I’m assuming there’s some stuff held in memory that speeds up tasks after you’ve done that initial run. Restarting your device clears all that.

0 Likes

#6

do you know why the bottleneck isn’t logged when I’m adding --profile flag?

0 Likes

#7

It’s not a Jekyll bottleneck. It’s your Ruby environment getting started from a fresh reboot. The delay is with that… and before Jekyll even runs.

0 Likes

#8

No. The delay is here:

I just don’t know what is being done there.

0 Likes

#9

Sure, but is that gap a 600s one? I’m willing to bet a bulk of the delay is before Jekyll even starts after you restart.

The other delays are normal content related things as Jekyll builds your site. I don’t think the profile flag will surface that as it just shows you the times of layouts and includes, not reading files… which is where those EntryFilter lines happen.

0 Likes

#10

Also it’s my understanding incremental builds have quirks to them. Which would explain why your results aren’t consistent.

0 Likes

#11

the build after restart takes 471 seconds and I think that delay I’m talking about is taking about 450 seconds.

to add more information, in this line Reading: _daddys-home-collection/index.md it seems jekyll is starting to read my collection files. I have too many collections. Somthing about 2k documents. And all of them have huge files like video and audio. Maybe jekyll is reading all of those static files and it takes so much time? :thinking:

0 Likes

#12

It most certainly is reading those static files and/or transferring them to _site.
If you have that many static assets I would suggest excluding them in _config.yml and use a task runner like Gulp or Grunt to move those assets to your _site folder instead.

You can also set keep_files: [DIR, FILE, ...] in your config to prevent those files from being flushed from _site so they can persist between builds.

I do this for my personal site that has over a 1GB of images. It’s an I/O intensive thing to have Jekyll try to read and copy those files ever build so I took them out of the equation.

0 Likes

#13

I used --incremental flag cause I thought its handling such situation. I mean with this flag, shouldn’t jekyll try not to transfer unchanged static files to _site?

0 Likes

#14

Maybe, but I think incremental in this case is related to what core Jekyll does… transforms your Markdown into HTML files.

That’s what it’s incrementally building and caching, not your static assets.

0 Likes