The first time I tried to build my website with --incremental flag, it took about 1000 seconds. Then immediately I tried to build again and it took 14 seconds and it is really a good result. However I restarted my pc and then tried rebuilding again. This time it took about 400 seconds to build while nothing was changed in my website. Although its still better than the initial 1000 second to build the website from scratch but its still too much while nothing is changed. I cant understand this behaviour.
I added --verbose and --profile flags to find the bottleneck however I couldnt find anything
bundle exec jekyll build -d public --incremental --profile --verbose
Logging at level: debug
Jekyll Version: 4.0.0.pre.alpha1
Configuration file: /home/hosna/Storage/repositories/zoom/_config.yml
Requiring: /home/hosna/Storage/repositories/zoom/_plugins/metareader.rb
Requiring: /home/hosna/Storage/repositories/zoom/_plugins/ext.rb
Requiring: /home/hosna/Storage/repositories/zoom/_plugins/jdate.rb
Requiring: /home/hosna/Storage/repositories/zoom/_plugins/include_absolute.rb
Requiring: /home/hosna/Storage/repositories/zoom/_plugins/jekyll-postfiles.rb
Requiring: /home/hosna/Storage/repositories/zoom/_plugins/file_exists.rb
Requiring: /home/hosna/Storage/repositories/zoom/_plugins/onebox_filter.rb
Requiring: jekyll-sitemap
Requiring: jekyll-paginate-v2
Requiring: jekyll-redirect-from
Source: /home/hosna/Storage/repositories/zoom
Destination: /home/hosna/Storage/repositories/zoom/public
Incremental build: enabled
Generating...
EntryFilter: excluded /Gemfile
EntryFilter: excluded /Gemfile.lock
EntryFilter: excluded /commit.sh
Reading: _posts/time/2018-03-15-time.markdown
EntryFilter: excluded /common/commit.sh
Reading: _daddys-home-collection/index.md
Reading: _daddys-home-collection/daddys-home-i/index.md
Reading: _daddys-home-collection/daddys-home-i/with-music/the-co-dads-battle-ii/index.md
Reading: _daddys-home-collection/daddys-home-i/with-music/index.md
Reading: _daddys-home-collection/daddys-home-i/with-music/co-dads-make-peace/index.md
////......deleted some bunch of reading log here
Reading: _thrive/part-02/index.md
Generating: ReadMetaFiles::Generator finished in 9.736257167 seconds.
Generating: Jekyll::PostFileGenerator finished in 0.002440468 seconds.
Generating: JekyllRedirectFrom::Generator finished in 2.191940422 seconds.
Generating: Jekyll::JekyllSitemap finished in 0.013961898 seconds.
Pagination: Starting
Pagination: Is enabled, but I couldn't find any pagination page. Skipping pagination. Pages must have 'paginate: enabled: true' in their front-matter for pagination to work.
Generating: Jekyll::PaginateV2::Generator::PaginationGenerator finished in 0.020414395 seconds.
Rendering: redirect.html
Pre-Render Hooks: redirect.html
Rendering Markup: redirect.html
Rendering Layout: redirect.html
Layout source: site
Rendering: redirect.html
Pre-Render Hooks: redirect.html
Rendering Markup: redirect.html
Rendering Layout: redirect.html
Layout source: site
Rendering: redirect.html
Pre-Render Hooks: redirect.html
Rendering Markup: redirect.html
Rendering Layout: redirect.html
Layout source: site
Rendering: redirect.html
Pre-Render Hooks: redirect.html
Rendering Markup: redirect.html
Rendering Layout: redirect.html
Layout source: site
Rendering: redirect.html
Pre-Render Hooks: redirect.html
Rendering Markup: redirect.html
Rendering Layout: redirect.html
Layout source: site
Rendering: redirect.html
///.... deleted some bunch of rendering logs here
Pre-Render Hooks: robots.txt
Rendering Liquid: robots.txt
Rendering Markup: robots.txt
Rendering Layout: robots.txt
Writing: /home/hosna/Storage/repositories/zoom/public/p/part/2066/index.html
Writing: /home/hosna/Storage/repositories/zoom/public/p/part/2059/index.html
///....deleted some bunch of other writing here
Writing: /home/hosna/Storage/repositories/zoom/public/p/part/2076/index.html
Writing: /home/hosna/Storage/repositories/zoom/public/redirects.json
Writing: /home/hosna/Storage/repositories/zoom/public/sitemap.xml
Writing: /home/hosna/Storage/repositories/zoom/public/robots.txt
Writing Metadata: .jekyll-metadata
Filename | Count | Bytes | Time
-----------------------+-------+----------+------
sitemap.xml | 1 | 310.97K | 0.849
_layouts/redirect.html | 2095 | 1555.54K | 0.066
robots.txt | 1 | 0.04K | 0.000
done in 471.909 seconds.
Auto-regeneration: disabled. Use --watch to enable.
I just relized that there is a great time gap between the line
EntryFilter: excluded /common/commit.sh
and
Reading: _daddys-home-collection/index.md
I dont know what is being done there.
In my experience, the first run (e.g. “cold start”) of Jekyll after a reboot is always slower.
I’ve noticed this with tools like Gulp as well. I think it has more to do with the environment getting setup and run that first time, than anything related to Jekyll or it’s incremental mode.
I don’t think it would make sense to be slower as much as 400 seconds.
You’d be surprised.
Every time I start up Jekyll after my computer was restarted, it takes awhile for it to get going. Each build after is about 50% faster, and I don’t use Jekyll’s incremental mode.
I don’t know the internals of Ruby, but I’m assuming there’s some stuff held in memory that speeds up tasks after you’ve done that initial run. Restarting your device clears all that.
do you know why the bottleneck isn’t logged when I’m adding --profile flag?
It’s not a Jekyll bottleneck. It’s your Ruby environment getting started from a fresh reboot. The delay is with that… and before Jekyll even runs.
No. The delay is here:
I just don’t know what is being done there.
Sure, but is that gap a 600s one? I’m willing to bet a bulk of the delay is before Jekyll even starts after you restart.
The other delays are normal content related things as Jekyll builds your site. I don’t think the profile flag will surface that as it just shows you the times of layouts and includes, not reading files… which is where those EntryFilter lines happen.
Also it’s my understanding incremental builds have quirks to them. Which would explain why your results aren’t consistent.
the build after restart takes 471 seconds and I think that delay I’m talking about is taking about 450 seconds.
to add more information, in this line Reading: _daddys-home-collection/index.md it seems jekyll is starting to read my collection files. I have too many collections. Somthing about 2k documents. And all of them have huge files like video and audio. Maybe jekyll is reading all of those static files and it takes so much time? 
It most certainly is reading those static files and/or transferring them to _site.
If you have that many static assets I would suggest excluding them in _config.yml and use a task runner like Gulp or Grunt to move those assets to your _site folder instead.
You can also set keep_files: [DIR, FILE, ...] in your config to prevent those files from being flushed from _site so they can persist between builds.
I do this for my personal site that has over a 1GB of images. It’s an I/O intensive thing to have Jekyll try to read and copy those files ever build so I took them out of the equation.
I used --incremental flag cause I thought its handling such situation. I mean with this flag, shouldn’t jekyll try not to transfer unchanged static files to _site?
Maybe, but I think incremental in this case is related to what core Jekyll does… transforms your Markdown into HTML files.
That’s what it’s incrementally building and caching, not your static assets.