That performance seems normal (really better than normal) – testing on a recent laptop performs at about half that speed. Jekyll is single-threaded, so all files are processed one at a time.
Have you tried incremental builds?
About static files: since Jekyll is just copying them, the performance is limited by how fast the file-system can copy. Most OSes now come with “Copy-on-Write” file-systems which makes all copying operations near-instantaneous (no matter what the file size).
The reason that I want to rebuild every minute or so is that there are many users of this site who might update some of the files as they use it. They would then have to wait for a rebuild to see the change - or for other users to be able to see their change… Of course not 500 files are changed on every build. Typically 1-2 files are. But most of the time there will be no changes.
I tried --profile
Not sure what to do with it though because the site render stats just lists filenames which all takes sub second time. The build process summary mentions that render and cleanup phases take about 13-14 seconds each… So this is the big part of the build time. Not sure if possible to do anything about render time. And I dont know what cleanup is?
What exactly did you do? Maybe some other users might benefit from your improvements.
But that is exactly how Jekyll works. Also, why would you want to rebuild the site if no changes were made? This is just wasting resources. I think it would be better to create some CI/CD solution to trigger the build when a file is changed, and that is quite easy to do. If the user is running the site locally, it is best to use jekyll serve instead of build.
why would you want to rebuild the site if no changes were made?
I dont. So I guess I should instead have a cron job that first checks the VCS is there are any new changes. And only if there are new changes does it rebuild using jekyll.
I have a few other jekyll sites that are somewhat similar but much much smaller. Those I simply have a cron job that updates the VCS and then runs jekyll build. But those only takes a couple of seconds.
Users are not running site locally. They commit pages from various other systems.
An alternative to a per-minute cron job might be to continuously run jekyll build with the --watch option, so it watches the source files and rebuilds when needed.
About incremental-builds, I think the benefit doesn’t start until the 2nd build. The first incremental build is a full build, but Jekyll records a dependency graph. After that, Jekyll uses the dependency graph to only rebuild what is needed. Of course, as @george-gca pointed out, it is still experimental, so careful testing is needed to see if it will work for a particular setup.
Don’t you use any git server like GitHub or Gitlab? Both support CI/CD. If not, your solution should be with jekyll build --watch as pointed by @chuckhoupt. Maybe you could also add --force_polling? For more options, see the docs.
Using globbed-paths (containing /* or /**) in front-matter defaults can have negative effect on build times on some OSs. The equivalent directive (in your specific use-case) is to use the key type:
Disclaimer: The above optimization may or may not yield noticeable gains. But is a better alternative regardless.
The huge time taken during the CLEANUP phase could be due to the large number of static files in your site. Jekyll 4 hasn’t really focused on optimizing that phase since it was never brought up as an issue. I will look into it and see if anything can be done.