How much time it will take to generate 500000 pages

Help
1

i have simple site with default theme. how much time it will take to generate 500000 pages

2

That’s too broad a question. There are factors like your computer speed. Are you running a Pentium from 1993 or a 10th gen i7? What’s the site structure? Do you have 15 includes in every document or are you passing the page content direct to the default layout?

You might find the article, How I reduced my Jekyll build time by 61%, interesting.

My best guess is, using Jekyll v4 and a decent processer on an *nix system, it’ll take you about 60-70 seconds. This supposes you aren’t running a boatload of plugins and have made some effort to optimize the site.

1 Like
3

60 seconds seems a bit optimistic. Below is a naive script that creates and builds a default-theme Jekyll 4 site with 1000 pages (dups of About page). On an i7 iMac (~5 years old), it builds in ~30 seconds.

If the speed scales linearly (O(n)), then 500K pages would take ~4 hours. Unfortunately, it appears to scale quadratically (O(n^2)), so it may take 80+ days!

Of course, this is an extremely naive approach on an older machine. A faster system and various optimizations might achieve linear speed with a few doublings – maybe it could be reduce to ~1hr?

Ultimately, rather then try to force Jekyll to build a 500K site on its own, it would be better to split the site into pieces, so Jekyll can be run in parallel. Then a 256-core system could probably build 500 1K page sites in a minute or two.

#!/bin/bash
set -Eeuxo pipefail

: CPU and Version Info
[[ -d /proc ]] && cat /proc/cpuinfo | grep 'model name' | uniq || sysctl -a | grep brand_string
ruby --version
jekyll --version

: Create a new Jekyll site with default theme
TEST=/tmp/jtest
rm -rf $TEST && jekyll new $TEST && cd $TEST

: Only list About page in menu 
printf "header_pages:\n  - about.markdown\n" >> _config.yml

: Create copies of About page without permalink
sed 's/permalink:.*//' about.markdown | tee page{1..1000}.md

: Build
time bundle exec jekyll build
1 Like
4

Just a quick anecdote, I have a 90k page Jekyll site that builds in 40-60 minutes.

It’s not optimized for build speed at all…lots of includes, bunch of plugins, multiple collections, etc.

I develop locally with a subset of the pages (~100) which builds in seconds. When I push it to production a CI/CD service takes over for the actual build, so build times don’t really matter. YMMV.

Hope that helps.

2 Likes
5

Wow! That’s a HUGE site. Is that site open-source via a public GitHub repository?
If it is I’d like to fork and run profilers on it…

6

Yup, here’s the repo: GitHub - grantmakers/profiles: Grantmakers.io Profiles - Summary profiles for all US-based foundations who have filed electronic IRS Form 990-PF.

There’s a smaller sample dataset in the repo, but if you (or anyone else) want to run profilers with the full dataset PM me and I’ll send you a link. I had to take the full dataset out of source control as I kept blowing past Github soft limits for repo size.

And @tejas456sawant, though my use of collections might potentially be useful as you explore how to pull data into your project, please note my project is basically a lesson in how NOT to optimize Jekyll code for build speed :slight_smile:

2 Likes