What is the maximum data file size?

Help
1

Hi all,

I am using Jekyll with JSON data files to generate a website that consists of roughly 250 pages for each language (~8 languages).

The JSON file I am using to pull in the contents is now 18 MB in size. I was able to compress it down to 7.4 MB with the help of jq but still, my builds fail. That is on Netlify with:

...
Jekyll Feed: Generating feed for posts
◈ Netlify Dev could not connect to localhost:4000.
◈ Please make sure your framework server is running on port 4000

and locally with:

Jekyll Feed: Generating feed for posts
/usr/lib/ruby/3.0.0/erb.rb:259: warning: already initialized constant ERB::Revision
/home/user/.gem/ruby/3.0.0/gems/erb-2.2.3/lib/erb.rb:260: warning: previous definition of Revision was here
/usr/lib/ruby/3.0.0/erb.rb:367: warning: already initialized constant ERB::Compiler::Scanner::DEFAULT_STAGS
/home/user/.gem/ruby/3.0.0/gems/erb-2.2.3/lib/erb.rb:369: warning: previous definition of DEFAULT_STAGS was here
/usr/lib/ruby/3.0.0/erb.rb:368: warning: already initialized constant ERB::Compiler::Scanner::DEFAULT_ETAGS
/home/user/.gem/ruby/3.0.0/gems/erb-2.2.3/lib/erb.rb:370: warning: previous definition of DEFAULT_ETAGS was here
/usr/lib/ruby/3.0.0/erb.rb:489: warning: already initialized constant ERB::Compiler::TrimScanner::ERB_STAG
/home/user/.gem/ruby/3.0.0/gems/erb-2.2.3/lib/erb.rb:491: warning: previous definition of ERB_STAG was here
/usr/lib/ruby/3.0.0/erb.rb:830: warning: already initialized constant ERB::NOT_GIVEN
/home/user/.gem/ruby/3.0.0/gems/erb-2.2.3/lib/erb.rb:832: warning: previous definition of NOT_GIVEN was here
                    done in 645.602 seconds.
 Auto-regeneration: enabled for '/home/user/Repositories/website'

Sometimes, locally, I get the same error as on netlify.

If I use a smaller version of my JSON data file, the build works as expected.

So, I am wondering, is there a maximum file size limit for data files? I was not able to find any information on that topic.

Thank you for your help in advance.
Juliana

2

I haven’t used Netlify much, but are you possibly running into the max build-time limit for free accounts? For free accounts, the limit is 15 minutes.

2 Likes
3

Was this error on Netlify logs?

Netlify Dev should only be run locally I think.

And there is no “localhost” on Netlify. Just a Jekyll build command. And any Netlify functionality stuff like Lambdas.

Make sure you build on Netlify, not serve

4

The issue might be a huge amount of pages and processing that Jekyll can’t do in 15min, not a limit the JSON files themselves.

5

You say you get the error locally, so it sounds like it is not related to a Netlify 15min cutoff but something else getting an error or timeout

6

I am getting this error, when running netlify dev locally:

and this error when I run jekyll serve locally:

I am sorry if my previous post caused confusions.

Still trying to figure out what’s going wrong with jekyll serve --trace --verbose. I’ll post an update if there’s an error.

7

Your netlify dev command says it needs Jekyll to be running so focus on fixing Jekyll

What I tend to do when facing a weird error is remove possible causes. Like make your JSON files empty and rebuild/serve.

Then comment out your config.

Keep going until your error disappears and you have found your file that causes the error.

Or start with an empty repo and add a config and JSON files into there and like one HTML page. And keep adding more JSON content until you find the one that breaks stuff.

Can you share a link to the repo so we can reproduce the error?

8

Unfortunately the (client) Repo is private but I will share if I find the issue. Thanks for the debugging tips.

1 Like
9

Just a heads up, the errors you’re seeing when running jekyll serve suggest your gems are out of whack. Using bundler will likely fix this.

So instead of jekyll serve, use bundle exec jekyll serve.

This of course assumes you already have bundler installed locally and a Gemfile in the project.

If you don’t, the top section of the docs will get you in the right direction.

2 Likes
10

I am using Bundler already. Everything’s up to date. I’ll try bundle exec jekyll serve, though.

2 Likes