Build posts to a text file?

BillRaymond · January 13, 2021, 6:55pm

When creating a blog in Jekyll, I write it in markdown. Then, Jekyll converts that markdown to HTML.

I would like to add an extra step in that process that converts the markdown to text, stripping out the yml front matter and any markdown items. Alternatively, it could convert the HTML output to text, but the idea is I would have a .txt file that anyone could read with no markup contained within.

Looking online, I see lots of web services and paid APIs that can do such a conversion, but I am looking to add the converter such that any Jekyll build does the conversion for me. Alternatively, rather than doing the conversion during a Jekyll build, it would do the conversion using a GitHub Actions workflow. Either way, I would prefer this to be an open-source solution.

Has anyone had any experience with this? I suppose it would not be incredibly difficult to create a converter since there are plenty of straightforward ways to strip markup language from a file, but if something already exists, I would appreciate using that.

Unfortunately, my searches have not pulled anything up on this matter so I am searching for the wrong thing or there is an approach I have not thought of

Thanks!

MichaelCurrin · January 13, 2021, 9:03pm

I think you can do this within Jekyll.

So you want to turn

## My heading

- A
- B

Into

My heading

A
B

Maybe keep structure of bullet points?

Anyway you can do this in a layout. Say your post layout.

---
layout: nil
---
{{ content | markdownify | strip_html }}

That would render markdown in the body as HTML and then strip the tags to make it plain text.

You might do something as extra step before stripping HTML that uses replace. For example to add an extra line where is single newline or bullet point. To add in a literal dash inside each li tag.

My approach above replaces all posts with plain text versions in .html files still.

You could use a Ruby plugin based on Generators in the docs, to give you two outputs for each one. One traditional post like foo.html and another as foo.txt with only text by using my logic above.

MichaelCurrin · January 13, 2021, 9:12pm

You could go a step further and use JSON instead of text to build a kind of API.

/api/posts/2020/01/02/foo.json

{ "content": "Body of the post.\nSecond line."}

I’d say a JSON API would be powerful. You could build a backend and then use a React app on the same or different domain to serve your content. Other people could load content of a given post on their frontend.

MichaelCurrin · January 13, 2021, 9:14pm

If you are happy with JSON approach, you can save yourself a lot of trouble around making a plugin and dealing with paths of text files which only contain a bit of data. And figuring out how to paths all those paths available like in a site map.

You could out a single data.json file at the root of your repo. With frontmatter and liquid. You can iterate through all posts and pages in your repo and output them as items in the list. Or as a hash, using path as a key.

The content field for each item can use the conversion approach in my first comment.

The result will be larger than many small files, but still reasonable for even a 100 posts because it is next and not styling.

It will be convenient to build and then on the other side to consume that single file. Using a single known path.

You can also more easily use it as a search index. Or page index for a React app. Or Next.js or whatever which reads pages from JSON data.

I have a gist to get you going. I’d help with the code or a proof of concept if you like.

gist.github.com

https://gist.github.com/MichaelCurrin/f8d908596276bdbb2044f04c352cb7c7

README.md

# Serving JSON as a Jekyll REST API

<!-- Note to self: This gist was created as part of a PR discussion for a tutorial on Jekyll docs. That tutorial links to this gist, which is kept here as a permalink and should not be moved to repo -->

How to make a read-only JSON REST API using Jekyll. This doesn't need any Ruby plugins - you just use some builtin templating features in Jekyll 3 or 4.

You might have data stored in your Jekyll project, such as in a YAML, CSV or JSON file in the `_data/` directory or in the frontmatter of your pages. This tutorial shows you how to render as JSON pages.

This is useful if you want to make your data available for another service or for others to consume (like GitHub, Twitter and Facebook make their data available on APIs). Or perhaps you have JavaScript single-page application that reads from your API backend to serve the app or build a search index.

This file has been truncated. show original

page.md

## Page

- `_layouts/page.html` - build up the JSON object, quoting all values and in the case of the body text we convert from markdown to HTML.
    ```liquid
    ---
    layout: none
    ---
    {"title":{{ page.title | jsonify }},"content":{{ page.body | markdownify | jsonify }},"links":{{ page.links | jsonify }}}
    ```
- `foo.json` - a Jekyll HTML/markdown page with a `.json` extension.

This file has been truncated. show original

post-file.md

## Post

### Input file

Here we make a post in the usual `_posts` directory.

The layout is also much lighter and more flexible - using the `data` object without caring about keys, values and formatting.

The downside is that you have to be more verbose in your post frontmatter to define a YAML variable with `&` and then use it with `*`. Based on this [YAML tutorial](https://idratherbewriting.com/documentation-theme-jekyll/mydoc_yaml_tutorial.html).

This file has been truncated. show original

There are more than three files. show original

BillRaymond · January 14, 2021, 12:41am

Thank you for the thoughts. I had been thinking if there is a post called “2021-01-01-my-blog.md”, there would be a new “2021-01-01-my-blog.txt”. However, your idea of doing a JSON output to a single file might be easier and more straightforward. Okay, I will look into this idea before I go too deep down the rabbit hole. Also, I like the idea of not having a lot of extra files.

Thanks for the idea!

MichaelCurrin · January 14, 2021, 8:26am

To clarify,

.md file would build as .html
And a generator would make the .txt at the same path (or another path)

If you start with naming your post .txt with frontmatter then it should output as .txt (but only one file). It might also not be recognized as a post if Jekyll needs .md for posts. But pages should be fine with any 3extension.

If you use data.json with frontmatter, it will build as a file data.json as well, keeping the extension. I’ve tested this out before.

MichaelCurrin · January 14, 2021, 8:26am

Glad to help. Hope one of my ideas works out

MichaelCurrin · January 15, 2021, 10:01am

@BillRaymond I put together this gist for you after local testing.

gist.github.com

https://gist.github.com/MichaelCurrin/7e75f6c04d1cd14f2c6583f9fad1d218

README.md

# Jekyll API
> Render content for all Jekyll pages as a single JSON endpoint


## Setup

1. Copy the content of the JSON file below.
2. Add to your project as JSON file e.g. `data.json` in your project root. Or `api/data.json`.
3. Start your Jekyll server.
4. View the page e.g. `http://localhost:4000/data.json`

This file has been truncated. show original

data.json.liquid

---
---
{%- assign pages = site.pages | where_exp: "p", "p.layout != nil" -%}
{
  {%- for p in pages %}
    {{ p.url | jsonify }} : {{ p.content | markdownify | strip_html | jsonify }}
    {%- unless forloop.last %},{% endunless -%}
  {% endfor %}
}

sample-output.json

{
    "/404.html" : "\n\n\n    404\n\n    Page not found :(\n    The requested page could not be found.\n\n",
    "/about/" : "This is the base Jekyll theme. You can find out more info about customizing your Jekyll theme, as well as basic Jekyll usage documentation at jekyllrb.com\n\nSource code:\n\n\n\n\n",
    "/" : "\n\nWelcome to my Jekyll Blog Demo\n\n\n  {{ site.description }}\n\n\n\n\n\n\n\n    \n        \n    \n\n"
}

I forgot about the limitation where Jekyll filters can’t render Liquid expressions. So you’re going to need to use a plugin if you want to evaluate {{ }} or {% %}.