Publishing a subset of pages from Roam Research

1

I’ve been using roam research to make a digital garden, a lot of people are interested in this too. I am the first to be using roam and Jekyll, based on my extensive googling and forum searching.

While blogs are oriented around chronological updates, with posts not updated very often. Digital gardens tend to be interconnected, with pages updated regularly.

Roam exports a json blob, not directly usable by Jekyll. So, I have a rake task to turn the blob into markdown pages with proper links between pages.

I store a lot of stuff in roam, and I’d only like to publish a subset of pages from roam, those linked to the page called ‘START HERE’. In python, I a script that parses the markdown and uses networkX to walk the graph of pages, and returns a subset around what I want my blog to be about.

The issue is this so called ‘build process’, requires python, and a ton of dependancies. Redoing the ego graph process in ruby would require some native extensions, which might be a turn off.

I feel like other people would be interested in topic based, roam research backed, digital gardens running Jekyll, so I’m interested in simplifying this process as much as possible, even releasing a plugin for the border community.

Should turning the export into pages be part of the Jekyll build process, or kept as a rake task?

Would having a python dependency (networkX and scipy) be less problematic than requiring a library that builds a native extension?

2

Hi. There are some recent conversations on digital gardens in the forum which you might be interested in, but I don’t recall hearing about Roam there.

I have experience with Jekyll, Python and Ruby plugins so would like to give my input.

Your flow is to filter markdown files and only render those in the Jekyll build step. Whether you do it with Python or Ruby, I’d first like to mention that this will not work on GH Pages (since you cannot use custom shell commands or custom plugins). Therefore you solution is suited to a CI flow like GitHub Actions (to support GH Pages), or Netlify, or some other flow where you run the code on a local machine or self-hosted server.

3

Between Ruby and Python, I think Python is more widely used and you have a solution that already works in Python, so that goes like a good bet. Yes there is a trade-off that it is not a Ruby plugin, but going Ruby or Python there will be developers in the community who can contribute to it.

Using Python means you can’t make a plugin, but I think this is fine. It means you treat your Roam filter service and your Jekyll build service as two distinct steps and you can update or replace either step without breaking the other, as long as you keep the formatting of the data consistent between them. If you make a plugin, then it runs whenever Jekyll build is run, which means makes it more opaque.

I actually have a project where I fetch API data using a script and then use the data to build my Jekyll site. I made it as a Ruby plugin as a challenge for myself, but the downside is that I only run the plugin as part of the build step and I can’t actually see the debug output. I am actually thinking of rebuilding in Python (which I am more fluent in), where the API step fetches data and writes out a JSON file, then the Jekyll plugin is very light - just reading in a JSON file of an expected format. This makes debugging a lot easier as I can run the API step independently and inspects its output, without even having Jekyll installed or running.

If you are familiar with Unix programming or SOLID programming or microservices, you’ll appreciate the simplicity and elegance of having two tasks which each do one task well. In Unix shell scripting, you pipe text from one command to the other, which means you can swap out commands. As opposed to one monolith command that does everything.

4

Regarding the Python implementation,

You can make a package hosted on PyPI or even just on GitHub.

You can install from GitHub like this:

pip install -e git+https://github.com/MichaelCurrin/python-package-quickstart.git@master

Or put this in requirements.txt, including the -e`.

-e git+https://github.com/MichaelCurrin/python-package-quickstart.git@master

Here are my Python quickstart project’s docs with a tutorial, if you are interested https://github.com/MichaelCurrin/python-package-quickstart/blob/master/docs/installation.md

When someone installs you Python package, they’ll get your networkX and scipy dependencies installed. Ideally all in a virtual environment at least locally. When comes to GH Actions and Netlify, Python is already installed and there is no need for a virtual environment. In fact, Netlify will find your Jekyll project’s requirements.txt file and install for you - so you could make your Jekyll site or a template available with deploy to Netlify button. Or just a GH Actions workflow file. If you need more info on Netlify or GH Actions, I have links to share and would recommend both for static sites and web apps.

Then to actually build the site, the build command would be (assuming your package is called roam_filter)

roam_filter # or maybe python -m roam_filter
bundle exec jekyll build

You can run them separately for easy local debugging - you can view the output generated from step 1 before running step 2. Assuming the first command outputs to a temporary unversioned directory of jekyll files which are used to build your site.

Or chain them together in a Makefile or Rakefile or Netlify build command.

python -m roam_filter && bundle exec jekyll build

Whichever way you go, if your install and run instructions are clear, then someone doesn’t even have to be a Python or Ruby dev to use your tool.

The disadvantage though is that it someone may not have Python installed. It’s only a few steps though.

# Install PY
sudo apt install python3
# New env for dependencies
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

See my mac and Windows instructions here

You can assume they have Ruby installed if they are using Jekyll.

So if you do want to go the Ruby route, you can write your tool in Ruby. And install as like

gem install 'roam_filter'

And then run as

bundle exec ruby roam_filter
bundle exec jekyll build

Your tool is still installable gem and you can run on the CLI, just not running as a Jekyll plugin.

By not using a plugin, you can freely move between a tool written in Python or Ruby and have your Jekyll build command read from an expected directory.