It seems like most blogs I read these days are built using a static site generator (SSG).
Some of these SSGs come with shortcodes for embedding content from popular websites, others don't, but shortcodes are generally the accepted way of embedding a reference to something someone else said on the internet in one of your posts.
I recently wrote about how brittle I find these shortcodes that call out to external websites, and instead wrote One Shortcode to Rule Them All which references things directly from my own personal knowledge base.
But we can still do better.
{{<tweet id="12345">}} and on first build, it downloads the profile image, name, tweet content, and timestamp and renders that, and it works forever.
I keep finding broken tweets on my Hugo site because Twitter's rate limiting something or the user deleted their tweet. I've resorted to just screenshotting, which feels sloppy.
This is also something that I have thought about before, but writing a middleware layer to cache referenced content from disparate sources to plug in to SSG codebases I'm not really familiar with was never an idea that really pulled me in.
I too had previously resorted to screenshotting and embedding images, but that process doesn't feel good, and I don't like adding more and more image files to my git repo. My requirements are also a bit simpler because I don't really care about rendering a user's profile picture from an external site or showing a timestamp.
Since I'm now pulling in all my quoted external content from a single data source, I can implement a pre-build step which will allow me to version control all of the external content referenced in my articles. I just need three things:
- A list of sources
# sources
https://m.mtlynch.io/@michael/115538492543985760
- A file which maps those sources to the data required to render them
// library.json
{
"https://m.mtlynch.io/@michael/115538492543985760": {
"title": "michael",
"source_display": "m.mtlynch.io",
"source_url": "https://m.mtlynch.io/@michael/115538492543985760",
"content": "@LGUG2Z I wish that SSGs would store the data from the remote URL in a file you could keep under version control. e.g., for tweets, I do something like `{{<tweet id=\"12345\">}}` and on first build, it downloads the profile image, name, tweet content, and timestamp and renders that, and it works forever.\n\nI keep finding broken tweets on my Hugo site because Twitter's rate limiting something or the user deleted their tweet. I've resorted to just screenshotting, which feels sloppy."
}
}
- A shortcode which can look up and render the data using the source URL
<!-- library.html -->
{% if url %}
{% set library = load_data(path="library.json") %}
{% set quote_data = library | get(key=url) %}
{% set use_markdown = markdown | default(value=false) %}
<div class="notado-quote"
style="border: 1px solid var(--border-color);
background-color: var(--bg-primary) !important;
position: relative;
margin-block: 1em;
border-radius: 5px;
padding-left: 1rem;
padding-right: 1rem;
padding-top: 1.25rem;
padding-bottom: 1.25rem;
{% if caption %} margin-bottom: 0em;
border-bottom-right-radius: 0px !important;
border-bottom-left-radius: 0px !important;
{% endif %}">
<div style="padding-bottom: 1.25rem">
<div style="display: flex; gap: 0.75rem">
<div style="min-width: 0;
flex: 1 1 0%;
display: flex;
flex-direction: column;
justify-content: center">
<p style="text-overflow: ellipsis;
overflow: hidden;
white-space: nowrap;
margin: 0em">
{{ quote_data.title }}
</p>
<p style="color: var(--text-1); margin: 0em">
{% if quote_data.source_url %}
<a href="{{ quote_data.source_url }}">{{ quote_data.source_display }}</a>
{% else %}
{{ quote_data.source_display | split(pat=" - ") | first }}
{% endif %}
</p>
</div>
<div style="flex-shrink: 0;
display: flex;
flex-direction: row-reverse;
align-items: center">
<a class="notado-icon" href="https://notado.app" style="border: none">
<img style="height: 3rem;
width: 3rem"
src="https://notado.app/static/notado-icon.png"
alt="notado" />
</a>
</div>
</div>
</div>
{# djlint:off #}
{% if use_markdown %}
<div>{{ quote_data.content | markdown(inline=true) | safe }}</div>
{% else %}
<div style="white-space: pre-line">{{ quote_data.content }}</div>
{% endif %}
{# djlint:on #}
</div>
{% if caption %}
<div class="notado-quote-caption"
style="border: 1px solid var(--border-color);
background-color: var(--bg-2) !important;
position: relative;
margin-bottom: 1em;
border-bottom-left-radius: 5px;
border-bottom-right-radius: 5px;
padding-left: 1rem;
padding-right: 1rem;
padding-top: 0.5rem;
padding-bottom: 0.5rem">
{{ caption }}
</div>
{% endif %}
{% endif %}
The first two pieces are the really interesting ones.
I maintain a CLI tool for my knowledge base which I extended with a command which takes a list of source URLs and updates a library file.
❯ notado-cli quote-gen
Adding quote data for https://bsky.app/profile/hmsnofun.bsky.social/post/3lmr6sm5k4k2b to library.json
Adding quote data for https://defcon.social/@corbden/113473397794111625 to library.json
Adding quote data for https://discourse.nixos.org/t/should-organizations-relating-to-the-defense-sector-being-able-to-sponsor-nixos/41252/6 to library.json
Adding quote data for https://lobste.rs/s/rzskjk/i_think_i_m_done_thinking_about_genai_for#c_l9x7we to library.json
Adding quote data for https://old.reddit.com/r/patientgamers/comments/udzo11/i_miss_the_days_of_server_browsers_and_community/i6lga1o/ to library.json
Adding quote data for https://programming.dev/comment/5789966 to library.json
Adding quote data for https://tildes.net/~tech/17xe/permanent_archival_formats_do_they_exist#comment-9feo to library.json
Adding quote data for https://twitter.com/mitchellh/status/1744850961309597855?s=12 to library.json
Adding quote data for https://www.youtube.com/watch?v=k0J0Dxf5JKc&lc=Ugwsmg7c0JpYiXyADQV4AaABAg to library.json
Whenever a new source is added and the command is run again, requests will only be made to fill in the library file with new data:
❯ notado-cli quote-gen
Skipping https://bsky.app/profile/hmsnofun.bsky.social/post/3lmr6sm5k4k2b as data is already in library.json
Skipping https://defcon.social/@corbden/113473397794111625 as data is already in library.json
Skipping https://discourse.nixos.org/t/should-organizations-relating-to-the-defense-sector-being-able-to-sponsor-nixos/41252/6 as data is already in library.json
Skipping https://lobste.rs/s/rzskjk/i_think_i_m_done_thinking_about_genai_for#c_l9x7we as data is already in library.json
Adding quote data for https://m.mtlynch.io/@michael/115538492543985760 to library.json
Skipping https://news.ycombinator.com/item?id=41841873 as data is already in library.json
Skipping https://old.reddit.com/r/patientgamers/comments/udzo11/i_miss_the_days_of_server_browsers_and_community/i6lga1o/ as data is already in library.json
Skipping https://programming.dev/comment/5789966 as data is already in library.json
Skipping https://tildes.net/~tech/17xe/permanent_archival_formats_do_they_exist#comment-9feo as data is already in library.json
Skipping https://twitter.com/mitchellh/status/1744850961309597855?s=12 as data is already in library.json
Skipping https://www.youtube.com/watch?v=k0J0Dxf5JKc&lc=Ugwsmg7c0JpYiXyADQV4AaABAg as data is already in library.json
In order to make diffs more pleasant after updates to the library file, I decided to serialize it using a BTreeMap so that the serialized JSON object will be sorted alphabetically by the URL keys.
With all these pieces in place, I can call my new shortcode to render this data without making any outgoing HTTP requests at build-time... Just like I did earlier in this article!
{{
library(
url="https://m.mtlynch.io/@michael/115538492543985760",
caption="Michael is on to something... Embedding his comment here didn't trigger a HTTP request when this article was built!",
markdown=true
)
}}
One of my greatest pleasures in life is building my own tools which work the way that I want them to work.
The initial feeling of gratification upon their completion soon fuels even more creativity in me as I slowly uncover new ways to integrate all of the different tools that I have built to target more and more of the papercuts that are just waiting to be addressed in the background of my life.
If you have any questions or comments you can reach out to me on Bluesky and Mastodon.
If you're interested in what I read to come up with solutions like this, you can subscribe to my Software Development RSS feed.
If you'd like to watch me writing code while explaining what I'm doing, you can also subscribe to my YouTube channel.
If you would like early access to komorebi for Mac, you can sponsor me on GitHub.