Efficiently inefficient

For years I have been manually taking notes as I read through articles. If I found the article worth remembering, I would reference the key parts in a note then file the note under a relevant topic. For instance, notes taken from an article on "Pytest best practices" would go under Python. Keeping notes organized like this was exhausting work. However, it was critical for searchability, particularly when I needed to recall the ideas again.

Inevitably, fractional notes started to form across my work computer, phone and laptop. It didn't help matters that I would infrequently glue the fragmented notes together, so most of the time, I would abandon orphaned notes altogether. I soon realized reading articles leisurely, without saving notes, was more productive as I could read more content in the same amount of time it took to write notes. The problem with that is that relying solely on long term memory isn't feasible.

There have been numerous articles I've pulled my hair out while scouring my browser's history attempting to locate an invaluable article. A good example of this was when I wanted a junior developer to have the same framing when writing tests as I had. I once read a series of articles by Kent C. Dodds on the fundamentals of React Testing Library which changed the way I write tests to this day. While Kent still has useful posts around the topic, I could no longer find the ones I had once revered. I suspect this is because the information has been moved into a testing course behind a paywall. This meant I couldn't rely on my browsing history.

Dipping into extensions

A sense of relief and excitement washed over me as I realized I could finally create a frictionless way to collect highlights across the web. This was possible because I had accrued an ecosystem of projects from a component system to a note taking system. These projects unlocked the ability to rapidly create my own browser extension-based bookmark manager. This bookmark manager could send slices of the web to my notes system via an API.

This tool can be found as Queso^1 — it's an open-source Chrome extension. Nowadays, when I'm browsing the web, I'll bookmark the articles I want to read later.

Beyond bookmarks

To help burn down articles I want to read, I have a job that generates a newsletter with articles I haven't read yet (totaling to 30 minutes of read time) and emails the newsletter to me daily. Each bookmark in the newsletter has a link to the article and a link to the bookmark in my notes system.

Queso is also built in a way that I can simultaneously read through an article and jot my musings in Chrome. After completing an article I can also mark that I have read it using Queso. The biggest time-saver came from automatically linking the web article's metadata. Queso pre-populates the article's URL, title, author, and generates estimated read time for every bookmark. As a start, Queso was a reading-based to-do list for unread content and a digital breadcrumb trail for the rest.

As I was dogfooding the app for a few months, I found that often I wanted to save quotes around the place. So I added a feature where on any webpage I highlight a portion of text, right click and Voilà!

The quote is now in the notes field and formatted as a markdown quote. Quotes work similarly to bookmarks, pre-populating the reference source's metadata with cited references to the URL and author. This feature will be particularly useful to embed quotes in my own blog posts!

Here's how that quote looks in my notes system:

Above and beyond

I had been using bookmarks and quotes for a couple of months when my digital hoarding kicked into overdrive. An itch to save the things I learned throughout the day needed to be scratched. Some might think this is a time sink. But as a software engineer, in the midst of an AI boom, I learn a plethora of new things regularly. Besides, bookmarking the things a one learns throughout time isn't novel. In fact, this is commonly known as a Today I Learned (TIL) list. Some developers have chosen to keep their own TIL as repositories on GitHub. My TIL would function similarly, but live in the same place as my bookmarks.

Adding support for TIL was straightforward, as it functions similarly to quotes and bookmarks. It was helpful that my notes system stores notes in Markdown, so when there are parts of code I find inspirational I can save code inside code blocks. Here's a TIL entry from when I learned about JSON-LD . Its tagged with SEO so I can quickly reference it next time I'm setting up a new marketing site.

An internet swiss knife

I've recently added support for Queso to take a snapshot of an article's written content when a bookmark is added. The snapshot takes the article's HTML content, pops it into Markdown format, and is uploaded with each bookmark. Thus preventing articles lost to link rot. So preserves not only my notes on the subject, but also the cited source's content!

Here's a demonstration of the snapshot feature in action using an article I wrote. The article contains relatively complex page elements for a blog, such as code blocks and tables. In one click with Queso I have now cached the article content and in a consecutive click I have uploaded it to my PKMS.

Preserving the article's content doesn't only prevent against data loss. It unlocks a heap of benefits such as the ability to read articles offline. Another benefit that may not be obvious is auto-linking related content using text-embeddings. As these links accrue, a network of inter-connected content highways start to form. This would act similar to manual cross-linking notes using Obsidian's Backlinks.

An important distinction from links in Obsidian is that regardless of when the note was made, it can easily be resurfaced when recent content intersects with old content. For example, 10 years ago you wrote about your Atkins diet, today you create a note about your new carnivore diet. The decade old Atkins diet note pops-up in relevancy and you revisit. This automated discoverability helps reinforce and compound knowledge over time.

Closing thoughts

There are plenty of options when it comes to retaining knowledge online. There are already several bookmark mangers that work for others. For me, Queso has drastically improved the way I retain web-based content, so I plan to keep using it for the foreseeable future.

For those interested in using Queso themselves, it's available for free on the Chrome web store. There are configurable settings so that it can be easily hooked up to upload bookmarks through an API. With services like Firebase, Supabase, or a DIY bookmark system can be set up quickly to start using Queso. Queso is also open-source for anyone interest in contributing to the project or repurposing the code.

"You can’t really know where you are going until you know where you have been." - Maya Angelou.

The age-old mission to capture and make sense of our knowledge and experiences echoes through history, from Thomas Jefferson's Commonplace book detailing his ideas on government to Marcus Aurelius's Meditations saturated with quotations and personal reflections.

However, this pursuit to preserve knowledge isn't without its own set of fears. Are our deepest thoughts private? Are we spending a disproportionate time customizing our notes system than benefiting from it? Will our preferred notes system last the sands of time?

These were the questions that motivated me to step outside conventional offerings and build my own solution. I'm sharing my story here not to prescribe, but to demonstrate it's okay to color outside the lines. Perhaps my journey to create a simple, secure, and lasting 'note vault' will spark ideas for how you can better cultivate your own knowledge garden.

Conventional knowledge gardens

A personal knowledge management systems (PKMS), also known as "second brain" is are central repositories for actively collecting one's meaningful insights, ideas and inspirations encountered throughout life. The benefit of decades of notes accruing and reinforcing knowledge is immense. That is why critically analyzing what you need your PKMS to achieve is crucial.

The most commonly used PKMS or note-taking apps today are Notion, Obsidian, Evernote and Logseq. The problem is that PKMS come and go. Could you see yourself using your note-taking app you use today in 30 years? Probably not. Do you ever have concerns around the privacy of your notes? Are you spending more time setting up your notes system rather than managing your notes? What does an effective and timeless PKMS even look like?

Pulling at a thread

My PKMS journey started with Obsidian. For those unfamiliar with Obsidian, it's a digital notebook that lives only on your computer. Its special feature is letting you internally link related notes together similar to Wikipedia. Beyond linking, its true power comes from the huge range of plugins. Out of all the great community plugins, Dataview is my personal favorite. Dataview is a powerful scripting tool which pulls data from markdown notes to generate tables, graphs, and other insights.

Obsidian was a great tool for me personally for a long time. But I felt frustrated when I wanted to access my notes on my phone while on-the-go and saw that I had to pay for this feature. Obsidian charges $8 a month to access the same notes across multiple devices. While not a huge amount for such a useful app, it adds up to an eye-watering amount - almost $1,000 if I planned to use Obsidian for a decade. I was surprised at this fee because I thought Obsidian was open-source. I later realized that the Obsidian community plugins are open-source, but Obsidian itself is, in fact, not. These were the same plugins I was spending an inordinate abount of time personalizing Obsidian with, rather than being productive.

I started to have concerns about the longevity of the plugins and app itself. Some of you may remember when Evernote aggressively limited free users to 50 notes, many users migrated their notes elsewhere. I was one of those users.

After some mental gymnastics weighing if I should continue with Obsidian, I found solace when asking myself "Can I see myself using this in 20 years?". I couldn't. The thought of cyclically migrating notes from one PKMS to another every 5 years, as I had done from Evernote to Notion to Obsidian, made me feel tired. So I grabbed some coffee and began my search for the digital equivalent of the Svalbard Global Seed Vault, but for my notes.

My perfect note vault

Key requirements were that my new PKMS needed to be easy to use, offer a plugin-like experience and be secure. Privacy and security were paramount. I didn't want my innermost thoughts being exposed to a 23andMe data leak scenario. Data security aside, I have a hard time trusting for-profit companies will keep their hands out of the cookie jar. I can imagine there's a killing to be made from targeted advertisements online using your notes or for them to be used to train AI. I wanted to know with certainty that I control my data. Entrusting a company to store my notes for free, couldn't give me that assurance.

Being a full-stack software engineer, the answer was obvious. I should step outside the commercial-PKM ecosystem entirely - I should build my own.

But if it's so obvious, why aren't other developers rolling out their own PKMS? Perhaps I'm the first to discover this or perhaps developers aren't writing about their custom PKMS. My guess is that commercial note-taking apps have larger, more vocal communities that drown out the murmurings of other DIY solutions.

It was here I decided to pull the trigger to build and host my own PKMS. It felt pertinent to write about it so others feel confident to do so too. Spoiler - it sounds like a daunting feat, but in reality it was so comically easy my only regret is that I hadn't done it years ago.

The note vault

Here's how my PKMS looks as of writing. I can create a note, update it in markdown format and then preview what the note looks like in rendered markdown.

Finally, I can access my notes from my phone too, right where I left off. No monthly fees for this feature either.

Since my PKMS is hosted online to manage notes across devices, I have multiple layers of security to ensure my notes are kept private.

And if I ever want to move to greener pastures, all my notes are stored in Markdown so I can export them with two command lines to my terminal. Here's an under-the-hood view of how my notes are stored in the database - just text! ^1

Benefits of a note vault

I've found that consistently collecting and reviewing valuable information promotes a deeper engagement with ideas. As a result, I've noticed my memory improving, and I started seeing surprising links between seemingly unrelated topics. Plus, it's created this incredible record of my learning and personal growth.

I found Ryan Holiday's take on the Commonplace Book particularly insightful for efficiently using a PKM. While Ryan advocates for analog notes, the same principles can be applied to digital notes. While physical writing has its merits, digital notes provide superior search capabilities and flexible organization. Ryan himself states how impracticability of a physical PKMS - "Because mine is a physical box with literally thousands of cards, I don’t carry the whole thing with me".

And that's the beauty of a digital PKMS – it all fits on my phone!

Additionally, now with AI code generation creating custom PKMS plugins are more accessible than ever before. No longer do we need to install a random developer's plugin that sends our private thoughts who knows where. We can vibe-code our own or use open-source tooling. I used to use an Obsidian plugin to pull uncompleted tasks from todo checklists that were scattered around various notes, but after 10 minutes of prompting Anthropic's Claude, I had my own private equivalent tool.

A look inside

Using essentially a wrapper around a database made creating my PKMS easy. This wrapper does the heavy lifting, keeping things simple and secure. I went with a platform called Directus, which can be used as a PKMS amongst other things such as a CMS. Directus has authentication and security layers already included, so I was able to start using it in less than a day.

For readers who have worked with SQL databases or docker containers - I wrote a step-by-step guide on building my own PKMS with Directus.

A final note

I’ve always believed that, much like a garden, how we manage our own knowledge system needs cultivation and continuity to flourish. My own knowledge garden was cultivated from a deep dive into the subject of note systems, driven by a search for a solution to those nagging frustrations I think many of us feel with commercialized systems.

A knowledge garden isn't always straightforward, and I'll be the first to admit I’ve been guilty of spending considerably more time fiddling with a system, rather than actually using it. It's easy to fall into analysis paralysis with these system as well as become plagued with paranoia worrying about privacy. So for any system to stick, it has to be genuinely secure and simple.

There isn't a one-size-fits-all approach to a sizeable amount of things in life, knowledge management being one of these. Inquisitive individuals are likely to be rewarded for exploring different avenues, even unconventional ones, as they lead to a system tailored to one's personal way of thinking and working.

The path I eventually took, building my own, put an end to exhausting cyclical migrations. It helped me reclaim control over my privacy, and significantly cut down on recurring costs. Especially because it's ran from my own VPS. Commercial apps certainly offer out-of-the-box convenience, but by stepping back and focusing on what mattered to me, I found a way that worked profoundly better. Here's how I built my PKMS step-by-step.

After dogfooding my PKMS for over a year now, I'm capturing and connecting ideas more efficiently than I ever managed with commercial apps. While I sometimes miss the community around commercial apps, I'm hopeful I'll find a niche community of like minded individuals one day.

If my PKMS journey has resonated with you, or if you're on a similar path, I'd genuinely love to connect. You can find me at any of the socials in the footer.

The simplicity, flexibility & centralization of these systems speak volumes to me for saving my todo lists, notes about things I've read and the like. However note-taking apps such as Obsidian, which are closed source, felt finite and restricted.

Concerns around vendor lock-in are why I created my own notes system. I've been using it as my core productivity framework for the past year and it's become indispensable to my work. Having the freedom to create custom tooling and integrations has impacted my day-to-day, and I’m excited to keep exploring its potential.

Features for the note taking app detailed in this guide include:

• Self-hosted
• Private
• Built to last
• Low maintenance
• Access in one place & from any device (Obsidian charges for this feature)
• Versioning
• Zero vendor lock-in
• Extendable (eg. passing text-embedded notes to AI)

The digital hoarder

Saving your notes digitally has many benefits. One being improving searchability across a corpus of notes. Now with advances in AI - notes saved digitally have the ability for smart categorization, summarization & more.

Saving anything online comes with a few points of concern. Privacy and provider lock-in are the most frequently discussed. Not to mention ensuring your notes don't get yeeted into the abyss when your computer stops working.

While these concerns are worrisome, they aren't enough for the mass population to go analog and use physical journals like it was the 1970s.

Easier than you think

I've used Directus to go from 0 to 60mph to build a secure, flexible and easy to use notes web application. Don't worry we're not locked into using Directus. The database and data inside it are yours to port how ever you wish in the future.

Directus is a data platform built by Benjamin Haynes. It provides a real-time API and intuitive no-code interface for managing SQL database content. No paid affiliation, just a fan of the platform. It's source code is available to read through and is free to use (unless you are using it to make millions in revenue through the platform).

Let's dive into the setup.

1. Hosting your app

You'll need a place to host your dockerized Directus app to start. For those without their own VPS - I've written an article here on the cheapest way to host your own web apps.

Optionally, there's Fly.io or Railway which make hosting web apps easy. Though there are trade-offs to hosting on these sort of providers as they are likely not long lived. At the time of writing neither Fly.io and Railway have Docker Compose support so your set up will look slightly different than this guide outlines.

2. Hosting your database

We'll need a place to save our notes data into a database. Directus allows you to use a variety of SQL databases such as PostgreSQL and SQLite. In this guide we'll use PostgreSQL.

I initially started with a $15/mo managed postgres instance from Digital Ocean for a few months. Then moved to a containerized postgres instance I self-manage and backup.

Use the option which makes the most sense for you.

3. Update app config

Use the docker compose yaml code below to start from. Its config is using a containerized postgres instance that our notes app can use to store it's data. It also uses a containerized redis instance for caching.

For those with managed databases - comment out the database service & postgres_data volume then update the database config in the directus service to use your managed database credentials.

    version: "3"
    services:
        database:
            image: postgres:15
            container_name: my_postgres
            restart: unless-stopped
            environment:
            POSTGRES_USER: "directus_user"
            # Add secure password here
            POSTGRES_PASSWORD: ""
            POSTGRES_DB: "directus_db"
            volumes:
            - postgres_data:/var/lib/postgresql/data
            ports:
            - "5432:5432"
            networks:
            - directus

        cache:
            container_name: cache
            image: redis:6
            networks:
            - directus
            restart: always

        directus:
            container_name: directus
            image: directus/directus:10.8.3
            ports:
            - 8000:8000
            volumes:
            - ./uploads:/directus/uploads
            networks:
            - directus
            depends_on:
            - cache
            restart: always
            environment:
            # Add key and secure secret
            KEY: ""
            SECRET: ""

            # Add your email & secure default password here
            ADMIN_EMAIL: ""
            ADMIN_PASSWORD: ""

            # Add your email to EMAIL_FROM and EMAIL_SMTP_USER
            EMAIL_FROM: ""
            EMAIL_TRANSPORT: "smtp"
            EMAIL_SMTP_HOST: "smtp-relay.brevo.com"
            EMAIL_SMTP_PORT: "587"
            EMAIL_SMTP_USER: ""
            # Add secure password here
            EMAIL_SMTP_PASSWORD: ""


            # Uncomment below - for S3 file store
            # STORAGE_LOCATIONS: ""
            # STORAGE_DIGITALOCEAN_DRIVER: "s3"
            # STORAGE_DIGITALOCEAN_ROOT: "/"
            # STORAGE_DIGITALOCEAN_BUCKET: ""
            # STORAGE_DIGITALOCEAN_REGION: ""
            # STORAGE_DIGITALOCEAN_ENDPOINT: ""
            # STORAGE_DIGITALOCEAN_KEY: ""
            # STORAGE_DIGITALOCEAN_SECRET: ""

            # This uses postgres for database
            DB_CLIENT: "pg"
            DB_HOST: "database"
            DB_PORT: "5432"
            DB_DATABASE: ""
            DB_USER: ""
            DB_PASSWORD: ""

            CACHE_ENABLED: "true"
            CACHE_STORE: "redis"
            REDIS: "redis://cache:6379"

    networks:
        directus:

    volumes:
        postgres_data:

4. Run your app

Its as easy as saving the docker-compose file to your hosting server and running docker compose up.

Once that's running you can login to your app with the ADMIN_ credentials from your docker-compose configuration.

5. Saving your notes

I've created a simple Notes table as shown. I went with Markdown language to save my notes in as it's widely used, has HTML support and multi-decade history.

If versioning is important to you be sure to select enable versioning on your Notes data model.

Ending notes

With this self-hosted notes system, you're now free from walled gardens and have full control over your notes.

As for my own notes app, I'm currently working on a chrome extension to help bookmark and add notes to various articles I'm reading. At some point, I'd like to build my own flavor of Obsidian's Dataview to work with a markdown file using Markdown frontmatter.

If you do create your own notes system using these steps, I’d love to see what you’ve built - especially any plugins or extensions. You can find me at any of the socials in the footer.

This guide supports Ubuntu as the operating system only.

1.Spinning up a VPS

Pick VPS that fits your system requirements. I generally go with the smallest option since providers allow you to scale up. I've used Digital Ocean and Hetzner myself and would recommend either providers.

This guide uses SSH to work with the server. If you don't know how to SSH into the server here's a guide on that.

I recommend running a benchmark test when you set up a new VPS. You’ll sometimes get older machines or machines that have been heavily used when using VPSs. If the performance is degraded, delete your VPS from that provider and try again. I usually use Yet Another Bench Script for this.

2.Set up docker

Digital ocean has option to spin up Droplet with Docker image pre-loaded. Your milage may vary with other providers.

For VPS without Docker installed follow this guide

3.Nginx setup

Once SSH'd into the server run the below in the command line to update the system's packages and install nginx so we can route requests to the server.

sudo apt-get update
sudo apt-get install nginx 
# press enter to restart default services

Run the following to clear out nginx defaults

cd /etc/nginx/sites-available/ && rm default
cd /etc/nginx/sites-enabled/ && rm default

4.Setting up a firewall

Providers often have firewall GUIs to simplify setting up a firewall. Here's an article on what you can expect setting up a firewall from a GUI to look like this one is from Hetzner's cloud offering.

If you prefer to set it up yourself I recommend this article here.

5.Set up your site's maintenance page

This is a page we'll use if your site offline. It's a good example that you can also use static files or static apps on your server. You can copy this starter one to test from here.

Run the command below, it will open up an editor. Once the editor is up paste in your html, save and exit the editor.

cd && sudo mkdir /var/www/mysite.com
cd /var/www/mysite.com
sudo nano maintenance.html

6.Routing network traffic

Let's pretend you have a docker container running on port 8008. Okay if you have an app running on that port or not since we have a maintaince page. We will now route traffic to your app running on the VPS from port 8008.

Create an nginx config file for the site running:

cd && sudo nano /etc/nginx/sites-available/mysite.com

Now paste the following in the editor, save and then exit the editor:

	server {
	    # Port 80 is used for incoming http requests

	    # The URL we want this server config to apply to
	    server_name  mysite.com;

	    location / {
	      proxy_pass http://127.0.0.1:8008;
	    }
	        location @static {
	        # This is the maintaince page saved as /var/www/mysite.com/maintenance.html on the server
	        root /var/www/mysite.com;
	        try_files /maintenance.html =404;
	    }

	}
	server {
	    listen 80;
	    listen [::]:80;
	    server_name  mysite.com;
	}

7.Enabling the site

Run the following to create a symbolic link from your nginx config to enable the config.

cd && sudo ln -s /etc/nginx/sites-available/mysite.com /etc/nginx/sites-enabled/mysite.com

Now let's test we have set up the nginx config correctly via sudo nginx -t. If that looks good we can now restart the nginx server via sudo systemctl restart nginx.

Finally, go to your site via http (eg. http://mysite.com) in a browser - you should see your running app if its running or the maintenance page if not.

8.Setting up HTTPS

We will use Certbot for this. Important to note the certification expires after 90 days so good to have a cron job set up to renew this automatically. This article doesn't cover automating the renewal but there's a plethora out there that do.

   snap install certbot --classic
   sudo apt-get install -y python3-certbot-nginx
   # ensure you have Nginx full added to firewall
   sudo certbot --nginx -d mysite.com
   # enter email & choose not to share -> then you're good to test a cert renewal 
   sudo certbot renew --dry-run

Wrapping up

I hope you found this guide easy to follow as it took me a few days to understand and get right myself. As a follow up, I recommend adding extra security set up to ensure your VPS is secure such as fail2ban.

Please reach out if you think I missed anything! You can find me at any of the socials in site's footer.

In mathematics, computer science and physics, a deterministic system is a system in which no randomness is involved in the development of future states of the system. - The Internet Encyclopedia of Science

LLMs are capable of producing a consistent output when the input and model version all match. However, change a single word from that same prompt and the LLM can output something drastically different. Because of this, most LLMs are limited to features where a degree of randomness is acceptable, such as chat bots. For this same reason, integrating LLMs has historically been impractical for mission-critical operations.

The good news is that with Structured Output LLMs become exceedingly reliable and less random. Structured Output leverages strongly-typed systems to put guardrails around what an LLM can output. Furthermore, when its used in combination with performance testing - companies can deliver stable AI-integrated products.

The reader can expect to walk away with an understanding of how to build dependable AI-integrations. The same concepts from this article were used to build HN Who's Hiring Database. Check out the original post on how the HN Who's Hiring Database website was was built here. It's a self-maintained open-source project - used by 100s of monthly active users to query Hacker News job postings. Hacker News is a social news website and online forum for computer science and entrepreneurship, run by the venture capital firm Y Combinator.

Structured outputs

At its core, Structured Output is a constraint mechanism for LLMs. It forces the model to adhere to a predefined schema, ensuring that the model's output follows a strict structure. For this article, I'll be using OpenAI's API in Python using Pydantic. The same can be done in Typescript with Zod and various other languages using JSON mode with JSON based schemas.

Clear advantages of structured output:

• Type safety: No more runtime type errors from unexpected LLM outputs.
• Efficient prompting: No more verbose instructions in your prompts about output format.
• Less code overhead: No more complex regex rules to extract data.

The example project for this article uses a cron job to scrape website data, gives large text blocks to an LLM to extract and clean data from, save and share the data online. We'll discuss only a portion of the project which uses LLM integration - to read more about those other parts check out the blog post on how HN Who's Hiring Database was built.

Let's have a look the data quality bump we can expect between LLM prompting with Structured Output and without. Not only does the quality go up, but the text based prompt greatly decreased. The original prompt's code with few-shot prompting was a total 134 lines of code. With Structured Output 134 lines were reduced to 7.

This reduction in prompting overhead is because the model uses the Structured Output schema as part of it's context. When schemas are named and related clearly, the model understands what it needs to do through a code based semantic meaning. You no longer need to explain linguistically to the model how a system's data model and and their relationships work.

That's not to say that Structured Output is always enough for the model to understand a system's data modeling. In these cases, we can add descriptions to the schema. JSON mode provides a way to add descriptions directly for this. However, Pydantic's built in field description is not supported. To work around this OpenAI's appears to read the Pydantic schema's docstring so I find it's more performant to add schema context there rather than putting in your prompt. Typescript using JSDoc in the Zod model may work similarly.

Let's see it in action

# Pydantic structures file
from typing import Optional, List
from enum import Enum

class RemoteEnum(str, Enum):
    YES = "yes"
    NO = "no"
    HYBRID = "hybrid"
    UNKNOWN = "unknown"


class CurrencyEnum(str, Enum):
    USD = "USD"
    EUR = "EUR"
    GBP = "GBP"
    CAD = "CAD"
    CNY = "CNY"
    AUD = "AUD"
    CHF = "CHF"
    MXN = "MXN"
    UNKNOWN = "unknown"


class EmploymentType(str, Enum):
    FULL_TIME = "full-time"
    PART_TIME = "part-time"
    CONTRACT = "contract"
    INTERN = "intern"
    UNKNOWN = "unknown"


class JobPosting(BaseModel):
    comment_id: int
    company_name: str
    job_title: str
    employment_type: EmploymentType
    currency: CurrencyEnum
    remote: RemoteEnum
    salary: Optional[str]
    remote_rules: Optional[str]
    how_to_apply: Optional[str]
    company_city: Optional[str]
    company_country: Optional[str]
    languages_and_frameworks: Optional[List[str]]


class JobPostings(BaseModel):
    postings: List[JobPosting]

from openai import OpenAI

from structures import JobPostings

client = OpenAI()


completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "You are a data extraction expert that extracts job posting data based on a list of job postings you are provided."},
        {"role": "user", "content": "..."}
    ],
    response_format=JobPostings,
)
job_postings = completion.choices[0].message.parsed

Now instead of JSON like typical REST requests, the response will a JobPostings Pydantic model. This not only adds a layer of validation, but can also be used to format or clean data further.

Enabling reliable testing

Before Structured Output, methods of extracting meaningful data from an LLM's wall of text, required excessive lines of regex.

Now that we have used Structured Output to enforce the shape of our data as opposed to a wall of tech - we can utilize test Evals to evaluate an LLM's performance. Evals are incredibly useful to validate new ideas, improve or lock-in AI integrated feature performance.

Before Structured Output, methods of extracting meaningful data from an LLM's wall of text, required excessive lines of regex. Testing an LLM's performance also prone to this same issue. Validating an LLM's wordy output required regex or similarity matching of text. Sometimes even more costly measures, like OpenAI's evals which uses another LLM to score our LLM’s output. Which always sounded like a snake eating its own tail from my perspective.

Well this is all much, much easier now that we can limit the LLM to a simple Yes / No only output if we like.

Turning your testing from looking like this...

...into this...

LLM output	Truth	Eval
apple	apple	✅
orange	banana	❌

Code below is how this could work in practice. You’ll notice there’s 2 similarity matching checks. This is where the output type would be considered an unstructured data type. When the output's longer than a sentence, we can generally consider as it as unstructured data.

# conftest.py - fixtures we’ll need for our tests
import pytest

from structures import JobPostings

JOB_POSTING_OUTPUT_FILE = 'job_posting_ouput.json'


@pytest.fixture
def jobs_struct() -> JobPostings:
    json_data = None
    script_dir = os.path.dirname(os.path.abspath(__file__))
    file_path = os.path.join(script_dir, JOB_POSTING_OUTPUT_FILE)

    with open(file_path, 'r') as f:
        json_data = json.load(f)
    if json_data is None:
        raise ValueError("No data found in the job_posting_ouput.json")
    return JobPostings(**json_data)
 

from fuzzywuzzy import fuzz

from structures import JobPosting


def test_response_passes_model_validation(jobs_struct):
    assert isinstance(assets_struct, JobPostings), "JobPosting structure is invalid"


def test_job_post_sample(jobs_struct):
    SOFT_ASSERT_THRESHOLD = 80  # 80% similarity threshold

    expected_job = JobPosting(
        comment_id=41133198,
        company_name="FusionAuth",
        job_title="Senior Java Software Engineer",
        employment_type=EmploymentType.FULL_TIME,
        currency=CurrencyEnum.USD,
        remote=RemoteEnum.HYBRID,
        salary="$120k-$180k",
        remote_rules="ONSITE or REMOTE in USA (location reqs listed on the job desc)",
        how_to_apply="Learn more, including about benefits and salaries, and apply here: https://fusionauth.io/jobs/",
        company_city="Denver",
        company_country="USA",
        languages_and_frameworks=["Java", "MySQL", "Docker"]
    )
    job = jobs_struct[5]

    # Strict assertions
    assert job.comment_id == expected_job.comment_id, "Comment ID does not match"
    assert job.company_name == expected_job.company_name, "Company name does not match"
    assert job.job_title == expected_job.job_title, "Job title does not match"
    assert job.employment_type == expected_job.employment_type, "Employment type does not match"
    assert job.currency == expected_job.currency, "Currency does not match"
    assert job.remote == expected_job.remote, "Remote does not match"
    assert job.salary == expected_job.salary, "Salary does not match"
    assert job.company_city == expected_job.company_city, "Company city does not match"
    assert job.company_country == expected_job.company_country, "Company country does not match"

    # Soft assertions
    similarity = fuzz.ratio(job.remote_rules, expected_job.remote_rules)
    assert similarity >= SOFT_ASSERT_THRESHOLD, f"Similarity {similarity}% is below {SOFT_ASSERT_THRESHOLD}% for line:\nActual: {job.remote_rules}\nExpected: {expected_job.remote_rules}"

    similarity = fuzz.ratio(job.how_to_apply, expected_job.how_to_apply)
    assert similarity >= SOFT_ASSERT_THRESHOLD, f"Similarity {similarity}% is below {SOFT_ASSERT_THRESHOLD}% for line:\nActual: {job.how_to_apply}\nExpected: {expected_job.how_to_apply}"

Long-term performance monitoring

When integrating LLMs into production systems, it's crucial to track performance over time. This generally is done using multiple test sets. The test sets can be created from snapshots of human verified successful cases. The larger test sets will improve reliability as will extending the test set as time goes on, and adding edge cases to the test set as they arise.

Here's why this matters and how structured output plays a key role:

• Detecting drift: LLM performance can drift over time due to changes in the underlying model, updates to prompt engineering, or shifts in user behavior.
• Seasonal variations: Some applications may have seasonal patterns. By maintaining separate test sets for different time periods (e.g., monthly), you can identify and account for these variations.
• Impact of updates: When you make changes to your prompts or fine-tuning, you need a way to measure the impact across different scenarios.
• Comprehensive evaluation: Different test sets can cover various edge cases, ensuring your LLM integration remains robust across a wide range of inputs.

To demonstrate LLM evals put into practice, let's imagine that we have shipped an AI-integration that takes a customer email, when the customer's email mentions an issue with the product the AI-integration will create a support ticket for our team to look into. We built-in test evals and are actively tracking the accuracy of our LLM integration over the course of a year. Each week, we run our test suite and record the overall accuracy.

We notice a dramatic dip in week 52. We looked into our source code from that day and can see an engineer upgraded the model's versioning which lead a 20% reduction in performance. Without this tracking in place, such a regression can not only go unnoticed but also looses the ability to pin point causality.

Test evals allows companies to make data-driven decisions around prompt engineering, model providers or versions, and fine-tuning strategies. All while ensuring that LLM integration remains robust and effective over the long term.

Wrapping things up

We've discussed how using Structured Outputs with LLMs adds stability to products using AI integration. Structured Output when used in combination with the right test evals, companies can start to integrate LLMs into production systems with confidence.

It's still not advisable to integrate critical systems with LLMs since, they cannot guarantee fully accurate data. However, LLMs can deliver greater consistency and efficiency than their human counter parts which otherwise become bored or tired. For projects that demand considerable manual efforts, LLM integrations are worth considering.

Feel free to reach out via any of the social links in the footer if this topic caught your interest. For those that enjoyed this article, you may also like to read how HN Who's Hiring Database was built here.

Repo link for those that wish to dive straight into the code.

As software engineers, we often find ourselves reinventing the wheel with each new project, particularly when it comes to UI components. This repetitive setup can be time-consuming and often leads to inconsistencies across projects. It's a challenge I've faced repeatedly, both in my personal projects and in professional settings.

To address this, I embarked on creating a custom component library. This approach has proven beneficial not only for solo developers like myself, but also for companies of various sizes.

Here's a handful of benefits off the top of my head:

1. Picking a base

Selecting the right component library was crucial. I evaluated several options based on key factors important for my needs.

Libraries considered:

• Open-source: MUI's Material-UI, Mantine, Ant Design, FAST (Microsoft), Base Web (UBER), Polaris (Shopify), Tamagui
• Open-source & Tailwind based: DaisyUI, Radix UI
• Paid licensing: Flowbite, Tamagui's Bento

To guide my decision, I prioritized these factors:

After careful consideration, I chose MUI's Material-UI as my base library. It stood out in several key areas, particularly in its extensive set of prebuilt components, robust documentation, and built-in design system based on Google's Material Design. As a bonus there's a Figma base MUI provides for free here that can be used prototype your projects ahead of build.

However, it's important to note that each base has its strengths, and what worked best for my needs might not be the optimal choice for everyone. I encourage you to evaluate these options based on your own criteria and project needs.

2. Making it my own

Next up, I tweaked the design system. Went for a kind of robotic text vibe, cooked up a color scheme that didn't make my eyes bleed, and made sure the dark and light modes looked related.

Here are examples of my customized themes:

It's worth noting that while these customizations worked well for my projects, your needs may differ. The beauty of using a flexible design system like is that it allows for extensive customization generally from single config file.

Worth checking if your base component library has a theme generator online to speed this step up.

3. Compile time

Selecting the right Javascript module bundler was a crucial decision in the development of my component library. The most common JavaScript compilers I considered were Rollup, Webpack and Parcel.

If you've ever had to maintain an ejected Create React App you know how complex Webpack can get. Given my Webpack battle scars I wanted to explore my options before committing to a long term relationship with a compiler. This exploration led me to Rollup, which stood out for several reasons. The popularity trend against it's peers included.

4. Package optimizations

Initial package size was an eye-watering 32.7 MB unpacked. I reduced this to 4.8 MB by removing sourcemaps and including only one javascript module system (I went with CommonJS).

Note that @mui/material alone is 10.8 MB unpacked.

It's crucial to consider your package size and optimize for the smallest size that meets your needs. While 4.8 MB is a significant improvement, there's certainly room for further optimization. The key is balancing functionality with performance.

5. Component playground

To streamline development and quickly iterate on components, I set up a local testing environment:

• Created a playground from a boilerplate React app
• Pointed the package.json file to the locally built component library: "@mb3r/component-library": "file:.."
• Added a Makefile with build and test-build commands for quick use

This step is particularly useful for verifying component behavior before publishing a package.

6. Publishing the package

For distributing the component library, I chose to publish it on NPM, the most widely used registry for open-source Node.js packages. Here's how I approached it:

• Created a company-type namespace: @mb3r
• Named the component library: component-library
• Resulting package name: @mb3r/component-library

This approach allows for hosting multiple related packages under a single namespace, providing a clean and organized structure in users' package.json files.

While NPM is ideal for open-source projects, the choice of registry depends on your specific requirements around privacy, integration with existing tools, and deployment processes. You may consider other package repository solutions such as AWS CodeArtifact, GitHub Packages or Verdaccio over NPM.

I update the versioning manually by running npm version 1.0.0 then publish to NPM by running npm publish.

In future releases I plan to automate package versioning with Commitizen. Commitizen essentially is a wrapper around your repo's Git history and creates semantic versioning based on Git commits along with a host of other useful features like automating your changelog.

Final thoughts

The ultimate goal is to find the right balance between consistency, efficiency, and flexibility in your workflow for a component library.

Personally creating a custom component library has significantly streamlined my development process. For example, I was able to build a web-clipper chrome extension for my PKMS / notes system in under a day. Here's what it looks like as of late clipping a web quote to send to my PKMS:

There's several similar solutions like this already, but let's build our own anyway.

Github repo here

Collecting Hacker News hiring tread data

I first performed a Google search to find the latest HackerNews hiring threads for a specific month and year, and then extracted the top link to the target thread. Next, I used BeautifulSoup to scrape data from this thread, focusing on comments structured like job listings. Finally, I compiled these comments into a dictionary and continued scraping across multiple pages to gather all relevant hiring data.

I realize now there is an id to these sorts of threads that makes the google search reduntant. Oh well

Token Size Estimation and GPT Prompt Formulation

Estimating the token size of each data point was done using OpenAI's tiktoken. This estimation was crucial for batching the rows into OpenAI requests, optimizing both speed and expenditure.

Then came the task of formulating the GPT prompt, which included:

• Data Extraction: Transforming the raw scraped rows into structured columns of interest, like salary.
• Few-Shot Prompting: A technique to enable in-context learning where we provide demonstrations in the prompt to steer the model to better performance.
• JSON Mode: Enforcing output to conform to valid JSON
• Error Handling

Processing, Cleaning

Once the GPT had processed the data, it was time to rejoin the batched data into a cohesive set. This step was followed by a thorough cleaning process, ensuring the data was accurate and useful.

Automation

To keep this process up-to-date without monthly manual intervention, I set up a monthly GitHub Action cron job. This job scrapes the latest data, processes it, and commits the updated table data to a repository, with a link provided in the README. Which finally opens an easy to manage pull request assigned to me.

Putting it all together

graph TB
    AAA[scrape hackernews] --> AA{Token counter} --> A
    A[scraped.csv] -. chunk data .-> B(batch-0.csv)
    A -. chunk data .-> C(batch-1.csv)
    A -. chunk data .-> D(batch-2.csv)

    B -..-> Q[Processing Queue]
    C -..-> Q
    D -..-> Q

    Q --> E{LLM System
 data extraction}
    E -. batch n .-> E

    E --> | Rejoin processed batches | F[summary.csv]
    F --> G{Clean data}
    G --> H[clean.csv]

Imperfect results

This project, while not perfect and certainly in need of improvements, showcases the practical application of web scraping, AI, and automation in extracting and analyzing data from unstructured sources. It was an engaging journey to automate this process, highlighting how these technologies can be effectively utilized for insightful data analysis. Looking ahead, I aim to refine the system and compare parsed data over time to reveal emerging trends.