bit-habit · platform infrastructure
One small server hosts every bit-habit.com website. This page explains how,
in five stages that get harder in order — each one starts by telling you
what you will be able to say when you finish it, and ends with a few questions to check
yourself. Stage 1 assumes you know nothing at all. Stop at whatever stage
answers your question; nothing later is a prerequisite for using the sites.
NO PRIOR KNOWLEDGE NEEDED
What this platform is, and the complete path a request takes from your browser to the page you see.
After this you can say which sites are up right now, and which one is not.
Each site is checked from the outside every 60 seconds. Full status page → status.bit-habit.com.
Loading live status…
After this you can say what this entire platform is, in one sentence.
One small server runs every bit-habit.com website. You type an address,
it finds the right app, and sends the page back.
bit-habit.comstartpage.bit-habit.comblog.bit-habit.com
wiki.bit-habit.complane.bit-habit.com… ~20 sites
Want to know how it picks "the right app"? Keep reading.
startpage.bit-habit.comAfter this you can say every step between typing an address and seeing a page.
Follow one request from your browser to the page. Six steps.
*.bit-habit.com name points to one server IP: 158.180.71.122.http (port 80), Traefik
sends you to secure https (port 443). It's a permanent redirect (status 308).startpage.bit-habit.com
and forwards the request to the matching app.Checkpoint · end of Stage 1
Answer these before moving on. Open each one to check yourself — if any answer is a surprise, that part is worth a re-read.
*.bit-habit.com address points at how many servers?One. A single Oracle Cloud box runs all ~20 sites, and wildcard DNS sends every name to its one IP.
Traefik. It holds ports 80 and 443 on the host and is the only thing listening on them.
http:// instead of https://. What happens?Traefik answers with a permanent redirect (status 308) to the https address, then serves the page over the encrypted connection.
THE VOCABULARY
The name of every moving part — Traefik, Ingress, Service, Pod, cert-manager, k3s — and what each one is for.
After this you can say the name of each moving part, and what each one does.
The server runs k3s, a small version of Kubernetes. Kubernetes is just a manager that starts apps, restarts them if they crash, and routes traffic to them. Here are its parts, in plain terms.
| Part | Think of it as | What it does |
|---|---|---|
| Traefik | the front door | receives every request on ports 80 / 443, adds HTTPS, and sends it to the right app |
| Ingress | the routing list | the table that says "this address → that app" |
| Service | a stable inside address | a fixed name for an app, even after it restarts |
| Pod | the running app | the container doing the actual work |
| cert-manager | the certificate robot | gets and renews the HTTPS certificate automatically |
| k3s | the manager | a small Kubernetes that starts, restarts, and watches every app |
And here is how those parts connect for a single app — a Deployment keeps Pods running, a Service gives them one address, an Ingress maps the hostname to that Service, and Traefik (with a cert from cert-manager) is the door:
About 20 apps share this one server. A few you can visit:
| Address | What it is |
|---|---|
bit-habit.com | the landing page (static site) |
startpage.bit-habit.com | a personal start dashboard |
blog.bit-habit.com | the blog (Ghost) |
wiki.bit-habit.com | the knowledge base (Wiki.js) |
habit.bit-habit.com | the habit tracker (web app + API) |
plane.bit-habit.com | project management (self-hosted) |
fider.bit-habit.com | feature-request voting |
quali-fit.bit-habit.com | an explainable staffing recommender (sign-in required) |
physical-spark.bit-habit.com | a Physical AI school — static site + an auth API |
llm-app-lab.bit-habit.com | an LLM course site |
headlamp.bit-habit.com | the live cluster dashboard (sign-in required) |
*.bit-habit.com. Every subdomain shares it,
so a new site is HTTPS with zero extra setup.
Checkpoint · end of Stage 2
Answer these before moving on. Open each one to check yourself — if any answer is a surprise, that part is worth a re-read.
The Service. That is the entire point of a Service: it is a stable name in front of Pods that come and go.
wiki.bit-habit.com should reach wikijs-svc?The Ingress — it is the routing rule. Traefik is the program that reads it and acts on it. Rule and reader are two different things.
cert-manager, automatically. One wildcard certificate for *.bit-habit.com covers every subdomain, so a new site needs no certificate work at all.
bit-habit.com page reaches your browser, Stages 1 and 2 are the whole
answer — you now know every part in the path and what each one does. Everything
below is about running and changing the platform, which is a different job.
Come back to it when you need it.
OPERATIONS
How your code actually gets onto the server, and what it looks like — and how you diagnose it — when something breaks.
After this you can say how code travels from your laptop to a live site — and why Kubernetes plays no part in noticing your push.
Everything above follows a request inward: browser → Traefik → app. This section follows code outward: laptop → GitHub → server → live. It is the half of the story people most often guess wrong.
Following one git push
main.authorized_keys with a forced command: no matter what command
arrives, only one script runs. Even a leaked key can do nothing else on the box.git reset --hard origin/main
— an exact mirror, so the server never drifts).
Most static sites deploy by baking the HTML into a container image, pushing it to a
registry, and rolling out new Pods. Here, nginx reads the git checkout on the server's
disk directly. So the moment git pull finishes, the new page is being
served — the same Pod, untouched, uptime intact. A build step buys nothing when the
artifact is the source.
Two layers, deployed differently
It matters which of these you changed — they do not travel the same road.
| What changed | Lives in | How it reaches the cluster |
|---|---|---|
| App content pages, code |
each app's own repo | automatic — GitHub Actions on every push (see above) |
| Infrastructure Ingress, certs, Deployments |
this repo, bit-habit-infra |
manual — kubectl apply by hand. It changes rarely, and a bad
Ingress can take every site down at once, so a human stays in the loop. |
Three repos ship themselves this way today:
| Repo | Site | What its deploy does |
|---|---|---|
portfolio-bithabit | bit-habit.com | git pull — that's the whole deploy |
llm-app-lab | llm-app-lab.bit-habit.com | git pull |
physical-spark |
physical-spark.bit-habit.com |
pull; rebuild the auth image only if auth/ changed; then
kubectl apply its own manifests
(ops/deploy.sh) |
kubectl apply is idempotent — re-applying an unchanged file costs one
unchanged line — so physical-spark applies its manifests on
every deploy, not only when they change. Anything someone poked by hand in the
cluster is quietly reset to what git says. The file, not the running cluster, is the
source of truth. That is the core idea behind GitOps tools like ArgoCD and Flux,
reached here with one extra line and nothing new installed.
After this you can say how to diagnose an outage one layer at a time, and why a 200 OK can be a lie.
A real outage on this cluster, written up with START — Situation, Task,
Action, Result, Takeaway. The site answered 200 OK the entire time, which
is exactly why nobody noticed for months. If you are new to infrastructure, this is
the most useful section on the page: everything above describes how things work, and this
describes what it looks like when they don't.
what was happening
Opening habit.bit-habit.com showed a personal portfolio page instead of the
BitHabit habit-tracking app. Not an error page — the wrong site, served
perfectly, with a valid certificate and a 200 OK status.
That detail matters more than it sounds. 200 OK is the code a server sends
when everything went fine. Uptime monitors, this page's own live status grid, and the
status page at status.bit-habit.com all ask "did it answer 200?" — and the
answer was yes, continuously, while the site was completely wrong.
The outage was invisible to every automated check.
what had to be done
Find out why, and restore the Flutter app — without breaking the other ~19 sites
that share this one server. That constraint shaped everything: the routing table
for every bit-habit.com site lived in a single file, so any careless edit
could take all of them down at once.
how it was actually tracked down
The method here is worth more than the answer: work inward one layer at a time, and after each step ask "is the problem above this layer or below it?"
| Step | What was checked | What it ruled out |
|---|---|---|
| 1 | Fetch the page from outside. Status was 200, but the page
<title> was the portfolio's, and main.dart.js —
the file that is the Flutter app — returned 404. |
Not a crash, not TLS, not DNS. Something is answering; it just isn't the app. |
| 2 | Fetch the neighbouring hostnames. bit-habit.com,
www.bit-habit.com and habit.bit-habit.com all returned
the same portfolio HTML. |
Not specific to the habit app. Three names are landing in one place — this is a routing problem. |
| 3 | Read the cluster's routing table (kubectl get ingress). The rule
said habit.bit-habit.com → static-web-svc. |
Found the wrong turn. static-web-svc is the portfolio, not the app. |
| 4 | Call the Flutter app's Service directly from inside the cluster, bypassing
the front door entirely. It returned <title>BitHabit and
main.dart.js → 200. |
The app was never broken. It had been running fine for 103 days with nothing routed to it. |
| 5 | Read the portfolio Pod's own nginx config. Its server_name listed
only bit-habit.com and www.bit-habit.com — not
habit.bit-habit.com. |
Explained the 200. See below: this is the second failure. |
Two independent failures had stacked, and it took both to produce this symptom:
One nginx can host many websites. It decides which one you get by reading the
Host: header your browser sends and matching it against each site's
server_name. If nothing matches, nginx does not return an
error. It quietly hands the request to the first site it has — the
default server.
That behaviour is intentional: a server on the public internet is reached by random
hostnames and IP scans all day, and it needs some answer. But it means a
hostname you forgot to configure does not fail loudly — it silently gets served
somebody else's website, with a 200. Whenever you see "the wrong site, but a
successful response," check server_name first.
what was changed
Traffic was restored in one line — but the fix that mattered was the one made
afterwards. The bad rule was not just live in the cluster; it was written that
way in base/ingress.yaml in this repo. Fixing only the running
cluster would have been undone the next time anyone applied that file.
bithabit-web — its Deployment,
Service and nginx ConfigMap — existed only inside the running cluster and in
no repository at all. It is now declared in
apps/bithabit-web/.apps/bithabit-web/ingress.yaml
for the site, apps/bithabit-api/ingress.yaml for
/api/ — and the shared table dropped from 15 hostnames to 14.what to carry forward
200 OK is not proof that anything is rightA status code says the server answered. It says nothing about what it answered. Health checks that only look at the status code will miss an entire class of outage — this one hid for months.
Do instead: assert on content. Gatus, already running here, can require the response body to contain an expected string. One extra line per service turns "it responded" into "it responded with itself."
Changing a live cluster directly makes the symptom go away and leaves the cause in
place. The next apply of the original file silently reintroduces the bug.
The gap between "what is running" and "what is written down" is called
drift.
Do instead: treat the repo as the truth and the cluster as its output. A direct patch is fine to stop the bleeding — but it is not done until the file says the same thing.
The Flutter app ran happily for 103 days with zero traffic routed to it. Nothing was alerting, because from Kubernetes' point of view nothing was wrong: a Pod that nobody visits is not an error.
Do instead: when you create something in a cluster by hand, write it into the repo in the same sitting. "I'll commit it later" is how an app becomes invisible.
The bad rule sat in a file about every other site. Nobody reviewing the habit app would open it, and anybody editing it risked the other 14. That is a large blast radius — the amount that can break from one mistake.
Do instead: keep each app's routing next to the app. Two
Ingresses can share one hostname (that is how / and /api/
split here), so splitting costs nothing and shrinks the blast radius to one site.
The decisive move was calling the app's Service from inside the cluster. That single test split the problem cleanly in half: the app is fine, so the fault is in front of it. Without it, the obvious guess — "the Flutter build is broken" — would have cost hours.
Do instead: at each layer, find the cheapest test that tells you which side the fault is on. Bisect the path, don't guess at it.
# 1. from outside: is it answering, and with what?
curl -sS https://habit.bit-habit.com | grep -o '<title>[^<]*'
curl -sS -o /dev/null -w '%{http_code}\n' https://habit.bit-habit.com/main.dart.js
# 2. what does the routing table say for this host?
sudo k3s kubectl get ingress -A
sudo k3s kubectl get ingress main-ingress -o yaml
# 3. bypass the front door — ask the Service directly, from inside
sudo k3s kubectl run t --rm -i --restart=Never --image=curlimages/curl -- \
curl -sS http://bithabit-web-svc/ | grep -o '<title>[^<]*'
# 4. read the nginx config of whichever Pod answered
sudo k3s kubectl exec deploy/static-web -- cat /etc/nginx/conf.d/default.confstatic-web — see
what to grow.daily_seongsu/infra/legacy-migration/04-ingress.yaml). It
describes 13 hostnames instead of the current 14 and predates several services.
Two files claiming to define the same resource is the same class of bug this
incident was. It should be deleted or clearly marked dead.Naming what you chose not to fix is part of a retrospective. An unwritten known issue is indistinguishable from an unknown one.
Checkpoint · end of Stage 3
Answer these before moving on. Open each one to check yourself — if any answer is a surprise, that part is worth a re-read.
main. Does Kubernetes notice?No — and this trips up almost everyone. Kubernetes never watches GitHub. GitHub Actions SSHes into the server and pulls, and only then is anything new.
200 OK but shows the wrong content. Is it broken?Yes. 200 only means the server answered — not that it answered correctly. This is exactly the outage in the case study, and it hid for months because every monitor was satisfied.
kubectl patch and the site works again. Are you done?No. The file that produced the bad route is still wrong, so the next kubectl apply silently undoes your fix. Fix the file, not just the cluster.
CONFIGURATION
Which file defines what, which of them are in git, and which single file can take every site down at once.
After this you can say whether this platform is really “all in git”, and exactly what is not.
A fair question to ask of any platform: is the whole thing in one git repo? Almost. Every service that matters is in git — either this repo, or its own. What is left over is small, and worth naming precisely rather than worrying about.
base/ingress.yaml) — plus 17 app manifests live in
bit-habit-infra. Two apps keep their manifests in their own repo alongside their
code (physical-spark, quali-fit), which is a perfectly standard
layout, not a gap. Nothing that matters exists only on this disk.
plane and fider
are off-the-shelf apps — stock images (makeplane/plane,
getfider/fider) driven by a couple of hand-written YAML files that sit in a home
directory and are in no repo. They are also barely used. Losing those files would mean
rewriting a Deployment and a Service from a public project's docs — an hour, not a disaster.
Worth tidying eventually; not worth calling a risk.
The cluster, right now
The standard goal is one line: the running cluster equals what git says, with nothing that exists only on one machine. GitOps tools — ArgoCD or Flux — enforce it by continuously watching a repo and re-applying it, so manual cluster edits are reverted automatically.
For one node run by one person, ArgoCD is deliberately not installed (kept only as reference): it adds more moving parts than it removes, and re-applying manifests on every deploy already buys the self-healing that makes GitOps worth having. Measured against the standard, this platform is closer than it looks — the remaining gap is secrets, not services. See what to grow.
After this you can say which file to open for any setting on this server.
"Where is that configured?" is the question you actually ask at 2am. There are three places a setting can live here, and only one of them is this git repo. Here is all three, with real paths.
1 · In this git repo
On GitHub as
bookseal/bit-habit-infra,
checked out on the server at /home/ubuntu/workspace/bit-habit-infra.
| Path | What it defines |
|---|---|
base/ingress.yaml | the routing table — every subdomain → its Service. The one file that can break every site at once. |
base/cert-manager/cluster-issuer.yaml | how TLS certificates get issued (Let's Encrypt, DNS-01 via Route53) |
base/cert-manager/certificate.yaml | the wildcard *.bit-habit.com certificate request |
base/cert-manager/aws-secret.example.yaml | a redacted template. The real one (Route53 keys) is gitignored and applied by hand |
base/middlewares/ | Traefik middlewares, e.g. strip the /api prefix before forwarding |
apps/<name>/deployment.yaml | one folder per service — 17 of them (ghost, wikijs, gatus, sentinel, llm-app-lab, …) |
apps/argocd/application.yaml | GitOps config, written but never installed — see the next section |
k3s-bootstrap/traefik-config.yaml | the edge — Traefik on host ports 80/443, http→https redirect |
docs/index.html | this page. nginx serves this folder straight off the disk |
2 · In the app's own git repo
Two services keep their Kubernetes manifests next to their code, and deploy themselves. This is a normal, defensible layout — the service stays self-contained, at the cost of this repo no longer being the complete picture.
| Service | Repo · path |
|---|---|
physical-spark |
bookseal/physical-spark
→ k8s/, auth/k8s/, applied by its own
ops/deploy.sh. It even brings its own Ingress. |
quali-fit |
KIBA-Automation/quali-fit
→ k8s/ (namespace, PVC, deployment, service, middleware, ingress).
The image is tagged with the git commit it was built from, so a running Pod always names
its own source. |
3 · On the host — k3s's own directories
These are not in the repo, and mostly cannot be. They belong to k3s itself.
| Path | What it is |
|---|---|
/var/lib/rancher/k3s/server/manifests/ |
A magic folder: k3s applies every YAML you drop in here, automatically.
It holds Traefik, CoreDNS, local-storage — and traefik-config.yaml,
a copy of the repo's file. To change the edge you
sudo cp the file here and k3s redeploys Traefik on its own.
This is a second source of truth, and ArgoCD can never manage it. |
/etc/rancher/k3s/k3s.yaml | the admin kubeconfig — the credentials + address kubectl uses to talk to the cluster. Root-only. |
/etc/rancher/k3s/registries.yaml | where to pull container images from |
~/.kube/config | the same kubeconfig, copied for a normal user |
~/.ssh/authorized_keys | the deploy keys GitHub Actions uses, each pinned to one script (see the deploys section) |
And the leftovers
/home/ubuntu/plane/k8s/ and
/home/ubuntu/fider/fider.yaml deploy stock upstream images
(makeplane/plane, getfider/fider) — a Deployment, a Service, a
Postgres. Nothing bespoke, nothing precious, and both are barely used.
Recreating them from the upstream docs is an hour of work, so this is a tidiness item,
not a risk. The real thing still outside git is secrets — see
what to grow.
Checkpoint · end of Stage 4
Answer these before moving on. Open each one to check yourself — if any answer is a surprise, that part is worth a re-read.
1) this infra repo, 2) an individual app’s own repo, 3) on the host itself, in k3s’s own directories — the last of which no GitOps tool can ever manage.
base/ingress.yaml, the shared routing table. That is why it is applied by hand, and why moving rules out of it into per-app files shrinks the blast radius.
Nobody owns it, nothing reviews it, and it cannot be rebuilt if the server is lost. It can also be quietly orphaned — the case study’s Flutter app ran for 103 days with no traffic routed to it.
GOING FURTHER
What GitOps actually means, what adopting ArgoCD here would cost, and what this platform should fix first.
After this you can say what GitOps means, and what installing ArgoCD here would actually cost.
GitOps is one idea in one sentence: put the desired state in git, and let a program inside the cluster continuously make reality match it. That's it. ArgoCD is the most common program for doing that. Here is what changes, and what it would cost here.
The one thing that changes: push becomes pull
Today, everything pushes into the cluster from outside — GitHub Actions SSHes in, or
a human runs kubectl apply. The cluster is passive; it never looks at git.
GitOps inverts that: a controller runs inside the cluster and pulls from git forever.
The loop, precisely
| Word | What it means |
|---|---|
| Application | ArgoCD's core config object. It says: "watch this path in this repo, and keep this namespace matching it." One per group of manifests. Already written here: apps/argocd/application.yaml. |
| Sync | The act of applying git to the cluster. Can be automatic or a button you press. |
| Synced / OutOfSync | Does the cluster currently match git? OutOfSync means someone changed one side. |
| Healthy / Degraded | A separate question: are the Pods actually running? You can be Synced and Degraded — git applied fine, but the app is crashing. |
| Drift | The cluster no longer matches git. Usually because a human ran kubectl edit at 2am. |
| selfHeal | Automatically undo that 2am edit — re-apply git over it. This is the property people actually want from GitOps. |
| prune | Delete things from the cluster when they're deleted from git. Powerful and dangerous: a bad file deletion becomes a production deletion. Off (prune: false) in this repo's config. |
| app-of-apps | One Application whose job is to create the other Applications. Adding a new service becomes "add one file", not "click around the UI". |
What installing it actually looks like
Five commands. The wildcard DNS and certificate already cover a new subdomain, so the UI is reachable with no DNS or TLS work.
# 1. a namespace to live in
kubectl create namespace argocd
# 2. install ArgoCD itself (one big official manifest)
kubectl apply -n argocd -f \
https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# 3. let Traefik handle TLS, so ArgoCD serves plain http internally
kubectl -n argocd patch deployment argocd-server --type json \
-p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--insecure"}]'
# 4. route argocd.bit-habit.com to it (wildcard cert already covers this name)
kubectl apply -f apps/argocd/ingress.yaml
# 5. tell ArgoCD which repo and paths to watch — this file already exists
kubectl apply -f apps/argocd/application.yamlStep 5 is the whole idea, and it is already written in this repo:
spec:
source:
repoURL: https://github.com/bookseal/bit-habit-infra.git
targetRevision: main # watch the main branch
path: base # watch this folder
syncPolicy:
automated:
prune: false # do NOT delete things removed from git (safety)
selfHeal: true # DO undo hand-edits made in the clusterselfHeal, which is the valuable half — and it costs nothing to run.
ArgoCD adds ~5 Pods (controller, repo-server, api-server, redis, dex) to a free-tier ARM box,
plus one more web UI to lock down. For one node run by one person, that trade is honestly close.
The real argument for adopting it is not the tool — it's that the tool forces
everything into git. You can get that benefit first, and for free, without installing anything.
After this you can say what this platform should fix next, and in what order.
Honest scorecard: what this platform would need before GitOps is even possible, ranked by what actually matters. There is exactly one real blocker, and it is not the services — it is the secrets.
Today: real secrets are gitignored and applied by hand — the Route53 keys,
quali-fit's basic-auth, oauth2-proxy's client secret. Git holds only redacted
*.example.yaml templates.
Why it blocks GitOps: a controller that syncs "everything in git" cannot sync what is deliberately absent from git. You would keep a manual step forever, and every fresh install would silently come up broken.
The options, cheapest first:
Today: plane and fider are deployed from YAML that
sits in a home directory and is in no repo. Both are stock upstream apps, and barely used.
Why it is low on this list: ArgoCD can only sync what is in a repo, so
strictly these would have to move before it could manage them. But there is nothing bespoke
in them — losing them costs an hour with the upstream docs, not a service.
Drop them in apps/ whenever it is convenient.
Today: almost everything is jammed into the default namespace.
Only plane, quali-fit, fider and headlamp have their own.
Why it matters: a namespace is the unit ArgoCD scopes an Application
to, and the unit prune deletes within. With one shared default,
turning on prune means one controller with permission to delete everything.
Namespaces are what make prune safe to ever enable.
/var/lib/rancher/k3s/server/manifests/ is applied by k3s itself,
before and below ArgoCD. Traefik's own configuration will always be a
sudo cp onto the host. That is fine — but it should be written down, not
discovered. Do not chase 100%: aim for "everything that can be in git, is."
With secrets encrypted in git and namespaces per app, installing ArgoCD becomes the five commands above and it just works. Do it in that order and it is a calm afternoon. Do it in the reverse order and you spend a week fighting a controller that keeps trying to sync a repo that does not contain the truth.
| Dimension | Today | Needed for GitOps |
|---|---|---|
| Edge, TLS, routing | ✅ in git | ✅ already there |
| Most app manifests | ✅ in git (17 apps) | ✅ already there |
| physical-spark · quali-fit | ✅ in their own git repos | ✅ fine — ArgoCD reads many repos |
| plane · fider | ⚠️ stock apps, YAML in no repo | commit them — an hour, low stakes |
| Secrets | ❌ gitignored, applied by hand | encrypted in git (Sealed Secrets) |
| Namespaces | ⚠️ mostly one shared default | one per app, so prune is safe |
| Self-healing | ✅ kubectl apply on every deploy | ✅ already have the property |
| Host-level (Traefik, CoreDNS) | manual sudo cp | stays manual — out of scope, by design |
Checkpoint · end of Stage 5
Answer these before moving on. Open each one to check yourself — if any answer is a surprise, that part is worth a re-read.
Git holds the desired state, and a program inside the cluster continuously makes reality match it.
selfHeal — automatically undoing hand-made cluster edits by re-applying git over them. Notably, re-applying manifests on every deploy already buys most of this without installing anything.
Secrets. A controller that syncs “everything in git” cannot sync what is deliberately kept out of git. It is not the services — those are already there.
REFERENCE
Not a stage to read in order — two references to come back to. How to add a site yourself, and every word on this page defined once.
After this you can say how to add a new site to this cluster yourself.
How a new site gets added
newsite.bit-habit.com → that Service.kubectl apply). That's all: the wildcard DNS and wildcard certificate already
cover the new subdomain, so it's reachable over HTTPS right away.158.180.71.122 (private 10.0.0.61 inside Oracle Cloud).80 and 443 on the host directly.http → permanent 308 redirect to https.*.bit-habit.com (Let's Encrypt, DNS-01 via Route53), stored in
the Secret tls-secret and auto-renewed before its 90-day expiry.
See the real configs
It is all plain YAML in the public repo. A whole small app is often just this:
apiVersion: apps/v1
kind: Deployment # keep N copies of my app running
metadata: { name: myapp }
spec:
replicas: 1
selector: { matchLabels: { app: myapp } }
template:
metadata: { labels: { app: myapp } }
spec:
containers:
- name: web
image: nginx:alpine
---
apiVersion: v1
kind: Service # one stable address for those copies
metadata: { name: myapp-svc }
spec:
selector: { app: myapp }
ports: [{ port: 80, targetPort: 80 }]The key files that run this platform (click to view on GitHub):
| File | What it defines |
|---|---|
k3s-bootstrap/traefik-config.yaml | the edge — Traefik on host ports 80/443, http→https redirect |
base/ingress.yaml | the routing table — every subdomain → its Service |
base/cert-manager/cluster-issuer.yaml | how TLS is issued — Let's Encrypt + Route53 DNS-01 |
base/cert-manager/certificate.yaml | the wildcard *.bit-habit.com certificate |
apps/llm-app-lab/deployment.yaml | a minimal app — Deployment + Service |
apps/gatus/deployment.yaml | the status page — Deployment + Service + ConfigMap |
Go further
This page uses jargon that sounds like it assumes you already know it. You don't have to. Here is every word, defined once, with where it actually shows up here.
| Term | In plain words |
|---|---|
| bootstrap | The one-time setup that brings a machine from "empty Linux box" to "running cluster". You do it once and mostly never again — which is exactly why it must be written down. That is what k3s-bootstrap/ is. |
| systemd service | How Linux keeps a program running in the background and restarts it after a reboot. k3s runs as one. systemctl status k3s asks "is it alive?" |
| SSH | The encrypted remote login used to reach the server from elsewhere. |
| deploy key | An SSH key given to an automated system (here, GitHub Actions) rather than a person, so it can log in without a password. |
~/.ssh/authorized_keys | The list of keys allowed to SSH into this server. If a key is in this file, it can log in. |
| forced command | The security trick this platform leans on. Next to a key in authorized_keys, you can pin command="/path/to/one-script.sh". Then no matter what the remote side asks to run, the server runs only that script and nothing else. So a leaked deploy key cannot open a shell, read files, or delete anything — it can only trigger a redeploy. |
sudo | "Run this as the administrator (root)." Needed to touch system directories like /var/lib/rancher/. |
git pull / git reset --hard origin/main | Fetch the latest code. The reset --hard form throws away any local change and makes the folder an exact mirror of GitHub — so the server can never quietly drift from the repo. |
| Term | In plain words |
|---|---|
| Kubernetes (k8s) | A program that runs your other programs: it starts them, restarts them when they crash, and routes traffic to them. k3s is a small single-binary version of it, which is what runs here. |
| manifest | Just a YAML file describing something you want to exist. "Manifest" and "YAML file" are used interchangeably. |
| declarative | You describe the end state ("I want 1 copy of nginx running"), not the steps to get there. Kubernetes figures out the steps and keeps that statement true forever. This is the single biggest idea in the whole system. |
kubectl | The command-line tool you use to talk to the cluster. Everything below is a kubectl subcommand. |
kubectl apply -f file.yaml | The main command in this whole platform. It means: "read this file, and make the cluster match it." Not "create" — match. If the thing doesn't exist it's created; if it exists but differs it's updated; if it already matches, nothing happens. |
| idempotent | Running it twice does the same thing as running it once. kubectl apply is idempotent, which is why a deploy can safely re-apply every manifest every single time. That habit is what quietly gives this platform self-healing without ArgoCD. |
| Pod | One running instance of your app (a container). The smallest unit Kubernetes runs. |
| Deployment | The manifest that says "keep N Pods of this app running." If a Pod dies, the Deployment makes a new one. This is what restarts your crashed app at 4am. |
| Service | A stable internal address for a set of Pods. Pods come and go with new IPs; the Service name never changes. |
| Ingress | The routing rule: "requests for wiki.bit-habit.com go to the wikijs-svc Service." All of them together are the routing table. |
| ingress controller | The program that actually does what the Ingress rules say. Here that's Traefik. The Ingress is the rule; Traefik is the doorman reading it. |
| namespace | A folder for cluster objects. Purely for grouping and permissions. Most things here sit in the one called default. |
| Secret / ConfigMap | Config handed to a Pod. A Secret is for passwords and keys (base64-obscured, not encrypted); a ConfigMap is for everything else. |
| hostPath | Letting a Pod read a folder from the server's actual disk. It's why this page deploys in ~10 seconds: nginx serves the git checkout directly, so a git pull is the deploy — no image to build, no Pod to restart. |
| kubeconfig | The file holding the cluster's address and your credentials, so kubectl knows where to connect and how to prove who you are. Lives at /etc/rancher/k3s/k3s.yaml. |
| CRD (Custom Resource Definition) | How a tool teaches Kubernetes a brand-new kind of object. cert-manager adds Certificate; ArgoCD adds Application. After a CRD is installed, kubectl apply handles the new type like any built-in one. |
| controller / reconcile loop | A program that never stops asking "does reality match what was declared? no? fix it." Every piece of Kubernetes is one of these. ArgoCD is simply one more, whose declared state happens to live in git. |
| desired state vs actual state | What you wrote down, versus what is really running. The gap between them is called drift, and closing it is the entire job. |
| Term | In plain words |
|---|---|
| wildcard DNS | One rule — *.bit-habit.com → 158.180.71.122 — that points every possible subdomain at this server. It's why adding a site needs no DNS change at all. |
| TLS certificate | The file that proves a site is genuinely who it claims to be. It's what produces the padlock, and what makes https possible. |
| wildcard certificate | One certificate valid for *.bit-habit.com, so every subdomain shares it instead of each needing its own. |
| Let's Encrypt | The free authority that issues those certificates, automatically, every 90 days. |
| DNS-01 challenge | How Let's Encrypt checks you really own the domain: it asks you to place a specific secret record in your DNS. It's the only method that can issue a wildcard certificate — which is why this setup needs Route53 API access at all. |
| Route53 | AWS's DNS service, which hosts this domain. cert-manager holds an API key for it so it can write that challenge record on its own. |
| cert-manager | The robot in the cluster that requests the certificate, answers the DNS-01 challenge, and renews it before it expires. Nobody touches certificates by hand. |
| 308 redirect | "This moved permanently — go to the https version." Traefik sends it for every plain http request. |
| Term | In plain words |
|---|---|
| GitOps | The rule that git is the only source of truth: if it isn't in the repo, it shouldn't be in the cluster. A controller enforces it continuously. |
| push vs pull | Push (today): something outside shoves changes in. Pull (GitOps): something inside the cluster fetches from git on its own, forever. |
| ArgoCD / Flux | The two common programs that do the pulling. They are alternatives to each other; neither is installed here. |
| Application | ArgoCD's config object: "watch this repo + this folder, keep this namespace matching it." |
| sync | Actually applying git to the cluster. |
| selfHeal | Automatically re-apply git over any hand-made change. The property everyone actually wants. |
| prune | Delete from the cluster whatever was deleted from git. Genuinely dangerous — deleting a file becomes deleting production. Off here. |
| Sealed Secrets / SOPS | Ways to encrypt a secret so the encrypted file is safe to commit. Needed because GitOps cannot sync what is deliberately kept out of git. |
| Term | In plain words |
|---|---|
server_name | The line in an nginx config that says which web addresses this site answers to. nginx compares it against the Host: header the browser sent. |
default_server | The trap in the case study. If no server_name matches, nginx does not error — it serves its first/default site instead. So a forgotten hostname quietly returns someone else's website with a successful 200. |
| status code | The three-digit result of a request. 200 = the server answered normally, 404 = no such page, 500 = the server broke. Note that 200 only means "answered" — not "answered correctly". |
| single source of truth (SSOT) | The rule that exactly one place defines each thing. When two files define the same resource — or when something exists only in the cluster and in no file — you can no longer tell which one is right, and the last person to run a command wins. |
| drift | The running system and the files describing it have quietly grown apart. Usually created by fixing something directly in production and not updating the file. |
| blast radius | How much breaks when one change goes wrong. A routing file covering 15 sites has a blast radius of 15; one file per app has a blast radius of 1. |
| root cause | The condition that let the bug exist, not the line that displayed it. Here the visible bug was one wrong Ingress line; the root cause was that the app was in no repo, so that line had no owner. |
| health check | An automated request that asks "is this site up?" on a timer. Whether it checks only the status code or also the content decides which outages it can see. |
| retrospective | A write-up after an incident, aimed at the system rather than the person. START is one format: Situation, Task, Action, Result, Takeaway. |
| bisecting | Debugging by repeatedly cutting the problem in half. Testing the Service from inside the cluster proved the app was healthy, which eliminated everything behind the front door in one step. |