Developer Documentation

Architecture, API contracts, data schemas, and the hard-won gotchas.

Flask + Next.js 16.1.6

Redis + Solr

Ollama on M1

Auth.js v5 Beta

Stack Overview

Backend: Python / Flask / gunicorn (port 5005)
Frontend: Next.js 16.1.6 / React 19 / TypeScript
Database: Redis (in-memory, port 6379)
Search: Apache Solr (full-text, port 8983)
AI Inference: Ollama on MacBook Air M1 (192.168.1.185)
Auth: Auth.js v5 beta — Google OAuth, JWT sessions
Proxy: Caddy (automatic TLS via Let's Encrypt)
Process Mgr: arc.sh + systemd itc-stack.service

Services & arc.sh

All services are managed by arc.sh. The stack auto-starts on boot via /etc/systemd/system/itc-stack.service (legacy name from itc era — paths are correct).

./arc.sh start|stop|restart [service]
./arc.sh status          # service states + log/backup sizes
./arc.sh build           # Docker build + restart frontend
./arc.sh backup          # fast SSD backup, code only (stack stops briefly)
./arc.sh backup-cold     # full archive to /mnt/data (stack stays up)
./arc.sh checkup         # health check + error scan + CPU/RAM
./arc.sh logs [service]  # tail logs for a service
./arc.sh prune [dry]     # rotate old backups

# Named service control:
./arc.sh restart gunicorn
./arc.sh restart scribe
./arc.sh restart linkedin_poster
./arc.sh restart frontend

# LinkedIn on/off (no restart needed):
redis-cli -a $REDIS_PASSWORD set linkedin:autopost 1
redis-cli -a $REDIS_PASSWORD set linkedin:autopost 0

gunicorn: Flask API — port 5005, must run before build
scribe: v50.0 — RSS scraper + full A.R.C. pipeline
manual_publisher: v5.1 — URL/text/file submissions
stream_consumer: Redis Streams consumer
analyzer: On-demand analysis worker
mailer: v1.0 ACTIVE — alerts + 7am digest
linkedin_poster: v1.0 — auto-posts to LinkedIn, on/off via Redis
frontend: Docker container arc-frontend — port 3000
watchdog: 60s check loop, restarts crashed services

Watchdog restart logic: Only restarts a service if its PID file exists. A missing PID file means intentionally stopped. A stale PID file means crashed.

Reverse Proxy: Caddy Routing

/api/auth/* and /api/user/* MUST appear before the /api/* catch-all. These routes go to Next.js (3000), not Flask (5005). Getting the order wrong silently breaks all auth and user prefs.

arc-codex.com, www.arc-codex.com {
  handle /api/auth/* { reverse_proxy localhost:3000 }  # Auth.js
  handle /api/user/* { reverse_proxy localhost:3000 }  # Prefs proxy
  handle /api/*      { reverse_proxy localhost:5005 }  # Flask
  handle             { reverse_proxy localhost:3000 }  # Next.js
  tls rossnesbitt@gmail.com
}

API Reference

GET/api/articlesPaginated feed — ?limit=N&offset=N
GET/api/articles/<id>Single article with full analysis
POST/api/submitSubmit URL/text/file for processing
GET/api/searchSolr full-text — ?q=query&limit=N
GET/api/translate/<id>Translate article + analysis — ?lang=<language>
DELETE/api/translate/<id>/cacheAdmin cache bust
GET/api/user/prefsFetch prefs (via Next.js proxy)
POST/api/user/prefsUpsert on login (loopback only)
PATCH/api/user/prefsUpdate preferred_lang (via proxy)
DELETE/api/user/prefsGDPR self-service deletion
GET/api/rssRSS 2.0 — full analysis per item

All blueprints registered in backend/main.py inside the Redis try block. Flask restart required after adding a new blueprint.

Redis Data Schemas

Authentication Architecture

Soft auth model — the site is fully public. Google login is optional and unlocks preferences only. No username/password fallback.

Browser
  → Next.js /api/auth/[...nextauth]  (Auth.js catch-all)
  → Google OAuth callback
  → JWT session cookie set (30 days)

Browser requests /api/user/prefs
  → Next.js app/api/user/prefs/route.ts  (server-side proxy)
  → Adds X-User-Id: {google_sub} header
  → Flask /api/user/prefs (loopback only — rejects if not 127.0.0.1)

trustHost: true is required in auth.ts. Without it, all auth routes return UntrustedHost error when behind a reverse proxy.

Use account.providerAccountId for the Google sub, not user.id. The JWT strategy populates these differently.

@auth/redis-adapter does not exist as a standalone package. The adapters.js in this beta is empty. JWT sessions are the correct approach — no adapter needed.

AI Pipeline

All AI inference routes through ollama_utils.py. Never duplicate call_ollama_with_fallback() in other files.

call_ollama_with_fallback() returns a TUPLE. Always use result[0] for text. Never unpack as text, duration = result — it returns more than 2 values and raises ValueError.

# Correct:
result = call_ollama_with_fallback(prompt, model)
text = result[0]

# Wrong — raises ValueError:
text, duration = call_ollama_with_fallback(prompt, model)

Models: devstral (cloud, primary) → gemma3:4b (local M1, fallback). gemma3:4b handles simple tasks but struggles with large JSON translation payloads. Translation failures on 429 are graceful — "model unavailable" shown to user.

DO NOT auto-translate on component mount in feed view. 33 article cards firing simultaneous Ollama requests blocks all gunicorn threads and takes down the site. preferred_lang is a click shortcut — it skips the language dropdown, it does not auto-fire.

Frontend Gotchas

FeedClient.tsxNEVER restructure. Surgical deletions only. Keep React.Fragment structure.
LayoutTheme.module.cssALWAYS check here first for color issues. It overrides everything else.
UserPrefsContextSingle source of truth for prefs. Import from @/components/UserPrefsContext — NOT @/hooks/useUserPrefs.
postcss.config.jsCommonJS (.js) only. The .mjs version references @tailwindcss/postcss which is not installed — build will fail.
npm installAlways use --legacy-peer-deps (set permanently in .npmrc — automatic).
Next.js version16.1.6 — not 14. App Router. Turbopack enabled.
spaCy installUse pip wheel URL. NOT python3 -m spacy download (typer conflict).
AdsFully removed. Do not re-add AdSense, GAM, or any ad network components.
CopyAllButtonClient component — import separately, never inline 'use client' in server components.

Search: Solr

Full-text search via pysolr. Endpoint: http://localhost:8983/solr/articles.

Schema fields: id, title, content, source, url, timestamp, directive, chimera_score.

Lazy reconnect: Both main.py and scribe.py use a global solr lazy reconnect pattern. This fixes the boot-order race condition where Solr starts after application services.

Admin tool: kasmir7.py functions 5–8 handle re-index, diagnostics, and orphan purging. 31,924 Solr orphans were purged during the v5.0 migration.

Planned Features

NEXT SESSIONarc.sh restore

List available backup tarballs, interactive selection, confirmation prompt, extract to stack root, auto-restart affected services. Fits existing arc.sh bash pattern.

NEXT SESSIONarc_admin.py — Curses TUI

Python stdlib curses. Password-protected terminal menu. Sections: System, Backups, Articles, Users, Ollama. Auth via Redis is_admin flag. Launch via ./arc.sh admin.

Future Roadmap

Auto-translate on article detail page /article/[slug] only (safe — one article)
Email digest notifications (mailer.py stub ready)
Topic/category preferences per user
GitHub SSO (second OAuth provider)
Article deduplication (SimHash/MinHash)
Ollama model auto-switching on credit exhaustion
Netdata integration for custom pipeline metrics
TLS for IMAP (port 993) via Let's Encrypt