init: mostly vibed

1 files changed, 173 insertions, 0 deletions

diff --git a/dev-docs/shireen.md b/dev-docs/shireen.md
new file mode 100644
index 0000000..85563a5
--- /dev/null
+++ b/dev-docs/shireen.md
@@ -0,0 +1,173 @@
+# shireen - developer documentation
+
+## Overview
+
+Shireen is a news and weather bot. It fetches weather from wttr.in (no API
+key required) and headlines from RSS/Atom feeds. It posts fresh headlines
+automatically at a configurable per-channel interval and deduplicates across
+restarts using a persisted GUID list.
+
+---
+
+## Module structure
+
+```
+shireen/
+  commands.py  - all logic, commands, broadcast tick
+  __init__.py  - empty
+```
+
+---
+
+## Data persistence
+
+All state is stored in a single JSON file at `data_path`. The structure is:
+
+```json
+{
+  "channels": {
+    "#example": {
+      "news_enabled": true,
+      "broadcast_interval": 90,
+      "broadcast_count": 1,
+      "last_broadcast": 0
+    }
+  },
+  "feeds": [...],
+  "seen": ["guid1", "guid2"]
+}
+```
+
+The file is read once at startup into `bot.memory["sh_data"]` and written
+back on every change. This means the in-memory dict is the live state and
+the file is the persistence layer. Changes made to the file while the bot
+is running will be overwritten on the next save.
+
+### _DEFAULT_CHANNEL mutation
+
+`setup()` mutates the module-level `_DEFAULT_CHANNEL` dict to apply the
+config-specified defaults. New channels added to `data["channels"]` after
+startup will use these mutated defaults. This is intentional: the config
+file controls what a "fresh" channel looks like.
+
+---
+
+## Feed parsing
+
+`_fetch_feed()` parses both RSS 2.0 and Atom feeds:
+
+- RSS: looks for `<item>` elements with `<title>`, `<link>`, and `<guid>`.
+- Atom: falls back if no items found, looks for `<entry>` elements with the
+  Atom namespace.
+
+The fallback order means Atom feeds are only tried if RSS parsing finds
+nothing. Feeds that mix both formats will be parsed as RSS.
+
+GUIDs are used as unique identifiers. If a feed does not provide a GUID,
+the link URL is used instead. This is less stable but still prevents
+immediate re-posting.
+
+---
+
+## URL cleaning
+
+`_clean_url()` strips query strings and fragments from feed item links before
+storing and posting them. RSS feeds from major outlets routinely include
+tracking parameters in article links (`utm_source`, `ref`, etc.). Stripping
+them produces cleaner, shorter URLs.
+
+---
+
+## Deduplication
+
+`data["seen"]` is a list of GUIDs that have already been posted. Before
+posting an item, its GUID is checked against this list. After posting,
+the GUID is added. The list is pruned to 1000 entries (oldest removed) on
+every save. This prevents the file from growing unbounded.
+
+The seen list persists across restarts so the same article is never reposted
+even if the bot is restarted between broadcast cycles.
+
+---
+
+## Broadcast tick and locking
+
+`@plugin.interval(60)` runs every minute and checks each channel.
+
+The original implementation held `sh_lock` for the entire tick including all
+HTTP fetches. This blocked `!headlines` and `!news` commands for the full
+duration of the RSS fetches (multiple feeds, multiple channels). On a slow
+network this could mean several seconds of blocked commands.
+
+The fix splits the tick into three phases:
+
+1. Under lock: read config, determine which channels need a broadcast,
+   snapshot `feeds` and `seen`.
+2. Outside lock: fetch headlines using the snapshot. HTTP happens here.
+3. Under lock: write seen GUIDs, update `last_broadcast`, save to disk.
+
+The `seen` set is updated between channels in step 3 to prevent the same
+article from being posted to multiple channels in the same tick cycle.
+
+`!headlines` still holds the lock during its fetch since it is a single
+user-triggered request. The wait is acceptable for interactive commands.
+
+---
+
+## Weather
+
+Weather data comes from `wttr.in` using the JSON format `?format=j1`. No
+API key is required. The response includes current conditions, hourly data
+for 3 days, and nearest area information.
+
+`_format_location()` builds a clean location string using a country code
+lookup. "United States of America" becomes "US", etc. If the country is not
+in the lookup dict, the full country name is used as-is.
+
+`!forecast` uses `day["hourly"][4]` for the weather description. Index 4
+corresponds to roughly midday (wttr.in hourly data is in 3-hour intervals
+starting at midnight, so index 4 = 12:00). This gives a more representative
+daily description than midnight or early morning conditions.
+
+---
+
+## Feed defaults
+
+Nine feeds are hardcoded in `_DEFAULT_DATA`. They are written to the data
+file on first run. After that, the feeds in the data file are used. To
+add, remove, or disable a feed, edit the `feeds` array in the data file
+while the bot is stopped.
+
+Each feed has an `enabled` field. Setting it to `false` excludes it from
+all headline fetches without removing it from the file.
+
+---
+
+## Per-channel configuration
+
+Each channel's config is stored separately in `data["channels"]`. `_chan()`
+creates a new channel entry with `_DEFAULT_CHANNEL` defaults if it does not
+exist. Channel settings are modified via `!news interval` and `!news count`.
+These write directly to the data dict and save immediately.
+
+---
+
+## Known issues and tradeoffs
+
+**No feed management via IRC.** Feeds can only be added or modified by
+editing the JSON file directly. There are no IRC commands for feed management.
+This was a deliberate simplicity choice.
+
+**RSS fetches can fail silently.** If a feed is unreachable or returns bad
+XML, the error is logged via `log.warning()` but not reported in IRC. The
+tick continues with the remaining feeds. This prevents a single broken feed
+from affecting all others.
+
+**Seen list is global, not per-channel.** An article posted in one channel
+will never be posted in another channel either. If you want different channels
+to receive the same articles independently, this would require reworking the
+seen list to be per-channel.
+
+**last_broadcast is stored in UTC epoch seconds.** The interval check is
+`now - last_broadcast < interval_sec`. This is wall-clock time, not adjusted
+for timezones. It works correctly as long as the bot's system clock is stable.