# shireen - developer documentation ## Overview Shireen is a news and weather bot. It fetches weather from wttr.in (no API key required) and headlines from RSS/Atom feeds. It posts fresh headlines automatically at a configurable per-channel interval and deduplicates across restarts using a persisted GUID list. --- ## Module structure ``` shireen/ commands.py - all logic, commands, broadcast tick __init__.py - empty ``` --- ## Data persistence All state is stored in a single JSON file at `data_path`. The structure is: ```json { "channels": { "#example": { "news_enabled": true, "broadcast_interval": 90, "broadcast_count": 1, "last_broadcast": 0 } }, "feeds": [...], "seen": ["guid1", "guid2"] } ``` The file is read once at startup into `bot.memory["sh_data"]` and written back on every change. This means the in-memory dict is the live state and the file is the persistence layer. Changes made to the file while the bot is running will be overwritten on the next save. ### _DEFAULT_CHANNEL mutation `setup()` mutates the module-level `_DEFAULT_CHANNEL` dict to apply the config-specified defaults. New channels added to `data["channels"]` after startup will use these mutated defaults. This is intentional: the config file controls what a "fresh" channel looks like. --- ## Feed parsing `_fetch_feed()` parses both RSS 2.0 and Atom feeds: - RSS: looks for `` elements with ``, `<link>`, and `<guid>`. - Atom: falls back if no items found, looks for `<entry>` elements with the Atom namespace. The fallback order means Atom feeds are only tried if RSS parsing finds nothing. Feeds that mix both formats will be parsed as RSS. GUIDs are used as unique identifiers. If a feed does not provide a GUID, the link URL is used instead. This is less stable but still prevents immediate re-posting. --- ## URL cleaning `_clean_url()` strips query strings and fragments from feed item links before storing and posting them. RSS feeds from major outlets routinely include tracking parameters in article links (`utm_source`, `ref`, etc.). Stripping them produces cleaner, shorter URLs. --- ## Deduplication `data["seen"]` is a list of GUIDs that have already been posted. Before posting an item, its GUID is checked against this list. After posting, the GUID is added. The list is pruned to 1000 entries (oldest removed) on every save. This prevents the file from growing unbounded. The seen list persists across restarts so the same article is never reposted even if the bot is restarted between broadcast cycles. --- ## Broadcast tick and locking `@plugin.interval(60)` runs every minute and checks each channel. The original implementation held `sh_lock` for the entire tick including all HTTP fetches. This blocked `!headlines` and `!news` commands for the full duration of the RSS fetches (multiple feeds, multiple channels). On a slow network this could mean several seconds of blocked commands. The fix splits the tick into three phases: 1. Under lock: read config, determine which channels need a broadcast, snapshot `feeds` and `seen`. 2. Outside lock: fetch headlines using the snapshot. HTTP happens here. 3. Under lock: write seen GUIDs, update `last_broadcast`, save to disk. The `seen` set is updated between channels in step 3 to prevent the same article from being posted to multiple channels in the same tick cycle. `!headlines` still holds the lock during its fetch since it is a single user-triggered request. The wait is acceptable for interactive commands. --- ## Weather Weather data comes from `wttr.in` using the JSON format `?format=j1`. No API key is required. The response includes current conditions, hourly data for 3 days, and nearest area information. `_format_location()` builds a clean location string using a country code lookup. "United States of America" becomes "US", etc. If the country is not in the lookup dict, the full country name is used as-is. `!forecast` uses `day["hourly"][4]` for the weather description. Index 4 corresponds to roughly midday (wttr.in hourly data is in 3-hour intervals starting at midnight, so index 4 = 12:00). This gives a more representative daily description than midnight or early morning conditions. --- ## Feed defaults Nine feeds are hardcoded in `_DEFAULT_DATA`. They are written to the data file on first run. After that, the feeds in the data file are used. To add, remove, or disable a feed, edit the `feeds` array in the data file while the bot is stopped. Each feed has an `enabled` field. Setting it to `false` excludes it from all headline fetches without removing it from the file. --- ## Per-channel configuration Each channel's config is stored separately in `data["channels"]`. `_chan()` creates a new channel entry with `_DEFAULT_CHANNEL` defaults if it does not exist. Channel settings are modified via `!news interval` and `!news count`. These write directly to the data dict and save immediately. --- ## Known issues and tradeoffs **No feed management via IRC.** Feeds can only be added or modified by editing the JSON file directly. There are no IRC commands for feed management. This was a deliberate simplicity choice. **RSS fetches can fail silently.** If a feed is unreachable or returns bad XML, the error is logged via `log.warning()` but not reported in IRC. The tick continues with the remaining feeds. This prevents a single broken feed from affecting all others. **Seen list is global, not per-channel.** An article posted in one channel will never be posted in another channel either. If you want different channels to receive the same articles independently, this would require reworking the seen list to be per-channel. **last_broadcast is stored in UTC epoch seconds.** The interval check is `now - last_broadcast < interval_sec`. This is wall-clock time, not adjusted for timezones. It works correctly as long as the bot's system clock is stable.