1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
|
# shireen - developer documentation
## Overview
Shireen is a news and weather bot. It fetches weather from wttr.in (no API
key required) and headlines from RSS/Atom feeds. It posts fresh headlines
automatically at a configurable per-channel interval and deduplicates across
restarts using a persisted GUID list.
---
## Module structure
```
shireen/
commands.py - all logic, commands, broadcast tick
__init__.py - empty
```
---
## Data persistence
All state is stored in a single JSON file at `data_path`. The structure is:
```json
{
"channels": {
"#example": {
"news_enabled": true,
"broadcast_interval": 90,
"broadcast_count": 1,
"last_broadcast": 0
}
},
"feeds": [...],
"seen": ["guid1", "guid2"]
}
```
The file is read once at startup into `bot.memory["sh_data"]` and written
back on every change. This means the in-memory dict is the live state and
the file is the persistence layer. Changes made to the file while the bot
is running will be overwritten on the next save.
### _DEFAULT_CHANNEL mutation
`setup()` mutates the module-level `_DEFAULT_CHANNEL` dict to apply the
config-specified defaults. New channels added to `data["channels"]` after
startup will use these mutated defaults. This is intentional: the config
file controls what a "fresh" channel looks like.
---
## Feed parsing
`_fetch_feed()` parses both RSS 2.0 and Atom feeds:
- RSS: looks for `<item>` elements with `<title>`, `<link>`, and `<guid>`.
- Atom: falls back if no items found, looks for `<entry>` elements with the
Atom namespace.
The fallback order means Atom feeds are only tried if RSS parsing finds
nothing. Feeds that mix both formats will be parsed as RSS.
GUIDs are used as unique identifiers. If a feed does not provide a GUID,
the link URL is used instead. This is less stable but still prevents
immediate re-posting.
---
## URL cleaning
`_clean_url()` strips query strings and fragments from feed item links before
storing and posting them. RSS feeds from major outlets routinely include
tracking parameters in article links (`utm_source`, `ref`, etc.). Stripping
them produces cleaner, shorter URLs.
---
## Deduplication
`data["seen"]` is a list of GUIDs that have already been posted. Before
posting an item, its GUID is checked against this list. After posting,
the GUID is added. The list is pruned to 1000 entries (oldest removed) on
every save. This prevents the file from growing unbounded.
The seen list persists across restarts so the same article is never reposted
even if the bot is restarted between broadcast cycles.
---
## Broadcast tick and locking
`@plugin.interval(60)` runs every minute and checks each channel.
The original implementation held `sh_lock` for the entire tick including all
HTTP fetches. This blocked `!headlines` and `!news` commands for the full
duration of the RSS fetches (multiple feeds, multiple channels). On a slow
network this could mean several seconds of blocked commands.
The fix splits the tick into three phases:
1. Under lock: read config, determine which channels need a broadcast,
snapshot `feeds` and `seen`.
2. Outside lock: fetch headlines using the snapshot. HTTP happens here.
3. Under lock: write seen GUIDs, update `last_broadcast`, save to disk.
The `seen` set is updated between channels in step 3 to prevent the same
article from being posted to multiple channels in the same tick cycle.
`!headlines` still holds the lock during its fetch since it is a single
user-triggered request. The wait is acceptable for interactive commands.
---
## Weather
Weather data comes from `wttr.in` using the JSON format `?format=j1`. No
API key is required. The response includes current conditions, hourly data
for 3 days, and nearest area information.
`_format_location()` builds a clean location string using a country code
lookup. "United States of America" becomes "US", etc. If the country is not
in the lookup dict, the full country name is used as-is.
`!forecast` uses `day["hourly"][4]` for the weather description. Index 4
corresponds to roughly midday (wttr.in hourly data is in 3-hour intervals
starting at midnight, so index 4 = 12:00). This gives a more representative
daily description than midnight or early morning conditions.
---
## Feed defaults
Nine feeds are hardcoded in `_DEFAULT_DATA`. They are written to the data
file on first run. After that, the feeds in the data file are used. To
add, remove, or disable a feed, edit the `feeds` array in the data file
while the bot is stopped.
Each feed has an `enabled` field. Setting it to `false` excludes it from
all headline fetches without removing it from the file.
---
## Per-channel configuration
Each channel's config is stored separately in `data["channels"]`. `_chan()`
creates a new channel entry with `_DEFAULT_CHANNEL` defaults if it does not
exist. Channel settings are modified via `!news interval` and `!news count`.
These write directly to the data dict and save immediately.
---
## Known issues and tradeoffs
**No feed management via IRC.** Feeds can only be added or modified by
editing the JSON file directly. There are no IRC commands for feed management.
This was a deliberate simplicity choice.
**RSS fetches can fail silently.** If a feed is unreachable or returns bad
XML, the error is logged via `log.warning()` but not reported in IRC. The
tick continues with the remaining feeds. This prevents a single broken feed
from affecting all others.
**Seen list is global, not per-channel.** An article posted in one channel
will never be posted in another channel either. If you want different channels
to receive the same articles independently, this would require reworking the
seen list to be per-channel.
**last_broadcast is stored in UTC epoch seconds.** The interval check is
`now - last_broadcast < interval_sec`. This is wall-clock time, not adjusted
for timezones. It works correctly as long as the bot's system clock is stable.
|
