Jiande Wu

AI Frontier Digest

An unattended LLM pipeline that turns AI news into a short daily digest in Chinese.

Live site · Code · RSS · Python (standard library only) · Gemini API · GitHub Actions

Problem

Keeping up with AI research and industry news means reading dozens of blogs, lab announcements, Hacker News, and paper feeds, most of it in English and much of it repeating the same event. The goal was a digest for readers with a science or engineering background who are not specialists: at most ten items a day, the ones most worth reading, each explained in plain Chinese, with no manual curation.

Pipeline

A scheduled GitHub Actions job runs every four hours. Each run:

  1. Collects from 31 researcher, lab, and news RSS/Atom feeds, the Hacker News front page (200+ points, AI-related titles), and Hugging Face Daily Papers (100+ upvotes).
  2. Filters to unseen items from the last three days. Items are keyed by a hash of the normalized link; undated items are skipped and future dates are clamped to the run time.
  3. Summarizes new items in batches of 15: the LLM writes a Chinese title, a one- or two-sentence summary, a category, and a 1–5 importance rating.
  4. Groups items from the last two days that report the same event, such as one model release covered by a lab blog, a newsletter, and Hacker News.
  5. Picks the daily list: a rating of 3 or more, at most two items per source, one item per event, and at most ten per day.
  6. Explains each pick from the article's full text in three or four sentences; when the page cannot be read, the short summary stays.
  7. Publishes the daily pages, the README, and an RSS feed as a commit; GitHub Pages serves the site.

Design decisions

Structured output with a fallback, then validate anyway

Every LLM call sends a per-batch JSON Schema in which item IDs and categories are enums, so the model can only refer to items it was given. The API sometimes rejects these enum-heavy schemas; on that error the call retries in plain JSON mode with the format spelled out in the prompt. Either way, every field is re-validated locally: unknown IDs are dropped, an invalid category gets a default, and an out-of-range rating is replaced.

Failures retry on the next run instead of being lost

Items are stored only after they are summarized. If a batch fails, its items stay unseen and the next run, four hours later, picks them up again. Transient HTTP errors (429 and 5xx) are retried with backoff within a run.

Keep raw model ratings; rank at render time

The CSV store keeps the model's raw rating. Ranking rules are applied when pages are rendered: one extra star for long-form posts by individual researchers, one for events covered by two or more sources, capped at five. Because the raw data never changes, the rules can be tuned without calling the model again.

One report per event

Grouping lets the digest show each event once. The highest-rated report wins, and on ties the earliest one, which is usually the original source. The other reports are listed under it as "see also." Coverage by several sources is itself a signal and raises the rating.

Article text is untrusted input

Full-text explanations pass scraped article text to the model. The prompt states that any instruction inside the text is content to be summarized, not followed. Extraction keeps the longest <article>, <main>, or <body> block, capped at 8,000 characters. Pages that yield fewer than 500 characters, usually paywalls or JavaScript-only pages, are treated as unreadable. Hugging Face papers use the API abstract instead.

Boring infrastructure

The pipeline uses only the Python standard library. The data store is a CSV per month in the repository, so every change shows up in a readable diff. Output is deterministic, so a run with nothing new produces no commit. Unit tests run before every update, and a monthly job fails when a feed is dead or has not updated for a year.

Limitations and next steps