Narrative

AI Trend Analyzer Pipeline

~50 (manual)→10,000+articles processed daily

Keeping up with technology trends required reading thousands of articles manually. The process took a senior engineer 3 hours per week and still produced incomplete, subjective results.

AI/MLPythonData
← Narrative Hub
// the problem

What Was Broken

  • ❌3+ hours/week of manual reading to produce a trend summary report
  • ❌Coverage was limited to ~50 sources — thousands of relevant publications ignored
  • ❌Output was subjective and inconsistent — different engineers drew different conclusions
  • ❌No real-time visibility — reports were weekly snapshots, not live intelligence
// required fix
  • Automated scraping fleet covering 500+ tech sources
  • LLM-powered extraction of technology trends with structured JSON output
  • Real-time dashboard surfacing trends by category and volume
  • Process 10,000+ articles daily with 98%+ extraction accuracy
// solution

How It Was Built

Built a scraping fleet on ECS, Kafka pipeline for decoupled ingestion, LangChain extraction layer with few-shot prompts, and a dashboard aggregating trend signals.

Decoupled Scraping → Kafka → Processing
  • Scrapers publish raw articles to a Kafka topic.
  • 📄 pipeline/extractor.py
Structured Output + Retry on Hallucination
  • Used Pydantic schema validation on LLM output — any response that doesn't match the schema is retried with explicit correction prompting.

Decoupled Scraping → Kafka → Processing

Scrapers publish raw articles to a Kafka topic. The LLM processing service consumes independently — scraping can scale or fail without affecting the analysis pipeline.

pipeline/extractor.py
python
from langchain.output_parsers import PydanticOutputParser
from models import TrendSignal

parser = PydanticOutputParser(pydantic_object=TrendSignal)

prompt = PromptTemplate(
    template="Extract technology trends.\n{format_instructions}\n{article}",
    input_variables=["article"],
    partial_variables={"format_instructions": parser.get_format_instructions()}
)

Structured Output + Retry on Hallucination

Used Pydantic schema validation on LLM output — any response that doesn't match the schema is retried with explicit correction prompting. Hallucination rate below 2%.

// results

What Changed

10,000+ articles processed daily. Report generation went from 3 hours (manual) to 5 minutes (automated). 98% extraction accuracy validated against held-out test set.

Articles processed daily
~50 (manual)
→
0
200× coverage
Report generation time
3 hours
→
0
36× faster
Extraction accuracy
Subjective
→
0
Measurable
"One system replaced hours of weekly manual research with continuous, consistent, and quantified trend intelligence."
// deep dive q&a

Common Questions

We used Pydantic schemas to strongly type the LLM output. If the response violated the schema, we automatically retried the prompt with explicit instructions about the failure, dropping hallucinations below 2%.
Scraping is inherently bursty and prone to rate limits. Kafka acts as a buffer. If the scrapers pull in a massive spike of data, the LLM consumers process it at their own pace without overwhelming the API rate limits.
Instead of relying purely on fragile CSS selectors, we use a hybrid approach that relies on readability-like extraction and LLMs to pull the main content, making the scrapers extremely resilient to minor DOM changes.