Deep Research

Architecture

research/{topic-slug}/
  plan.md                    # Research plan with sub-questions
  sources-index.md           # URL registry: what's been scraped, metadata
  sources/                   # Raw scraped content (one file per source)
    001-source-slug.md
    002-source-slug.md
    ...
  notes/                     # Extracted findings per sub-question
    question-1.md
    question-2.md
    ...
  report.md                  # Final compiled report

Research Parameters

• Breadth (default: 3): Parallel search queries per sub-question. Range: 2-6.
• Depth (default: 2): Recursive deepening rounds. Range: 1-4.

Step-by-Step Workflow

Phase 0: Initialize

1. Create the research directory:

   mkdir -p research/{topic-slug}/sources research/{topic-slug}/notes
   

2. Search local filesystem first. Before any web search, check if relevant content already exists:
   • Past research in research/ directories in the working directory
   • Relevant files in the working directory or project
   • Use grep, glob, or semantic search (lss) as appropriate for the task
   • If prior research exists on this topic, read it and build on it -- don't start from scratch
3. Initialize sources-index.md:

   # Sources Index
   
   | # | URL | Title | Date | Type | Credibility | File |
   |---|-----|-------|------|------|-------------|------|
   

Phase 1: Planning

1. Analyze the query. Identify:
   • Core research question(s)
   • Required evidence types (empirical data, expert opinion, primary sources, statistics)
   • Potential biases to watch for
2. Decompose into 2-5 independently researchable sub-questions.
3. Write plan.md with the sub-questions, search strategy, and scope decisions.
4. Create a todo list tracking each sub-question.

Phase 2: Search-Read-Extract Loop

2a. Generate Search Queries

• Phrasing (synonyms, technical vs. lay terms)
• Angle (supporting evidence, counter-evidence, meta-analyses)
• Source type (academic site:scholar.google.com, government site:gov, news, industry)
• Recency (add year filters for time-sensitive topics)

2b. Read and Extract

1. Check sources-index.md first. If the URL is already scraped, skip it. Do not re-scrape pages already processed in this session.
2. Scrape the page. Batch URLs for efficiency:
   scrape_webpage("url1, url2, url3")
3. Save raw content to disk immediately:

   # Write scraped content to sources/NNN-slug.md
   

   Include a header with the URL, title, and scrape date. This gets the raw content out of your context.
4. Extract key findings from the scraped content and append to notes/question-N.md:
   • Key claims and findings (with brief quotes when important)
   • Data points: numbers, statistics, dates
   • The source number (for citation mapping later)
   • Methodology notes if relevant (sample size, study design)
5. Update sources-index.md with the new source entry:
   | 003 | https://example.com/article | Article Title | 2025-06-15 | news | medium | sources/003-article-title.md |
6. Free your context. After extracting findings and saving to disk, you do not need to retain the raw scraped content. Move on to the next source.

2c. Deepen (Recursive)

1. Read notes/question-N.md to assess coverage.
2. Identify gaps: what remains unanswered? What claims lack corroboration?
3. Generate follow-up questions based on findings.
4. If depth > 0: recurse with follow-up questions at depth - 1 and ceil(breadth / 2).
5. If depth == 0: stop, move to next sub-question.

Phase 3: Synthesis

1. Read all notes/*.md files (not raw sources -- the notes are already distilled).
2. Cross-reference claims -- flag where sources agree vs. contradict.
3. Resolve contradictions by source quality:
   • Peer-reviewed > government data > industry reports > news > blogs
   • Meta-analyses > individual studies
   • Multiple independent sources > single source
4. Identify consensus vs. uncertainty. Be explicit about what is well-established vs. debated vs. unknown.
5. Check for bias: Did search results skew toward one viewpoint? Are key perspectives missing?

Phase 4: Report Generation

# [Research Title]

## Executive Summary
[2-3 paragraph overview of key findings and conclusions]

## Key Findings

### [Finding 1 Title]
[Discussion with inline citations [1][2]]

### [Finding 2 Title]
[Discussion with inline citations [3][4]]

## Analysis
[Cross-cutting patterns, contradictions, consensus areas]

## Limitations & Caveats
[What this research couldn't determine, methodological limitations]

## Conclusions
[Evidence-based conclusions with confidence levels]

## Sources

[1] Author (Date). "Title." *Publication*. URL
[2] Author (Date). "Title." *Publication*. URL

Phase 5: Finalize

1. Save report.md in the research directory.
2. Report to user: main conclusions (2-3 sentences), number of sources, confidence levels, file path.

Citation Rules

1. Every factual claim must have a citation. No uncited assertions of fact.
2. Use numbered inline citations [1], [2] that map to the Sources section.
3. Never fabricate sources. Every URL in Sources must be a real page you actually visited and read.
4. Quote directly when the exact wording matters. Use > blockquote for direct quotes.
5. Distinguish levels of evidence:
   • "Studies show..." (cite the studies)
   • "According to [Source]..." (single source, acknowledge it)
   • "The evidence suggests..." (multiple corroborating sources)
   • "It remains debated whether..." (conflicting evidence)
6. Date every source. If no date is found, write "(n.d.)".
7. Note source type in the bibliography: academic paper, government report, news article, blog post, etc.

Scientific Rigor Standards

• Falsifiability: Actively search for counter-evidence. Don't just confirm priors.
• Replication: A claim supported by multiple independent sources is stronger than one from a single source.
• Recency awareness: Newer isn't always better. Note when older foundational work is more relevant.
• Correlation vs. causation: Flag when sources conflate the two.
• Sample size and methodology: Note these for empirical claims.
• Conflicts of interest: Note when a source has a stake in its conclusions.

Tool Usage Patterns

Search (always use search_depth "advanced" for deep research)

Scrape (batch URLs)

Save scraped content to disk

# After scraping, immediately write to file and extract findings
# This is critical -- do not hold raw scraped content in context

Academic Paper Search (OpenAlex)

# Find highly-cited papers on a topic
curl -s "https://api.openalex.org/works?search=topic+keywords&filter=cited_by_count:>50,type:article,has_abstract:true&sort=cited_by_count:desc&per_page=15&select=id,display_name,publication_year,cited_by_count,doi,authorships,abstract_inverted_index&mailto=agent@beeuni.ai"

# Find recent preprints
curl -s "https://api.openalex.org/works?search=topic&filter=type:preprint,publication_year:2025&sort=publication_date:desc&per_page=10&mailto=agent@beeuni.ai"

# Find review/survey papers
curl -s "https://api.openalex.org/works?search=topic&filter=type:review,cited_by_count:>20&sort=cited_by_count:desc&per_page=10&mailto=agent@beeuni.ai"

# Follow citation chains: who cites a seminal paper?
curl -s "https://api.openalex.org/works?filter=cites:WORK_ID&sort=cited_by_count:desc&per_page=10&mailto=agent@beeuni.ai"

Web Source Hunting

Data Source Hunting

Fact-Checking Pattern

Local FS Search (check before web search)

# Search past research
grep -r "keyword" research/

# Semantic search if available
lss "research question" -p /workspace --json -k 10

Handling Edge Cases

• No good results found: State what was searched and that evidence is lacking. This is a valid finding.
• Highly technical topics: Search for both technical papers and accessible explainers.
• Rapidly evolving topics: Prioritize recency. Note publication dates prominently.
• Controversial topics: Present all major viewpoints with evidence. Don't editorialize.
• Paywalled sources: Extract what's available from abstracts and secondary reporting. Note when full text was not accessible.
• Resuming prior research: If a research directory already exists, read the existing plan, notes, and sources-index. Continue from where the previous session left off rather than starting over.

Research For Assets

1. Research phase -- gather facts, dates, numbers, citations, and source URLs using the workflow above.
2. Asset collection phase -- gather verified images, logos, screenshots, or media inputs if the deliverable needs them.
3. Build phase -- only then create the website, deck, report, or other asset from the completed research.

Parallel and Long-Running Research