read_large_webpage_skill

Overview

This skill performs disciplined, segment-by-segment reading of large web pages or knowledge bases and turns raw text into structured, traceable knowledge entries. It captures original fragments, produces real-time summaries, and incrementally writes refined points into a knowledge store to maintain order and recoverable progress. The workflow supports iterative reads, cross-segment traceability, and periodic global reviews.

How this skill works

Before reading, the skill records the resource identifier, desired segment size, focus topic, and output format. It fetches text ranges via get_knowledge_by_lines, extracts and annotates key points in real time, and stores summaries with context using add_knowledge. Progress indices, source line numbers, timestamps, and fragment IDs are kept to enable continuation, cross-segment comparison, and traceability. At checkpoints it merges partial summaries into higher-level context maps and flags unresolved questions for follow-up.

When to use it

• Ingesting long technical reports, white papers, or policy documents that exceed single-pass processing limits.
• Building or updating a searchable knowledge base from lengthy web pages or document archives.
• Performing focused research where you need summaries tied to exact source lines for citation or verification.
• Gradually training agents or building world models from large corpora while preserving provenance.
• Recovering interrupted reads and resuming from precise offsets without data duplication.

Best practices

• Specify the resource ID, segment size, topic focus, and desired output format before starting to read.
• Use conservative segment sizes to keep fetches fast and summaries precise; record offsets for robust recovery.
• Preserve original fragment IDs and source line numbers to enable cross-segment comparisons and traceability.
• Write intermediate summaries promptly with add_knowledge so partial results are usable and durable.
• Record unanswered questions or potential data gaps in each summary so follow-up reads can prioritize them.

Example use cases

• Summarize a 200-page technical standard into linked 500-line summaries with source offsets for later verification.
• Read a large web encyclopedia entry in 100-line segments, store concise bullet summaries, and merge into a topic map for an agent training dataset.
• Extract and annotate policy document sections, flag ambiguous clauses, and deposit resolved interpretations into a knowledge base.
• Continuously ingest news archives by date ranges, maintain progress indices, and run periodic global reviews to build a temporal knowledge graph.