cnki-exp-search-automation_skill

Overview

This skill automates advanced searches on CNKI (China National Knowledge Infrastructure) using a headful browser automation approach. It performs professional query syntax searches, iterates result pages, and extracts structured metadata and abstracts. The skill is designed to handle common anti-bot verifications and supports exporting results to JSON or CSV for downstream analysis.

How this skill works

The skill drives a browser instance to the CNKI advanced search page, fills the expert search syntax, submits the query, and waits for result tables to load. It scrapes each result row for title, authors, date, citation count, and source, paginates with keyboard navigation to reduce detection, and optionally navigates to article detail pages to extract full metadata and abstracts. Anti-bot events are detected and handled by pausing, retrying, or invoking alternate JavaScript strategies while preserving the session.

When to use it

• Large-scale academic literature surveys where CNKI is a primary source
• Batch extraction of CNKI metadata for bibliometric or systematic reviews
• Automated retrieval of abstracts and keywords for text analysis
• When you have a headful browser environment to handle anti-bot checks
• Periodic updates to an existing CNKI dataset using incremental runs

Best practices

• Run in a headful browser environment and reuse the same browser session to reduce triggers
• Use CNKI professional search syntax (e.g., SU='term' and SU='term') for precise results
• Throttle requests and add 1–2 second waits between page extractions
• Prefer keyboard pagination (ArrowRight) to avoid click-based detection
• Save results incrementally (per page) to avoid data loss on failures

Example use cases

• Extract all search results for SU='conjunctival laxity' AND SU='treatment' into a CSV for a review paper
• Collect metadata and abstracts for a set of CNKI article URLs to build a training corpus
• Run scheduled incremental crawls to update citation and download counts in a local database
• Bulk-export CNKI search results to JSON for downstream bibliometric pipelines