nealcaren/social-data-analysis
Overview
This skill provides step-by-step computational text analysis tailored for sociology and social science research. It guides users through study design, corpus exploration, method selection (classical and neural), systematic validation, and production of reproducible, publication-ready outputs. Support is provided for workflows in R and Python with concrete recommendations for when to use each language.
How this skill works
I lead you through five phased workflows: research design, corpus preparation, method specification, main analysis, and validation/interpretation. For each phase I prompt for decisions, recommend algorithms (LDA/STM, BERTopic, BERT, dictionaries, classifiers), and produce memos, diagnostics, and code-ready specifications. I emphasize reproducibility: documented preprocessing, fixed seeds, and human validation steps.
When to use it
- Exploring themes in large interview, news, or social media corpora (topic modeling).
- Measuring sentiment or concept prevalence with lexicons or supervised models.
- Building reproducible classification pipelines for coding or predictive tasks.
- Applying neural embeddings or transformer models for semantics or clustering.
- Producing publication-ready figures, tables, and replication materials.
Best practices
- Start with a design memo: clarify research question, corpus scope, and method choice.
- Explore the corpus thoroughly before modeling (descriptives, rare/common terms).
- Pre-specify preprocessing and evaluation metrics; record all parameters and seeds.
- Validate algorithmic outputs with human coding, sensitivity checks, and diagnostics.
- Choose language by method: use R for STM/LDA/ggplot workflows, Python for transformers/BERTopic.
Example use cases
- Use STM in R to study topic prevalence across demographic covariates in survey responses.
- Run BERTopic in Python to discover emergent themes in social media posts and cluster semantically similar messages.
- Validate a dictionary-based sentiment measure with human-coded subsamples and alternative lexicons.
- Train a classifier (tidymodels or sklearn) to label policy documents and report precision/recall across folds.
- Produce replication materials: cleaned DTM/embeddings, analysis scripts, and memoized decisions.
FAQ
Prefer R (stm package) for topic models with covariates and richer diagnostics; use Python for neural topic tools like BERTopic.
Do I need human validation for automated results?
Yes. Human coding, label checks, and sensitivity analyses are essential to establish that model outputs map to meaningful constructs.
11 skills
This skill guides sociology researchers through reproducible text analysis with R or Python, covering topic models, sentiment, classification, embeddings, and
This skill guides abductive qualitative analysis, helping you generate novel theories from interview anomalies by applying theory-first reasoning.
This skill helps instructors transform textbook chapters into engaging, evidence-based lectures with Google Slides, guiding outcomes, narrative design,
This skill helps you draft precise case justification sections for interview-based sociology articles by aligning context with theory and cluster benchmarks.
This skill helps you draft publication-ready Theory sections by guiding structure, turn development, and hedging calibrated to sociology norms.
This skill helps sociologists craft publication-ready interview writeups by guiding methods drafting, findings structure, and evidence presentation.
This skill helps researchers build a reproducible literature review using OpenAlex by guiding search, screening, snowballing, annotation, and synthesis.
This skill coordinates manuscript revision by routing feedback to specialized writing skills, ensuring coherent, section-specific improvements across the full
This skill helps sociologists draft publication-ready Methods sections for interview-based studies, guiding pathway selection, content coverage, and
This skill simulates peer review by constructing reviewer personas from Zotero sources to provide focused, actionable manuscript feedback.
This skill helps researchers generate genre-analysis-based writing skills from corpus conclusions, producing structured phases, cluster guides, and technique
