KDD 2026 · Reproductive Health · Social Media · LLM Evaluation

RELIANCE: Curating and Evaluating Reproductive Health Information on Social Media

Vaibhav Balloli1, Laura Peyton Ellis2, Vishala Mishra3, Alice Chi1, Alex Peahl1, Elizabeth Bondi-Kelly1

1University of Michigan · 2University of Connecticut School of Medicine · 3Duke University School of Medicine

Paper Proceedings GitHub Dataset Contact Us
Overview of the RELIANCE data collection, annotation, analysis, and LLM evaluation pipeline.

Abstract

Social media platforms like TikTok have become a key source of health information, including for questions from the peripartum period: before, during, and after pregnancy. Inaccurate health information in this setting can have adverse consequences. As Large Language Model (LLM) providers increasingly integrate LLMs into digital platforms to fact-check content, with Perplexity on WhatsApp and xAI's Grok on X (formerly Twitter) now openly prompted to "fact-check this," these systems need to be evaluated on real-world multimodal videos rather than on standardized medical exams alone.

We introduce RELIANCE, an expert-annotated dataset of reproductive health information surfaced by TikTok search. The dataset starts from 56 clinician-reviewed natural-language questions about the peripartum period, collects the top six TikTok video results for each question, and asks expert clinicians to identify medically relevant sentences or paragraphs in the transcripts.

RELIANCE contains 409 annotated sentences or paragraphs from 336 videos. Clinicians label each one for inaccuracy and harmfulness, making it possible to separate information that is not supported by scientific evidence or standard clinical practice from medically dangerous information that can cause adverse consequences. We use the same annotations to evaluate whether LLMs can detect inaccurate and harmful information.

Dataset

Pie chart showing RELIANCE query distribution by pregnancy stage.
Each search question was assigned to a stage in the peripartum period so the dataset could be summarized by stage.
Pie chart showing RELIANCE query distribution by reproductive health category.
Clinicians also assigned each search question to a reproductive health category so the dataset could be summarized by query topic.
Correlation heatmap comparing accuracy and harmfulness labels with engagement metrics.
The heatmap compares clinician labels with views, likes, shares, comments, and creator followers; popularity was not a reliable signal of medical accuracy or harm.

What Search Returned

Videos found via TikTok search were largely relevant to the clinician-reviewed questions, and the collected videos averaged roughly 273K views.

What Clinicians Labeled

About 69.9% of annotated sentences or paragraphs were accurate, while 16.2% were harmful. Inaccurate information can be harmless or harmful, so the dataset keeps those labels separate.

Popularity Is Not Reliability

Views, likes, shares, comments, and creator followers do not reliably indicate whether a video contains accurate reproductive health information.

What the Dataset Covers

Clinicians assigned each search question to a stage of the peripartum period and to one or more reproductive health categories. The four stages are trying to conceive, during pregnancy, labor and delivery, and postpartum and infant care. The seven categories are physiology; symptoms; diet and exercise; supplements, medications, and vaccines; labor and pain management; open-ended "can I do this?" questions; and other.

The 409 annotated sentences and paragraphs are unevenly distributed across these groupings. Questions about pregnancy and about labor and delivery account for most of the annotations, and the two largest categories are open-ended "can I do this?" questions and physiology. Because a single question can touch several categories, the category counts overlap. Individual questions range across topics such as preeclampsia and aspirin, gestational diabetes screening, breastfeeding and lactation, contraception, fetal movement, epidurals and pain relief, prenatal supplements, and postpartum recovery.

LLM Evaluation

RELIANCE evaluates general-purpose LLMs on two granularities. In claim-level verification, the model checks a specific clinician-highlighted sentence or paragraph from a video. In video-level detection, the model receives the query and complete transcript and must decide whether the video contains inaccurate or harmful reproductive health information.

Gemini-2.5-Flash performs best overall in these evaluations, while most models are better at identifying harmful information than identifying inaccuracies. Performance drops when moving from specific claim verification to whole-video judgment, with a consistent gap of about 15% across tasks.

RELIANCE also tests safeguard models that flag user questions, video transcripts, and LLM responses for violations of their internal guidelines. These systems behave inconsistently on reproductive health content. The omni-moderation model flagged the query "How much kicking should a pregnant woman feel?" as violence, while Llama Guard often flagged responses as specialized advice.

Line charts showing LLM performance on detecting inaccurate reproductive health information.
Model performance on detecting inaccurate information, measured with accuracy, precision, recall, and calibration metrics.
Line charts showing LLM performance on detecting harmful reproductive health information.
Model performance on detecting harmful information; harmfulness is evaluated separately from factual accuracy.
Hamming distance heatmap comparing LLM predictions for the inaccuracy detection task.
Hamming distance shows how often pairs of models disagree on the same inaccuracy labels, which helps identify whether an ensemble would add complementary judgments.
Qwen model scaling panels for accuracy, precision, recall, calibration, and Brier score.
Comparing Qwen models of different sizes shows that larger models in the same family did not consistently improve reproductive health fact-checking performance.

Takeaways

Real-World Data Matters

Medical exam and question-answering benchmarks do not fully capture real-world multimodal data from TikTok videos.

Claim Granularity Matters

Models perform differently when checking one highlighted sentence or paragraph versus judging the full transcript of a video.

Safety Needs Domain Context

Safeguard models flagged different reproductive health queries and contexts under different violation categories, showing a need for consistency in this domain.

Where This Work Sits

RELIANCE connects several lines of research: health misinformation detection and automated fact-checking, benchmark design for large language model evaluation, content moderation and guardrail design, and public health communication on social media. Because the labels come from practicing clinicians in obstetrics, gynecology, and internal medicine rather than from general-purpose annotators, the dataset also speaks to work on expert annotation and annotator agreement in medical settings, and to AI for social good more broadly.

The source material is video rather than text, so the annotations also support research on multimodal and video-based health information understanding, and on how ranked search results shape what patients encounter when they look for answers about fertility, pregnancy, labor and delivery, and postpartum care.

Citation

RELIANCE was accepted at the KDD 2026 Datasets and Benchmarks track. If you use the dataset, the collection tool, or the evaluation setup, please cite the paper. 

@inproceedings{balloli2026reliance,
  title     = {RELIANCE: Curating and Evaluating Reproductive Health Information on Social Media},
  author    = {Balloli, Vaibhav and Ellis, Laura Peyton and Mishra, Vishala and
               Chi, Alice and Peahl, Alex and Bondi-Kelly, Elizabeth},
  booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery
               and Data Mining V.2 (KDD '26)},
  year      = {2026},
  pages     = {8624--8635},
  publisher = {Association for Computing Machinery},
  doi       = {10.1145/3770855.3817447},
  url       = {https://doi.org/10.1145/3770855.3817447}
}

Frequently Asked Questions

Can large language models (LLMs) fact-check health misinformation on social media?

RELIANCE evaluates whether general-purpose LLMs such as Gemini, GPT-4o, Qwen, and Llama Guard can detect inaccurate and harmful reproductive health information in real-world TikTok videos. Gemini-2.5-Flash performs best overall, but accuracy drops by about 15% when a model moves from verifying a single clinician-highlighted claim to judging an entire video, and most models are better at flagging harmful information than at catching factual inaccuracies.

What is the RELIANCE dataset?

RELIANCE (Reproductive hEaLth Information Access in oNline Content Environments) is an expert clinician-annotated dataset of reproductive health information surfaced by TikTok search. It contains 409 annotated sentences or paragraphs from 336 videos across 56 clinician-reviewed peripartum queries, each labeled separately for medical inaccuracy and harmfulness by expert clinicians in obstetrics, gynecology, and internal medicine.

What reproductive health topics does RELIANCE cover?

The 56 clinician-reviewed search questions span four stages of the peripartum period, namely trying to conceive, during pregnancy, labor and delivery, and postpartum and infant care, and seven reproductive health categories: physiology; symptoms; diet and exercise; supplements, medications, and vaccines; labor and pain management; open-ended "can I do this?" questions; and other. Most annotated claims come from questions about pregnancy and about labor and delivery.

How were the videos and transcripts collected?

Each clinician-reviewed question was issued as a TikTok search query and the top six video results were collected, yielding 336 videos. Collection used TikTik, the semi-manual tool released alongside the dataset on GitHub. Clinicians then read the video transcripts and marked the sentences or paragraphs carrying medical information, producing the 409 annotated spans. Clinicians agreed on 82.97% of accuracy labels and 86.91% of harm labels.

Is popular social media health content more medically accurate?

No. Engagement metrics such as views, likes, shares, comments, and creator followers show no strong correlation with medical accuracy or harmfulness. Popularity is not a reliable signal of whether a reproductive health video contains accurate information.

Do LLM safeguard and guardrail models handle reproductive health content reliably?

Not consistently. RELIANCE tests safeguard models including Llama Guard and the omni-moderation model and finds they flag reproductive health queries and responses inconsistently. The omni-moderation model, for instance, flagged the query "How much kicking should a pregnant woman feel?" as violence, showing that content-moderation guardrails need domain-specific context for reproductive and maternal health.

How is RELIANCE different from medical exam or question-answering benchmarks?

Medical exam and question-answering benchmarks test models on curated, text-only questions with known answers. RELIANCE instead uses the health information people actually encounter: transcripts of real TikTok videos returned by ordinary search queries, where claims sit inside longer narration and where inaccurate information may or may not be clinically harmful. RELIANCE keeps inaccuracy and harmfulness as separate labels for exactly that reason.

Which models does RELIANCE evaluate?

The benchmark covers general-purpose LLMs including Gemini, GPT-4o, and Qwen on two tasks: claim-level verification of a single clinician-highlighted sentence or paragraph, and video-level detection over a complete transcript. It separately evaluates safeguard models, including Llama Guard and the omni-moderation model, on user queries, video transcripts, and model responses. Comparing Qwen models of different sizes shows that scaling within a single model family did not consistently improve reproductive health fact-checking.

Why does health misinformation on TikTok matter for pregnancy and postpartum care?

Nearly 59% of American adults under 30 use TikTok, and many turn to social media for pregnancy, postpartum, and fertility questions. About 30% of the health claims surfaced by search in this dataset are inaccurate, and roughly half of those are clinically harmful, the kind of misinformation that can cause real adverse consequences during the peripartum period.

How can I access and cite the RELIANCE dataset and paper?

The dataset and collection tool are openly available on GitHub, the full paper is on arXiv, and the published version is in the KDD '26 proceedings. RELIANCE was accepted at the KDD 2026 Datasets and Benchmarks track. For questions, contact us.