Political Ideology and Polarization: A Multi-dimensional Approach

Probably not a substantial claim: may not pass S3 (load-bearing).

The proposed computational models classify the partiality of media sources without quantifying their ideology.

“The proposed computational models classify the partiality of media sources without quantifying their ideology (Elejalde et al., 2018).”

• cites Elejalde et al., 2018 sourceDisputed

  The Related work section says the authors propose a methodology that quantifies news outlets’ political leaning, contrary to the claim that their proposed model does not quantify ideology. The Abstract likewise says they automatically compute outlets’ political and socio-economic orientation. Although it says earlier studies often classified partiality without quantifying it, that does not describe this paper’s proposed method; page numbers were not provided in our copy. (LLM, conf=0.87)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

May combine two claims.

Agreements among annotators were moderate for economic, substantial for foreign, and fair for social.

“The agreements among annotators for the economic and foreign dimensions are moderate & substantial (Artstein and Poesio, 2008), respectively; for social, the 'fair' agreement was noticed during annotation, and additional discussion for each paragraph was then held.”

• cites Artstein and Poesio, 2008 sourceCould not verify

  The citation is attached to the economic and foreign agreement ratings; the citing sentence presents the social rating as its own observation. Section 5.3 discusses how to interpret agreement coefficients but gives no ratings for economic or foreign annotations. Searches for those dimensions and for “moderate” and “substantial” found no supporting passage in our copy. (LLM, conf=0.73)

  Experimental — accuracy not yet measured

May combine two claims.

In contrast to our multi-dimensional approach that examines the ideology of the issue being discussed instead of the author stance, much of the recent work in computational linguistics has been dedicated to the latter while collapsing ideology to one dimension.

“In contrast to our multi-dimensional approach that examines the ideology of the issue being discussed instead of the author stance, much of the recent work in computational linguistics has been dedicated to the latter (detection of ideological bias in news media) while collapsing ideology to one dimension”

• cites Budak et al., 2016 sourceNot checked — no full text of the source
• cites Kulkarni et al., 2018 sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Kiesel et al., 2019 sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

May combine two claims.

The dataset consists of 721 fully adjudicated annotated paragraphs from 175 news articles, covering an average of 7.86 articles per year.

“After 150 hours of annotation, we present a dataset of 721 fully adjudicated annotated paragraphs, from 175 news articles and covering an average of 7.86 articles per year (excerpts shown in Tables 1, 2, and 3).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Creating the dataset cost approximately 150 hours of manual multi-dimensional labeling.

“In total, creating this dataset cost ∼150 hours of manual multi-dimensional labeling.”

May combine two claims.

Inter-annotator agreement (Krippendorff's α) for 135 articles after training and before adjudication was economic 0.44, foreign 0.68, social 0.39.

“To assess the inter-annotator agreement of stage 1, we report Krippendorf's α (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39).”

• cites Hayes and Krippendorff, 2007 sourceNot checked — no full text of the source

May combine two claims.

The annotated articles originate from five US news outlets (Chicago Tribune, Christian Science Monitor, New York Times, Time Magazine, Wall Street Journal) related to the US Federal Budget from 1947-1975, covering center-left, center, and center-right spectrum.

“These articles originate from 5 news outlets related the US Federal Budget from 1947-1975 covering the center-left, center, center-right spectrum: Chicago Tribune (CT), Christian Science Monitor (CSM), the New York Times (NYT), Time Magazine (TM), and the Wall Street Journal (WSJ).”

• AssumptionRests on an assumption: Using proclaimed center, center-left, and center-right media outlets that claim to be objective allows focusing exclusively and more objectively on the ideology of the issue being discussed, without the subjectivity of author stance annotation.

Probably not a substantial claim: may not pass S3 (load-bearing).

After additional independent annotation of 25 articles, agreement was α = 0.53.

“Afterwards, 25 more articles were independently annotated and assessed with an α of 0.53.”

May combine two claims.

On both paragraph and article level, the economic dimension is often neutral, and this tends to co-occur with both liberal and conservative positions in other dimensions.

“Still, we see interesting nuances; for example, on both the paragraph and the article level, the economic dimension is often neutral, and this tends to co-occur with both liberal and conservative positions in other dimensions.”

May combine two claims.

The task of ideology prediction is distinct from, although correlated with, labels automatically derived from news outlets.

“exploring pretraining with existing data in news ideology/bias identification, we found that this task is distinct from, although correlated with, labels automatically derived from news outlets.”

May combine two claims.

At the paragraph level, co-occurrences of labels from different dimensions are less frequent than at the article level, and are more likely to be along the same side of ideology.

“Unlike the article-level, the co-occurrences are less frequent and we are more likely to observe co-occurrences along the same side of ideology.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The number of paragraphs per dimension across 175 articles is Economics 558, Social 291, Foreign 338.

“The number of paragraphs per dimension in total is: Economics (558), Social (291), Forign (338), across the 175 articles.”

Classified as probably not a substantial claim. May combine two claims.

The paper presents baseline models for ideology prediction, outlining a challenging task distinct from stance detection.

“We further present baseline models for ideology prediction, outlining a challenging task distinct from stance detection.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

On average, 21%-54% of sentences in a paragraph were highlighted by annotators as relevant to each dimension.

“On average, 21%-54% of the sentences in a paragraph were highlighted.”

May combine two claims.

All the models do better than randomly guessing, and better than predicting the majority class.

“It is important to note that all the models do better than randomly guessing, and better than predicting the majority class.”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Automatic segmentation with Topic Tiling resulted in articles with 1 to 6 paragraphs, average 4 paragraphs per article.

“The segmentation resulted in articles with 1 to 6 paragraphs.The average number of paragraphs per article was 4.”

May combine two claims.

In the sample, left-leaning news outlets were closest to their proclaimed ideological bias over time, whereas neutral outlets were more liberal before 1957 and after 1964, and right-leaning outlets were more conservative at that time than their proclaimed ideological bias.

“The figure shows that in our sample of articles, the left-leaning news outlets were closest to their proclaimed ideological bias measure over time, whereas the neutral outlets were more liberal before 1957 and after 1964.The rightleaning outlets were more conservative at that time than their proclaimed ideological bias.”

• AssumptionRests on an assumption: The news outlets' proclaimed ideological bias ratings from Adfontes, Allsides, and MBFC accurately reflect their bias.

May combine two claims.

Party affiliation was infrequent and was mostly associated with social issues.

“Finally, party affiliation was infrequent and was mostly associate with social issues.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After this curation, the total number of articles is 5,706 from the 5 outlets.

“After this curation, the total number of articles is 5,706 from the 5 outlets.”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

May combine two claims.

Task-guided pre-training improved the BERT (no fine-tuning) model for 1 of the 3 dimensions, but it led to worse performance than BERT (fine-tuned).

“Although task-guided pre-training improved the BERT (no fine-tuning) model for 1 of the 3 dimensions, it led to worse performance than BERT (fine-tuned).”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

Probably not a substantial claim: may not pass S3 (load-bearing).

Each article in the second annotation batch was annotated by 2 annotators.

“Then, the three annotators received the second batch and each article was annotated by 2 annotators.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Annotation was composed of two stages to account for possible subjectivity, with stage 2 involving adjudication by a third annotator as consolidator.

“This annotation was composed of two stages to account for possible subjectivity.”

• Also stated in: § 3 Data collection and annotation · plain

  “In stage 2, the two annotators examine paragraphs that they disagree, and met with the third annotator acting as consolidator to adjudicate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All dimensional labels were adjudicated after discussions between annotators.

“Although the agreements were not perfect and reflected a degree of subjectivity in this task, all dimensional labels were adjudicated after discussions between annotators.”

Classified as probably not a substantial claim. May combine two claims.

In Western politics, ideology is a multi-dimensional construct that includes equal opportunity versus economic individualism, respect for tradition/hierarchy/stability versus social change, and beliefs about the fairness/efficiency of markets.

“In Western politics, these worldviews translate into a multi-dimensional construct that includes: equal opportunity as opposed to economic individualism; general respect for tradition, hierarchy and stability as opposed to advocating for social change; and a belief in the un/fairness and in/efficiency of markets (Jost et al., 2009).”

• cites Jost et al., 2009 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

At least a bidimensional ideology distinguishing economic and social preferences is appropriate for domestic politics.

“Many political scientists and political psychologists argue for the use of at least a bidimensional ideology for domestic politics that distinguishes between economic and social preferences (Carsey and Layman, 2006;Carmines et al., 2012;Feldman and Johnston, 2014).”

• cites Carsey and Layman, 2006 sourceNot checked — no full text of the source
• cites Carmines et al., 2012 sourceNot checked — no full text of the source
• cites Feldman and Johnston, 2014 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The annotation manual was developed by a political scientist, and the data were annotated by three linguists after an elaborate training phase.

“Our extensive annotation manual is developed by a political scientist, and the data then annotated by three linguists after an elaborate training phase (Section 3).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The development and test sets contain articles uniformly distributed from 1947 to 1974 such that no particular decade is predominant.

“The development and test sets contain articles uniformly distributed from our time period (1947 to 1974) such that no particular decade is predominant.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The importance of taxation and defense is evident in conservative ideology in words such as tax, business, missile, and force.

“The importance of the topic of taxation and defense is evident in conservative ideology in words such as tax, business, missile, and force.”

May combine two claims.

Ideology varies frequently within an article, showing that a single article-level label will not be fine-grained enough to capture variances within an article.

“The figure indicates that ideology varies frequently within an article, showing that a single article-level label will not be fine-grained enough to capture variances within an article.”

Probably not a substantial claim: may not pass S2 (checkable).

The framework enables nuanced, quantitative analyses into polarization.

“the above analysis illustrates that our framework enables nuanced, quantitative analyses into polarization.”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

Probably not a substantial claim: may not pass S3 (load-bearing).

Individual congresspeople are more concerned with specific issues.

“This also is not surprising as it suggests that individual congresspeople are more concerned with specific issues.”

Classified as probably not a substantial claim.

All paragraphs belonging to the same article are present in a single split.

“To ensure the integrity of the modeling task, all paragraphs belonging to the same article are present in a single split.”

Classified as probably not a substantial claim.

The polarization conclusions are drawn only for the annotated articles.

“Hence, the analysis are demonstrations of what our framework enables, which we discuss at the end of this section, and the conclusions are drawn for our annotated articles only.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The importance of education and labor to liberals is evident in the economic and social dimensions in words like school, education, and wage.

“The importance of education and labor to liberals is also evident in the economic and social dimensions in words like school, education, and wage.”

May combine two claims.

The polarization analyses use only fully adjudicated data and refrain from model predictions because baseline experiments show predicting ideology is challenging.

“Limitations: We use only the fully adjudicated data and refrain from using model predictions, since our baseline experiments in Section 6 show that predicting ideology is challenging.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

A γ of 0 essentially turns a focal loss into a cross entropy loss, so it has less potential to hurt performance than to improve it.

“Since a γ of 0 essentially turns a focal loss into a cross entropy loss, it has less potential to hurt performance than to improve it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The conclusion applies only to the set of articles in the dataset.

“While this conclusion applies to only the set of articles in our dataset”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

Probably not a substantial claim: may not pass S3 (load-bearing).

Words like federal and Senator allude to the topic being at the federal level.

“For example, words like federal and Senator allude to the fact that the topic is at the federal level.”

May combine two claims.

There is a potential correlation to be exploited by the ideology of the news outlet, but such labels are not that informative for multi-dimensional prediction.

“The improvement on the no fine-tuning setting indicates that there is a potential correlation to be exploited by the ideology of the news outlet, but such labels are not that informative for multi-dimensional prediction.”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

Probably not a substantial claim: may not pass S3 (load-bearing).

Our corpus is relatively small.

“Further, our corpus is relatively small”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

We take the first step in studying multi-dimensional ideology and polarization over time and in news articles relying on the major political science theories and tools of computational linguistics.

“We take the first step in studying multi-dimensional ideology and polarization over time and in news articles relying on the major political science theories and tools of computational linguistics.”

Probably not a substantial claim: may not pass S4 (based).

We believe this period is fitting because it includes various ideological combinations of the tripartite composition of the American government, Congress and presidency.

“We believe this period is fitting because it includes various ideological combinations of the tripartite composition of the American government, Congress and presidency.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The classification tasks still remain challenging for neural models, leaving plenty of room for improvement in future work.

“However, the classification tasks still remain challenging for neural models, leaving plenty of room for improvement in future work.”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Because federal budget stories touch on all aspects of the federal activity, this topic appeals to both liberal and conservative media and thus can provide a good testing ground to showcase our proposed ideological annotation method.

“Because federal budget stories touch on all aspects of the federal activity, this topic appeals to both liberal and conservative media and thus can provide a good testing ground to showcase our proposed ideological annotation method.”

• AssumptionRests on an assumption: The LDA topic model with 50 topics yields well-deliminated topics.

Probably not a substantial claim: may not pass S3 (load-bearing).

The models are capturing some of the complex underlying phenomena in the data.

“This shows that the models are capturing some of the complex underlying phenomena in the data.”

• AssumptionRests on an assumption: The corpus annotations are reliable and accurate.

Classified as probably not a substantial claim. May combine two claims.

In addition to the core federal budget topic, we also include other topics such as health and education that are integral parts of ideological beliefs in the United States, and when discussed at the federal government level, are typically related to the federal budget.

“In addition to the core federal budget topic (topic 5 of Table 7), we also include other topics such as health and education that are integral parts of ideological beliefs in the United States, and when discussed at the federal government level, are typically related to the federal budget.”

• AssumptionRests on an assumption: The LDA topic model with 50 topics yields well-deliminated topics.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

We only focus on news whereas these dynamics might be different in other forms of communication such as social media posts or online conversations, and the legislature.

“We only focus on news whereas these dynamics might be different in other forms of communication such as social media posts or online conversations, and the legislature.”