Exploring Siri’s Content Diversity Using a Crowdsourced Audit
DOI:
https://doi.org/10.33621/jdsr.v4i1.115Keywords:
Siri, content diversity, crowdsourced audit, voice assistants, US politics, politically controversial issues, algorithmic bias, search algorithms, search results, long tail distribution, concentration, fragmentationAbstract
This study aims to describe the content diversity of Siri’s search results in the polarized context of US politics. To do so, a crowdsourced audit was conducted. A diverse sample of 170 US-based Siri users between the ages of 18-64 performed five identical queries about politically controversial issues. The data were analyzed using the concept of algorithmic bias. The results suggest that Siri’s search algorithm produces a long tail distribution of search results: Forty-two percent of the participants received the six most frequent answers, while 22% of the users received unique answers. These statistics indicate that Siri’s search algorithm causes moderate concentration and low fragmentation. The age and, surprisingly, the political orientation of users, do not seem to be driving either concentration or fragmentation. However, the users' gender and location appear to cause low concentration.
References
Agichtein, E., Brill, E., Dumais, S., & Ragno, R. (2006). Learning user interaction models for predicting web search result preferences. SIGIR ’06: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 3–10. https://doi.org/10.1145/1148170.1148175
Anderson, C. (2016). Assembling publics, assembling routines, assembling values: Journalistic self-conception and the crisis in journalism. In J. Alexander, E. Breese, & M. Luengo (Eds.), The Crisis of Journalism Reconsidered: Democratic Culture, Professional Codes, Digital Future (pp. 153-169). Cambridge University Press. https://doi.org/10.1017/CBO9781316050774.010
Barker, R. (2018). Trapped in the Filter Bubble? Exploring the Influence of Google Search on the Creative Process. Journal of Interactive Advertising, 18(2), 85–95. https://doi.org/10.1080/15252019.2018.1487810
Bechmann, A., & Nielbo, K. L. (2018). Are We Exposed to the Same “News” in the News Feed? An empirical analysis of filter bubbles as information similarity for Danish Facebook users. Digital Journalism, 6(8), 990–1002. https://doi.org/10.1080/21670811.2018.1510741
Beer, D. (2009). Power through the algorithm? Participatory web cultures and the technological unconscious. New Media & Society, 11(6), 985–1002. https://doi.org/10.1177/1461444809336551
Benson, R. (2013). Shaping immigration news: A French-American comparison. Cambridge University Press. https://doi.org/10.1017/CBO9781139034326
Blaikie, N., & Priest, J. (2019). Designing Social Research. Polity Press.
Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in Neural Information Processing Systems 29, 4349–4357.
Boothby, H. (2018, October 15). Radio Production Autumn 2018 [Lecture recording]. Canvas@Malmö University. https://mau.se/canvas/
Bozdag, E. (2013). Bias in algorithmic filtering and personalization. Ethics and Information Technology, 15(3), 209–227. https://doi.org/10.1007/s10676-013-9321-6
Bruns, A. (2019). Filter bubble. Internet Policy Review, 8(4), 1–14. https://doi.org/10.14763/2019.4.1426
Bucher, T. (2017). The algorithmic imaginary: exploring the ordinary affects of Facebook algorithms. Information, Communication & Society, 20(1), 30–44. https://doi.org/10.1080/1369118x.2016.1154086
Chams, Z. (2020, October 15). Republicans vs Democrats: Where do the two main US political parties stand on key issues?. ABC News. https://www.abc.net.au/news/2020-10-15/us-election-political-parties-explained-democrats-vs-republicans/12708296
Cho, J., Ahmed, S., Hilbert, M., Liu, B., & Luu, J. (2020). Do Search Algorithms Endanger Democracy? An Experimental Investigation of Algorithm Effects on Political Polarization. Journal of Broadcasting & Electronic Media, 64(2), 150–172. https://doi.org/10.1080/08838151.2020.1757365
Curtois, C., Slechten, L., & Coenen, L. (2018). Challenging Google Search filter bubbles in social and political information: Disconforming evidence from a digital methods case study. Telematics Informatics, 35(7), 2006–2015. https://doi.org/10.1016/j.tele.2018.07.004
Dambanemuya, H. J., & Diakopoulos, N. (2020, March 20-21). “Alexa, what is going on with the impeachment?” Evaluating smart speakers for news quality [Paper presentation]. Computation and Journalism Symposium, Boston, MA, USA. https://cj2020.northeastern.edu/research-papers/
Dellinger, A. J. (2019, April 30). Survey says that Siri and Google Assistant are the most used voice assistants. Digitaltrends. https://www.digitaltrends.com/home/siri-google-asistant-most-used-voice-assistants-alexa/
Diakopoulos, N. (2015). Algorithmic Accountability. Digital Journalism, 3(3), 398–415. https://doi.org/10.1080/21670811.2014.976411
Diakopoulos, N., Trielli, D., Stark, J., & Mussenden, S. (2018). I vote for – How search informs our choice of candidate. In M. Moore & T. Tambini (Eds.), Digital Dominance: The Power of Google, Amazon, Facebook, and Apple (pp. 320–341). Oxford University Press.
Dorfman, R. (1979). A Formula for the Gini Coefficient. The Review of Economics and Statistics, 61(1), 146–149. https://doi.org/10.2307/1924845
Ekstrand, M. D., Burke, R., & Diaz, F. (2019). Fairness and discrimination in retrieval and recommendation. RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems, 576-577. https://doi.org/10.1145/3298689.3346964
Fabris, A., Purpura, A., Silvello, G., & Susto, G. A. (2020). Gender stereotype reinforcement: Measuring the gender bias conveyed by ranking algorithms. Information Processing & Management, 57(6), 102377. https://doi.org/10.1016/j.ipm.2020.102377
Feezell, J. T., Wagner, J. K., & Conroy, M. (2021). Exploring the effects of algorithm-driven news sources on political behavior and polarization. Computers in Human Behavior, 116, 1–11. https://doi.org/10.1016/j.chb.2020.106626
Fletcher, R., & Nielsen, R. K. (2017). Are News Audiences Increasingly Fragmented? A Cross-National Comparative Analysis of Cross-Platform News Audience Fragmentation and Duplication. Journal of Communication, 67(4), 476–498. https://doi.org/10.1111/jcom.12315
Fletcher, R., Cornia, A., & Nielsen, R. K. (2020). How Polarized Are Online and Offline News Audiences? A Comparative Analysis of Twelve Countries. The International Journal of Press/Politics, 25(2), 169–196. https://doi.org/10.1177/1940161219892768
Friedman, B., & Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems, 14(3), 330–347. https://doi.org/10.1145/230538.230561
Geschke, D., Lorenz, J., & Holtz, P. (2019). The triple-filter bubble: Using agent- based modelling to test a meta-theoretical framework for the emergence of filter bubbles and echo chambers. British Journal of Social Psychology, 58(1), 129–149. https://doi.org/10.1111/bjso.12286
Gillespie, T. (2014). The relevance of algorithms. In: T. Gillespie. T., P. Boczkowski & K. Foot (Eds.), Media Technologies: Essays on Communication, Materiality, and Society (pp. 167–194). MIT Press. https://doi.org/10.7551/mitpress/9780262525374.003.0009
Gillespie, T. (2015). Platforms Intervene. Social Media & Society, 1–2. https://doi.org/10.1177/2056305115580479
Gillespie, T. (2017). Algorithmically recognizable: Santorum’s Google problem, and Google’s Santorum problem. Information, Communication & Society, 20(1), 63–90. https://doi.org/10.1080/1369118X.2016.1199721
Gitlin, T. (1998). Public sphere or public sphericules? In J. Curran & T. Liebes, (Eds.). Media, ritual and identity (pp. 168–174). Routledge. https://doi.org/10.4324/9780203019122
Gutmann, A., & Thompson, D. F. (2004). Why Deliberative Democracy?. Princeton University Press.
Haim, M., Graefe, A., & Brosius, H. (2018). Burst of the Filter Bubble?. Digital Journalism, 6(3), 330–343. https://doi.org/10.1080/21670811.2017.1338145
Introna, L. D., & Nissenbaum, H. (2000). Shaping the web: Why the politics of search engines matters. The Information Society, 16(3), 169–185. https://doi.org/10.1080/01972240050133634
Jurkowitz, M., Mitchell, A., Shearer, E., & Walker, M. (2020). U.S. Media Polarization and the 2020 Election: A Nation Divided: Deep partisan divisions exist in the news sources Americans trust, distrust and rely on. Pew Research Center. https://www.journalism.org/2020/01/24/u-s- media-polarization-and-the-2020-election-a-nation-divided/
Just, N., & Latzer, M. (2017). Governance by algorithms: reality construction by algorithmic selection on the Internet. Media, Culture & Society, 39(2), 238–258. https://doi.org/10.1177/0163443716643157
Kay, M., Matuszek, C., & Munson, S. A. (2015). Unequal representation and gender stereotypes in image search results for occupations. CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 3819–3828. https://doi.org/10.1145/2702123.2702520
Kinsella, B. (2020, November 5). Voice Assistant Use on Smartphones Rise, Siri Maintains Top Spot for Total Users in the U.S.. Voicebot. https://voicebot.ai/2020/11/05/voice-assistant-use-on-smartphones-rise-siri-maintains-top-spot-for-total-users-in-the-u-s/
Kitchin, R. (2017). Thinking critically about and researching algorithms. Information, Communication & Society, 20(1), 14–29. https://doi.org/10.1080/1369118X.2016.1154087
Krafft, T. D., Gamer, M., & Zweig, K. A. (2019). What did you see? Personalization, regionalization and the question of the filter bubble in Google's search engine. EPJ Data Science, 8(38), 1–23. https://doi.org/10.1140/epjds/s13688-019-0217-5
Lambert, P. J. (2001). The Distribution and Redistribution of Income. Manchester University Press.
Lehn J., Müller-Gronbach T., & Rettig S. (2000). Konzentrationsmaße. In J. Lehn, T. Müller-Gronbach & S. Rettig (Eds.), Einführung in die Deskriptive Statistik. Teubner Studienbücher Mathematik (pp. 50-64). Vieweg & Teubner Verlag. https://doi.org/10.1007/978-3-322-80099-2_3
McQuail, D. (1992). Media performance: Mass communication and the public interest. Sage.
Milanovic, B. (1997). A simple way to calculate the Gini coefficient, and some implications. Economics Letters, 56(1), 45-49. https://doi.org/10.1016/S0165-1765(97)00101-8
Najle, M., & Jones, R. P. (2019, February 19). American Democracy in Crisis: The Fate of Pluralism in a Divided Nation. PRRI. https://www.prri.org/research/american-democracy-in-crisis-the-fate-of-pluralism-in-a-divided-nation/
Natale, S., & Cooke, H. (2021). Browsing with Alexa: Interrogating the impact of voice assistants as web interfaces. Media, Culture and Society, 43(6), 1000–1016. https://doi.org/10.1177/0163443720983295
Nechusthai, E., & Lewis, S. C. (2019). What kind of news gatekeepers do we want machines to be? Filter bubbles, fragmentation, and the normative dimensions of algorithmic recommendations. Computers in Human Behavior, 90, 298–307. https://doi.org/10.1016/j.chb.2018.07.043
Olson, K. (2011). Deliberative democracy. In B. Fultner (Ed.), Jürgen Habermas: Key Concepts (pp. 140–155). Acumen Publishing. https://doi.org/10.1017/UPO9781844654741.008
Otterbacher, J., Bates, J., & Clough, P. D. (2017). Competent Men and Warm Women: Gender Stereotypes and Backlash in Image Search Results. CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 6620–6631. https://doi.org/10.1145/3025453.3025727
Pariser, E. (2011). The Filter Bubble: What the Internet Is Hiding from You. Penguin Press.
Pasquale, F. (2015). The Black Box Society: The Secret Algorithms that Control Money and Information. Harvard University Press.
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018). Auditing Partisan Audience Bias within Google Search. Proceedings of the ACM on Human-Computer Interaction, 2, 1–22. https://doi.org/10.1145/3274417
Sandvig, C., Hamilton, K., Karahalios, K., & Langbort, C. (2014, May 22). Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. [Paper presentation]. 64th Annual Meeting of the International Communication Association. Seattle, WA, USA.
Seaver, N. (2013). Knowing Algorithms. Media in Transition, 8, 412–422.
Sunstein, C. R. (2002). Republic.com. Princeton University Press.
Trielli, D., & Diakopoulos, N. (2019). Search as News Curator: The Role of Google in Shaping Attention to News Information. CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–15. https://doi.org/10.1145/3290605.3300683
Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76(2), 105–110. https://doi.org/10.1037/h0031322
Valishery, L. S. (2021, January 22). Share of voice assistant users in the U.S. 2020 by device. Statista. https://www.statista.com/statistics/1171363/share-of-voice-assistant-users-in-the-us-by-device/
Vike-Freiberga, V., Däubler-Gmelin, H., Hammersley, B., & Pessoa Maduro, L. M. P. (2013). A free and pluralistic media to sustain European democracy. High Level Group on Media Freedom and Pluralism. http://ec.europa.eu/digital-agenda/sites/digital-agenda/files/HLG%20Final%20Report.pdf
Wagner, P. (2018, June 29). Siri Remains The Most Used Mobile Voice Assistant. Statista. https://www.statista.com/chart/14505/market-share-of-voice-assistants-in-the-us/
Webster, G. J., & Ksiazek, T. B. (2012). The Dynamics of Audience Fragmentation: Public Attention in an Age of Digital Media. Journal of Communication, 62(1), 39–56. https://doi.org/10.1111/j.1460-2466.2011.01616.x
Yitzhaki S., & Schechtman E. (2013). More Than a Dozen Alternative Ways of Spelling Gini. In S. Yitzhaki & E. Schechtman (Eds.), The Gini Methodology: A Primer on a Statistical Methodology (pp. 11–31). Springer. https://doi.org/10.1007/978-1-4614-4720-7_2
Zuiderveen Borgesius, F. J., Trilling, D., Möller, J., Bodó, B., de Vreese, C. H., & Helberger, N. (2016). Should we worry about filter bubbles? Internet Policy Review, 5(1), 1–16. https://doi.org/10.14763/2016.1.401
Downloads
Published
Issue
Section
License
Copyright (c) 2021 The Author
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
