The United Nations 2030 agenda for sustainable development launched in 2015 and introduced 17 sustainable development goals (SDGs) (Sachs 2012; UN 2015). Ever since then, Universities have been interested in evaluating how their research relates to these goals (for examples see Al-Raeei, 2023; Nature, 2021; Pakkan, et al., 2023; Repiso, et al., 2023). In University of Helsinki we are developing ways of monitoring and analysing the share of publications related to sustainable development.

Several analytical tools for tracking SDG contributions have cropped up in recent years (see e.g., Borchardt, et al., 2022; Confraria, et al., 2024; Hajikhani and Suominen, 2022; Our World in Data team, 2023; Rodenburg, et al., 2021). Major citation databases such as Scopus, Web of Science and Dimensions now come with SDG-information. However, these databases tend to use different methods for relating publications under various SDGs.

Scopus based analytical database SciVal uses AI-enhanced keyword searches. There have been many iterations on this approach. The first published search queries were presented in 2019 and contained relatively simple search strings (Jayabalasingham, et al., 2019). The 2021 queries saw the introduction of machine learning techniques (Rivest, et al., 2021). 2022 and 2023 queries introduced further refinements (Bedard-Vallee, et al., 2023; Roberge, et al., 2022).

Web of Science based analytical database InCites uses citation-based clusters (Garcia, 2022). There is a set of initial keyword searches, which are then used to identify micro citation topics. If a citation topic is strongly associated with an SDG, then all publications in that citation topic will be attributed to that SDG. This method was originally introduced by the STRINGS project (see Confraria et al., 2024 for details).

The Dimensions database uses curated machine learning approaches (Wastl, et al., 2020). Curated data was fed to a machine learning algorithm to generate an automatic classification model. This approach also incorporated natural language processing and a quality control process.

The problem is that different approaches tend to yield different results (Armitage, et al., 2020; Purnell, 2022; Raman, et al., 2023). All classification contains an interpretative element, but since SDGs are related to important policy goals and come with the promise of numerical indicators, this case of ambiguity is more troublesome than most. SDGs are also very general in their scope and do not benefit from unifying mechanisms such as having a long historical tradition like traditional disciplines. The connection between SDG-research and SDG-related practical solutions is also strained at best (Ludwig, et al., 2022). This lack of an external cohesion makes it very difficult to evaluate the different SDG-mapping methods. In fact, it has been suggested that one should not assume there is some correct mapping method and instead just interpret the different methods as merely different perspectives (Rafols, et al., 2021).

We wanted to find out what the differences in SDG-mapping methods look like at the level of a local institution. Our target institution, our university, has a very broad range of research fields and a well-maintained current research information system. The University of Helsinki has roughly 31,000 degree students, 4,500 teaching and research staff, around 9,000 peer-reviewed publications per year and faculties in agriculture and forestry, biological sciences, humanities, education sciences, medicine, natural sciences and mathematics, law, pharmacy, social sciences, theology and veterinary sciences. Our institution allows for a broad evaluation of SDG-attributions over a range of research topics with accurate coverage information. The more local focus also allows us to cover all SDG and see firsthand the kind of anomalies that can show up on a more restricted scale.

To find out our SDG distributions, we used our Pure based Current Research Information System (CRIS), Tuhat. Scientific publications in our CRIS are manually verified and researchers are required to submit their publications to Tuhat. For this reason, publication information in our CRIS is highly reliable and has high coverage.

We gathered all peer-reviewed scientific publications from the University of Helsinki between the years 2020-2022. This included articles, review articles, conference articles, peer-reviewed notes and letters as well as books and edited volumes. Our university also published 1,452 dissertations during 2020-2022. Only 15 of them had a Digital Object Identifier (DOI). Dimension contained 13 of them, while InCites and SciVal contained none. Of those 13 dissertations, only one had an SDG- attribution. Dissertations were therefore not included.

In total, there were 26,639 publications. The publications were collected in April 2023.

22,563 publications (85%) had a DOI. The publications were found in the Dimensions, InCites and SciVal databases using their respective DOIs. In InCites and Scival, the databases DOI importing feature was used. For Dimensions, we performed direct DOI-searches. InCites found 18,761 (83%) of these publications, SciVal 20,169 (89%) and Dimensions 21,365 (95%) respectively. We exported these DOI-based datasets from the databases to get the SDG-attribution information. That information was then compared on an article-to-article level. Overlaps were calculated through simple Excel functions and the formulas manually checked to eliminate possible errors.

SciVal and InCites also come with ready-made SDG-information based on affiliate information, which can be used to quickly gather SDG-information on a particular research organisation. When comparing different research organisations, this is likely the most straightforward way to proceed. However, we noticed that using such affiliation data can lead to varying results. In our testing,

SciVal found between 25%-72% of our DOI-based publication SDG-counts while InCites found between 96%-125%. So, for SciVal, a significant number of SDG-publications was left out if pure affiliation information was used instead of validated DOIs. For InCites, affiliation information gave a sizeable number of additional publications, which may not actually belong to our research organisation.

Use of additional identifiers such as WOS ID can be used to improve the coverage of the publications sets obtained from the databases.

However, this requires careful manual checking of the results and tends to result in very few additional publications for the case of our university. This is because we have a very well- maintained CRIS and hence the missing publications tend to either have wrong metadata or lack proper identifiers (DOI).

Furthermore, identifiers from all the three databases have not been comprehensibly collected into our CRIS. Use of database related identifiers could then introduce additional bias. We believe that using pure DOIs makes the analysis fairer and more transparent.

For each SDG, we determined the total number of publications in each database and overlaps between each of the three SDG-attributions. There are two factors that affect the overlaps: 1) different SDG-attributions and 2) differences in database coverage. To demonstrate both factors, we created two sets of SDG values. The first set contains all SDG-publications from each database. The second set contains only those SDG-publications that are covered by all three databases (15,786 publications, marked as SDG*).

The total numbers of SDG-publications were the following:

There are rather large variances between different SDG-counts. InCites has by far the highest number of University of Helsinki publications with associated SDGs, while Dimensions tends to have far fewer. For individual SDGs, InCites tends to have the highest number of articles, but there are also SDGs in which InCites has the least number of attributed publications. Dimensions has the most SDG-attributed publications for SDGs 4 and 7, but this seems to be caused by Dimensions having better overall coverage.

There are a total of seventeen SDGs, but we only considered the first sixteen. SciVal had 388 publications attributed to SDG 17, but Dimensions only had 4 and InCites had zero. Based on these numbers, SDG 17 was omitted from the analysis.

Restricting the analysis to SDG-publication that are covered by all three databases, we find that InCites and SciVal SDG-publications can be moderately affected, but Dimensions can have rather drastic reductions in their SDG-totals (over 50%). This can be due to Dimensions having better overall coverage, while also having the least total number of SDG-attributed publications.

SDG-overlaps in terms of total numbers were the following:

Of note is that, due to low overall SDG-publication counts, Dimensions occasionally has very few SDG-publications that are not similarly SDG-attributed in at least one of the two other databases. This is particularly true for SDGs 1, 5, 6 and 9 and especially if we restrict to publications that are covered by all three databases. The exceptions seem to be SDG 4, 7, 14 and 16.

The proportional overlaps were:

An immediate observation is that InCites and SciVal have rather different SDG-attributions. At best, Scival can match only 40% of InCites SDG-publications and at worst it only matches 5%. SciVal is a bit better matched by InCites but InCites also has a lot more SDG-attributed publications in total. Dimensions is matched reasonably well by both InCites and SciVal but Dimensions itself does not cover either InCites or SciVal that well. This is probably doe to Dimensions just having less overall SDG-publications. SDG-publications matched by all databases are low across the board, signalling that SDG-attributions are rather inconsistent.

We can also look at individual SDGs to get a better understanding of the various overlaps. We constructed Venn diagrams for the overlaps and coloured the bubbles based on database thematic colours (Dimensions = blue, InCites = violet and SciVal = orange). The Venn diagram on the left contains all publications covered by individual databases and the Venn diagram on the right contains only publications that are covered by all three databases.

SDG 1: No Poverty.

The first SDG 1 has low overlap between InCites and SciVal (10% and 15%). It is even a bit lower when we account for coverage differences (8% and 14%). Dimensions has far fewer publications attributed to this SDG than the other two databases and they mostly overlap with SDG- publications in SciVal (54%/88%). Nevertheless, even here the overlap is not complete.

SDG 2: Zero Hunger.

SDG 2 has a quite clear difference between InCites and Scival (19% and 35%/39%), but SciVal and dimensions have average overlaps (26%/28% and 54%/61%). InCites covers Dimensions rather well (44%/54%) but InCites also has a lot more SDG 2-attributed publications (659/611 compared to 167/137).

SDG 3: Good Health and Well-being.

SDG 3 has by far the most overall publications (InCites: 9425/8495, SciVal: 4512/3794 and Dimensions: 3167/2717). This is because SDG 3 is of a very general nature, covering human health and welfare and therefore very closely connected to the field of medicine. Our medical faculty has high publication counts averaging around 3000 peer-reviewed publications yearly with high coverage in international publication databases. High SDG 3 publication counts were thus to be expected. InCites has a very broad take on SDG 3 and it ends up covering both SciVal and Dimensions here. However, SciVal and Dimensions only cover at most 40% of InCites publications. Due to the high publication counts, database differences here can lead to very different overall SDG counts.

SDG 4: Quality Education.

SDG 4 is evenly spread out. Each database has their own take on quality education. Overlaps are low to average (41%/47% to 23%/27%). It is worth noting that education related publications are not that well covered in InCites or Scival. Dimensions has better coverage and consequently loses proportionally more publications when we consider only publications found in all three databases. Better coverage of education research could yield more consistent results.

SDG 5: Gender Equality.

SDG 5 has a particularly small overlap between InCites and Scival (5% and 18%/23%). SciVal and Dimensiosn have higher overlaps (50%/87% and 17%/21%). InCites also has a lot more SDG 5- attributed publications than SciVal or Dimensions. A cursory study of the publications indicates that the underlying reason seems to be that InCites attributes a lot of medical research to SDG 5 where as SciVal and Dimensions are more focused on social sciences.

SDG 6: Clean Water and Sanitation.

InCites and SciVal have rather similar publications numbers, but their overlap is low (22% and 15%/20%). Dimensions has few publications in comparison and publications overlap with both InCites and SciVal evenly.

SDG 7: Affordable and Clean Energy.

For SDG 7, the publication counts are similar. The overlap between Dimensions and Scival (38%/40% and 36%/42%) is a bit higher than overlaps with InCites (24%/26% and 15%/17% for SciVal and 28%/32% and 17%/22% for Dimensions).

SDG 8: Decent Work and Economic Growth.

SDG 8 has low overlaps with InCites. SciVal has the most SDG-publications. Interestingly, 54% of Dimensions SDG-publications are not covered by the other databases. Dimensions has decent overlap with SciVal, but it also has far fewer SDG-attributed publications.

SDG 9: Industry, Innovation and Infrastructure.

SDG 9 is another example of a very low overlap between InCites and Scival (8%/11% and 6%/7%). Dimensions has average overlaps with both InCites and SciVal, but with a very few publications (15/9). Only 64% of InCites SDG-attributed publications are also found in SciVal and Dimensions.

SDG 10: Reduced Inequalities.

SDG 10 has rather low overlaps with InCites, but Dimensions and SciVal have average to good overlaps.

SDG 11: Sustainable Cities and Communities.

SDG 11 has a big difference between InCites and Scival (19%/23% and 40%/46%), but SciVal covers Dimensions rather well (66%/80%). The publications counts are also rather different for the three databases.

SDG 12: Responsible Consumption and Production.

SDG 12 has very low overlaps between InCites and Scival. Dimensions has better overlaps, but it also has less a lot less publications. Total SDG-publication numbers are also rather different.

SDG 13: Climate Action.

SDG 13 has rather high overlap between SciVal and Dimensions. They also have very similar publication counts. InCites, however, has a much broader conception of Climate Action. InCites covers SciVal and Dimensions well, but this is probably due to a much larger SDG-publication count.

SDG 14: Life Below Water.

SDG 14 has a fairly even spread and overlaps. The publication counts are also very similar. However, the overlaps are at most average (46%).

SDG 15: Life on Land.

SDG 15 has fairly good overlaps between Dimensions and SciVal (53%/75% and 54%/59%). InCites has a lot more SDG-publications signalling a wider take on the SDG. Consequently, the overlaps for InCites are low. Publication coverage is rather similar across the databases.

SDG 16: Peace, Justice and Strong Institutions.

SDG 16 is unusual in the sense that unlike most SDGs, InCites has the least amount of SDG attributed publications. InCites SDG-publications also do not overlap here that well with SciVal (34%/37% and 8%/9%) and Dimensions (32%/44% and 8%/16%)). SciVal and Dimensions also seem to have their own take on the SDG and only have 24%-28% and 28%-44% overlaps.

In general, we can observe that there is a lot of variance at the level of individual SDGs between the different SDG-mapping methods. Overlaps varied but tended to be small to at most average. SDG counts also varied with InCites having the most overall SDG-attributed publications followed by SciVal and Dimensions.

SDG-overlaps are higher when taking only publications that are found in all three databases. However, this might give a misleading impression for a regular use case since SDG-numbers are not typically filtered from multiple databases. Actual variance in SDG-counts between databases also contains difference in coverage.

During the analysis, we also noticed that SDG-attributions can go through rather dramatic changes. Before the InCites 2023 April update, InCites had a total of 7,888 SDG-publications for our university. After the update, that number rose to 15,109 SDG-publications, which is a 91% increase. Similarly, SciVal SDG-attributions have gone through many updates with large effects on total SDG numbers. For Dimensions, we do not have comparative data.

It comes as a no surprise that different databases give different SDG-attributions. The databases use different approaches, contain different metadata and have different coverages. There is also an interpretive element in attributing SDGs and interpretation is never an exact activity. As Armitage et al. (2020) has already demonstrated, different SDG-attribution approaches can make a big difference. However, we were still surprised as to the extent of the differences.

Another problem we encounter was that single version updates could have drastic effects on SDG- counts. This meant that the attributions were also highly unstable. This instability might get alleviated in the future once the approaches bed down. There is a reason to be a bit sceptical though. Many of the commonly used classifications in bibliometrics have a more or less direct relation to scientific practice. Classificatory terms such as keywords are shaped by their use and utility in scientific practice. Scientific fields have related organisations, journals and teaching. Research topics and clusters have citation links. SDGs, on the other hand, have no such unifying mechanisms apart from the UN descriptions, which demonstrably admit very differing interpretations. Furthermore, as science progresses and society evolves, our understanding of what research relates to which SDGs is prone to change and this should be reflected in the mappings. The need for interpretation is constant and that makes constancy hard to come by.

It should be pointed out that, in the absence of shared external standards, it is not possible to evaluate which of the SDG-classification schemes are superior (Turunen et al. 2023). If it would so happen that all the approaches would coincide, we could say that the particular choice of a classification scheme is mostly irrelevant. This could have some practical benefits, and possible detriments, but we would still not be able to say that we now have the correct SDG-attributions. This is because we would still lack the external criteria we need for evaluating whether we indeed have the correct attributions. After all, it is not just about getting the bubbles to overlap, but about what we want them to overlap for.

SDG-attributions are supposed to tell us about how research relates to sustainable development, a crucial endeavour, but their source dependence and instability makes them unsuitable for use in indicators and evaluation. At worst, use of SDG-based indicators in evaluation can actively misdirect science policy. This is especially important for the societal impact of science as SDGs are related to sustainable development.

This is not to say that SDG-classification don’t have their uses for universities. In fact, there are already a myriad of studies that utilise various SDG-classification for non-evaluative purposes. For example, Aswathy et al. (2023) used keyword co-occurrence mappings between different SDGs and between the topics of a singular SDG, Meschede (2020) looked at author country affiliation co-occurrence clusters for SDG-publications, Payumo et al. (2021) looked at repeated collaboration in SDG-related research and Roy et al. (2022) performed a deep dive into SDG6. Care must be taken to ensure that one is not merely dealing with artefacts due to a choice of a particular classification scheme. However, if we go beyond rigid evaluation, we can take SDG-classifications more as perspectives (see Rafols, et al., 2021) that can highlight important aspects of sustainable development. From this perspective, the differences become a boon.