SUSTAIN-2: Impact study of the European Social Survey
Publication date: 14 March 2022 | Report language: EN
This report presents the findings of the second impact study of the European Social Survey (ESS) ERIC. It is a follow-up to the original impact study of the ESS, which was conducted in 2016/17.
This study was commissioned by the ESS ERIC and conducted by Technopolis, with bibliometric analysis from the centre for Science and Technology Studies (CWTS) at the University of Leiden. The study was funded under the European Commission Horizon2020 grant SUSTAIN-2 (reference 871063).
The purpose of this study is to identify the academic, non-academic and teaching impacts of the ESS. We note below our headline conclusions in brief.
User numbers and trends
- As of June 2021, there were 182,778 registered ESS users. This means that since the start of the original impact study (June 2016) the number of registered ESS users has almost doubled
- The ESS user base has grown consistently by around 14-15% each of the last five years and its composition has largely stabilised to around two thirds students, one quarter academics (faculty/research or PhD), and just under 10% other (typically non-academic) user types
- Indicators of ESS use-intensity are promising. In the year up to June 2021, 74% registered users had downloaded ESS data. The proportion of downloaders has grown steadily, increasing by 5% since the original impact study in 2016
- Whilst there is substantial ‘churn’ in the overall user numbers, there is also evidence of much repeat-use. Of the roughly 50,000 non-student ESS users registered by 2020, 7,712 had logged in and downloaded data with the calendar year. We estimate that around 3,500 of these were repeat-users, who registered in a previous year and have since returned
- Current ESS member/observer countries generally have among the highest numbers of ESS users, although some current guest countries or former participating countries also feature in the ‘top-30’, as well as three fully non-European countries: the USA, Canada and China
- There is substantial variation on how fast the user bases have grown in different countries during this period. Interestingly, the highest rates of growth have occurred in countries fully outside of Europe (China and Chile). Other high-growth countries include former participants or current guest countries (Russia, Spain, Denmark). In short, while this study largely focuses on core ESS members, there are signs of growing ESS use much further afield
Teaching impacts and institutional hotspots
The ESS is being used increasingly widely as a teaching tool, featuring across participating countries and in an increasing number of institutions for both methodological and thematic courses.
While overall ESS user numbers are strongly driven by users registering as students, the ‘true’ number of students using the ESS is likely higher than the ESS user data suggests, as many lecturers may download ESS data and convert the data into materials for their students, who themselves never actually register with the ESS. Nevertheless, ESS user data provide a useful proxy to identify teaching hotspots:
- In 2016, there were 18 institutions with more than 500 confirmed users (including four with over 1,000). By the start of 2021, this increased to 42 (including 15 institutions with over 1,000 confirmed users). Many cluster around the Benelux countries, England and western Germany. However, there are at least some such user hotspots in most European regions
- The user base at several of the current top-30 has grown by between 60% and 100% compared with 2016 – roughly in line with the overall growth of the ESS user-base. However, there are also several institutions where the number of ESS users has increased at a rate far and above what would be expected. There is therefore a clear sense that new hotspots of high ESS-use rapidly emerge (indicating widespread use for teaching at those institutions), whilst established hotspots continue to grow at steady rates
Academic impacts
The ESS continues to be viewed positively for its exceptionally high methodological standards. Even in direct contrast to other national and international social values and attitudes surveys, the ESS continues to be viewed as a gold standard.
Various measures we have taken suggest that the overall volume of ESS-based publication output has increased by at least 150% since the first ESS impact study of 2016/17. Including various different publication types and non-English language publications, data collected by staff at the University of Ljubljana suggest that there are over 7,500 ESS-based publications in existence (equivalent figure in the first impact study was 2,704). CWTS was able to identify 2,448 ESS-based items listed on Web of Science, which compares to around 1,000 in a similar analysis conducted in 2016.
Drawing on the 2,448 ESS-based items listed on Web of Science, bibliometric analysis performed by CWTS yields the following findings:
- The citation impact of ESS publications is well above average, being about 70% more highly cited than average, with 21% of all ESS publications belonging to the top 10%. The journals in which work is being published have a citation impact of 40% above the world average. The citation impact seems to be highest around 2008, gradually decreasing in more recent times, but still staying well above average
- Whilst output volume and citation metrics are generally highest in western European countries with especially strong research systems and, within those, in the most prestigious universities, ESS-based work performs well (i.e. above average) on citation metrics when we variously adjust for field, year, institution or journal
- The ESS is mostly used by ESS member countries, producing almost three quarters of all ESS publications. The United Kingdom, Germany and the Netherlands are the largest producers of ESS publications with respectively 436, 337 and 292 publications. All three countries show a fairly high impact, roughly twice as high as average. The largest producer of ESS publications outside of the ESS member countries is the USA, with 315 publications, and has a very high impact, of almost three times the average
- The three largest producers are the Katholieke Universiteit Leuven, University of Oxford and the University of Amsterdam with respectively 119, 75 and 69 publications. The impact at Oxford is especially high, being over three times higher than average. These three institutions show a high impact also in general in the social science and humanities (SSH), but the impact of ESS publications is higher still. Indeed, for all institutions the impact of their ESS publications is about 1-3 times higher than the impact of their SSH publications
- Most ESS-based publications are being published in the WoS subject categories of Sociology and Political Science representing about 40% of all ESS publications. Economics is the third largest subject category, representing only about 6% of all publications. The impact is higher in Political Science than in Sociology. The USA is most active in Political Science while Germany and the Netherlands are most active in Sociology
- Using a more granular field classification consisting of 4,140 so-called micro-fields, we find that ESS publications are dispersed across quite a number of different micro-fields. The four largest micro-fields concern voter turnout, social capital, the welfare states and social values, together representing about one third of all publications. The number of publications in smaller micro-fields tapers off gradually. Additional topics that emerge from text-mining titles and abstracts from ESS publications relate to data gathering and measurements, (im)migration, labour, education, family composition and health, including also gender aspects
Non-academic impacts
Our research could not identify a meaningful way of quantifying the non-academic impacts of a research infrastructure such as the ESS. However, our country-level research yielded many examples of non-academic impacts of many different types and across different domains. These take many different shapes, including
- General intelligence and insight for NGOs or government ministries, agencies or advisory bodies
- Agenda setting by using ESS data to highlight a particular problem or challenge, triggering various types of policy action
- Influence on public debate or highlighting certain issues to the general public through presentation of ESS data or ESS-based findings in the news media
- Monitoring, i.e. using ESS data as indicators to track certain aspects of societal progress, e.g. to help assess whether certain policies are achieving the desired outcomes
To better understand outreach and the ESS’s presence on social media, we conducted a social media analysis, which yielded the following main findings:
- The performance of the ESS’ own social media accounts is overall good compared to similar organisations. The average engagement rate of the official ESS Twitter account over the analysed period was 1.02%, which means that 10.2 ESS Twitter followers out of 1,000 engaged with an ESS tweet on average (a Twitter engagement rate between 0.33% and 1% is generally considered to be high)
- Academia and researchers are the most active communities online when it comes to the ESS. This is in line with the objectives of the ESS and with its intended target groups. Overall, the ESS’ main online audience tends to be in the UK. This is also somewhat reflected in the UK’s comparatively high share of non-academic ESS users
- The social media analysis revealed that there are areas which resonate very significantly with online audiences. The ESS data on the public’s attitude towards the EU membership is one of them
