EGU Blogs

Nature

A thought on impact factors

OK, bear with me on this one. It’s a bit of a thought dump, but it would be interesting to see what people think.

You can’t go anywhere in academia these days without hearing about impact factors. An impact factor is a metric assigned to a journal that measures the average number of citations per article over the preceding two year interval. It was originally designed to help libraries select which journals were being used by academics in their research, and therefore which ones they could not renew subscriptions to. However, in modern day academia, it is often used to measure the individual ‘impact’, or quality, of a single paper within a journal – that is, the metric assigned to a journal is used as a proxy for the value of each article inside. It doesn’t make much sense on the face of things, especially when you here stories about how much impact factors are gamed (read: purchased) by journals and their publishers (see link below), to the extent that they are at the least meaningless, and at the worst complete lies.

The evidence suggests that the only thing that an impact factor, and journal rank, is reflective of is academic malpractice – that is, fraud. The higher an impact factor, the higher the probability that there has been data fudging of some sort (or higher probability of detection of such practice). A rather appealing option seems to be to do away with journals altogether, and replace them with an architecture built within universities that basically removes all the negative aspects of assessment of impact factors, at the same time as removing power from profit-driven parasitic publishers. It’s not really too much a stretch of the imagination to do this – for example, Latin America already uses the SciELO platform to publish its research, and is free from the potential negative consequences of the impact factor. University College London also recently established it’s own open access press, the first of its kind in the UK. The Higher Education Funding Council for England (HEFCE) recently released a report about the role of metrics in higher education, finding that the impact factor was too often mis-used or ‘gamed’ by academics, and recommended its discontinuation as a measure of personal assessment. So there is a lot of evidence that we are moving away from a system where impact factors and commercial publishers are dominating the system (although see this post by Zen Faulkes).

But I think there might be a hidden aspect behind impact factors that has often been over-looked, and is difficult to measure. Hear me out.

Impact factors, whether we like it or not, are still used as a proxy for quality. Everyone equates a higher impact factor with a better piece of research. We do it automatically as scientists, irrespective of whether we’ve even read an article. How many times do you hear “Oh, you got an article in Nature – nice one!” I’m not really sure if this means well done for publishing good work, or well done for beating the system and getting published in a glamour magazine. Either way, this is natural now within academia, it’s ingrained into the system (and by system, I include people). The flip side of this is that researchers then, following this practice, submit their research which they perceive to be of ‘higher quality’ (irrespective of any subjective ties or a priori semblance of what this might mean) to higher impact factor journals. The inverse is also true – research which is perceived to be less useful in terms of results, or lower quality, will be sent to lower impact factor journals. Quality in this case can refer to any combination of things – strong conclusions, a good data set, relevance to the field.

Now, I’m not trying to defend the impact factor and it’s use as a personal measure for researchers. But what if there is some qualitative aspect of quality that it is capturing, based on this? Instead of thinking “It’s been published in this journal, therefore it’s high quality”, it’s rethinking it as “This research is high quality, therefore I’m going to submit it to this journal.” Researchers know journals well, and they submit to venues for numerous reasons – among them is the appropriateness of that venue based on its publishing history and subject matter. If a journal publishers hardcore quantitative research, large-scale meta-analyses and the sort, then it’s probably going to accrue more citations because it’s of more ‘use’ – more applicable to a wider range of subjects or projects.

For example, in my field, Palaeontology, research typically published in high impact factor journals involves fairly ground-breaking new studies regarding developmental biology, macroevolution, extinctions – large-scale patterns that offer great insight into the history of life on Earth. On the other hand, those published in lower impact factor journals might be more technical and specialist, or perhaps regarding descriptive taxonomy or systematics – naming of a new species, for example. An obvious exception to this is anything with feathers, which makes it’s way into Nature, irrespective of it’s actual value in progressing the field (I’ll give you a clue: no-one cares about new feathered dinosaurs any more. Get over it, Nature).

So I’ll leave with a question: do you submit to higher impact factor journals if you think your research is ‘better’ in some way. And following this, do you think that impact factors capture a qualitative aspect of research quality, that you don’t really get if you think about what impact factors mean in a post-publication context? Thoughts below! Feel free to smash this thought to shreds.

Plotting for the Earth. Sciences.

This was originally posted at: http://www.nature.com/scitable/blog/earthbound/plotting_for_the_earth_sciences

So a cool paper came out a while back about using plots when attempting to construct stories as a mode of communicating in Earth Science. I cannot, as always, emphasise my frustration when someone writes an article that’s supposed to be broadly educational, and sticks it behind a paywall. In this case, it might have reached the target audience of practising institutionalised Earth scientists (hello), but not the many who aren’t fortunate to have a subscription.

[Read More]

Social Media for Science Outreach – A Case Study: That social media thang

This was initially posted at: http://www.nature.com/spoton/2013/04/social-media-for-science-outreach-a-case-study-that-social-media-thang/ as part of a series of case studies exploring how academics use social media.

Jon began university life as a geologist, following this with a treacherous leap into the life sciences with a course in biodiversity and taxonomy. Now undertaking a PhD in tetrapod biodiversity and extinction at Imperial College London, there was a brief interlude were Jon was sucked into the world of science policy and communication. He blogs at https://blogs.egu.eu/network/palaeoblog/, tweets as Protohedgehog and co-runs an [infamous, probably] podcast series called Palaeocast. Jon can usually be found procrastinating in pubs, trying to exchange bad science, usually about dinosaurs, in exchange for food and beer.

Tell us a bit about you and your social media project

I’m currently a PhD student at Imperial College London, investigating the biodiversity patterns of tetrapods (anything with four limbs/wings/flippers) about 145 million years ago to see what we can figure out in a macroevolutionary sense, and whether we can find a ‘hidden’ mass extinction in the fossil record. I commit some of my time to 3 major social media platforms: blogging, tweeting, and podcasting, with a bit of Facebook, LinkedIn, Google+ and others on the side.  These activities are less of a project, per se, and more just stuff I do in parallel, and often with overlap, with my PhD research.

[Read More]

SpotOn London – a global conference

SpotOn London was held this weekend at the Wellcome Trust, shockingly, in London. The name stands for Science Policy, Outreach and Tools Online, with each of these representing three individual but strongly interwoven strands during the two days. As far as conferences go, it was pretty interactive. Each session was live-streamed, and through that and the power of Twitter –many people in each session either had an ipad, laptop, or mobile phone out tracking the online conversation, and drawing in additional comments from those who couldn’t be there in person.

It was also nice to briefly meet Barbara and Edvard of the EGU network (*waves at*). Apart from the small geo-cadre, the rest of the delegates represented in total the entire scientific discipline, from PhD students to BBC science reporters, communications managers, and those who work in science policy. When you consider science as being an interactive process, pretty much everyone present can be classed as some brand of scientist, representing the huge social range of the field. That’s intended as a compliment to any attendees who read this!

Typical attendee at the conference

So, why as a PhD student and palaeontologist, did I go to this conference? The primary reason is that I was actually invited to co-co-ordinate a session with Michelle Brook, the Head of Policy at the Physiological Society. As well as this, the ‘science communication network’, consisting rather loosely of those who do various outreach initiatives, work in science policy, reporting, editing, publishing, are an insanely valuable resource for personal development as an early career scientist. I only ‘joined’, or became aware of, this community around a year ago, and since then have begun to learn about issues such as gender bias in science, the open access movement, the role of science in government, the role of learned societies, as well as the value of blogging, and various other forms of communication, among other things. This has largely been fueled by being pretty active on Twitter, which is a pretty awesome way of staying in touch with the world beyond the desk/lab, and made a whole lot easier by living in London, which is undoubtedly the UK’s hub of science communication-relevant events, people, and organisations (ranging from CaSE to BioMed Central’s publishing house, and everything in between).

So yeah, the session. We decided to strike for the pretty hot topic of increasing engagement between scientists and policymakers. The panel consisted of Nic Bilham, Head of External Relations and Strategy at the Geological Society (and my old boss, who taught me everything I know about geoscience policy), Julian Huppert, the MP for Cambridge, and strong supporter of science within government, and Anna Zecharia, a post-doc at Imperial College, Head of the Science Communication Forum there, and someone with pretty big ideas for science communication and policy in the future. Michelle chaired the session. You can see their discussion here:

As well as this, we also organised a workshop to follow on from this discussion. If you want to see what any of this was like on Twitter, check out the #solo12sp hash tag. The video feed for this session was a bit fuzzy, seeing as how we had several discussions happening at the same time, based on the format idea of a ‘marketplace of discussion’, with the three panelists fueling debate and conversation by acting as focal areas of expertise. The idea is to take the notes and ideas made during this session, and collate them into some sort of guide document that scientists and policymakers can use to increase reciprocal engagement between the two. That’s not to say the two ‘sides’ are mutually distinct, and that interaction doesn’t happen already. It’s just that there could be more of it, and if some sort of strategic framework were established whereby communications between the two fields increased, for the benefit of both, then it would be mutually beneficial in terms of having more evidence-informed policies, where needed, and more policy-aware scientists. As far as how geology/geoscience fits in to this, I think, and I’m sure many would agree, that in times of issues such as climate change, energy security, natural hazards, radioactive waste, the more geoscientific input we have, the more informed we can be when it comes to tackling these. Where does palaeontology come in to all this? I’ll let you know when I’ve figured that one out.

For sticking with it, here’s an image of the dinosaur Tenontosaurus tilletti that I played with a couple of years ago.