EGU Blogs

higher education

A thought on impact factors

OK, bear with me on this one. It’s a bit of a thought dump, but it would be interesting to see what people think.

You can’t go anywhere in academia these days without hearing about impact factors. An impact factor is a metric assigned to a journal that measures the average number of citations per article over the preceding two year interval. It was originally designed to help libraries select which journals were being used by academics in their research, and therefore which ones they could not renew subscriptions to. However, in modern day academia, it is often used to measure the individual ‘impact’, or quality, of a single paper within a journal – that is, the metric assigned to a journal is used as a proxy for the value of each article inside. It doesn’t make much sense on the face of things, especially when you here stories about how much impact factors are gamed (read: purchased) by journals and their publishers (see link below), to the extent that they are at the least meaningless, and at the worst complete lies.

The evidence suggests that the only thing that an impact factor, and journal rank, is reflective of is academic malpractice – that is, fraud. The higher an impact factor, the higher the probability that there has been data fudging of some sort (or higher probability of detection of such practice). A rather appealing option seems to be to do away with journals altogether, and replace them with an architecture built within universities that basically removes all the negative aspects of assessment of impact factors, at the same time as removing power from profit-driven parasitic publishers. It’s not really too much a stretch of the imagination to do this – for example, Latin America already uses the SciELO platform to publish its research, and is free from the potential negative consequences of the impact factor. University College London also recently established it’s own open access press, the first of its kind in the UK. The Higher Education Funding Council for England (HEFCE) recently released a report about the role of metrics in higher education, finding that the impact factor was too often mis-used or ‘gamed’ by academics, and recommended its discontinuation as a measure of personal assessment. So there is a lot of evidence that we are moving away from a system where impact factors and commercial publishers are dominating the system (although see this post by Zen Faulkes).

But I think there might be a hidden aspect behind impact factors that has often been over-looked, and is difficult to measure. Hear me out.

Impact factors, whether we like it or not, are still used as a proxy for quality. Everyone equates a higher impact factor with a better piece of research. We do it automatically as scientists, irrespective of whether we’ve even read an article. How many times do you hear “Oh, you got an article in Nature – nice one!” I’m not really sure if this means well done for publishing good work, or well done for beating the system and getting published in a glamour magazine. Either way, this is natural now within academia, it’s ingrained into the system (and by system, I include people). The flip side of this is that researchers then, following this practice, submit their research which they perceive to be of ‘higher quality’ (irrespective of any subjective ties or a priori semblance of what this might mean) to higher impact factor journals. The inverse is also true – research which is perceived to be less useful in terms of results, or lower quality, will be sent to lower impact factor journals. Quality in this case can refer to any combination of things – strong conclusions, a good data set, relevance to the field.

Now, I’m not trying to defend the impact factor and it’s use as a personal measure for researchers. But what if there is some qualitative aspect of quality that it is capturing, based on this? Instead of thinking “It’s been published in this journal, therefore it’s high quality”, it’s rethinking it as “This research is high quality, therefore I’m going to submit it to this journal.” Researchers know journals well, and they submit to venues for numerous reasons – among them is the appropriateness of that venue based on its publishing history and subject matter. If a journal publishers hardcore quantitative research, large-scale meta-analyses and the sort, then it’s probably going to accrue more citations because it’s of more ‘use’ – more applicable to a wider range of subjects or projects.

For example, in my field, Palaeontology, research typically published in high impact factor journals involves fairly ground-breaking new studies regarding developmental biology, macroevolution, extinctions – large-scale patterns that offer great insight into the history of life on Earth. On the other hand, those published in lower impact factor journals might be more technical and specialist, or perhaps regarding descriptive taxonomy or systematics – naming of a new species, for example. An obvious exception to this is anything with feathers, which makes it’s way into Nature, irrespective of it’s actual value in progressing the field (I’ll give you a clue: no-one cares about new feathered dinosaurs any more. Get over it, Nature).

So I’ll leave with a question: do you submit to higher impact factor journals if you think your research is ‘better’ in some way. And following this, do you think that impact factors capture a qualitative aspect of research quality, that you don’t really get if you think about what impact factors mean in a post-publication context? Thoughts below! Feel free to smash this thought to shreds.

Week 3, and the rising of a new dawn

It hit me. As I stared in to the depths of the ~500 or so papers I’d carefully curated in Mendeley, the gravity of a PhD came down like a tonne of dinosaur bones. This is big. Even simply in terms of background reading, there was so much to do it would probably take a year just to get through it. It was time for a pondering and a pint, and a reassessment of strategy.

Week 3 in the Big Brother House wasn’t all speculative despair though. This Palaeontologist has been [relatively] busy! Between the deciphering of all compendiums on mass extinctions, the sourcing of new material to add to the never-diminishing pile of ‘stuff I have to read’, quite a lot has been going on! There’s always something going on in London.

Despite being thrust back in to academia, I’m still maintaining a pretty active interest in policy, largely with regards to education and science. So naturally, when a new ‘Policy Lunchbox‘ was announced looking into research careers, I was there. I think it’s important that, even if students aren’t aware of specific policies governing higher education, they should at least be aware of what people are saying, and any changes that might be lurking on the horizon. Hosted by the Brisitsh Ecological Society, Society for Experimental Biology, and Biochemical Society, this seemed like a good chance to see the direction that future of academic careers were heading. And have a free lunch. Imagine the anguish, then, when asides from one member of the Panel, I was the only actual academic to turn up, in an audience of 40-50, and one of I think only 4 males in the audience (naturally, the Panel consisted of 4 middle-aged white men). That was odd. Why weren’t there more researchers here to hear about what people had to say about their future? Slightly bewildering. I’d encourage academics, particularly those just starting out on the path, to get involved more in events and discussions of this nature. It’s important to venture outside of university once in a while to see what’s happening in these external but highly relevant spheres.

Turns out there is such a thing!

The Society of Vertebrate Palaeontology held their annual meeting this week in Raleigh, North Carolina. A swarm of hammer-wielding loonies, with various animals hides adorning their sun-beaten torsos descended on the unsuspecting city, and by the sounds of it, had a damn good time! For twitterers, the hash tag feed #2012SVP has plenty of 140-character long gems, and Bora Zivkovic has blogged the event.

Official logo, of what appears to be a crocodile noshing a leaf. Not sure about that, but then, I wasn’t at the conference!

Bora is also the co-host of an annual event called Science Online, also held in North Carolina. Tickets for this illustrious event have already been gobbled up by a hungry mob of scientists, reporters, journalists and science communicators of every breed. NESCent have procured a couple of tickets however, and are giving them away as a prize in a blog competition along with a travel subsidy, with the theme focused on evolution. I’ve submitted, and with a bit of luck, will be heading over early next year! This is about the time when C3-PO tells me how slim the odds of this occurring are.

Damn straight.

ORCID is a new tool for identifying and connecting with researchers. It seems like any other networking tool atm, but is quite young so might be worth signing up for and following any developments. Like any shiny new online toy, I signed up, naturally.

London also have an equivalent to the Science Online in NC, called Science Online London. The event series has been re-branded as SpotON, but the premise is still the same: a bunch of people who love science getting together to discuss how to boost it forward, with themes such as science communication and outreach, online and digital tools, and science policy. I’m co-coordinating a couple of sessions in the science policy strand for this year’s event. Stay tuned for more details! Again, I think it’s quite important that young researchers grasp these opportunities when they’re presented. I haven’t even presented orally at a conference before, and now, will have helped organise a session on increasing reciprocal engagement between policy makers and scientists, in the hope that we can take steps forward in evidence-informed policy. Neat eh!

It was also Earth Science Week last week! If you didn’t know, it’s not too late, but the whole idea was that you give a geologist a beer and a hug. It’s not too late! The real concept was to increase public engagement with the geosciences, and help foster an appreciation of the geosphere, geodiversity, and geoconservation. Note, you can add the prefix geo- to most words to make them georelevant. There was an event on at the Science Museum in London called ‘Science on a Sphere’, where a series of specially-designed short films about various aspects of geoscience were three-dimensionally projected onto a large sphere suspended in mid-air. The films themselves were pretty cool, albeit largely themed around meteorology, strangely, and the visuals were amazing. Unfortunately, there must have been a communications meltdown, as none of the people there for the half hour showings realised that the footage was anything beyond the usual material projected there. This was a pity, as pretty much everyone stayed for about a minute to watch, took a photo, and moved on, instead of enjoying the whole experience. Did anyone else manage to do anything for Earth Science Week this year (UK or elsewhere)? Dave, the creator and co-runner of Palaeocast managed to get a blog post out of it for the Geological Society – good publicity for our project!

Earth Science Week is supposed to get people of all ages engaged with geoscience! Maybe next year.. (2013, not what this image says)

A final point of mention, is that the Royal Veterinary College held their first RVC Lates this week, as part of Biology Week (which strangely coincides with Earth Science Week). There were some cool engagement activities there, including a highly suspect strawberry DNA extraction experiment (mush it up, add washing up liquid and 100% ethanol, and extract the DNA. I thought it was more complex than this..), evolution of the the V-formation in avian flight, zebrafish embryology. The piece to resistance was surely the dissection. A pony was uncovered and butchered by one of the professional anatomists of the RVC, much to the delight and horror of the audience members. My one memory will be the smell. And the blood. And the artificially breathing lungs. It was awesome, and I hope they do it again in the future! No photos unfortunately, as there were warnings about photos being mus-interpreted by certain parties should they be distributed. I also got to meet awesome Palaeo-artist Sam Barnett (@Palaeosam), who demonstrated some skill at sketching the various anatomical elements on display (including the only surviving image of the poor pony!)

This is pretty much what it looked like. That stuff on top is DNA apparently!

So yeah, a busy week generally for this Palaeontologist. London gives you the chance to uncover so many aspects of science, largely for free, and it’s so worth embracing them. You get to meet great new people, and learn a lot, and it makes a nice break from the current information-pump lifestyle of background reading (not that I’m complaining – my project is pretty awesome!). My final words will just be a note of encouragement to PhD students to get out there, break beyond the barriers of your research project and explore science and the other networks and opportunities out there!

Science!