Abstract

Keywords

Ranking; law journals; Department of Higher Education; bibliometric data; citations; quality.

……………………………………………………….

1 Introduction: A changing research landscape

There is something fundamentally absurd about the idea of ranking research. At the same time, no one can seriously argue that all research is equal in importance and quality. Either way, we are doubtless witnessing a dramatic change in the management and organisation of research. One aspect of the change is a move towards the ranking of research.1

The assessment of the quality of research in higher education is a relatively new international phenomenon.2 Academics are required to publish more, to publish faster,3 and to demonstrate the quality and significance of their research to their employers and state funders.4 This trend is prevalent in South Africa as well, and is part of a larger corporatisation movement to introduce managerial mechanisms. Initially developed to measure performance in profit-making enterprises, quality assessment was extended into academia to "improve efficiency and economy"5 at universities and to advance capacity, quality and innovation.6 The merits of this trend and its potential impact on academic freedom are excluded from this discussion.7

In developing any national research policy:

It … is worth considering three fundamental issues: first, whether there is a need to assess the quality of research outputs and, if so, whether it is better to control the assessment process centrally or devolve the process to individual institutions; second, if research assessment is deemed useful, then

what form it should take; third, whether outcomes should be explicitly linked to the distribution of research funding.8

The need for the quality assessment of research output is recognised and accepted in South Africa as well as internationally, and a form of such assessment is already in use.9 Research assessment is seen as useful and necessary, and is currently practised in the form of rewarding all contributions to accredited peer-reviewed journals in the form of the government "publication output subsidy".10 Over the past decade or so public universities have successfully encouraged their academics to publish peer-reviewed11 research in accredited journals in line with this policy, even though the actual number of academics has remained fairly stable.12 Official opinion will have it that although the policy has resulted in increased output, it has not necessarily led to an improvement in quality.13

Academic performance measurement generally leads to a growth in measurable output but seldom results in a higher quality of the research or direct reallocation of funds to the best performers.14

Both the 2013 White Paper for Post-School Education and Training15 and particularly the 2015 DHET Research Output Policy16 indicate that a different government approach is to be adopted in an attempt to reward better quality and higher impact peer-reviewed research.17

Leaving aside the problematic definitions of "quality" and "impact",18 the question may well be asked why the specific focus on quality and impact is

currently so prevalent. Should not the fact that an article has been peer-reviewed and published in an accredited journal be a sufficient indication that the publication meets the set minimum quality standards?19 After all, ASSAf does engage in random expert post-publication evaluations of accredited South African law journals. The ASSAf review of law journals in 2014 resulted in a finding that all (but one) of the accredited law journals on the DHET list were in line with the DHET policies and had met the minimum standards vis-à-vis inter alia the peer-review process.20 Thus, all these accredited journals had been found, in principle, to be publishing appropriate research, in line with the DHET policy as "original, systematic investigation undertaken to gain new knowledge and understanding".21

Although the journals may meet the minimum criteria, it has become evident that peer-reviewed published research is not always of uniform standard.

While peer-review is one of the most fundamental indicators of the quality of a research journal, the way it is applied is what reflects the journal's standards and indicates the overall quality of the research presented in its pages.22

The variation in the standard of published research could be ascribed to numerous factors such as the diverse peer-review processes employed by editors, differences in the assessment standards used by peer-reviewers, and/or possibly the proliferation of accredited peer-reviewed journals recognised by the DHET, each with its own selection nuances. 23 Another factor that could negatively influence the standards of output is the increase

in the number of predatory publishers and journals, some of which also appear on accredited lists. These seem to focus on profit rather than on an adherence to strict peer-review quality standards, making it easier for authors to get published.24

In addition, there is a negative perception of the quality of South African journals.25 Because of the small pool of academics in any specific discipline, it is inevitable that the number of journal submissions as well as of peer reviewers will be limited – casting doubt over the independence of the peer-review system.26

The negative perceptions about quality prompted the DHET to amend its research assessment policy as from 2016 to ensure that it received value for its subsidy investment in academic research.27 In terms of this policy, the DHET will continue to determine the quality of research output by proxy.28 However, from 2016 the quality assessment of research output may include additional mechanisms such as biometric/bibliometric data (including citations),29 discipline-specific panels of experts and post-publication reviews by the DHET.30 The DHET "may consider introducing such measures as the categorising of journals as 'high' or 'low' impact; citation indexes or other relevant and appropriate quality measurements as prerequisites, after due and extensive consultation process with this sector."31 No formal consultation process with legal academia on this issue has yet begun at the time of the writing of this article, but certain universities

are already differentiating between legal journals on national and international accredited lists.32

It must be stated that whatever form any additional research assessment takes, it could never be truly objective.33 Then, the additional burdens of the cost of the implementation of such a system and the inconvenience to individuals and universities by diverting attention to non-core business issues must also be reckoned with.34

The task facing the South African legal fraternity is to agree on additional quality improvement measures that would generally be regarded as suitable and that could be implemented successfully as a means of determining the impact or quality (or perceptions of the impact or quality) of the South African legal research output, or as a means of distinguishing between the exceptional and the average. The consequences of non-engagement may result in the DHET determining measure(s) that may or may not be appropriate for legal academics.

When one considers the terms "high" and "low" quality and "impact", it stands to reason that a ranking system, identifying the better quality journals could be introduced. 35 However, such a system has not been universally acceptable for all legal jurisdictions.

In the USA, for instance, quality is assessed through a direct journal ranking system as opposed to an assessment of individual articles.36 This system is well developed with an extensive published discourse about the types of journal ranking systems employed. More about these will form the basis of the discussion in the latter part of this article.

Belgium and Australia have experimented with ranking systems, albeit not too successfully.37 Van Gestel notes that in Belgium the 2004 ranking list

was so controversial that it was never implemented and in Australia the 2010 formal journal rankings were abandoned by 2011.38 The Netherlands and the UK have not even attempted to implement a ranking system.39 The Netherlands uses "qualitative reviews by panels of international experts for its external reviews" to assess research outputs.40 The UK implemented an external post-publication evaluation process where selected individual articles submitted by universities are assessed on merit.41 The 2014 UK Law sub-committee confirmed the view "that peer review remains the most reliable method of assessing research quality in law".42 In the UK the external evaluation process of selected research output submissions, rather than the ranking of journals, serves as quality-control.43 In 2001 and again in 2008 the law panel in the Research Assessment Exercise (England) concluded (my emphasis):

Work of internationally-recognised excellence was found in a wide range of types of output and places, and in both sole and jointly authored works …. First-rate articles were found in both well-known journals and relatively little-known ones. Conversely, not all the submitted pieces that had been published in ‘prestigious’ journals were judged to be of international excellence. These two points reinforced the Panel's view that it would not be safe to determine

the quality of research output on the basis of the place in which they have been published or whether the journal was ‘refereed’.44

It is submitted that should the DHET in South Africa implement a system of differentiating among journals based on their "low" and "high" impact, whether by a panel of experts and/or the use of bibliometric data and/or through other methods, it is conceivable that a journal ranking system would follow, either officially or unofficially, unless a suitable alternative could be found.

2 Journal ranking systems

It has been argued that a journal ranking system could serve to increase the quality and impact of scholarship and create incentives for journal editors to select and publish only quality submissions; which, in turn, would motivate academics to strive to produce work of higher quality.45 However, this is not always the case as "ground-breaking 'must read' articles are as likely to be published in less prestigious journals as those held in particular high regard".46 Research has shown that a ranking system may "stifle diversity and innovation", as journal editors may prefer to publish mainstream articles to increase their rankings rather than new and experimental research.47 An unintended consequence (also of incentivising editors to publish only high quality outputs) may be a chilling effect on young academics, wary of rejection, or, where those young academics do submit outputs, editors may reject them. In the light of the above observation regarding the relatively small pool of persons working in legal fields in South Africa, this could be catastrophic going forward. And this would be even more devastating in respect of the project of transforming academia in respect of the development of black academics.

Academics could benefit from a ranking system as it could act as a guide to their choice of journal.48 Publication in higher-ranking journals would afford prestige, as it would signal potential superior quality,49 which could lead to

favourable outcomes regarding promotions and career paths.50 The reputations of law schools would benefit51 if their journals obtained high-ranking status.52 Higher-ranked journals in turn could benefit other stakeholders: they would be more widely purchased,53 read and cited, with accompanying benefits, as serious scholars are likely to prefer making use of more prestigious journals.54 From a journal editor's perspective, it could also translate into potential reviewers approached for review being more likely or inclined to review submissions.

For funders and managers, exceptional quality could be rewarded and promoted.55 For the journals, a negative change in ranking may give rise to self-evaluation and reflection.56

If journal rankings become established and respected in the legal and academic community, they can have a significant effect on the content of legal scholarship produced nationwide. This conclusion suggests that attempts to rank journals are extremely significant to the scholarly enterprise …57

According to Perry58 a ranking system should adhere to certain minimum requirements.59 First, it should be based on quality-sensitive criteria.60 This could be problematic because, as mentioned above, one journal may contain both excellent and mediocre articles.61 Secondly, the ranking methodology must be sensitive to changes in quality and must make allowance for regular revision and updating.62 Thirdly, the ranking must be based on objective criteria, free from bias,63 and practical, with enough

available data to fulfil the goals of the ranking.64 The data should be readily verifiable and not susceptible to manipulation.65 For a ranking system to be successful, it should thus be "carefully designed. If the ranking method is not defensible, then the resultant ranking will not fulfil its goals".66 These requirements will be used hereunder to evaluate the ranking systems discussed.

It has been acknowledged that a multi-factor combination ranking rather than a single factor system is preferred,67 although using a combination of factors may be simply "too burdensome" – especially where a single factor data system has not yet been collected and coded.68 The use of a multi-factor method is, however, not beyond criticism, as the person determining the ranking has:

… to determine how the different factors should be combined to generate the ultimate ranking. The weight that is assigned to each factor is crucial, and since this determination is purely subjective (and most likely controversial), a complex ranking method can [also] never be objective.69

The question arises whether it is possible to achieve a successful ranking system in South Africa, given the limited number of law journals published in the country, including those of a highly specialised nature, and taking into account the limited number of legal academics in the country.

The ASSAf Report on Grouped Peer Review of Scholarly Journals in Law notes the following special considerations for the evaluation of law journals.70 One: legal content is more locally-orientated than other disciplines, as the legal principles under scrutiny are mostly jurisdiction-specific.71 Two: the limited number of active academics in a specific sub-

area impacts on the number of possible submissions72 and qualified peer-reviewers. Three: the type of research is generally not primary research or empirical in nature73 and can be carried out with minimal resources, unlike other types of research that require extensive funding.74 Four: a premium is placed on the writing of books and chapters in books, resulting in the fact that not all legal scholarship can be found in journal articles.75 And five: research can be focused on applied legal practice, where research academics are agents for legal changes or developments.76 Law is always a "discipline in transition" where quality and impact should not rely solely on historical accuracy and data.77 Legal commentary is aimed at the legal profession and a case note suggesting an alternative approach may have a profound effect on the law that more "lengthy and academic papers published in prestigious journals do not have".78 Similarly, textbooks clarifying a complex legal issue in an accessible manner may "appear trivial to researchers from other disciplines [but] are in fact highly valuable and more sophisticated than they may seem at first glance".79

Such work, [law review articles] … has earned the real respect of the bench. We admire the law review for its scholarship, its accuracy, and, above all, for its excruciating fairness. We are well aware that the review takes very seriously its role as judge of judges – and to that, we say, more power to you. By your criticisms, your views, your appraising cases, your tracing the trends, you render the making of 'new' law a little easier. In a real sense, you thus help to keep our system of law an open one, ever ready to keep pace with the changing patterns.80

Taking the above into consideration, the aim of this article is to focus on the consequences of a possible law journal ranking system for South Africa in determining perceived quality and impact. The author will assess how the existing journal information for the period from 2009 to 2014 would have been evaluated, had the various US-type journal ranking systems been

imposed. The article includes direct and indirect quality evaluations.81 The direct evaluation consists of the ASSAf expert panel evaluation. Indirect evaluations through biometric/bibliometric data are based on whether the journal is featured in internationally accredited lists, the size of its print run, author prominence, the rejection rate, usage studies based on the perusal of library and electronic databases, and evaluations based on citations – in other journals as well as by the courts. An additional and uniquely South African ranking system is also considered, based on the five best outputs submitted to the National Research Foundation by applicants applying for an NRF rating.

In this article the principles, advantages and disadvantages, and the outcome of the principles applied (with some nuances) to the South African law journals will be discussed. Finally, the spread of the rankings per journal and an average of all the rankings are shown, flawed as they may be. The article concludes with an assessment of whether the application of the various systems resulted in a consistent ranking outcome or whether the results showed a marked difference in ranking depending on the ranking system used. This information could form the basis of a more informed decision about the viability of ranking systems for law journals in South Africa or whether an alternate evaluation system by the DHET is called for.

Of the twenty-three peer-review law journals that met the ASSAf minimum criteria for accreditation, twenty-one will be considered for this article.82 It should be noted from the outset that the immediate problem was the "lack of reliable, comparable and comprehensive data",83 and this article should therefore be treated as exploratory – as a starting point for a debate about the quality and impact evaluation of South African law journals. It is not intended to be comprehensive and neither the parts nor the whole is without fault or beyond criticism.84 The various aspects could and should be improved upon by additional research and debate.85 That said, the author

expects to make a useful contribution by exposing the dangers of a ranking system and the need for engagement with the DHET.

3 Direct quality and impact assessment through expert panels

In an ideal world, direct quality evaluation for all law journals would be performed by a panel of experts who regularly evaluate journal contributions objectively, according to prescribed criteria.86 However, it is unrealistic to expect academics who are over-burdened as it is and not expert in all areas of the law to find the time to devote to additional and continuous quality peer reviewing.87 Even where sub-specialisation panels are utilised and the number of specialised journals is limited, the process would remain time-consuming, subjective and therefore problematic.88 The logical conclusion would be that academics would evaluate their own articles as well as those of their peers, making the system inherently subjective, biased89 and potentially "self-perpetuating".90

[Peer-review, w]hile this is an important step in the right direction … can, nevertheless give unreliable assessments. Studies have shown peer reviews can produce inconsistent results …91

The process of ranking journals is complex and even experts might find it challenging to evaluate the varying quality of different journals.92

The South African ASSAf panels mentioned above have been active for some time and periodically assess sample journals through the ASSAf Committee on Scholarly Publishing in South Africa to determine whether, post-publication, journals still meet the minimum set criteria for inclusion in

the accreditation lists.93 This process is not a journal ranking system and it does not designate "low" or "high" impact status. For now, it provides the best available data for the purposes of this paper.

South African law journals were assessed by the ASSAf in 2014 and the results were published in their Report on Grouped Peer Review of Scholarly Journals in Law. The evaluation of the journals was based on the best practices set out in the ASSAf Editorial Process-related Criteria,94 which are aimed at eventually promoting available quality open-access online research.95 As mentioned earlier, the criteria used by the panels included editorial process-related criteria set out in the Code of Best Practices,96 as

well as business-related criteria97 and bibliometric assessments.98

Although the process does not rank journals, the panels do comment on the quality of the contributions in their individual assessments. Descriptions of the quality of the journals are used in this article to determine if and/or how the journals could/should be ranked. They were, however, not standardised and objective, and as such neither very accurate nor very helpful.99 For instance, would a journal described as "high" quality be the same as "generally high"? Is there a difference in quality between being "good overall" and "generally good"? If the first is inconsistent and of varied quality, and the other included "very good contributions", does that mean that the latter should be rated higher? Where the evaluating panel described a journal as "generally good" would that imply that some contributions were poorer and others better?

If the ranking of law journals should become inevitable, these panels may be in the best position to carry out such a task, although a truly accurate ranking may remain elusive and subjective.

For the ranking of journals based on the available information, flawed and subjective as it may be, five different groups are identified – a sort of scale of perceived quality as expressed by the ASSAf Report:

The SALJ referees were unanimous that the SALJ publishes articles of a high quality.100 In fact, it noted that the SALJ is "South Africa's premier law journal".101

The second group included the Annual Survey and the ILJ, which were regarded as examples of "the best work done in South Africa".102

The SAJHR,103 the SAYIL104 and Fundamina105 fell into the third group, being described as of a "high" quality without reservations.

The fourth group, including Stell LR, LDD, TSAR, JJS, SACJ, Obiter, THRHR and CILSA, was described as publishing good material, subject to a qualification. Articles in Stell LR106 received special mention and the quality was labelled "generally high". LDD107 articles were noted to be "generally very good". The quality of the articles in TSAR108 was described as "good overall" and those in JJS,109 SACJ110 and Obiter111 as "generally good". The contributions in the THRHR112 were also described as "high", although there was concern that some articles seemed to be primarily descriptive and to have very little theoretical content. The CILSA publications were described as of a "high quality", but concerns were raised about the lack of variety and about not keeping pace with changes in the area.113

The fifth group consisted of journals that were regarded as publishing articles of varying quality, but as being nonetheless worthy of accreditation. This group includes AHRLJ, De Jure, PER, SA Public Law, Acta Juridica, Merc LJ and Speculum Juris. The Report noted that in AHRLJ114 the quality of the contributions varied between and within issues, with some being excellent whilst others are average, but most were judged as being "good". The articles in De Jure,115 PER,116 SA Public Law 117 and Acta Juridica118 were described as varied, ranging from adequate, acceptable or average to

good. SA Public Law occasionally had "very good contributions",119 and the Merc LJ was "generally good with some exceptions".120 Speculum Juris121 contributions were described as "a mixture of more academic and more practical articles [that] compares well with general national law journals in Europe, America and the UK".

In conclusion, it is submitted that the information generated through the ASSAf Report does not meet the Perry minimum standards for a trustworthy and acceptable ranking system, mainly because it is neither objective nor free from bias, but also as it is impractical and not easily verifiable. A ranking based on the five groups was nonetheless included in the summary of the data under 5 hereunder.

As an aside, this article would not be complete without mentioning that an alternative method of determining the perceived quality of journals exists in the form of perception-based questionnaires or surveys similar to those of the Crespi122 and Campbell, Goodacre and Little123 studies. However, this system is also controversial because of its subjectivity and the fact that discretionary viewpoints cannot be standardised.124 In addition, the respondents may not be equally familiar with all of the journals125 and research has shown that "geographical origin, research orientation and affiliation with a journal" play a significant role in the assessment made by of the respondent.126 As far as the outcome of these studies is concerned, there may be some consensus about who should make the list, but the ranking of the journals remains unclear.127 As no South African academics or other role players have taken part in such surveys between 2009 and 2014, none could be included in this article.128

4 Indirect quality evaluation by bibliometric data

Not everything that counts can be counted, and not everything that can be counted counts.129

4.1 Introduction

Although there has been a rise in the use of bibliometric indicators in legal scholarship, it is not yet regarded as being on a par with or as effective as the expert peer review process.130 This type of data is also biased in favour of English publications and older legal sub-disciplines.131 Van Gestel and Vranken132 ask the following question:

What problems are bibliometric research indicators really meant to solve? … The purpose seems to have shifted [from furthering the scholarly quality of individual publications] towards creating an instrument for oversight, management and policy, which is just as ineffective in guaranteeing a lasting high quality of scholarly publications as substantive assessment by peers. Implementing both systems cumulatively would only add to the burden on the time and efforts of researchers to justify their work, leaving less time for research and education.

Bibliometric data serve the purpose of gauging a journal from another perspective to get an indication of the productivity and depth of impact amongst discipline peers. It is regarded as being more objective, as it circumvents the "old boy's network", is cheaper and more transparent.133 The systematic use of bibliometric data is, by its very nature, rooted in history. It assesses the past as a possible indication of future performance, but excludes new discoveries, new researchers and new universities.134

Obtaining reliable data is problematic135 because law journals themselves do not always present a full picture of academic endeavour. As mentioned

earlier and discussed later, books and chapters in books are also important sources of information in the legal field.136

With the available information, the following eight bibliometric systems were considered for this article: inclusion in international accredited lists; the print-run; author prominence; the rejection rate; library usage; the citation index; court citations; and NRF rating choices.

4.2 International accredited lists

Although this article is limited to the ranking of law journals on the DHET list of accredited journals, some of these journals also appear on international accreditation lists recognised by the DHET. Could it be argued that journals appearing on international lists should be ranked higher? It may well be that inclusion on numerous accreditation lists could have a positive influence on their impact because it makes the journals more accessible. But are these journals necessarily of a higher quality?

On the one hand it could be argued that they are not necessarily so, as the criteria used for inclusion in any of the lists are similar to those used by the DHET. No information is available about the reasons why all the journals are not on international lists. For instance, did they apply and were they rejected, or did they not apply at all?

Certain South African public institutions award greater financial incentives to academics who publish in journals accredited in international lists, indicating perceptions of their better quality or greater impact.137 Most importantly, to be accepted for and remain on these international lists the journal must undergo an additional systematic and continuous evaluation by experts, using set criteria of scholarly expertise,138 including peer-

reviewed139 high quality analytical research under an international editorial board of academics. Most of these lists have their own areas of expertise that journals must adapt to for inclusion. NSD and ISI focus on the diversity of authorship. ISI specifically considers the citation index of the journal within the context of the discipline. SciELO is focused on the DHET criteria as confirmed by the ASSAf evaluation panel reports; and Scopus on indexing and citations.

For the purposes of this article and because these journals have been subjected to additional external scrutiny and evaluation and have been found to meet their specific criteria, the journals that appear on international lists, and the number of times they appear are ranked higher for the purposes of this article.

Only one South African law journal appeared in four additional lists in 2014,140 namely the SAJHR, which is ranked first. AHRLJ, CILSA and PER were each listed in two of these lists,141 and are jointly ranked second. The titles of ten journals appeared in only one international list: SALHR, SALJ, JJS, THRHR and AHRJ in IBSS; and SAJHR, SAYIL and the SACJ in the NSD list. The SciELO list also included De Jure, Fundamina and LDD.

In conclusion, it is reiterated that this system is not a true reflection of the quality of a journal's research output and as such does not meet the Perry minimum requirement for quality-sensitive criteria. It is nevertheless included in the summary under 5 hereunder.

4.3 Ranking based on the print run

The print run of a journal used to be an indication of its popularity, coverage and visibility, including in South Africa, as market forces generally differentiate between journals based on their relevance and impact.142

The ASSAf Report noted the print-run of all hard-copy law journals, but the information could not be verified independently. Print run as a measuring tool has limited usefulness, as the number of copies of the journals printed

does not necessarily indicate the quality of the research per volume. Serial journal subscription should also be taken into account. Moreover, in later years some journals have decreased their print runs as they became available electronically. Others, like PER, have never been available in hard copy, while TSAR was unwilling to release its print run information to ASSAf,143 this making the inclusion of these two journals in the ranking impossible.

The stated number of hard copies printed is as follows: SALJ (1150),144 ILJ (1000),145 Annual Survey (750),146 AHRLJ (650),147 Acta Juridica (600),148 THRHR and Merc LJ (550 each),149 SAYIL (400),150 SAJHR (391),151 LDD (375),152 SACJ (350),153 Obiter and Stell LR (315 each),154 CILSA (300), Fundamina (300),155 De Jure (260),156 JJS (250),157 SA Public Law and Speculum Juris (200 each).158

Although seemingly straightforward, print run is not an effective measuring tool and it is evident that it does not meet the Perry minimum requirements for an acceptable ranking system, in that the criteria are not quality-sensitive. In addition, in the light of the move towards open online access, the print run will become less relevant. It is still included in the ranking data hereunder at 5, however.

4.4 Ranking based on author prominence

We do not believe that we need to provide a detailed justification. Right or wrong, good or bad, justified or unjustified, prestige speaks volumes in the legal – and legal academic – world … Accordingly, we think our decision to

attempt a prestige-based ranking of specialised reviews will strike most readers as intuitive.159

In 1997 Jarvis and Coleman160 ranked law reviews in the US by author prominence and in 1999 George and Guthrie161 did the same for speciality law journals.162 The rationale for this ranking methodology is that it "reflects the common-sense intuition that the prestige of a journal depends largely upon the prestige of the authors whose articles it publishes".163 Law journals were ranked over a five-year period, using a 1 000-point contributor scale according to the prestige of the authors of lead articles at the time of publication.164 The scale ranged from 1 000 points for an article by the US President to 750 for a US Circuit Court judge to 625 for a law professor at a first-tier law school.165 Although the creators acknowledged that the scale itself was subjective, they argued that the exact points were not as important as the consistency in comparing journals.166

The main flaw in this method is its subjectivity,167 which is highlighted by Crespi,168 Perry169 and Korobkin.170 It is the status or prominence of the author which determines the rankings rather than the scholarliness or quality of the article published.171 Author prominence does "not necessarily correlate with creativity, innovation, profundity, style, usefulness, or impact on legal thought or practice".172 It incentivises editors to select articles based on the prestige the author might lend the journal, rather than the quality of the submission.173

Applying the system in South Africa, a slightly amended scale based on the Jarvis and Coleman contributor scale is adopted for this article.174 However, it remains subjective, unsatisfactory and open to criticism.175

Table 1: South African Rating Scale

The average contributor score for each volume per year was used for the calculations.

Table 2: Author Prominence Calculations per Journal per Year

Should this system be used for ranking, TSAR and Annual Survey would be ranked jointly first (586), with THRHR third (551), Fundamina fourth (549) and SA Public Law fifth (546). The next group would contain Acta Juridica (527), De Jure (525), Merc LJ (524) and Stell LR (524) with SACJ (521) next.

In conclusion, even though this ranking system does not meet the Perry minimum criteria for its lack of quality-sensitivity, it is nonetheless included under 5 hereunder. Furthermore, in the South African historical context focussing on author prominence as the basis of a ranking system would not be an appropriate measure. It might dis-incentivise editors from publishing articles by younger (black) academics as they may not yet register high on the prominence scale, resulting in the possibility of constraining the academic transformation process.

4.5 Ranking based on the rejection rate

Another ranking system for journals might be based on the rejection or acceptance rates of submitted articles. The argument is that the more selective the journal, the higher its rejection rate, which could indicate a higher quality of scholarly publication. Perry176 argues that this method is unreliable for the following reasons: cooperation from the editors is required in collecting and producing the data, and the information may be unreliable and unverifiable; and differences in rejection policies may distort the

rejection rates and result in the artificial alteration of the rates. He asks, for instance, if the response "revise and re-submit" is a rejection or not?177 Is the non-acceptance of a quality article because of systematic preferences a rejection? The rejection rate focuses only on the selection made by the journal among the articles submitted to it, and not the selection made by the authors of the journals to which they will submit their articles; and the solicitation of articles may decrease the rejection rate, which would therefore not represent a true reflection of the quality of the material.178 These arguments are also relevant vis-à-vis South African law journals. In addition, one should not lose sight of the fact that a journal with a greater output of volumes per annum may receive more submissions179 and that the subjective nature of peer review may influence the rejection rate.180

I refer yet again to the ASSAf Report on Grouped Peer Review of Scholarly Journals in Law as the only available information on this topic. The Report reflects the number of rejections by most of the law journals, although the data could not be independently verified.

Table 3: Rejection rate according to ASSAf

The ranking according to the rejection rate result in Merc LJ being ranked first, AHRLJ second and JJS third. SALJ and LDD would be ranked fourth and fifth respectively.

In conclusion, apart from the data’s being unverifiable and subjective, it does not meet the Perry minimum requirement of quality-sensitivity. The data are nonetheless included under 5 hereunder.

4.6 Ranking based on library and electronic database usage

This type of ranking is based on the library usage of journals – either through actual physical viewing of the journal in the library or online by way of electronic downloads.200 This method cannot be regarded as very effective in South Africa, as there are no national or other verifiable statistics available for the physical access to journals. Research in the US has shown that the data would be university-, region- or database-specific, depending on the interest and focus area of academics at a specific university.201 The results would be demand-driven and dependent on what the "hot" topics202 are, and could ultimately be manipulated by an in-house academic who requires his/her students to use specific materials.203 Online databases are not very reliable sources of statistics, as companies are competitive and secretive.204 When accessing information from outside the system, there is always the possibility of a manipulated result because, without individual logon identification, multiple views or author views may not be highlighted.205 The frequency of downloading is also not a true reflection of the frequency of

actual use206 and may not be indicative of the quality or impact of the article.207

As it was not possible to obtain any data of electronic usage per journal for the period, no rating could be made on this basis.

4.7 Ranking based on journal citations

One of the standard ways of determining the quality of a journal for ranking purposes in various disciplines is to look at the number of citations it receives:208

The overall impact of a given journal is equivalent to the aggregate impact of all articles that were published in it during the relevant time period. Consequently, the citation frequency of all articles published in a certain journal in a given set of subsequent texts constitutes a rough measure of that journal's impact on the professional discourse within a specific circle, although … it needs to be adjusted to serve as an approximate measure of the academic value of this journal.209

The argument is that the more often the journal is cited, the higher the quality and the more influential the journal.210 The citation frequency of law journals can be seen as an indication of important scholarship, although it may not always be the case.211 Although citation is regarded as a more objective measure of quality,212 it is not without its problems, such as technical difficulties in the "citing–cited matching process"213 underlying the bias214 and limitations that underpin the system.215

The quality of each journal may vary and an article published in a frequently cited journal may not necessarily be of a high quality itself.216 In addition, this ranking system is "overly sensitive to the presence of one or two

remarkable articles" getting numerous citations.217 In addition, editors could be tempted to publish articles that they know would be controversial and thus more likely to be cited218 or to publish articles by more famous authors likely to be cited and not necessarily because of the quality of the article itself.219 A journal may also be cited for the "wrong" reason.220 Korobkin inter alia names a few instances where the citing of a journal "provides little or no evidence of scholarly value." For instance, the article may contain an unoriginal argument that is easy to access; the source may be a friend or ally of the author; the author may wish to impress the source;221 the source may be incorrect or present an opposite view from that of the author.222 Other negatives of the citation system have been noted by Tome and Lipu:223 Scholars working in the same field tend to cite one another, resulting in "strong geographical and regional tendencies"224 and favouring English as the medium.225

Other problems are that the citations may include negative citations, self-citation, collegial citation or selective citation without necessarily reflecting the quality of the journal.226

Citations are inherently predisposed towards older journals,227 although the number of citations may decline over the years.228 The extent of the size of

the publication may impact on the number of citations because of an increased word-count, even though the material published may be more mediocre.229 Articles available online in their entirety are more likely to be cited than those available in hard copy only.230

Impact factors also differ for each subject field, document type and journal size.231 The citation conventions in the legal field vary: articles are comprehensively referenced and every citation may not be of equal scholarly value.232 In addition, there may be inconsistencies in the citations of general law journals vis-à-vis specialist law journals.233 Certain areas of the law could be regarded as more interesting or popular at any given time and this could impact positively on the number of citations.234

Low impact indices are not unusual in law. The London School of Economics Maximising the impact of academic research noted that the average h-index score in law is 1.25 as opposed to 4.83 in Economics and 5.04 in Geography.235

Research in the US has highlighted that various citation indices could provide inconsistent results.236

Those concerned with citations237 in different disciplines have attempted to nuance or adapt their citation systems accordingly to mitigate the negatives.238 Eisenberg and Wells argued that the use of ISI is inappropriate

in the legal field as it is under-represented and slanted towards the sciences, and may be detrimental to scholarship.239

The most popular social sciences index in South Africa is the Web of Science SSCI Social Science Citation Index, but a certain preference for country and discipline is present.240 As only the SAJHR was included in this list during the relevant period, this index is disregarded in this article.

The Washington and Lee citation databases count only citations by US Law journal articles in the Westlaw Database.241 As a result, the citation counts of smaller legal jurisdictions such as South Africa, which tend to be jurisdiction-specific, are not well-represented.242

In this article the existing Washington and Lee Law Journal Ranking Project243 data will be used as an example. It has been argued that its so-called Combined Factor balances the raw citations with the impact factor to give a more realistic measure of influence.244 This system considers the wider readership of legal materials such as practitioners, judges, academics and policy makers.245 It should be noted that compared to ISI it has been described as "surprisingly inconsistent, with no statistically significant correlation".246

The rankings described in this article are based on citations found in the full-text Westlaw journals database of journals published in the preceding eight years.247 For our purpose, it is important to note that fourteen of the relevant journals appear on the list. By 2014 the South African journals were ranked as follows:

Table 4: Ranking by citation by Washington and Lee

In conclusion, the citation system does not meet the Perry minimum requirements: comprehensive South African data is unavailable and it is not quality-sensitive. In addition, it may not be productive to rank South African law journals and pit the editors against each other in such a small pool of legal academia:

The notion of competition among editors has its disturbing side, in that behaviour modification to manipulate a journal's citation ranking may not be a positive feature. Without improving the intellectual quality of a general law review it is possible to increase a journal's citation ranking by excluding less popular topics …251

4.8 Court citations

The first legal citation index dates back to the Shepard's Citations of 1873, wherein the index listed court cases and publications referred to.252 Because of the nature of legal research, it was argued that it may be possible to rank the journals according to the number of citations of the journal by the courts.253 As such, it may be a measure of the value of legal academic scholarship,254 especially if the aim of legal research is seen to be legal change and the development of the law.

The question, however, is which courts should be used for the purposes of this research. In the US "there are at least two paradigmatic versions of the method":255 one, the citations in the appellate courts, which would reflect the influence the journal exercises on the understanding and development of the law and especially the resolving of practical legal problems – which shows creativity and innovation;256 and two, the frequency of citations in the trial courts because of the nature and variety of work done in these courts –

reflecting the analysis of existing law.257 The court citation may also be negative.258

A rudimentary attempt was made to ascertain the citation rate of journals by using the Juta, Lexis Nexis and Saflii judgment databases259 in an online search. It included all the online facilities in 2014. The search was not limited to a specific number of years, resulting in a possible slant towards older journals.260 This was counteracted by considering the number of years a journal has been in existence. The citations of certain journals could not be calculated because the title picked up references to the word in other contexts as well, such as Obiter, De Jure and PER. These three had to be ignored for the purposes of this exercise. These searches did not make provision for any possible duplication of reported cases in the various databases.

Table 5: Court citations

Although the data obtained are flawed and incomplete, if this system is used the ILJ would be ranked first, with SALJ second, THRHR third and TSAR and SAJHR fourth and fifth respectively. The high ranking of the ILJ is not unexpected as it includes labour judgments, and possibly because the databases searched include a high number of CCMA awards and where labour law sources are cited a slanting towards more citations from the ILJ is inevitable.

Because of the unreliability of the information, this cannot be a proper basis for an official ranking system. It does not meet the Perry minimum requirements of reliability and quality-sensitivity. The data are nonetheless included under 5 hereunder.

4.9 Ranking based on the outputs chosen by successful NRF-rating applicants as their best261

In terms of the NRF rating application requirements, applicants who apply for rating must submit what they regard as their five best publications of the past eight years for peer-evaluation.262 These submissions may include locally or internationally published books, chapters in books, or journal articles.

When one looks at the NRF-rating data over a period of six years, 2009–2014, the raw data reveal the frequencies with which the journals relevant

to this study were selected by successful NRF-rating applicants: SALJ (40); TSAR (32); Merc LJ (31); THRHR (29); Stell LR (26); SAJHR (24); PER (20); CILSA (17); SA Public Law (16); De Jure (12); Obiter (9); SACJ (8); AHRLJ (8); Acta Juridica (7); Fundamina (6); LDD (6); ILJ (5); JJS (5); SAYIL (5) and Speculum Juris (5). It is not surprising to note that specialised journals such as Fundamina, SACJ and the ILJ were selected less frequently than the generalist law journals, keeping in mind that there are fewer specialist academics in South Africa.

When one breaks down the data further into rating categories, the information becomes more nuanced. The A-rated legal academics, who are in the minority, displayed a preference for submitting internationally published books and international journal articles for the NRF review process.263 B-rated academics, in contrast, submitted a variety of materials. Apart from a modicum of conference proceedings and reports, there was an almost equal distribution on average of South African journal articles, international articles and chapters in books.264 There was also a noteworthy number of book submissions in this class (15%).265 The C-rated academics concentrated on submissions published in South African journals – at all levels.266 Although chapters in books and international journals also played a role, these were substantially fewer than the South African journals.267 The submission of chapters and books increases as the rating becomes higher.268 P- and Y1-rated scientists generally included more international journals in their submission lists.269

From the above it may be inferred that there seems to be a trend that the higher an academic's NRF-rating, the less significant the role South African law journals play in the assessment of career-defining publications. This

does not detract, however, from the submission above, namely that the esteem in which a journal is held is reflected in the number of highly-rated academics quoting it.

An additional caveat should be added to this analysis, namely that fewer than twenty per cent of law academics in South Africa were rated in 2014.270 A reason for this could be inter alia a pre-conception that applicants will be given a C-rating, irrespective of their real standing in the field nationally, as many fields in law do not lend themselves to publication in international journals. Whatever the merit of the argument, rating is not compulsory at most universities, and many legal academics are not interested in applying for a rating. And, going forward, recent developments at the NRF to reduce incentive funding may suggest that very few academics will apply for rating in future.

Table 6: Rated law academics (2014)

If this system were to be used as a guide, the ranking of the top five journals would be as follows: SALJ, TSAR, Merc LJ, THRHR and Stell LR. However, apart from the reasons given above, the data do not meet the Perry minimum requirements for an overall quality-sensitive measurement. The data are nevertheless included under 5 hereunder.

5 Consolidated data

The evidence above that none of the systems discussed is flawless, trustworthy and quality-sensitive, but it also highlights the problems related to ranking systems in general as well as specific ranking systems in particular.

The question can rightly be asked whether the various systems should not be weighted, as some of the measures could be regarded as inherently more important than others. The choice of systems and weightings could be in the interests of credibility and trustworthiness.271 However, any exclusions or weightings would remain purely subjective.272 For the purposes of illustration, all the available data are included.

To consider the consistency in the outcome of the various systems of ranking South African law journals, the spread of the outcomes per journal and the average of all the imperfect results are shown.

Table 7: Spread of the ranking outcomes (highest, lowest and average ranking)

So what does all of this mean? The spread per journal of the outcome of the various ranking systems used shows that there is very little consistency in the outcome of the data.273 The ranking of a journal will thus depend largely on the ranking system(s) chosen, making the use of ranking as a quality and impact measure random, unfair, irrational and unreliable.

6 Conclusion

The aim of this article is to consider whether a journal ranking system would be useful to assist the DHET in determining quality and impact in the legal field. None of the journal ranking systems discussed meets the Perry minimum requirements for a reliable and trustworthy system. None can claim to be objective, with quality-sensitive criteria free from bias. In addition the data are difficult to retrieve, unverifiable and possibly open to manipulation. For South Africa as a small jurisdiction the negatives clearly outweigh any possible benefits of the use of a data ranking system.274

The short response to the proposal of a ranking system for South African law journals would be negative, making consultation with the DHET urgent,

especially since the decision is directly linked to government funding. Not acting could result in an inappropriate and prejudicial system’s being imposed on the legal fraternity, especially in the light of the tendency to use journal citations as a fall-back position.

More research and debate are required. These may indicate that expert panels should be appointed to determine quality and impact, which may or may not result in the giving of an expanded brief to ASSAf. New developments on the electronic front, particularly around databases, and a usable, verifiable and trustworthy citation system may in future be developed for law to create indicators of quality and impact.

The quest for a solution should be part of a serious national debate on substantive quality and impact and the development of a matrix to give effect to the complexity of the legal discipline, as suggested by Van Gestel.275 The South African Law Deans' Association may have a role to play in this regard.

Whatever the final decision, there are deep concerns currently about a law journal ranking system for South Africa and I can only agree with the sentiment eloquently expressed by Svantesson and White:

[N]one of the ranking methodologies ... has the sort of scientific rigour that can be expected from an exercise with such profound implications. … [T]he results… are bound to be influenced by intentional and/or unintentional biases. Consequently, there may be a lack of 'procedural fairness' as to who will be the winners and who will be the losers under any particular ranking scheme. Finally, we are concerned about the consequences that inevitably will flow from research ranking. Those consequences may be particularly detrimental for a small jurisdiction …276

[In addition …] we have little faith that existing research ranking schemes are capable of producing accurate and scientifically valid ranking of research. At the same time, we are convinced that attempts to rank research carry with them serious detrimental effects on research diversity, and potentially, research quality. This is particularly so where research funding is based on the outcome of the ranking exercise.277

In addition, any ranking system could probably be open to challenge should it have a detrimental effect for any given journal. Administrative justice requires, at least, rationality, and it would not appear that the implementation of any of the systems discussed here and found to be wanting.

Bibliography

Literature

Altbach 2001 Higher Education

Altbach PG "Academic Freedom: International Realities and Challenges" 2001 Higher Education 205-219

ASSAf Report on Strategic Approach to Research Publishing

ASSAf Report on a Strategic Approach to Research Publishing in South Africa (ASSAf Pretoria 2006)

ASSAf Report on Grouped Peer Review of Scholarly Journals in Law

ASSAf Report on Grouped Peer Review of Scholarly Journals in Law and Related Legal Fields (ASSAf Pretoria 2014)

Brophy 2006 Connecticut Law Review

Brophy AL "Law [Review]'s Empire: The Assessment of Law Reviews and Trends in Legal Scholarship" 2006 Connecticut Law Review 101-107

Budden et al 2008 Trends Ecol Evol

Budden AE et al "Double-Blind Review Favours Increased Representation of Female Authors" 2008 Trends Ecol Evol 4-6

Campbell, Goodacre and Little 2006 J L & Soc'y

Campbell K, Goodacre A and Little G "Ranking of UK Law Journals. An Analysis of the Research Assessment Exercise (RAE) 2001 Submissions and Results" 2006 J L & Soc'y 335-363

Carnelley 2015 Obiter

Carnelley M "Identifying Predatory Open-Access Academic Journal Publishers, in Light of the South African Department of Higher Education and Training's Decision to Retrospectively De-accredit Certain Journals" 2015 Obiter 519-538

Crespi 1997 Int'l Law

Crespi GS "Ranking International and Comparative Law Journals: A Survey of Expert Opinion" 1997 Int'l Law 869-886

Crespi 1998 Wm & Mary Envtl L & Pol'y Rev

Crespi GS "Ranking the Environmental Law, Natural Resources Law and Law Use Planning Journals: A Survey of Expert Opinion" 1998 Wm & Mary Envtl L & Pol'y Rev 273-297

Crespi 1999 Fla St U L Rev

Crespi GS "Ranking Specialised Law Reviews: A Methodological Critique" 1999 Fla St U L Rev 837-849

Currie and Pandler 2011 J Bank Finance

Currie RR and Pandler GS "Survey. Finance Journal Rankings and Tiers: An Active Scholarly Assessment Methodology" 2011 J Bank Finance 7-20

Curtis 2008 Globalisation, Societies and Education

Curtis B "The Performance‐Based Research Fund: Research Assessment and Funding in New Zealand" 2008 Globalisation, Societies and Education 179-194

Datt, Tran and Tran-Nam 2009 ATF

Datt K, Tran A and Tran-Nam B "Ranking Tax Journals – The Way Forward" 2009 ATF 341-369

DHET Research Output Policy

Department of Higher Education and Training Research Output Policy (DHET Pretoria 2015)

DHET White Paper

Department of Higher Education and Training White Paper for Post-School Education and Training: Building an Expanded, Effective and Integrated Post-School System (DHET Pretoria 2013)

DHET Directorate Communiqué

Department of Higher Education and Training Directorate: University Policy and Development Support Communiqué 2 of 2015 (DHET Pretoria August 2015)

Doyle 2004 Leg Ref Serv Q

Doyle J "Ranking Legal Periodicals and some other Numeric Uses of the Westlaw and Lexis Periodical Databases" 2004 Leg Ref Serv Q 1-396

Editorial 2012 EJIL

Editorial 2012 "Impact Factor – The Food is Bad and What's More There is not Enough of It: EJIL – The Beginning of the Existential Debate; Masthead Changes; In this Issue" 2012 EJIL 607-612

Editorial 2015 Nature

Editorial 2015 "Nature Journals Offer Double-Blind Review" 2015 Nature 274

Eisenberg and Wells 2014 Economic Inquiry

Eisenberg T and Wells MT "Ranking Law Journals and the Limits of Journal Citation Reports" 2014 Economic Inquiry 1301-1314

European Union Expert Group on Assessment of University-based Research Assessing University-based Research

European Union Expert Group on Assessment of University-based Research Assessing University-based Research (Publications Office of the EU Luxembourg 2010)

Fuld 1953 NYU L Rev

Fuld SH "A Judge Looks at the Law Review" 1953 NYU L Rev 915-921

George and Guthrie 1999a Fla St U L Rev

George TE and Guthrie C "An Empirical Evaluation of Specialised Law Reviews" 1999 Fla St U L Rev 813-836

George and Guthrie 1999b Fla St U L Rev

George TE and Guthrie C "In Defence of Author Prominence: A Reply to Crespi and Korobkin" 1999 Fla St U L Rev 877-896

Given, Kelly and Wilson "Bracing for Impact"

Given LM, Kelly W and Wilson R "Bracing for Impact: The Role of Information Science in Supporting Societal Research Impact" in Proceedings of the 78th ASIS&T Annual Meeting: Information Science (6-10 November 2015 St Louis, MO) 1-10

Grossman 2003 Colum J Gender & L

Grossman JL "Feminist Law Journals and the Ranking Conundrum" 2003 Colum J Gender & L 522-531

HEFCE Research Excellence Framework

Higher Education Funding Council for England Research Excellence Framework 2014: Overview Report by Main Panel C and Sub-panels 16 to 26 (HEFCE Bristol 2015)

HEFCE Research Assessment Exercise 2001

Higher Education Funding Council for England Research Assessment Exercise 2001 Law Panel General Overview (HEFCE Bristol 2001)

Jarvis and Coleman 1997 Arizona L Rev

Jarvis RM and Coleman PG "Ranking Law Reviews: An Empirical Analysis Based on Author Prominence" 1997 Arizona L Rev 15-24

Jarvis and Coleman 2007 L Libr J

Jarvis RM and Coleman PG "Ranking Law Reviews by Author Prominence – Ten Years Later" 2007 L Libr J 573-588

Kahn 2011 SAJS

Kahn M "A Bibliometric Analysis of South Africa's Scientific Outputs – Some Trends and Implications" 2011 107(1/2) SAJS 1-6

Korobkin 1999 Fla St U L Rev

Korobkin R "Ranking Journals: Some Thoughts on Theory and Methodology" 1999 Fla St U L Rev 851-876

Liefner 2003 Higher Education

Liefner I "Funding, Resource Allocation and Performance in Higher Education System" 2003 Higher Education 469-489

Mallard, Lamont and Guetzkow 2009 Sci Technol Hum Values

Mallard G, Lamont M and Guetzkow J "Fairness as Appropriateness. Negotiating Epistemological Differences in Peer Review" 2009 Sci Technol Hum Values 573-606

Marashi et al 2015 SAJS

Marashi S-A et al "Can Scientific Journals be Classified Based on their 'Citation Profiles'?" 2015 SAJS 1-3

Maru 1976 Am B Found Res J

Maru O "Measuring the Impact of Legal Periodicals" 1976 Am B Found Res J 227-249

McWhirter Legal 100

McWhirter DA The Legal 100: A Ranking of Individuals Who Have Most Influenced the Law (Birch Lane Press Secaucus 1998)

Ministry of Education Policy and Procedures

Ministry of Education Policy and Procedures for Measurement of Research Output of Public Higher Education Institutions (Department of Education Pretoria 2003)

Moed, Van Leeuwen and Reedijk 1998 J Doc

Moed HF, Van Leeuwen Th N and Reedijk J "A New Classification System to Describe the Ageing of Scientific Journals and Their Impact Factors" 1998 J Doc 387-419

Mouton Bibliometric Analysis

Mouton J A Bibliometric Analysis of the State of Research at UKZN. Final Report (Centre for Research on Evaluation, Science and Technology Stellenbosch 2015)

Mouton and Valentine 2017 SAJS

Mouton J and Valentine A "The Extent of the South African Authored Articles in Predatory Journals" 2017 SAJS 1-9

NRF/RISA Evaluation and Rating

NRF/RISA Evaluation and Rating. Guidelines and Instructions for Completing the Application for Evaluation and Rating via the NRF Online Submission System (NRF/RISA Pretoria 2015)

OCLC Research Assessment Regimes

Online Computer Library Centre A Comparative Review of Research Assessment Regimes in Five Countries and the Role of Libraries in the Research Assessment Process. A Pilot Study (OCLC Truro 2009)

Osterloh and Frey Research Governance in Academia

Osterloh M and Frey BS Research Governance in Academia: Are There Alternatives to Academic Rankings? (Institute for Empirical Research in Economics Zurich 2009)

Perry 2006 Va J Law Technol

Perry R "The Relative Value of American Law Reviews: A Critical Appraisal of Ranking Methods" 2006 Va J Law Technol 1-40

Pouris and Pouris 2015 SAJS

Pouris AEM and Pouris A "An Assessment of South Africa's Research Journals: Impact Factors, Eigenfactors and Structure of Editorial Boards" 2015 111(3/4) SAJS 1-8

Pritchard 1998 Minerva

Pritchard RMO "Academic Freedom and Autonomy in the United Kingdom and Germany" 1998 Minerva 101-124

Ramsay and Stapledon 1997 MULR

Ramsay I and Stapledon GP "A Citation Analysis of Australian Law Journals" 1997 MULR 676-692

Smyth 1998 U Tas LR

Smyth R "Academic Writing and the Courts: A Qualitative Study of the Influence of Legal and Non-Legal Periodicals in the High Court" 1998 U Tas LR 164-182

Smyth 2012 UNSWLJ

Smyth R "Who Publishes in Australia's Top Law Journals?" 2012 UNSWLJ 201-245

Steinbuch 2009 Loy LA L Rev

Steinbuch R "On the Leiter Side: Developing a Universal Assessment Tool for Measuring Scholarly Output by Law Professors and Ranking Law Schools" 2009 Loy LA L Rev 87-123

Svantesson 2009 Legal Studies

Svantesson DJB "International Ranking of Law Journals – Can it be done and at What Costs?" 2009 Legal Studies 678-691

Svantesson and White 2009 Bond LR

Svantesson DJB and White P "Entering an Era of Research Ranking – Will Innovation and Diversity Survive?" 2009 Bond LR 173-191

Van Gestel 2015 Legal Studies

Van Gestel R "Sense and Non-Sense of a European Ranking of Law Schools and Law Journals" 2015 Legal Studies 165-185

Van Gestel and Vrancken 2011 GLJ

Van Gestel R and Vrancken J "Assessing Legal Research: Sense and Nonsense of Peer Review Versus Bibliometrics and the Need for a European Approach" 2011 GLJ 901-929

WITS Research Publication Incentive (RINC) Policy

University of the Witwatersrand Faculty of Commerce, Law and Management Faculty Research Committee Research Publication Incentive (RINC) Policy (WITS Johannesburg circa 2015)

Yoon 2013 JLA

Yoon AH "Editorial Bias in Legal Academia" 2013 JLA 309-339

Internet sources

Akademie van Wetenschappen 2014 http://www.knaw.nl/nl/actueel/nieuws/

wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek

Akademie van Wetenschappen 2014 Standard Evaluation Protocol 2015 – 2021 http://www.knaw.nl/nl/actueel/nieuws/wetenschapsorganisaties-presenteren-nieuw-evaluatieprotocol-voor-onderzoek accessed 10 October 2017

Elsevier date unknown https://www.elsevier.com/solutions/scopus

Elsevier date unknown About Scopus https:// www.elsevier.com/solutions/scopus accessed 10 October 2017

IBSS 2013 http://media2.proquest.com/documents/IBSS+ Editorial+Policies+and+ Principles.pdf

International Bibliography of the Social Sciences 2013 IBSS Editorial Policies and Principles http://media2.proquest.com/ documents/ IBSS+Editorial+Policies+and+Principles.pdf accessed 10 October 2017

London School of Economics 2010 http://blogs.lse.ac.uk/impactofsocial sciences/files/ 2010/11/chart3.jpg

London School of Economics 2010 Maximising the Impact of Academic Research http://blogs.lse.ac.uk/impactofsocialsciences/files/2010/11/chart3.jpg accessed 10 October 2017

NSD date unknown https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.

action?requestlocale=en

Norwegian Register for Scientific Journals, Series and Publishers date unknown Criteria for Inclusion of New Scientific Publication Channels https://dbh.nsd.uib.no/publiseringskanaler/OmKriterier.action?request_locale=en accessed 10 October 2017

PBRF date unknown http://www.tec.govt.nz/funding/funding-and-performance/

New Zealand Performance-Based Research Fund date unknown Funding and Performance http://www.tec.govt.nz/Funding/funding-and-performance/ accessed 10 October 2017

SciELO date unknown http://www.scielo.org.za/avaliacao/avaliacao_en.

htm

Scientific Electronic Library Online date unknown 'SciELO SA Criteria': Criteria, Policy and Procedures for Admission and Permanence of Scientific Journals in the South African Collection of SciELO http://www.scielo.org.za/avaliacao/avaliacao_en.htm accessed 10 October 2017

Testa 2016 http://wokinfo.com/essays/journal-selection-process/

Testa J 2016 Journal Selection Process http://wokinfo.com/essays/journal-selection-process/ accessed 10 October 2017

Tome and Lipu 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi= 10.1.1.897.6809&rep=rep1&type=pdf

Tome L and Lipu S 2004 Research and Development Discussion Paper #6: Indicators of Journal Quality http://citeseerx.ist.psu.edu/viewdoc/download ?doi=10.1.1.897.6809&rep=rep1&type=pdf accessed 19 December 2017

UKZN Research Office date unknown http://research.ukzn.ac.za/DoHETAccredited Journals.aspx

UKZN Research Office date unknown DHET Accredited Journals http://research. ukzn.ac.za/DoHETAccreditedJournals.aspx accessed 10 October 2017

Washington and Lee University 2009-2014 http://lawlib.wlu.edu/LJ/

Washington and Lee University 2009-2014 Law Journals: Submissions and Ranking, 2009 – 2014 http://lawlib.wlu.edu/LJ/ accessed 10 October 2017

WITS date unknown https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf

University of the Witwatersrand date unknown South African Journal on Human Rights https://www.wits.ac.za/sajhr/#sthash.OYn340c9.dpuf accessed 10 October 2017

LIST OF ABBREVIATIONS