4 Literature review and citations/references

Literature reviews and references

Figure 4.1: Literature reviews and references

Your may have done a literature survey as part of your proposal. This will be incorporated into your dissertation, not left as separate stand-alone. Most economics papers include a literature review section, which may be a separate section, or incorporated into the paper’s introduction. (See organising for a standard format.)

Some disambiguation:

A ‘Literature survey’ paper: Some academic papers are called ‘literature surveys’. These try to summarise and discuss the existing work that has been done on a particular topic, and can be very useful. See, for example, works in The Journal of Economic Perspectives, the Journal of Economic Literature, the “Handbook of [XXX] Economics”

Many student projects and undergraduate dissertations are mainly literature surveys.

4.1 What is the point of a literature survey?

Your literature review should explain:

  1. what has been done already to address your topic and related questions, putting your work in perspective, and

  2. what techniques others have used, what are their strengths and weaknesses, and how might they be relevant tools for your own analysis.

Take notes on this as you read, and write them up.

Figure 4.2: Take notes on this as you read, and write them up.

4.2 What previous work is relevant?

Focus on literature that is relevant to your topic only.

But do not focus only on articles about your exact topic! For example, if your paper is about the relative price of cars in the UK, you might cite papers (i) about the global automobile market, (ii) about the theory and evidence on competition in markets with similar features and (iii) using econometric techniques such as “hedonic regression” to estimate “price premia” in other markets and in other countries.

Consider: If you were Colchester a doctor and wanted to know whether a medicine would be effective for your patients, would you only consider medical studies that ran tests on Colchester residents, or would you consider more general national and international investigations?

4.3 What are “good” economics journal articles?

You should aim to read and cite peer-reviewed articles in reputable economics journals. (Journals in other fields such as Finance, Marketing and Political Science may also be useful.) These papers have a certain credibility as they have been checked by several referees and one or more editors before being published. (In fact, the publication process in Economics is extremely lengthy and difficult.)

Which journals are “reputable”? Economists spend a lot of time thinking about how to rank and compare journals (there are so many papers written about this topic that they someone could start a “Journal of Ranking Economics Journals”. For example, “REPEC” has one ranking, and SCIMAGO/SCOPUS has another one. You may want to focus on journals ranked in the top 100 or top 200 of these rankings. If you find it very interesting and relevant paper published somewhere that is ranked below this, is okay to cite it, but you may want to be a bit more skeptical of its findings.

Any journal you find on JSTOR is respectable, and if you look in the back of your textbooks, there will be references to articles in journals, most of which are decent.

You may also find unpublished “working papers”; these may also be useful as references. However, it is more difficult to evaluate the credibility of these, as they have not been through a process of peer review. However, if the author has published well and has a good reputation, it might be more likely that these are worth reading and citing.

Unpublished “working papers”

You may also find unpublished “working papers” or ‘mimeos’; these may also be useful as references. In fact, the publication process in Economics is so slow (six years from first working paper to publication is not uncommon) that not consulting working papers often means not being current.

However, it is more difficult to evaluate the credibility of this ‘grey literature’, as they have not been through a process of peer review. However, if the author has published well and has a good reputation, it might be more likely that these are worth reading and citing. Some working paper series are vetted, such as NBER; in terms of credibility, these might be seen as something in between a working paper and a publication.

Which of the following are “peer-reviewed articles in reputable economics journals”? Which of the following may be appropriate to cite in your literature review and in your final project?8

  1. Klein, G, J. (2011) “Cartel Destabilization and Leniency Programs – Empirical Evidence.” ZEW - Centre for European Economic Research Discussion Paper No. 10-107

  2. Spencer, B. and Brander, J.A. (1983) “International R&D Rivalry and Industrial Strategy”, Review of Economic Studies Vol. 50, 707-722

  3. Troisi, Jordan D., Andrew N. Christopher, and Pam Marek. “Materialism and money spending disposition as predictors of economic and personality variables.” North American Journal of Psychology 8.3 (2006): 421.

  4. The Economist,. ‘Good, Bad And Ugly’. Web. 11 Apr. 2015. [accessed on…]

  5. Mecaj, Arjola, and María Isabel González Bravo. “CSR Actions and Financial Distress: Do Firms Change Their CSR Behavior When Signals of Financial Distress Are Identified?.” Modern Economy 2014 (2014).

  6. Universities, U. K. “Creating Prosperity: the role of higher education in driving the UK’s creative economy.” London Universities UK (2010).

4.4 How to find and access articles

You should be able to find and access all the relevant articles online. Leafing through bound volumes and photocopying should not be neededs. (Having been a student in the late 90’s and 2000’s, I wish I could get those hours back.)

The old way!

Figure 4.3: The old way!

Good online tools include Jstor (jstor.org) and Google Scholar (scholar.google.co.uk). Your university should have access to Jstor, and Google is accessible to all (although the linked articles may require special access). You will usually have the ‘most access’ when logged into your university or library computing system.If you cannot access a paper, you may want to consult a reference librarian.

It is also ok, if you cannot access the journal article itself, to use the last working paper version (on Google scholar find this in the tab that says “all X versions”, where X is some number, and look for a PDF). However, authors do not always put up the most polished versions, although they should do to promote open-access. As a very last resort, you can e-mail the author and ask him or her to send you the paper.

When looking for references, try to find ones published in respected refereed economics journals (see above).

4.5 Good starting points: Survey article, course notes, and textbooks

A “survey article” is a good place to start; this is a paper that is largely a categorization and discussion of previous work on a particular topic. You can often find such papers in journals such as

  • the Journal of Economic Perspectives,
  • the Journal of Economic Surveys,
  • and the Journal of Economic Literature.

These will be useful as a “catalog” of papers to read and considers citing. They are also typically very readable and offer a decent introduction to the issue or the field.

It is also helpful to consult module (course) notes and syllabi from the relevant field. Do not only limit yourself to the ones at your own university; many of universities make their course materials publicly accessible online. These will not only typically contain reading lists with well-respected and useful references, they may also contain slides and other material that will help you better understand your topic and the relevant issues.

However, be careful not to take material from course notes without properly citing it. (Better yet, try to find the original paper that the course notes are referring to.)

Textbooks serve as another extremely useful jumping off point. Look through your own textbooks and other textbooks in the right fields. Textbooks draw from, and cite a range of relevant articles and papers. (You may also want to go back to textbooks when you are finding the articles you are reading too difficult. Textbooks may present a simpler version of the material presented in an article, and explain the concepts better.)

4.6 Backwards and forwards with references

When you find a useful paper, look for its “family.” You may want to go back to earlier, more fundamental references, by looking at the articles that this paper cited. See what is listed as “keywords” (these are usually given at the top of the paper), and “JEL codes”. Check what papers this paper cites, and check what other papers cited this paper. On Google scholar you can follow this with a link “Cited by…” below the listed article. “Related articles” is also a useful link.

4.7 Citations

Keep track of all references and citations

You may find it helpful to use software to help you manage your citations

  • A storage “database” of citations (e.g., Jabref, Zotero, Endnote, Mendeley); these interface well with Google Scholar and Jstor

  • An automatic “insert citation” and “insert bibliography” in your word processing software

Use a tool like Endnote to manage and insert the bibliographies, or use a bibliography manager software such as Zotero or Jabref,

Further discussion: Citation management tools

List of works cited

Put your list of references in alphabetical order by author’s last name (surname).

Include all articles and works that you cite in your paper; do not include any that you don’t cite.

Avoiding plagiarism and academic offenses**

Here is a definition of plagiarism

The main point is that you need to cite everything that is not your own work. Furthermore, be clear to distinguish what is your own work and your own language and what is from somewhere/someone else.

Why cite? Not just to give credit to others but to make it clear that the remaining uncited content is your own.


Here are some basic rules:

(Rephrased from University of Essex material, as seen in Department of Economics, EC100 Economics for Business Handbook 2017-18, https://www1.essex.ac.uk/economics/documents/EC100-Booklet_2017.pdf accessed on 20 July 2019, pp. 15-16)

  • Do not submit anything that is not your own work.

  • Never copy from friends.

  • Do not copy your own work or previously submitted work. (Caveat: If you are submitting a draft or a ‘literature review and project plan’ at an earlier stage, this can be incorporated into your final submission.

  • Don’t copy text directly into your work, unless:

    • you put all passages in quotation marks: beginning with ’ and ending with ’, or clearly offset from the main text
    • you cite the source of this text.

It is not sufficient merely to add a citation for the source of copied material following the copied material (typically the end of a paragraph). You must include the copied material in quotation marks. … Ignorance … is no defence.’ (ibid, pp. 15 )

(‘Ibid’ means ‘same as the previous citation’.)

WARNINGS

Your university may use sophisticated plagiarism-detection software. Markers may also report if the paper looks suspect

Before final submission, they may ask you to go over your draft and sign that you understand the contents and you have demonstrated that the work is your own.

Not being in touch with your supervisor may put you under suspicion.

Your university may give a Viva Voce oral exam if your work is under suspicion. It is a cool-sounding word but probably something you want to avoid.

Your university may store your work in its our database, and can pursue disciplinary action, even after you have graduated.

Penalties may be severe, including failure with no opportunity to retake the module (course). You may even risk your degree!


Comprehension questions; answers in footnotes

True or false: “If you do not directly quote a paper you do not need to cite it”9


You should read and cite a paper (choose all that are correct)…10

  1. If it motivates ‘why your question is interesting’ and how it can be modeled economically
  2. Only if it asks the same question as your paper
  3. Only if it is dealing with the same country/industry/etc as you are addressing
  4. If it has any connection to your topic, question, or related matters
  5. If it answers a similar question as your paper
  6. If it uses and discusses techniques that inform those you are using

4.8 How to write about previous authors’ analysis and findings

Use the right terminology

Oops:

“Johnson et al. (2000) provide an analytical framework that sheds substantial doubt on that belief. When trying to obtain a correlation between institutional efficiency and wealth per capita, they are left with largely inconclusive results.”

They are not trying to “obtain a correlation”; they are trying to measure the relationship and test hypotheses.

“Findings”: Critically examine sources

Don’t take everything that is in print (or written online) as gospel truth. Be skeptical and carefully evaluate the arguments and evidence presented. Try to really survey what has been written, to consider the range of opinions and the preponderance of the evidence. You also need to be careful to distinguish between “real research” and propaganda or press releases.

The returns to higher education in Atlantis are extremely high. For the majority of Atlanian students a university degree has increased their lifetime income by over 50%, as reported in the “Benefits of Higher Education” report put out by the Association of Atlantian Universities (2016).

But don’t be harsh without explanation:

Smith (2014) found a return to education in Atlantis exceeding 50%. This result is unlikely to be true because the study was not a very good one.

“Findings:” “They Proved”

A theoretical economic model can not really prove anything about the real world; they typically rely on strong simplifying assumptions.

Cringe:

Through their economic model, they prove that as long as elites have incentives to invest in de facto power, through lobbying or corruption for example, they will invest as much as possible in order to gain favourable conditions in the future for their businesses.

Better:

In their two period model, which assumes \[details of key assumptions here\], they find that when an elite Agent has an incentive to invest in de facto power, he invests a strictly positive amount, up to the point where marginal benefit equals marginal cost”

Empirical work does not “prove” anything (nor does it claim to).

It relies on statistical inference under specific assumptions, and an intuitive sense that evidence from one situation is likely to apply to other situations.

Shudder:

“As Smith et al (1999) proved using data from the 1910-1920 Scandanavian stock exchange, equity prices always increase in response to reductions in corporate tax rates.”

Better:

“Smith et al (199) estimated a VAR regression for a dynamic CAP model using data from the 1910-1920 Scandanavian stock exchange. They found a strongly statistically significant negative coefficient on corporate tax rates. This suggests that such taxes may have a negative effect on publicly traded securities. However, as their data was from a limited period with several simultaneous changes in policy, and their results are not robust to \[something here\], further evidence is needed on this question.”

Use the language of classical11 statistics:

Hypothesis testing, statistical significance, robustness checks, magnitudes of effects, confidence intervals.

Note that generalisation outside the data depends on an intuitive sense that evidence from one situation is likely to apply to other situations.

“Findings”: How do you (or the cited paper) claim to identify a causal relationship?

This policy was explained by Smith and Johnson (2002) in their research on subsidies and redistribution in higher education. Their results showed that people with higher degree have higher salaries and so pay higher taxes. Thus subsidizing higher education leads to a large social gain.

The results the student discusses seem to show an association between higher degrees and higher salaries. The student seems to imply that the education itself led to higher salaries. This has not been shown by the cited paper. Perhaps people who were able to get into higher education would earn higher salaries anyway. There are ways economists used to try to identify a “causal effect” (by the way, this widely used term is redundant as all effects must have a cause), but a mere association between two variables is not enough

As inflation was systematically lower during periods of recession, we see that too low a level of inflation increases unemployment.

Economists have long debated the nature of this “Phillips curve” relationship. There is much work trying to determine whether the association (to the extent it exists) is a causal one. We could not rule out reverse causality, or third factor that might cause changes in both variables.

4.9 …Stating empirical results

Don’t write: “I accept the null hypothesis.”

Do write: “The results fail to reject the null hypothesis, in spite of a large sample size and an estimate with small standard errors” (if this is the case)

Note: The question of what to infer from acceptance/rejection of null hypotheses is a complex difficult one in Classical (as opposed to Bayesian) statistics. This difficulty is in part philosophical: classical hypothesis testing is deductive, while inference is necessarily inductive.

4.10 What to report

You need to read this paper more clearly; it is not clear what they conclude nor what their evidence is.

4.11 Organising your literature review

A common marking comment:

These papers seem to be discussed in random order – you need some structure organising these papers thematically, by finding, by technique, or chronologically perhaps.

How should you organise it? In what order?

  • Thematically (usually better)

  • By method, by theoretical framework, by results or assumptions, by field

  • Chronologically (perhaps within themes)

Exercise: Compare how the literature review section is organized in papers you are reading.

Organising a set of references

Figure 4.4: Organising a set of references

Q: What sort of structure am I using in the above outline?

It may also be helpful to make a ‘table’ of the relevant literature, as in the figure below. This will help you get a sense of the methods and results, and how the papers relate, and how to assess the evidence. You may end up putting this in the actual paper.

Organisational table from Reinstein and Riener, 2012b

Figure 4.5: Organisational table from Reinstein and Riener, 2012b

4.12 What if you have trouble reading and understanding a paper?

  • Consult a survey paper, textbook, or lecture notes that discuss this paper and this topic

  • Try to find an easier related paper

  • Ask your supervisor for help; if he or she can

  • Try to understand what you can; do not try to “fake it”

4.13 Some literature survey do’s and don’ts

Do not cite irrelevant literature.

Do not merely list all the papers you could find.

  • Discuss them, and their relevance to your paper.

  • What are their strengths and weaknesses? What techniques do they use, and what assumptions do they rely on? How do they relate to each other?

Use correct citation formats.

Try to find original sources (don’t just cite a web link).

Don’t just cut and paste from other sources. And make sure to attribute every source and every quote. Be clear: which part of your paper is your own work and what is cited from others? The penalties for plagiarism can be severe!

4.14 Comprehension questions: literature review

How to discuss empirical results: “Causal” estimation, e.g., with Instrumental Variables

Which is the best way to state it?12

  1. “As I prove in table 2, more lawyers lead to slower growth (as demonstrated by the regression analysis evidence).”

  2. “Table 2 provides evidence that a high share of lawyers in a city’s population leads to slower growth.”

3.“Table 2 shows that a high share of lawyers in a city’s population is correlated with slower growth.”

Which is better?13

  1. “However, when a set of observable determinants of city growth (such as Census Region growth) are accounted for, the estimate of this effect becomes less precise.”

or

  1. “In the correct regression I control for all determinants of city growth and find that there is no effect of lawyers on growth”

Stating empirical results: descriptive

Do write:

“Using the US data from 1850-1950, I find that inflation is lower during periods of recession. This is statistically significant in a t-test [or whatever test] at the 99% level, and the difference is economically meaningful. This is consistent with the theory of …, which predicts that lower inflation increases unemployment. However, other explanations are possible, including reverse causality, and unmeasured covarying lags and trends.”

“I find a significantly lower level of inflation during periods of recession, and the difference is economically meaningful. This relationship is statistically significant and the data is accurately measured. Thus I find that inflation increases unemployment.”

Some tips on writing a good paper– relevant to literature reviews

  1. Answer the question
  2. Provide clear structure and signposting
  3. Demonstrate an ability for critical analysis
  4. Refer to your sources
  5. Produce a coherent, clear argument
  6. Take time to proofread for style and expresssion
  • Source “Assignment Writing Skills EBS 3rd year 2012”"

  1. Answer: only b is a ‘peer reviewed article in a reputable economics journal’. All of these might be useful to cite, however.↩

  2. False. You need to cite any content and ideas that are not your own.↩

  3. Answers: 1, 5, and 6. Note that 2 and 3 are too narrow criteria, and 4 is too broad.↩

  4. or Bayesian if you like↩

  5. The second one; if this is really causal evidence.↩

  6. The first one. There is no ‘correct regression’. It is also not really correct in classical statistics to ‘find no effect’.↩