Digital Libraries and Multi-Disciplinary Research Skills

 

Mike Thelwall, School of Computing and Information Technology, University of Wolverhampton , Wulfruna Street , Wolverhampton WV1 1SB , UK .

E-mail: m.thelwall@wlv.ac.uk

 

Digital libraries and the Web have brought enormously powerful search mechanisms to the desktops of many researchers. With the use of insights from cultural studies and the history and philosophy of science, it is argued that one likely outcome is a change in the acceptable standards for literature reviews, leading to changes in article contents to reflect a broader academic basis. As a result, researchers should be trained to read and evaluate material from a much wider range of subjects than previously necessary. The broadening base may impact science and non-science subjects differently.

Introduction

Progress in the management of documentation has clearly affected the research process. Seventeenth century scholars may have had access to only one or two journals and a few books of relevant work. Perhaps they read everything they could get their hands on from cover to cover in an indiscriminate fashion. In contrast, by the twentieth century, the massive explosion in scholarly output (Price, 1963) had led to specialisation, with researchers needing to read journals mainly from their own narrow field (Meadows, 1998). An effective researcher needed to know which journals and books were likely to contain information related to his/her field and would also have needed a strategy for manually searching these publications for relevant articles or passages.

The organisation of documentation for scholars has rapidly evolved in the early twenty-first century. Many journals are now available on free Web sites or in a wide variety of broad-based digital libraries (see below). Moreover, there are increasingly sophisticated searchable interfaces to digital libraries that allow authors to search by keywords across wide subject coverage or even across all disciplines. The keyword search is a crucial tool that can have a major impact on the way research is conducted and the skills needed to be an effective researcher. Keyword searches can increase serendipity by retrieving relevant documents from outside the main topic area that a traditional literature search would not have found. This may yield unexpected insights into the problem from an unknown research field. In the case of the discovery of a new pertinent theory, this may change the nature of the research itself, pushing it outside of disciplinary boundaries and making it multidisciplinary. This certainly happens (Foster & Ford, 2003), but it is not known how frequently.

Hine (2000) rightly mocks those who foretell the future based upon a purely technology-driven analysis. With this in mind, drawing upon theories from sociology and the history and philosophy of science, the approach adopted in the rest of the article is to look for changes that are likely based upon an understanding of scientific communication as human culture.

Scholarly Communication

The journal article appears to be a relatively stable and evolved form (Gross, Harmon & Reidy, 2002), but will scholars resist attempts to change it? To answer this, research must be placed in the social setting to which it belongs.

The History and Philosophy of Science

Kuhn, a historian and philosopher of science, sets the context for a modern understanding of science. Prior to the popularisation of Kuhn’s theories, the common-sense understanding of scientific progress was as a series of incremental additions to the existing body of knowledge, a slow “accumulation of facts.” Kuhn (1970), in contrast, characterised science as long periods of pursuing research based upon common sets of shared assumptions (paradigms) interspersed with short bursts of disagreement and debate leading to a new set of shared assumptions (a paradigm shift) that are partly incompatible with the assumptions of the previous set. Each framework of shared assumptions is widely accepted at one time but then rejected (at least in part) when replaced. Science, therefore, does not discover “the truth” but operates within paradigms that are at the time effective in pursuing goals, such as obtaining more accurate measurements or developing new chemicals. Scientific truth is therefore at its core a temporal and a social phenomenon (a nominalist rather than an essentialist perspective [Woolgar, 1988]). This approach is still accepted (e.g., Hess, 1995). On a much smaller scale, the classic sociological study by Latour and Woolgar (1986) of the activities in a single scientific laboratory underlines the social element of the day-to-day routine of scientists. This has helped to spawn a new research area, the social studies of science, which is a sociology of scientists rather than of science.

Non-scientific scholarly areas such as the arts, humanities, and social sciences are more transparently social activities than science. The products of scholars in these areas vary more widely, with art and books being valued in some communities and journal articles in others (Hyland, 2000), but dominant ideas nevertheless typically change over time, reflecting the importance of the same temporal and social factors as those in science. Examples of identifiable changes are movements in art (e.g., romanticism, expressionism), changes in dominant social theories (e.g., modernism, post-modernism), philosophical approaches (e.g., structuralism, post-structuralism), and methodologies in social science (e.g., positivism, post-positivism).

One important end product of scientific (and some non-scientific) research is the journal article. In some fields, books are more important, and, in the modern era, technological progress is also often a desired end product. Nevertheless, the continued major significance of journal articles can be seen in the UK ’s Research Assessment Exercise (Bessant et al., 2003; McNay, 2003). The Gross, Harmon, and Reidy (2002) explanation of the evolution of the journal article over time since its origins in the seventeenth century French Journal des Sçavans denies that the current form is stable and unlikely to change because it is not an improvement over previous formats in an absolute sense. Each earlier form was adapted to the milieu in which it developed. For example, the relatively verbose personal presentational style of the seventeenth and early eighteenth century could be argued to be appropriate to the typical readership of amateur enthusiasts. In this context it is natural that the form of articles may evolve as a result of technological changes rather than to become in some sense closer to optimal.

In summary, sociological, historical, and philosophical perspectives all place research in a social context, hence able to change over time in response to external factors.

Scholarly Communication and Peer Review

Since “the truth” or accepted theories and ideas are socially constructed both inside and outside of science, scholars must communicate to claim a share in their discovery. The almost simultaneous founding of the first scholarly journals in seventeenth-century France and England (Gross, Harmon, & Reidy, 2002) was not motivated by this, however, but by the need to share information and to establish the precedence of claims of “discovery.” In order to stake an academic claim to a particular piece of knowledge today, a scientist may publish a journal article, and a social scientist or humanities scholar may either publish a journal article or book. There are variations on this: conference proceedings are respectable in computer science and works of art in the arts. The importance of refereed journal articles is that they are quality controlled and, in theory, represent validated additions to the store of knowledge. In practice, however, even published articles can be wrong and are subject to obsolescence or reappraisal after paradigm shifts. Others are ignored and thus have no impact at all.

Peer review is still the gold standard for most scholarly publishing, particularly in the sciences. This involves subject specialists reading submissions and judging them on a number of criteria, including whether methods are correct, whether they are understandable and appropriate for the target audience, and whether they significantly add to current knowledge (Wager, Godlee, & Jefferson, 2002). Additionally, conferences and art exhibitions are typically subject to a kind of peer review although the form varies in both cases. External funding grants are also normally peer reviewed, and in some fields these are necessary to conduct any research at all. Perhaps less obviously, academic books are also typically peer reviewed, both as part of the publisher’s quality control and through published book reviews. Hyland (2000), backed by Faigley (1986) and Geertz (1983), argues that successful academic writing depends upon following approved discourses: choosing even the text structures and writing styles as part of a strategy of persuasion. Authors’ strategies reflect their perceptions of key peers and reviewers in particular. In summary, peer review is of critical importance to scholars. Successful researchers will be those who match the expectations of others in their field. Note that this is an explicitly social statement: success does not accrue from merely discovering new knowledge.

Method correctness is, like the rest of science, socially constructed. “Criteria for judging explanations vary from discipline to discipline, and within disciplines from time to time, but with some contact with material reality” (House, 1994). This is perhaps seen most clearly from the perspective of a statistician. A subject involving numbers is likely to need statistics, and this simple observation explains why many social scientists employ complex statistics, perhaps without understanding the foundation of statistics. In such a situation a particular technique can be identified as “correct” for a given type of data by a research community. In concrete terms, a researcher using the accepted technique will not get a paper rejected by his/her peers for using it, whatever a statistician (who presumably will not get to review the paper) may think of the merits of the method. In fact, the field of statistics itself changes over time, as evidenced by a recent shift towards the greater use of non-parametric tests (Howell, 2002).

An important but typically untrumpeted component of peer review is the concept of “reasonable effort.” A paper may be published with flawed methods if the peer group accepts that the researchers have made a reasonable effort to minimise the effects of the flaws. A clear example of this comes from media studies where researchers wishing to analyse media coverage (e.g., of science) would restrict their attention to one or two major newspapers simply because a full scale sample of all types of newspapers would present an impractical data collection exercise (Weigold, 2001). Given the impossibility of a genuine sample, the logical options are either not to conduct this kind of research or to accept the limitations and proceed, resisting a metonymic pull to treat the small sample as being representative. This example also illustrates the temporal nature of such accepted practices: in the digital era large scale content analysis of a wide range of types of newspaper is now practical, so the old fashioned style of study is now unacceptable because better alternatives are available. Indeed, whole new fields have been built upon a technological innovation leading to a new data source, including citation analysis (Garfield, 1972)—predicated upon the existence of a large academic citation database such as that of the Institute for Scientific Information—and webometrics (Almind & Ingwersen, 1997), which analyses Web data.

As noted, the journal article is an evolving form. Changes in the form and content of research can be triggered by social and technological developments. Peer review is an important gatekeeper of quality for scientific publications, but its judgements change over time so that practices acceptable in one era are forbidden in another.

Digital Libraries and the Literature Search

The Literature Search

Previous research into digital libraries has tended to focus on design and use rather than on their potential impact on the behaviour of users (Fox & Urs, 2002). Digital libraries affect most immediately the part of research that involves searching for related work. The expression “literature search” will be used here to mean the process of identifying relevant scholarly work to support new research. Literature searches are most concretely expressed through references, so the two are closely connected. Literature reviews form an important part of most types of scholarly publication, having gradually evolved in mainstream science since the seventeenth century (Gross, Harmon, & Reidy, 2002). In some articles there are explicit “literature review” sections. In others the literature review is either in the introduction or implicit in the early part of the paper through a discussion of background topics. At opposite extremes, whole articles can be literature reviews, or a review can be virtually non-existent, with references used exclusively to support individual facts (or mathematical theorems). A team of researchers has developed, with some success, tools to identify potentially useful connections between previously unrelated research fields (Swanson & Smalheiser , 1997), but such connections can now occur by accident due to the power of keyword searches. This seems particularly applicable to science fields, where different fields may have different types of information about the same subject.

One structure for a literature review section is to start with a general description of the topic area and then to progressively narrow the scope to research that relates as directly as possible to the research question. We call this the traditional literature review. Literature reviews in science do not conform to a single monolithic structure but are often in a separate section or a small number of sections (e.g., Davis , 1997, p. 41). Humanities scholars may be unfamiliar with this style and the need for explicit literature reviews in science and social science. With the trend towards multidisciplinary research (Gibbons et al., 1994), traditional literature reviews may also take the form of multiple topic-based sections.

Citations are more common objects of study than literature reviews, but from citation theory it can be surmised that literature reviews serve many functions, both cognitive and social (Borgman & Furner, 2002). Cognitive connections can be expressed through citing an earlier work that is extended, elaborated, generalised, or refuted. They can also serve other purposes such as providing the sources for methods used. This is the traditional view of the purpose of citations (Merton, 1973). There are many social reasons for citations, for example, their use as part of a wider attempt to convince an audience of the value of the research. A hybrid social-cognitive role is the use of literature reviews to justify the importance of research by citing other work that has discussed similar questions, normally a function of an article’s introduction. References can also be used to establish the credibility of an author. Self-citation is an example of a credibility-establishing tactic (Hyland, 2003), but there is a second one—credibility in terms of establishing scholarly competence. An author is more credible if he/she has demonstrated an awareness of the literature in the field and demonstrated his/her membership in the community by the choice of citations (Hyland, 2000). In cultural studies terminology, the author is performing the identity of a scholarly community member (du Gay, Evans & Redman, 2000).

Since the reasons for referencing vary by discipline (Borgman & Furner, 2002) as does the entire structure and form of scholarly publishing (Hyland, 2000), the above are of necessity general descriptions of phenomena that are not universal even though they have been identified and studied in particular contexts.

Types of Digital Libraries

A traditional manual literature review might have involved selecting a range of journals and visiting a library to scan its indexes to identify relevant articles. Such a process is clearly time consuming and likely to yield narrow subject coverage. Paper-based aids used to speed the process included review journals, which publish a few full-length reviews or many short précis of individual articles. These made the search more efficient, thus allowing some of the time savings to be devoted to widening and broadening the literature review.

Digital tools to facilitate literature searches predate the Web (Fox & Urs, 2002). Historically, these were mainly subject-specific, for example, ERIC (education), Medline (medicine), and INSPEC (science). These considerably aided the power and scope of searching by allowing database keyword searches. The resulting searches could cover all publications in the database, not just a few selected journals or a single specific disciplinary category. This was an advantage because articles on any given topic are likely to be widely scattered in the literature even if most are concentrated in a “core” set ( Bradford , 1934). Keyword searches can find the widely scattered articles. The power of a keyword search actually depends upon several factors: the power of the query interface engine, the content indexed (titles, abstract, full-text), the review topic (common words such as “statistics” would be more difficult than “Italian ethnomethodologists” to review comprehensively), and the competence of the searcher.

A different type of database is the Institute for Scientific Information’s (ISI) citation index, which has a much broader non-subject specific coverage. When it did not include full abstracts, keyword searches would have been based upon titles and article keywords alone, but the search process could also be expanded to include cited references and citing articles. Today, the ISI’s Web of Science provides index abstracts and attempts to index the top journals in every field in addition to some important books and conference proceedings, making it a highly useful and very broad source. Scholars may also have access to subject-specific Web archives that are similar in scope, for instance CiteSeer.com (computer science) and MedWeb.com (medicine). These may give more comprehensive coverage than the ISI for a specific field and may also index full-text articles. Publishers’ digital libraries can now provide similar searching and electronic referencing facilities (Arms, 2001; Halliday & Oppenheim, 2001). Jascó (2003) has remarked upon the apparent contradiction of commercial publishers’ providing free search interfaces and abstracts online, which is due to a new publishing model of purchase-on-demand for individual articles. In fact, publishers now routinely allow free access to selected articles as a marketing ploy, which impacts consumer behaviour (Nicholas, Huntington & Watkinson, 2003). Some publishers also use digital libraries such as SwetsNet and WestLaw that store full-text articles from many different sources.

Commercial search engines are important interfaces for some scholarly digital information, particularly Google for its indexing of PDF and PostScript. If allowed, search engines can potentially index publishers’ digital libraries in addition to scholars’ self-published articles. Since academics can also publish publication lists online, individual journal titles are likely to be frequently found in Google’s index (Vaughan & Shaw, 2003). The Web itself can therefore be conceptualised as a huge, partly-organised digital library. The success of global Web searches will depend upon factors such as users being able to use sufficiently specific and rare keywords. A potentially revolutionary trend has been the digital online preprint archive (Harnad & Carr, 2000) , which appears to be hegemonic in some fields (Rees, 2002). Some have seen this as threatening the heart of the peer review system (Magueijo, 1998). The long-term impacts of the digital revolution in this respect remain to be seen. Withey (2003), for example, has hypothesised regarding the emergence of “promiscuous consumers,” i.e., people who exploit the ease of navigation of the Web to move between information sources.

Consequences of Digital Library Growth

One implication of the now widespread availability of searchable interfaces for large, broad-content digital libraries is that it is reasonable to expect a scholar to use them. Technologies, however, do not emerge fully formed into the world; they interact with consumers at many stages of production and design (du Gay, Hall, Janes, Mackay & Negus, 1997; Storey, 1996). Nevertheless, digital libraries are now out of their infancy (Fox & Urs, 2002) and seem to be stabilising to the provision of a common set of features in different roles. These features include publishers’ allowing free searchable access to the titles and abstracts of journal articles but not to full text. As argued above, research communities are social organisations and are not solely driven by logic, so the possibility that something might happen or that it might be advantageous does not make a strong case that it will actually occur.

Refereeing is a core element of much scholarly communication. A referee could justifiably look for evidence that digital library search facilities have been used and that related research from other fields has been recognised and used as appropriate. Clearly the referee can check this using the same interfaces. With referees typically being established researchers, it is likely that there will be a time gap between the availability of the technology and its full assimilation into the practice of refereeing. Probably this time lag will also vary with discipline. Nevertheless, there are logical consequences that follow from the necessity for broad-based literature reviews.

Searching competence. Authors will need to be able to use current search interfaces effectively. This is a non-trivial task that typically involves keyword selection and perhaps Boolean queries. It is not clear how much librarianship training is appropriate for scholars for this purpose, but it is logical to assume that this should form part of higher education at some stage. The need for searching competence (c.f. Cruickshank, 2002) is a transparent outcome of digital libraries and one that dovetails with the general increase in the importance of electronic information. The importance of search skills in education is widely recognised, becoming a part of information literacy and triggering the publication of entire books devoted to various forms of online searching. Our finding serves to underline the centrality of online searching to the research process.

Increased serendipity. Literature reviews can now be expected to produce papers using materials from previously untapped sources. For example, an information scientist wishing to study personal home pages on the Web and conducting a traditional disciplinary literature review, perhaps targeting key information science journals to look for related articles, would tend to find only information science contributions. With serendipity he/she could perhaps also find one or two other fields discussing the same topic (Foster & Ford, 2003). A new style keyword search of digital libraries will produce different results. For example, the indexing software may scan tens of thousands of journals across the spectrum of disciplines and yield a very disparate set of matching publications, perhaps including some from computer science, gender studies, and sociology. Should the researcher choose to use the extra articles from outside of his/her field, he/she will produce a paper with a much broader scholarly base.

Two Types of Serendipity

For simplicity of argument, this study differentiates between two extreme types of serendipitous input to research: fact-based and theory-based.

  • A fact from a different research field could be taken at face value and incorporated into the new research as supporting evidence or may connect two phenomena in such a way as to suggest new solutions to a problem. In either case, the researcher does not necessarily need to fully understand the research (e.g., theories, experimental apparatus, and design) that produced the results.
  • The discovery of relevant theory (e.g., a thesis or a model) from a different field imposes a more daunting task on the author: to assimilate philosophy from areas of scholarship with which he/she is unfamiliar. An academic conducting a particular piece of research may have limited time in which to attempt to understand the new theory. This may be a difficult task if the theory draws upon field-specific or subject-specific knowledge and understanding that would be standard for investigators already in the field.

Both types of serendipity may have an important impact upon research in terms of helping to generate better answers to research questions, but theory-based serendipity also imposes requirements upon researchers and calls for skills that were not necessarily needed for research in the past (or perhaps not so frequently). It would not be reasonable to expect an author to plough through the equivalent of a new degree course in order to fully understand an article from a field that overlaps with the author’s area of interest, but it would clearly be helpful and perhaps necessary to have background knowledge of the field or subject. This is potentially a problem that will occur with increasing frequency. The individual academic facing this problem has three possible solutions:

  • Ignore the new research
  • Use the new research without fully understanding the theory
  • Take the time necessary to understand the new theory
  • Identify a collaborator who can understand the new theory

The first two options are clearly undesirable and should be stopped by journal referees. If this problem is a frequent one, then future researchers may need to have (a) a broader basic education than is now the case, particularly in the UK’s specialised education system (something that may itself be changed [BBC, 2003]) and/or (b) expertise to collaborate effectively with researchers in other fields. In practice, the second option also seems likely to require broader academic knowledge. This would be in line with the apparent general tendency at the end of the twentieth century toward greater interdiscipliniarity (Gibbons, Limoges, Nowotny, Schwartzman, Scott & Trow, 1994). Of course, interdiscipliniarity in groups and individuals is not a new phenomenon. The claim being made here is that digital libraries are accelerating this phenomenon to the extent that it can be seen as the norm for research. As cultural studies indicate, changes in technology and in its use tend to evolve in cycles. Consequently, if the broadening of research is not perceived to be successful, this may lead to a reversion to a more narrow typical focus. Since disciplinary differences do greatly impact the adoption and use of technology (Kling & McKim, 2000), it is likely that the changes discussed above will vary in practice among fields and disciplines.

Summary and Conclusions

Two consequences of the modern generation of broad-based, partly open-access digital libraries are likely to be an increased need for the searching skills to use them and for the interdisciplinary skills to author articles, even when the topic is apparently firmly within a single field. The latter phenomenon is the more fundamental. The stereotypical product of Kuhn’s (1970) “normal research” is an article expressed within a single paradigm, whereas interdisciplinary research calls upon multiple paradigms that may partially contradict each other. This is a poststructuralist phenomenon (Foucault, 1972) and is in keeping with other trends away from monolithic research such as the move from positivist or constructivist to mixed-model methodologies (Tashakkori & Teddlie, 1998). The emphasis in the mixed-model methodology is on selecting techniques to achieve the researcher’s goals. The broad-based literature review offers the potential to select fields that support these goals, but this would be poor research. Better practice would be to incorporate all relevant findings but to critically analyse them to assess the impact of those that are in conflict with the goals. Such a critical analysis involving a range of different fields is likely to be a non-trivial task and may be more onerous for the humanities scholar than the scientist because of the latter’s greater reliance upon facts. It would be possible to claim that it will make research more difficult to conduct, but this would not be in keeping with the concept of research as a discourse culture with concepts such as “good enough” rather than “perfect” or “absolute truth.” The reality, then, is more likely to be that the most effective researchers of the future will have a different skill set than those of today. Perhaps there is an advantage to be gained by the scholars who recognise this first and particularly for universities that incorporate training on how to critically evaluate research from a wide range of fields into their doctoral research methods courses. As a final point, from a historical point of view, each new generation of researchers enters a different world, needing a different optimal skill set. Researchers and research managers need to recognise the necessary changes in a timely manner and respond appropriately.

References

Almind, T.C., & Ingwersen, P. (1997). Informetric analyses on the World Wide Web: Methodological approaches to “webometrics.” Journal of Documentation, 53(4), 404-426.

Arms, W.Y. (2001). Uniform resource names: Handles, PURLs, and digital object identifiers. Communications of the ACM, 44(5), 68.

BBC. (2003). Baccalaureate could replace A-levels, available at: http://news.bbc.co.uk/1/hi/education/2680099.stm

Bessant, J., Birley, S., Cooper, C., Dawson, S., Gennard, J., Gardiner, M., Gray, A., Jones, P., Mayer, C., McGee, J., Pidd, M., Rowley, G., Saunders, J., & Stark, A. (2003). The state of the field in UK management research: Reflections of the Research Assessment Exercise (RAE) panel. British Journal of Management, 14(1), 51-68.

Borgman, C., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science & Technology36, 3-72.

Bradford, S.C. (1934). Sources of information on specific subjects. British Journal of Engineering, 137, 85-86.

Cruickshank, J. (2002). The role of scientific literature in scholarly communication. Science & Technology Libraries, 22(3-4), 71-100.

Davis , M. (1997). Scientific papers and presentations. London : Academic Press.

du Gay, P., Evans, J., & Redman, P. (2000). Identity: A reader. London : Sage.

du Gay, P., Hall, S., Janes, L., Mackay, H., & Negus, K. (1997). Doing cultural studies: The story of the Sony Walkman. London : Sage.

Faigley, L. (1986). Competing theories of a process: A critique and a proposal. College Composition and Communication, 48, 527-542.

Foster, A., & Ford, N. (2003). Serendipity and information seeking: An empirical study. Journal of Documentation, 59(3), 321-340.

Foucault, M. (1972). The Archaeology of Knowledge. NY: Pantheon.

Fox, E.A., & Urs, S.R. (2002). Digital libraries. Annual Review of Information Science and Technology, 36, 503-589.

Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178, 471-479

Geertz, C. (1983). Local knowledge: Further essays in interpretive anthropology. New York : Basic Books.

Gibbons, M., Limoges , C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The New Production of Knowledge. London : Sage.

Gross, A.G., Harmon, J.E., & Reidy, M.S. (2002). Communicating Science: The Scientific Article from the 17th Century to the Present. Oxford : Oxford University Press Inc.

Halliday , L., & Oppenheim, C. (2001). Developments in digital journals. Journal of Documentation, 57(2), 260-283.

Harnad, S., & Carr, L. (2000). Integrating, navigating, and analysing open eprint archives through open citation linking (the OpCit project). Current Science, 79(5), 629-638.

Hess, D.J. (1995). Science and technology in a multicultural world: The cultural politics of facts and artefacts. New York : Columbia University Press.

Hine, C. (2000). Virtual Ethnography. London : Sage.

House, E.R. (1994). Integrating the quantitative and qualitative. In: C. S. Reichhardt & S. F. Rallis (Eds.) The qualitative-quantitative debate: New Perspectives. San Francisco : Jossey-Bass (pp. 13-22).

Howell , D.C. (2002). Statistical methods for psychology. Duxbury: Pacific Grove , USA .

Hyland, K. (2000), Disciplinary discourses: Social interactions in academic writing. Harlow : Longman.

Hyland, K. (2003). Self-citation and self-reference: Credibility and promotion in academic publication. Journal of the American Society for Information Science and Technology, 54(3), 251-259.

Jascó, P. (2003). Savvy searching. Online Information Review, 27(3), 207-210.

Kling, R., & McKim, G. (2000). Not just a matter of time: Field differences in the shaping of electronic media in supporting scientific communication. Journal of the American Society for Information Science, 51(14), 1306-1320.

Kuhn, T. (1970). The Structure of Scientific Revolutions. (2nd. edition). Chicago : University of Chicago Press.

Latour, B., & Woolgar, S. (1986). Laboratory Life: The Construction of
Scientific Facts. 2nd ed
. Princeton : Princeton University Press.

Magueijo, J. (1998). Electronic archives and the death of journals, available at: http://theory.ic.ac.uk/~magueijo/com.pdf

McNay, I. (2003). Assessing the assessment: An analysis of the UK Research Assessment Exercise, 2001, and its outcomes, with special reference to research in education. Science and Public Policy30(1), 47-54.

Meadows, A.J. (1998). Communicating research. Boston : Academic Press.

Merton, R.K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago : University of Chicago Press.

Nicholas, D., Huntington, P., & Watkinson, A. (2003). Digital journals, Big Deals and online searching behaviour: A pilot study. Aslib Proceedings55(1/2), 84-109.

Price, D. J. (1963). Little science, big science. Columbia University Press.

Rees, M. (2002). Not worth the paper. New Scientist, 2370, 27.

Small, H. (1999). Visualising science through citation mapping. Journal of the American Society for Information Science, 50(9), 799-812.

Storey, J. (1996). What is cultural studies?: A reader. London : Edward Arnold.

Swales, J. M., & Lindemann, S. (2001). Teaching the literature review to international graduate students. In: A. Johns, (Ed.), Genre in the classroom: Multiple perspectives. Mahwah , NJ : Lawrence Erlbaum.

Swanson , D.R., & Smalheiser, N.R. (1997). An interactive system for finding complementary literatures: A stimulus to scientific discovery. Artificial Intelligence, 91(2), 183-203.

Tashakkori, A., & Teddlie, C. (1998). Mixed methodology. London : Sage.

Vaughan, L., & Shaw, D. (2003). Bibliographic and web citations: What is the difference? Journal of the American Society for Information Science and Technology, 54 (14), 1313-1322.

Wager, E., Godlee F., & Jefferson, T. (2002). How to survive peer review. London : BMJ Books.

Weigold, M.F. (2001). Communicating science: A review of the literature. Science Communication, 23(2), 164-193.

Withey, R. (2003). (Mis)understanding the digital media revolution. Aslib Proceedings55(1/2), 18-22.

Woolgar, S. (1988). Science: The very idea. London : Routledge.