Evaluating Digital Scholarship: Experiences in New Programmes at an Irish University
Michael Cosgrave, Anna Dowling, Lynn Harding, Róisín O’Brien and Olivia Rohan
While we have used digital research in teaching at University College Cork for many years, the central role played by digital artefacts in the new Digital Humanities programmes is a relatively recent addition. This pivotal shift is new for both staff and students who, by the nature of new media technologies, cannot benefit from generations of received wisdom on assessment and evaluation. In this piece, we undertake a frank and personal investigation from both a pedagogical and scholarly perspective.
Image credit: Róisín O’Brien
Experiences of Digital Humanities: A Shikari’s Perspective
Assessing digital artefacts in academe is driven by the problem of grading the digital work being produced in our Phd and MA programmes in Digital Arts and Humanities. When the DAH Phd consortium developed the proposal for a structured Phd, there was agreement that the outputs could include digital artefacts but no detailed discussion on what those might be and what criteria might be applied.
With that Phd in its second year and our own one year MA course in UCC in its first, this issue needs to be addressed in a very immediate way. Our students are currently working on digital products which we need to guide, and that guidance needs to be shaped by a clear awareness of what our expectations are, and how we will assess their digital work.
Disciplines not only have signature pedagogies, they also have signature assessments, and the skill of grading those is often handed down from generation to generation as an artisan craft. This is understood across the community of the discipline so external examiners have no problem validating the marks assigned in their discipline. Colleagues who have never needed to explicitly consider grade descriptors or grading rubrics find it difficult to conceive of how one might grade an as yet undefined assessed digital object.
Grading is part art, and never wholly science, but in an interdisciplinary field like digital humanities, where we must assess new types of student work, some frameworks are necessary. In the National University of Ireland, we have clear and well established guidelines in the NUI grade descriptors (PDF). The descriptors clearly lay out, in a general, non-discipline specific way, the sort of ‘evidence of a mind at work’ we should expect at various grade brackets.
The NUI descriptors require no real modification for application to essays and other traditional work. Following on from traditional rubrics for “regular” student essays, people have produced countless rubrics for assessing blog posts and posting in discussion forums. These are, after all, written work and different from traditional academic writing mostly in extent, and sometimes in the formality of tone or voice.
When we move into less conventional forms, matters become more complicated. How do you assess a database, a critical edition, a performance, a piece of multimedia or an ‘app’? The optimal manner to build a relational database is, at one level, defined by a set of normalisation rules which are pretty clear and preclude, it seems, developing an argument. At another level, the choice of data to capture, the varieties of datatypes, indices and relations, are all driven by the questions which are informed by the particular inquiry being pursued.
Digital Artefacts — databases, corpora, and other things — designed by different students for different inquiries can and should differ. One would expect to see individual choices about analysis and synthesis of the material to suit particular analytical questions being asked of the original raw, pre-digital material.
A common criticism of digital work is that “if it were a proper academic essay, it would of course have footnotes and so on” as if it was not a proper academic essay. But a work like ‘The Phoenix Tapes” are a very fine collection of six essays on Hitchcock: they choose themes, extract examples, arrange those examples in a structure which includes a clear progression from introduction through development, to show the often horrific end result of the obsessions, highlighting along the way the manner in which Hitchcock visually expresses these themes.
If we skate over the detail that the original footage was not digital, the Phoenix tapes are an artefact, an essay of sorts, albeit in a medium which doesn’t easily permit footnotes. Nevertheless, submitted with a copy of the script, including references, they should, under the NUI grade descriptors, merit a clear first class mark.
Part of our problem with assessment of digital artefacts is that many academics have never explicitly considered their instinctive grading rubrics. When challenged in discussion on a particular essay, most academics can explain why they gave it the assigned mark, but do not have a set of grading rubrics to hand, nor do they provide students with copies of grading rubrics at the start of courses.
As leaders in digital humanities, we are asking our students to leave accepted pathways and march into the desert; we have a responsibility to know enough to help them draw a new map.
Experiences of Digital Humanities: An apprentice’s perspective
Introduction
This piece will focus on personal experiences of recent entrants into digital humanities scholarship thus far, setting them in a framework of evaluating digital scholarship as learners. It will highlight the challenges faced by digital humanities novices in assessing scholarly literature. It will also refer to the digital tools utilized in scholarly endeavours and publications, while synopsising how these have affected learning to date. Instead of focusing on the technical shell of standardising evaluation and assessment of digital scholarship, this piece will concentrate on the innards of the issue; that is to say the main principle of a free-form approach which will guide evaluation as opposed to regulating it.
This is particularly indicative in the language used to discuss and communicate within the current digital humanities, as well as the audience’s reading when engaging with, not only the written material, but meaning-making in general.
Current debates in the field of the digital humanities about the divergent practices of ‘close’ and ‘distant’ reading are really a screen for deeper changes called for by the advent of new media. Digital technologies do more than propose new ways of thinking, as did theory; they require new modes of being (Schreibman 126).
Digital Humanities Identity: Different Voices
The 2010 Digital Humanities Conference saw Kathleen Fitzpatrick define digital humanities as a “nexus of fields within which scholars use computing technologies to investigate the kinds of questions that are traditional to the humanities, or . . . who ask traditional kinds of humanities-oriented questions about computing technologies”. It is this unrestricted classification that makes digital humanities such an attractive field, but also, unsurprisingly, presents problems for students who wonder whether their contribution is a valid one. For instance, words like collaboration, TEI, temporality, remediation, and xml have been passed back and forth in our classes, while taking for granted the fact that we were all aware of their meanings.
Depending on who you read, digital humanities is a minefield: it can be riddled with “charlatanism . . . that . . . undersells the market by providing a quick-and-dirty simulacrum of something that, done right, is expensive, time-consuming, and difficult” (Unsworth); consequently, the most earnest of students, worried they may be tarred with such a brush, can experience what Mullen terms “digital-humanities impostor syndrome”. The desire to be certified and qualified in something jars with the field’s characteristic lack of structure, and many students require an awareness of the workings of assessment and evaluation to feel secure in their chosen area.
Arguably, trying to find a sense of identity in the digital world presents one of the main struggles which has led to the emergence of Digital Arts and Humanities. After all, the Humanities need to carve out a digital persona just as much as any large corporation, in order to make their presence felt in an ever-changing digital atmosphere. There is a sense of not wanting to be left behind evident in this move into the realm of IT. Jaron Lanier explores this concern in his book, You Are Not a Gadget. His concluding thoughts express this Humanist need to stay true to oneself while entering into the digital:
The most important thing about postsymbolic communication is that I hope it demonstrates that a humanist softie like me can be as radical and ambitious as any cybernetic totalist in both science and technology, while still believing that people should be considered differently, embodying a special category.
Encountering evaluation
From the beginning of the Masters Degree in Digital Arts and Humanities, an open-minded approach to scholarship was promoted. A first encounter was with digital publications of literature, and the second was in the setting up of academic blogs for the purpose of open, online discourse.
It was at these two points in this year’s scholarly endeavour that the issue of criteria for standards of evaluation was raised, namely:
- What are the standard practices in digital scholarship?
- What is the “right” form of best practice with regard to these standard practices?
- What are the standard practices for research in particular?
- How should a student evaluate standards of literature?
- What are the standards expected in our own online text communication in the higher level environment?
The opportunity exists to also set a precedent for intervarsity communication. If seized upon, and the result acknowledged by each institution, this chance could provide students with a wider pool from which to form connections, build projects, and review each other’s work on a structured basis (though still less formal than official journals), as the community grows cumulatively larger. In this collaborative sense, digital humanities encompasses the acknowledgement that the physical days of education can no longer stand alone as a means for learning.
Last October’s Digital Archiving in Ireland (DRI) National Survey of the Humanities and Social Sciences (PDF) saw one respondent state
[w]hen I see the word[s] Digital Repository Ireland, I would expect to find born-digital records are stored there and preserved there so that they can be migrated forward into new formats and then preserved and made accessible at the right time. And I really think that is where the gap is more than any other gap.
The demand is there for Ireland to stockpile and standardise — not homogenise — digital scholarly work. In relation to library archiving systems, the report notes one institution’s emphasis on being aware of any “broad national perspective on things…so if there are a lot of institutions moving…[in the same direction], we would move in a very coherent way.’” As Priego writes, “core critical and practical skills applicable to a wide variety of web tool scenarios would be a great thing to have a structured, recognised framework for.”
Digital tools
It soon became clear that standard practices in carrying out digital scholarship include technical skills and digital tools such as XML, TEI and databases. There is also an array of less daunting tools that are available to postgraduate students for research in any discipline. The problem lies in the fact that the sheer volume of digital tools can, at times, make the digital humanities realm awkward to navigate. To this end, a general, online, instructive directory, with general guidelines to popular software or particularly useful blogs would — though perhaps tedious to maintain and regularly update — be a great help. Self-directed learning, while expected at a postgraduate level, can become problematic if the student feels he or she is left without a map. With the impostor complex facing many at the beginning of their digital humanities explorations, how can we implement a structure that will reassure the budding digital humanist that, by the end of her studies, she will be qualified to actually do anything?
The solution, as with any set of tools, is to be discerning in one’s choices; rather than indulging in experimentation for its own sake and at the risk of confusion, one should build up to a knowledge of more esoteric tools. The following are some examples which have featured in our academic involvement this year, and which have a strong possibility of becoming standard practice for pedagogy and reflective learning outside of digital humanities, that is if their value can be demonstrated to institutions. When evaluating digital humanities, the benefits cannot be ignored.
Moodle, for example, acts as an online classroom and discussion forum, allowing a significant depth of reflection on course material. Similarly, Blackboard acts as a virtual library for course readings, as well as being another forum for discussion. Tools such as Skype and Google Hangouts enable the perimeters of the classroom to be endlessly extended; similarly, the eReader has now gained widespread availability. Even those who do not shop online are now inundated by its display in their local Tesco.
Clearly, the process of reading and learning has, to paraphrase Yeats, “changed, changed utterly”, but is this change a “terrible beauty” (The Collected Poems, 193) or welcome evolution? D. Randy Garrison’s E-Learning in the 21st Century summarizes the change in educational focuses which one can argue that digital humanities represents: “To be constrained by the restricted frame of traditional classroom presentational approaches is to ignore the capabilities and potential of e-learning” (54).
Process
As the first academic term nears its end, MA DAH students have already started to shed insecurities about personal judgement in assessing academic literature. Learning that the reader’s response to literature is not a trivial feature in terms of assessment of the quality of its contribution to a digital library was a formative experience. One may not feel practiced in the art of evaluation, however it is true to say that there is worth in every reader’s response. No matter what form the scholar’s interpretation takes, the exercise of assessing literature for its scholarly worth is a vital part of the process of handing responsibility back to the learner and relates directly to the vital strive for experiential learning. The student not only learns how to identify insightful literature, but also takes the early steps in laying a foundation for their own autonomous learning.
What does all this mean for the evaluators? How can any individual possess the abilities to work and evaluate across such a broad spectrum of practise? The issue of interpretation and intended meaning requires further interrogation and greater development through discourse. According to Schreibman, Mandell, and Olsen (2011), humanities scholars are, for the most part, “ill equipped … to recognize the scholarship” or the “intellectual content” of projects in which theoretical and technical choices inform project design. This issue is highlighted in Clement’s “Half-Baked; The State of Evaluation in the Digital Humanities”, in which she asserts that academic works relating to evaluation in the digital humanities have given rise to “a conversation that has very few listeners or readers in the humanities capable of appreciating the scholarship represented in this interdisciplinary work” (2012). This is supported by Browner (2011), who states that“[o]wning a computer and being able to click on a link is only the first and perhaps most easily addressed issue in assuring a real democracy of knowledge. Having intellectual access is much harder”.
On that note, there are three main features of scholarship which encapsulate the main characteristics of the process of learning in the backdrop of web 2.0. Each one is spurred by the use of digital tools, such as the aforementioned blogs. These are:
- Independent learning
- Collaboration
- Insightfulness
In order to assess the understanding of a scholar, one must also assess their cognitive presence, both in terms of critical thinking and discourse. An important point of reference in evaluating this is the Practical Inquiry Descriptors and Indicators model, as illustrated below and in D. Randy Garrison’s E-Learning in the 21st Century – A Framework for Research and Practice (52).
Practical Inquiry Descriptors and Indicators from “E-Learning in the 21st Century: A Framework for Research and Practice” by D. Randy Garrison (2011).
Garrison suggests that “practical inquiry is the model within which we operationalize and assess cognitive presence” (51). The aim is to offer “a practical means to judge the nature and quality of critical reflection and discourse in a community of inquiry”. The question of standard is, understandably, a hot topic for the budding digital humanist in particular, given that there can be such disparity between articles, studies or blog entries that all file themselves under the same digital humanities umbrella. By using a related, but less fixed, model in evaluating digital scholarship, we can tread the middle ground between a laissez-faire stance and a forced setting in stone of standards. One should not establish a system of rating but as an alternative examine the qualities already expected from scholarship and simply allow these to standards to homogenise in a digital setting.
Accessibility is a characteristic that must be stamped on digital scholarship. This applies to scholarship both in terms of publication of literature, and also the accessibility of data for the purposes of XML analysis. Without accessibility of data, information is as useful as a piece of chalk on an interactive whiteboard. The notion of access must be a guiding light for one’s own academic goals, to make a conscious move to live and breathe accessibility, thus exposing one’s work for the theoretical benefit of the academic community and allowing standards to grow from it.
Not only must the situation of ‘intellectual access’, or as Stefiks terms it “sensemaking”(Liu, 2011), be remedied through education and an increased academic and industrial awareness, but a more urgent predicament must be answered, in fact demands a response; “a reader could easily ask of these books what humanities scholars everywhere consistently ask of digital humanities writ large: So what? Is that it? And what does this have to do with our research?” (Clement, 2012).The origin of this issue of relevancy may stem from a predicament identified by Bartscherer and Coover (2011): “scholars and artists understand little about the technologies that are so radically transforming their fields, while IT specialists have scant or no training in the humanities or traditional arts”. Is it any wonder that a difficulty has arisen with regards to evaluation and perceived value within the digital humanities?
Previous university graduates will have had an understanding of the requirements of a postgraduate degree. However, web 2.0 has modified the reality of higher level academia, and offered an opportunity to reshape the traditional structures in education. In the MA DAH at UCC, the digital evolution has been fully embraced as an appropriate setting for the rounded learning of a 21st century student in line with communities of practice. The campus is now both physical and virtual. Web 2.0, and social media in particular, has changed the reality of the academy into a virtual experience, with room for immediate distribution of relevant, up-to-date knowledge. Such practice is essential to the promotion of accessibility. It is up to scholars to harness the energy of web 2.0. However, the linking of scholarship with the digital realm is an individual choice that each researcher will have to make.
Collaborative-learning environments, and the learner as a sounding board for standards, may be the main catalysts in the development towards an empowered learner and an adequate set of learner-centred standards, as opposed to the decree of an elite crew. In other words, e-learning should be part of an organic process in terms of developing standards for evaluation of scholarship in all its digital manifestations. In order for this to be fully realised, one must begin at the first marker of making accessibility an integral part of the academic world, or, in other words, a widely accepted standard.
An area which stirs up an array of controversy is the use of blogging within digital humanities, and indeed education in general. Issues exist surrounding the question of whether the blog can be considered a legitimate tool for research, or citation in an academic paper. The reality is that an amount of time and effort, equivalent to that which is being put into scholarly publications, is now being directed into the blogosphere. Alan Liu offers an entry point for such examples of social media to become more respected:
In the digital humanities, cultural criticism — in both its interpretive and advocacy modes — has been noticeably absent by comparison with the mainstream humanities. . . . How the digital humanities advance, channel, or resist the great postindustrial, neoliberal, corporatist, and globalist flows of information-cum-capital, for instance, is a question rarely heard in the digital humanities associations, conferences, journals, and projects with which I am familiar (Where is Cultural Criticism in The Digital Humanities, np).
Engagement with more thoughtful scholarship which directs itself towards cultural criticism could strengthen the consideration of blogs and other social media tools for digital humanities scholarship, through the fusion of discussions of data use with cultural commentary. Social media is fast becoming the leading publishing house for new material. Could one go so far as to argue that web 2.0 is the Humanities life-support system?
Evaluation
In terms of our experience of evaluation, there are several sides to evaluating literature, many of which were encountered through e-learning by the questions that were raised:
- How does one identify insightful literature, and how do we develop insight in our own online contributions?
- What is the etiquette of text communication as a scholar?
- Will knowledge be diluted when presented in digital form?
- Will design distract from data?
In assessing digital scholarship, the following points are also taken into consideration:
- What is the relationship between literature and communities of practice?
- Is literature always best served in an Open Access type of environment?
- Is digital information impermanent or simply up-to-date?
Price (2011) takes this issue a step further when it is revealed that this issue of evaluation and perceived relevancy carries right up the academic ladder to “tenure and promotion committees [that] have a notoriously difficult time in the humanities with multi-authored projects (characteristic of digital humanities projects)”. Clement (2012) supports this, indicating “And so the game continues: players lay their claims on the table and the winner is the person who makes the claims deemed most insightful by respondents”. Another significant hurdle against progress in adopting a new system of evaluation is indicated by Browner (2011): that “the habits, biases, power centers, and economics that shaped print over the last 500 years are also shaping the digital world”.
There needs to be some synergy between disciplinary standards of governance and the increasing use of more liberal forms of research using technology. Julia Fraser conceives that “digital humanities as a whole has revealed precisely how interwoven and mutually consequential ‘technical’ and ‘disciplinary’ standards often are” (Collaborative Research in the Digital Humanities, 68). Digital humanities demands these sectors strike the right balance when merging in research.
Conclusion
Some of the questions raised in evaluating broad-spectrum scholarship can be applied to any form of learning. However, most disciplines now collide with the challenges and enhancements of digital scholarship. In the MA DAH, feedback from teachers and interactivity through virtual text and verbal discussion allows standards to form in an organic way. This is the radical crux of the argument: that defining digital standards in any form should come from those who both produce scholarly outputs and read them. To do this, one cannot simply hypothesise. Instead, one must tear down speculation and examine the plain evidence. One must then share this knowledge in order to establish a true form of best practice, as derived from pedagogical practices, and, perhaps more importantly, our own innate learning experiences. If one shares these models of learning, instead of theorising ad infinitum, one will be able to demonstrate their actual implementation on a personal level and therefore on realistic terms.
In reality, the basic criteria for assessment and evaluation will reflect the standards which have always existed in any form of scholarship, including a cohesive, well-formed argument, presented in an accessible manner. It is not for scholarship to be clinically assessed in any hypothetical way; rather, feedback can be drawn from an existing set of evaluation principles, and refined to establish a pattern of acceptable forms for the digital version. Creation, data, collaboration, innovation and publication appear in new media forms, but the core elements of best practice remain the same, regardless of the medium. While digital scholarship allows for an enrichment of existing principles, the most important category that we cannot neglect is again the accessibility of work to all interested parties, as this is where standards of evaluation and assessment are born.
Part of the digital humanities utopian view is that of a democratic world of collaborative, open source, non-hierarchical understanding. As Professor William Pannapacker conveyed, “with leaders who have never known a time when scholarship in the humanities wasn’t in crisis, digital humanities is moving us — finally — from endless hand-wringing toward doing something to create positive change throughout academe”. If we smother digital humanities and digital scholarship’s free-form, shape-shifting attributes by attacking scholarship and delving into the task of structuring standards, we are treading the dangerous ground of inhibiting the organic growth thereof, and consequently stifling digital scholarship and goals of accessibility. Although organisation is a necessary feature of scholarship, we first need to start with a hands-off approach and make adjustments along the way where necessary. After all, the internet began as a communication device — what if we had tried to tighten our grip and to slam it down with definitions of its existence? To use an artistic analogy, instead of enforcing a theme of standards, we need to move towards freeform brushwork and bring out the features of optical art, which mutates before our scholarly eyes.
Clement (2012) takes a positive posture with regard to “Interdisciplinary conversations, on the other hand, are much harder: they are fruitful and productive when, in our attempt to understand each other, we produce knowledge”, while Spiro and Segal (2011) when investigating the field of digital scholarship in American literature, observed that within digital humanities scholarship ‘using’ digital infrastructure provides for more innovative scholarship than ‘making’, and Judith Donath (2011) who likened the current changes in scholarship within the digital humanities as a mutation, where “the richness of life comes from a myriad of accidental yet advantageous mutations — at the cost of the many that failed. As we enter the digital era, we are able to program the level of risk we are willing to take with unexpected changes”.
Digital humanities is a field in transit. It is moving from a world of constraint to a world of scholarly freedom. So far, digital humanities appears to the novice to be a culture of collaboration and experimentation. Perhaps it is too early to pinpoint what exactly it is; or, perhaps, what new material regularly unfolds persists in proving the field too rich to be confined by definition. The solution for the novice may be to content herself, for now, with the uncertain process of trial and error. It is up to us all to work out how to navigate this transition in a thoughtful, cautious manner.
Originally published by Mike Cosgrave, Anna Dowling, Lynn Harding, Róisín O’Brien & Olivia Rohan on December 3, 2012.
Works Cited
Bartscherer, T., and R. Coover, eds. Switching Codes: Thinking Through Digital Technology in the Humanities and the Arts. Chicago: University of Chicago Press, 2011.
Browner, S.P. “Digital Humanities and the Study of Race and Ethnicity.” The American Literature Scholar in the Digital Age. Eds. A.E. Earheart and A. Jewell. Ann Arbor: University of Michigan Press and University of Michigan Library, 2011. 209-28.
Cohen, M. “Design and Politics in Electronic American Literary Archives.” The American Literature Scholar in the Digital Age. Eds. A.E. Earheart and A. Jewell. Ann Arbor: University of Michigan Press and University of Michigan Library, 2011. 228-49.
Deegan, Marilyn, and Willard McCarty. Collaborative Research in the Digital Humanities: A Volume in Honour of Harold Short. Surrey: Ashgate Publishing Ltd, 2012.
Earheart, A.E., and A. Jewell, eds. The American Literature Scholar in the Digital Age. Ann Arbor: University of Michigan Press and University of Michigan Library, 2011.
Garrison, D. Randy. E-Learning in the 21st Century: A Framework for Research and Practice. New York: Routledge, 2011.
Gold, Matthew K. Debates in the Digital Humanities. Minneapolis: University of Minnesota Press, 2012.
Lanier, Jaron. You Are Not A Gadget: A Manifesto. New York: Alfred A. Knopf, 2010.
Liu, Alan. “Where is Cultural Criticism in the Digital Humanities.” The History and Future of the Digital Humanities. Modern Language Association Convention. Los Angeles, 7 January, 2011.
—, “We Will Really Know.” Switching Codes: Thinking Through Digital Technology in the Humanities and the Arts. Eds. T. Bartscherer and R. Coover. Chicago: University of Chicago Press, 2011. 89-94.
Price, K. M. “Collaborative Work and the Conditions for American Literary Scholarship in a Digital Age.” The American Literature Scholar in the Digital Age. Eds. A.E. Earheart and A. Jewell. Ann Arbor: University of Michigan Press and University of Michigan Library, 2011. 9-26.
Rockwell, G. “On the Evaluation of Digital Media as Scholarship.” Profession 1 (2011): 52-68. <http://www.mlajournals.org/doi/pdf/10.1632/prof.2011.2011.1.152> (PDF)
Schreibman, S., L. Mandell, S. Olsen. “Evaluating Digital Scholarship: Introduction.” Profession 1 (2011): 123-201. <http://www.mlajournals.org/doi/pdf/10.1632/prof.2011.2011.1.123> (PDF)
Spiro L., J. Segal. “Scholars’ Usage of Digital Archives in American Literature.” The American Literature Scholar in the Digital Age. Eds. A.E. Earheart and A. Jewell. Ann Arbor: University of Michigan Press and University of Michigan Library, 2011. 101-24.
Yeats, W.B. The Collected Poems of W.B.Yeats. Ed. Richard J.Finneran. New York: Simon and Schuster, 1989.
About Michael Cosgrave, Anna Dowling, Lynn Harding, Róisín O’Brien, and Olivia Rohan
