Closing the Evaluation Gap

The Editors

With this fourth issue we wrap up the first year of the Journal of Digital Humanities, and with it, our first twelve months of attempting to find and promote digital scholarship from the open web using a system of layered review. The importance of assessment and the scholarly vetting process around digital scholarship has been foremost in our minds, as it has in the minds of many others this year. As digital humanities continues to grow and as more scholars and disciplines become invested in its methods and results, institutions and scholars increasingly have been debating how to maintain academic rigor while accepting new genres and the openness that the web promotes.

Some scholarly societies, universities and colleges, and departments have called for a redefinition — or at least an expansion — of what is considered creditable scholarship. There have been scattered initial attempts to understand how digital scholarship might be better assessed, but the editors of JDH felt, and many of our readers agreed, that there was not a single place to go for a comprehensive overview of proposals, guidelines, and experiences. We attempt to provide a single location here, with an issue and living bibliography that will grow as additional examples are published across the web.

We begin with an identification of the scope of the problem, some reasons for the difficulty assessing digital scholarship, and a call for action. First, Sheila Cavanagh explains how the expectations of traditional scholarship and the breadth of support required for successful and creative scholarly and pedagogical projects restrict younger scholars. Bethany Nowviskie suggests that modifying outdated modes of peer review to recognize and credit the intellectual and technical labor of the many participants who produce ambitious and collaborative projects will positively influence the evolution of scholarship writ large. The collaboratively-written “Call to Redefine Historical Scholarship in the Digital Turn,” led by Alex Galarza, Jason Heppler, and Douglas Seefeldt, was submitted as a formal request for the American Historical Association to recognize and address these particular issues.

In the next section, practitioners from across the academy and the world offer their perspectives on assessment and evaluation. Todd Presner, Geoffrey Rockwell, and Laura Mandell propose evaluation criteria specifically for tenure and promotion. James Smithies details a typology of digital humanities projects to ensure proper evaluation. Shannon Christine Mattern advises that the same detailed criteria used to evaluate multimodal work in her classroom can serve the larger academy. Zach Coble offers the view from the library, which is the home of many collaborators and creators of digital humanities projects. Finally, Sheila Brennan suggests that we further highlight the intellectual goals and achievements of digital humanities projects declared, but perhaps buried, in administrative documents and reports.

Several practitioners then offer their personal experiences with evaluation and assessment to help others in this unchartered territory. Mark Sample explains the approach to digital scholarship he used in his tenure and promotion case, while Katherine D. Harris offers her tenure and promotion statements as a resource for others. Finally, students at the new digital humanities program at University College Cork remind us that evaluation ultimately is meant to encourage conversation, so practitioners need to be involved directly in the definition of any standards.

Already some organizations and scholars have produced good beginning guidelines for assessment. The Modern Language Association in particular has solicited in-depth discussions among its membership and outside scholars who have long worked in new media on how to assess new forms of scholarship involving digital media and technology. Other institutions, such as the Organization of American Historians and the National Council on Public History, have made some entreaties to broaden the definition of scholarly communication that will require fleshing out in the years to come. We have reproduced some of that content at the end of this issue. We end the issue with a bibliography of additional suggested readings on the evaluation and assessment of digital humanities work.

For the broadest possible understanding of the assessment of digital scholarship, we asked the community to help us find good case studies, personal accounts, and departmental and institutional efforts. This issue brings the best of these into one place that we can continue to update as other guidelines and experiences are shared. We hope that scholars in digital humanities and related fields will be able to point to this volume of the Journal of Digital Humanities as a resource for digital assessment and a starting place for further conversations.

 

Daniel J. Cohen and Joan Fragaszy Troyano, Editors