'How do we evaluate this?' : Perspectives on evaluation criteria for digital scholarship from the digital humanities community

Since the advent of the World Wide Web, there has been an increasing influx of digital scholarship. Such scholarship is not always recognized as legitimate, in part because digital work is still in its 'incunabula phase' and also because the staggering variety in tools, user communities, etc. engenders a host of potentially competing evaluation priorities. These concerns have created a pressing need for appropriate evaluation criteria to fairly assess digital projects. Though this topic has received substantial attention in the scholarly literature, discrete solutions and the establishment of firm yet flexible evaluation criteria remain elusive. This paper presents a pilot study that sought to clarify the following: what criteria participants use to evaluate digital scholarship, the place of digital tools in the evaluation of scholarship, who should evaluate digital projects, the role of stated intentions in the formation of evaluation criteria, what role the TEI might play in evaluation of text encoding, and finally how this role would be practically implemented. The study indicated that despite the complex nature of the topic, a number of practical solutions may aid in the legitimization of digital scholarship. In particular, including a statement of intent that explains the methodology of the project goes a long way in establishing the relationship between the content and the tools and the criteria to evaluate both components. Two potential roles for the TEI community also emerged: (1) to provide counsel and formative assistance with ongoing projects in a manner targeted towards project evaluations and (2) to consider including dedicated reviews section in the Journal of the Text Encoding Initiative to feature project evaluations and accept submissions for review. This publication is an ideal online platform for the discussion of review guidelines and may help to clarify what evaluation criteria are necessary to promote fair and accurate assessments of digital projects. Determining what to evaluate and how to do so are perennially relevant questions, and as digital scholarship continues to develop, it will become more important than ever to develop a better of understanding of what we value and why we value it.