If you are forced into evaluating scholarly work, consider the Linguistic Society of America‘s resolution on annotated language documentation materials (and see the RNLD list on this).
“Therefore the Linguistic Society of America supports the recognition of these materials as scholarly contributions to be given weight in the awarding of advanced degrees and in decisions on hiring, tenure, and promotion of faculty. It supports the development of appropriate means of review of such works so that their functionality, import, and scope can be assessed relative to other language resources and to more traditional publications.”
To which add grant applications [ update [ tx Claire!]- as places where track records get brutally evaluated by strangers ].
An article in an edited conference proceedings published by a scholarly vanity press counts for something. But it counts for NOTHING to have a carefully recorded, annotated, translated and archive corpus of recordings from say a year’s fieldwork, which would have taken at least a further year to create. The one creates something that most people will never read; the other creates something whose value is enduring. Result: scholarly vanity presses are springing up to cater for people’s need for publications. Possible solution: set up a scholarly vanity press for ‘publishing’ corpora. Call them books?
Follow
Jane I’m not sure what the “to which add grant applications” refers to here. I’m not sure that grant applications should count; grants clearly should (and do), but the applications themselves? Isn’t that a little like saying that rejected articles should also count for tenure?
Lincom is essentially a vanity press and they publish text collections. I get the impression though that they are sufficiently much of a vanity press that publishing with them doesn’t count, since they are totally non-reviewed and not even cursorily edited.
For what it’s worth (and I don’t think this is a widely held view, but I’ll say it anyway), I think we need to be really careful with arguments like “this took a year to produce”. That takes the argument away from the value of the materials and the thought and theory they take to produce, and pushes the argument towards the hours alone. That makes the argument more similar to the one we all see from students, that “I put a lot of time into this and I don’t think my grade reflects my effort”. (I don’t care how much time they put into it, it’s getting the right answer that counts, though that may be increasingly pre-modernist, absolutist and downright old-fashioned…)
Oops! I was quite unclear. I MEANT to add evaluating someone’s track record in refereeing a grant application to the list of places where track record is evaluated.
Yes there needs to be some kind of independent measure of corpora of annotated texts. But the value of many collections is in the presence of linked audio/video. So an online publisher would solve part of the problem. But many text collections can’t be published via online archives because of restrictions on distribution by the speakers. So perhaps the big digital archives could institute a rating system which could count as ‘publication’. Maybe using peer refereeing, together with a measure based on hours, detail, meta-data etc.
And I agree about the need for caution over hours. But I am happy to accept that annotating a hitherto un-documented language with an unusual phonology and lots of irregularity takes more time than annotating a German dialect, and that there must be some way of giving weight to that.
Clare and Jane: I agree that the issue of how to evaluate the scholarly merit of a documentary corpus is really interesting. In a field like Hittite, say, or Greek epigraphy, the people (archaeologists) who actually uncover the objects with primary language data on them don’t necessarily publish text corpora or even editions, but they publish excavation reports and the like. One strategy in our case might be to develop something (a journal? an online encyclopedia?) that has language-documentation counterparts of those excavation reports.
Jane: There are West Germanic dialects with interesting phonological features, and (now I’m wearing my Indo-Europeanist hat) I’d like to encourage good documentary or text-corpus work on endangered and distinctive dialects of even well-known languages (or languages that are better known than most endangered ones).
I couldn’t agree more about the value of having a collection of transcribed and linked material in West Germanic dialects. And I’d also agree that a collection with a very detailed narrow phonetic transcription would be a lot of work. What I was trying to find some measure for was assessing the amount of work involved in a particular corpus and assigning a value accordingly. Maybe this is not achievable – or not desirable. But there has to be a distinction between what could be done by an online transcription service, and what a linguist does in annotating and analysing texts and linking them to sound/video files
It seems (to me) that presenting and analyzing texts, and at the same time presenting the underlying source of the material (an audio file, a linguist’s legacy manuscript, a clay tablet), is the traditional work of text editing. Text editions can be published, and getting linguists credit for doing text editions shouldn’t be all that different from what’s customary in universities. (Sometimes linguists don’t value that kind of work, but I think that’s a different problem.) It seems (again, to me) that the more serious problem is getting scholarly credit for doing the documentation work: the work that’s analogous to digging up new texts. What if you’ve created a great documentary resource for the next text editor who comes along? How do you get credit for that? (Or should you?)
One possibility is a recognition not of the “digging up”, but of the fact that because “digging” is time-consuming, especially if done well, that has an effect on output of other types. I think that’s an important distinction. When I think of the time that goes into corpus creation and materials preparation, a lot of it is fairly basic formatting, cutting and pasting, etc. It doesn’t require much skill (beyond an ability to understand the materials, which is what makes it necessary for the linguist to do it and not a random undergrad or a “mechanical Turk”) but it’s a necessary evil, like assays or making reagents. Getting “credit” for it seems to me to make the job more about that, rather than that being the annoying and boring stuff we do to prep or postprocess the actual, interesting research. I guess the analogous item in theoretical economics would be how many notepads of algebra the person filled on the way to the published article.
I think we should not set up a “vanity press” for publishing corpora, but a rigorously refereed journal for publishing corpora. Technically that’s easy, and it shouldn’t be very expensive. Likewise, there should be a refereed journal for publishing dictionaries (see http://colab.mpdl.mpg.de/mediawiki/Living_Sources_in_Lexical_Description for a proposal).
This sounds like a brilliant well-thought-out proposal for something we could all benefit from. I guess it did not get funded when you applied first time round. Are you applying again?