View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Workshop onQuality/Selectivity of the DLESE
Collections
• Framing the Question• History of the Discussion
Kim Kastens, June 30, 2003
Framing the Question
• DLESE has Broad Collection and a Reviewed Collection. We are (mostly) talking about the Broad Collection.
Framing the Question
• Resources enter the DLESE Broad Collection via two routes:
– Individually, via the DLESE Cataloging Tool (the “Community Collection”)
– As part of an aggregated or themed collection, a collection accessioned into DLESE in its entirety.
• We are concerned with quality and relevance of resources entering via both routes.
Framing the Question
• Anyone can submit a resource to DLESE via the cataloging tool, which is an open set of web forms.
• This has given rise to concerns that “junk” could get into the DLESE Collections.
Framing the Question
• This workshop needs to make recommendations on two issues:
– What should be the criteria by which resources are approved for inclusion in the DLESE Broad Collection?
– What should be the procedures by which these criteria are implemented?
Framing the Question: Criteria
• Resources submitted for the DLESE Broad collection currently must meet two criteria:
– The resource is relevant to Earth System Education
– The resource works (i.e. it has no conspicuous bugs).
Framing the Question: Criteria
• Other possible criteria that have been suggested for the DLESE Broad Collection:
• No cost or low cost for educational users
• Resource is in English
• No commercial message
• No intrusive advertising
• No blatant religious message
• No blatant political message
• No blatant errors of fact
• Educational effectiveness
• Well documented
Framing the Question: Procedures
• With respect to procedures, we have two issues:
– By what process shall we identify problematic resources?
– What shall we do when we find a problematic resource?
Framing the Question: Procedures
• By what process shall we identify problematic resources?
– Ask the resource contributor (current system)?
– Screening by the community? – Screening by paid staff?
Framing the Question: Procedures
• What shall we do when we find a problematic resource?
– Exclude it from the collection?
– Include it in the Broad Collection but annotate it?
This is what an annotation might look like in theDiscovery System:
Framing the Question: Procedures
Screening bypaid staff
Screening bycommunity
Problematicresourcesexcluded fromlibrary
1 2
Problematicresourcesannotated inDiscoverySystem
3 4
History of the Discussion
• Coolfont: August 1999
– Collections Policy drafted
– There shall be a Reviewed Collection and an “Unreviewed” Collection
– Collection Committee established
History of the Discussion
• Coolfont: August 1999 (cont’d)
– Rationale for Reviewed Collection: • Users’ Perspective: “…. recognized, efficient source for
quality teaching and learning materials.”
• Creators’ Perspective: “…. a recognized stamp of professional approval at the level of publication in a peer-reviewed journal.”
– Rationale for the “Unreviewed” Collection:• “Users are seeking materials on a huge range of topics.
The DL provides added value by being inclusive while providing powerful search and classification capability.”
History of the Discussion
• Coolfont: August 1999 (cont’d)
– Criteria for Reviewed Collection:
• Accuracy, as evaluated by scientists• Importance/significance• Pedagogical effectiveness. • Well documented. • Ease of use for students and faculty• Inspirational or motivational for students• Robustness/sustainability
History of the Discussion
• Coolfont: August 1999 (cont’d)
– No Criteria established for “Unreviewed” Collection
– After debate, it was decided that there would be a human-mediated step between submission of resource and ingestion into library.
History of the Discussion
• Spring 2000: Academic Career Recognition Task Force Web Survey
– Seven selection criteria for the Reviewed Collection met approval of prospective DLESE users, resource creators, and department Chairs.
History of the Discussion
• Mid-late 2000: Collecting began
– DPC: testbed collection for exercising metatdata framework
– Montana State: Dave Mogk & students
– Foothill College: Chris DiLeonardo & students
History of the Discussion
• October 2000: Collections Meeting at Boulder:
– DLESE Community Cataloger tool introduced to non-DPC collecting groups (AGI, Montana State, others?)
History of the Discussion
• November 2000: Steering Committee Meeting at Lamont:
– Contentious discussion about “filters” at the gateway to the Broad Collection
– Agreement on only two of the discussed “filters”: (1) relevant to Earth System Education(2) “It works”, e.g. no conspicuous bugs
- Contentious discussion of how to apply “filters”; clarity seemed to emerge when John Snow described a “holding tank” system used in his history group.
History of the Discussion
• Nov-Dec 2000: Steering Committee Meeting at Lamont (cont’d):
- Meeting Minutes: • “The general concept of a 30-day public comment period on new
resources was agreed to. This will allow a time for the community to review resources….”
• “In the short term, partners collecting resources …. will review them to make sure they are appropriate
• “….the Collections Committee, collection proposal team and the DPC will work together to investigate mechanisms for encouraging review….”
History of the Discussion
• February 2001 Collections Meeting:
– Joint meeting of Collections Committee, “Collections Partners”, and Community Review System Editorial Review Board
– DLESE Community Cataloging Tool open to the world
– Collections Committee drafted Deaccession Policy
History of the Discussion
• February 2001 Collections Meeting (cont’d):
– Collections Committee discussed “filters” at gateway to Broad Collection. Imperfect consensus:
• Relevance Filter – Is the resource relevant to Earth System Science education?
• Integrity Filter – Are there no blatant errors of fact in the resource? – Are there no blatant political, religious, or commercial messages in
the resource? – Does it function reasonably; i.e., seem to be basically bug-free?
History of the Discussion
• April 2001 Steering Committee meeting at Biosphere 2:
– Collections Committee/DPC Collections group presented fleshed out version of the “holding tank” or “provisional status” plan.
– Many questions and issues. Who are reviewers? How mobilized and overseen? No $ to oversee the “army of filterers.”
– No resolution.
History of the Discussion• July-Aug 2001 Steering Committee meeting at Flagstaff:
– 850 resources in library. Metadata QA streamlined.
– “Mike Mayhew indicated a concern …about the broad collection. ….Where is the quality control in developing the collection? Do we dilute the value of library with variable quality?”
– Holding tank idea revisited, in simpler form without “designated reviewers”
– Action item: “Boyd …. will develop a draft proposal/set of guidines to implement a holding tank in which resources are discoverable in the system and identified as accessioned within a 30-day period with some mechanism to accept comments. The proposal for implementation will not include a designated reviewer”
History of the Discussion
• February 2002 Steering Committee meeting at Boulder:
– Draft Collections Accession Policy presented
– Revised throughout spring
– DLESE oversight would be review of review process, rather than review of individual resources
History of the Discussion
• July 2002 Steering Committee & Annual meeting at Cornell:
– Deaccession Policy approved– Interim Collection Accession Policy approved– First annotation service demo’d within DLESE– Faulker reported that NSDL content philosophy was:
“Educational value …to be manifest in capabilities for annotation and selective filtering, rather than an accession threshhold”
– Possibility raised that annotation option might be solution to ongoing dilemma about quality of DLESE Broad Collection.
History of the Discussion
• Fall 2002:
– Sumner et al focus group study of Educators’ perceptions of Quality.
– Best Practices for Resources summited to the DLESE Reviewed Collection begins to take shape.
History of the Discussion
• Spring 2003:
– Ad hoc Collections group met in Boulder, worked on how to implement Interim Collections Accession Policy and on Pathways to Reviewed Collection document
– June 13: 12 collections met documentation requirements to be accessioned as collections.