Open Data

There are 8 posts filed in Open Data (this is page 1 of 1).

Are You Paying Attention?

Not for the first time, the glut of incoming information threatens to push out useful knowledge into merely a cloud of data. And there’s no doubt that activity streams and linked data are two of the more interesting things to aid research in this onrushing surge of information. In this screen-mediated age, the advantages of deep focus and hyper attention are mixed up like never before, since the advantage accrues to the company who can collect the most data, aggregate it, and repurpose it to willing marketers.

N. Katherine Hayles does an excellent job of distinguishing between the uses of hyper and deep attention without privileging either. Her point is simple: Deep attention is superb for solving complex problems represented in a single medium, but it comes at the price of environment alertness and flexibility of response. Hyper attention excels at negotiating rapidly changing environments in which multiple foci compete for attention; its disadvantage is impatience with focusing for long periods on a noninteractive object such as a Victorian novel or complicated math problem.

Does data matter?

The MESUR project is one of the more interesting research projects going, now living on as a product from Ex Libris called bx. Under the hood, MESUR looks at the research patterns of searches, not simply the number of hits, and stores the information as triples, or subject-predicate-object information in RDF, the resource description framework. RDF triple stores can put the best of us to sleep, so one way of thinking about it is smart filters. Having semantic information available allows computers to distinguish between Apple the fruit and Apple the computer.

In use, semantic differentiation gives striking information gains. I picked up the novel Desperate Characters, by Paula Fox. While reading it, I remembered that I first heard it mentioned in an essay by Jonathan Franzen, who wrote the foreward to the edition I purchased. This essay was published in Harper’s, and the RDF framework in use on harpers.org gave me a way to see articles both by Franzen, as well articles that were about him. This semantic disambiguation is the obverse of the firehose of information that is monetized from advertisements.

Since MESUR is pulling information from CalTech and Los Alamos National Laboratory’s SFX link resolver service logs, there’s a immediate relevance filter applied, given the scientists who are doing research in those institutions. Using the information contained in the logs, it’s possible to see if a given IP address belonging to faculty or department) goes through an involved research process, or a short one. The researcher’s clickstream is captured, and fed back for better analysis.  Any subsequent researcher who clicks on a similar SFX link has a recommender system seeded with ten billion clickstreams. This promises researchers a smarter Works Cited, so that they can see what’s relevant in their field prior to publication. Competition just got smarter.

Standards based way of description

Attention.xml, first proposed in 2004 as an open standard by Technorati technologist Tantek Celik and journalist Steve Gilmor, promised to give priority to items that users want to see. The problem, articulated five years ago, was that feed overload is real, and the need to see new items and what friends are also reading requires a standard that allows for collaborative reading and organizing.

The standard seems to have been absorbed into Technorati, but the concept lives on in the latest beta of Apple’s browser Safari, which lists Top Sites by usage and recent history, as does Firefox’s Speed Dial. And of course, Google Reader has Top Recommendations, which tries to leverage the enormous corpus of data it collects into useful information.

Richard Powers’ novel Galatea 2.2 describes an attempt to train a neural network to recognize the Great Books, but finds socializing online to be a failing project: “The web was a neighborhood more efficiently lonely than the one it replaced. Its solitude was bigger and faster. When relentless intelligence finally completed its program, when the terminal drop box brought the last barefoot, abused child on line and everyone could at last say anything to everyone else in existence, it seemed to me we’d still have nothing to say to each other and many more ways not to say it.” Machine learning has its limits, including whether the human chooses to pay attention to the machine in a hyper or deep way.

Hunch, a web application designed by Caterina Fake, known as co-founder of Flickr, is a new example of machine learning. The site offers to “help you make decisions and gets smarter the more you use it.” After signing up, you’re given a list of preferences to answer. Some are standard marketing questions, like how many people live in your household, but others are clever or winsome. The answers are used to construct a probability model, which is used when you answer “Today, I’m making a decision about…” As the application is a work in progress, it’s not yet a replacement for a clever reference librarian, even if its model is quite similar to the classic reference interview. It turns out that machines are best at giving advice about other machines, and if the list of results incorporates something larger than the open Web, then the technology could represent a leap forward. Already, it does a brilliant job at leveraging deep attention to the hypersprawling web of information.

How to Achieve True Greatness

Privacy has long returned to norms first seen in small-town America before World War II, and our sense of self is next up on the block.  This is as old as the Renaissance described in Baldesar Castiglione’s The Book of the Courtier and as new as twitter, the new party line, which gives ambient awareness of people and events.

In this age of information overload, it seems like a non sequitur that technology could solve what it created. And yet, since the business model of the 21st century is based on data and widgets made of code, not things, there is plenty of incentive to fix the problem of attention. Remember, Google started as a way to assign importance based on who was linking to who.

This balance is probably best handled by libraries, with their obsessive attention to user privacy and reader needs, and librarians are the frontier between the machine and the person. The open question is, will the need to curate attention be overwhelming to those doing the filtering?

Galatea 2.2 Galatea 2.2 Richard L. Powers; Farrar, Straus, Giroux 1995

Lock-in leads to lockdown

What goes up must come down. This simple law of gravity can been seen in baseball, and these days, the stock market.

As I attended the Web 2.0 conference in New York recently, I had occasion to ask Tim O’Reilly what he thought about libraries. “Well, OCLC’s doing some good things,” he said. I encouraged him to continue looking at library standards, as the 2006 Reading 2.0 conference pulled together a number of interesting people who have been poking at the standards that knit libraries and publishers together.

But the phrase Web 2.0, coined by O’Reilly, was showing signs of age. From the halycon days, where every recently funded website showed rounded corners and artful form submission fades, the new companies were a shadow of their former booth size. Sharing space with the Interop conference, Web 2.0 was the bullpen to the larger playing field.

Interoperability

What helps companies to grow and expand? Some posit that the value of software is estimated by lock-in, that is, the number of users who would incur switching costs by moving to a competitor or another platform.

In the standards world, lock-in is antithetical to good functioning. Certainly proprietary products and features play a role to keep innovation happening, but cultural institutions are too important to risk balkanization of data for short-term profits.

Trusted peers

It seems to me that curation has moved to the network level, and a certain amount of democratization is now possible. The cautions about privacy and users as access points are true and useful, but librarians and publishers have a role in recommending information, and this is directly correlated to expert use of recommender systems. Web 2.0 applications like del.icio.us for bookmarks, last.fm for music, and Twitter and Facebook for social networks provide a level of personal guidance that was algorithmically impossible before data was easily collectible.

Prior to last.fm’s 2007 purchase by CBS Music, public collective data about listening habits was deemed “too valuable” to be mashed up by programmers any longer. In the library world, there’s a unique opportunity to give users the ability to see recommendations from trusted people. Though del.icio.us does this quite well for Internet-accessible sources, there’s an opportunity extant for the scholarly publishers to standardize on a method. Elsevier’s recent Article 2.0 conte shows encouraging signs of moving towards a release of control back to the authors and institutions that originally wrote and sponsored the work.

In the end, though, companies that are forced to choose between opening up their data or paying their employees will not likely choose the long-term reward. Part of this difficulty, however, has been tied to the lack of available legal options, standards, or licenses for releasing data into the public domain. The Creative Commons project has pointed many people to defined choices if they choose to enable their works into the public domain or for reuse.

Jonathan Rochkind of Johns Hopkins University points out that “A Creative Commons license is inappropriate for cataloging records, precisely because they are unlikely to be copyrightable. The whole legal premise of Creative Commons (and open source) licenses is that someone owns the copyright, and thus they have the right to license you to use it, and if you want a license, these are the terms. If you don’t own a copyright in the first place, there’s no way to license it under Creative Commons.

The Open Data Commons has released a set of community norms for sharing data. This is a great step towards a standard way of separating profit concerns from the public good, and also frees companies from agonizing legal discussions about liability and best practices.

Standard widgets

If sharing entire data sets isn’t feasible, one practice that was nearly universal in Web 2.0 companies was the use of widgets to embed data and information.

In his prescient entry, “Blogs, widgets, and user sloth,” Stu Weibel describes the difficulty he had installing a widget, a still-depressing reality today.

Netvibes, a company that provides personalized start pages, has proposed a standard for a universal widget API. The jOPAC, an “integrated web widget,” uses this suggestion to make its library catalog embeddable in several online platforms and operating systems. Since widgets are still being used for commercial ventures, there seems to be an opportunity to define a clear method of data exchange. The University of Pennsylvania’s Library Portal is a good example of where this future could lead, as its portal page is flexible and customizable.

Perhaps a widget standard would give emerging companies and established ventures a method to exchange information in a way that promotes profits, privacy, and potential.

Here Comes Everybody Here Comes Everybody: The Power of Organizing Without OrganizationsClay Shirky; Penguin Press HC 2008

WorldCat•Read Online•LibraryThing•Google Books•BookFinder 

Mining for Meaning

In David Lodge’s 1984 novel, Small World, a character remarks that literary analysis of Shakespeare and T.S. Eliot “would just lend itself nicely to computerization….All you’d have to do would be to put the texts on to tape and you could get the computer to list every word, phrase and syntactical construction that the two writers had in common.”

This brave new world is upon us, but the larger question for Google and OCLC, among other purveyors of warehoused metadata and petabytes of information, is how to achieve meaning. One of the brilliant insights derived from Terry Winograd‘s research and mentoring is that popularity in the form of inbound links does matter for web pages, at least. In the case of all the world’s books turned into digitized texts, it’s a harder question to assign meaning without popularity, a canon, or search queries as a guide.

Until recently, text mining wasn’t possible at great scale. And as the great scanning projects continue on their bumpy road, the mysteries of what will come out of them have yet to emerge into meaning for users.

Nascent standards

Bill Kasdorf pointed out several XML models for books in his May NISO presentation, including NISO/ISO 12083, TEI, DocBook, NLM Book DTD, and DTBook. These existing models have served publishers well, though they have been employed for particular uses and have not yet found common ground across the breath of book types. The need for a standard has never been clearer, but it will require vision and a clear understanding of solved problems to push forward.

After the professor in Small World gains access to a server, he grows giddy with the possibilities of finding “your own special, distinctive, unique way of using the English language….the words that carry a distinctive semantic content.” While we may be delighted about the possibilities that searching books afford, there is the distinct possibility that the world of the text could be changed completely.

Another mechanism for assigning meaning to full text has been opened up by web technology and science. The Open Text Mining Interface is a method championed by Nature Publishing Group as a way to share the contents of their archives in XML for the express purpose of text mining while preserving intellectual property concerns. Now in a second revision, the OTMI is an elegant method of enabling sharing, though it remains to be seen if the initiative will spread to a larger audience.

Sense making

As the corpus lurches towards the cloud, one interesting example of semantic meaning comes in the Open Calais project, an open platform by the reconstituted Thomson Reuters. When raw text is fed into the Calais web service, terms are extracted and fed into existing taxonomies. Thus, persons, countries, and categories are first identified and then made available for verification.

This experimental service has proved its value for unstructured text, but it also works for extracting meaning from the most recent weblog posting to historic newspapers newly scanned into text via Optical Character Recognition (OCR). Since human-created metadata and indexing services are among the most expensive things libraries and publishers create, any mechanism to optimize human intelligence by using machines to create meaning is a useful way forward.

Calais shows promise for metadata enhancement, since full text can be mined for its word properties and fed into taxonomic structures. This could be the basis for search engines that understand natural language queries in the future, but could also be a mechanism for accurate and precise concept browsing.

Glimmers of understanding

One method of gaining new understanding is to examine solved problems. Melvil Dewey understood vertical integration, as he helped with innovations around 3×5 index cards, cabinets, as well as the classification systems that bears his name. Some even say he was the first standards bearer for libraries, though it’s hard to believe that anyone familiar with standards can imagine that one person could have actually been entirely responsible.

Another solved problem is how to make information about books and journals widely available. This has been done twice in the past century—first with the printed catalog card, distributed by the Library of Congress for the greater good, and the distributed catalog record, at great utility (and cost) by the Online Computer Library Center.

Pointers are no longer entirely sufficient, since the problem is not only how to find information but how to make sense of it once it has been found. Linking from catalog records has been a partial solution, but the era of complete books online is now entering its second decade. The third stage is upon us.

Small world: an academic romance

David Lodge; Penguin 1985

WorldCat•LibraryThing•Google Books•BookFinder 

Is there a bibliographic emergency?

The Bibliographic Control Working Group held its third and final public meeting on the future of bibliographic control July 9 at the Library of Congress, focusing on “The Economics and Organization of Bibliographic Data.” The conclusion of the meetings will come in a report issued in November. No dramatic changes were issued from this meeting, and public comment is invited until the end of July.

With several panels, invited speakers, and an open forum (including a public webcast), Deanna Marcum, Library of Congress associate librarian for library services, framed the discussion by saying “Worries about MARC as an international standard make it seem like we found it on a tablet.” She went on to say, “Many catalogers believe catalogs…should be a public good, but in this world, it’s not possible to ignore economic considerations.” Marcum said there is no LC budget line that provides cataloging records for other libraries, though the CIP program has been hugely successful.

Value for money
Jose-Marie Griffiths, dean of the library school at the University of North Carolina, Chapel Hill, said there are three broad areas of concern: users and uses of bibliographic data, different needs for the data, and the economics and organization of the data. “What does free cost?” she asked, “Who are the stakeholders, and how are they organizationally aligned?”

Judith Nadler, library director, University of Chicago, moderated the discussion and said the format of the meetings was based on the oral and written testimony that was used to create Section 108 of the Copyright Law. Nadler joked that “We will have authority control–if we can afford it.”

Atoms vs bits
Rick Lugg, partner, R2 Consulting, has often spoken of the need for libraries to say no before saying yes to new things. His Powerpoint-free (at Marcum’s request–no speakers used it) presentation focused on how backlogs are invisible in the digital world. “People have difficulty managing what they have,” Lugg said, “There is a sense of a long emergency, and libraries cannot afford to support what they are doing.”

Using business language, since R2 often consults for academic libraries on streamlining processes, Lugg said libraries are not taking advantage of the value chain. Competitors are now challenging libraries in the area of search, even as technical services budgets are being challenged.

In part, Lugg credited this pressure to the basic MARC record becoming a commodity, and he estimated the cost of an original cataloged record to be $150-200. He challenged libraries to abandon the “cult of perfection,” since “the reader isn’t going to read the wrong book.”

Another area of concern is the difficulty of maintaining three stove-piped bibliographic areas, from MARC records for books, to serials holdings for link resolvers, to an A-Z list of journals. With separate print and electronic records, the total cost of bibliographic control is unknown, particularly with a lifecycle that includes selection, access, digitization, and storage or deaccession.

There is a real question about inventory control vs. bibliographic control, Lugg said. The opportunity cost of the current processes lead to questions if libraries are putting their effort where it yields the most benefit. With many new responsibilities coming down the pike for technical services, including special collections, rare books, finding aids, and institutional repositories, libraries are challenged to retrain catalogers to expand their roles beyond MARC into learning new formats like MODS, METS, and Dublin Core.

Lugg said R2 found that non-MLS catalogers were often more rule-bound than professional staff, which brings about training questions. He summarized his presentation by asking:

  1. How do we reduce our efforts and redirect our focus?
  2. How can we redirect our expertise to new metadata schemes?
  3. How can we open our systems and cultures to external support from authors, publishers, abstract and indexing (A&I) services, etc?

The role of the consortium
Lizanne Payne, director of the WRLC, a library consortia for DC universities, said that with 200 international library consortia dedicated to containing the cost of content, the economics of bibliographic data is paramount. Saying that shared catalogs and systems date from a time when hardware and software was expensive, “IT staff is the most expensive line item now.”

Payne said storage facilities require explicit placement for quick retrieval, not a relative measure like call numbers. She called for algorithms to be written beyond FrBR that dedupe for unique and overlapping copies that go beyond OCLC or LCCN numbers.

Public libraries are special
Mary Catherine Little, director of technical services, Queens Library (NY), gave a fascinating overview of her library system. With 2.2 million items circulated in 2006 in 33 languages, 45,000 visitors per day, and 75,000 titles cataloged last year, Queens is the busiest library in the United States and has 66 branches within “one mile of every resident.”

Little said their ILS plans are evolving, “Heard about Sirsi/Dynix?” With its multilingual and growing collection, Little detailed their process. First, they ask if they are the first library to touch the record. Then, they investigate whether the ILS can function with the record “today, then tomorrow,” and ask if the record can be found from an outside source. The library prefers to get records from online vendors or directly from the publishers, and has 90 percent of English records in the catalog prior to publication.

Queens Public Library has devised a model for international providers which revolve around receiving monthly lists of high-demand titles, especially from high demand Chinese publishers, and standing orders. With vendors feeling the push from the library, many then enter into partnerships with OCLC.

“Uncataloged collections are better than backlogs,” Little said, and many patrons discover high-demand titles by walking around, especially audio and video. “We’ve accepted the tradeoffs,” she said.

Little made a call for community tagging, word clouds, and open source and extensible catalogs, and said there is a continuing challenge to capture non-Roman data formats.

“Global underpinnings are the key to the future, and Unicode must be present,” Little said, “The Library of Congress has been behind, and the future is open source software and interoperability through cooperation.”

Special libraries harmonize
Susan Fifer Canby, National Geographic Society vice president of library and information services, said her library contains proprietary data and works to harmonzie taxonomies across various content management systems (CMS), enhancing with useful metadata to give her users a Google-like search.

Canby said this work has led to a relative consistency and accuracy, which helps users bridge print and electronic sources. Though some special libraries are still managing print collections, most are devoting serious amounts of time to digital finding aids, competitive information gathering, and future analysis for their companies to help connect the dots. The library is working to associate latitude and longitude information with content to aid with mashups.

The National Geographic library uses OCLC records for books and serials, and simple MARC records for maps, and more complex records for ephemera, “though [there’s] no staff to catalog everything.” The big challenge, however, is cataloging photographs, since the ratio used to be 100 shots for every published photo, and now it’s 1000 to 1.”Photographers have been incentivized to provide keywords and metadata,” Canby said. With the rise of IPTC embedded data, the photographers are adding terms from drop-down menus, free-text fields, and conceptual terms.

The library is buying digital content, but not yet HD content, since it’s too expensive due to its large file size. Selling large versions of its photos through ecommerce has given the library additional funds for special librarians to do better, Canby said.

Special libraries have challenges to get their organizations to implement digital document solutions, since most people use email as a filing strategy instead of metadata-based solutions. Another large challenge is that most companies view special libraries as a cost center, and just sustaining services is difficult. Since the special library’s primary role isn’t cataloging, which is outsourced and often assigned to interns, the bottom line is to develop a flexible metadata strategy that includes collaborating with the Library of Congress and users to make it happen.

Vendors and records
Bob Nardini, Coutts Information Services, said book vendors are a major provider of MARC records, and may employ as many catalogers as the Library of Congress does. Coutts relies on the LC CIP records, and said both publishers and LC are under pressure to do more with less. Nardini advocated doing more in the early stages of a book’s life, and gave an interesting statistic about the commodity status of a MARC record from the Library of Congress: With an annual subscription to the LC records, the effective cost is $.06 per record.

PCC
Mechael Charbonneau, director of technical services at Indiana University Libraries, gave some history about how cataloging was under threat in 1996 because of budget crunches. In part, the Program for Cooperative Cataloging (PCC) came about to extend collaboration and to find cost savings. Charbonneau said that PCC records are considered to be equivalent to LC records, “trustworthy and authoritative.” With four main areas, including BIBCO for bibliographic records, NACO for name authority,  SACO for subject authority, and CONSER for serial records, international participants have effectively supplemented the Library of Congress records.

PCC’s strategic goals include looking at new models for non-MARC metadata, being proactive rather than reactive, reacting with flexibility, achieving close working relationships with publishers, and internationalizing authority files, which has begun with LC, OCLC, and the Deutsche Bibliotek.

Charbonneau said in her role as a academic librarian, she sees the need to optimize the allocation of staff in large research libraries and to free up catalogers to do new things, starting with user needs.

Abstract and indexing
Linda Beebe, senior director of PsycINFO, said the American Psychological Association (APA) has similar goals with its database, including the creation of unique metadata and controlled vocabularies. Beebe sees linking tools as a way to give users access to content. Though Google gives users breadth, not precision, partnerships to link to content using CrossRef’s DOI service has started to solve the appropriate copy problem. Though “some access is better than none,” she cautioned that in STM, a little knowledge is a dangerous thing.

Beebe said there is a continuing need for standards, but “how many, and can they be simplified and integrated?” With a dual audience of librarians and end-users, A&I providers feel the need to make the search learning curve gentle while preserving the need for advanced features that may require instruction.

A robust discussion ensued about the need for authority control for authors in A&I services. NISO emerging standards and the Scopus author profile were discussed as possible solutions. The NISO/ISO standard is being eagerly adopted by publishers as a way to pay out royalties.

Microsoft of the library world?
Karen Calhoun, OCLC VP for WorldCat and Metadata Services, listed seven economic challenges for the working group, including productivity, redundancy, value, scale, budgets, demography, and collaboration. Pointing to Fred Kilgour, OCLC founder, as leading libraries into an age of “enhanced productivity in cataloging,” Calhoun said new models of acquisition is the next frontier.

With various definitions of quality from libraries and end users, libraries must broaden their scale of bibliographic control for more materials. Calhoun argued that “narrowing our scope is premature.” With intense budget pressure “not being surprising,” new challenges include retirements building full strength starting in 2010.

Since libraries cannot work alone, and cost reductions are not ends in themselves, OCLC can create new opportunities for libraries. Calhoun compared the OCLC suite of services to the electric grid, and said remixable and reusable metadata is the way of the future, coming from publishers, vendors, authors, reviewers, readers, and selectors.

“WorldCat is an unexploited resource, and OCLC can help libraries by moving selected technical services to the network,” Calhoun said. Advocating moving library services to the OCLC bill “like PayPal,” Calhoun said libraries could reduce its manpower costs.

Teri Frick, technical services librarian at the Orange County Public Library (VA), questioned Calhoun, saying her library can’t afford OCLC, Calhoun admitted ” OCLC is struggling with that,” and “I don’t think we have the answers.”

Frick pointed out that her small public library has the same needs as the largest library, and said any change to LC cataloging policy would have a large effect on her operations in southwestern Virginia. “When LC cut–and I understand why–it really hurt.”

Library of Congress reorganizes
Beacher Wiggins, Library of Congress director for acquisitions and bibliographic control, read a paper that gave the LC perspective. Wiggins cited Marcum’s 2005 paper that disclosed the costs of cataloging at $44 million per year. LC has 400 cataloging staff (down from 750 in 1991), who cataloged 350,000 volumes last year.

The library has reorganized acquisitions and cataloging into one administrative unit in 2004, but cataloger workflows will be merged in 2008, with retraining to take place over the next 12-36 months. New job descriptions will be created, and new partners for international records (excluding authority records) are being selected. After an imbroglio about redistribution of Italian book dealer Casalini records, Wiggins said, “For this and any future agreements, we will not agree to restrict redistribution of records we receive.”

In further questioning, Karen Coyle, library consultant, pointed out that the education effort would be large, as well as the need to retrain people. Wiggins said LC is not giving up on pre-coordination, which had been questioned by LC union member Thomas Mann and others, but that they are looking at streamlining how it is done.

Judith Cannon, Library of Congress instruction specialist, said “We don’t use the products we create, and I think there’s a disconnect there. These are all interrelated subjects.”

NLM questions business model
Dianne McCutcheon, chief of technical services at the National Library of Medicine, agreed that cataloging is a public good and that managers need to come up with an efficient cost/benefit ratio. However, McCutcheon said, “No additional benefit accrues to libraries for contributing unique records–OCLC should pay libraries for each use of a unique record.”

McCutcheon spoke in favor of incorporating ONIX from publishers in place or to supplement MARC, and “to develop the appropriate crosswalks.” With publishers working in electronic environments, libraries should use the available metadata to enhance records and build in further automation. Since medical publishers are submitting citation records directly to NLM for inclusion in Medline, the library is seeing a significant cost savings, from $10 down to $1 a record. The NLM’s Medical Text Indexer (MTI) is another useful tool, which assits catalogers in assigning subject headings, with 60 percent agreement.

NAL urges more collaboration
Christopher Cole, associate director of technical services at the National Agricultural Library (NAL), said like the NLM, the NAL is both a library and a A&I provider. By using publisher supplied metadata as a starting point and adding additional access points and doing quality control, “quality has not suffered one bit.” Cole said the NAL thesaurus was recreated 6-7 years ago after previously relying on FAO and CAB information, and he advocated for a similar reinvention. Cole said, “Use ONIX. The publishers supply it.”

Tagging and privacy
Dan Chudnov, Library of Congresss Office of Strategic Initiatives, made two points, first saying that social tagging is hard, and its value is an emergent phenomenon with no obvious rhyme or reason. Chudnov said it happens in context, and referenced Tim Spalding’s talk given at LC. “The user becomes an access point, and this is incompatible with the ALA Bill of Rights on privacy that we hold dear,” he said.

Finally, Chudnov advocated for the inclusion of computer scientists from the wider community, perhaps in a joint meeting joined by vendors.

Summing up
Robert Wolven, Columbia University director of library systems and bibliographic control and working group member, summarized the meeting by saying that the purpose was to find the “cost sinks” and to find “collective efficiencies,” since metadata has a long life cycle. Cautioning that there are “no free rides,” libraries must find ways to recoup its costs.

Marcum cited LC’s mission, which is “to make the world’s creativity and the world’s knowledge accessible to Congress and the American people,” and said the LC’s leadership role can’t be diminished. With 100 million hidden items (including photos, videos, etc), curators are called upon in 21 reading rooms to direct users to hidden treasures. But “in the era of the Web, user expectations are expanding but funding is not. Thus, things need to be done differently, and we will be measuring success as never before,” Marcum said.

IDPF: Google and Harvard

Libraries And Publishers
At the 2007 International Digital Publishing Forum (IDPF) in New York May 9th, publishers and vendors discussed the future of ebooks in an age increasingly dominated by large-scale digitization projects funded by the deep pockets of Google and Microsoft.

In a departure from the other panels, which discussed digital warehouses and repositories, both planned and in production from Random House and HarperCollins, Peter Brantley, executive director of the Digital Library Federation and Dale Flecker of Harvard University Library made a passionate case for libraries in an era of information as a commodity.

Brantley began by mentioning the Library Project on Flickr, and led with a slightly ominous series of slides: “Libraries buy books (For a while longer), followed by “Libraries don’t always own what’s in the book, just the book (the “thing” of the book).¨

He then reiterated the classic rights that libraries protect: The Right to Borrow, Right to Browse, Right to Privacy, and Right to Learn, and warned that “some people may become disenfranchised in the the digital world, when access to the network becomes cheaper than physical things.” Given the presentation that followed from Tom Turvey, director of the Google Book Search project, this made sense.

Brantley made two additional points, saying “Libraries must permanently hold the wealth of our many cultures to preserve fundamental Rights, and Access to books must be either free or low-cost for the world’s poor.”¨ He departed from conventional thinking on access, though, when he argued that this low-cost access didn’t need to include fiction. Traditionally, libraries began as subscription libraries for those who couldn’t afford to purchase fiction in drugstores and other commercial venues.

Finally, Brantley said that books will become communities as they are integrated, multiplied, fragmented, collaborative, and shared, and publishing itself will be reinvented. Yet his conclusion contained an air of inevitability, as he said, “Libraries and publishers can change the world, or it will be transformed anyway.”

A podcast recording of his talk is available on his site.

Google Drops A Bomb
Google presented a plan to entice publishers to buy into two upcoming models for making money from Google Book Search, including a weekly rental “that resembles a library loan” and a purchase option, “much like a bookstore,” said Tom Turvey, director of Google Book Search Partnerships.¨ The personal library would allow search across the books, expiration and rental, and copy and paste. No pricing was announced. Google has been previewing the program at events including the London Book Fair.

Turvey said Google Book Search is live in 70 countries and eight languages. Ten years ago, zero percent of consumers clicked before buying books online, and now $4 billion of books are purchased online. “We think that’s a market,”Turvey said, “and we think of ourselves as the switchboard.”

Turvey, who previously worked at bn.com and ebrary, said publishers receive the majority of the revenue share as well as free marketing tools, site-brandable search inside a book with restricted buy links, and fetch and push statistical reporting.¨He said an iTunes for Books was unlikely, since books don’t have one device, model or user experience that works across all categories. Different verticals like fiction, reference, and science, technology and medicine (STM), require a different user experience, Turvey said.

Publishers including SparkNotes requested a way to make money from enabling a full view of their content on Google Books, as did many travel publishers. Most other books are limited to 20 percent visibility, although Turvey said there is a direct correlation between the number of pages viewed and subsequent purchases.

This program raises significant privacy questions. If Google has records that can be correlated with all the other information it stores, this is the polar opposite of what librarians have espoused about intellectual freedom and the privacy of circulation records. Additionally, the quality control questions are significant and growing, voiced by historian Robert Townsend and others.

Libraries are a large market segment to publishers. It seems reasonable to voice concerns about this proposal at this stage, especially those libraries who haven’t already been bought and sold. Others at the forum were skeptical. Jim Kennedy, vice president and director at the Associated Press, said, “The Google guy’s story is always the same: Send us your content and we’ll monetize it.”

Ebooks Ejournals And Libraries
Dale Flecker of the Harvard University Library gave a historical overview of the challenges libraries have grappled with in the era of digital information.

Instead of talking about ebooks, which he said represent only two percent of usage at Harvard, Flecker described eight challenges about ejournals, which are now “core to what libraries do” and have been in existence for 15-20 years. Library consultant October Ivins challenged this statistic about ebook usage as irrelevant, saying “Harvard isn’t typical.” She said there were 20 ebook platforms demonstrated at the 2006 Charleston Conference, though discovery is still an issue.

First, licensing is a big deal. There were several early questions: Who is a user? What can they do? Who polices behavior? What about guaranteed performance and license lapses? Flecker said that in an interesting shift, there is a move away from licenses to “shared understandings,” where content is acquired via purchase orders.¨

Second, archiving is a difficult issue. Harvard began in 1630, and has especially rich 18th century print collections, so it has been aware that “libraries buy for the ages.” The sticky issues come with remote and perpetual access, and what happens when a publisher ceases publishing.

Flecker didn’t mention library projects like LOCKSS or Portico in his presentation, though they do exist to answer those needs. He did say that “DRM is a bad actor” and it’s technically challenging to archive digital content. Though there have been various initiatives from libraries, publishers, and third parties, he said “Publishers have backed out,” and there are open questions about rights, responsibilities, and who pays for what. In the question and answer period that followed, Flecker said Harvard “gives lots of money” to Portico.”

Third, aggregation is common. Most ejournal content is licensed in bundles and consortia and buying clubs are common. Aggregated platforms provide useful search options and intercontent functionality.

Fourth, statistics matter, since they show utility and value for money spent. Though the COUNTER standard is well-defined and SUSHI gives a protocol for exchange of multiple stats, everyone counts differently.

Fifth, discovery is critical. Publishers have learned that making content discoverable increases use and value. At first, metadata was perceived to be intellectual property (as it still is, apparently), but then there was a grudging acceptance and finally, enthusiastic participation. It was unclear which metadata Flecker was describing, since many publisher abstracts are still regarded as intellectual property. He said Google is now a critical part of the discovery process.

Linkage was the sixth point. Linking started with citations, when publishers and aggregators realized that many footnotes contained links to articles that were also online. Bilateral agreements came next, and finally, the Digital Object Identifier (DOI) generalized the infrastructure and helped solve the “appropriate copy” problem, along with OpenURL. With this solution came true interpublished, interplatform, persistent and actionable links which are now growing beyond citations.

Seventh, there are early glimpses of text mining in ejournals. Text is being used as fodder for computational analysis, not just individual reading. This has required somewhat different licenses geared for computation, and also needs a different level of technical support.¨Last, there are continuing requirements for scholarly citation that is: • Unambiguous •Persistent • At a meaningful level. Article level linking in journals has proven to be sufficient, but the equivalent for books (the page? chapter? paragraph?) has not been established in an era of reflowable text.

In the previous panel, Peter Brantley asked the presenters on digital warehouses about persistent URLS to books, and if ISBNs would be used to construct those URLs. There was total silence, and then LibreDigital volunteered that redirects could be enabled at publisher request.

As WorldCat.org links have also switched from ISBN to OCLC number for permanlinks, this seems like an interesting question to solve and discuss. Will the canonical URL for a book point to Amazon, Google, OCLC, or OpenLibrary?

Open Data: What Would Kilgour Think?

The New York Public Library has reached a settlement with iBiblio, the public’s library and digital archive at the University of Chapel Hill, North Carolina, for harvesting records from its Research Libraries catalog, which it claims is copyrighted.

Heike Kordish, director of the NYPL Humanities Library, said a cease and desist letter was sent because a 1980s incident by an Australian harvesting effort which turned around and resold the NYPL records.

Simon Spero, iBiblio employee and technical assistant to the assistant vice chancellor at UNC-Chapel Hill, said NYPL requested that its library records be destroyed, and the claim was settled with no admission of wrongdoing. “I would characterize the New York Public Library as being neither public nor a library,” Spero said.

It is a curious development that while the NYPL is making arrangements under private agreements to allow Google to scan its book collection into full-text that it feels free to threaten other research libraries over MARC records.

The price of open data
This follows a similar string of disagreements about open data with OCLC and the MIT Simile project. The Barton Engineering Library catalog records were widely made available via Bit Torrent, a decentralized network file sharing format.

This has since been resolved by making the Barton data available again, though in RDF and MODS, not MARC, under a Creative Commons license for non-commercial use.

OCLC CEO Jay Jordan said the issues around sharing data had their genesis in concerns about the Open WorldCat project and sharing records with Microsoft, Google, and Amazon. Other concerns about private equity firms entering the library market also drove recent revisions to the data sharing policies.

OCLC quietly revised its policy about sharing records, which had not been updated since 1987 after numerous debates in the 1980s about the legality of copyrighting member records.

The new WorldCat policy, reads in part, “WorldCat® records, metadata and holdings information (“Data”) may only be used by Users (defined as individuals accessing WorldCat via OCLC partner Web interfaces) solely for the personal, non-commercial purpose of assisting such Users with locating an item in a library of the User’s choosing… No part of any Data provided in any form by WorldCat may be used, disclosed, reproduced, transferred or transmitted in any form without the prior written consent of OCLC except as expressly permitted hereunder.”

Looking through the most recent board minutes, it looks like concerns have been raised about “the risk to FirstSearch revenues from OpenWorldCat,” and management incentive plans have been approved.

What is good for libraries?
Another project initiated by Simon Spero, entitled Fred 2.0 after recently deceased Fred Kilgour of OCLC, Yale, and Chapel Hill fame, recently released Library of Congress authority file and subject information, which was gathered by similar means as the NYPL records.

Spero said the purpose of the project is “dedicated to the men and women at the Library of Congress and outside, who have worked for the past 108 years to build these authorities, often in the face of technology seemingly designed to make the task as difficult as possible.

Since Library of Congress data by definition cannot be copyrighted as free government information, the project was more collaborative in nature and has received acclaim for its help in pointing out cataloging irregularities in the records. OCLC also offers a linked authority file as a research project.

Firefox was born from open source
While the purpose of releasing library data has not yet reached consensus about what will be built as a result, it can be compared to Netscape open-sourcing the Mozilla code in 2000, which eventually brought Firefox and other open source projects to light. It also shows that the financial motivations of library organizations by necessity dictate the legal mechanisms of protection.

code4lib 2007

Working Code Wins
Responding to increasing consolidation in the ILS market, library developers demonstrated alternatives and supplements to library software at the second annual code4lib conference in Athens, GA, February 27-March 2, 2007. With 140 registered attendees from many states and several countries, including Canada and the United Kingdom, the conference was a hot destination for a previously isolated group of developers.

Network connectivity was a challenge for the Georgia Center for Continuing Education, but the hyperconnected group kept things interesting and the attendees coordinated by Roy Tennant artfully architected workarounds and improvements as the conference progressed.

In a nice mixture of emerging conference trends, code4lib combined the flexibility of the unconference with 20 minute prepared talks, keynotes, five minute Lightning Talks, and breakout sessions. The form was derived from Access, the Canadian library conference.

Keynotes
The conference opened with a talk from Karen Schneider, associate director for technology and research at Florida State University. She challenged the attendees to sell open source software to directors in terms of solutions it provides, since the larger issue in libraries is saving digital information. Schneider also debated Ben Ostrowsky, systems librarian at the Tampa Bay Library Consortium, about the importance of open source software from the stage, to which Ostrowsky responded, “Isn’t that Firefox [a popular open source browser] you’re using there?”

Erik Hatcher, author of Lucene in Action, gave a keynote about using the full-text search server, Apache Solr, open-source search engine Lucene and faceted browser, Flare, to construct a new front-end to library catalog data. The previous day, Hatcher led a free preconference for 80 librarians who brought exported MARC records, including Villanova University and the University of Virginia.

Buzz
One of the best-received talks revolved around BibApp, an “institutional bibliography” written in Ruby on Rails by Nate Vack and Eric Larson, two librarians at the University of Wisconsin-Madison. The prototype application is available for download, but currently relies on citation data from engineering databases to construct a profile of popular journals, publishers, citation types, and who researchers are publishing with. “This is copywrong, which is sometimes what you have to do to construct digital library projects. Then you get money to license it,” Larson said.

More controversially, Luis Salazar gave a talk about using Linux to power public computing in the Howard County (MD) public library system. A former NSA systems administrator, he presented the pros and cons of supporting 300 staff and 400 public access computers using Groovix, a customized Linux distribution. Since the abundant number of computers serves the public without needing sign up sheets, “patrons are able to sit down and do what they want.”

Salazar created a script for monitoring all the public computers, and described how he engaged in a dialog with a patron he dubbed “Hacker Jon,” who used the library computers to develop his nascent scripting skills. Bess Sadler, librarian and metadata services specialist at the University of Virginia, asked about the privacy implications of monitoring patrons. “Do you have a click-through agreement? Privacy Policy?” she asked. Salazar joked that “It’s Maryland, we’re like a communist country” and said he wouldn’t do anything in a public library that he wouldn’t expect to be monitored.

Casey Durfee presented a talk on “Endeca in 250 lines of code or less,” which showed a prototype of faceted searching at the Seattle Public Library. The new catalog front-end sits on top of a Horizon catalog, and uses Python and Solr to present results in an elegant display, from a Google-inspired single search box start to rich subject browse options.

The future
This year’s sponsors included Talis, LibLime, OCLC, Logical Choice Technologies, and Oregon State University. OSU awarded two scholarships to Nicole Engard, Jenkins Law Library (2007 LJ Mover and Shaker), and Joshua Gomez, Getty Research Institute.

Next year’s conference will be held in Portland, OR.

NetConnect Winter 2007 podcast episode 2

This is the second episode of the Open Libraries podcast, and I was pleased to have the opportunity to talk to some of the authors of the Winter netConnect supplement, entitled Digitize This!

The issue covers how libraries can start to digitize their unique collections. K. Matthew Dames and Jil Hurst-Wahl wrote an article about copyright and practical considerations in getting started. They join me, along with Lotfi Belkhir, CEO of Kirtas Technologies, to discuss the important issue of digitization quality.

One of the issues that has surfaced recently is exactly what libraries are receiving from the Google Book Search project. As the project grows beyond the initial five libraries into more university and Spanish libraries, many of the implications have become more visible.

The print issue of NetConnect is bundled with the January 15th issue of Library Journal, or you can read the articles online.

Recommended Books:
Kevin
Knowledge Diplomacy

Jill
Business as Unusual

Lotfi
Free Culture
Negotiating China
The Fabric of the Cosmos

Software
SuperDuper
Google Documents
Arabic OCR

0 Music and Intro
1:59 Kevin Dames on his weblog Copycense
2:48 Jill Hurst-Wahl on Digitization 101
4:16 Jill and Kevin on their article
4:34 SLA Digitization Workshop
5:24 Western NY Project
6:45 Digitization Expo
7:43 Lotfi Belkhir
9:00 Books to Bytes
9:26 Cornell and Microsoft Digitization
11:00 Scanning vs Digitization
11:48 Google Scanning
15:22 Michael Keller’s OCLC presentation
16:14 Google and the Public Domain
17:52 Author’s Guild sues Google
21:13 Quality Issues
24:10 MBooks
26:56 Public Library digitization
27:14 Incorporating Google Books into the catalog
28:49 CDL contract
30:22 Microsoft Book Search
31:15 Double Fold
39:20 Print on Demand and Digitization
39:25 Books@Google
43:14 History on a Postcard
45:33 iPRES conference
45:46 LOCKSS
46:45 OAIS