Latest blog posts
Martyn Rittman
2024 November 15

Research Integrity Roundtable 2024

For the third year in a row, Crossref hosted a roundtable on research integrity prior to the Frankfurt book fair. This year the event looked at Crossmark, our tool to display retractions and other post-publication updates to readers. Since the start of 2024, we have been carrying out a consultation on Crossmark, gathering feedback and input from a range of members. The roundtable discussion was a chance to check and refine some of the conclusions we’ve come to, and gather more suggestions on the way forward.

...Find out more
Dominika Tkaczyk
2024 November 06

How good is your matching?

In our previous blog post in this series, we explained why no metadata matching strategy can return perfect results. Thankfully, however, this does not mean that it’s impossible to know anything about the quality of matching. Indeed, we can (and should!) measure how close (or far) we are from achieving perfection with our matching. Read on to learn how this can be done! How about we start with a quiz? Imagine a database of scholarly metadata that needs to be enriched with identifiers, such as ORCIDs or ROR IDs.

...Find out more
Kornelia Korzec
2024 October 28

Update on the Resourcing Crossref for Future Sustainability research

We’re in year two of the Resourcing Crossref for Future Sustainability (RCFS) research. This report provides an update on progress to date, specifically on research we’ve conducted to better understand the impact of our fees and possible changes. Crossref is in a good financial position with our current fees, which haven’t increased in 20 years. This project is seeking to future-proof our fees by: Making fees more equitable Simplifying our complex fee schedule Rebalancing revenue sources In order to review all aspects of our fees, we’ve planned five projects to look into specific aspects of our current fees that may need to change to achieve the goals above.

...Find out more
Lucy Ofiesh
2024 September 24

Meet the candidates and vote in our 2024 Board elections

On behalf of the Nominating Committee, I’m pleased to share the slate of candidates for the 2024 board election. Each year we do an open call for board interest. This year, the Nominating Committee received 53 submissions from members worldwide to fill four open board seats. We maintain a balanced board of 8 large member seats and 8 small member seats. Size is determined based on the organization’s membership tier (small members fall in the $0-$1,650 tiers and large members in the $3,900 - $50,000 tiers).

...Find out more
  • Menu
search

Documentation

Text and data mining

Text and data mining (TDM) is the automatic (bot) analysis and extraction of information from large numbers of documents. TDM is more effective than screen-scraping, which is inefficient, error-prone, and fragile. Screen-scraping puts an unnecessary load on member sites (downloading html, css, javascript and other superfluous web assets), will often break if members (even slightly) redesign their websites, and typically is tied to specific members’ page layouts (and therefore need to be adapted on a member-by-member basis).

Using the DOI as the basis for TDM in a common API provides several benefits:

  • An easy way to de-duplicate documents that may be found on several sites. Processing the same document on multiple sites could easily skew TDM results and traditional techniques for eliminating duplicates (such as hashes) will not work reliably if the document in question exists in several representations (such as PDF, HTML, ePub) and/or versions (such as author’s accepted manuscript, and version of record)
  • Persistent provenance information. Using the DOI as a key allows researchers to retrieve and verify the provenance of the items in the TDM corpus, many years into the future when traditional HTTPS URLs will have already broken
  • An easy way to document, share, and compare corpora without having to exchange the actual documents
  • A mechanism to ensure the reproducibility of TDM results using the source documents
  • A mechanism to track the impact of updates, corrections, retractions, and withdrawals on corpora.

Researchers are increasingly interested in performing TDM with scholarly content. This requires automated access to the full-text content of large numbers of articles. The format of the full-text content varies by member. Our metadata helps researchers get access to this content and enables members to provide it.

How TDM works

  1. A member deposits URLs for their full-text and license/waivers (along with other publication metadata) weith us
  2. A researcher finds relevant content registered with us (such as journal articles) using a discovery service
  3. The researcher retrieves metadata for each item of registered content, including license information
  4. The researcher makes a full-text request from the member
  5. The member checks the subscription rights of the researcher and returns the full-text to them.

Researchers and text miners can access content URLs and license information via our API. If you are a member and would like to begin depositing URLs and access indicators, please contact us.

Page owner: Patrick Polischuk   |   Last updated 2020-April-08