The Big Picture of Data Management in an Electronic Lab Notebook (ELN)

by Jan Cheetham

The ELN is designed to be a single place to hold the different types of digital data you produce in your research. Now that UW-Madison researchers have unlimited storage on the LabArchives platform, there are no physical limits on the amount of data you include in your ELN notebook. It may be time to take a holistic view of how to manage all those data files in the ELN.

To get a big picture view, it helps to consider the different ways to add data and information to a LabArchives notebook and the options for getting it out again, as a backup copy or to archive it for the long haul. In an ELN system, a notebook consists of essentially three different types of data: notebook pages, attached files, and linked files.

 

 

Getting data in

This schematic illustrates the different ways to add data to a notebook: manually uploading it to an attachment entry; typing, pasting, or dragging it into a rich text entry, or using an automated workflow to “push” it from instruments or software into an attachment entry (blue arrows).  Once a data file is attached to an entry, LabArchives keeps track of the file name and versions, including times and dates of changes.

Linking to data

If your data files need to live on an external server outside of the ELN, you can create a link to them from inside a notebook entry. However, if the files move, this link will break.

Getting data out

The green arrows in the diagram represent the export process, which is manual. LabArchives has two types of export packages: Offline Notebook, a zipped folder that includes notebook pages in HTML format plus attached data files (the most recent version only) and Print to PDF, a single PDF file of the pages that includes names and icons of attached files only. Of course, neither option contains a copy of linked files that are stored outside the ELN– only the path to the external files gets recorded in the ELN.

Backing up and archiving notebooks require some planning. There are several factors to consider:

  1. How often will you need to back up your notebook? Which type(s) of export package(s) will be most useful for this?
  2. Where will you store these export package files? Will you also print out paper copies and retain them? Is there a digital repository or data archiving platform you can keep them in for the long term?
  3. Will you keep a backup copy of attached data files outside of the ELN? If yes, you may find that the PDF format is sufficient for periodic backups of your notebook. If no, you may want to have a regular schedule for exporting the Offline Notebook as a backup of both notebook pages and attached data files.
  4. Are the external data files you link to from the ELN in a permanent location? If not, consider moving them into the ELN or storing them in a more permanent location.

For more information

ELN Archiving

Scenarios to help you create an archival and backup plan.

Archiving Electronic Lab Notebooks

Components of a ELN notebook and what information in each needs to be archived.

Manage Your Data with LabArchives

More details about export packages as well as data management tips for sharing, organizing, and documenting data inside the ELN.

 Data Storage and Backup

Storage and backup resources at UW-Madison for exported ELN files.

 Digital Archiving Platforms

A sampling of digital data repositories that may be useful for archiving ELNs

Tool: Electronic Lab Notebook

by Cameron Cook

UW-Madison nows offers an electronic lab notebook (ELN) service called LabArchives.

What is an electronic lab notebook?

An electronic lab notebook is software designed to replace traditional paper lab notebooks and allows you to contain all the different types of digital data your research produces in one place. It also makes your research easier by helping you organize, document, and share your work.

What can LabArchives help you do?

Per the university ELN website and the DoIT ELN Information, some of the available features are:

  • Organize research by structuring files and folders, as well as allow versioning and tracking.
  • Designate multiple user roles and customize permission settings.
  • Share and collaborate with other researchers whether internally or externally.
  • Integrate it with “Microsoft Office (Windows only), Google docs, ChemDoodle, PubMed, and GraphPad Prism”.
  • Attach files of many different formats.
  • Link to files living on an external server.
  • Unlimited storage on the LabArchives platform.
  • Securely access research  anywhere in the world via an internet connection.
  • Access to basic drawing and chemistry tools.
  • Export your data with ‘Offline Notebook’ or ‘Print to PDF’ options.
  • Using LabArchives through the university provides legal protections for your research like the one’s provided through UW-Google Apps and UW-Madison Box. It also helps with compliance to UW’s Policy on Data Stewardship, Access, and Retention.
How do you get it?

It is available at no cost to students, staff, faculty, and researchers upon request or approval by a Principal Investigator or other appropriate designee.

What else should you know?

If you are interested in using the LabArchives service with data that is subject to campus IRB or federal oversight, it is recommended that you consult with your IRB or information security officer. At this time, the campus LabArchives service is not considered a suitable place for sensitive data elements, such as personal health information.

For more information LabArchives and ELN best practices suggested by UW visit eln.wisc.edu. For more information about ELN implementation at UW-Madison, visit UW-Madison News’ recent article.

 

Tools: Archiving Electronic Lab Notebooks

Electronic Lab Notebooks are becoming important data management tools for researchers in a number of fields. Since ELNs are replacing paper lab notebooks in many labs, can we anticipate a future in which boxes and shelves of decades-old notebooks are replaced with a digital archive of ELN entries? Since ELNs are relative newcomers to the data management ecosystem, some basic discussion about what an ELN archive should contain seems relevant.

There are four general types of data “assets” that can be recorded in a ELN and each has a separate set of considerations for archiving.

DoIT AT LTDE - Blog 100x100 Icons-131. Notebook pages/entries and folders

In ELNs, pages and entries are containers in which text, symbols, equations, and other entities are entered using tools in the ELN interface. ELN pages/entries may be further organized within folders in the notebook.

What needs to be preserved?

All the information entered in ELN notebook fields, including tags and comments. In addition, the organizational structure of the page and hierarchical structure of folders and subfolders needs to be preserved. Therefore, an export package should include notebook page files in formats such as xml, html, or PDF that preserve the content, appearance, and layout of notebook pages and folders. It should also retain the naming schemas and folder hierarchies with the notebook.

DoIT AT LTDE - Blog 100x100 Icons-112. Attached data files

These are data files and documents that were not created in the ELN interface but uploaded to the ELN platform and attached to an ELN entry. These can include things like images, spreadsheets, and data files from lab instruments. ELN platforms generally allow the user to add annotations and comments and associate them with these data files.

What needs to be preserved?

All the  data files in their original, native formats plus any annotations added in the ELN interface. Annotations and comments should be preserved as either  separate files linked to the data files or as components of page/entry files in the export package, rather than altering the data files themselves. If multiple versions of individual files were attached to an ELN entry/page, metadata about the versions, including dates, should be also be preserved.

DoIT AT LTDE - Blog 100x100 Icons-073. Linked data files

These are files and documents that are linked to an ELN entry but reside on other systems such as lab or department servers.

What needs to be preserved?

Although linked files are located external to the ELN platform, an archive of all the data associated with a notebook should include a record of the server address of the linked file plus evidence of whether the server location is still accurate for the file at the time of archiving. One mechanism to assure that the file associated with an ELN entry is valid is to generate a checksum using common algorithms like MD5 or SHA-1 that would be stored with the file location. Ideally, the ELN platform would manage this checksum generation. In addition, it would be beneficial for the ELN platform to perform periodic link checking even before archiving is done to assure the continued presence of the remote file.

DoIT AT LTDE - Blog 100x100 Icons-094. Metadata

This is information about the provenance of an ELN page/entry and includes things such as date and time, name of the individual creating/editing, the version history of attached data files, etc.

What needs to be preserved?

Provenance information that is viewable in the ELN interface should be included in archives of the ELN pages. More detailed metadata is contained in log files collected on the database and application servers of the ELN platform and some components of this information that provide evidence of user access and actions may also need to be preserved in an ELN archive.

Tools: Electronic Lab Notebooks

What are they?

Electronic Lab Notebooks (ELNs) are software counterparts to paper lab notebooks. Although the name suggests a physical notebook device, ELNs are actually just software that runs on a computer, although some ELNs have apps for tablets and phones. The ELN interface resembles a notebook page, with fields for creating text entries and for attaching and annotating data files. Most allow you to create and modify templates for frequently used protocols. Other functionalities may include inventory tools that allow you to track samples and reagents.  Some have chemistry tools for drawing and searching on chemical structures.

UW-Madison has piloted a couple of ELNs (http://academictech.doit.wisc.edu/ideas/electronic-lab-notebooks) and we are currently evaluating a few others that have recently emerged. Interest in ELNs at the UW has grown over the past 2 years from a few interested researchers at the start of the pilot to hundreds of interested researchers. In response to this interest, a campus wide effort spearheaded by the Office of the CIO, WARF, CALS, and DoIT is seeking software, funding, and infrastructure to establish an enterprise ELN service.

Are ELNs useful for data management?

In general, ELNs let you keep data and descriptive information (e.g. materials, methods, analysis, and interpretations of the data) in digital form and all in one place. (An exception may be digital data files that are too large to upload/attach to the ELN. In that case, links can be added in the ELN to the server location of these files.) For simplicity in the following discussion, we’ll refer to both data files and the descriptive information as “data,” although the descriptive information might more accurately be called metadata.

Storing data in ELNs

From a data storage perspective, ELNs come in two flavors: those that can be locally hosted and those are hosted and store data in the cloud.

Locally hosted ELNs have the advantage of keeping data on campus servers. However, they usually consist of application, file system, and database layers and can be fairly complex to install, administer, and maintain for individual labs and departments. An enterprise level ELN service could provide some economies of scale by providing a common infrastructure to provide the hardware and services needed for a locally hosted ELN.

Other ELNs are cloud services that store data in the cloud, external to UW servers.  However, there are a lot of questions about security, protection of intellectual property and other issues when research data is moved to the cloud.  An advantage of an exploring an enterprise cloud ELN would be that the university could negotiate for favorable terms with vendors to secure agreements about the geographic location of cloud servers, segmentation of the our data from that of other institutions, encryption and advanced security provisions, etc., through a purchasing exercise.

Organizing and Tracking Versions of Files

When it comes to keeping an audit trail and preserving versioning information for entries and attached data, ELNs have a lot going for them. ELNs log time, date, user names and actions, and track version information. In many ELNs, these audit trails are designed to meet Federal government requirements for electronic records: the Code of Federal Regulations Title 21, Part II Electronic Records; Electronic Signatures.

Sharing and Collaborating
Similar to collaborative tools such as Google Drive, ELNs let you tag, comment, define workgroups, and share entries, files, folders, and templates with specified groups and individuals. In addition, a few labs on campus have been showing data recorded in their ELN at lab meetings and find that generally works out well.

Most ELNs also have features that allow electronic signing and witnessing of entries, enabling a level of legal documentation needed for research leading to patents.

Exporting and Archiving
Most ELNs let you export (and print) entries as PDF files and data files that are attached in their native formats. Some have xml and/or html export formats. However, there are no standards for exports which would make it easier to move records from one vendor’s ELN to another. We’d like to see that happen.

One of the strongest arguments for using an ELN system is its promise as a solution for data stewardship. Since the campus data stewardship policy specifies data be retained for at least 7 years (longer for some types of research), an enterprise ELN service would need to have a backend archival system. This would allow older data in the ELN to be moved to cheaper storage but would still allow search and retrieval from the archive with the appropriate access permissions in place. Researchers can also save ELN entries in PDF format and retain both print outs and electronic versions of these files.

Publishing
ELNs also offer the potential for publishing data. For example, one ELN provides a permanent digital object identifier (DOI) for each entry, which have been used in a publication.