What is HDF5? The “Supercharged R List” Analogy

If you aren’t familiar with HDF5, the easiest way to think about it is as a persistent, cross-platform R list. In R, a list is powerful because it can hold a matrix of numbers, a data frame, and a nested list all in one object. HDF5 brings this same flexibility to your hard drive, allowing you to:

• Store Heterogeneous Data Together: Keep arrays, tidy data frames, and raw bytes inside a single .h5 file.

• Perform Efficient Partial I/O: HDF5 allows you to “reach in” and read only the specific slice of a dataset you need. This makes it possible to work with multi-gigabyte objects that would otherwise crash your R session.

• Preserve Metadata with Attributes: HDF5 attributes are the direct analog to R attributes. You can attach custom metadata - like units, timestamps, or descriptions - directly to any dataset or group.

• Organize with Groups: Just as you nest lists in R, HDF5 uses a filesystem-like hierarchy of “groups” to organize your data.

• Share Across Languages: Because HDF5 is a universal standard, that “list” you saved in R can be opened in Python as a dictionary or in Julia as a named tuple, with all your attributes and dimensions intact.

The Motivation: Navigating the Dependency Minefield

The primary goal of h5lite and hdf5lib is to eliminate the “Dependency Minefield” that R users often face. Historically, existing HDF5 interfaces in R presented significant hurdles for anyone trying to build a stable, redistributable package:

• The Bioconductor Barrier: Relying on the rhdf5 ecosystem requires your users to navigate BiocManager, which complicates the standard install.packages() workflow and can cause issues on CRAN check farms.

• System Library Fragility: The hdf5r package typically requires users to manually install libhdf5-dev on their OS. This is notoriously difficult for Windows users and creates an unpredictable environment where your package might fail if the user has a mismatched library version.

By providing a guaranteed, zero-dependency environment, these packages allow R developers to simply add LinkingTo: hdf5lib to their DESCRIPTION file. For the first time, you can ship a package that uses HDF5 and be certain it will “just work” for every user, on every platform, immediately upon installation.

Choosing Your Path: Which Package to Use?

Depending on whether you are analyzing data or building tools for others, your entry point into this ecosystem will differ:

• Interactive Users: You only need h5lite. It provides a simplified R interface (h5_read, h5_write) to quickly store and retrieve data, removing the need to manage complex HDF5 objects or system dependencies.

• Package Developers: You have two powerful ways to leverage this infrastructure:

  • Imports: h5lite: Use this to call the streamlined R interface within your own package functions.
  • LinkingTo: hdf5lib: Use this if your package contains C or C++ code and requires direct, high-performance access to the native HDF5 C API.

The Interface: h5lite

h5lite is an opinionated, streamlined interface that handles the complexities of type mapping and interoperability so you can focus on your data. Below is a summary of the type translation logic handled by the h5lite interface:

The Engine: hdf5lib

Under the hood, h5lite is powered by hdf5lib. While most users won’t interact with it directly, it serves as the rock-solid foundation for the entire ecosystem.

• Bundled Source: hdf5lib bundles the complete HDF5 2.0.0 source code, compiling natively on macOS, Windows, and Linux with no external system libraries required.

• Comprehensive Compression: In the next release, hdf5lib will bundle gzip (zlib), szip (libaec), lz4, zstd, and blosc. This ensures these high-performance filters “just work” out-of-the-box on all platforms and are available to users of h5lite.

• Thread-Safety: Compiled with thread-safety enabled by default, allowing for safe parallel I/O regardless of your chosen threading system.

• Versioning & API Stability: Developers can specify LinkingTo: hdf5lib (>= 2.0) to ensure HDF5 2.0 features are available. Built-in API versioning also safeguards your code against breaking changes in future HDF5 releases.

Integrated Compression Support

To provide world-class compression while maintaining a minimal footprint and zero-dependency promise, we are integrating advanced filters directly into the engine.

• Bundled Libraries: Instead of a standalone package, the gzip (zlib), szip (libaec), lz4, zstd, and blosc compression libraries will be bundled in the next release of hdf5lib.

• How it Works: h5lite will automatically leverage these libraries bundled within hdf5lib to provide seamless, high-performance compression at runtime.

• Immediate Alternatives: If you need these formats today, h5lite can utilize external plugins. You can point the HDF5_PLUGIN_PATH environment variable to the plugin directories provided by Bioconductor’s rhdf5filters package.

Stability and Minimal Footprint

Reliability is paramount for a storage interface. Both packages minimize their dependency footprint by not importing any other R packages. To ensure a stable experience, h5lite maintains a 100% code coverage test suite. Both packages have been rigorously validated across standard CRAN platforms (Linux, Windows, macOS) and compilers (GCC, Clang).

Comparison at a Glance

Try It Out

install.packages("h5lite")

library(h5lite)
file <- tempfile(fileext = ".h5")

# Write various R objects with smart defaults
h5_write(1:10, file, "ints")                   # Integer Vector
h5_write(I("example"), file, "/", attr = "id") # Scalar Attribute
h5_write(matrix(1:20, nrow = 5), file, "mtx")  # Numeric Matrix
h5_write(factor(letters), file, "letters")     # Factor -> ENUM
h5_write(list(a = 1, b = 2), file, "config")   # List -> GROUP
h5_write(iris, file, "flowers/iris")           # Data Frame -> COMPOUND

# Write with specific type coercions
h5_write(iris, file, "flowers/coerced",
  as = c(Sepal.Length = "bfloat16", .numeric = "float32"))

# Inspect the file structure
h5_str(file)
#> /
#> ├── @id <utf8[7] scalar>
#> ├── mtx <uint8 × 5 × 4>
#> ├── letters <enum × 26>
#> ├── config/
#> │   ├── a <uint8 × 1>
#> │   └── b <uint8 × 1>
#> ├── ints <uint8 × 10>
#> └── flowers/
#>     ├── iris <compound[5] × 150>
#>     │   ├── $Sepal.Length <float64>
#>     │   ├── $Sepal.Width <float64>
#>     │   ├── $Petal.Length <float64>
#>     │   ├── $Petal.Width <float64>
#>     │   └── $Species <enum>
#>     └── coerced <compound[5] × 150>
#>         ├── $Sepal.Length <bfloat16>
#>         ├── $Sepal.Width <float32>
#>         ├── $Petal.Length <float32>
#>         ├── $Petal.Width <float32>
#>         └── $Species <enum>

For a deeper dive, explore the Getting Started with h5lite guide and the hdf5lib documentation.