platform-mem

A Rust library for low-level memory management with unified interface for allocator-backed and memory-mapped file storage.

Overview

platform-mem provides the RawMem trait that abstracts over different memory backends:

• Allocator-based memory (Global, System, Alloc<T, A>) - uses Rust's allocator API
• Memory-mapped files (FileMapped, TempFile) - uses mmap for persistent or temporary file-backed storage

This allows writing generic code that works with any memory backend, making it easy to switch between heap allocation and file-mapped storage.

Features

• Unified RawMem trait - common interface for growing, shrinking, and accessing memory
• Type-erased memory via ErasedMem - enables dynamic dispatch with Box<dyn ErasedMem<Item = T>>
• Memory-mapped files - persistent storage with automatic page management
• Temporary file storage - anonymous file-backed memory that's cleaned up on drop
• Safe growth operations - grow_filled, grow_zeroed, grow_from_slice, and more
• Thread-safe - all memory types implement Send + Sync
• Async memory operations (optional) - async mmap access with AsyncFileMem via dedicated I/O thread

Installation

Add to your Cargo.toml:

[dependencies]
platform-mem = "0.1"

Note: This crate uses Rust edition 2024 and works on stable Rust. It uses the allocator-api2 crate to provide allocator API functionality on stable Rust.

Optional Features

To enable async memory operations:

[dependencies]
platform-mem = { version = "0.1", features = ["async"] }

Usage

Basic Example with Global Allocator

use platform_mem::{Global, RawMem};

fn main() -> Result<(), platform_mem::Error> {
    let mut mem = Global::<u64>::new();

    // Grow memory and fill with value
    mem.grow_filled(10, 42)?;
    assert_eq!(mem.allocated(), &[42u64; 10]);

    // Grow more from a slice
    mem.grow_from_slice(&[1, 2, 3])?;
    assert_eq!(mem.allocated().len(), 13);

    // Shrink by 5 elements
    mem.shrink(5)?;
    assert_eq!(mem.allocated().len(), 8);

    Ok(())
}

Memory-Mapped File Storage

use platform_mem::{FileMapped, RawMem};

fn main() -> Result<(), platform_mem::Error> {
    // Create memory mapped to a file
    let mut mem = FileMapped::<u64>::from_path("data.bin")?;

    // Data persists across program runs
    unsafe {
        mem.grow_zeroed(1000)?;
    }

    // Modify the memory
    mem.allocated_mut()[0] = 123;

    Ok(())
}

Temporary File Storage

use platform_mem::{TempFile, RawMem};

fn main() -> Result<(), platform_mem::Error> {
    // Anonymous temporary file - cleaned up on drop
    let mut mem = TempFile::<u8>::new()?;

    mem.grow_from_slice(b"hello world")?;
    assert_eq!(mem.allocated(), b"hello world");

    Ok(())
}

Generic Code with RawMem

use platform_mem::RawMem;

fn process_data<M: RawMem<Item = u32>>(mem: &mut M) -> Result<(), platform_mem::Error> {
    mem.grow_filled(100, 0)?;

    for (i, slot) in mem.allocated_mut().iter_mut().enumerate() {
        *slot = i as u32;
    }

    Ok(())
}

Type-Erased Memory with ErasedMem

use platform_mem::{ErasedMem, Global, RawMem};

fn main() {
    // Use dynamic dispatch when the memory type isn't known at compile time
    let mem: Box<dyn ErasedMem<Item = u64> + Send + Sync> =
        Box::new(Global::<u64>::new());
}

Async File Memory (requires async feature)

use platform_mem::AsyncFileMem;

#[tokio::main]
async fn main() -> Result<(), platform_mem::Error> {
    // Create async mmap-backed memory with a dedicated I/O thread.
    // Page faults are handled by the I/O thread, not the async runtime.
    let mem = AsyncFileMem::<u64>::from_path("async_data.bin")?;

    // Async grow operations (dispatched to I/O thread)
    mem.grow_filled(100, 42).await?;

    // Read/write via I/O thread (non-blocking for async callers)
    mem.set(0, 123).await?;
    assert_eq!(mem.get(0).await?, Some(123));

    // Data is synced to disk when AsyncFileMem is dropped
    Ok(())
}

API Overview

RawMem Trait

The core trait providing memory operations:

Method	Description
allocated()	Returns a slice of the initialized memory
allocated_mut()	Returns a mutable slice of the initialized memory
grow(addition, fill)	Grows memory by addition elements with custom initialization
shrink(cap)	Shrinks memory by cap elements
grow_filled(cap, value)	Grows and fills with cloned values
grow_zeroed(cap)	Grows and zero-initializes (unsafe for non-zeroable types)
grow_from_slice(src)	Grows and copies from a slice
grow_with(addition, f)	Grows and initializes with a closure
grow_within(range)	Grows by cloning a sub-range of existing data
grow_assumed(cap)	Grows assuming data is already initialized (unsafe)

Memory Types

Type	Description
Global<T>	Uses Rust's global allocator
System<T>	Uses the system allocator
Alloc<T, A>	Generic over any Allocator
FileMapped<T>	Memory-mapped file storage
TempFile<T>	Temporary file-backed memory
AsyncFileMem<T>	Async mmap access via dedicated I/O thread (requires async feature)

Error Handling

The crate defines an Error enum with these variants:

• CapacityOverflow - Requested capacity exceeds isize::MAX bytes
• OverGrow - Tried to grow more than available space
• AllocError - Allocator failed to allocate/reallocate
• System - I/O error from file operations

License

This project is released into the public domain under the Unlicense.

Related Projects

• doublets-rs - Doublet links data structure using this memory library
• LinksPlatform - The Links Platform organization