GDAL

Typedefs | Enumerations | Functions
cpl_virtualmem.h File Reference

Virtual memory management. More...

`#include <stddef.h>`
`#include "cpl_port.h"`
`#include "cpl_vsi.h"`

Go to the source code of this file.

## Typedefs
typedef struct CPLVirtualMem | CPLVirtualMem
| Opaque type that represents a virtual memory mapping.
typedef void(* | CPLVirtualMemCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)
| Callback triggered when a still unmapped page of virtual memory is accessed. More...
typedef void(* | CPLVirtualMemUnCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)
| Callback triggered when a dirty mapped page is going to be freed. More...
typedef void(* | CPLVirtualMemFreeUserData) (void *pUserData)
| Callback triggered when a virtual memory mapping is destroyed. More...

## Enumerations
enum | CPLVirtualMemAccessMode { VIRTUALMEM_READONLY , VIRTUALMEM_READONLY_ENFORCED , VIRTUALMEM_READWRITE }
| Access mode of a virtual memory mapping. More...

## Functions
size_t | CPLGetPageSize (void)
| Return the size of a page of virtual memory. More...
CPLVirtualMem * | CPLVirtualMemNew (size_t nSize, size_t nCacheSize, size_t nPageSizeHint, int bSingleThreadUsage, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemCachePageCbk pfnCachePage, CPLVirtualMemUnCachePageCbk pfnUnCachePage, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
| Create a new virtual memory mapping. More...
int | CPLIsVirtualMemFileMapAvailable (void)
| Return if virtual memory mapping of a file is available. More...
CPLVirtualMem * | CPLVirtualMemFileMapNew (VSILFILE *fp, vsi_l_offset nOffset, vsi_l_offset nLength, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
| Create a new virtual memory mapping from a file. More...
CPLVirtualMem * | CPLVirtualMemDerivedNew (CPLVirtualMem *pVMemBase, vsi_l_offset nOffset, vsi_l_offset nSize, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
| Create a new virtual memory mapping derived from an other virtual memory mapping. More...
void | CPLVirtualMemFree (CPLVirtualMem *ctxt)
| Free a virtual memory mapping. More...
void * | CPLVirtualMemGetAddr (CPLVirtualMem *ctxt)
| Return the pointer to the start of a virtual memory mapping. More...
size_t | CPLVirtualMemGetSize (CPLVirtualMem *ctxt)
| Return the size of the virtual memory mapping. More...
int | CPLVirtualMemIsFileMapping (CPLVirtualMem *ctxt)
| Return if the virtual memory mapping is a direct file mapping. More...
CPLVirtualMemAccessMode | CPLVirtualMemGetAccessMode (CPLVirtualMem *ctxt)
| Return the access mode of the virtual memory mapping. More...
size_t | CPLVirtualMemGetPageSize (CPLVirtualMem *ctxt)
| Return the page size associated to a virtual memory mapping. More...
int | CPLVirtualMemIsAccessThreadSafe (CPLVirtualMem *ctxt)
| Return TRUE if this memory mapping can be accessed safely from concurrent threads. More...
void | CPLVirtualMemDeclareThread (CPLVirtualMem *ctxt)
| Declare that a thread will access a virtual memory mapping. More...
void | CPLVirtualMemUnDeclareThread (CPLVirtualMem *ctxt)
| Declare that a thread will stop accessing a virtual memory mapping. More...
void | CPLVirtualMemPin (CPLVirtualMem *ctxt, void *pAddr, size_t nSize, int bWriteOp)
| Make sure that a region of virtual memory will be realized. More...
void | CPLVirtualMemManagerTerminate (void)
| Cleanup any resource and handlers related to virtual memory. More...

## Detailed Description

Virtual memory management.

This file provides mechanism to define virtual memory mappings, whose content is allocated transparently and filled on-the-fly. Those virtual memory mappings can be much larger than the available RAM, but only parts of the virtual memory mapping, in the limit of the allowed the cache size, will actually be physically allocated.

This exploits low-level mechanisms of the operating system (virtual memory allocation, page protection and handler of virtual memory exceptions).

It is also possible to create a virtual memory mapping from a file or part of a file.

The current implementation is Linux only.

## Typedef Documentation

## ◆ CPLVirtualMemCachePageCbk

typedef void(* CPLVirtualMemCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)

Callback triggered when a still unmapped page of virtual memory is accessed.

The callback has the responsibility of filling the page with relevant values

Parameters

  ctxt | virtual memory handle.
  nOffset | offset of the page in the memory mapping.
  pPageToFill | address of the page to fill. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() nOffset.
  nToFill | number of bytes of the page.
  pUserData | user data that was passed to CPLVirtualMemNew().

## ◆ CPLVirtualMemFreeUserData

typedef void(* CPLVirtualMemFreeUserData) (void *pUserData)

Callback triggered when a virtual memory mapping is destroyed.

Parameters

  pUserData | user data that was passed to CPLVirtualMemNew().

## ◆ CPLVirtualMemUnCachePageCbk

typedef void(* CPLVirtualMemUnCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)

Callback triggered when a dirty mapped page is going to be freed.

(saturation of cache, or termination of the virtual memory mapping).

Parameters

  ctxt | virtual memory handle.
  nOffset | offset of the page in the memory mapping.
  pPageToBeEvicted | address of the page that will be flushed. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() nOffset.
  nToBeEvicted | number of bytes of the page.
  pUserData | user data that was passed to CPLVirtualMemNew().

## Enumeration Type Documentation

## ◆ CPLVirtualMemAccessMode

enum CPLVirtualMemAccessMode

Access mode of a virtual memory mapping.

Enumerator
VIRTUALMEM_READONLY | The mapping is meant at being read-only, but writes will not be prevented. Note that any content written will be lost.
VIRTUALMEM_READONLY_ENFORCED | The mapping is meant at being read-only, and this will be enforced through the operating system page protection mechanism.
VIRTUALMEM_READWRITE | The mapping is meant at being read-write, and modified pages can be saved thanks to the pfnUnCachePage callback

## Function Documentation

## ◆ CPLGetPageSize()

size_t CPLGetPageSize | ( | void |  | ) |

Return the size of a page of virtual memory.

Returns
  the page size.

Since
  GDAL 1.11

## ◆ CPLIsVirtualMemFileMapAvailable()

int CPLIsVirtualMemFileMapAvailable | ( | void |  | ) |

Return if virtual memory mapping of a file is available.

Returns
  TRUE if virtual memory mapping of a file is available.

Since
  GDAL 1.11

## ◆ CPLVirtualMemDeclareThread()

void CPLVirtualMemDeclareThread | ( | CPLVirtualMem * | ctxt | ) |

Declare that a thread will access a virtual memory mapping.

This function must be called by a thread that wants to access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemUnDeclareThread().

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Since
  GDAL 1.11

## ◆ CPLVirtualMemDerivedNew()

CPLVirtualMem * CPLVirtualMemDerivedNew | ( | CPLVirtualMem * | pVMemBase,
|  | vsi_l_offset | nOffset,
|  | vsi_l_offset | nSize,
|  | CPLVirtualMemFreeUserData | pfnFreeUserData,
|  | void * | pCbkUserData
| ) |  |

Create a new virtual memory mapping derived from an other virtual memory mapping.

This may be useful in case of creating mapping for pixel interleaved data.

The new mapping takes a reference on the base mapping.

Parameters

  pVMemBase | Base virtual memory mapping
  nOffset | Offset in the base virtual memory mapping from which to start the new mapping.
  nSize | Size of the base virtual memory mapping to expose in the the new mapping.
  pfnFreeUserData | callback that is called when the object is destroyed.
  pCbkUserData | user data passed to pfnFreeUserData.

Returns
  a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since
  GDAL 1.11

## ◆ CPLVirtualMemFileMapNew()

CPLVirtualMem * CPLVirtualMemFileMapNew | ( | VSILFILE * | fp,
|  | vsi_l_offset | nOffset,
|  | vsi_l_offset | nLength,
|  | CPLVirtualMemAccessMode | eAccessMode,
|  | CPLVirtualMemFreeUserData | pfnFreeUserData,
|  | void * | pCbkUserData
| ) |  |

Create a new virtual memory mapping from a file.

The file must be a "real" file recognized by the operating system, and not a VSI extended virtual file.

In VIRTUALMEM_READWRITE mode, updates to the memory mapping will be written in the file.

On Linux AMD64 platforms, the maximum value for nLength is 128 TB. On Linux x86 platforms, the maximum value for nLength is 2 GB.

Supported on Linux only in GDAL <= 2.0, and all POSIX systems supporting mmap() in GDAL >= 2.1

Parameters

  fp | Virtual file handle.
  nOffset | Offset in the file to start the mapping from.
  nLength | Length of the portion of the file to map into memory.
  eAccessMode | Permission to use for the virtual memory mapping. This must be consistent with how the file has been opened.
  pfnFreeUserData | callback that is called when the object is destroyed.
  pCbkUserData | user data passed to pfnFreeUserData.

Returns
  a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since
  GDAL 1.11

## ◆ CPLVirtualMemFree()

void CPLVirtualMemFree | ( | CPLVirtualMem * | ctxt | ) |

Free a virtual memory mapping.

The pointer returned by CPLVirtualMemGetAddr() will no longer be valid. If the virtual memory mapping was created with read/write permissions and that they are dirty (i.e. modified) pages, they will be flushed through the pfnUnCachePage callback before being freed.

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Since
  GDAL 1.11

## ◆ CPLVirtualMemGetAccessMode()

CPLVirtualMemAccessMode CPLVirtualMemGetAccessMode | ( | CPLVirtualMem * | ctxt | ) |

Return the access mode of the virtual memory mapping.

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Returns
  the access mode of the virtual memory mapping.

Since
  GDAL 1.11

## ◆ CPLVirtualMemGetAddr()

void * CPLVirtualMemGetAddr | ( | CPLVirtualMem * | ctxt | ) |

Return the pointer to the start of a virtual memory mapping.

The bytes in the range [p:p+CPLVirtualMemGetSize()-1] where p is the pointer returned by this function will be valid, until CPLVirtualMemFree() is called.

Note that if a range of bytes used as an argument of a system call (such as read() or write()) contains pages that have not been "realized", the system call will fail with EFAULT. CPLVirtualMemPin() can be used to work around this issue.

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Returns
  the pointer to the start of a virtual memory mapping.

Since
  GDAL 1.11

## ◆ CPLVirtualMemGetPageSize()

size_t CPLVirtualMemGetPageSize | ( | CPLVirtualMem * | ctxt | ) |

Return the page size associated to a virtual memory mapping.

The value returned will be at least CPLGetPageSize(), but potentially larger.

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Returns
  the page size

Since
  GDAL 1.11

## ◆ CPLVirtualMemGetSize()

size_t CPLVirtualMemGetSize | ( | CPLVirtualMem * | ctxt | ) |

Return the size of the virtual memory mapping.

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Returns
  the size of the virtual memory mapping.

Since
  GDAL 1.11

## ◆ CPLVirtualMemIsAccessThreadSafe()

int CPLVirtualMemIsAccessThreadSafe | ( | CPLVirtualMem * | ctxt | ) |

Return TRUE if this memory mapping can be accessed safely from concurrent threads.

The situation that can cause problems is when several threads try to access a page of the mapping that is not yet mapped.

The return value of this function depends on whether bSingleThreadUsage has been set of not in CPLVirtualMemNew() and/or the implementation.

On Linux, this will always return TRUE if bSingleThreadUsage = FALSE.

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Returns
  TRUE if this memory mapping can be accessed safely from concurrent threads.

Since
  GDAL 1.11

## ◆ CPLVirtualMemIsFileMapping()

int CPLVirtualMemIsFileMapping | ( | CPLVirtualMem * | ctxt | ) |

Return if the virtual memory mapping is a direct file mapping.

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Returns
  TRUE if the virtual memory mapping is a direct file mapping.

Since
  GDAL 1.11

## ◆ CPLVirtualMemManagerTerminate()

void CPLVirtualMemManagerTerminate | ( | void |  | ) |

Cleanup any resource and handlers related to virtual memory.

This function must be called after the last CPLVirtualMem object has been freed.

Since
  GDAL 2.0

## ◆ CPLVirtualMemNew()

CPLVirtualMem * CPLVirtualMemNew | ( | size_t | nSize,
|  | size_t | nCacheSize,
|  | size_t | nPageSizeHint,
|  | int | bSingleThreadUsage,
|  | CPLVirtualMemAccessMode | eAccessMode,
|  | CPLVirtualMemCachePageCbk | pfnCachePage,
|  | CPLVirtualMemUnCachePageCbk | pfnUnCachePage,
|  | CPLVirtualMemFreeUserData | pfnFreeUserData,
|  | void * | pCbkUserData
| ) |  |

Create a new virtual memory mapping.

This will reserve an area of virtual memory of size nSize, whose size might be potentially much larger than the physical memory available. Initially, no physical memory will be allocated. As soon as memory pages will be accessed, they will be allocated transparently and filled with the pfnCachePage callback. When the allowed cache size is reached, the least recently used pages will be unallocated.

On Linux AMD64 platforms, the maximum value for nSize is 128 TB. On Linux x86 platforms, the maximum value for nSize is 2 GB.

Only supported on Linux for now.

Note that on Linux, this function will install a SIGSEGV handler. The original handler will be restored by CPLVirtualMemManagerTerminate().

Parameters

  nSize | size in bytes of the virtual memory mapping.
  nCacheSize | size in bytes of the maximum memory that will be really allocated (must ideally fit into RAM).
  nPageSizeHint | hint for the page size. Must be a multiple of the system page size, returned by CPLGetPageSize(). Minimum value is generally 4096. Might be set to 0 to let the function determine a default page size.
  bSingleThreadUsage | set to TRUE if there will be no concurrent threads that will access the virtual memory mapping. This can optimize performance a bit.
  eAccessMode | permission to use for the virtual memory mapping.
  pfnCachePage | callback triggered when a still unmapped page of virtual memory is accessed. The callback has the responsibility of filling the page with relevant values.
  pfnUnCachePage | callback triggered when a dirty mapped page is going to be freed (saturation of cache, or termination of the virtual memory mapping). Might be NULL.
  pfnFreeUserData | callback that can be used to free pCbkUserData. Might be NULL
  pCbkUserData | user data passed to pfnCachePage and pfnUnCachePage.

Returns
  a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since
  GDAL 1.11

## ◆ CPLVirtualMemPin()

void CPLVirtualMemPin | ( | CPLVirtualMem * | ctxt,
|  | void * | pAddr,
|  | size_t | nSize,
|  | int | bWriteOp
| ) |  |

Make sure that a region of virtual memory will be realized.

Calling this function is not required, but might be useful when debugging a process with tools like gdb or valgrind that do not naturally like segmentation fault signals.

It is also needed when wanting to provide part of virtual memory mapping to a system call such as read() or write(). If read() or write() is called on a memory region not yet realized, the call will fail with EFAULT.

Parameters

  ctxt | context returned by CPLVirtualMemNew().
  pAddr | the memory region to pin.
  nSize | the size of the memory region.
  bWriteOp | set to TRUE if the memory are will be accessed in write mode.

Since
  GDAL 1.11

## ◆ CPLVirtualMemUnDeclareThread()

void CPLVirtualMemUnDeclareThread | ( | CPLVirtualMem * | ctxt | ) |

Declare that a thread will stop accessing a virtual memory mapping.

This function must be called by a thread that will no longer access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemDeclareThread().

Parameters

  ctxt | context returned by CPLVirtualMemNew().

Since
  GDAL 1.11