Typedefs | Enumerations | Functions

# cpl_virtualmem.h File Reference

`#include "cpl_port.h"`
`#include "cpl_vsi.h"`

Go to the source code of this file.

## Typedefs
typedef struct CPLVirtualMem | CPLVirtualMem
typedef void(* | CPLVirtualMemCachePageCbk )(CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)
typedef void(* | CPLVirtualMemUnCachePageCbk )(CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)
typedef void(* | CPLVirtualMemFreeUserData )(void *pUserData)
## Enumerations
enum | CPLVirtualMemAccessMode { VIRTUALMEM_READONLY, VIRTUALMEM_READONLY_ENFORCED, VIRTUALMEM_READWRITE }
## Functions
size_t | CPLGetPageSize (void)
CPLVirtualMem * | CPLVirtualMemNew (size_t nSize, size_t nCacheSize, size_t nPageSizeHint, int bSingleThreadUsage, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemCachePageCbk pfnCachePage, CPLVirtualMemUnCachePageCbk pfnUnCachePage, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
int | CPLIsVirtualMemFileMapAvailable (void)
CPLVirtualMem * | CPLVirtualMemFileMapNew (VSILFILE *fp, vsi_l_offset nOffset, vsi_l_offset nLength, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
CPLVirtualMem * | CPLVirtualMemDerivedNew (CPLVirtualMem *pVMemBase, vsi_l_offset nOffset, vsi_l_offset nSize, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
void | CPLVirtualMemFree (CPLVirtualMem *ctxt)
void * | CPLVirtualMemGetAddr (CPLVirtualMem *ctxt)
size_t | CPLVirtualMemGetSize (CPLVirtualMem *ctxt)
int | CPLVirtualMemIsFileMapping (CPLVirtualMem *ctxt)
CPLVirtualMemAccessMode | CPLVirtualMemGetAccessMode (CPLVirtualMem *ctxt)
size_t | CPLVirtualMemGetPageSize (CPLVirtualMem *ctxt)
int | CPLVirtualMemIsAccessThreadSafe (CPLVirtualMem *ctxt)
void | CPLVirtualMemDeclareThread (CPLVirtualMem *ctxt)
void | CPLVirtualMemUnDeclareThread (CPLVirtualMem *ctxt)
void | CPLVirtualMemPin (CPLVirtualMem *ctxt, void *pAddr, size_t nSize, int bWriteOp)
void | CPLVirtualMemManagerTerminate (void)

## Detailed Description

Virtual memory management.

This file provides mechanism to define virtual memory mappings, whose content is allocated transparently and filled on-the-fly. Those virtual memory mappings can be much larger than the available RAM, but only parts of the virtual memory mapping, in the limit of the allowed the cache size, will actually be physically allocated.

This exploits low-level mechanisms of the operating system (virtual memory allocation, page protection and handler of virtual memory exceptions).

It is also possible to create a virtual memory mapping from a file or part of a file.

The current implementation is Linux only.

## Typedef Documentation

typedef struct CPLVirtualMem CPLVirtualMem

Opaque type that represents a virtual memory mapping.

typedef void(* CPLVirtualMemCachePageCbk)(CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)

Callback triggered when a still unmapped page of virtual memory is accessed. The callback has the responsibility of filling the page with relevant values

Parameters:

  | ctxt | virtual memory handle.
  | nOffset | offset of the page in the memory mapping.
  | pPageToFill | address of the page to fill. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
  | nToFill | number of bytes of the page.
  | pUserData | user data that was passed to CPLVirtualMemNew().

typedef void(* CPLVirtualMemFreeUserData)(void *pUserData)

Callback triggered when a virtual memory mapping is destroyed.

Parameters:

  | pUserData | user data that was passed to CPLVirtualMemNew().

typedef void(* CPLVirtualMemUnCachePageCbk)(CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)

Callback triggered when a dirty mapped page is going to be freed. (saturation of cache, or termination of the virtual memory mapping).

Parameters:

  | ctxt | virtual memory handle.
  | nOffset | offset of the page in the memory mapping.
  | pPageToBeEvicted | address of the page that will be flushed. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
  | nToBeEvicted | number of bytes of the page.
  | pUserData | user data that was passed to CPLVirtualMemNew().

## Enumeration Type Documentation

enum CPLVirtualMemAccessMode

Access mode of a virtual memory mapping.

Enumerator:

  VIRTUALMEM_READONLY | The mapping is meant at being read-only, but writes will not be prevented. Note that any content written will be lost.
  VIRTUALMEM_READONLY_ENFORCED | The mapping is meant at being read-only, and this will be enforced through the operating system page protection mechanism.
  VIRTUALMEM_READWRITE | The mapping is meant at being read-write, and modified pages can be saved thanks to the pfnUnCachePage callback

## Function Documentation

size_t CPLGetPageSize | ( | void |  | ) |

Return the size of a page of virtual memory.

Returns:
  the page size.

Since:
  GDAL 2.0

int CPLIsVirtualMemFileMapAvailable | ( | void |  | ) |

Return if virtual memory mapping of a file is available.

Returns:
  TRUE if virtual memory mapping of a file is available.

Since:
  GDAL 2.0

void CPLVirtualMemDeclareThread | ( | CPLVirtualMem * | ctxt | ) |

Declare that a thread will access a virtual memory mapping.

This function must be called by a thread that wants to access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemUnDeclareThread().

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Since:
  GDAL 2.0

CPLVirtualMem* CPLVirtualMemDerivedNew | ( | CPLVirtualMem * | pVMemBase,
|  | vsi_l_offset | nOffset,
|  | vsi_l_offset | nSize,
|  | CPLVirtualMemFreeUserData | pfnFreeUserData,
|  | void * | pCbkUserData |
| ) |  |  |

Create a new virtual memory mapping derived from an other virtual memory mapping.

This may be usefull in case of creating mapping for pixel interleaved data.

The new mapping takes a reference on the base mapping.

Parameters:

  | pVMemBase | Base virtual memory mapping
  | nOffset | Offset in the base virtual memory mapping from which to start the new mapping.
  | nSize | Size of the base virtual memory mapping to expose in the the new mapping.
  | pfnFreeUserData | callback that is called when the object is destroyed.
  | pCbkUserData | user data passed to pfnFreeUserData.

Returns:
  a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since:
  GDAL 2.0

CPLVirtualMem* CPLVirtualMemFileMapNew | ( | VSILFILE * | fp,
|  | vsi_l_offset | nOffset,
|  | vsi_l_offset | nLength,
|  | CPLVirtualMemAccessMode | eAccessMode,
|  | CPLVirtualMemFreeUserData | pfnFreeUserData,
|  | void * | pCbkUserData |
| ) |  |  |

Create a new virtual memory mapping from a file.

The file must be a "real" file recognized by the operating system, and not a VSI extended virtual file.

In VIRTUALMEM_READWRITE mode, updates to the memory mapping will be written in the file.

On Linux AMD64 platforms, the maximum value for nLength is 128 TB. On Linux x86 platforms, the maximum value for nLength is 2 GB.

Only supported on Linux for now.

Parameters:

  | fp | Virtual file handle.
  | nOffset | Offset in the file to start the mapping from.
  | nLength | Length of the portion of the file to map into memory.
  | eAccessMode | Permission to use for the virtual memory mapping. This must be consistant with how the file has been opened.
  | pfnFreeUserData | callback that is called when the object is destroyed.
  | pCbkUserData | user data passed to pfnFreeUserData.

Returns:
  a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since:
  GDAL 2.0

void CPLVirtualMemFree | ( | CPLVirtualMem * | ctxt | ) |

Free a virtual memory mapping.

The pointer returned by CPLVirtualMemGetAddr() will no longer be valid. If the virtual memory mapping was created with read/write permissions and that they are dirty (i.e. modified) pages, they will be flushed through the pfnUnCachePage callback before being freed.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Since:
  GDAL 2.0

CPLVirtualMemAccessMode CPLVirtualMemGetAccessMode | ( | CPLVirtualMem * | ctxt | ) |

Return the access mode of the virtual memory mapping.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Returns:
  the access mode of the virtual memory mapping.

Since:
  GDAL 2.0

void* CPLVirtualMemGetAddr | ( | CPLVirtualMem * | ctxt | ) |

Return the pointer to the start of a virtual memory mapping.

The bytes in the range [p:p+CPLVirtualMemGetSize()-1] where p is the pointer returned by this function will be valid, until CPLVirtualMemFree() is called.

Note that if a range of bytes used as an argument of a system call (such as read() or write()) contains pages that have not been "realized", the system call will fail with EFAULT. CPLVirtualMemPin() can be used to work around this issue.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Returns:
  the pointer to the start of a virtual memory mapping.

Since:
  GDAL 2.0

size_t CPLVirtualMemGetPageSize | ( | CPLVirtualMem * | ctxt | ) |

Return the page size associated to a virtual memory mapping.

The value returned will be at least CPLGetPageSize(), but potentially larger.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Returns:
  the page size

Since:
  GDAL 2.0

size_t CPLVirtualMemGetSize | ( | CPLVirtualMem * | ctxt | ) |

Return the size of the virtual memory mapping.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Returns:
  the size of the virtual memory mapping.

Since:
  GDAL 2.0

int CPLVirtualMemIsAccessThreadSafe | ( | CPLVirtualMem * | ctxt | ) |

Return TRUE if this memory mapping can be accessed safely from concurrent threads.

The situation that can cause problems is when several threads try to access a page of the mapping that is not yet mapped.

The return value of this function depends on whether bSingleThreadUsage has been set of not in CPLVirtualMemNew() and/or the implementation.

On Linux, this will always return TRUE if bSingleThreadUsage = FALSE.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Returns:
  TRUE if this memory mapping can be accessed safely from concurrent threads.

Since:
  GDAL 2.0

int CPLVirtualMemIsFileMapping | ( | CPLVirtualMem * | ctxt | ) |

Return if the virtal memory mapping is a direct file mapping.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Returns:
  TRUE if the virtal memory mapping is a direct file mapping.

Since:
  GDAL 2.0

void CPLVirtualMemManagerTerminate | ( | void |  | ) |

Cleanup any resource and handlers related to virtual memory.

This function must be called after the last CPLVirtualMem object has been freed.

Since:
  GDAL 2.0

CPLVirtualMem* CPLVirtualMemNew | ( | size_t | nSize,
|  | size_t | nCacheSize,
|  | size_t | nPageSizeHint,
|  | int | bSingleThreadUsage,
|  | CPLVirtualMemAccessMode | eAccessMode,
|  | CPLVirtualMemCachePageCbk | pfnCachePage,
|  | CPLVirtualMemUnCachePageCbk | pfnUnCachePage,
|  | CPLVirtualMemFreeUserData | pfnFreeUserData,
|  | void * | pCbkUserData |
| ) |  |  |

Create a new virtual memory mapping.

This will reserve an area of virtual memory of size nSize, whose size might be potentially much larger than the physical memory available. Initially, no physical memory will be allocated. As soon as memory pages will be accessed, they will be allocated transparently and filled with the pfnCachePage callback. When the allowed cache size is reached, the least recently used pages will be unallocated.

On Linux AMD64 platforms, the maximum value for nSize is 128 TB. On Linux x86 platforms, the maximum value for nSize is 2 GB.

Only supported on Linux for now.

Note that on Linux, this function will install a SIGSEGV handler. The original handler will be restored by CPLVirtualMemManagerTerminate().

Parameters:

  | nSize | size in bytes of the virtual memory mapping.
  | nCacheSize | size in bytes of the maximum memory that will be really allocated (must ideally fit into RAM).
  | nPageSizeHint | hint for the page size. Must be a multiple of the system page size, returned by CPLGetPageSize(). Minimum value is generally 4096. Might be set to 0 to let the function determine a default page size.
  | bSingleThreadUsage | set to TRUE if there will be no concurrent threads that will access the virtual memory mapping. This can optimize performance a bit.
  | eAccessMode | permission to use for the virtual memory mapping.
  | pfnCachePage | callback triggered when a still unmapped page of virtual memory is accessed. The callback has the responsibility of filling the page with relevant values.
  | pfnUnCachePage | callback triggered when a dirty mapped page is going to be freed (saturation of cache, or termination of the virtual memory mapping). Might be NULL.
  | pfnFreeUserData | callback that can be used to free pCbkUserData. Might be NULL
  | pCbkUserData | user data passed to pfnCachePage and pfnUnCachePage.

Returns:
  a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since:
  GDAL 2.0

void CPLVirtualMemPin | ( | CPLVirtualMem * | ctxt,
|  | void * | pAddr,
|  | size_t | nSize,
|  | int | bWriteOp |
| ) |  |  |

Make sure that a region of virtual memory will be realized.

Calling this function is not required, but might be usefull when debugging a process with tools like gdb or valgrind that do not naturally like segmentation fault signals.

It is also needed when wanting to provide part of virtual memory mapping to a system call such as read() or write(). If read() or write() is called on a memory region not yet realized, the call will fail with EFAULT.

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().
  | pAddr | the memory region to pin.
  | nSize | the size of the memory region.
  | bWriteOp | set to TRUE if the memory are will be accessed in write mode.

Since:
  GDAL 2.0

void CPLVirtualMemUnDeclareThread | ( | CPLVirtualMem * | ctxt | ) |

Declare that a thread will stop accessing a virtual memory mapping.

This function must be called by a thread that will no longer access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemDeclareThread().

Parameters:

  | ctxt | context returned by CPLVirtualMemNew().

Since:
  GDAL 2.0

Generated for GDAL by 1.7.1.