Chunk memory (The Polipo Manual)

Next: Malloc memory, Previous: Memory usage, Up: Memory usage [Contents][Index]

### 5.1 Chunk memory

Most of the memory used by Polipo is stored in chunks, fixed-size blocks of memory; the size of a chunk is defined by the compile-time constant `CHUNK_SIZE`, and defaults to 4096 bytes on 32-bit platforms, 8192 on 64-bit ones. Chunks are used for storing object data (bodies of instances) and for temporary I/O buffers. Increasing the chunk size increases performance somewhat, but at the cost of larger granularity of allocation and hence larger memory usage.

By default, Polipo uses a hand-crafted memory allocator based on `mmap`(2) (`VirtualAlloc` under Windows) for allocating chunks; while this is very slightly faster than the stock memory allocator, its main benefit is that it limits memory fragmentation. It is possible to disable the chunk allocator, and use `malloc`(3) for all memory allocation, by defining `MALLOC_CHUNKS` at compile time; this is probably only useful for debugging.

There is one assumption made about `CHUNK_SIZE`: `CHUNK_SIZE` multiplied by the number of bits in an `unsigned long` (actually in a `ChunkBitmap` — see `chunk.c`) must be a multiple of the page size, which is 4096 on most systems (8192 on Alpha, and apparently 65536 on Windows).

As all network I/O will be performed in units of one to two chunks, `CHUNK_SIZE` should be at least equal to your network interface’s MTU (typically 1500 bytes). Additionally, as much I/O will be done at `CHUNK_SIZE`-aligned addresses, `CHUNK_SIZE` should ideally be a multiple of the page size.

In summary, 2048, 4096, 8192 and 16384 are good choices for `CHUNK_SIZE`.