When the system is low on memory, we want to shrink any cached
system memory ion is holding. Previously we were shrinking memory
in the page pools, but not in the deferred free list. This patch
makes it possible to shrink both. It also moves the shrinker
code into the heaps so they can correctly manage any caches they
might contain.
Change-Id: I177f587f999a5220eddbf7af94745aae736cac75
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
The high variable was sometimes used uninitialized
Change-Id: I2f51413fd2d063fdff325047e824dc8c749d9e0a
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Allocations from the ion heap need to be zeroed to protect userspace
from seeing memory belonging to other processes. First allocations
from this heap were not zero'd allowing users to see memory from other
processes on a warm reset.
Change-Id: I524a7b79cb76c390c870fcf8b30d213185fc85a0
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Add the ability for a heap to free buffers asynchrounously. Freed buffers
are placed on a free list and freed from a low priority background thread.
If allocations from a particular heap fail, the free list is drained. This
patch also enable asynchronous frees from the chunk heap.
Change-Id: Idfdbc8608b6cbd9e27d2e31ea4fd84fea9f69f7d
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Currently ion can only share buffers with dma buf fd's. Fd's can not be
used inside the kernel as they are process specific so support for
sharing buffers with dma buf kernel handles is needed to support kernel
only use cases. An example use case could be a GPU driver using ion
that wants to share its output buffers with a 3d party display
controller driver supporting dma buf.
Change-Id: If1b3753ddbd5b44c5a3e622055d5473e16fc1c48
Signed-off-by: Johan Mossberg <johan.mossberg@stericsson.com>
Refactor the code in the system heap used to map and zero the buffers
into a seperate utility so it can be called from other heaps. Use it from
the chunk heap.
Change-Id: I706341ae42b80bc4aae8a8614b4f73435bbf05d9
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
vmap/vunmap spend a significant amount of time allocating the
address space to map into. Rather than allocating address space
for each page, instead allocate once for the entire allocation
and then just map and unmap each page into that address space.
Change-Id: I4a5c850717c80f75506a36b7ec2bcd55857b8dea
Signed-off-by: Rebecca Schultz Zavin <rschultz@google.com>
The heapmask in the client generally wasn't being used. This
patch removes it.
Change-Id: I3526723fbf8f2e81c28c0733deb583ea14bdd837
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
There is some confusion between when to use the heap type and when
the id. This patch clarifies this by using clearer variable names
and describing the intention in the comments. Also fixes the client
debug code to print heaps by id instead of type.
Change-Id: Ie8b3dadded52e18590fcb2ca94001f6ed46ef07d
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
This patch adds support for a chunk heap that allows for buffers that are
made up of a list of fixed size chunks taken from a carveout. Chunk sizes
are configured when the heaps are created by passing the chunk size in the
priv field of the heap platform data.
Change-Id: Ia9e003f727b553a92804264debe119dcf78b14e0
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
The system heap contained several general purpose functions to map
buffers to the kernel and userspace. This patch refactors those
into ion_heap.c so they can be used by other heaps.
Change-Id: If64591798bdc2c248bf9064ace2c927909d7adb8
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Switches the rbtree tree of heaps for a plist. This significantly
simplifies the code and the list is small and is modified only at
first boot so the rbtree is unnecessary. This also switches
the traversal of the heap list to traverse from highest to lowest
id's. This allows allocations to pass a heap mask that falls
back on the system heap -- typically id 0, which is the common case.
Change-Id: I715be6f4cf020a84ca4f1947c30ee3d2559fb523
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
This patch allows you to specify a heap that requires carveout memory
but that doesn't specify a start address. Memblock_alloc will be called
to find a location for these heaps.
Change-Id: I9c79b30e3105e796060fc74b058f04093ee5e96e
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
use atomic_read to get the refcount value to avoid compilation warning
Change-Id: I9bf0a07c787af8f87ac19314f996a78142f72f6b
Signed-off-by: Benjamin Gaignard <benjamin.gaignard@linaro.org>
when using carveout heap ion_buffer_create function failed because
map_dma and unmap_dma operations aren't set by carveout heap.
Change-Id: I817bfad742abfab10b01d9b3d2e18bdf812a1307
Signed-off-by: Benjamin Gaignard <benjamin.gaignard@linaro.org>
Pages are zeroed for security purposes when returned to the
ion heap. There was a bug in this code preventing this
from happening.
Bug: 7573871
Change-Id: I79c22ee1da98f306199f3a192eaec4e81d5fd059
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
When the requested mmap length was not an integer number of
chunks or the buffer, or if an offset was provided, a bug
would cause extra or incorrect pages of the buffer to be mapped.
Change-Id: I2766763d86048f026eeef0e0388b7de0e25c2093
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
This will prevent the kernel from kicking off compaction
when higher order allocations are made. Instead we will
get these high order allocations only if they are readily
available.
Change-Id: I5c038781ef4028c1c0a1a52b6cb549d26550a124
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Removes contention for lock between allocate and free by reducing
the length of time the lock is held for. Split out a seperate
lock to protect the list of heaps and replace it with a rwsem since
the list will most likely only be updated during initialization.
Change-Id: I3017ccaa2126c7a24ab22b18e1ee553983977c37
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
The single shrink function will free lower order pages first. This
enables compaction to work properly.
Change-Id: Icbeefa9e84c2eacf6962d9536ab66a8b059cc941
Currently the mutex is held while kmalloc is called, under a low memory
condition this might trigger the shrinker which also takes this mutex.
Refactor so the mutex is not held during allocation.
Change-Id: Ic1d3b2f69e61209191bac84724ba56f6b98e2bc4
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Split out low and high mem pages so they are correctly reported
when the shrinker is called.
Fix potential deadlock caused by holding the page pool lock while
allocationg and also needing that lock from the shrink function
Change-Id: I6def3a3111f59914ef6b5b59d81a6fde37cb60c5
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
When allocations larger than order 4 are made, use _GFP_NORETRY
and __GFP_NO_KSWAPD so kswapd doesn't get kicked off to reclaim
these larger chunks. For smaller allocaitons, these are
unnecessary, as the system should be able to reclaim these.
Change-Id: I6b45cd1a595d25cc39622e3dbcc04d84fa1b7db8
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
With this change the system heap will use pagepools to avoid
having to invalidate memory when it is allocated, a
significant performance improvement on some systems.
Change-Id: I96ab778d67c4b19805883dcfa44d750811cdff48
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
This patch adds a new utility heaps can use to manage
memory. In the past we have found it can be very
expensive to manage the caches when allocating memory,
but it is imposible to know whether a previous user of a
given memory allocation had a cached mapping. This patch
adds the ability to store a pool of pages that were
previously used uncached so that cache maintenance
only need be done when growing this pool. The pool also
contains a shrinker so memory from the pool can be
recovered in low memory conditions.
Change-Id: I686fd7d192060fd32d34ef66471f62b7829006ee
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
When ion_map_kernel is execute the system must allocate
an array large enough to hold a pointer to each page in
the buffer. If the buffer is very large and the system
memory has become very fragmented, there may not be
sufficient high order allocations available from kmalloc.
Use vmalloc instead.
Change-Id: I5fabf79be6cfd158f7805bfca6267a60c4708582
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
With this patch the system heap will only try to allocate from each
order as long as allocations succeed. If it failes to obtain a higher
order allocation, it doesn't retry that order.
Change-Id: I0d9144b4c30cc0e427acdcad2f1f12ae7f37f827
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
If a buffer's user mappings are not going to be faulted
in it need not be allocated page wise. We can optimize
this common case by allocating an sglist of larger chunks
rather than creating an entry for each page in the
allocation.
Change-Id: I47814990e55c7bdb7abeaa2af824744b0a97602d
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
We have found that faulting in the mappings for cached
allocations has a significant performance impact and is
only a benefit if only a small part of the buffer is
touched by the cpu (an uncommon case for software rendering).
This patch introduces a ION_FLAG_CACHED_NEEDS_SYNC
which determines whether a mapping should be created by
faulting or at mmap time. If this flag is set,
userspace must manage the caches explictly using the SYNC ioctl.
Change-Id: I227561f49e0f382a481728fb55ac5c930fc26025
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
It is possible for a buffer to exist only as a dma_buf file
descriptor without it being held in any handles. When this
occurs it is impossible to track where the buffer is in the
system (without traversing every process in the system and
inspecting its file table). When buffers are orphaned like
this, copy the task comm and pid of the last client to hold
them into the buffer so we have a debugging hint as to where
this buffer came from. In practice this will probalby be
the process that allocated the buffer.
Change-Id: I49d179f9ccdee36982f365af15d0f0b577c6347e
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
If preemted during ion_free after the refcount is updated but
before the handle can be removed from the rb_tree, import
might find that handle in the tree and try to reuse it
when execution returns to free, the handle will be cleaned
up leaving the caller of import with a corrupt handle.
This patch modifies the locking to protect agains this race.
Change-Id: I31d18cc6398f0ca18e05cd919e2bcf86fa18d568
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
Previously, metadata was stored in the allocated pages themselves
during allocation. However the system can only have a limited
number of kmapped pages. A very large allocation might exceed
this limit.
Change-Id: Ibe972096e83924bf5e621d8282c4cd133ca75b0a
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
If dma_buf_fd fails, the dma_buf needs to be cleaned up by
calling dma_buf_put. dma_buf_put will call ion_dma_buf_release
which in turn calls ion_buffer_put to clean up the buffer
reference. Calling ion_buffer_put after dma_buf_put drops the
reference count by one more which is incorrect. Fix this by
getting rid of the extra ion_buffer_put call.
Change-Id: Ieb3a7b781bf2dc299c45384dbf0a12d186adeb22
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
ION_IOC_MAP, ION_IOC_SHARE, and ION_IOC_IMPORT may return
success when an error occurs.
Add correct error handling to ION_IOC_MAP, ION_IOC_SHARE, and
ION_IOC_IMPORT.
Change-Id: I61c6b0c574da425604235b2d39e13e4a27f7b6f3
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
Several functions in the ion interface is missing
EXPORT_SYMBOL. This is needed to allow clients to
use these functions from kernel modules.
Add EXPORT_SYMBOL to functions that are supposed
to be exposed.
Change-Id: I25611f7c367fd131daf5fb652abbea77e89a3d06
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
The condition argument to the WARN call in ion_free and
ion_share_dma_buf are missing. Add the argument to
allow correct printing of warning message.
Change-Id: I9b233992d5034f8780296cae6b42d77699f14cce
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
This is deprecated in favor of using the dma_buf api which will
automatically sync a buffer to memory when it is mapped to a device.
However, that functionality is not ready, so this patch adds the
ability to sync a buffer explicitly.
Change-Id: Ia15810a13cd5c4b939f4afa5c8e721c89fac76d4
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
On some systems there is a performance benefit to reducing tlb pressure
by minimizing the number of chunks in an allocation.
Change-Id: Ie6ff215a5f32a837bf3103f79a09748e38759f0e
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
This patch adds cache maintenance operations to ion. As per mailing
list discussions regarding dma_buf, cache operations are done implicitly.
At buffer allocaiton time the user can select whether he'd like mappings
(both kernel and user) to be cached. When cached mappings are selected,
no mappings will be created for a buffer at mmap time. Instead pages will
be faulted in one at a time so we can track which pages require flushing
before dma. When the buffers are mapped for dma (via the dma_buf apis)
any pages which were touched will be synced for device.
Change-Id: Id5d6894e8bb52af038c91dd895143bf3b4203b0b
Signed-off-by: Rebecca Schultz Zavin <rebecca@android.com>
