[GAP] Effect of using malloc for temporary allocations in GAP

Frank Lübeck frank.luebeck at math.rwth-aachen.de
Tue Jan 3 17:32:54 GMT 2017


Dear All,

I'll give some comments on the original question of this thread.

On Mon, Jan 02, 2017 at 01:45:33PM +0100, Max Horn wrote:
> Hi there,
> 
> I would like to double-check what the status of using malloc/free in GAP
> for allocations is. So far I operated under the following assumptions,
> based on how I recall past discussions on this:
>
> (1) on 32 bit systems, using malloc/free can interfere negatively with
> the GC, as a malloc'ed region of memory can prevent the GC from growing
> its heap. 
> (2) on 64 bit systems, this is also the case, but the chance
> for this is much, much lower (due to the larger address space), and hence
> it is mostly safe to use malloc/free -- but this is only a heuristic, in
> practice you could be unlucky and still end up with a GAP crashing because
> it cannot extend its heap anymore, despite there being plenty of free RAM.

This was the case until I introduced (around 2007) a '-s' option. 
With this option a huge amount of memory if allocated at startup; only a
part of it is usually used and this used part is extended if necessary.

In about 2012 we made this option the default and Max N. improved the code
to use 'mmap' and 'madvise' for the initial allocation and a possible later
extension of that space (if available).

You can see with 'top' that a 32-bit GAP by default has allocated 1.5 GB of
virtual space but is using only a part of it (128 MB?). A 64-bit GAP
allocates by default 4 GB of virtual memory. These amounts can be changed
with the -s option.

I think, this is currently not used by the Windows version, where 'sbrk' is
still used. I don't know why. So, Windows may a problem for the remarks
below.

> Next, I was wondering about *temporary* allocations... Is the following
> true or false?
>
> (3) It is OK to malloc some space, as long as you free it before the GC
> runs next (resp. before the GC tries to grow its heap)
> 
> It would be nice if this could be either confirmed or refuted by those who
> know our GC well enough, and also the memory managers .

In the case that the -s option is used you can use malloc and free as you
like and it will not cause a problem for GASMAN because it uses the
mentioned virtual memory to extend its workspace if needed.

In the case that GAP gets its memory with 'sbrk' from the system, you can
still be lucky with using malloc/free if you use the '-a' option with a big 
enough value (which preallocates some memory at startup and frees it 
afterwards). But in detail this depends on the 'malloc' implementation you
use, and probably on the chunk sizes you want to malloc.

> Grepping through the GAP kernel reveals that GAP_rl_func sometimes mallocs
> some temporary space. So I am inclined to think (3) holds, but would like
> to know for sure. (I also note some malloc() calls in InitSystem() from
> system.c, but those are only executed if SyAllocPool==0, which can only
> happen on CYGWIN32).

Yes, for small chunks of memory as in these examples the technique with the 
-a option already worked fine in practice. And with the -s option this is no
longer a problem.

> The reason why I am asking is because I've been recently looking into our
> GMP interface, and I noticed two things:
>
> (a) We are using the low-level GMP interfaces, but these can in fact also
> make temporary allocations -- e.g. mpn_gcd may do so. So I guess this is
> another data points towards (3) being true.
>
> (b) If (3) is true, then we could use the mpz_* APIs, too -- while this
> would require us to perform some copying around, I am pretty sure that
> we would still see a noticeable net-benefit for computing things like
> factorials, binomials, roots, PowerModInt, various number theoretical
> stuff (see <https://gmplib.org/manual/Number-Theoretic-Functions.html>)
> etc. It is quite tempting to just try this, but it only makes sense if it
> does not cause issues down the road.

So far, GMP was only used on mpn level to get improved versions of the
functions which were implemented in the old src/integer.c file.
I support the idea to abandon this old code. And so it makes now sense to 
make more of the higher level functions of GMP available at GAP level.

Best regards,
  Frank

PS concerning a change of GAP's GC: It may be nice to have a new GC which is 
easier to use with libraries that use malloc/free and other software
with its own memory management. But this is a highly non-trivial task. 
It is certainly not necessary in the context asked here by Max. But the
question may arise (again) when we work on efficient interfaces to new
functionality for GAP.

-- 
///  Dr. Frank Lübeck, Lehrstuhl D für Mathematik, Pontdriesch 14/16,
\\\                    52062 Aachen, Germany
///  E-mail: Frank.Luebeck at Math.RWTH-Aachen.De
\\\  WWW:    http://www.math.rwth-aachen.de/~Frank.Luebeck/



More information about the Gap mailing list