Tangential, but the complexity and over-head of the malloc abstraction is precisely what made me very intrigued by "stack-only" languages, such a recently published Goose. There is something very elegant and exciting about the idea of (almost) completely getting rid of it. Just as modern programming languages have developed good ergonomics that replaces a garbage collector, the same could hopefully be done to replace heap-allocated memory.
This is actually what first drew me to zig. Most people talk about error handling, but the explicit allocator handling and making choosing an allocator a deliberate consideration has pushed me to better coding habits in general. I don't think it is possible or practical to completely abandon heap-allocated memory, but explicitly choosing between allocation types and styles that most precisely fit a job has great potential for meeting that spirit.
Avoiding the heap entirely isn't required and will prevent you from working with substantial amounts of data. The real key is to allocate all the memory you need up front and use that large block for a long duration.
When getting into wasm when it was new and had clang support, I wrote my own libc and malloc, which was so a great learning all these low level functions and tricks (free lists for example), I also like that I can read wasm quite well, and tooling is good to just return a .wat and read it.
This shows you how you could "secure" memory code in C by using more robust implementations of these functions. But it doesn't show you that you don't even need them anyway. I think in my entire career I've only used dynamic memory in C twice. The rest of the time I use functions with static allocations, buffers, unrolled loops, etc. Maybe it's paranoia, or habits from embedded programming, or being a control freak... But my C programs don't get memory leaks or crash from memory issues.
One of my pet peeves is when people, trying to make a
point, use code that doesn't compile. I mean, it's fine
when you're sketching out code on a white board. But on
an article you've had time to prepare, it's a little
annoying.
Take for instance, this code fragment:
.. static void *ptrs[1000];
..
.. int main(void) {
.. for (int i = 0; i < 1000; i++)
.. ptrs[i] = malloc(16);
.. }
.. }
I'm pretty sure what they //meant// to write was:
.. #include <stdlib.h>
..
.. static void *ptrs[1000];
..
.. int main(void) {
.. for (int i = 0; i < 1000; i++)
.. ptrs[i] = malloc(16);
.. }
.. return( 0 );
.. }
I understand why the author produced the code on top
instead of the code at the bottom. The code on the bottom
includes features extraneous to the conversation. The
include directive and return statement only serve to
distract from the point the author is making.
If you read far enough down, you'll see he has links to a
broken GitHub repo presumably with code fragments. (or
maybe GitHub itself is broken today or it decided it
didn't like me trying to access the repo from an AT&T
fiber line.)
I'm sure when GitHub decides to behave and serve bits like
it's supposed to, there will be compilable code so readers
can understand the author's intent within the context it
was intended to be viewed.
Which is to say:
1. It's probably okay to have only code fragments in the
main part of your discussion. Please provide a note
that there is code which shows your snippet in the
context of a compilable program. And put that note
somewhere near the code fragment.
Or... consider putting a note at the beginning of the
article saying something like "Compilable versions of
code fragments in this article are available at a
GitHub repository referenced at the bottom of the
page."
2. Do not depend on GitHub to hold your code fragments.
You do not know when it will be down or which readers
it will randomly decide to block.
3. I really do think you should include more of the
"layout" code. When reading "The Address Space"
section, it was unclear where the initialized,
uninitialized and local came from. I initially thought
the author was using a version of C I had not yet
encountered that defined these symbols.
4. Since the code uses ``sbrk()``, the author should
probably have cautioned readers the code isn't POSIX
compliant, but should compile and run on Linux systems
using gcc. There's a comment about demonstrating it
with glibc, but I think that's not a strong enough
warning since you can compile programs with clang that
are intended to link against glibc.
I doubt there are many QNX users out there who expect
random code from the internet to run on their systems
without alteration, but it seems good form to describe
the environment you are targeting.
Now I'm wondering if this is expected to work on Darwin
as well as Linux.
5. It might be a decent idea to order the SHOW()
statements in the code in "The Address Space" section
to follow the order of the diagram at the bottom of
that section:
.. SHOW( "local (stack)", &local );
.. SHOW( "printf (libc)", printf );
.. SHOW( "program break", sbrk(0) );
.. SHOW( "malloc(16) (heap)", malloc(16) );
.. SHOW( "uninitialised (bss)", &uninitialised);
.. SHOW( "initialised (data)", &initialised );
.. SHOW( "main(text)", main );
6. In the "Chunks and Headers" section, it may be
interesting to mention what the flags in the allocation
headers actually mean.
7. glibc does late fill, right? I can't remember. If it
does, it might be useful to mention in the Demand
Paging section since it affects the program logic.
Now that I've yelled at the cloud a bit, allow me to say,
I enjoyed the article. Things I liked:
1. The author used the exact strace command they used to
produce the output discussed in the text of the
article.
2. The graphics are quite nice.
3. Despite grousing about fragments being fragments, they
do a good job of illustrating points made in the text.
EDIT: After complaining about GitHub not being available,
it's finally come back. The web interface is still borked
for me, but I can clone the repo mentioned at the bottom
of the page and view the local copy of the source.
like many C functions you must remember that C is not low level, you won’t learn how memory is allocated by inspecting malloc() anymore than you would learn how characters are printed to the screen by reading printf() or how a file is written by reading write().
you will only learn about the thin or heavy wrapper that C is responsible for, the real magic is happening in the kernel and the device firmware
Disagree, I found malloc source useful, and writing a toy allocator especially useful because you learn about the brk() syscall which is a great breadcrumb into kernel land and a great bridge to connect the memory mapping lessons from computer architecture with application code.
Depending on how low-level you're going, you don't even need to call brk.
My toy stop/copy GC and allocation routines just have this:
%define HEAP_SIZE (4 * 1024 * 1024 * 1024)
.section bss
;;
;; Two heaps so we can flip-flop between them
;;
heap_a:
resb HEAP_SIZE
heap_a_end:
heap_b:
resb HEAP_SIZE
heap_b_end:
Of course there are other syscalls involved in my compiler/runtime, but for memory allocations the static area sufficed for me.
> and writing a toy allocator especially useful because you learn about the brk() syscall
Nobody in their right mind (except musl) use brk(2) nowadays; mmap(2) is pretty much universally accessible, and brk() has been non-portable for more than 20 years already (POSIX ditched it in 2001) anyhow. For goodness sake, stop keeping this stupid API from VM-lacking PDP-7 alive, and let it go away!
Malloc needs the kernel to obtain memory, but in any reasonable implementation (you do not want to make a syscall for every malloc and free) is mostly user space, and studying that will tell you a lot about how your malloc works.
It's a consumer market scarcity. Since you're supposed to run everything on rented hardware, own nothing and be happy, I'd expect the malloc to be the right place to invest into once it is IPO.
Tangential, but the complexity and over-head of the malloc abstraction is precisely what made me very intrigued by "stack-only" languages, such a recently published Goose. There is something very elegant and exciting about the idea of (almost) completely getting rid of it. Just as modern programming languages have developed good ergonomics that replaces a garbage collector, the same could hopefully be done to replace heap-allocated memory.
This is actually what first drew me to zig. Most people talk about error handling, but the explicit allocator handling and making choosing an allocator a deliberate consideration has pushed me to better coding habits in general. I don't think it is possible or practical to completely abandon heap-allocated memory, but explicitly choosing between allocation types and styles that most precisely fit a job has great potential for meeting that spirit.
I don't think we can get rid of heap memory, but I do think we can get rid of a shared heap.
Avoiding the heap entirely isn't required and will prevent you from working with substantial amounts of data. The real key is to allocate all the memory you need up front and use that large block for a long duration.
Embedded programming has taught me that almost anything can be a stack-only language if you want it to be, even C++. :)
Most stack oriented languages also have a heap.
the stack is just a bump allocator. the unique thing about stack-only languages is lifetime management, not really how they allocate.
A stack only language that called into malloc to service allocations would not be very interesting.
When getting into wasm when it was new and had clang support, I wrote my own libc and malloc, which was so a great learning all these low level functions and tricks (free lists for example), I also like that I can read wasm quite well, and tooling is good to just return a .wat and read it.
I found this one very well explained time ago, for those looking for more details: https://sploitfun.wordpress.com/2015/02/10/understanding-gli...
That's a great reference, thanks!
This shows you how you could "secure" memory code in C by using more robust implementations of these functions. But it doesn't show you that you don't even need them anyway. I think in my entire career I've only used dynamic memory in C twice. The rest of the time I use functions with static allocations, buffers, unrolled loops, etc. Maybe it's paranoia, or habits from embedded programming, or being a control freak... But my C programs don't get memory leaks or crash from memory issues.
like many C functions you must remember that C is not low level, you won’t learn how memory is allocated by inspecting malloc() anymore than you would learn how characters are printed to the screen by reading printf() or how a file is written by reading write().
you will only learn about the thin or heavy wrapper that C is responsible for, the real magic is happening in the kernel and the device firmware
Disagree, I found malloc source useful, and writing a toy allocator especially useful because you learn about the brk() syscall which is a great breadcrumb into kernel land and a great bridge to connect the memory mapping lessons from computer architecture with application code.
Depending on how low-level you're going, you don't even need to call brk.
My toy stop/copy GC and allocation routines just have this:
Of course there are other syscalls involved in my compiler/runtime, but for memory allocations the static area sufficed for me.> and writing a toy allocator especially useful because you learn about the brk() syscall
Nobody in their right mind (except musl) use brk(2) nowadays; mmap(2) is pretty much universally accessible, and brk() has been non-portable for more than 20 years already (POSIX ditched it in 2001) anyhow. For goodness sake, stop keeping this stupid API from VM-lacking PDP-7 alive, and let it go away!
Malloc needs the kernel to obtain memory, but in any reasonable implementation (you do not want to make a syscall for every malloc and free) is mostly user space, and studying that will tell you a lot about how your malloc works.
If the DRAM scarcity continues, not much.
It's a consumer market scarcity. Since you're supposed to run everything on rented hardware, own nothing and be happy, I'd expect the malloc to be the right place to invest into once it is IPO.