brk is used to set the end of the data segment to the value you specify. It says "set the end of my data segment to this address". Of course, the address you specify must be reasonable, the operating system must have enough memory, and you can't make it point to somewhere that would otherwise exceed the process maximum data size. Thus,
brk(0) is invalid, since you'd be trying to set the end of the data segment to address
0, which is nonsense.
On the other hand,
sbrk increments the data segment size by the amount you specify, and returns a pointer to the previous break value. Calling
sbrk with 0 is valid; it is a way to get a pointer to the current data segment break address.
malloc is not a system call, it's a C library function that manages memory using
sbrk. According to the manpage,
malloc(0) is valid, but not of much use:
If size is 0, then malloc() returns either NULL, or a unique pointer value that can later be successfully passed to free().
malloc(0) are not equivalent: the first of them is invalid, the second is used to obtain the address of the program's break, and the latter is useless.
Keep in mind that you should never use both
sbrk throughout your program.
mallocassumes it's got full control of
sbrk, if you interchange calls to
brk, very weird things can happen.
On success, brk() returns zero, and sbrk() returns a pointer to the start of the new area. On error, -1 is returned, and errno is set to ENOMEM.