Using "gdb and other tools to find out which system calls" is not reverse engineering. These are used for debugging a program.
Reverse engineering (properly) is a three to four step operation and requires at least three people.
1. identify all input, outputs of a program, and document them (person 1)
2. identify the functions carried out by the program and document them (person 1)
3. re-implement the functions based on the documentation from steps 1 and 2. (person 2)
4. have a lawyer review the process as the project continues. (person 3)
The problem with "reengineering" using the debugging tools is that this only gives you a derivative of the original programming, along with the copyright restrictions of the original program.
Even doing it properly is still subject to software patents (even if they are stupid).
The problem with one person doing it is that the implementation person MAY include copyrighted code from steps 1 and 2 when using some of the debugging tools/dump analysis tools, thus contaminating the result with copyright restrictions. This was the charge against the Samba project as it remimplemented AD and CIFS, even though the only thing being examined was the network packets being passed around.
Now for finding out what most Linux do (the open source ones), just get the source and look. Besides seeing the code, you also get any comments that might explain why something was done the way it was done. Who knows - you may even come up with a better way. If that happens, submit it as an improvement...
As for books, you can start with:
https://www.google.com/search?q=text...utf-8&oe=utf-8
or the PDF format
https://beginners.re/
NOTE: this one is with a Russian point of view, and does not cover any legal ramifications. Some of the activities would be counted as illegal (as in breaking any license you have on proprietary software - specifically about dissassembling the code via either debugger or other tools).
but google for others - there are quite a few