I'm sorry to disappoint you, but nested awk does not exist! The awk statement inside the awk program is interpreted as a variable name and since it has not been assigned before, it is a null variable which is interpreted as false. In other words it does absolutely nothing! Regarding the single quotes following awk in your first working example, they simply close the shell quotes and reopen it again, so that in the awk program they result as nothing. When you put the brackets between these quotes, it results obviously in a syntax error.
Maybe if you explain what are you trying to do, we may help. Moreover if you provide an example of the input file and the desired output, it will result more clear what you're trying to achieve.