In your command the pattern /home/mahmood/a/b/ is not matched by a/b, unless you give the full path to the directory you want to archive:
tar cvjf compressed.tar.bz2 --exclude=/home/mahmood/a/b /home/mahmood/a
or using the relative path:
tar cvjf compressed.tar.bz2 --exclude=a/b a
Also notes that the trailing slash at the end of the exclude pattern does not work. I cannot explain exactly this behavior, anyway.