With no details about your hardware config, it is a difficult question to be answered.
Basically agpgart module/driver manages the agp bus of your video card.
Agpgart can be included into the kernel or compiled as module. (nothing strange 'til now).
Many video cards and related drivers (ie nvidia) have their own agp internal module but if available their primary choice is to load the agp kernel module, leaving the built in module as a secondary choice.
They also prefer the agp kernel module to be loaded by the X server, instead having the module loaded at boot time.
If the agp kernel module doesn't work properly, you are allowed to specify what agp module you want the X server to load by editing your XF86Config file.
ATI cards/drivers work slighly different, and you should compile into the kernel the dedicated ati agp and dri modules as well for the ATI cards supporting 3D acceleration , but to be honest I do not have much experience with ATI cards, so I cannot give details on that.