LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 07-15-2009, 01:27 PM   #1
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Rep: Reputation: 2
Need advice on building/developing complex projects


Hello All

I have been developing this C++ based engineering application for many years now. The code is contained within 100's of source files. I was smart to have separate .h and .cpp files.

I don't understand make and other similar tools, so I have been using various #define #ifndef and #include "<definition_file>.cpp" directives so that it is built in one g++ command.

The complexity of the program is starting to overwhelm me. When ever I add new files or something, I have to spend many hours de-bugging all my directives to ensure that the right *.h and *.cpp files are included in the correct order.

I would like to strip out all the #include "<definition_file>.cpp" directives, so that each *.cpp file is compiled just with the needed *.h files, then linked later. Yet this would require a multiple file compilation process and supporting tools. So I have been looking into the GNU suite of build tools.

I toyed with some tutorials on the automake/autoconf tools. I am disappointed/discouraged that I am getting compilation errors from the resulting makeFile's. Being completely new to makeFiles I don't know where to start debugging this new language.

Also, the fact that I have to define the Makefile.am for every source directory, will make testing very awkward. I have gotten into the habit of testing every source file in some test.cpp file that only includes the necessary *.h files to test that specific source file. I don't want to be making multiple Makefile.am files for every little test program.

I am hoping that experienced developers might be able to give me some advice on managing the development of complex projects. If you know of some alternative tools for compiling multiple files, let me know, I am looking to try anything. Any advice will be appreciated. Thanks,

pilotmm
 
Old 07-15-2009, 01:46 PM   #2
bannock
LQ Newbie
 
Registered: Jun 2009
Location: Toronto
Distribution: Ubuntu, Fedora
Posts: 24

Rep: Reputation: 18
Take a look at SCons. I've found it quite easy to use.
 
Old 07-15-2009, 02:16 PM   #3
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Slackware 10.1/10.2/12, Ubuntu 12.04, Crunchbang Statler
Posts: 3,786

Rep: Reputation: 282Reputation: 282Reputation: 282
I'm not very good with makefiles either

I've posted a makefile in http://www.linuxquestions.org/questi...project-380437.

I think that you need to change the linkcc to be able to use it with g++ and obviously the name of the executable.

For a bigger project, it's probably advisable to split the project over different subdirectories in which case this one will no longer work.
 
Old 07-15-2009, 02:25 PM   #4
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by pilotmm View Post
Any advice will be appreciated.
I recommend learning to use make. The online manual for GNU make is very useful. I find the section on automatic prerequisites very handy.

You can use "targets" to build your different configurations. For example, you could have a "test" target that goes through your sub-directories and builds only your test programs.

The downside to using make is that it takes quite a while to get to the point where you can write them on your own without referring to examples and documentation. The upside is that make is well documented, and there are many examples available on the internet to show you how to solve almost any problem.
 
Old 07-15-2009, 09:54 PM   #5
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,377

Rep: Reputation: 1108Reputation: 1108Reputation: 1108Reputation: 1108Reputation: 1108Reputation: 1108Reputation: 1108Reputation: 1108Reputation: 1108
In this case, I do think that SCons {http://www.scons.org/} might well be what you're looking for.

Let's face it: the problem here is figuring out what your Makefile needs to consist of. With hundreds of source-files, that becomes very difficult for a human to do, but it is a straightforward problem for the computer. The goal of "smart build" utilities, such as this one, is not simply "to figure out what needs to be in a Makefile," but to largely eliminate the need to build (and to separately maintain) one. As projects grow bigger and bigger, this becomes more and more of a problem.

Your requirement is not new. Look around.
 
Old 07-16-2009, 10:51 AM   #6
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by sundialsvcs View Post
In this case, I do think that SCons {http://www.scons.org/} might well be what you're looking for.
I looked at the manual and tutorial for SCons. It looks very powerful, but it looks just as hard to learn as make. Also, using SCons seems to require some knowledge of how make works. You have to understand targets:

Quote:
Originally Posted by SCons
This sets up a basic environment. Afterwards, you can set up targets.

1 env.Program(target='bar', source=['foo.c'])
You have to understand standard make variables:
Quote:
Originally Posted by SCons
To do that you can specify named parameters such as CCFLAGS for C files or CPPFLAGS for the C++ Preprocessor. More of these can be seen below in this article and also in the Configuration File Reference section of the man page.

1 # directly when constructing your Environment
2 env = Environment(CCFLAGS='-O3')
3
4 # ... or appending it later
5 env.Append(CCFLAGS='-O3')
The vast majority of open source projects rely on make, so I still stand by my suggestion.

Learning to use make will help a person to understand how a typical project gets built (e.g. the Linux kernel). It will also help a user debug build problems in open source projects. If a person is going to spend much time working with open source software, that person will have to acquire some knowledge of how make works.
 
Old 07-16-2009, 12:22 PM   #7
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Hello All

I appreciate all the advice. There seems to be a debate on whether make or SCons is the best route. I am going to experiment with both. I would like to go the GNU route because it would be better accepted by the community. However, as my original post states, I was thrown some errors on a simple tutorial. Hence I am discouraged it will simplify things. I will report back with my experiences as things develop. Maybe it will be useful for others. Once again thanks all for your comments.

Take Care
Pilotmm
 
Old 07-18-2009, 10:40 PM   #8
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Hello All

So I have tried scon. Seems to work Ok, there are some settings you have to set for it to find source files in other directories. I was disappointed that I still had to specify the source files for the dependencies. Thus I still had to search through my source code to determine what *.cpp files should be compiled. This is what I was trying to avoid.

I don't think make would be any easier to use if I have to manually specify dependencies. Given the errors I received before, I am not going to go down that route any further.

I did manage to find a solution to my problem. I created a little utility program that traverses my source code looking for #include <file>.h directives. When a new one is found it will print #include <file>.cpp and then traverse through that file to find more #include directives. I pipe this output to a separate file that I include last in my program. This guarantees that I have everything declared before anything else is defined. I would like to know your opinions, would this utility be useful? It might be added to scon or something else, so source code is automatically found for the dependencies. Let me know and I will post the source.

One complication arose with my template class header files. I had to keep the #include *.cpp directives in the header files of the template class. This is the C++ standard, and the reason for this is obvious, but mentioning it for others facing similar issues.

When multiple file compilation is applied, I received many "duplicate definition" errors with templates classes that are used in more than one object file. The problem was fixed by adding the inline keyword for all my template function specialization. Again notes for other frustrated programmers.

I wonder, are all template functions compiled inline? Also, is there a away to avoid multiple definition errors without forcing specialized templates to be inline? This is not a problem, but some of these functions are large and I would like to keep bloat down.

Using this utility, I managed to simplify the compilation problem without getting any errors about things not being defined. At the same time I did not have to rely on a multiple file compilation process. Could someone please explain why the multiple file compilation process used? What are the benefits? The only thing that I can see is when fixing/debuging you only need to compile the erroneous *.cpp file and link. Speeding up the compilation process. Yet, compiling is so fast on my computer I don't see any really value, given the head aches of keeping track of the dependencies. I am curious why do people use make and scon and all those builders? Just some questions/curiosities.

Thanks for all your help so far. I would still appreciate more comments. I always want to be a better programmer so any advice is always appreciated.

Take Care
pilotmm
 
Old 07-19-2009, 05:29 AM   #9
mmatt
Member
 
Registered: Nov 2005
Location: UK
Distribution: archlinux
Posts: 90

Rep: Reputation: 19
As for why people use make and other such utilities to only compile what's required... I think you can probably appreciate that even small projects may one day grow into larger ones and if the methods of managing the build are bad to start with it only causes headaches later on. Would you be happy to compile every file if your project grew twice the size? Also, if you ever plan to distribute the project as source (which we all hope you will... ) then not everyone has a lightening fast computer should they wish to modify your code. People are also used to tar, make, make install.
 
Old 07-19-2009, 07:12 AM   #10
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian
Posts: 2,455

Rep: Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843Reputation: 843
Quote:
Originally Posted by pilotmm View Post
I did manage to find a solution to my problem. I created a little utility program that traverses my source code looking for #include <file>.h directives. When a new one is found it will print #include <file>.cpp and then traverse through that file to find more #include directives. I pipe this output to a separate file that I include last in my program. This guarantees that I have everything declared before anything else is defined. I would like to know your opinions, would this utility be useful? It might be added to scon or something else, so source code is automatically found for the dependencies. Let me know and I will post the source.
You can also use gcc for this:
http://gcc.gnu.org/onlinedocs/gcc-4....cessor-Options
Quote:
-M
Instead of outputting the result of preprocessing, output a rule suitable for make describing the dependencies of the main source file. The preprocessor outputs one make rule containing the object file name for that source file, a colon, and the names of all the included files, including those coming from -include or -imacros command line options.

Quote:
I wonder, are all template functions compiled inline? Also, is there a away to avoid multiple definition errors without forcing specialized templates to be inline? This is not a problem, but some of these functions are large and I would like to keep bloat down.
See How can I avoid linker errors with my template functions?
 
Old 07-19-2009, 01:14 PM   #11
pilotmm
LQ Newbie
 
Registered: Aug 2007
Distribution: openSuse
Posts: 26

Original Poster
Rep: Reputation: 2
Hi MMat, ntubski

MMat, I understand what you are saying. It is my intention at some point to release my code. Naturally I want to have it in a familiar form for other users/developers. At this point I don't have the time to package it nicely for make and such. I have deadlines just to get working code.

ntubski, thanks a lot for addressing my questions. I overlooked C++ FAQ-Lite when I was trying to solve that problem, instead came across bad advice on other forums. That makes me happy, I can have a seperate *.cpp implementation file that tells the compiler what template functions to compile. Hence templates will be compiled as regular functions, just as intended.

ntubski, once again thanks for passing on what obscure gcc options is required to determine the dependencies. I was always suspicious that gcc would have that, but I am always overwhelmed when I go researching the gcc manual.

I am now encouraged by the prospects of make and scon. There is hope that I can apply conventional build tools for my application. It is something I will do eventually. For now I have some deadlines, and I have a hacked method of re-solving dependencies that seems to work and let me go back to programming. So instead I am going to go back to that and hopefully get the program running in good time.

Since not all reasons are always obvious, let me explain why template definitions need to be included when compiling. A template by itself does not compile into code. The template parameters (Typically referred to as T) need to be defined for this to happen. Typically they are only defined at instantiation (ie when the function is called). That is why terms like <long> are appended to functions names and such (Note that template parameter definition can be inferred by the arguments to the various templates functions themselves). Thus, the template definition is only compiled when the compiler comes across a call to the template in other code. If the definition is not available at that point, then the code will not be able to compile.

Thanks to advice from ntubski, you can get around this problem. The problem is that the template parameter needs to be defined for the template definition to be compiled into code. You can define the template parameters in function/class declarations (ie: `int func<int>();'). When the compiler sees this, it will compile the template definition with the whatever you declared for the template parameters (`int' for the example above). This allows you to separate the template declaration with the definition. You just need a third file that tells the compiler what versions of the template are needed.

Note a third file is not absolutely needed, the declarations could be in the template definition file. However, this can be problematic if multiple programs are using different versions of the same template. Under these circumstances, if the declarations are in the definition file, the other programs will be forced to compile versions of the template that are not needed by that program.

All in all I hope that clears up the template issue.

Forgive me for the verbose posts. I like to explain everything for future frustrated and confused people, who have problems just like me.

Take Care
pilotmm
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Building your own memory manager for C/C++ projects alan_ri Programming 16 07-19-2009 09:24 AM
Are there any Open Source Building Automation Projects going on? aclavi Programming 4 04-08-2008 12:06 AM
Advice for developing graphical menu Irving Programming 2 05-25-2006 09:04 AM
Advice on Router-Building JamesGolick Linux - Wireless Networking 1 05-17-2005 08:22 PM
Hosting for open source projects (advice) unholy Programming 1 12-04-2004 06:36 PM


All times are GMT -5. The time now is 08:54 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration