LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Blogs > rainbowsally
User Name
Password

Notices

Rate this Entry

Simple C: Beyond Bash Part 2 - Dependency lister for dir branch (recursive)

Posted 01-27-2012 at 07:09 AM by rainbowsally
Updated 08-06-2014 at 11:33 AM by rainbowsally

[Note to myself: Needs rewrite using libLQ.]

Simple C: Beyond Bash Part 2 - Dependency lister for dir branch (recursive)

CHANGELOG

* Sat Feb 18, 2012 -rs

find-deps version 1.2

In 'find_elf()' changed pipecmd to "find * -executable -o name *.so*" because some libs
may not have have their executable flag set, such as an unpacked from an rpm package that
doesn't have the perms set until install time. This change will set off error messages about the files that need corrected permissions.

Also quotes around the "*.so*" for the 'find' call. (It couldn't have worked without it, but somehow this was mis-posted). (corrected feb 21) -rs

Features:
  • Read a pipe (uses slist functions from Part 1).
  • Commandline switch reordering (non-destructive).
  • Non-regex parsing & translating in do{...}while(0) AND block (see 'elfpaths_only') ,
  • Unsorted 'uniq' example in C
  • -- A tool to read dependencies from all elf executables and libraries in a directory branch.
  • -- Can be used to check make-based[tm] :-) installations or rpm and deb packages before packing up.
Build Requires:
  • bash
  • make
  • g++
Run Requires:
  • ldd
  • find
This is interesting, because I discovered that we can't run ldd in a pipe to read /usr/lib or /usr/bin (openSUSE 11.4). Possibly due to security settings. Try it with the --info flag. It CAN, however read files in directories below /usr/lib (such as /usr/lib/dri on my system).

Seems to work great when used "normally" (to read binary trees in package folders), and it's pretty fast. Here's part of a dump of my /bin deps.

Code:
[list of 45 unique deps deleted -rs]
---------------------------------------------------
binaries found : 125
ldd pipe errors: 0
external deps  : 45
---------------------------------------------------
real    0m1.667s -- less than two seconds (cmd: 'time find-deps /bin -i')
user    0m0.727s
sys     0m0.778s
The "external deps" listed are libs loaded at run time that are not resolved in or below the prefix branch (i.e., /bin in this case).

I have installed the code and executable in my sandbox so it's linked into my path and is easy to edit. (see early blog entries re. new.symlink, etc.). And I gave the folder a version number so I can easily see which one is current if this needs any changes later.
Code:
HOME
 \-- bin
      \-- src
//           \-- find-deps-1.0 [everything goes 
here]
            \-- find-deps-1.2 ...
The slib.cpp and slib.h files are in Part 1. Put those in the same folder with the new ones. Don't worry, the compiler will tell you which are missing if you forget. :-)

There are three new files (five total). Here's what's new.
  • version.h - version and changelog in one
  • usage_str.dat - easier to edit a text file, this is the output, the non-C type file extension avoids potential linker errors with tools such as AutoBake (makefile creator).
  • find-deps.cpp - the program

file: version.h
Code:
// initial app created
//#define VERSION "1.0"

// #define VERSION "1.0"

// #define VERSION "1.1"
// added --unres switch and printouts for unresolved libs
// #define VERSION "1.2"

// #define VERSION "1.2"
// added *.so* files to elf file search command
#define VERSION "1.2"
file: usage_str.dat
Code:
/* usage.txt converted with txt2cstr */
const char* usage_str =
    "\n"
    "Usage: find-deps <branch>\n"
    "\n"
    "  Primary usage is for discovering files needed by a binary package before\n"
    "  installation/building, etc.\n"
    "\n"
    "  Prints out a list of all libs used by executable files and libs \n"
    "  found in the directory <branch> (recursive). \n"
    "\n"
    "  Does not list files that are resolved by other files in the same \n"
    "  branch.\n"
    "\n"
    "  Does not look in any other branches for resolvers.\n"
    "\n"
    "Switches:\n"
    "  -i | --info         display info about files found\n"
    "  -v | --version      show version\n"
    "\n"
    "Notes: \n"
    "\n"
    "  1. Needs linux tools 'find' and 'ldd' to run.\n"
    "\n"
    "  2. If a symlink points to a non-existent file (as encountered in \n"
    "  development packages) this will be still be counted as a self-resolved \n"
    "  dependency.\n"
    "\n"
    "  3. The error message \"No elf binaries found\" sent to stderr could mean\n"
    "  that there really aren't any or just that the path doesn't exist.\n"
    "\n"
    "  4. May not work for system dirs (e.g. /usr/bin, /usr/lib, /lib) due to \n"
    "  ldd quirks but we can get 'info' on the number of elf files.\n"
    "\n"
    ;

file: find-deps.cpp
Code:
// find-deps.cpp

// shotgun load some commonly used headers
#include <stdio.h>    // printf, sprintf, etc.
#include <stdlib.h>   // exit()
#include <unistd.h>   // tons of stuff
#include <malloc.h>   // memory allocation
#include <string.h>   // strings and block comparisons and moves

// our own includes
#include "slist.h"
#include "version.h"

void dbg(){}          // a breakpoint for kdbg/gdb

//////////////////////////////////////////////////////
// prototypes section

// saved current directory path
extern char* HERE;

// true if we want to know how many files were processed
extern int infoflg;

// number of pipe to ldd errors
extern int ldderrs;

// name of currently running app
extern char* appname;

// reorder args so switches precede args and return
// # of switches.
int reorder_args(int argc, char** argv);

// find and store names of elf executables in elflist,
// returns 0 on success
int find_elf(char*** pelflist, const char* prefix);

// find and store dependency libs in depslist, not including files
// in elflist, returns 0 on success
int find_deps(char*** depslist, const char* prefix, char** elflist);

// process recognized switches and return errcode if bad switch
// value.
int handle_switches(int n, char** argv);

// usage note
int usage(int errcode);

//////////////////////////////////////////////////////
// the program

// saved current directory path
char* HERE;

// true if we want to know how many files were processed
int infoflg = 0;

// number of pipe to ldd errors
int ldderrs = 0;

// name of currently running app
char* appname;

int main(int argc, char** argv)
{
  dbg();
  
  appname = basename(argv[0]);
  
  int err = 0;
  char** elflist = slist_new();
  char** depslist = slist_new();  
  int offset = 0; // increases for each commandline switch
  
  // syntax checks, should return usage(1) if argc < 2, but... whatever :-)
  if((argc < 2) || (strcmp(argv[1], "--help") == 0))
    return usage(0);
      
  // switches
  offset = reorder_args(argc, argv);
  err = handle_switches(offset, argv);
  if(err)
    return err;
  
  // setup
  const char* prefix = argv[1 + offset];
  HERE = getenv("PWD");
  
  // get list of all elf executables in prefix dir
  err = find_elf(&elflist, prefix);
  
  int num_elf = slist_count(elflist);

  if(num_elf == 0)
  {
    fprintf(stderr, "No elf binaries found.\n");
    return 1;
  }
  
  // get list of deps in all elf files that are not in the
  // list already in the prefix dir
  err = find_deps(&depslist, prefix, elflist);
  
  int num_deps = slist_count(depslist);

  for(int i = 0; i < slist_count(depslist); i++)
    printf("%s\n", depslist[i]);
  
  if(infoflg)
  {
    const char* hline =
       "---------------------------------------------------";
    printf("%s\n", hline);
    printf("binaries found : %d\n"
           "ldd pipe errors: %d\n"
           "external deps  : %d\n",
           num_elf, ldderrs, num_deps);
    printf("%s\n", hline);
  }
  
  // cleanup and return
  slist_delete(depslist);
  slist_delete(elflist);
  return 0;
}

//////////////////////////////////////////////////////
// the library of functions and various utils

// utils - misc

// returns true (1) if filename is an elf file
int is_elf(const char* fname)
{
  FILE* fp = fopen(fname, "r");
  if(! fp) return 0; // error, file not found
  
  // read the first few bytes of the file and look for the
  // signature of a valid elf binary.
  char buf[8] = {0};
  fread(buf, 1, 4, fp);
  if(memcmp(buf, "\177ELF", 4) == 0)
    return 1; // true
  return 0;   // false
}

int reorder_args(int argc, char** argv)
{
  // make literal copy of argv list in temporary list
  char** tmp_args = (char**)malloc(argc * sizeof(char**));
  memcpy(tmp_args, argv, argc * sizeof(char**));
  
  int cnt = 1; // current copy back number, app path is at [0] (no change)
  int nswitches = 0; // the value to return
  
  // copy switches (anything starting with '-') back first
  for(int i = 1; i < argc; i++)
    if(tmp_args[i][0] == '-')
      argv[cnt++] = tmp_args[i];
  
  nswitches = cnt - 1; // number of actual switches
  
  // copy non-switches back next
  for(int i = 1; i < argc; i++)
    if(tmp_args[i][0] != '-')
      argv[cnt++] = tmp_args[i];
  
  free(tmp_args);
  return nswitches;
}

/////////////////////////////////////////////////////////

// find and store names of elf executables in elflist,
// returns 0 on success
int find_elf(char*** pelflist, const char* prefix)
{  
  // init the flag and the list which is an array of 
  // pointers terminated by a zero and save current
  // directory so we can always return to it.
  int err = 0;
  char pipcmd[512];
  char** elflist = *pelflist; /// remove after test
  char** tmp = slist_new();

  // Create a non-looping do-while block we can leave 
  // at the first error.  Functionally equivalent to a 
  // string of ORs using err as the flag.
  do
  {  
    char pipecmd[512];
    err = chdir(prefix);
    if(err) break;
  
    // get complete list of all executables, then
    // remove those not elf type.

    // sprintf(pipecmd, "find * -executable");
    // added all *.so files, some libs in rpm packages may not be executable -rs
    sprintf(pipecmd, "find * -executable -o -name \"*.so*\"");

    err = slist_readPipe(&elflist, pipecmd);
    if(err)
    {
      slist_clear(elflist);
      break;
    }
    
    // remove non-elf files
    for(int i = 0; i < slist_count(elflist); i++)
    {
      if(is_elf(elflist[i]))
        slist_append(&tmp, elflist[i]);
    }
  }while(0);
  if(err) return err;
  
  // elflist = tmp;
  slist_delete(elflist);
  //elflist = tmp->list();
  elflist = tmp;
  *pelflist = elflist;
  // delete tmp;
  chdir(HERE);
  return 0;
}

void elfpaths_only(char*** plist)
{
  int state = 0;
  char** list = *plist;
  char** tmp = slist_new();
  
  for(int i = 0; i < slist_count(list); i++)
  {
    // THANK YOU :-) to the svanah non-gnu BNF project for this trick
    char buf[256];
    sprintf(buf, list[i]);
    char* ip = buf, *op = buf;
    do
    {
      // skip whitespaces
      while(*ip <= ' ')
        ip++;
      
      // don't want blank lines
      if(!*ip)
        break;
      
      // want lines starting with slash
      if(*ip == '/')
        break;
      
      // don't want lines not pointing to lib path
      ip = strstr(ip, " => ");
      if(!ip)
        break;
      
      ip += 4; // move to file path, should start with '/'
      
      // don't want lines not having a lib path
      if(*ip != '/')
        break;
      
      // MATCHED - now copy to first non text and terminate string
      while(*ip > ' ')
        *op++ = *ip++;
      *op = 0;
      
      state = 1; // signale that we got one
    }while(0);
      
    if(state)
      slist_append(&tmp, buf);
  }
  
  // save new result and return
  slist_delete(list);
  *plist = tmp;
}

// find and store dependency libs in depslist, not including files
// in elflist, returns 0 on success
int find_deps(char*** pdepslist, const char* prefix, char** elflist)
{
  // init
  char** depslist = *pdepslist;
  int err = 0;
  char pipecmd[512];
  
  err = chdir(prefix);
  if(err) return 1;
  
  int len = slist_count(elflist);
  for(int i = 0; i < len; i++)
  {
    // get list of ldd libs loaded by each elf file and add
    // to the depslist, temporarily before removing dups.
    
    char** sublist = slist_new();
    
    sprintf(pipecmd, "/usr/bin/ldd %s", elflist[i]);
    err = slist_readPipe(&sublist, pipecmd);
    
    // count ldd errors
    if(err)
      ldderrs++;
    
    elfpaths_only(&sublist);
    
    // "you will be assimilated" said depslist to sublist....
    slist_addList(&depslist, sublist);
    slist_delete(sublist);
  }while(0);
  
  int ip;
  int op;
  
  // remove duplicates if any items to copy
  ip = 0;  
  for(ip = 0; ip < slist_count(depslist); ip++)
  {
    op = ip + 1;
    while(op < slist_count(depslist))
    {
      // shrink list or advance pointer...
      if(strcmp(depslist[op], depslist[ip]) == 0)
        slist_remove(&depslist, op);
      else
        op++;
    }
  }
  
  // remove self-resolved and hope the paths are right 
  // after installation because we won't (can't) check them at
  // this point without setting LD_LIBRARY_PATH and that can 
  // be as error prone as this, so... here we go.  If it 
  // exists at all in our branch, we count is as 'resolved'.
  for(int elf = 0; elf < slist_count(elflist); elf++)
  {
    char* elffile = basename(elflist[elf]);
    for(int dep = 0; dep < slist_count(depslist); dep++)
    {
      char* depfile = basename(depslist[dep]);
      if(strcmp(elffile, depfile) == 0)
        slist_remove(&depslist, dep);
    }
  }

  *pdepslist = depslist;
  chdir(HERE);
  return 0;
}

int usage(int errcode)
{
#include "usage_str.dat"
  printf(usage_str); 
  return errcode;
}

// process recognized switches and return errcode if bad switch
// value.
int handle_switches(int n, char** argv)
{
  n++; // don't count argv[0] (app path)
  
  // we already know the first char is a '-' so we look for
  // substrings common to both long and short switches.
  
  for(int i = 1; i < n; i++)
  {
    // -v or --version
    if(strstr(argv[i], "-v"))
    {
      printf("%s version %s\n", appname, VERSION);
      exit(0);
    }
    
    // -i or --info
    else if(strstr(argv[i], "-i"))
      infoflg = 1;
    
    else
    {
      fprintf(stderr, "Unrecognized switch %s\n", argv[i]);
      return 1; // not a valid switch
    }
  }
  return 0; // ok
}

// if you must, here's a way to do this without linking explicitly. :-)
// #include "slist.cpp" // uncomment this line.
// COMPILE: g++ find-deps.cpp -o find-deps
Compile as you did with the toy at the bottom of Part 1, or if you like we can cheat and simply
Code:
#include "slist.cpp"
in the deps-list.cpp file itself. This trick works sometimes. And it will this time.

But after this, having a bit of a taste of what's possible with C/C++, you might want to tool up and do it for real with
  1. a good debugger. I HIGHLY recommend kdbg version 5.0 or greater. (insight genre.)
  2. a decent programming editor. With macro keys if you can get em. kdevelop3 is my pick. (Haven't tried Jen's programming editor with Wine yet -- that's also a good one.)
  3. and a way to create your own makefiles if you don't like the default. I like AutoBake.
That's so you don't have to live with anyone else's bad idea.

And here's my own bad idea if you want to check it out. It's easy to edit the templates, but I should have done that myself. "autobake new c-32" is ok out-of-box tho.

http://rainbowsally.org/rainbowsally...1-12-29.tar.gz

Also needs a type lister like 'new.image' has and some other improvements/reorganizations. It's easier to delete stuff than write stuff though, so this is on the back burner as far as my own needs are concerned.

But that's one bad idea. And it may give you some bad ideas of your own!

:-)

Now we have find-deps.

We'll put it to good use a bit later, if all continues to go well here in blog-land.

Some more interesting stuff you can do once you've got a way to compile this and run it.
  • Compile the app and run 'strip find-deps' and check the file size after it's stripped. Compare to the -s switch (sent to the linker).
  • Add the -g3 switch to the compile and run in a good debugger. Find where the ldd pipe errors come from when checking /usr/lib if your system does the same thing as mine. (Same whether you chdir to the folder or not. You'll see what I mean.)
  • Create a TREE folder containing the directory image of an installation you might want to generate. Probably in TREE/usr/local, for a somewhat safe place if you decide to actually install it later. And 'find-deps TREE'. :-)
Posted in Uncategorized
Views 578 Comments 0
« Prev     Main     Next »
Total Comments 0

Comments

 

  



All times are GMT -5. The time now is 12:45 PM.

Main Menu
Advertisement

My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration