LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Blogs > rainbowsally
User Name
Password

Notices

Rate this Entry

A runc script to create a debian repository

Posted 09-07-2013 at 06:25 PM by rainbowsally
Updated 09-07-2013 at 06:29 PM by rainbowsally (added link)

Today's Features
  • A debian package repository creator.
  • Testing runc as a shell function (direct execution of C/C++ code).

[See also "Run C/C++ code as a script" http://www.linuxquestions.org/questi...script-35691/]

I took kubuntu for a spin (installed on an old computer) to see if I could convert it to mint after kubuntu was installed. The kubuntu installer was a little better.. by a mainingful notch, at least for the old HP computer I used. Some of the other stuff does NOT work as well as mint though, such as updating the system configuration. Nevermind.

But I discovered that the custom repository setup I had used for mint was not adequate for kubuntu.

So... What do we need to do to create an 'official' looking repository simply enough that it's not hard to set up but it will look right in muon, synaptic, etc.?

Here's the basic setup taken from firefox which may include some undocumented features, This is not likely to be perfect for source builds (because the installed size will be off, for one thing), but it's probably pretty usable for user generated packages that install and uninstall using standard menu functions (and also with apt-get).

We need our repository (named DEB for this example) somewhere on our hard drives with a directory tree like this

DEB/dists/pool/<various subfolders and deb files>

The Packages.gz files contain text files named Packages. And their format appears to be based on but not exactly the same as the control file in the deb packages themselves.

So... here's the plan.
Code:
Package: <copy>
Priority: <copy>
Section: <copy if present, else 'Optional'>
Installed-Size: <compute blocks>
Maintainer: <copy>
Architecture: <copy and save for Package subdir to write into>
Version: <copy>
Depends: <copy if present else ??>
Filename: <DEB subdir and filename>
Size: <package size in bytes (for size checks)>
MD5sum: <compute>
SHA1: <compute>
SHA256: <compute>
Description: <copy to end>
The above tags are key-value pairs. Wanna try to do this in bash? Be my guest, so for today's experiment we'll use the previously uploaded runc library to write the function to do this in C/C++ runnable as a shell program, easy to edit and modify as issues arise, if any.

We'll unpack the deb file and copy what we can from the control file and fill in the rest.

Here's the expected procedure.
Code:
0. Init key value pairs that we'll need.

1. get the check sums for the package itself
  compute and save key-value pairs for:
    MD5SUM
    SHA1
    SHA256

2. save the deb package sub path as Filename and <sub path>

3. unpack the deb package

4: copy some control file key value pairs, saving 'Description" to the end.

5. delete the control files.

6. compute the installed size on the remaining files and save as Installed-Size and <size in blocks>

7. Append the key/value pairs to either the binary-<arch> or sourc Package files.

8. gzip the Package files.
Now let's write the program, in C/C++ THEN let's save it as a runnable script calling /usr/local/runc to execute it.

[You can also remove the top line and compile this is C++ to compare execution timss, but you'll have to remember to -I/usr/local/include and -lrunc.]

file: mkrepo (for debian packages to appear in synaptic, apt-get, etc.)
purpose: easily edited utility (executable)
Code:
#!/usr/local/bin/runc

#include <stdio.h>
#include <malloc.h>
#include <string.h>
#include <stdlib.h>   // system()
#include <libgen.h>   // dirname()
// #include <assert.h>
#include <unistd.h>   // sleep()

#include <runc.h>     // tmpstr(), slist funcs, peculiar to runc

void dbg(){}          // for a non-moving breakpoint

#define streq(a, b) (0 == strcmp(a, b))
#define error(s) (fprintf(stderr, "%s", s), exit(1))
#define BLKSIZE 1024 /* used for installed size estimates */
// The following macro can be used in system calls to suppress unwanted commandline feedback
// do NOT put a comma between the command and this macro.  It appends to the string being executed.
#define SILENT " >/dev/null 2>&1"

// a place to unpack the packages to get sizes and control info
static const char* tempdir = "tMp";
// the output folder for generated Package files
static const char* mkrepo_output = "dists-temp/generic/custom";
// an error clue if found in in any output data
static const char* UNKNOWN = "[Unknown]";
// used for system calls (using linux tools as a lib).
static int err;

int usage(int errcode);
int init();
int finish();
void runner(); // calls run() on all deb packages found
void run(const char* filepath);

void init_kv_pairs(char*** pkeys, char*** pvalues);
void check_required_apps();
// also sets filename field
void get_check_sums(char*** pkeys, char*** pvalues, const char* filepath);
int create_tempdir();
void remove_tempdir();
void get_size(char*** pkeys, char*** pvalues, const char* filepath);
void read_control(char*** pkeys, char*** pvalues, char*** pdescription, const char* tempdir);
void get_installed_size(char*** pkeys, char*** pvalues);  
void write_package_file(char*** pkeys, char*** pvalues, char*** pdescription);

void slist_printout(char*** pkeys, char*** pvalues, char*** pdescription);

int main(int argc, char** argv)
{
  dbg();
  
  err = system("rm -rf tMp" SILENT); // for test
  
  // check args
  if(! argv[1])
       return usage(1);
  if(streq(argv[1], "--help") || streq(argv[1], "help")) 
    return usage(0);
  
  if(streq(argv[1], "init"))
     return init();
  
  if(streq(argv[1], "finish"))
     return finish();
  
  if(! streq(argv[1], "run"))
    return(usage(1));
  
  runner();
  
  return 0;
}

void runner()
{
  // for i in `find *.deb`; do run done
  // allows symlinks to real repository

  char** files = slist_new();
  slist_pipeRead(&files, "find */* -type f -name *.deb");
  int cnt = slist_count(&files);
  printf("[%d packages found]\n", cnt);
  for(int i = 0; i < cnt; i++)
  {
    printf("%3d%%", (i * 100)/cnt); fflush(stdout);
    run(slist_get(&files, i));
    printf("\b\b\b\b");
  }
  
  if(cnt)
  {
    printf("%3d%% done\n", 100);
    printf("You may edit the generated 'Package' files as text files now\n"
        "as may be required before 'finish'.\n");
  }
  else
    printf("No deb packages found???  [Maybe files should be in a subdir or at a symlink.]\n");
  
  slist_delete(&files);
}


void run(const char* filepath)
{
  // check filepath
  //  const char* filepath = argv[1];
  if((err = system(tmpstr(256, "test -f %s", filepath))))
  {
    fprintf(stderr, "Can't find file:\n  %s\n", filepath);
    exit(1);
  }
  
  check_required_apps();
  
  // init key pairs
  char** keys = slist_new();
  char** values = slist_new();
  char** description = slist_new();
  
  init_kv_pairs(&keys, &values);
  
  // start loading fields
  get_check_sums(&keys, &values, filepath);
  
  // unpack the deb package
  err = create_tempdir();
  if(err)
    error("Need write permissions in current directory.");
  // ar x ../filename
  err = system(tmpstr(256, "cd %s && ar x ../%s", tempdir, filepath));
  if(err)
  {
    error("??? Need a relative path for the input file name, maybe?\n");
    remove_tempdir();
  }
  
  read_control(&keys, &values, &description, tempdir);
  
  get_size(&keys, &values, filepath);
  get_installed_size(&keys, &values);
  
  write_package_file(&keys, &values, &description);
  
  // just for form in this context because we're leaving anyway..
  slist_delete(&keys);
  slist_delete(&values);
  slist_delete(&description);  
  return;
}


typedef struct
{
  const char* key;
  const char* value;
}KV_PAIR;

KV_PAIR kv_pairs[] =
{
  {"Package",  UNKNOWN},
  {"Priority",  "Optional"},
  {"Section",  "Custom"},
  {"Installed-Size",  UNKNOWN},
  {"Maintainer",  UNKNOWN},
  {"Architecture",  "All"},
  {"Version",  UNKNOWN},
  {"Depends",  "bash"},
  {"Filename",  UNKNOWN},
  {"Size",  UNKNOWN},
  {"MD5sum",  UNKNOWN},
  {"SHA1",  UNKNOWN},
  {"SHA256",  UNKNOWN},
//  {"Description",  UNKNOWN},
  {0, 0}
};

void init_kv_pairs(char*** pkeys, char*** pvalues)
{
  KV_PAIR *p = kv_pairs;
  while(p->key)
  {
    slist_append(pkeys, p->key);
    slist_append(pvalues, p->value);
    p++;
  }
}

static int prog_error(const char* progname)
{
  return system(tmpstr(256,"which %s" SILENT, progname));
}

void check_required_apps()
{
  if(
        prog_error("md5sum")
     || prog_error("sha1sum")
     || prog_error("sha256sum")
    )
    exit(1);
}

int usage(int errcode)
{
  printf(
"\n"
"  Run this inside the repository directory containing a pool of debian\n"
"  packages and use 'mkrepo' + 'init', 'run' and 'finish' to make visible in a list\n"
"  for synaptic/muon, or some other package manager.\n"
"\n"
"  You'll need to add the repo to your etc/apt/sources.list file manually,\n"
"  most likely, but then the packages should show.\n\n"
  );
  return errcode;
}

int slist_find(char*** plist, const char* s, int exact);
void slist_replace(char*** plist, int indx, const char* s);

void get_sum(char*** pkeys, char*** pvalues, const char* keyname, const char* prog, const char* filepath)
{
  char** tmp = slist_new();
  int idx = slist_find(pkeys, keyname, true);
  // pipeRead() always appends
  slist_pipeRead(&tmp, tmpstr(256, "%s %s", prog, filepath));
  // cut last string at first space
  char* s = tmpstr(256, "%s", slist_get(&tmp, 0));
  char* p = strchr(s, ' ');
  if(p)
    *p = 0;
  slist_replace(pvalues, idx, s);
  slist_delete(&tmp);
}



int slist_find(char*** plist, const char* s, int exact)
{
  int cnt = slist_count(plist);
  int match;
  char* text;
  for(int i = 0; i < cnt; i++)
  {
    text = slist_get(plist, i);
    match = 0;
    if(exact)
      match = 0 == strcmp(text, s);
    else
      match = 0 != strstr(text, s);
    if(match)
      return i;
  }
  return -1; // no match
}

void slist_replace(char*** plist, int index, const char* s)
{
  slist_remove(plist, index);
  slist_insert(plist, index, s);
}

int create_tempdir()
{
  return system(tmpstr(256, "mkdir %s", tempdir));
}


int find_key(char*** pcontrol, const char* key)
{
  int slen = strlen(key);
  int n = slist_count(pcontrol);
  char* text;
  char* p;
  for(int i = 0; i < n; i++)
  {
    text = slist_get(pcontrol, i);
    if(memcmp(text, key, slen) == 0)
    {
      p = text + slen;
      if((*p == ':') || (*p < 32))
        return i;
    }
  }
  return -1; // not found
}


void remove_tempdir()
{
    err = system(tmpstr(256, "rm -rf %s/*" SILENT, tempdir));
    err = system(tmpstr(256, "rmdir %s" SILENT, tempdir));
}


void remove_control()
{
  // remove everything except data.*
  char** list = slist_new();
  slist_pipeRead(&list, tmpstr(256, "ls %s", tempdir));
  int save = slist_find(&list, "data.", false);
  slist_remove(&list, save);
  for(int i = 0; i < slist_count(&list); i++)
    system(tmpstr(256, "rm %s/%s", tempdir, slist_get(&list, i)));
  slist_delete(&list);
}


void read_control(char*** pkeys, char*** pvalues, char*** pdescription, const char* tempdir)
{
  err = system(tmpstr(256, "cd %s && tar xaf control.*", tempdir));
  
  char** control = slist_new();
  slist_fileRead(&control, tmpstr(256, "%s/control", tempdir));
  
  // move strings at the Description line from list to description strings
  int idx = find_key(&control, "Description");
  int cnt;
  while(1)
  {
    cnt = slist_count(&control);
    if(idx < cnt)
    {
      slist_append(pdescription, slist_get(&control, idx));
      slist_remove(&control, idx);
    }
    else
      break; }
  
  ///////////////////////////////////////////////////////////
  // for all other tags in the control file try to find the tag in the 
  // keys list and if found, replace the temp value with the real one.
  
  char* key;
  char* value;
  char* p;
  for(int i = 0; i < slist_count(&control); i++)
  {
    // key is the first word followed by ';' or spaces and ':'
    key = tmpstr(256, "%s", slist_get(&control, i));
    value = strchr(key, ':');
    *value = 0;
    // retry finding space
    p = strchr(key, ' ');
    if(p)
      *p = 0;    
    // skip whitespaces and punct
    while(*value <= 32)
      value++;
    
    idx = slist_find(pkeys, key, true);
    if(idx >= 0)
      slist_replace(pvalues, idx, value);
  }
  slist_delete(&control);
  remove_control();
}

void get_size(char*** pkeys, char*** pvalues, const char* filepath)
{
  // exact size used for error checking
  size_t size;
  FILE* fp = fopen(filepath, "r");
  // we know it exists
  fseek(fp, 0, SEEK_END);
  size = ftell(fp);
  fclose(fp);
  int idx = slist_find(pkeys, "Size", true);
  slist_replace(pvalues, idx, tmpstr(256, "%u", size));
}


void get_installed_size(char*** pkeys, char*** pvalues)
{
  long long size;
  char** tmp = slist_new();
  err = system(tmpstr(256, "cd %s && tar xaf data.*", tempdir));
  err = system(tmpstr(256, "rm %s/data.*", tempdir));
  slist_pipeRead(&tmp, tmpstr(256, "du -cbs %s/*", tempdir));
  sscanf(slist_get(&tmp, 0), "%lld", &size);
  int idx = find_key(pkeys, "Installed-Size");
  slist_replace(pvalues, idx, tmpstr(256, "%lld", (size+BLKSIZE/2)/BLKSIZE));
  remove_tempdir();
  slist_delete(&tmp);
}

// also sets filename field
void get_check_sums(char*** pkeys, char*** pvalues, const char* filepath)
{
  int idx = find_key(pkeys, "Filename");
  slist_replace(pvalues, idx, filepath);
  get_sum(pkeys, pvalues, "MD5sum", "md5sum", filepath);
  get_sum(pkeys, pvalues, "SHA1", "sha1sum", filepath);
  get_sum(pkeys, pvalues, "SHA256", "sha256sum", filepath);
}

void copy_value(char* buf, char*** pkeys, char*** pvalues, const char* key)
{
  *buf = 0;
  int idx = find_key(pkeys, key);
  if(idx >= 0)
    strcpy(buf, slist_get(pvalues, idx));
}

void write_package_file(char*** pkeys, char*** pvalues, char*** pdescription)
{
  char pkgtype[64];
  copy_value(pkgtype, pkeys, pvalues, "Architecture");
  // mkdir(pkgtype, 07550); // owner: read|write|modify, group: exec|read, others: exec|read
  err = system(tmpstr(256, "mkdir -p %s/%s" SILENT, mkrepo_output, pkgtype));
  FILE* fp = fopen(tmpstr(256, "%s/%s/Packages", mkrepo_output, pkgtype), "a"); // append
  KV_PAIR* p = kv_pairs;
  int idx;
  while(p->key)
  {
    idx = find_key(pkeys, p->key);
    if(idx >= 0)
      fprintf(fp, "%s: %s\n", slist_get(pkeys, idx), slist_get(pvalues, idx));
    p++;
  }
 
  // Description is a literal copy of the originl including key name "Description:"
  for(int i = 0; i < slist_count(pdescription); i++)
    fprintf(fp, "%s\n", slist_get(pdescription, i));
    
  fprintf(fp, "\n");
  fclose(fp);
}

int init()
{
  err = system("rm -r dists-temp/*" SILENT);
  err = system("rmdir dists-temp" SILENT);
  return 0;
}

int finish()
{
  char** packagesfile = slist_new();
  slist_pipeRead(&packagesfile, "find dists-temp/* -name Packages");
  int cnt = slist_count(&packagesfile);
  
  for(int i = 0; i < cnt; i++)
    system(tmpstr(256, "gzip %s" SILENT, slist_get(&packagesfile, i)));
  
  if(cnt)
    printf(
    "'mkrepo finish' is not yet fully implemented\n"
    "\n"
    "You may manually change the directory dists-temp to dists and change the\n"
    "subfolder names to match your system and change the <arch> folder if it\n"
    "exists to 'binary-<arch>' [where <arch> is the architecture] and update\n"
    "the sources.list file in /etc/apt to point to this repository."
        );
    else
      printf("Nothing to do in folder dists-temp\n");
  return 0;
}
For my tests I made a symlink to the 'pool' of deb files. They don't need to be in alphabetical folders.

And all I changed was dists-temp to dists, and then moved the created folder to the same folder containing the 'pool'.

Seems to work so far.

The Computer Mad Science Team

:-)
Posted in Uncategorized
Views 344 Comments 0
« Prev     Main     Next »
Total Comments 0

Comments

 

  



All times are GMT -5. The time now is 07:21 AM.

Main Menu
Advertisement

My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration