LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 01-28-2011, 01:12 PM   #16
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301Reputation: 1301

I saw the results, and I don't see huge improvement.

As for /proc/sys/net/ip_conntrack, I don't have the file, so I didn't know.

But then, how does wc -l work if the file has no size ?
 
Old 01-29-2011, 12:45 AM   #17
frater
Member
 
Registered: Jul 2008
Posts: 121

Original Poster
Rep: Reputation: 23
Quote:
Originally Posted by H_TeXMeX_H View Post
I saw the results, and I don't see huge improvement.
I'm looking for any improvement, but you are right.. not huge.
As I wrote in my original post. On a normal system this file contains about 600 lines.
There's no need for improvement then (0.01 second)
Code:
# time wc -l /proc/net/ip_conntrack
1498 /proc/net/ip_conntrack

real    0m0.009s
user    0m0.000s
sys     0m0.008s
But on that popular server we're talking about +/- 38000 lines which takes 10 seconds.
If I could bring this down to 6 seconds I'll be more happy than I am already.

This code is being used by a monitoring server which executes this every minute.
I don't want it to become an extra burden on such an already stressed system.
Quote:
Originally Posted by H_TeXMeX_H View Post
As for /proc/sys/net/ip_conntrack, I don't have the file, so I didn't know.
Sorry, it's /proc/net/ip_conntrack
You will probably have it.
Quote:
Originally Posted by H_TeXMeX_H View Post
But then, how does wc -l work if the file has no size ?
It works fine ;-)

BTW, today I got an email with those C-programs I wrote more than 25 years ago which I published later as shareware on fidonet. Just found them on the Internet.
http://pastebin.com/37UncgHf
This soundex function was done by Nantucket in assembly language using a complete different algorithm.
I wrote it in my own way in C (I found assembly too cumbersome) and made it 3 times as fast returning a 5 byte code instead of 4. Speed was a necessity because the function was used as an index key for a database.

Although I was an experienced programmer in Clipper, the only things I did in C were these little functions. All original work as I had no-one I could learn from (I didn't even know there was a thing called Internet). Later in my life I created some functions in 'C' managing sets (union, intersection...) (http://en.wikipedia.org/wiki/Set_%28mathematics%29).

Last edited by frater; 01-30-2011 at 04:18 AM.
 
Old 01-29-2011, 02:02 PM   #18
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,774

Rep: Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081
You went and changed the benchmark! For 1.3GB.txt, it looks like most of the time is spent waiting for the disk since sys+user is much less than real time. I would suggest that you on test ip_conntrack since that is what you are actually interested in.
 
Old 01-30-2011, 04:10 AM   #19
frater
Member
 
Registered: Jul 2008
Posts: 121

Original Poster
Rep: Reputation: 23
Quote:
Originally Posted by ntubski View Post
You went and changed the benchmark! For 1.3GB.txt, it looks like most of the time is spent waiting for the disk since sys+user is much less than real time. I would suggest that you on test ip_conntrack since that is what you are actually interested in.
I would never have done this if I had a choice in this. I have no access to that server nor will I ever have.
I don't have any server at my disposal that handles that many connections.

Last edited by frater; 01-30-2011 at 04:19 AM.
 
Old 02-03-2011, 08:56 AM   #20
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,774

Rep: Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081
I tried writing a script to open 40000 connections but I found things slowed to a crawl around 15000. I'll give it another shot on the weekend.

I also found conntrack while searching:
Quote:
conntrack provides a full featured userspace interface to the netfilter connection tracking system that is intended to replace the old /proc/net/ip_conntrack interface.
Might be a faster method of listing connections if you have it installed on your server.
 
Old 02-06-2011, 08:51 PM   #21
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian, Arch
Posts: 3,774

Rep: Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081Reputation: 2081
So it turns out increasing the buffer size doesn't help, I rewrote lc.c to use open() and read(), and discovered that it doesn't read() more than 4096 bytes at a time.

Code:
~/tmp$ ./lc /proc/net/ip_conntrack
nread = 4080, BUFFER_SIZE = 4194304
nread = 4080, BUFFER_SIZE = 4194304
nread = 4080, BUFFER_SIZE = 4194304
nread = 3060, BUFFER_SIZE = 4194304
    76 /proc/net/ip_conntrack
Code:
/* Sample implementation of wc utility. */

#include <string.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <stdarg.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>



typedef unsigned long count_t;  /* Counter type */

static char *buffer;
/* make this a bit bigger than the size of the file */
#define BUFFER_SIZE (4*1024*1024) /* 3 891 335 */

/* Current file counters: chars, words, lines */
count_t ccount;
count_t wcount;
count_t lcount;

/* Totals counters: chars, words, lines */
count_t total_ccount = 0;
count_t total_wcount = 0;
count_t total_lcount = 0;

/* Print error message and exit with error status. If PERR is not 0,
   display current errno status. */
static void
error_print (int perr, char *fmt, va_list ap)
{
    vfprintf (stderr, fmt, ap);
    if (perr)
        perror (" ");
    else
        fprintf (stderr, "\n");
    exit (1);
}

/* Print error message and exit with error status. */
static void
errf (char *fmt, ...)
{
    va_list ap;

    va_start (ap, fmt);
    error_print (0, fmt, ap);
    va_end (ap);
}

/* Print error message followed by errno status and exit
   with error code. */
static void
perrf (char *fmt, ...)
{
    va_list ap;

    va_start (ap, fmt);
    error_print (1, fmt, ap);
    va_end (ap);
}

/* Output counters for given file */
void
report (const char *file, count_t lcount)
{
    printf ("%6lu %s\n", lcount, file);
}

/* Process file FILE. */
void
counter (const char *file)
{
    int fd = open (file, O_RDONLY);
    int total_read = 0, lcount = 0;

    if (fd < 0)
        perrf ("cannot open file `%s'", file);

    for (;;) {
        int nread = read(fd, buffer+total_read, BUFFER_SIZE - total_read);
        if (nread < 0) perrf("read");
        if (nread == 0) break;  /* eof */
        total_read += nread;
        /* printf("nread = %d, BUFFER_SIZE = %d\n", nread, BUFFER_SIZE); */
    }

    for (const char *p = buffer; p; p = memchr(p+1, '\n', total_read - (p+1 - buffer)))
        lcount++;

    report (file, lcount);
    total_lcount += lcount;

    close(fd);
}

int
main (int argc, char **argv)
{
    int i;

    if (argc < 2)
        errf ("usage: lc FILE [FILE...]");

    buffer = malloc(BUFFER_SIZE);

    for (i = 1; i < argc; i++)
        counter (argv[i]);

    if (argc > 2)
        report ("total", total_lcount);
    return 0;
}
The program to open a bunch of ports, if anyone is interested:
Code:
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <unistd.h>
#include <netinet/in.h>
#include <sys/epoll.h>

int client_socket(int port)
{
    int status, yes = 1, fail;
    struct addrinfo hints;
    struct addrinfo *servinfo, *si;  // will point to the results
    char portname[8];
    int sockfd;

    snprintf(portname, sizeof portname, "%d", port);

    memset(&hints, 0, sizeof hints); // make sure the struct is empty
    hints.ai_family = AF_INET;
    hints.ai_socktype = SOCK_STREAM; // TCP stream sockets
    hints.ai_flags = AI_NUMERICSERV; /* numeric address */

    if ((status = getaddrinfo("127.0.0.1", portname, &hints, &servinfo)) != 0) {
        fprintf(stderr, "getaddrinfo error: %s\n", gai_strerror(status));
        exit(1);
    }

    // servinfo now points to a linked list of 1 or more struct addrinfos
    for (si = servinfo; si; si = si->ai_next) {
        sockfd = socket(si->ai_family, si->ai_socktype, si->ai_protocol);
        if (sockfd < 0) {
            perror("socket");
            continue;
        }

        fail = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
        if (fail) { perror("fail"); exit(1); }

        break;
    }

    if (!si) {
        exit(1);
    }
    freeaddrinfo(servinfo); // free the linked-list

    fail = connect(sockfd, si->ai_addr, si->ai_addrlen);
    if (fail) { perror("connect"); exit(1); }
    return sockfd;
}

int server_socket(int port)
{
    int status, yes = 1, fail;
    struct addrinfo hints;
    struct addrinfo *servinfo, *si;  // will point to the results
    char portname[8];
    int sockfd;

    snprintf(portname, sizeof portname, "%d", port);

    memset(&hints, 0, sizeof hints); // make sure the struct is empty
    hints.ai_family = AF_INET;
    hints.ai_socktype = SOCK_STREAM; // TCP stream sockets
    hints.ai_flags = AI_PASSIVE;     // fill in my IP for me

    if ((status = getaddrinfo(NULL, portname, &hints, &servinfo)) != 0) {
        fprintf(stderr, "getaddrinfo error: %s\n", gai_strerror(status));
        exit(1);
    }

    // servinfo now points to a linked list of 1 or more struct addrinfos
    for (si = servinfo; si; si = si->ai_next) {
        sockfd = socket(si->ai_family, si->ai_socktype, si->ai_protocol);
        if (sockfd < 0) continue;

        fail = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
        if (fail) { perror("fail"); exit(1); }
        fail = bind(sockfd, si->ai_addr, si->ai_addrlen);
        if (!fail) break;
        else close(sockfd);
    }

    if (!si) {
        perror("bind");
        exit(1);
    }
    freeaddrinfo(servinfo); // free the linked-list

    fail = listen(sockfd, 1);
    if (fail) {
        perror("listen");
        exit(1);
    }

    return sockfd;
}

int main(int argc, char **argv)
{
    assert(argc == 4);
    int portlo = atoi(argv[2]);
    int porthi = atoi(argv[3]);
    assert(portlo && porthi);

    if (strcmp("--client", argv[1]) == 0) {
        for (int p = portlo; p < porthi; p++) {
            client_socket(p);
            if (p % 100 == 0) printf(".");
            if (p % 1000 == 0) printf("%d\n", p);
        }
        goto hang;
    } /* else serve */

    int epollfd;
#define MAX_EVENTS 1024
    static struct epoll_event ev, events[MAX_EVENTS];

    epollfd = epoll_create(porthi - portlo);
    if (epollfd < 0) { perror("epoll_create"); exit(1); }

    ev.events = EPOLLIN;
    for (int p = portlo; p < porthi; p++) {
        int sock = server_socket(p);
        ev.data.fd = sock;
        if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sock, &ev) == -1) {
            perror("epoll_ctl(add, listener)");
            exit(1);
        }
        if (p % 100 == 0) printf(".");
        if (p % 1000 == 0) printf("%d\n", p);
    }

    for (;;) {
        int nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
        if (nfds < 0) { perror("epoll_wait"); exit(1); }
        for (int n = 0; n < nfds; n++ ) {
            int tcpsock = accept(events[n].data.fd, NULL, NULL);
            if (tcpsock < 0) {
                perror("accept"); exit(1);
            }
            if (epoll_ctl(epollfd, EPOLL_CTL_DEL, events[n].data.fd, &events[n]) == -1) {
                perror("epoll_ctl(del, listener)"); exit(1);
            }
            close(events[n].data.fd); /* done listening */
        }
    }

 hang:
    for (;;) sleep(1);
    return 0;
}
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to count # of chars per line (if line meets certain criteria) and get avg #? kmkocot Linux - Newbie 3 09-13-2009 11:05 AM
write line count to a variable? tatoosh Linux - Newbie 9 07-30-2009 03:44 PM
word count in a line pantera Programming 4 08-25-2004 01:14 PM
How to count line numbers recursively in a directory? puzz_1 Linux - General 1 07-01-2004 09:43 AM
Count number of line in a file. philipina Programming 7 03-18-2004 05:04 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 05:34 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration