Equivalent to Windows batch file

Sanmayce · 07-01-2019, 02:28 PM

Need the equivalent .sh script to Windows batch file that I use:

Nakamichify_16MB.bat:

Code:

@if '%1'=='' goto usage
Chunkerito.exe "%1" 16777216
dir "chunkerito.???,???"/b/on>q

@echo Processing these...
@FOR /F %%i in (q) do @echo %%i
@FOR /F %%i in (q) do "Nakamichi_Ryuugan-ditto-1TB_btree.exe" %%i x 25 13000 i

copy/b *.Nakamichi "%1.NKM"
ren "%1.NKM" "%1.Nakamichi"
"Nakamichi_Ryuugan-ditto-1TB_btree.exe" "%1.Nakamichi" /bench

:finish
goto realend

:usage
@echo Usage: Nakamichify filename
@echo.

:realend

Also, please point out the help resource describing used commands:

- how to concatenate (as binary) files;
- how to handle command line options (the %1 and %2 in Windows);
- how to print files (one-by-one) from a list file;
- how to rename a file.

Having the correct counterpart will allow me to look the right places.

rtmistler · 07-01-2019, 02:37 PM

Quote:

Originally Posted by Sanmayce

Need the equivalent .sh script to Windows batch file that I use:

Have you started anything at all? Please consider reading up on The Bash Beginners Guide and if you find that an easy read, then there's also the Advanced Bash Scripting Guide.

Or if you are interested in using Python, there's resources to review there.

Please understand that people here are volunteers, like yourself, and not necessarily here to write on-demand scripts for you. As you know the way the LQ site works is that you should be learning how to help yourself on technical Linux topics. Therefore some background effort would be very helpful so that people can make script suggestions where you'll understand what they are about.

Quote:

Originally Posted by Sanmayce

Also, please point out the help resource describing used commands:

- how to concatenate (as binary) files;
- how to handle command line options (the %1 and %2 in Windows);
- how to print files (one-by-one) from a list file;
- how to rename a file.

Having the correct counterpart will allow me to look the right places.

Are you familiar with the Linux command line? All of those capabilities are commands which you'd encounter and use as part of the command line on an everyday basis. Afraid I don't have a favorite resource to recommend for the command line, however all of it is highly visible/accessible with web searches.

Sanmayce · 07-01-2019, 02:50 PM

The problem is that I still don't have a working *nix environment, after some days I intend to play with RUFUS which will make me a bootable LIVE UBUNTU USB stick, hopefully then I will be able to try making the counterpart. I already downloaded the .ISO file but booting UBUNTU and working in its shell is a new thing to me. Also, by "script" I don't know what to expect, I need the basic and portable scripting language, as far as I know, BASH is only one variant, don't know whether it is the standard for common shell tasks as in Windows. Don't need advanced scripting just as a start this one.

Thanks for the https://www.tldp.org/LDP/Bash-Beginn...ect_09_01.html
It was helpful to see at the bottom 'Similar Threads', there found

this command will do what you want:

for i in `ls *.deb`; do dpkg -i $i; done

note the use of `backticks` and not 'apostrophes'

https://www.linuxquestions.org/quest...4/#post3995896

rtmistler · 07-01-2019, 07:28 PM

You really can't do much until you have an environment which provides a shell.

There are a number of options to get you a shell environment, cygwin, virtual box, or do what you plan and install Ubuntu.

At that point you'll be ready to start learning how to write and debug some bash scripts.

chrism01 · 07-02-2019, 01:07 AM

Actually, the 'Advanced Bash..' is really just part 2 of the Beginner's one; don't be put off by the name...

ehartman · 07-02-2019, 01:20 AM

Quote:

Originally Posted by Sanmayce

for i in `ls *.deb`; do dpkg -i $i; done

note the use of `backticks` and not 'apostrophes'

You can avoid backticks AND the use of the ls with

Code:

for i in *.deb; do dpkg -i $i; done

JJJCR · 07-04-2019, 02:39 AM

Quote:

The problem is that I still don't have a working *nix environment

Is Virtualbox not an option?

But try this one it might suit your needs, it will copy the files while changing the file extension.

Code:

for file in *.src; do
    cp "${file}" "${tgt_dir}"/"${file/%.src/.tgt}"
done

Found on this link:
https://stackoverflow.com/questions/...ging-extension

evo2 · 07-04-2019, 03:22 AM

Hi,
something like the following I guess.

Code:

#!/bin/bash

function usage
{
    echo "Usage ${0} filename"
}

if [ "$1" = "" ] ; then
    usage
    exit 1
fi

split --bytes=16777216 -d $1 $1_

for f in $1_* ; do
    echo $f
done

for f in $1_* ; do
    Nakamichi_Ryuugan-ditto-1TB_btree.exe $f
done

# I don't really understand the point of thesecopy and rename lines.
cp *.Nakamichi $1.NKM
mv $1.NKM $1.Nakamichi

Nakamichi_Ryuugan-ditto-1TB_btree.exe $1.Nakamichi --bench

Please note that I don't really understand the point of your original script or the commands used in it.

Evo2.

Sanmayce · 07-04-2019, 06:47 AM

Thank you ehartman, JJJCR and evo2, that is helpful, learning from actual examples is the way I like to do things.

>Is Virtualbox not an option?

Heard how good it is, but never have seen it, on top of that currently I am limited to using old Windows XP 32bit, kinda all the software seems to deprecate it. Have to try it, but having worked a bit with an old 32bit Knoppix live CD, years ago, I prefer some full fledged environment as UBUNTU, in too many cases I see people using it, maybe it is the primary choice!
When have time, want to have a modern 64bit *nix GCC environment (LIVE - on a RAM drive) with GCC 7.3.0 as a minimum in order to create ELFs (the counterparts for my EXEs).

Exactly what I needed, evo2, will try it when enter the *nix prompt and share whether it does the same as my .BAT file.

> echo "Usage ${0} filename"

Oh, this ${0} actually stands for the executed script name, I guess, nice refinement it is.

>if [ "$1" = "" ] ; then
> usage
> exit 1

Good, just now I saw the counterpartS of handling the parameter and "goto" - using a function within the "batch" is new to me.

>split --bytes=16777216 -d $1 $1_

Oof-oof, nice, never heard of 'split', wonder are the $1_ files "numbered".

># I don't really understand the point of thesecopy and rename lines.
>cp *.Nakamichi $1.NKM

The task is to glue them (in BINARY mode) into $1.NKM, the actual ordering is also important, could you say whether 'cp' copies in binary mode as it is given.

Also, if the first line "#!/bin/bash" is omitted, in order not only bash (but other shells) to be able to exectute it, will it be more "portable" i.e. universal.

See, this script is important to me (to have and share it with a small compression package), since Perl/Python/Bash is Dark India to me, I PREFER some fellow experienced member to write it instead of my crippled attempt done.

Thanks

Sanmayce · 07-04-2019, 06:59 AM

>Please note that I don't really understand the point of your original script or the commands used in it.

For what is worth, here is the 100% FREE compression package (with C source code) where the *nix script has to go:
https://community.centminmod.com/posts/75533/

Not knowing that *nix has its own splitter, I wrote a fast one - it is in the package:

Code:

// Chunkerito.c, written by Kaze, 2014-Aug-14
// Purpose: To chunkize/split any file to 'ChunkSize' long chunks.
// The funny 'bring.me.the.head.of.the.machinegun.woman' movie gave the name... ChinchinERO/ChinchinERITO.

#ifndef NULL
#ifdef __cplusplus
#define NULL 0
#else
#define NULL ((void*)0)
#endif
#endif

// Windows: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//  _CRTIMP size_t __cdecl fread(void *, size_t, size_t, FILE *);
//  _CRTIMP size_t __cdecl fwrite(const void *, size_t, size_t, FILE *);
//  _CRTIMP int __cdecl fgetpos(FILE *, fpos_t *);
//  _CRTIMP int __cdecl fsetpos(FILE *, const fpos_t *);

//  _CRTIMP __int64 __cdecl _lseeki64(int, __int64, int);
//  _CRTIMP __int64 __cdecl _telli64(int);
//  _CRTIMP __int64 __cdecl _filelengthi64(int);
//  above 3 are in 'io.h'

//  _CRTIMP int __cdecl fseek(FILE *, long, int);
//  _CRTIMP long __cdecl ftell(FILE *);
//  _CRTIMP int __cdecl fclose(FILE *);

//  #ifndef _SIZE_T_DEFINED
//  #ifdef  _WIN64
//  typedef unsigned __int64    size_t;
//  #else
//  typedef _W64 unsigned int   size_t;
//  #endif
//  #define _SIZE_T_DEFINED
//  #endif

//  typedef __int64 fpos_t;

// Linux: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//  size_t fread (void *data, size_t size, size_t count, FILE *stream)
//  size_t fwrite (const void *data, size_t size, size_t count, FILE *stream)
//  int fgetpos (FILE *stream, fpos_t *position)
//  int fsetpos (FILE *stream, const fpos_t *position)

//  FILE * fopen64 (const char *filename, const char *opentype)
//  int fseeko64 (FILE *stream, off64_t offset, int whence)  
//  off64_t ftello64 (FILE *stream)  
//  int fclose (FILE *stream)  

//  off_t lseek (int filedes, off_t offset, int whence)
//  above 1 is in 'unistd.h'


// ============== MUST work both for Windows and Linux ==============
//Only one must be uncommented:
//#define _WIN32_ENVIRONMENT_
//#define _POSIX_ENVIRONMENT_

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h> // uint64_t needed
#include <time.h>
#include <string.h>

#if defined(_WIN32_ENVIRONMENT_)
#include <io.h> // needed for Windows' 'lseeki64' and 'telli64'
#else
#endif

typedef unsigned char char_t;
typedef char_t *string;


void x64toaKAZE (      /* stdcall is faster and smaller... Might as well use it for the helper. */
        unsigned long long val,
        char *buf,
        unsigned radix,
        int is_neg
        )
{
        char *p;                /* pointer to traverse string */
        char *firstdig;         /* pointer to first digit */
        char temp;              /* temp char */
        unsigned digval;        /* value of digit */

        p = buf;

        if ( is_neg )
        {
            *p++ = '-';         /* negative, so output '-' and negate */
            val = (unsigned long long)(-(long long)val);
        }

        firstdig = p;           /* save pointer to first digit */

        do {
            digval = (unsigned) (val % radix);
            val /= radix;       /* get next digit */

            /* convert to ascii and store */
            if (digval > 9)
                *p++ = (char) (digval - 10 + 'a');  /* a letter */
            else
                *p++ = (char) (digval + '0');       /* a digit */
        } while (val > 0);

        /* We now have the digit of the number in the buffer, but in reverse
           order.  Thus we reverse them now. */

        *p-- = '\0';            /* terminate string; p points to last digit */

        do {
            temp = *p;
            *p = *firstdig;
            *firstdig = temp;   /* swap *p and *firstdig */
            --p;
            ++firstdig;         /* advance to next two digits */
        } while (firstdig < p); /* repeat until halfway */
}

/* Actual functions just call conversion helper with neg flag set correctly,
   and return pointer to buffer. */

char * _i64toaKAZE (
        long long val,
        char *buf,
        int radix
        )
{
        x64toaKAZE((unsigned long long)val, buf, radix, (radix == 10 && val < 0));
        return buf;
}

char * _ui64toaKAZE (
        unsigned long long val,
        char *buf,
        int radix
        )
{
        x64toaKAZE(val, buf, radix, 0);
        return buf;
}

char * _ui64toaKAZEzerocomma (
        unsigned long long val,
        char *buf,
        int radix
        )
{
                        char *p;
                        char temp;
                        int txpman;
                        int pxnman;
        x64toaKAZE(val, buf, radix, 0);
                        p = buf;
                        do {
                        } while (*++p != '\0');
                        p--; // p points to last digit
                             // buf points to first digit
                        buf[26] = 0;
                        txpman = 1;
                        pxnman = 0;
                        do
                        { if (buf <= p)
                          { temp = *p;
                            buf[26-txpman] = temp; pxnman++;
                            p--;
                            if (pxnman % 3 == 0)
                            { txpman++;
                              buf[26-txpman] = (char) (',');
                            }
                          }
                          else
                          { buf[26-txpman] = (char) ('0'); pxnman++;
                            if (pxnman % 3 == 0)
                            { txpman++;
                              buf[26-txpman] = (char) (',');
                            }
                          }
                          txpman++;
                        } while (txpman <= 26);
        return buf;
}

char * _ui64toaKAZEcomma (
        unsigned long long val,
        char *buf,
        int radix
        )
{
                        char *p;
                        char temp;
                        int txpman;
                        int pxnman;
        x64toaKAZE(val, buf, radix, 0);
                        p = buf;
                        do {
                        } while (*++p != '\0');
                        p--; // p points to last digit
                             // buf points to first digit
                        buf[26] = 0;
                        txpman = 1;
                        pxnman = 0;
                        while (buf <= p)
                        { temp = *p;
                          buf[26-txpman] = temp; pxnman++;
                          p--;
                          if (pxnman % 3 == 0 && buf <= p)
                          { txpman++;
                            buf[26-txpman] = (char) (',');
                          }
                          txpman++;
                        } 
        return buf+26-(txpman-1);
}


int main( argc, argv )
int argc; char *argv[];
{

#if defined(_WIN32_ENVIRONMENT_)
      unsigned long long size_inLINESIXFOUR;
#else
      size_t size_inLINESIXFOUR;
#endif 

//#define ChunkSize 1024*128*2
//#define ChunkSize 1024*1024*257

      FILE *fp_outLOG, *fp_inLINE;
      unsigned long long i;              
      unsigned char workbyte; // unsigned in order to index ASCII
//      char workK[ChunkSize];
      char *workK=NULL;
      int ChunkSize=0;
      long workKoffset = -1;
      int ReallyRead;
      int Melnitchka=0;

      unsigned long long NumberOfLines=0;              
      unsigned long long ChunkCounter=0;              

/* minimum signed 64 bit value */
#define _I64_MIN    (-9223372036854775807i64 - 1)
/* maximum signed 64 bit value */
#define _I64_MAX      9223372036854775807i64
/* maximum unsigned 64 bit value */
#define _UI64_MAX     0xffffffffffffffffui64

/* minimum signed 128 bit value */
#define _I128_MIN   (-170141183460469231731687303715884105727i128 - 1)
/* maximum signed 128 bit value */
#define _I128_MAX     170141183460469231731687303715884105727i128
/* maximum unsigned 128 bit value */
#define _UI128_MAX    0xffffffffffffffffffffffffffffffffui128

      char llTOaDigits[27]; // 9,223,372,036,854,775,807: 1(sign or carry)+19(digits)+1('\0')+6(,)
      // below duplicates are needed because of one_line_invoking need different buffers.
      char llTOaDigits2[27]; // 9,223,372,036,854,775,807: 1(sign or carry)+19(digits)+1('\0')+6(,)
      char llTOaDigits3[27]; // 9,223,372,036,854,775,807: 1(sign or carry)+19(digits)+1('\0')+6(,)
      char llTOaDigits4[27]; // 9,223,372,036,854,775,807: 1(sign or carry)+19(digits)+1('\0')+6(,)

      //char *CurrentSolid = "Chunkerito.000,000\0";
      char CurrentSolid[19];
      char *CurrentSolidEXT;

char *Auberge[4] = {"|\0","/\0","-\0","\\\0"};
char *ngram[11] = {"NULLleton\0","singleton\0","doubleton\0","tripleton\0","quadrupleton\0","quintupleton\0","sextupleton\0","septupleton\0","octupleton\0","nonupleton\0","decupleton\0"};

/*
	size_in64_L14 = 1024 * (unsigned long long)Thunderwith + 14;
BufEnd_64 = 0+14;
// The tag plays two roles, the second to avoid existence of SeekPosition equal to 0. The 0 cannot be used as a free slot FLAG without the TAG.
	printf( "Allocating/ZEROing %s bytes swap file ... ", _ui64toaKAZEcomma(size_in64_L14, llTOaDigits, 10) );
	fsetpos(fp_outRG, &fsetpos_ZERO); // SOMETHING ROTTEN with lseeki64/fseeko and fsetpos ??!! So DO-IT-OVER.
	memset(OneCkusterZEROES,0,1024*4);

	for (ThunderwithL64_L14=0; ThunderwithL64_L14 < size_in64_L14/(1024*4); ThunderwithL64_L14++)
        	fwrite(OneCkusterZEROES, 1024*4, 1, fp_outRG);
	for (ThunderwithL64_L14=0; ThunderwithL64_L14 < size_in64_L14%(1024*4); ThunderwithL64_L14++)
        	fwrite(&OneChar_ieByte, 1, 1, fp_outRG);
	fsetpos(fp_outRG, &BufEnd_64); // SOMETHING ROTTEN with lseeki64/fseeko and fsetpos ??!! So DO-IT-OVER.
*/

printf("Chunkerito, revision 1+, written by Kaze.\n");
printf("Purpose: To chunkize/split any file to 'ChunkSize' long chunks.\n");
printf("Usage: Chunker filename ChunkSize\n");
printf("Note: For 128MB chunks use ChunkSize = 134217728\n");

if (argc!=3) exit (13);

	ChunkSize = atoi(argv[2]);
	workK = (char*)malloc(ChunkSize);

if( ( fp_inLINE = fopen( argv[1], "rb" ) ) == NULL )
{ printf( "Chunkerito: Can't open file %s \n", argv[1] ); return( 1 ); }

//fseek( fp_inLINE, 0L, SEEK_END );  //Rev. 12
//size_inLINE = ftell( fp_inLINE );  //Rev. 12
//fseek( fp_inLINE, 0L, SEEK_SET );  //Rev. 12

#if defined(_WIN32_ENVIRONMENT_)
   // 64bit:
_lseeki64( fileno(fp_inLINE), 0L, SEEK_END );
size_inLINESIXFOUR = _telli64( fileno(fp_inLINE) );
_lseeki64( fileno(fp_inLINE), 0L, SEEK_SET );
#else
   // 64bit:
fseeko( fp_inLINE, 0L, SEEK_END );
size_inLINESIXFOUR = ftello( fp_inLINE );
fseeko( fp_inLINE, 0L, SEEK_SET );
#endif 

printf( "Size of Input TEXTual file: %s\n", _ui64toaKAZEcomma(size_inLINESIXFOUR, llTOaDigits, 10) );

/*
        for( i = 0; i < size_inLINESIXFOUR; i++ )
	{

                // ~~~~~~~~~~~~ Buffering fread [
                if (workKoffset == -1) {
                        if (i + ChunkSize < size_inLINESIXFOUR) {
                                fread( &workK[0], 1, ChunkSize, fp_inLINE );
                                workKoffset = 0;
                                workbyte = workK[workKoffset];
                        } else 
                        fread( &workbyte, 1, 1, fp_inLINE );
                } else {
                        workKoffset++;
                        workbyte = workK[workKoffset];
                        if (workKoffset == ChunkSize - 1) workKoffset = -1;
                }
                // ~~~~~~~~~~~~ Buffering fread ]
                        if (workbyte == 10) {NumberOfLines++;}
        }
*/
	
	memcpy(CurrentSolid, "Chunkerito.000,000\0", strlen("Chunkerito.000,000\0")+1);
        // Chunkization [
        for( i = 0; i < size_inLINESIXFOUR; i++ )
	{
		ReallyRead = fread( &workK[0], 1, ChunkSize, fp_inLINE );
		i=i+ReallyRead;
		//if (ReallyRead == ChunkSize)
		ChunkCounter++;

		CurrentSolidEXT = _ui64toaKAZEzerocomma(ChunkCounter, llTOaDigits, 10)+(26-7);
		memcpy(CurrentSolid+(12-1), CurrentSolidEXT, 7);

		if( ( fp_outLOG = fopen( CurrentSolid, "wb" ) ) == NULL )
		{ printf( "Chunkerito: Can't open %s file.\n", CurrentSolid ); exit( 1 ); }
		fwrite( &workK[0], ReallyRead, 1, fp_outLOG );
		fclose(fp_outLOG);

		Melnitchka = Melnitchka & 3; // 0 1 2 3: 00 01 10 11
		printf( "%s; Chunk # %s has been created ...\r", Auberge[Melnitchka++], _ui64toaKAZEcomma(ChunkCounter, llTOaDigits, 10));
        }
        // Chunkization ]
	printf( "\n");

//printf( "Number Of LFs (ASCII 010): %s\n", _ui64toaKAZEcomma(NumberOfLines, llTOaDigits, 10) );
free(workK);
exit(0);
}

To compile it:

Code:

gcc -O3 -static -msse4.1 -fomit-frame-pointer Chunkerito.c -o Chunkerito.elf -D_POSIX_ENVIRONMENT_

evo2 · 07-04-2019, 07:51 PM

Hi,

Quote:

Originally Posted by Sanmayce

>if [ "$1" = "" ] ; then
> usage
> exit 1

Good, just now I saw the counterpartS of handling the parameter and "goto" - using a function within the "batch" is new to me.

goto should be avoided whenever possible IMHOI.

Quote:

>split --bytes=16777216 -d $1 $1_

Oof-oof, nice, never heard of 'split', wonder are the $1_ files "numbered".

https://linux.die.net/man/1/splint

Quote:

># I don't really understand the point of thesecopy and rename lines.
>cp *.Nakamichi $1.NKM

The task is to glue them (in BINARY mode) into $1.NKM, the actual ordering is also important, could you say whether 'cp' copies in binary mode as it is given.

It is not the dark ages. cp does what it should
https://linux.die.net/man/1/cp

Quote:

Also, if the first line "#!/bin/bash" is omitted, in order not only bash (but other shells) to be able to exectute it, will it be more "portable" i.e. universal.

If you want to make it more portable you can use /bin/sh but then you should remove any "bashisms".

Quote:

See, this script is important to me (to have and share it with a small compression package), since Perl/Python/Bash is Dark India to me, I PREFER some fellow experienced member to write it instead of my crippled attempt done.

Hmm, not sure what is so special about the compression you are using. There is already a huge number of well established ones to choose from.

Evo2.

Sanmayce · 07-05-2019, 07:28 AM

>It is not the dark ages. cp does what it should

I experienced problems with 'cp' and couldn't do it!

My first attempt:

Code:

>cp -v *.Nakamichi q
cp: target `q' is not a directory

My second attempt:

Code:

>cp -T -v *.Nakamichi q
cp: extra operand `Chunkerito.000,003.Nakamichi'

Looked up the man page and couldn't figure out how to copy it, then I had a flashback and remembered 'cat' being somewhat the counterpart of 'type' in Windows:

Code:

>cat *.Nakamichi>q

Looking into current directory, I saw 'q' being exactly the size of concatenated 6 Chunkerito files. Success. Why 'cp' failed me?

Also, I experienced problem with $f, don't know why but instead of 6 instances:
Nakamichi_Ryuugan-ditto-1TB_btree.exe "Chunkerito.000,001"
Nakamichi_Ryuugan-ditto-1TB_btree.exe "Chunkerito.000,002"
Nakamichi_Ryuugan-ditto-1TB_btree.exe "Chunkerito.000,003"
Nakamichi_Ryuugan-ditto-1TB_btree.exe "Chunkerito.000,004"
Nakamichi_Ryuugan-ditto-1TB_btree.exe "Chunkerito.000,005"
Nakamichi_Ryuugan-ditto-1TB_btree.exe "Chunkerito.000,006"

It executed one instance with $f being:
Nakamichi_Ryuugan-ditto-1TB_btree.exe "Chunkerito.000,001 Chunkerito.000,002 Chunkerito.000,003 Chunkerito.000,004 Chunkerito.000,005 Chunkerito.000,006"

In that fragment:

Code:

for f in $1_* ; do
    Nakamichi_Ryuugan-ditto-1TB_btree.exe $f
done

Finally, I succeeded in making Nakamichify_32MB.sh:

Code:

./Chunkerito.elf $1 33554432

for f in `ls Chunkerito.???,???` ; do ./Nakamichi_Ryuugan-ditto-1TB_btree.elf $f x 20 14000 i; done
#for f in `ls Chunkerito.???,???` ; do echo $f; done

cat *.Nakamichi>$1.NKM
mv $1.NKM $1.Nakamichi

./Nakamichi_Ryuugan-ditto-1TB_btree.elf $1.Nakamichi /bench

IT WORKS!

>Hmm, not sure what is so special about the compression you are using.

Nothing special really, just a Zennish Microdeduplicator, a natural continuation of Prof. Okumura's shared LZSS.C back in 1989, 30 years later instead of 256 Binary Trees we now can have 2^31 or 2 billion B-trees - to boost the matchfinding. The purpose of Nakamichi is to speed up huge textual traversals (full-text) by reducing the I/O, f.e. if 3 TB DNA files are to be traversed in full-text mode then Nakamichi screams - BUT! You need first a bunch of tiny supercomputers to compress them.

Heh-heh, love koans - the only way to express the truth - via paradoxes:

Tao Te Ching Chapter 56-7 Kanshiketsu / Like dust
https://www.youtube.com/watch?v=pgwKn7sMUeo

CASE 21. UMMON'S DRIED DUNG
A monk asked Ummon,"What is Buddha?" Ummon answered him, "Dried dung."
http://www.angelfire.com/electronic/.../mumonkan.html

Funny guys were those Zen/Tao masters, borrowing wisdom from them, my next variant will be named 中道乾屎橛 Nakamichi 'Kanshiketsu' - a multi-threaded (OpenMP) decompressor of these chunks able to compete with the fastest ones.

>There is already a huge number of well established ones to choose from.

A huge number you say, curious I am what is your TOP 5, regarding superfast textual decompression?

These days I am playing with DNA corpora, despite Nakamichi targeting English Texts. A quick look what above script should deliver:

https://www.linuxquestions.org/quest...1&d=1562331938

Benchmark machine:
Our reference machine is a reasonably standard Linux workstation.
CPU: dual Xeon E5-2643v3 (3.4 GHz, 6 cores)
RAM: 128 GB DDR4-2133 ECC Registered

Benchmark machine for Nakamichi:
Sanmayce's reference machine is a reasonably standard Windows laptop.
CPU: i7-3630QM (3.4 GHz, 4 cores)
RAM: 16 GB DDR3-1600

The used 'SILVA' corpus is here:
https://www.arb-silva.de/no_cache/do...e_132/Exports/

ehartman · 07-05-2019, 01:56 PM

Quote:

Originally Posted by Sanmayce

>It is not the dark ages. cp does what it should

I experienced problems with 'cp' and couldn't do it!

My first attempt:

Code:

>cp -v *.Nakamichi q
cp: target `q' is not a directory

cp isn't cat.
It has two modes:
EXACTLY two arguments: copy the first filename to the 2nd, optionally to another directory too.
When the first argument contains wildcards (*, ?, etc) it must evaluate to exactly one single filename for this mode to work.
MORE then 2 arguments (your *.Nakamichi probably expands to multiple filenames):
the destination argument must be a directory where to copy all of these files TO, under their present filenames.
So it never will concatenate files together, that is what cat is for, cp will copy files as (separate) files.

evo2 · 07-05-2019, 10:28 PM

Lol, do you have a tldr version of that.

Sanmayce · 07-12-2019, 04:45 PM

For all rookies like me, my suggestions, these are the two books I chose to learn from:
- Apress.Pro.Bash.Programming.Scripting.the.GNU.Linux.Shell.Oct.2009.pdf
- Aho A., Kernighan B., Weinberger P. - The AWK Programming Language - 1988.pdf

The first one is exactly the style I like - a practical cookbook covering many usecases that I need, as if it was written for me :P
Next one, I chose in order to get portability, my inclination is to write sh scripts, not bash ones, also while looking for a way to get the free RAM into a variable found an one-liner using awk:

Code:

phymem=$(free|awk '/^Mem:/{print $2}')

Source: https://stackoverflow.com/questions/...-to-a-variable

As far as I get, using 'free' also delivers portability, like such tiny techniques, intend to gather similar ones in order to enrich my sh/awk vocabulary.