LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 08-11-2019, 01:54 PM   #16
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 8,985

Original Poster
Rep: Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876

Quote:
Originally Posted by pan64 View Post
again, I know exactly what is ARG_MAX, I know bash ARRAYS, just I don't understand how comes these things together.
I think it is a totally wrong approach and you want to construct something useless (= that is the syntax error) caused by a limitation.
The solution could be to avoid passing such a huge array (but its name), or use a different language.

You can easily implement a perl/python/whatever oneliner to invoke any program and pass the full array to it without these limitations.
From the other hand there is a command xargs to do the split you are trying to implement.

What you try to implement in bash is a very good exercise to learn bash and to understand what should not be written in bash.
then please explain to me why I am getting the error I am getting when I add files with there paths included in there into an array that when totaled up the amount of chars against the ARG_MAX I get this error, and when i cause it to limit it below that ARG_MAX I no longer get this error.

I do not want to incorporate anything else Perl, python or anything It is just a simple script that takes in all of my images and gives me back on on a random to display in a terminal. No need to call in the army to deal with it.

again this is in FreeBSD where there ARG_MAX is less than as I already posted in here a few times on it.

as this was explained to me by someone in FreeBSD about ARG_MAX that I wasn't aware of until then. Because in Linux I wasn't getting this error, it wasn't until I used this same script IN FreeBSD that the error showed up.

it is just a simple question:

how to create arrays dynamically in bash. the reasoning behind it is actually mute, and plays no relevance to that question.

Last edited by BW-userx; 08-11-2019 at 02:34 PM.
 
Old 08-11-2019, 04:03 PM   #17
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 8,985

Original Poster
Rep: Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876
Quote:
Originally Posted by teckk View Post
That's what post 8 does. Takes 100's-1000's of files and puts them into 10 item array. Then do what you want with each array. So that you don't reach a item limit.

Loop through the arrays and do the same thing with each.
found this,

Getting Bashed by Dynamic Arrays
 
Old 08-11-2019, 05:41 PM   #18
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 2,241

Rep: Reputation: 474Reputation: 474Reputation: 474Reputation: 474Reputation: 474
That's good info. Thanks. More to add to my notes.

Anyone who wants a good formated .pdf of that for your wiki.
(Already had one made, just plugged in url)
Code:
#! /usr/bin/env python

import sys
from PyQt5.QtWidgets import QApplication
from PyQt5.QtGui import QPageLayout, QPageSize
from PyQt5.QtCore import QUrl, QSizeF, QMarginsF, pyqtSignal
from PyQt5.QtWebEngineWidgets import (QWebEngineView, 
                    QWebEngineProfile, QWebEngineSettings)
                                               
url = 'http://www.ludvikjerabek.com/2015/08/24/getting-bashed-by-dynamic-arrays/'

agent = ('Mozilla/5.0 (Windows NT 10.0; x86_64; rv:67.0)'
            ' Gecko/20100101 Firefox/67.0')

class PdfPrint():
    def __init__(self, url, out_file):
        super(PdfPrint, self).__init__()

        self.agent = QWebEngineProfile()
        self.agent.defaultProfile().setHttpUserAgent(agent)
            
        def print_pdf():
            self.printer.show()
            self.printer.page().printToPdf(out_file, layout)

        self.printer = QWebEngineView()
        self.printer.settings().setAttribute(
                QWebEngineSettings.JavascriptEnabled, False)
        self.printer.settings().setAttribute(
                QWebEngineSettings.AutoLoadImages, False)
        self.printer.settings().globalSettings().setFontSize(
                QWebEngineSettings.MinimumFontSize, (14))
                
        margins = QMarginsF(5, 5, 5, 5)
        layout = QPageLayout(QPageSize(QSizeF(279, 432),
                    QPageSize.Millimeter),QPageLayout.Portrait, 
                        margins, QPageLayout.Millimeter, margins)
                
        self.printer.load(QUrl(url))
        self.printer.loadFinished.connect(print_pdf)

if __name__ == '__main__':
    app = QApplication([])       
    out_file = "AraysDyn.pdf"
 
    PdfPrint(url, out_file)
    sys.exit(app.exec_())
 
1 members found this post helpful.
Old 08-12-2019, 01:03 AM   #19
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 12,943

Rep: Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080Reputation: 4080
Quote:
Originally Posted by BW-userx View Post
then please explain to me why I am getting the error I am getting when I add files with there paths included in there into an array that when totaled up the amount of chars against the ARG_MAX I get this error, and when i cause it to limit it below that ARG_MAX I no longer get this error.
This is not related to arrays, just bash cannot construct/handle lines longer than that. This is that simple.

Quote:
Originally Posted by BW-userx View Post
I do not want to incorporate anything else Perl, python or anything It is just a simple script that takes in all of my images and gives me back on on a random to display in a terminal. No need to call in the army to deal with it.
You need to understand the requirements are based on what you want to implement and not what you wish. It looks like there is a limitation. There can be workarounds - you got several tips to do that, but you cannot ignore it.

Quote:
Originally Posted by BW-userx View Post
it is just a simple question:

how to create arrays dynamically in bash. the reasoning behind it is actually mute, and plays no relevance to that question.
You still need to specify what do you mean by that. I can't see any problem with arrays (as long as you don't want to use very long lines).
 
1 members found this post helpful.
Old 08-12-2019, 08:04 AM   #20
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 8,985

Original Poster
Rep: Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876
Quote:
Originally Posted by pan64 View Post
This is not related to arrays, just bash cannot construct/handle lines longer than that. This is that simple.

You need to understand the requirements are based on what you want to implement and not what you wish. It looks like there is a limitation. There can be workarounds - you got several tips to do that, but you cannot ignore it.


You still need to specify what do you mean by that. I can't see any problem with arrays (as long as you don't want to use very long lines).
as this part is all new to me then yes, there is a certain level of ignorance here. Seeings that Linux is not having any issues with this script in how i take in a bazillion filenames with there absolute paths attached to them.

The issue did not pop up until I ran that script in FreeBSD, which I just installed and am learning the ins and outs of it, for the most part, just by setting ups its basic things I need and by using it.
 
Old 08-12-2019, 04:32 PM   #21
Sefyir
Member
 
Registered: Mar 2015
Distribution: Linux Mint
Posts: 593

Rep: Reputation: 288Reputation: 288Reputation: 288
Hi,

This is striking me as a XY problem, so I apologize for not answering the question directly.

From what I have gathered:

You are getting this error

Code:
'line 175: /usr/local/bin/shuf: Argument list too long'
Code:
	 ### HERE IS WHERE I AM GETTING THE 
	 #
	 #   /usr/local/bin/shuf: Argument list too long'
	 #
	 ###################################################
	#temp4=( $( shuf -e -n "$smallest" "${temp1[@]}" ) )
My question is, have you tried process substitution?

shuf cannot read from such a long argument list.. so don't read from the argument list


I have a directory lots_of_files that has 12000040 files in it

Code:
$ > stat lots_of_files/
  File: lots_of_files/
  Size: 12000040
Using shuf with command line arguments
Code:
$ > a="$(find lots_of_files/ -type f)"
$ > shuf "$a"
bash: /usr/bin/shuf: Argument list too long
Process substitution
Code:
$ > shuf <(find lots_of_files/ -type f) | head -n2
lots_of_files/file_462404
lots_of_files/file_260371
<(command) takes the output of command and puts it in a temporary file. You can see this with
Code:
echo <(find lots_of_files/ -type f)
/dev/fd/63
Basically, you trick shuf to treat find's output like a file, where it doesn't mind unlimited lines. Also, it's asynchronous so you don't have variables using a lot of memory sitting around. I like using it for that reason.

Quote:
Process Substitution
Process substitution allows a process's input or output to be referred to using a filename. It takes the form of <(list) or >(list). The process list is run asynchronously, and
its input or output appears as a filename. This filename is passed as an argument to the current command as the result of the expansion. If the >(list) form is used, writing to
the file will provide input for list. If the <(list) form is used, the file passed as an argument should be read to obtain the output of list. Process substitution is supported
on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.
Hope this helps!

Last edited by Sefyir; 08-12-2019 at 04:35 PM.
 
1 members found this post helpful.
Old 08-12-2019, 06:40 PM   #22
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 8,985

Original Poster
Rep: Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876Reputation: 1876
Thanks everyone for all of your input.
I just simplified it.
Code:
#!/usr/bin/env bash
#Aug. 12, 2019

#to give a random background image to terminal
#by chaning its resource file
#xfce4-terminal config file

wd1=$HOME/data/ScreenResizedImages/tallImages
wd2=$HOME/data/ScreenResizedImages/wideImages
wd3=$HOME/data/wallhaven-papers
SysColors=$HOME/bin/colorsOfASystem
templist=$HOME/.templist

RandomColor()
{
	cat "$SysColors" | shuf | shuf > tempColors
	mv tempColors "$SysColors"
	mapfile -t colors < "$SysColors"
}

MixRandomImages()
{
	touch $templist
	find "$wd1" "$wd3" "$wd2" -type f -name "*.jpg" | shuf >> $templist
}
#write changes to terminal config file
UpdateXFCE4()
{
	PAT=$HOME/.config/xfce4/terminal/terminalrc
	sed -ibak 's|BackgroundImageFile=.*|BackgroundImageFile='"$(sort -R $templist | head -n1)"'|' "$PAT"
}
#run functions
#RandomColor
#MixRandomImages
UpdateXFCE4
#rm -v $templist
#start the terminal.
xfce4-terminal &
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
BASH-Adding array element: Naming issue using array[${#array[*]}]=5 calvarado777 Programming 8 07-26-2013 09:48 PM
Remove dynamically created .html pages as soon as they are viewed? Or stream them? jlarsen Linux - Server 1 07-11-2013 01:47 PM
Extracting numbers from "" quoted char array and save to different array zme2000 Programming 8 11-02-2012 02:28 PM
Unable to dynamically create an array of params for execv in C nindzadza Programming 9 01-06-2011 12:51 PM
dynamically expanding an array in c++ a1ghagh05t Programming 4 03-07-2004 09:00 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 12:14 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration