Python/Perl Command Equivalents
 

Recently I asked for some ideas about porting a shell script to a cross platform scripting language.

I have no experience with Python or Perl, although learning both is on my to-do list.

I'm looking for a chart of some sort or mini how-to that explains which shell built-in and standard command line tools are available directly within the Python and Perl languages.

Specifically I am interested in these commands:

Shell built-in commands:
exit, for/done, if/then/fi, cd

Coreutils:
echo, ls, grep, cut, sort, uniq, head, tail, rm, cat, mv

Other:
(g)awk, sed

I am aware of and have used the GNUWin32 ports of many standard command line tools, but the idea here is to reduce as much external command dependency as possible.

Thank much!

I think you're missing the point, you'll need to do things differently because it's not a shell script. Most languages have loop statements (for, if/else, while) but you won't have or because they're not shell scripts, in the case of perl and python the scripts are compiled on the fly and then executed.

In both languages you find the mentioned flow controls like if, else, while and such. Both allow to execute system commands and use the results.
in Perl you can say

system("echo 'aha'");

whereas in Python you need the subprocess module to do the same. (I havn't used Python very often so I can't tell in detail)

OP will have to avoid calling system commands or the script won't be cross platform, just use the capabilities of the chosen language.

@Woodsman

It looks like you have a fundamentally wrong idea of what a "real" programming language is like. It's not at all like shell scripting, and there are no "equivalents" for stuff like coreutils commands and shell builtins.

Moved: This thread is more suitable in <Programming> and has been moved accordingly to help your thread/question get the exposure it deserves.

You might have to specify exactly what your script is currently doing and how it's using the features of those coreutils. As I think has been expressed, languages like perl and python provide many robust ways to control the flow of a program written in them, but you typically use the features found in their libraries rather than call shell utilities and therefore you need to know the basics of the language to performs the basics like echo or mv.

For python:
For ls, rm, mv, see:
http://docs.python.org/py3k/library/shutil.html And note the mention of the os library/module/package/whatever.

For cat/tail/head it might be:
http://docs.python.org/py3k/library/io.html See the readline() methods.

For grep/cut/sort/unique you'll need:

• Regular Expressions http://docs.python.org/py3k/library/re.html
• Data types/structures/containers/collections http://docs.python.org/py3k/library/collections.html
• And builtin or handwritten algorithms http://docs.python.org/py3k/library/functions.html See reversed(), sorted(), slice().
  Also see the basic tutorial for fun simple notation for things like slices. http://docs.python.org/py3k/tutorial...ion.html#lists

All of which depends on an understanding of the basics, like flow control.

Lastly, are you tied to perl or python? Because most IT departments would be happier supporting something like a Java app than getting python and tk packaged if you must create a nice GUI and remain crossplatform. The Java Runtime Environment is typically on most Windows machines for regular office builds. The Java documentation's very similarly structured. Java, or rather Oracle, dependency is becoming a concern, but for this small app, you might find it much quicker and easier to create and deploy a Java Swing GUI than something in other languages. But I haven't touched python in quite a while and just remember struggling a bit with tk and wxwindows/wxwidgets for GUIs. Also I have no real experience of perl, but afaik it's made for text manipulation, it's RE syntax is inherited in python and Java iirc.

Just happened across this for python PDF manipulation http://pybrary.net/pyPdf/
For Java you might need http://pdfbox.apache.org/ or possibly can use http://itextpdf.com/
As for perl: http://www.perl.com/pub/2007/09/20/p...with-perl.html

Based on some of the replies, I did not explain myself well. :(

As I mentioned in my last sentence, the idea here is to reduce external command dependencies.

Shell scripts are wonderful in native 'nix environments but are not portable to other operating systems.

If I port a shell script to another language and that port must run on Windows, then I no longer want to depend upon common Linux tools and commands because I can't expect those tools to exist natively in Windows. Yes, there is Cygwin and GNUWin32 tools, but that requires installing that software, which creates external dependencies.

Therefore I am looking for how to perform the same tasks I perform in my shell script without the external dependencies.

Yeah, I know that. I realized a few hours after posting that I should not have even mentioned those requirements. :)

That is part of what I did not explain properly. I am presuming I need to find libraries, functions, and procedures that perform the same results as the external commands used in a shell script. As I am unfamiliar with languages such as Perl, Python, or Java, I don't know how easily those same tasks can be rewritten.

Some preliminary reading indicates that Perl has some built in methods to simulate or perform the equivalent of external commands such as grep and sed. I was hoping I could find a more exhaustive list of these "equivalents."

Python string manipulation
Perl string manipulation

Hi Woodsman,

Perl has many commands adapted from the shell/s and other tools like grep, sed, awk and many others. But Perl often does things on it's own way. Perl-regex for example has become more or less the de facto standard for regular expressions at all (this is not directly related to your question about shell commands).

I've written some Perl-scripts which I use on Windows-servers and these run on my Slackware-machine as well.

to your list from your first post: as mentioned above the loops and conditional statements are not OS-related. For commands like cd or mv you can use the system command.
For "ls" you can chose between the Perl-commands opendir/readdir and the so called "backticks". Other than system the backticks make the ouput of a command available in the Perl-script, an example
will create an array @filesinthisdirectory with all the filenames.

When it comes to compatibility with Windows you'll have to make sure that different systemcommands are handled correct and that in pathes we us "/" in Linux but "\" in Windows. But you can let Perl find out which OS it is and put the various commands in an if-statement.

Markus

Not quite. I.e. 'cd' through 'system' will affect the child process, not the script itself.
...
To the OP: just start patiently reading http://perldoc.perl.org/ -> http://perldoc.perl.org/index-overview.html , and then items one by one in the left column of http://perldoc.perl.org/ .

If you're going to do that, you might as well use a shell script. At least you won't need Perl as a dependency.

https://duckduckgo.com/?q=perl+move+file

Almost all common languages come with functions for cross-platform file system manipulation. And they also usually automatically convert "/" to the OS's native path separator, so you don't even need to worry about that.

well, I just wanted to give an example ;)

That's interesting, I wasn't aware of that, thanks for the information.

Markus

For perl:

Cheers,
Tink

Just to stir the pot ... I used to do everything in perl but I find python a lot cleaner - meaning it's easier to read and maintain.

I'm starting to use Python more frequently because of XML-rpc;
while it's fairly easy to pull everything needed in from CPAN
on my Slackware box, in a corporate environment, running RHEL
on s390 it's a pain in the proverbial. Most of the dependencies
aren't available on EPEL for s390; Python does XML-rpc out of
the box (ancient as the version in RHEL is).

I'm starting to like some of the python ways of doing things ...



Cheers,
Tink

Here are some of the shell equivalents (non-exhaustive) for Python. For os specific tasks, use the os module. (check the docs, not going to link for you here )

1) listing files/directories (ls equivalent)
2) To move/copy files. Use the shutil module. There are also methods like copytree etc similar to cp -r
3) To rename/remove files
4) head/tail/cat etc. All these involve opening file handle, doing something with the file handle, and closing it
5) simple grepping for text files
6) cut equivalent. Basically, cutting is just splitting up text according to fields and processing
7) ftp (use ftp module)
8) telnet (use telnetlib module)

and many more.........

I think I got you, I don't know if you got me.

You really need to clarify if having your users/IT people install perl/python plus tk/wxwindows is going to be ok and look ok, or if you need something easier to work with like Java's Swing. Because you don't want to have to port a half/mostly working solution from perl/python after you've got to the point of trying to make a GUI and finding you/users/IT don't like it.
Does this still stand?

Quicker and easier than a perl or python script?
Definetely not.

Typically such claims indicate lack of Perl knowledge.

Because Perl is notoriously easy to read.

j-ray, I'm talking just to make a GUI or some popup dialogs that have a nice look&feel, and ensuring all the dependencies are set up on the system. JRE means you're good to go. http://download.oracle.com/javase/tu...og.html#create

I remember posting in a different forum a snippet of Python code stolen from here. A very short snippet. People after that were laughing big time about Python readability.

It's the programmer who mostly causes unreadable code.

Seriously, I reject Python first and foremost for lack of lexical scoping. And for lack of arrays. And for whitespaces being used to denote code blocks - and I do indent in Perl/"C".

Come on, perl's nicknamed a write-only language, the rep is for a reason even if in jest.
Kinda hard to write python that's super bad being as how whitespace has meaning especially for flow control. And why have to use curly braces when you'd have been in error without them anyway if you do as you say and indent correctly.

I might need to brush up on lexical scoping and python but I'm not clear where it's lacking sane scoping. http://docs.python.org/py3k/referenc...tionmodel.html

Also what are you on about lack of arrays? You wish to edit lists/sequences in place explicitly aka manage the memory behind them, and permit uninitialised entries?

Well, Larry Wall, the creator of Perl, is a Linguist. Perl is designed to be as context-sensitive as any human language. As in any human language it is necessary to learn the language before one can "see" the context.

This is in my opinion the main difference between Perl and the so called "easier to read" languages like Python or Ruby.

Btw, Perl has never been easy for me, though I'm "learning" it since version 4.

Markus

Python with PyQt4 is just as simple (if not simpler):

EDIT: And it's not just the GUI functions that make it easier to write a program, it's the language itself. Python is much easier to work with than Java.

At the moment:

If you have difficulties reading, don't blame Perl. Or you want to say all those module are written for nothing, nobody uses them, nobody submits patches ?

As I wrote many times here, my first acquaintance with Python happened in 2000, and it was through a very intelligent guy who knew both Perl and Python.

The acquaintance was through the following process: "Here are things I do in Perl this way for this reason - what is the equivalent in Python ?". At that moment for many things there was no equivalent. Furthermore, we took a production data structure occupying ~250MBytes in memory and translated it into Python. Python simply crashed with segmentation fault.

Python has evolved since, I am watching it. Python-3.x.y is quite different from Python-2.x.y - backward compatibility is broken. Opposed to that, Perl developers pay a lot of attention to regression testing.

Python is slowly catching up with Perl, for example: http://en.wikipedia.org/wiki/Function_object#In_Python - closures were introduced in Perl 5 in mid nineties.

...

- how ugly is that "nonlocal" - exactly because of lack of lexical scoping. How ugly is the "return" statement.

In Perl

.

The above code can be rewritten without '$x':

.

Of course, a name can be given in the consuming scope, e.g.

.

And this indicate a lack of Python knowledge. Before you take a dig at other people's preferences, maybe you should take a look at it yourself.

Hey, anyone, the OP is looking for advice how to make his scripts platform independent, not another Python/Perl flamewar.
Both languages are able to achieve what the OP want, no need for that discussions about readability, there is only one person here that can decide which language suits him better: the OP.

Thanks everybody for the responses. I get the big picture that at least for Perl and Python, there are ways to perform the same tasks that are performed in shell scripts. Might require some digging into various external modules, but seems porting an equivalent and functional script to either is possible. That observation likely is true for many languages.

I think we are getting each other, just focusing on different objectives. :)

I don't think that topic was discussed in detail in either thread. I am aware that portability of a script does not mean the script can be run in other environments. When I started this project one of the first questions I asked is what scripting languages the IT people support. I haven't yet received an answer. :( Bottom line is if IT personnel do not support Perl or Python, or refuse to support, then portability is a moot issue. In my other thread I addressed that I might have to port the shell script to VB Script because that is a native scripting language in Windows. That option remains open because although I installed Cygwin in my testing system to run my shell script, and I created two desktop shortcuts so the non computer people don't have to deal with the terminal or command line, I still don't yet have permission to implement that solution. :(

Yes, but that statement was in another thread and the topic in this thread is different although somewhat related :). User skills drive these decisions. You and I might dislike how typical end-users are treated with respect to learning new computer skills, but after three decades I have seen no general change in that attitude. Users will learn new apps but only as much as they need to get by. Most will not dig deeper into an app or process. The people I have worked with through the years are smart people, they just don't care to become more skilled with computers than they are interested in changing the oil on their car or repairing a leaky faucet. Likewise with terminals and the command line. Ain't gonna happen with most of these people. They simply have different priorities in life. These people are all born and bred in a Windows environment and that is what they use at home. Point-and-click. Using a terminal is a horrifying idea to them. :)

Oh, really ?! How about:


http://docs.python.org/library/array.html :

.

In Perl arrays are built-in, come with language, not with a module.

You probably don't care about what IT people support - Perl can be used under both Linux and Windows without being installed centrally. For Windows have a look at http://strawberryperl.com -> http://strawberryperl.com/download/5...0-portable.zip .

I.e. under Windows you'll need a .bat script that would bootstrap everything as needed.

FWIW Windows Script Host can do Javascript/JScript out of the box as well as VBScript (plus scripting engines for perl, python, etc). And there's Windows PowerShell with a v2 for newer OSs. Also C#/.Net stuff can be crossplatform with Mono iirc.
Something written in perl/python/Java should be able to be almost write once, run anywhere. At least just between Windows and linux on x86 machines.

As a syntax I say that python example with common words is far easier to read than the perl use of characters. Understanding of either does completely depend on knowledge of the language.
Breaking things between language version 2 and 3 doesn't mean they don't pay a lot of attention to regression testing.
I've never quite got the hard-on some people have for closures/anonymous functions/lambdas in imperative languages. Nor do I see a problem with return statements. Seems there's been decades and many successful languages that have things the way you don't prefer, so can we all be so wrong? :)
Yes CPAN is well known as a valuable resource that any language would be blessed to have.
Apparently there's no written spec or standard for perl prior to 6, which isn't ready yet. ;)
As for ~250MB data structures in python, I'd say it's possibly not really the best tool for that job, and that there's the C API/bindings of the mainstream implementation CPython. Also, EVE Online's python/stackless python.
I don't think anyone's going to disagree that no language is perfect, some aim for more practical features, others to stick to more abstract or theoretical ideals of computation expression. Always good to check you're up to date on the state of different tools, sorry for any derailment OP.

Let us know if/when you get those firm answers about language options and user interaction and if you still need help.

Edit: Sergei, is there really an issue about a feature being in the language itself vs the standard library?
And the OP may very well care what IT say even if they can be technically sidestepped, for support and legal reasons.

Anonymity means less broken naming conventions and names contention. As I wrote, the consuming scope, i.e. the end user decides on names, so developers of "modules" (not Perl modules, just reusable pieces of code) don't care about each others naming decisions.

Closures is a fundamental expression of stateful (opposed to stateless) code. All those fancy OO data class members are just variables at outer scopes of closures.

I came to all this not at school, but through practice - luckily, met a couple of younger guys who suggested to try, and after trying I realized I really liked them - they made life easier. Also, not only my life. When, say, 20 people on project need to write configuration files, and I work as an integrator, my own interest is to make sure that by construction there are no contentions.

I think that they're called "lists" in Python. And AFAIK they are part of the language, they even have their own syntax:

A lot of problems can be made a lot simpler with closures and first-class functions. I really wish that more common languages would support them. And I really wonder why they don't, I can't believe that it's that difficult. I even implemented them in my own working interpreter, and it wasn't really complicated.

I care in as much that ignoring IT policies could lead to termination. :)

If you mean a Perl script can be compiled into a stand-alone app, then that is good to know. I notice Strawberry Perl is available as a portable app. I also notice something called Perl2Exe. Yet even in those cases, installing software in an enterprise network could lead to termination. I don't want this thread to be about office politics, but cooperating with IT personnel is presumed in these discussions. :)

Bear in mind too that if money is involved then the IT folks become a party in the solution because such software needs evaluation and approval. Popular scripting languages are free/libre software and potentially avoid much of that red tape.

With that said, generally I prefer the principle of "seeking forgiveness rather than asking permission." :) One method to bypass locked systems is a USB flash drive. I have suggested that option if the IT people refuse to let us install Cygwin --- we can run the scripts from a USB stick.

Another option is to install to a shared network directory that belongs to the group. Generally IT people let users in such groups do as they please inside those directories. I have suggested that option too.

Another option is to run the Cygwin environment and scripts from user's home directories, although that consumes more network drive space than a common installation.

All of these work-arounds require scripts to know path locations for any such unique installation. As long as any such work-around does not introduce viruses, spyware, or anything else that raises the ire of the IT people, then likely there are no repercussions and everybody lives happily ever after. :)

I am aware that Windows Script Host supports JavaScript, but if I am cornered into supporting a native Windows method then I'll have to stick with VB Script. I have no experience with JavaScript. Although I'm rusty after several years I have sufficient experience with the similar Visual Basic for Applications (VBA). I don't know whether PowerShell is an option. As I mentioned previously I have yet to receive a reply about which scripting options these folks support. From what I have read thus far, porting my shell script to PowerShell will be much work. Like Perl and Python, I have to learn PowerShell syntax from scratch, whereas VB Script will be more or less familiar.

Yet even then, my script uses external tools such as awk and sed. I would have to simulate those external calls as internal functions or find a way to run the GNUWin32 equivalent of those tools from within any script. So once again I'm back to installing software unless I can perform everything internally in the ported script. If installing software is approved, then I'll just stick with the Cygwin software.

With respect to this thread, I basically wanted to know how much is involved to port a shell script to a different scripting language on a different operating system. Doable, but lots of work. :) With respect to my other thread, I have a solution in place (Cygwin), but now wait for approval.

No, I don't mean that "a Perl script can be compiled into a stand-alone app". I mean that portable Perl does not require installation in central places, but that it can be put into any directory an run from there, and one doesn't even need to have Administrator privileges to do that.

Whatever 'awk' and 'sed' do can be done by Perl built-in functions without spawning child processes.

Yes I thought we'd covered that almost everything you're using common utilities for can be expressed within each programming language. And probably not much work, generally you just have a little more declaration of variables and flow control, a little more verbosity about which features/functions you're using.

I'm not getting why something in e.g. Java would care how one has named one's methods and variables within a useful library any more than they would if it were in perl. Pass the arguments as however you've named them, be oblivious to how the method's implemented, assign the result as you wish. The only thing seems to be the name of the method, and that's down to having a sane creator of the library you're using. As for naming contention, namespaces. If you really want to rename the method, subclass, wrapper method with the name you want just passing the arguments to the other method, done?
In a similar vein, Java seems fine with interfaces and anonymous classes whenever anonymous functions are needed.
There must be some class of algorithm that I'm just not used to, that works best with passing function references around, rather than just object references to implementing class instances.

I still can only think that Sergei's talk of arrays in python is the lack of control around consecutive memory locations, to perform in-place inserts/sorts in the most manual and efficient manner. Beyond that, afaik python gives you several sequential mutable data types that you can place empty or null values in, but just can't micromanage the memory for behind the scenes. Well, unless you subclass them iirc.

I don't know anywhere that's got VB of any flavour as an Invest tech, thought I'm sure there's some. I'd strong suggest you at least look at C# if you're going with anything using the .Net framework & libraries.
I have little experience of it, but Cygwin sounds a rather hefty solution to your problem, though as you say you can leave your script as is, calling those common separate utilities.

Translate this:


into Java - the code returns an anonymous code reference and does not pollute global namespace. The code is used like this (assuming the code resides in 'function.prl' file) :

Pay attention that there are two variables outside the function body: '$_number_of_retries', '$_network_timeout', but still they can not have a contention with other variables with the same name - because of Perl scoping rules.

This is a piece of code I'm using in my tool building stuff from sources.

What global namespace is there to pollute? Stuff in Java is always in a class, constants and methods are imported and references to them get qualified as required if there's contention.

Your 2 variables outside the function would probably be object variables instantiated for each independant use of this method that you'd need, or just static variables for the class the methods are defined in. I don't see the problem your syntax is solving, perhaps Java doesn't have it to start with, being OO the way it is and compiled to bytecode.

Again, define this class as implementing whatever interface your code with the $f equiv needs, and you can pass it an object instance that has the one or more methods, with you naming the reference as you wish. Not seeing the big win here. No you can't pass completely arbitrary objects around (well without deeming them simply of type Object and later casting them back to classes/interfaces you know of), but that's stopping you mess something up, instead you define the interface aka method signatures the passed object must implement, and beyond that people are free to implement those as they wish, using whatever variable and private method naming schemes, and your caller need know or care nothing about them, while the compiler checks at compile time that you've not passed the wrong reference.

If you want to get more into arbitrary method calls, Java has reflection, you can list an object's method names/signatures as text and then build a call to them dynamically afaik. Couples in with web services and other remote method invocation iirc.

I'm no expert, I already said no language is perfect, but I don't see how perl's solving a problem that Java's stumped by here. And strong typing and compiler-time checks are good.

Gotcha. Class names are global. I.e. using class names causes a headache of having their names unique.

And if you are going to tell me there are namespaces on top of classes, I'll tell you that namespaces are global, i.e. using namespaces causes a headache of having their names unique.

Languages without anonymity are defective by design.

@Proud

This is slightly unrelated, but still relavent:

One thing I don't like about many OOP languages without first class functions if that they force OOP down your throat when first-class functions would make it so much easier.

For example, in an OOP language, if you wanted to make a customizable sorting function, you would have to define a class (that pollutes the global namespace) that has a comparison method, create an instance of it, and pass it to your sorting function.

In a language with first-class functions, you just pass an anonymous function, like this:

Isn't it far simpler and more elegant?

EDIT: If the language supports closures, another advantage is that the sorting function can use variables in the scope in which it was created. Try doing that in Java.

This has nothing to do with anonymity. OCaml, for example, is a strictly typed languages - much stricter than Java, OCaml doesn't even have RTTI if any kind, but it still has anonymous functions and lexical scoping. And no 'return' statement - everything is an expression.

---------- Post added 08-22-11 at 12:45 AM ----------

Oh, and Java-7 finally has closures (IIRC).

And Scala is functional - there should be a way of modernizing Java world.

Okay. That matches my comments about installing to a USB stick, or user home or group network directory.

Yes, I think we pretty much have established that. I was just being verbose. :)

OP: apologies, I didn't mean to get your thread hijacked by a language war

If you are too forgetful, let me bring you back in time to what you said:
You said "lack of arrays" is one of the reasons you reject Python. But arrays are supported in Python, both as built in lists (1D) , and a thin wrapper module (array) around C arrays. You did not specifically mention that you reject Python because it doesn't come built in like that in Perl. And you quickly change your argument in this post. And, if people finds it easier to maintain and read, then its their business. You have no reason to sound arrogant and insult their intelligence. Not sure what I am talking about? See post #20

Perl and Python are not the only ones. There's also Ruby.

Let me tell you one piece of advice. I am a Linux Sys Admin and I used to only shell script. Me and another guy I went in took it upon ourselves to "step our game up." Him with perl, me with python. Let me tell you what, within 1 month, both of us were making "scripts" that out performed a shell script by far. Think on, combining awk,sed and bash together with phenomenal error handling and full lists and dictionaries with direct access to shell commands and there results similar to $?. MUCH more readable code (python, not Perl), with the availability of an entire test sweet "IDLE" to test all lines of code as you develop it. You cant go wrong!!! By the way, I stomped ol' perl boy as he continually tried to find "There's more than one way to do it..."

Because of it I have received praise on the error handling and power of Python's simplicity over Perl/ksh/bash/awk/sed scripting!

BTW im not saying use python. Perl is a awesome and powerful language to replace shell scripting with. Anything "upper" language is better than old school shell scripting.

...or PHP

I'd like to mention that any programmer can write perl code that is readable for C-, Java-, PHP-programmers without much difficulties.

why do you even suggest that? PHP is a bloat, too many similar functions, no namespaces (until recently? ) ... and whole lot more problems..