I found an example that uses a Perl one-liner to remove duplicate entries in an authorized_keys file. However as you can see below the example is set up as an alias.

How do you run this one liner against a file without using an alias?

Thanks,
Gary

perl -i -ne 'script' filename
\$* belongs to the alias, that should be dropped....

I'm not convinced "-i" should be considered here.

perl -ne 'print $_ unless $a{$_}++' authorized_keys

You don't need to escape the $'s because they're inside the single quote. Equivalently, you could do:

perl -ne "print \$_ unless \$a{\$_}++" authorized_keys

It could be just as simple as the ubiquitous sort | uniq . . .

After all, this is a text-file, and what you want is to eliminate the duplicates from it. Therefore, first sort the file, then pass it through a filter which removes consecutive duplicate records. Direct that to a new file. Then, swap the new file for the old one. Q.E.D.

Thanks everyone. OP here. I've been kind of slammed and haven't been able to test but these look good.

Very new to Perl and trying to come up with something similar. I am getting dups in the authorized_keys file. Looking for a simple way to just flag there are dups or even possibly remove them. What was the final solution on this thread?

Brent

There are two solutions presented - one of them perl.
If you want to learn, look at post #4 - if you aren't confident look at post #5.

In either case better to write to a new file and replace the original manually when convinced.

Thanks for the reply. I guess my problem is slightly different than the one listed, but very similar. I have users recreating their keys and copying their new public key from the primary host to the remote host authorized_key file. So the two keys are not identical, but the signature of the user@hostname is and it causes the authentication to fail.

So what I really need to do is look for duplicate user@hostname, not the entire key as the new key is different. Only the last part of each key is duplicate.

any idea how to look for dups on just part of the file? See the example below.

Brent

>cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvJPhWjzLlxLrPW94T86a/JRvP+HHWjIfERQUj8eBm3sg tHNyfMZLtiLfMGe9d/zGuFHVzm3R7g1Pv+88ZFQjZ0AjRZal+2NXtEQovVs88UjDdaZBXEO3VPWzktER URkkrInSk7LKvMXHOjIggzG+O1tiExJyKjXTxuRhTpsm2kSd6mXhkowh0wCIpG57GTGviCdC7Ut1uwKN VHBYBq0JNNblYf09JuA7vNzWjaJjSnSt8bLCvyIgzsAyLp6igM1urxXNepNwCvqwHXTEES0adLP2yLm+ tA/IPU17zDMX7DUqcDbxk+R0txgtoW/M1GcJX7n2gep1Mdwp1Ft40m7RfQ== brent@XYZSERVR
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAty/iV4YP1RhqZ0dL/HgFzma8wJwiWf8S1SMtLVvV5k4s wayDBZR8v0lVMeUXIMPcR6kL98KQAT9oxpBAxKDRl1HHBfddHu97jNpNl7RgFkPSF/jRkTBaY3O1iRYV SeOBKDOuoGc/JQuyFnZj/Rg4V1OHvFH2uojb9aHZcr+4F7yfqupdOTinf50UIXm809e5u6UH6hpm3YPE 60JcdFzEMtK7yHur7nB0onEDFo8SYXzNRUpVdv5WAJ521YYdlbURhJleS6lwyPLGwz/9mxaSm7ry8M3S o3aXrRKG74LdRFVVUbsoLKXI697xEw3BhXD6XaJscouloIlmuod4pA9rFw== brent@XYZSERVR

Several.
These days I'd use awk rather than perl - it splits into fields based on whitespace, so the string you are looking for would be the last field - which awk also gives you a variable for.
Could also be done easily in bash using parameter substitution.

Try thisSuccinct and clean.
awk is (also) well worth studying.

That's a great idea, but I've been told, "Though shall write it in Perl". Considering calling awk from Perl though?

you need to improve the original perl script. Have you tried that?

as sundial said, 'sort | uniq' and its done.

or try painting your house with a toothbrush.