Core utilities/Tips and tricks
See Core utilities for the main article.
grep
Colored output
grep
's color output can be helpful for learning regexp and additional grep
functionality.
To enable grep coloring write the following entry to the shell configuration file (e.g. if using Bash):
~/.bashrc
alias grep='grep --color=auto'
To include file line numbers in the output, add the option -n
to the line.
The environment variable GREP_COLOR
can be used to define the default highlight color (the default is red). To change the color find the ANSI escape sequence for the color liked and add it:
export GREP_COLOR="1;32"
GREP_COLORS
may be used to define specific searches.
less
Colored output through environment variables
Add the following lines to your shell configuration file:
~/.bashrc
export LESS=-R export LESS_TERMCAP_mb=$'\E[1;31m' export LESS_TERMCAP_md=$'\E[1;36m' export LESS_TERMCAP_me=$'\E[0m' export LESS_TERMCAP_se=$'\E[0m' export LESS_TERMCAP_so=$'\E[01;44;33m' export LESS_TERMCAP_ue=$'\E[0m' export LESS_TERMCAP_us=$'\E[1;32m'
Change the values (ANSI escape code) as you like.
Colored output through wrappers
You can enable code syntax coloring in less. First, install source-highlight, then add these lines to your shell configuration file:
~/.bashrc
export LESSOPEN="| /usr/bin/source-highlight-esc.sh %s" export LESS='-R '
Frequent users of the command line interface might want to install lesspipe.
Users may now list the compressed files inside of an archive using their pager:
$ less compressed_file.tar.gz
==> use tar_file:contained_file to view a file in the archive -rw------- username/group 695 2008-01-04 19:24 compressed_file/content1 -rw------- username/group 43 2007-11-07 11:17 compressed_file/content2 compressed_file.tar.gz (END)
lesspipe also grants less the ability of interfacing with files other than archives, serving as an alternative for the specific command associated for that file-type (such as viewing HTML via python-html2text).
Re-login after installing lesspipe in order to activate it, or source /etc/profile.d/lesspipe.sh
.
Vim as alternative pager
Vim (visual editor improved) has a script to view the content of text files, compressed files, binaries, directories. Add the following line to your shell configuration file to use it as a pager:
~/.bashrc
alias less='/usr/share/vim/vim74/macros/less.sh'
There is also an alternative to less.sh macro, which may work as the PAGER
environment variable. Install vimpager and add the following to your shell configuration file:
~/.bashrc
export PAGER='vimpager' alias less=$PAGER
Now programs that use the PAGER
environment variable, like git, will use vim as pager.
Colored output when reading from stdin
When you run a command and pipe its standard output (stdout) to less for a paged view (e.g. pacman -Qe | less
), you may find that the output is no longer colored. This is usually because the program tries to detect if its stdout is an interactive terminal, in which case it prints colored text, and otherwise prints uncolored text. This is good behaviour when you want to redirect stdout to a file, e.g. pacman -Qe > pkglst-backup.txt
, but less suited when you want to view output in less
.
Some programs provide an option to disable the interactive tty detection:
# dmesg --color=always | less
In case that the program does not provide any similar option, it is possible to trick the program into thinking its stdout is an interactive terminal with the following utilities:
- stdoutisatty — A small program which catches the
isatty
function call.
- https://github.com/lilydjwg/stdoutisatty. || stdoutisatty-gitAUR
- Example:
stdoutisatty program | less
- unbuffer — A tclsh script comes with expect, it invokes desired program within a pty.
- http://expect.sourceforge.net/example/unbuffer.man.html || expect
- Example:
unbuffer program | less
Alternatively, using zpty module from zsh: [1]
~/.zshrc
zmodload zsh/zpty pty() { zpty pty-${UID} ${1+$@} if [[ ! -t 1 ]];then setopt local_traps trap '' INT fi zpty -r pty-${UID} zpty -d pty-${UID} } ptyless() { pty $@ | less }
Usage:
$ ptyless program
To pipe it to other pager (less in this example):
$ pty program | less
ls
Colored output
Colored output can be enabled with a simple alias. File ~/.bashrc
should already have the following entry copied from /etc/skel/.bashrc
:
alias ls='ls --color=auto'
The next step will further enhance the colored ls output; for example, broken (orphan) symlinks will start showing in a red hue. Add the following to your shell configuration file:
eval $(dircolors -b)
tar
Backup with parallel compression
To back up using parallel compression (SMP), use pbzip2 (Parallel bzip2).
First back up the files to a plain tarball with no compression:
# tar -cvf /destionation_path/etc-backup.tar /etc
Then use pbzip2 to compress it in parallel:
$ pbzip2 /path/to/chosen/directory/etc-backup.tar.bz2
Store etc-backup.tar.bz2
on one or more offline media, such as a USB stick, external hard drive, or CD-R. Occasionally verify the integrity of the backup process by comparing original files and directories with their backups. Possibly maintain a list of hashes of the backed up files to make the comparison quicker.
Restore corrupted /etc
files by extracting the etc-backup.tar.bz2
file in a temporary working directory, and copying over individual files and directories as needed. To restore the entire /etc
directory with all its contents execute the following command as root:
tar -xvjf etc-backup.tar.bz2 -C /
Restore name of a tar.gz archive
Here describes about how you can restore normal names for the most of a recovered tar.gz archives by photorec. First you need to create a file with a more detailed information about files, see: post recovery tasks.
Restored tar.gz files with photorec might look like ./recup_dir.996/f864593944_wmmaiload-1.0.5.tar.gz
or ./recup_dir.996/f864589184.tar.gz
.
The tar.gz archive includes inside a tar archive whose name is used to restore filename of the tar.gz if it is possible or name of a first folder/file from inside of the archive.
The ways that might be used to restore the name of the tar.gz archives:
- From part of a name by cutting everything before '_' in files with created by photorec with pattern: f864593944_wmmaiload-1.0.5.tar.gz
- From name of a tar inside of the tar.gz archive.
- From the name of the folder inside it. In many cases it usually compressed a whole folder that might be similar to the archive name.
Collect info data about tar.gz
Collecting needed data from inside of the archives in order to restore their names.
collect-info-about-tar-gz.sh
#!/bin/bash FileName="$1" IntCheck=$(7z t "$1" | grep ^"Everything is Ok"$ ) AddIt="False" if [ 'XX'"$IntCheck" != 'XXEverything is Ok' ]; then ErrorFlag="Damaged";else ErrorFlag='OK';fi; if [ 'XX'"$(echo "$FileName" | grep -i tar.gz$ )" != 'XX' ];then AddIt='True' BaseFileName=$(tar -tvf "$FileName" | head -1 | awk '{ print substr($0, index($0,$6)) }' ) OutOfOrig="$( echo "$1" | awk '{ print substr($0, index($0,$4)) }' )" if [ 'XX'"$(echo "${BaseFileName}" | grep '/')" == 'XX' ]; then BaseFileName="$(echo "${BaseFileName}"| sed 's/\///m')" fi fi if [ $AddIt == 'True' ]; then if [ 'XX'"${OutOfOrig}" != 'XX' ]; then IfOrig=$(echo "$OutOfOrig" | grep -v '/') ;fi if [ 'XX'"${IfOrig}" != 'XX' ]; then BaseFileName="$IfOrig" ; fi BF="$(echo "$BaseFileName" | sed 's/\///g' | sed 's/^\.//m' )" echo "$1"'|'"${BF}"'.tar.gz''|'Errors:'|'$ErrorFlag'|' >> /tmp/tmpDB.txt fi;
Run find -type f -name *.tar.gz -exec collect-info-about-tar-gz.sh "{}"\;
Restore filenames
This will restore file names based on collected ínfo about files.
restore-tar-gz-names.sh
#!/bin/bash CountAll=0 Destination="$HOME/tar-gz" mkdir -v "$(echo $Destination)" -p ArrayFillCount=0; while read line ; do ArrayFillCount=$((ArrayFillCount+1)) ArrayOfFiles[$ArrayFillCount]=$line done < /tmp/tmpDB.txt; XX=${#ArrayOfFiles[@]} echo $XX while [ $XX != $CountAll ] ; do FolderName="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'|' '{print $2}' | cut -c1-4 | tr '[:lower:]' '[:upper:]' )" PathToFile="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'|' '{print $1}')" FileName="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'|' '{print $2}')" #SourceFolder="$(echo $PathToFile | sed s/$(basename $PathToFile)//m )" SourceFolder='./Sorted/' if [ "$(echo $FolderName)" == '.TAR' ];then FileName="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'_' '{ print substr($0, index($0,$4))}'|awk -F'|' '{print $1}' )" FolderName="$(echo $FileName | cut -c1-4 | tr '[:lower:]' '[:upper:]' )" fi if [ -f "$FileName" ]; then FIX="$(echo $PathToFile | sed s/$(basename $PathToFile)//m )" BadName="$(echo $PathToFile | awk -F"$FIX" '{print $2}' )" if [ -d "${Destination}/BadName" ];then echo A > /dev/null;else mkdir "${Destination}/BadName";fi cp -fv "${PathToFile}" "${Destination}"'/'BadName'/'"$BadName" else IfExist="$(echo $Destination/${SourceFolder}${FolderName})" if [ -d "$IfExist" ]; then echo a > /dev/null;else mkdir -vp "$(echo $Destination/${SourceFolder}${FolderName})"; fi if [ -f "$Destination/${SourceFolder}${FolderName}"'/'"$FileName" ]; then DupName="${FileName}"_Duplicate"${CountAll}"; IfExist="$(echo $Destination/Duples)" if [ -d "$IfExist" ]; then echo a > /dev/null; cp -fv "${PathToFile}" "$Destination/Duples"'/'"$DupName"; else mkdir -vp "$(echo $Destination/Duples)"; cp -fv "${PathToFile}" "$Destination/Duples"'/'"$DupName";fi else cp -fv "${PathToFile}" "$Destination/${SourceFolder}${FolderName}"'/'"$FileName" fi fi CountAll=$((CountAll+1)) echo $CountAll done; echo $XX
Files will be restored with pattern like $HOME/tar-gz/Sorted/MIXE
where MIXE is the first 4 letters of a filename. You can adjust it with the cut -c1-4
command inside the script.
In the $HOME/tar-gz/BadName
places damaged files or files where it was impossible to get filename.
In the $HOME/tar-gz/Duples
places duplicates of files with pattern filename.tar.gz_Duplicate123 where 123 is a number of a processed file.