29 Jan 2014

Linux Command line tips and Bash stuff

Linux Command Line Tips and Many More

Here is a selection of command-line tips that I've found useful when working on Linux. The emphasis is on somewhat less-known techniques that are generally important or useful to technical users. It's a bit long, and users certainly don't need to know all of them, but I've done my best to review that each item is worth reading in terms of projected time savings, if you use Linux heavily.

To get more information on a command mentioned, first try "man <command name>". In some cases, you must install a package for this to work -- try aptitude or yum. If that fails, Google it.

Basics

  • Learn basic Bash. Actually, read the whole bash man page; it's pretty easy to follow and not that long. Alternate shells can be nice, but bash is powerful and always available (learning mainly zsh or tcsh restricts you in many situations).
  • Learn vim. There's really no competition for random Linux editing (even if you use Emacs or Eclipse most of the time).
  • Know ssh, and the basics of passwordless authentication, via ssh-agent, ssh-add, etc.
  • Be familiar with bash job management: &, Ctrl-Z, Ctrl-C, jobs, fg, bg, kill, etc.
  • Basic file management: ls and ls -l (in particular, learn what every column in "ls -l" means), less, head, tail and tail -f, ln and ln -s (learn the differences and advantages of hard versus soft links), chown, chmod, du (for a quick summary of disk usage: du -sk *), df, mount.
  • Basic network management: ip or ifconfig, dig.
  • Know regular expressions well, and the various flags to grep/egrep. The -o, -A, and -B options are worth knowing.
  • Learn to use apt-get or yum (depending on distro) to find and install packages.


Everyday use

  • In bash, use Ctrl-R to search through command history.
  • In bash, use Ctrl-W to kill the last word, and Ctrl-U to kill the line. See man readline for default keybindings in bash. There are a lot. For example Alt-. cycles through prevous arguments, and Alt-* expands a glob.
  • To go back to the previous working directory: cd -
  • If you are halfway through typing a command but change your mind, hit Alt-# to add a # at the beginning and enter it as a comment (or use Ctrl-A, #, enter). You can then return to it later via command history.
  • Use xargs (or parallel). It's very powerful. Note you can control how many items execute per line (-L) as well as parallelism (-P). If you're not sure if it'll do the right thing, use xargs echo first. Also, -I{} is handy. Examples
                                   find . -name \*.py | xargs grep some_function
                                   cat hosts | xargs -I{} ssh root@{} hostname
  • pstree -p is a helpful display of the process tree.
  • Use pgrep and pkill to find or signal processes by name (-f is helpful).
  • Know the various signals you can send processes. For example, to suspend a process, use kill -STOP [pid].  For the full list, see man 7 signal
  • Use nohup or disown if you want a background process to keep running forever.
  • Check what processes are listening via netstat -lntp. See also lsof.
  • In bash scripts, use set -x for debugging output. Use set -e to abort on errors. Consider using set -o pipefail as well, to be strict about errors (though this topic is a bit subtle). For more involved scripts, also use trap.
  • In bash scripts, subshells (written with parentheses) are convenient ways to group commands. A common example is to temporarily move to a different working directory, e.g.
                                        # do something in current dir
                                             (cd /some/other/dir; other-command)
                                        # continue in original dir
  • In bash, note there are lots of kinds of variable expansion. Checking a variable exists: ${name:?error message}. For example, if a bash script requires a single argument, just write input_file=${1:?usage: $0 input_file}. Arithmetic expansion: i=$(( (i + 1) % 5 )). Sequences: {1..10}. Trimming of strings: ${var%suffix} and ${var#prefix}. For example if var=foo.pdf, then echo ${var%.pdf}.txt prints "foo.txt".
  • The output of a command can be treated like a file via <(some command). For example, compare local /etc/hosts with a remote one: diff /etc/hosts <(ssh somehost cat /etc/hosts)
  • Know about "here documents" in bash, as in cat <<EOF ....
  • In bash, redirect both standard output and standard error via: some-command >logfile 2>&1. Often, to ensure a command does not leave an open file handle to standard input, tying it to the terminal you are in, it is also good practice to add "</dev/null".
  • Use man ascii for a good ASCII table, with hex and decimal values.
  • On remote ssh sessions, use screen or dtach to save your session, in case it is interrupted.
  • In ssh, knowing how to port tunnel with -L or -D (and occasionally -R) is useful, e.g. to access web sites from a remote server.
  • It can be useful to make a few optimizations to your ssh configuration; for example, this .ssh/config contains settings to avoid dropped connections in certain network environments, not require confirmation connecting to new hosts, forward authentication, and use compression (which is helpful with scp over low-bandwidth connections):
    • TCPKeepAlive=yes
    • ServerAliveInterval=15
    • ServerAliveCountMax=6
    • StrictHostKeyChecking=no
    • Compression=yes
    • ForwardAgent=yes

Data processing

  • To convert HTML to text: lynx -dump -stdin
  • If you must handle XML, xmlstarlet is good.
  • For Amazon S3, s3cmd is convenient (albeit immature, with occasional misfeatures).
  • Know about sort and uniq (including uniq's -u and -d options).
  • Know about cut, paste, and join to manipulate text files. Many people use cut but forget about join.
  • It is remarkably helpful sometimes that you can do set intersection, union, and difference of text files via sort/uniq. Suppose a and b are text files that are already uniqued. This is fast, and works on files of arbitrary size, up to many gigabytes. (Sort is not limited by memory, though you may need to use the -T option if /tmp is on a small root partition.)

                                 cat a b | sort | uniq > c   # c is a union b
                                 cat a b | sort | uniq -d > c   # c is a intersect b
                                 cat a b b | sort | uniq -u > c   # c is set difference a - b

  • Know that locale affects a lot of command line tools, including sorting order and performance. Most Linux installations will set LANG or other locale variables to a local setting like US English. This can make sort or other commands run many times slower. (Note that even if you use UTF-8 text, you can safely sort by ASCII order for many purposes.) To disable slow i18n routines and use traditional byte-based sort order, use export LC_ALL=C (in fact, consider putting this in your .bashrc).
  • Know basic awk and sed for simple data munging. For example, summing all numbers in the third column of a text file: awk '{ x += $3 } END { print x }'. This is probably 3X faster and 3X shorter than equivalent Python.
  • To replace all occurrences of a string in place, in files:
                               perl -pi.bak -e 's/old-string/new-string/g' my-files-*.txt
  • Use shuf to shuffle or select random lines from a file.
  • Know sort's options. Know how keys work (-t and -k). In particular, watch out that you need to write -k1,1 to sort by only the first field; -k1 means sort according to the whole line.
  • Stable sort (sort -s) can be useful. For example, to sort first by field 2, then secondarily by field 1, you can use sort -k1,1 | sort -s -k2,2
  • If you ever need to write a tab literal in a command line in bash (e.g. for the -t argument to sort), press Ctrl-V <tab> or write $'\t' (the latter is better as you can copy/paste it).
  • For binary files, use hd for simple hex dumps and bvi for binary editing.
  • Also for binary files, strings (plus grep, etc.) lets you find bits of text.
  • To convert text encodings, try iconv. Or uconv for more advanced use; it supports some advanced Unicode things. For example, this command lowercases and removes all accents (by expanding and dropping them):

       uconv -f utf-8 -t utf-8 -x '::Any-Lower; ::Any-NFD; [:Nonspacing Mark:] >; ::Any-NFC; ' < input.txt > output.txt

  • To split files into pieces, see split (to split by size) and csplit (to split by a pattern).


System debugging

  • For web debugging, curl and curl -I are handy, and/or their wget equivalents.
  • To know disk/cpu/network status, use iostat, netstat, top (or the better htop), and (especially) dstat. Good for getting a quick idea of what's happening on a system.
  • To know memory status, run and understand the output of free and vmstat. In particular, be aware the "cached" value is memory held by the Linux kernel as file cache, so effectively counts toward the "free" value.
  • Java system debugging is a different kettle of fish, but a simple trick on Sun's and some other JVMs is that you can run kill -3 <pid> and a full stack trace and heap summary (including generational garbage collection details, which can be highly informative) will be dumped to stderr/logs.
  • Use mtr as a better traceroute, to identify network issues.
  • For looking at why a disk is full, ncdu saves time over the usual commands like "du -sk *".
  • To find which socket or process is using bandwidth, try iftop or nethogs.
  • The ab tool (comes with Apache) is helpful for quick-and-dirty checking of web server performance. For more complex load testing, try siege.
  • For more serious network debugging, wireshark or tshark.
  • Know strace and ltrace. These can be helpful if a program is failing, hanging, or crashing, and you don't know why, or if you want to get a general idea of performance. Note the profiling option (-c), and the ability to attach to a running process (-p).
  • Know about ldd to check shared libraries etc.
  • Know how to connect to a running process with gdb and get its stack traces.
  • Use /proc. It's amazingly helpful sometimes when debugging live problems. Examples: /proc/cpuinfo, /proc/xxx/cwd, /proc/xxx/exe, /proc/xxx/fd/, /proc/xxx/smaps.
  • When debugging why something went wrong in the past, sar can be very helpful. It shows historic statistics on CPU, memory, network, etc.
  • For deeper systems and performance analyses, look at stap (systemtap) and perf.
  • Use dmesg whenever something's acting really funny (it could be hardware or driver issues).

16 Jan 2014

BASH TRICKS



Here are some awesome bash trick which will make you more productive on bash:

History expansion
    !! = previous command
    !$ = last word of previous command
    !-n = nth previous command
    !#$ = last word of current line
    !<start of command> will execute the command from history starting with letters after "!"

Brace Expansion:
    {a..b} = numbers a to b in order.
    {a,b,c} = words a, b, c. Useful for paths: touch /tmp/{foo,bar,baz}

Parameter Expansion:
Suppose that foo=/usr/local/blah.txt
    ${variable#word} = removes word from the beginning of variable. For example, ${foo#*/} = usr/local/blah.txt
    ${variable##word} = same thing, but removes longest pattern matching word. For example, ${foo##*/} = blah.txt
    ${variable%word} = removes word from end of variable. For example: ${foo%.txt} = /usr/local/blah
    ${variable%%word} = same thing but longest matching suffix

Process Substitution:
    <(command) = treats the output of command as a file. diff -u <(ssh web{1,2} cat /etc/passwd)) shows you a           unified diff between /etc/passwd on web1 and 2

Basically, the entire bash manpage is useful and awesome. Enjoy


Linuxskool thanks Daniel Chan for spreading words of trick.


3 Dec 2013

Simple Python WebServer: Share/View your Linux Filesystem files over Http Server

A quick tip for Linux lovers.

Python allows you to start a quick http server by a single command.


Suppose you want to share files and directory of current working directory over http. No need to set up a complete web server for that. Python  can do this magic in a Single command.

 [root@clone-1 /]# python -m SimpleHTTPServer

Serving HTTP on 0.0.0.0 port 8000 ...
localhost.localdomain - - [02/Dec/2013 10:15:18] "GET / HTTP/1.1" 200 -
localhost.localdomain - - [02/Dec/2013 10:15:22] "GET /boot/ HTTP/1.1" 200 -
localhost.localdomain - - [02/Dec/2013 10:15:24] "GET /boot/symvers-2.6.18-164.el5.gz HTTP/1.1" 200 -
localhost.localdomain - - [02/Dec/2013 10:15:32] "GET /home/ HTTP/1.1" 200 -
localhost.localdomain - - [02/Dec/2013 10:15:34] "GET /home/yogi/ HTTP/1.1" 200 -
192.168.91.1 - - [02/Dec/2013 10:20:12] "GET / HTTP/1.1" 200 -
192.168.91.1 - - [02/Dec/2013 10:20:12] code 404, message File not found
192.168.91.1 - - [02/Dec/2013 10:20:12] "GET /favicon.ico HTTP/1.1" 404 -
192.168.91.1 - - [02/Dec/2013 10:20:14] "GET /etc/ HTTP/1.1" 200 -
192.168.91.1 - - [02/Dec/2013 10:20:17] "GET /etc/auto.misc HTTP/1.1" 200 -
192.168.91.1 - - [02/Dec/2013 10:20:27] "GET /etc/modprobe.d/ HTTP/1.1" 200 -
192.168.91.1 - - [02/Dec/2013 10:20:29] "GET /etc/modprobe.d/blacklist HTTP/1.1" 200 -

This one-liner starts a web server on port 8000 with the contents of current directory on all the interfaces (address 0.0.0.0), not just localhost. If you have "index.html" or "index.htm" files, it will serve those, otherwise it will list the contents of the currently working directory.
It works because python comes with a standard module called SimpleHTTPServer. The -m argument makes python to search for a module named SimpleHTTPServer.py in all the possible system locations (listed in sys.path and $PYTHONPATH shell variable). Once found, it executes it as a script. If you look at the source code of this module, you'll find that this module tests if it's run as a script if __name__ == '__main__', and if it is, it runs the test() method that makes it run a web server in the current directory.

If you are not sure that on which specific port this tiny webserver will listen, You can simple ensure it by

[root@clone-1 ~]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:8000                0.0.0.0:*                   LISTEN      9136/python      
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      3519/portmap    

Here you can see local http server is listening on port 8000.
Remember to configure your linux firewall or "iptables" service to access from remote host.

If you want to try some different port the syntax would be:

$ python -m SimpleHTTPServer 8080

Try it out.

21 Nov 2013

Using MTS MBlaze on Mint Linux 14 / Ubuntu 12.04


Yes, Indeed ubuntu and Mint Linux both provides inbuilt support for MTS MBlaze , But sometime network GUI may not recognize your MTS dongle. You can configure "wvdial" to get online.


Install wvdial using
    sudo apt-get install wvdial

Yes, you are right you should install it in advance ;).

You will also need usb-modeswitch and usb-modeswitch-data. this might come preinstalled with your distribution, You can check this under Synaptics Package Manager.
 
Then you need to modify /etc/wvdial.conf configuration file. This might not present on your system by default, But no worries You can generate it by using wvdialconf command

Yogi-Lenovo ~ # wvdialconf
Editing `/etc/wvdial.conf'.

Scanning your serial ports for a modem.

Modem Port Scan<*1>: S0   S1   S2   S3   S4   S5   S6   S7  
Modem Port Scan<*1>: S8   S9   S10  S11  S12  S13  S14  S15 
Modem Port Scan<*1>: S16  S17  S18  S19  S20  S21  S22  S23 
Modem Port Scan<*1>: S24  S25  S26  S27  S28  S29  S30  S31 
ttyUSB0<Info>: Device or resource busy
Modem Port Scan<*1>: USB0
ttyUSB1<*1>: ATQ0 V1 E1 -- failed with 2400 baud, next try: 9600 baud
ttyUSB1<*1>: ATQ0 V1 E1 -- failed with 9600 baud, next try: 9600 baud
ttyUSB1<*1>: ATQ0 V1 E1 -- and failed too at 115200, giving up.
ttyUSB2<*1>: ATQ0 V1 E1 -- OK
ttyUSB2<*1>: ATQ0 V1 E1 Z -- OK
ttyUSB2<*1>: ATQ0 V1 E1 S0=0 -- OK
ttyUSB2<*1>: ATQ0 V1 E1 S0=0 &C1 -- OK
ttyUSB2<*1>: ATQ0 V1 E1 S0=0 &C1 &D2 -- OK
ttyUSB2<*1>: ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0 -- OK
ttyUSB2<*1>: Modem Identifier: ATI -- Manufacturer: +GMI: HUAWEI TECHNOLOGIES CO., LTD
ttyUSB2<*1>: Speed 9600: AT -- OK
ttyUSB2<*1>: Max speed is 9600; that should be safe.
ttyUSB2<*1>: ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0 -- OK

Found a modem on /dev/ttyUSB2.
Modem configuration written to /etc/wvdial.conf.
ttyUSB2<Info>: Speed 9600; init "ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0"

 


This will generate a raw wvdial configuration file, You will need to modify this with MTS parameters.

Alternatively you can use below configuration in /etc/wvdial.conf:

[Dialer mts]
Stupid Mode = 1

Inherits = Modem0

Password = mts

Username = internet@internet.mtsindia.in

Phone = #777

[Modem0]

Init1 = ATZ

SetVolume = 0

Modem = /dev/ttyUSB0

Baud = 115200

FlowControl = Hardware (CRTSCTS)

Dial Command = ATDT


Next, Save it and run $sudo wvdial mts 
You will see somthing like below:

Yogi-Lenovo ~ # wvdial mts
--> WvDial: Internet dialer version 1.61
--> Initializing modem.
--> Sending: ATZ
OK
--> Modem initialized.
--> Sending: ATDT#777
--> Waiting for carrier.
ATDT#777
CONNECT 3100000
--> Carrier detected.  Starting PPP immediately.
--> Starting pppd at Thu Nov 21 23:32:50 2013
--> Pid of pppd: 15289
--> Using interface ppp0

 


Once you are successful, Open Browser and check connectivity, Leave this terminal as it is, to disconnect, return back to terminal and Ctrl+C will disconnect you.

For Tata Photon and Reliance Netconnect you only need to replace USERNAME, PASSWORD and Phone parameter in wvdial.conf.

Hope it helps you get online. 

  

 

29 Oct 2013

Linux Bash : Remove only numbers or patterns from a long list using SED and more


Q: I want to remove only numbers/specific patterns from a long list ?


You can use "sed" for the above task. Below is the command.

#cat filename | sed 's|[0-9]||g'
Or if you want to KEEP ONLY numbers.

# cat filename | sed 's|[^0-9]||g'

Or anything you wish to skip in whole text. for eg "-" and "." in below example

#cat filename | sed 's|[-.-]||g'

A more BASH specific way can be ,if you do not wish to use sed:

while read line; do \
    echo ${line//[0-9]/}; \
done < filename

Well only grep can also perform same stuff:

#cat test | grep -o '[^0-9-]*'


Hope you this little tricks helpful.






Enhanced by Zemanta

25 Oct 2013

Check RPM installed recently on RedHat/CentOS



Q: How do I check that which rpm's have installed recently on my RedHat/CentOS machine ?

RPM provide various ways to manage, update, installation, uninstall of packages.
To check what rpm's installed recently we can use following command.

[root@linuxskool]# rpm -qa --last
perl-rrdtool-1.4.7-1.el5.rf                   Thu 11 Jul 2013 05:16:36 PM IST
rrdtool-1.4.7-1.el5.rf                          Thu 11 Jul 2013 05:16:35 PM IST
ruby-1.8.5-29.el5_9                            Thu 11 Jul 2013 05:16:34 PM IST
libdbi-0.8.1-2.1                                   Thu 11 Jul 2013 05:16:34 PM IST

ruby-libs-1.8.5-29.el5_9                      Thu 11 Jul 2013 05:16:33 PM IST

or more clearly

[root@linuxskool]# rpm -qa --last | less


Hope you find this quick tip useful.


24 Oct 2013

Ubuntu 13.10 (Saucy Salamander)

Canonical Ubuntu 13.10 Now Available

Posted By: Yogesh Upadhyay

The latest Ubuntu is now available to download.
                                 
Canonical said on Thursday that Ubuntu 13.10 for desktop and smartphone is nowavailable. This release includes a wide range of mobile core apps created by the Ubuntu developer community, including a browser, calendar, clock, weather, and calculator. The company also said that it's working with partners to bring Ubuntu smartphone devices to market in 2014.
"This is a milestone in Ubuntu's history; the exact same Ubuntu OS runs on ARM phones and modern HP Moonshot ARM servers, and provides exactly the same capability as x86 platforms," said Rick Spencer, who leads Ubuntu's consumer-facing engineering. "Ubuntu 13.10 is a full server-grade OS that offers a mobile experience and is lean enough to support mobile devices, kicking off a new era in mobile security and computing convergence."
The release of Ubuntu 13.10 also brings the full SDK for developing apps for all Ubuntu form factors. This toolset includes templates and extensions, theming, automatic orientation, and easy to use UI tools for rapid application development. This toolset also supports native and HTML5 development, and a "responsive app design" that, according to Canonical, should make it easy for developers to target phones, tablets and PCs with a single database.
The updated OS also introduces Smart Scopes into the Dash. Simply type any query into the Dash Home, and the Smart Scopes server will determine which categories of content are the most relevant. The server constantly improves the results by learning which categories and results are the most helpful over time.
Ubuntu 13.10 also sets MIR, the new open source graphics stack which supports higher frame-rates, enabled by default for smartphones, and provides it as an option for desktops. MIR promises improved performance in games, with a simplified driver model for widespread hardware support, and better access to the latest underlying graphics capabilities of today's devices.
"The Ubuntu Dash brings content straight to your desktop, searching more than 50 online sources through scopes," reads the press release. "Ubuntu 13.10 introduces a Smart Scope on both desktop and phone which combines results from many different scopes automatically and learns individual user preferences so that search results improve for each user over time. In 13.10, the Dash includes many new search scopes including Wikipedia, Amazon, Google News and Flickr, and can be configured for privacy or specific search preferences."
Ubuntu 13.10 is available to download now right here. Canonical is accepting donations, but you can still grab the new OS for free. 

Further, Ubuntu 14.04 "Trusty Tahr" daily builds are available to download. Worth mentioning that 14.04 will be a Long Term Support release (LTS).
You can get 14.04 first builds at Right-Here
Courtesy:TomsHardware,Ubuntu Official Page.

Enhanced by Zemanta