Difference between revisions of "Linux"

From Server Knowledge Base
Jump to navigationJump to search
Line 113: Line 113:


Cut counts the first number/letter as 1, not 0.
Cut counts the first number/letter as 1, not 0.
=== eval ===
Use this of you want run a variable after a pipe and to shorten down your scripts. e.g. in http://serverkb.co.uk/tools/getdns.sh
<pre>ns="ns.nameserver.co.uk"
d="domain.co.uk"
g="grep $d | grep -v 'DiG\|;'"
echo "dig @$ns $d A"
dig @$ns $d A | eval $g</pre>
It is a good replacement instead of $() / `` or running just a variable.


=== Find ===
=== Find ===

Revision as of 17:27, 26 February 2013

To see help commands do command --help or man command. To paste into PuTTY, use SHIFT + INSERT.
For detailed software and hardware info do apt-get install hardinfo then hardinfo. For CentOS 6 use this.
To write to a user in the same SSH server, do w, get their tty session and then do write user ttySession. If they are root, do write root Session
Awk introduction, If manual.

Binary/binaries

These are normally in the bin folder for a program.

Calendar

apt-get install gcal
gcal -K -q GB_EN December/2012-January/2013 # Holidays for Dec/2012 and Jan/2013 with week numbers

Credit to User MPB

Check Linux version/kernel

For Ubuntu do cat /etc/issue or for some CentOS distributions use cat /etc/redhat-release

uname -a
uname -r #for just the kernel
uname -rs #for OS and kernel

Check if SELinux is enabled

apt-get install chkconfig
yum install chkconfig

chkconfig --list
cat /etc/sysconfig/selinux
sestatus
selinuxenabled

Check if Virtualization is available for server

CentOS:

egrep '(vmx|svm)' --color=always /proc/cpuinfo

Ubuntu:

apt-get install cpu-checker
kvm-ok

CLI/bash Commands and scripting

- The $() syntax allows you to load anything echoed to STDOUT to be loaded into a variable, and the single quotes ensure that if there are any spaces in the directory name, it still gets loaded into a single variable. To use command substitution, enclose any command that generates output to standard output inside parentheses and precede the opening parenthesis with a dollar sign, $(command). Command substitution is useful when assigning a value to a variable. It is handy for using the output of one command as an argument to another command. Why is $(...) preferred over `...` (backticks)?

  • For variables with multiple pipes "|", use tacs `` instead of quotes ""
  • If you have an if statement and its requirements are not met (e.g. greater than) and it then moves onto the next if statement, if it uses a mail function it may send it to the root user's email or the Admin/Administrator's email address. To avoid this use > /dev/null . 2>&1 likely won't work.
  • It seems if you are using the read function, you can only call a variable that is inside of it (subshell), from here.
  • To stop a ping after x amount of responses, do ping -c x or use ping -oc y (where y equals a maximum amount of tries)
  • Use >> to append an output to the end of the file.
  • The use of && means that you only echo the name of the directory if the directory creation is successful.

$?

This is the exit status/code of the last executed function/program/command.

Awk

To print out a list of just folders/files you want, do;

ls -l filepath | awk '{print $9;}'

This won't work correctly if there is spaces in the filename. To resolve this, use this command whilst in the directory itself.

To do multiple sections of a result, do:

awk '{print $1,$2,$4,$X;}'

If you want to get rid of/cut certain/specific lines/rows from STDOUT aka print a specific line, use awk in this way:

awk 'NR==22'

This is for use without a file. This example will only display the 22nd row from your output.

Notes: Blank lines count as a row. To do multiple lines do awk 'NR==22,NR==25' . This will output line 22-25.

The command sed '22 ! d' would do the same as the awk 'NR==22' example.

To get rid of pipe symbols in a file, do:

awk -F'|' '{print $1,$10}' FileWithPipes > FileWithoutPipes

Alternatively if you get a list, e.g of domains from MySQL with only one column selected, put them in a file and you can remove the pipes by doing:

cat domainlist | awk '{print $2}' > list

awk '{ printf "%-20s %-40s\n", $1, $2}' allows you to print information in columns

Cut

This tool can be used in an example where you get an output but want to strip it to exactly what you need e.g.

lookupipscript.sh <IPaddress>

Output below:

Plan : Bronze, Silver, Gold
Type : IPv4 or IPv6
URL  : http
IP   : 0.0.0.0(primary)

lookupipscript.sh <IPaddress> | sed 'row ! d' | awk '{print $column;}'

This would output 0.0.0.0(primary). To get just the IP address and not the "(primary)" section, do:

| cut -c 1-7

Cut counts the first number/letter as 1, not 0.

eval

Use this of you want run a variable after a pipe and to shorten down your scripts. e.g. in http://serverkb.co.uk/tools/getdns.sh

ns="ns.nameserver.co.uk"
d="domain.co.uk"
g="grep $d | grep -v 'DiG\|;'"

echo "dig @$ns $d A"
dig @$ns $d A | eval $g

It is a good replacement instead of $() / `` or running just a variable.

Find

Find Help (this includes mtime commands)

find . -maxdepth 1 -type f -printf "%f\n" | sed s/,$//

Finds all files that contain "some string". This command is useful in a directory (e.g. mail), so you would do ls -lah | grep year-mm then:

find . -type f -exec grep -l "some string" {} \;

for loop examples

Grep

Do grep for multiple terms (or exclude), do:

grep -v 'chroot\|default\|fs\|fs-passwd\|httpsdocs'

To look for IP addresses, do this courtesy of SO:

grep -E -o '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)'

To get rid of pipe symbols from MySQL, do:

mysql -pPASSWORD admin -e"select domain from domains;" grep -v "|" > FileWithoutPipes

Head

If for example you do /var/qmail/bin/qmail-qstat and it prints two separate lines, do this to get just the first line:

head -1

If and Else

Advanced if usage

Use -f for files, -d for directories

if [ -f $VARIABLE ]
then
parameters (e.g. echo, mkdir, touch, rm)
fi

Combine if and $? to get to do something with the exit status of the last executed command.

if [ $? -eq 0 ]; then
commands
fi

If you want to prompt for both y and Y for yes input, use:

if [[ $variable1 = y || $variable2 = Y ]];

sed

If you want to get rid of/cut certain/specific lines/rows from STDOUT aka print a specific line, use sed in this way:

sed '22 ! d'

This is for use without a file. This example will only display the 22nd row from your output.

Notes: Blank lines count as a row. The command awk 'NR==22' would do the same.

To print multiple lines do:

sed '22,39 ! d'

To put a variable inside sed, use quotes instead of apostrophes:

sed "22,$variable ! d"

To delete parenthesis/brackets, use this:

sed 's/[()]//g'
sed 's/[)]//g'
sed 's/[(]//g'
sed 's/[[]]//g'
sed 's/[]]//g'
sed 's/[[]//g'

Shells and subshells

If you want a script to make changes to your current shell, not the subshell a bash script works in, execute the script by doing the following (sometimes you may need to do ./ still):

. script

This information was provided by these sources: 1, 2

Use [[ instead of [

Compare a remote file with a local file

ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

Credit to User Root

Compare files in a directory

diff -bur folder1/ folder2/

Warning: when doing diff on folders, if the timestamps are different it will think the files are different. If you actually compare the files, they will be the same.

Console Clock in corner

while true; do echo -ne "\e[s\e[0;$((COLUMNS-27))H$(date)\e[u"; sleep 1; done &

Warning, this auto scrolls your SSH session. To kill it, do:

ps aux | grep bash | grep -v grep

Then kill the bash session at the time you ran it:

kill <processid>

Credit to User Squiidux

Create backup of file whilst in vim

vim file
:!cp % %-

Press enter and continue to edit

Credit to User MPB

Cronjob/Crontab

Generator

crontab -e
crontab -l

*     *     *   *    *        command to be executed
-     -     -   -    -
|     |     |   |    |
|     |     |   |    +----- day of week (0 - 6) (Sunday = 0 or 7)
|     |     |   +------- month (1 - 12)
|     |     +--------- day of        month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)

Guides:

http://www.adminschoice.com/crontab-quick-reference

To do a job every X minutes, do */X

To do a job every minute in a specific hour, do * X * * *

To do a job every day at X hour on the hour, do 0 9 * * * for 9am each day.

To do a job every week at 2am on Sunday, do 0 2 * * 0

To monitor active cron jobs, do tail -f /var/log/cron

/usr/local/bin/php: No such file or directory

Do whereis php

Generally it is actually in /usr/bin/php

Cron Daemon email

If you get the below email:

cd / && run-parts --report /etc/cron.daily
/etc/cron.daily/sysklogd:
chown: cannot access `/var/log/mail.warn': No such file or directory
chown: cannot access `/var/log/syslog': No such file or directory
chown: cannot access `/var/log/mail.info': No such file or directory

Do the following:

cd /var/log
touch /var/log/mail.warn /var/log/syslog /var/log/mail.info
/etc/init.d/sysklogd restart

Execute one off command whenever

echo "ls -l" | at midnight

This is an alternative to cron which allows a one-off task to be scheduled for a certain time.

Credit to User Root

File Permissions

One of the best permissions guide

The values and their equivalent permissions. R is Read, W is Write and X is Execute. There are three sets on a file or folder (e.g. -rwx-w--w- , -rw-r--r-- , -rwxr-xrwx) so you need to put in a value of three (or four in rare occasions) for a file/folder e.g. 644.

0  ---<br>
1  --x
2  -w-
3  -wx
4  r--
5  r-x
6  rw-
7  rwx

The syntax for chmod is (for example) 755 fileorfolder. The -R option can also be used to set the permissions on anything below a folder.
The syntax for chown is chown user:group fileorfolder . To apply the user:group to anything below a folder you need to use the -R option.

Permissions Calculator
Help Guide 1
Help Guide 2
Unix Notation
File protection with chmod

If you use PHP Support as Apache module, it will use the third value of -rw- r-- r-x to permissions. If it is using Fast CGI it will use the user:group for the first two values -rwx rw- --x

If you are using Plesk a good place to check if you are getting Forbidden errors on your website is /var/www/vhosts/yourdomain.com/httpdocs/statistics/logs/error_log . Access your site and tail -f that file. If you get .htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable name your .htaccess file correctly, set your httpdocs to 757 and see this link.

Special permissions

There is also a, g & s which are not widely used as it just needs additional representation at the special/setid permission to the group.

To set a file to -rws--x--x for example use

4 = setuid - s
2 = setgid - s
4 + 2 = S
1 = Sticky Bit
4 + 2 + 1 = T
2511  -r-x--s--x (e.g. /var/qmail/bin/qmail-remote or /var/qmail/bin/qmail-queue)
4655  -rwSr-xr-x
4711  -rws--x--x
4744  -rwsr--r--
4755  -rwsr-xr-x
6411  -r-S--s--x.
6511  -r-s--s--x
6644  -rwSr-Sr--
6666  -rwSrwSrw-
7000  ---S--S--T
7644  -rwSr-Sr-T
7711  -rws--s--t
7744  -rwsr-Sr-T
7755  -rwsr-sr-t

Owner and Group advice for websites

File Locations (index priority)

/etc/apache2/mods-enabled/dir.conf (on Ubuntu) or /etc/httpd/conf/httpd.conf (on CentOS, ~line 402) and it should show something like:

DirectoryIndex index.html index.html.var index.shtml index.cfm index.php index.htm

This shows the default priority in which the index page is picked up and this can be set in the .htaccess file as well with:

DirectoryIndex index.html index.php

These are good places to check if you are using all of your disk space.

cd /var/log
cd /var/www/vhosts (website and statistics)
cd /var/lib/mysql (database) or mysql/mysql
cd /usr/bin/mysqldump (mysql dump)
cd var/qmail (mail logs, queue and configuration)
cd /var/lib/psa/dumps (physical Plesk backups)
cd opt/psa/tmp/ (Plesk stores temporary files here for backups)

Find command guide

FreeBSD

  • mysqldump location: /mysql/bin/mysqldump
  • Remove syntax:

- rm -r folderName

  • Generic tunneling interface starts with gif

Check software versions

fetch instead of wget

To download a file.

whereis instead of locate command

To find files/folders.

Full Directory listing

apt-get install tree
yum install tree
tree > tree.txt

If you feel adventurous do cat tree.txt , it will take a while ;)

Find the deepest directory in your server/file structure

find . -type d -printf '%d:%p\n' | sort -n | tail -1

Find and remove specific file types from current directory

cd into the directory
find . -type f -name '*.filetype' -exec rm -v {} \;

Line count a file

wc -l /file/path

Generate random number

Between 1 and 10:

seq 10| shuf | head -1

GeoIP - Block countries accessing website

Ubuntu: apt-cache search geoip ; apt-get install geoip-database libgeoip-dev libgeoip1 python-geoip geoip-bin libapache2-mod-geoip tclgeoip
CentOS: yum list |grep -i geo , yum install GeoIP.x86_64
http://www.webhostingtalk.com/showthread.php?t=959646
http://askmatt.co.uk/blog/2010/05/block-countries-using-apache-mod-geo-ip-list-of-countries-by-fraud-risk

.htaccess

HyperText access

inode usage

df --si
df -ih

A lot of the time the cause can be /tmp (/var/lib/php/session/ on Plesk) due to sessions. You may want to delete the files in there.

Linux Container

This install below is for an Ubuntu physical server, I may update this in the future for CentOS, Fedora and others.

https://help.ubuntu.com/12.04/serverguide/lxc.html

Installation of LXC

apt-get install lxc
cat /etc/init/lxc-net.conf | grep USE_LXC_BRIDGE

If true set to false unless you want the containers to NAT to your servers real IP addresses, and to be accessible externally.

Pre-container creation steps

To reduce errors pre-container creation do the following:

dpkg-reconfigure locales
locale-gen en_GB
update-locale LANG=en_GB.UTF-8

Creating/deleting containers

OpenVZ Template list

Check the templates below and pick one:

cd /usr/lib/lxc/templates/ ; ls -lah

Create a container from one of the templates:

lxc-create -t ubuntu -n NameOfTheContainer

If you want to install the fedora package, do apt-get install yum

To delete it just do:

lxc-destroy -n NameOfTheContainer

Start a container

lxc-start -n NameOfTheContainer -d

Access the container

The default user is 'ubuntu' with the password 'ubuntu', to enter a container, do:

lxc-console -n NameOfTheContainer
sudo -i

Exit using Ctrl + a, then press q

To re-enter the container, do lxc-console -n NameOfTheContainer and then press enter (you may have to a few times)

Configuration settings

  • By default you can ping a container from the host, and vice versa, and you can ping the outside world from the container.
  • You can set the hostname just like a normal server, if you want to rename the container.

Default configuration of system files

cd /var/lib/lxc/nameofcontainer/rootfs/etc/sysconfig/network-scripts
vim ifcfg-eth0
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
HOSTNAME=phptester
NM_CONTROLLED=no
TYPE=Ethernet
MTU=
vim /var/lib/lxc/nameofcontainer/config
lxc.network.type=veth
lxc.network.link=lxcbr0
lxc.network.flags=up
lxc.network.hwaddr = MAC Address
lxc.utsname = MT
vim /etc/lxc/lxc.conf
lxc.network.type=veth
lxc.network.link=lxcbr0
lxc.network.flags=up

You can add the below to /etc/network/interfaces

auto br1
iface br1 inet dhcp
    bridge_ports eth0

Fedora

The mirrors/repositories the container uses may be broken by default, don't try to install anything. You'll likely get:

Error: Cannot retrieve repository metadata (repomd.xml) for repository: fedora. Please verify its path and try again

And you likely won't be able to ping anything except the host machine and localhost/127.0.0.1

iptables -t nat -A POSTROUTING -s ContainerIP/24 -j SNAT --to-source PhysicalHostIP iptables -t nat -A PREROUTING -m tcp -p tcp --dport 10022 -j DNAT -i eth0 --to-destination ContainerIP:80 iptables -t nat -A PREROUTING -m tcp -p tcp --dport 10443 -j DNAT -i eth0 --to-destination ContainerIP:443

libvirt

vi /etc/yum.repos.d/fedora.repo
vi /etc/yum.repos.d/fedora-updates.repo

Uncomment (#) the lines starting with "baseurl"

yum update

OpenSUSE

Version history

http://www.lacerta.be/d7/content/opensuse-lxc-container-inside-ubuntu

List containers

lxc-list

Set passwords

Log in as the root user of the container:

Fedora container:

Username: root
Password: root

Set the root password to something different:

passwd

You will need to do yum install vim when inside the server.

Ubuntu container:

sudo -i
Username: ubuntu
Password: ubuntu

Set the user's password:

passwd ubuntu

Set the root user's password:

passwd

SSH in externally

To route from externally through the host to the container, just do the below iptables rule:

iptables -t nat -A PREROUTING -p tcp --dport 1337 -j DNAT --to 10.0.4.60:22
iptables-save

In the above case we are saying:

- You want to SSH in on port 1337
- The container's eth0 IP address is 10.0.4.60
- Then below we are saying the physical machine has an IP address of 110.111.112.113

Then externally from the server do:

ssh [email protected] -p2222

And bingo! You should be in the container.

passwd

LSB Init Scripts

Kill tty session

w
ps aux | grep bash | grep -v grep
ps aux | grep tty | grep -v grep
kill -HUP <processid>

Further info

Maldet

Documentation
Maldet .tar.gz
Install and Configure

Always run your scans from chroot environment (if possible) and in Screen (screen -S NameIt), and detach by doing CTRL + A, then press D

Scan reports are normally stored in /usr/local/maldetect/sess/ as session. files.

maldet -a -e -l filepath

Manually alter time/date

Ubuntu

date
date mmddtimeyear #as seen below

Cent OS

cd /etc/
ls -lah
rm localtime
ln -s /usr/share/zoneinfo/Europe/London /etc/localtime
date mmddtimeyear
     | |  |   | ---> 2012
     | |  | 24hr --> XX:XX without :
     | | dd -------> day
     | mm ---------> month

/etc/init.d/ntpd restart or start
date

PID

Under construction.

Process ID.

To determine the usage of a specific process, do top -p PID

The maximum number of pids can be obtained from /proc/sys/kernel/pid_max

Remove/rename file called tilde

mv '~' newfilename
rm '~'

Restart service or service

Useful for differently named ones, e.g.

[ -f /etc/init.d/mysqld ] && service mysqld restart ; [ -f /etc/init.d/mysql ] && service mysql restart

[ -f /etc/init.d/httpd ] && service httpd restart ; [ -f /etc/init.d/apache2 ] && service apache2 restart

Rough Hostname guide

If the server runs Plesk and Virtuozzo, the permanent one needs setting on the hardware or in Virtuozzo.

Run dig -x IP.IP.IP.IP from any Linux server and it will show you the PTR/hostname.

Note: Most servers by default come with a non resolving hostname of localhost, localhost.localdomain or something generic.

The hostname should never be left as the default value on a server sending mail, as it is one of three things mail recipient's mailservers see to determine if mail is spam or not. The other two are reverse DNS and the SMTP banner.

If Plesk throws an error when clicking Websites & Domains tab regarding hostname -f, see this resolution.

Roughly list file count

This includes nested directories:

find /full/file/path -type f | wc -l

SCP Command - Secure Copy

To secure copy a file from one Linux server to another, use the following syntax form:

scp -P PORT file user@IPAddress:/filepath

For example:

scp -P 22 index.html [email protected]:/

To move a folder, put -r in between the port and the folder. If you receive the following error while trying to SCP a file from one server to the other:

stdin: is not a tty

You can solve the issue quickly by doing the following on the destination server:

vi ~/.bashrc
  
if [ $(expr index "$-" i) -eq 0 ]; then
return
fi

Screen

Manage Multiple sessions in one Terminal
Tips and Tricks

Setting the time

http://geoffhankerson.com/node/112
http://codeghar.wordpress.com/2007/12/06/manage-time-in-ubuntu-through-command-line/

System Logs and Shutdown troubleshooting

If you are having an issue, a reboot should not be performed if the server can be accessed in any way (e.g. locally in the data centre, only if it is a Dedicated server). This is because after a reboot there is little you can find out from the logs as the important logs get cleared on restart.

last reboot
last

/var/log/syslog
/var/log/kern.log
/var/log/dmesg
/var/log/messages

grep -i error /var/log/messages ; grep -i panic /var/log/messages ; grep -i warning /var/log/messages

errpt

To find .log files, run updatedb and then locate *.log

To restart and on boot do a disk check do (or -rF):

shutdown -Fr now

tar command

To archive and compress a folder/files do:

tar -czvf files.tgz files/
tar -czvf folder.tar.gz folder/
tar cvf mubackup.tar mu/ ; gzip -9 mubackup.tar

#.tgz is same as .tar.gz

c creates the archive (tar), z compresses it into the gzip, v is verbose, f is the file/folder

To extract do:

#for tar
tar xvf file.tar

#for .tgz or tar.gz
tar zxvf file.tar.gz

Guide 1
Guide 2
3 and 4

User ID 99

This is most commonly the user nobody. This can be caused by PHP or done on purpose. cat /etc/passwd | grep 99 ; vim /etc/passwd

User is not in the sudoers file

If you try to sudo into a server or run sudo and get the following message:

"is not in the sudoers file. This incident will be reported."

Do the following command as root and add the username into the file in the same format as the root user:

visudo

Use unusual characters in filenames

If you want to specify a space, lets say the file = /usr/local/etc/testingdatabase.sql

You need to name the file/folder as follows:

mv /usr/local/etc/testingdatabase.sql /usr/local/etc/testing\ database.sql

This will make the file be testing database.sql , on command line this will appear as testing\ database.sql .

A backslash symbol \ needs to be used before an apostrophe ' , bracket (), exclamation/bang ! symbol or question marks ?:

01\ -\ It\'s\ You.txt
#How it appears: 01 - It's You.txt

02\ -\ Boom\!.m3u
#How it appears: 02 - Boom!.m3u

03\ -\ Why\ Wont\ You\ Work\?.sh
#How it appears: 03 - Why Wont You Work?.sh

04\ -\ Musical\ Playlist\ For\ \(VPS\)\ Server.m3u
#How it appears: 04 - Musical Playlist For (VPS) Server.m3u

updatedb (locate command)

This command is used when you cannot locate a file and you get "locate: warning: database /var/lib/slocate/slocate.db' is more than 8 days old". It is advised to run updatedb at least once a month. However if you get the following error when using it:

updatedb: fatal error: load_file: Could not open file: /etc/updatedb.conf: No such file or directory

You need to create or edit this file

vim /etc/updatedb.conf

and put the following inside of it:

PRUNE_BIND_MOUNTS="yes"PRUNEPATHS="/tmp /var/spool /media"PRUNEFS="NFS nfs nfs4 rpc_pipefs afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre_lite tmpfs usbfs udf"

Alternatively run the below script after reading this file http://serverkb.co.uk/tools/README.txt:

http://serverkb.co.uk/tools/updatedbscript.sh

To vim a file you locate, see this. vim $(command)

VPS Hints and Tips

To check for the filepath of a command run top and then press c.

If it is a container on a node, there are generally no datacentre / rack level restrictions as the container is virtualised on a node. The only restrictions are what is put in place via the container itself effectively.

To list all open Internet, x.25 (HP-UX), and UNIX domain files, use:

lsof -i -U

WHOIS script

Pipe Viewer, PV info page

apt-get install pv ; cd /var/www/vhosts ; ls -l | awk ' {print $9}' > domainlist ; wget serverkb.co.uk/tools/findregistrar.sh ; chmod +x findregistrar.sh

For CentOS use yum -y install jwhois.x86_64

Edit the file and replace domain registrar with the one you want to find. Then do:

./findregistrar.sh

write error

If you get the following error when doing write user TTYsession or write user Session:

write: write: you have write permission turned off.

Do this to fix the problem:

mesg y

Writing Shell Scripts