Linux Useful Commands

Posted: May 23, 2019 in Uncategorized
List of Linux commands – Useful to regular job
SNO
Context
Command
Notes
Reference
1
Sum of files in a directory
du -ach *
Total size of files is shown at the bottom of the output along with individual file size
2
Grep a string in dir and sub dir
1. grep -R ‘string’ dir/
2. find dir/ -type f -exec grep -H ‘string’ {} +
3
Convert date in squid log
perl -pe ‘s/\d+/localtime($&)/e’ access.log
Squid access.log generally shows in a format of 1178695088.774. This helps to convert it in readable time manner
4
Logs between 2 time stamps
sed -n ‘/Feb 12 00:00:02/,/Feb 14 23:59:59/P’ access.log
5
Count the no’of occurance in a file
grep -o ‘503’ access.log |wc -l
6
Count the time_wait in server
netstat -aon | grep tcp | grep TIME_WAIT | wc -l
7
Search for error codes
zgrep -P ‘\t503\t’ access_log.20190213* |wc –
To search in zip/archived files
8
S3 bucket/object size
aws s3 ls s3://Bucket-name/ –recursive –human-readable –summarize
9
Curl output with timestamp of reqeuest time with new line prints
curl -s –insecure -L -X GET -A “ios” –proxy 2.3.4.1:8080 https://wgetip.com -w “\n” | ts ‘[%Y-%m-%d %H:%M:%S]’
10
Internet speed test through CLI
For Installation: Download – tool
2. chmod +x speedtest-cli
3. ./speedtest-cli
11
tcpdump
tcpdump -c 8 -tttt -i eth0 -w tcpdump-8080.pcap port 8080
12
sbt – package installation
2. yum install sbt-0.13.5.rpm
SBT – Scala Build Tool – Required for make create on credit stack of iOS proxy
13
Compare 2 files and print matched data
awk ‘NR==FNR{a[$0];next} ($0 in a)’ file1.out file2.out
14
netstat -s
15
netstat -ant | awk ‘{print $6}’ | sort | uniq -c | sort -n
16
History with date
1. export HISTTIMEFORMAT=’%F %t ‘
2. history
17
sbt version
sbt sbtVersion
18
Search for a string/text in multiple files
grep -rnw ‘/etc/datadog-agent/’ -e ‘squid’
19
Extract field based data from a file
awk ‘{print $1}’ filename.txt
Prints column field in a file
20
Print full line of a file based on search field criteria
awk ‘$3 == “11903” { print $0 }’ log.txt
It prints whole line of a file where ever it has the string value
$3 -> For 3rd field/column
print $0 -> to print whole line
21
Prints uniq data from a file
awk ‘{print $3} log.txt | sort | uniq
print 3rd column/field
22
Service monitoring
if service is down, it restarts automatically
23
Extract 2nd column from a file (Usecase: Log file )
awk ‘{print $2}’ error.log > error.out
Extract and print in other file
24
Print only the chars/string after a delimiter
awk -F’.’ ‘{print $1,$2,$3}’ OFS=’.’ error.log
Prints only 1st ($1), 2nd and 3rd chars after a delimiter .
25
Print number of occurrences in a file while checking the string/char/pattern in one file
awk ‘NR==FNR{arr[$0];next} $0 in arr’ file1 file2 | sort | uniq -c
It reads line by line in File1 and compare it with File2. If matches prints number of occurrences in File2
26
Find files with 0 size and delete them all from a directory
  1. find . -type f -size 0 -delete
  2. find ./ -type f -size 0 | xargs rm -f
  3. find ./ -type f -size 0 -exec rm -f {} \;

 

S3 static website hosting

Posted: May 14, 2019 in Uncategorized

Steps to host s3 static website
1. Create a bucket – Ensure the bucket name should be same as the domain for which you want to create a subdomain or alias record
Ex: If the domain is example.com bucket name should be testbucket.example.com. So that S3 end point will be shows under target list while creating alia A recrd set.
2. Bucket permissions.
Block all public access – Off
Block public access to buckets and objects granted through new access control lists (ACLs) – Off
Block public access to buckets and objects granted through any access control lists (ACLs) – Off
Block public access to buckets and objects granted through new public bucket policies – Off
Block public and cross-account access to buckets and objects through any public bucket policies – On

3. Enable Bucket properties – Static website hosting

4. Get the Domain from the static website hosting

5. Route 53
> Select hosted zone > Create record set > Edit record set > name should be bucket name >
Type > A > Alias > Yes > Alias target > Select the s3 website end point > Save record

  1. export HISTTIMEFORMAT=’%F %t ‘

Root partition extension online:
This article may help you when you have a situation of disk running out of space, especially a root partition online.
Note: Below are the steps we can follow, when we have the partitions which are created with LVM only.
Step 1: Check the current disk space or partitions with below command
# df -h

Step 2: Check the available devices or partitions with below command
# lsblk

Step 3 : Ensure the available/free space with below command
# pvdisplay

From the output, find the element Free PE. This value shows the available free space where we can use/extend it for root partition.
If you have free space then follow the next steps.

Step 4:   Check the device name with below command
# fdisk -l
You may find device name something like: /dev/sda

Step 5: Now create a new partition with below command
# fdisk /dev/sda
When you run the above command, it asks for different input parameters(param) that needs to supply in order to create a new partition by the system.
All bold letters are the input we need to enter.

Param 1 Need to create a new partition so enter the option:                          n 

Param 2: We need to have a primary partition (It asks for primary or extended – choose primary)- For that enter the option:                               p

Param 3: Primary device number – You would know this from lsblk command where it shows like sda1 sda2 if the partitions are already created and in use – Enter 2 or 3 depends on the next available device partitions –

Let say if you have only 1 partition is created already and the name of the partition is /dev/sda1, then for the next partition number that you must choose is 2:
So here in this case you option for param 3 is : 2

Param 4: Simply it prompts for acceptance – so just press enter

Param 5: No need to supply any input, simply just enter

Param 6: With the above steps and params input, it creates a partition. We can check the change/newly created partition from partition table and that can be printed by entering a param:   p

Param 7: From the above steps device is created as a normal Linux partition but we need to make it as LVM partition to extend it.
Input the param to make it as Linux LVM partition – to change the filesystem type :  t

Param 8: It asks for a partition number: Ensure to enter the new partition number that we supplied at param 3:  8e

Param 9: Now make changes/write to partition table for the newly created partition:    w

With all above steps we have created a new partition and that can be shown with below command.
# lsblk

Some times it may now show with the above command and there is a requirement of reboot the server to get the partition table.

But this is not recommended when there is no downtime and the server might be in a production cluster pool.

Step 5: Below command helps to show up the newly created device or partition which is ready use to extend the root partition.  This command helps to add or make changes to partition table without reboot.
# partx -v -a /dev/sda

Step 6: Now list the devices again with below command
# lsblk

Step 7: Check the free space and which ready to use with below command.
# pvdisplay

Step 8: Now create a physical volume with the newly created partition with below command
# pvcreate /dev/sda2

Step 9: Ensure the volume is created with the new partition
# pvdisplay

Step 10: Check the root volume group name with below command – This is needed to extend the root volume
# lvdisplay

Step 11: Now, need to extend the volume group from the physical volume created from Step 8.
# vgextend VolGroup /dev/sda3

Step 12: Check the LVM of root size (LV Size ) with below command
# lvdisplay

Step 13: Run below command to see the extend volume – Ensure free space – Free PE where we can use the Free PE to extend the LVM partition of root.
# vgdisplay

Output: From the output ensure the below parameter and use the free space PE value in extending the root partition

Example value would be as follows
Free PE / Size 15232 / 59.50 

In the above example use 15232 as a parameter/option to extend the partition.

Step 14: Now extend the logical volume with below command
# lvextend -l 15232  /dev/VolGroup/lv_root (+4482 a number displays at free space)

Options:
-l 15232 – We get this from Step 13
/dev/VolGroup/lv_root – We get this from step 13

Step 15: Resize the volume group with below command
# resize2fs /dev/VolGroup/lv_root

Step 16: Now check the newly available/free space of root partition with below command
# df -h

That’s all about extending a root partition online – without reboot the server.

Forgot or lost the root password of you Linux box.

Here you go some simple steps to reset your root password

Step 1: Press Ctrl+Alt+Delete to reboot the machine
Step 2: Press ESC to enter into the Grub menu options while the Linux server is booting up. This action avoids your server to boot as normal
Step 3: Select the server Kerenl and press e
Step 4: Then it shows Kerenl argument. At the end of the line Press Space bar and type single
Step 5: Press enter again
Step 6: Press b to boot the machine into single user mode

Now its time to reset the root password as it boots into a single user mode of root user. There you can setup the password with below commands

Step 7: passwd root (Simple you can run passwd command as it logs in with root user)

Type your custom password and Press enter. It prompts to enter the password again.

Enter the new password again. That’s it. It resets the password.

Step 8: Reboot the box and keep quiet until it asks for the root password. (It selects the kernel and boots the machine)

Now try with your new password. Hope it works.

Once logged in with your new password, try to check the runlevel position.  It shows multiuser mode.

We can check this with below command.

Step 9: runlevel

Might shows the output as N 3

That’s it 🙂 You all set to go run your box.

 

Step 1: Edit the file bashrc  ( vi ~/.bashrc ) and add below line into that

alias python=python3

Step 2: Save the file and run below command

source ~/.bashrc

Step 3: Now check the current version of python

python –version

 

Increase the ulimit in Linux

Posted: January 9, 2018 in Uncategorized

Sometimes we may encounter few issues like application is slow, server consuming lot of cpu/memory and too many openfiles etc..

To avoid those kind of issues, from the server side we can increase the ulimit.

By default it is 1024. So we can increase upto 4096

Steps to increase the ulimit on server.

  1. Log on to server
  2. Check the current ulimit with below command

ulimit -n  

If it shows 1024 then increase that with below command

3. Open the file

/etc/security/limits.conf

and add below lines at the end of the file

* soft nofile 4096
* hard nofile 4096

Save the file and exit

4. Logout from the current session and login back to server

5. Check the ulimit value again. It shows current ulimit set for the server is 4096

ulimit -n