# Shell & Command Line 

## 1. `ls` - List Directory Contents

```bash
ls                      # list files in current directory
ls /path/to/dir         # list specific directory
ls -a                   # list ALL files including hidden (dot files) ✅
ls -l                   # long format (permissions, size, date, owner)
ls -h                   # human-readable sizes (KB, MB) - NOT hidden files ❌
ls -la                  # long format + hidden files ✅ (most useful)
ls -lh                  # long format + human-readable sizes
ls -lt                  # sort by modification time
ls -lS                  # sort by file size
ls -lr                  # reverse order
ls -R                   # recursive (list subdirectories too)
```

### Hidden Files

- Hidden files start with `.` (dot): `.gitignore`, `.env`, `.bashrc`, `.ssh/`
- `ls` without `-a` does NOT show hidden files
- `ls -a` shows ALL files including hidden ✅
- ❌ `ls -h` shows human-readable sizes, NOT hidden files
- ❌ `dir /ah` is Windows only, does NOT work on Unix/Linux

### Long Format Output (`ls -la`)

```
drwxr-xr-x  5 user group 4096 Dec 14 16:45 .
-rw-r--r--  1 user group  220 Dec 14 09:00 .bashrc    <- hidden
-rw-r--r--  1 user group 2048 Dec 14 16:45 script.py
# Columns: permissions | links | owner | group | size | date | name
```

---

## 2. `find` - Search for Files by NAME

```bash
find . -name "*.py"             # all Python files ✅
find . -name "data.csv"         # specific file
find /home -name "*.log"        # in specific directory
find . -type f -name "*.csv"    # files only (not dirs)
find . -type d -name "data"     # directories only
find . -size +10M               # files larger than 10MB
find . -size -1k                # files smaller than 1KB
find . -mtime -7                # modified in last 7 days
find . -mtime +30               # modified more than 30 days ago
find . -name "*.log" -exec cat {} \;   # cat all log files
find . -name "*.pyc" -exec rm {} \;   # delete all .pyc files
```

### `find` vs `grep` (Exam Critical)

| Command | Searches | Example |
| :--- | :--- | :--- |
| `find` | File NAMES | `find . -name "*.log"` |
| `grep` | File CONTENTS | `grep "error" logfile.txt` |

- ❌ `find -name "Pine Ridge"` finds a FILE named Pine Ridge, NOT content inside files
- ✅ `grep "Pine Ridge" file.txt` finds lines CONTAINING "Pine Ridge"

---

## 3. `grep` - Search Text INSIDE Files

```bash
grep "search_term" file.txt       # search in file ✅
grep "Pine Ridge" evacuation.txt  # find all lines with "Pine Ridge" ✅
grep "error" app.log              # find error lines

# Important flags
grep -i "term" file.txt     # case-insensitive search ✅
grep -n "term" file.txt     # show line numbers
grep -v "term" file.txt     # invert: lines NOT containing term
grep -c "term" file.txt     # count matching lines
grep -l "term" *.txt        # list FILES containing term
grep -r "term" ./dir/       # recursive search in directory ✅
grep -w "term" file.txt     # whole word match only
grep -A 2 "term" file.txt   # show 2 lines AFTER match
grep -B 2 "term" file.txt   # show 2 lines BEFORE match
grep -E "pattern" file.txt  # extended regex ✅

# grep with regex
grep -E "[0-9]+" file.txt           # lines with numbers
grep -E "^ERROR" file.txt           # lines starting with ERROR
grep -E "(GET|POST)" access.log     # lines with GET or POST ✅
grep -E "1[2-5]:[0-5][0-9]" log    # time range 12:xx-15:xx
```

---

## 4. `awk` - Field Extraction & Processing

```bash
awk '{print $1}' file.txt          # print first field
awk '{print $3, $7}' file.txt      # print 3rd and 7th fields ✅
awk -F',' '{print $2}' file.csv    # comma delimiter
awk '{print NR, $0}' file.txt      # with line numbers

# Filter and extract
awk '$9 == 200 {print $0}' access.log    # lines where field 9 = 200
awk '$7 ~ /\/checkout/' access.log       # URL contains /checkout/

# Calculate
awk '{sum += $5} END {print sum}' file.txt           # sum column 5
awk '{count[$7]++} END {for (url in count) print count[url], url}' log
```

### `awk` Special Variables

| Variable | Meaning |
| :--- | :--- |
| `$0` | Entire line |
| `$1`, `$2`... | Field 1, 2... |
| `NR` | Current line number |
| `NF` | Number of fields in current line |
| `FS` | Field separator (default: whitespace) |

---

## 5. `sort`, `uniq`, `wc` - Text Processing

```bash
# sort
sort file.txt              # alphabetical sort
sort -n file.txt           # numeric sort
sort -r file.txt           # reverse order
sort -nr file.txt          # numeric reverse ✅ (most exam-relevant)
sort -k2 file.txt          # sort by 2nd field
sort -k2 -n file.txt       # sort by 2nd field numerically
sort -u file.txt           # sort and remove duplicates
sort -t',' -k2 file.csv    # sort CSV by 2nd column

# uniq (MUST sort first!) ✅
sort file.txt | uniq       # remove consecutive duplicates
sort file.txt | uniq -c   # count occurrences ✅ (exam answer)
sort file.txt | uniq -d   # show only duplicates
sort file.txt | uniq -u   # show only unique lines

# wc - count
wc file.txt                # lines, words, characters
wc -l file.txt             # count lines only ✅
wc -w file.txt             # count words
wc -c file.txt             # count bytes
```

---

## 6. `head`, `tail`, `cat`, `less`

```bash
head file.txt              # first 10 lines (default)
head -n 20 file.txt        # first 20 lines
tail file.txt              # last 10 lines
tail -n 20 file.txt        # last 20 lines
tail -f file.txt           # follow file in real-time (logs) ✅

cat file.txt               # display entire file
cat -n file.txt            # display with line numbers
cat file1 file2            # display multiple files
cat > newfile.txt          # create file (Ctrl+D to end)

less file.txt              # view page-by-page (better for large files)
# In less: /term=search, n=next, q=quit
```

---

## 7. Shell Pipelines (Exam Critical)

The `|` pipe operator passes output of left command as input to right command.

```bash
# Log Analysis Pipeline ✅ (exam-relevant)
grep -E "(0[89]|1[0-8]):[0-5][0-9]" traffic.log \
  | awk '{print $3, $7}' \
  | sort \
  | uniq -c \
  | sort -nr

# Breakdown:
# grep -E "..."  -> filter lines matching time pattern
# awk '{...}'    -> extract fields 3 and 7
# sort           -> sort (required before uniq)
# uniq -c        -> count unique occurrences
# sort -nr       -> sort by count, highest first

# Count HTTP status codes in log
awk '{print $9}' access.log | sort | uniq -c | sort -nr

# Top 10 most common IP addresses
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -10

# Find most common POST URLs
grep "POST" access.log | awk '{print $7}' | sort | uniq -c | sort -nr

# Count errors by hour
grep "ERROR" app.log | awk '{print substr($2,1,2)}' | sort | uniq -c

# Count lines matching pattern
grep -c "404" access.log

# Extract unique URLs
awk '{print $7}' access.log | sort -u
```

---

## 8. Directory & File Operations

```bash
# Navigation
pwd                         # print working directory
cd /path/to/dir             # change directory
cd ..                       # go up one level
cd ~                        # go to home directory
cd -                        # go to previous directory

# Directory management
mkdir new_folder            # create directory
mkdir -p a/b/c              # create nested directories ✅
rmdir empty_folder          # remove empty directory
rm -rf folder/              # remove directory + contents (careful!)

# File operations
cp source.txt dest.txt      # copy file
cp -r source/ dest/         # copy directory recursively
mv old.txt new.txt          # move/rename file
rm file.txt                 # remove file
rm -f file.txt              # force remove (no confirmation)
touch newfile.txt           # create empty file or update timestamp
```

---

## 9. File Permissions

```bash
chmod 755 script.py         # rwxr-xr-x
chmod +x script.py          # make executable
chmod 644 file.txt          # rw-r--r--
chown user:group file.txt   # change owner

# Permission notation: r=4, w=2, x=1
# 755 -> rwxr-xr-x (owner: all, group+others: read+execute)
# 644 -> rw-r--r-- (owner: read+write, others: read only)
```

---

## 10. Network & Downloads

```bash
curl https://api.example.com/data              # make HTTP GET request
curl -O https://example.com/file.zip           # download file
curl -H "Authorization: Bearer TOKEN" url      # with header ✅
curl -X POST -d '{"key":"value"}' url          # POST request
wget https://example.com/file.zip              # download file

# API testing
curl -s "https://api.openweathermap.org/data/2.5/weather?q=Delhi&appid=KEY"
```

---

## 11. Environment Variables

```bash
export API_KEY="your_key"    # set environment variable ✅
echo $API_KEY                # print variable
env                          # list ALL environment variables
unset API_KEY                # remove variable
printenv API_KEY             # print specific variable

# In Python
import os
key = os.getenv('API_KEY')             # ✅
key = os.environ.get('API_KEY', 'default')
```

---

## 12. Shell Scripting Basics

```bash
#!/bin/bash
# Variables
NAME="John"
echo "Hello, $NAME"

# Conditionals
if [ -f "file.txt" ]; then
    echo "File exists"
elif [ -d "folder" ]; then
    echo "Folder exists"
else
    echo "Neither"
fi

# Loops
for i in {1..5}; do echo "Number: $i"; done
for file in *.csv; do echo "Processing: $file"; done

# While loop reading file
while IFS= read -r line; do
    echo "$line"
done < file.txt

# Check exit code
python process_data.py > temp_output.csv
if [ $? -eq 0 ]; then
    mv temp_output.csv final_output.csv   # ✅ atomic move
    echo "Success"
else
    echo "Failed"
    exit 1
fi
```

---

## 13. Process Management

```bash
ps aux                      # list all running processes
kill -9 PID                 # force kill process
top                         # real-time process monitor
htop                        # improved process monitor
nohup python script.py &    # run in background
jobs                        # list background jobs
fg %1                       # bring job 1 to foreground
```

---

## 14. Quick Reference Card

```
Navigation & Files:
  pwd       -> current directory
  ls -a     -> list all including hidden ✅
  ls -la    -> long format + hidden
  find      -> search FILE NAMES (not content)

Viewing Content:
  cat       -> display file (small)
  less      -> page-by-page (large files)
  head -n N -> first N lines
  tail -f   -> follow file real-time ✅

Searching:
  grep "term" file    -> search CONTENT ✅
  grep -i             -> case-insensitive
  grep -r "term" dir  -> recursive
  grep -E "pattern"   -> extended regex

Processing:
  awk '{print $N}'    -> extract field N
  sort -nr            -> numeric reverse sort ✅
  uniq -c             -> count unique (sort first!) ✅
  wc -l               -> count lines

Pipeline Pattern:
  grep -E "pattern" log | awk '{print $field}' | sort | uniq -c | sort -nr ✅

Log Analysis Exam Pattern:
  grep | awk | sort | uniq -c | sort -nr | head
```

---

## 15. Exam Scenario Answers

| Scenario | Command |
| :--- | :--- |
| Show hidden files | `ls -a` ✅ |
| Search content in file | `grep "term" file.txt` ✅ |
| Search file names | `find . -name "*.py"` ✅ |
| Count lines with "ERROR" | `grep -c "ERROR" app.log` ✅ |
| Top 5 most common IPs | `awk '{print $1}' log | sort | uniq -c | sort -nr | head -5` ✅ |
| Follow log in real-time | `tail -f logfile.log` ✅ |
| Count total lines | `wc -l file.txt` ✅ |
| Make script executable | `chmod +x script.py` ✅ |
| Recursively find CSVs | `find . -type f -name "*.csv"` ✅ |
| Sort numerically descending | `sort -nr` ✅ |