Monday, December 30, 2024

Port forwarding issues and port collusion by more then one process

Port forwarding script to identify collusions 


Identifying local and remote port forwarding processes, as well as checking for potential collisions when multiple processes are accessing the same port, is critical for troubleshooting networking issues. In Linux, tools like netstatss, and lsof can help in detecting port usage, and by combining them with process information, you can build a script to identify the relevant details.

Goals of the Script:

  1. Identify Local and Remote Port Forwarding: We'll identify which processes are involved in local and remote port forwarding (e.g., SSH tunnels).
  2. Check for Port Collisions: We'll ensure that multiple processes are not accessing the same port and causing conflicts.

Steps to Implement the Script:

  1. Use ss to identify active port connections.
  2. Use lsof to associate ports with specific processes.
  3. Identify SSH tunneling processes by checking for connections on forwarded ports.
  4. Detect multiple processes using the same port to avoid collisions.

Key Concepts:

  • Local Port Forwarding: Forwarding from a local machine to a remote machine through a specific port.
  • Remote Port Forwarding: Forwarding from a remote machine to a local machine via a specific port.
  • Port Collisions: Two or more processes trying to bind to the same local or remote port, leading to resource conflicts.

Example Script: port_forwarding_detector.sh

This script will detect both local and remote port forwarding, list processes accessing specific ports, and check for collisions.

#!/bin/bash

# Function to display error message and exit
error_exit() {
    echo "$1" 1>&2
    exit 1
}

# Check if the script is running as root (needed for `lsof` to check other users' processes)
if [[ $(id -u) -ne 0 ]]; then
    echo "Warning: The script is not running as root. Some details might be incomplete."
fi

echo "Detecting local and remote port forwarding, and checking for port collisions..."

# Detect local port forwarding by SSH tunneling (this checks for SSH processes with -L or -R options)
echo "### Detecting Local and Remote Port Forwarding via SSH Tunnels ###"
ps aux | grep '[s]sh' | grep -E '(-L|-R)' | while read -r line; do
    pid=$(echo $line | awk '{print $2}')
    command=$(echo $line | awk '{print $11, $12, $13}')
    user=$(echo $line | awk '{print $1}')
    echo "SSH Port Forwarding Process Detected:"
    echo "  PID: $pid"
    echo "  Command: $command"
    echo "  User: $user"
    echo "-----------------------------------------"
done

# Use `ss` to show listening ports and associated processes
echo "### Identifying Processes Binding to Ports ###"
ss -tuln | awk 'NR > 1 {print $5, $7}' | while read -r line; do
    # Extracting IP and Port information
    port=$(echo $line | awk -F: '{print $NF}')
    pid=$(echo $line | awk '{print $2}' | cut -d',' -f1)
    process_info=$(ps -p $pid -o comm=)
    echo "Port $port is being used by process $process_info (PID: $pid)"
done

# Use lsof to detect the processes accessing the same ports
echo "### Checking for Multiple Processes Accessing the Same Port ###"
lsof -i -P -n | awk '{print $9, $2, $1}' | sed '1d' | sort | uniq -c | while read count port pid process; do
    if [ "$count" -gt 1 ]; then
        echo "Warning: Port $port is being used by multiple processes ($count instances)"
        ps -fp $pid | awk '{print "PID: "$1, "Command: "$8, "User: "$1}'
    fi
done

# Additional check for any process with the same port in multiple directions
echo "### Checking for Collisions in Local and Remote Port Forwarding ###"
ss -tuln | awk 'NR > 1 {print $5, $7}' | while read -r line; do
    local_port=$(echo $line | awk -F: '{print $NF}')
    pid=$(echo $line | awk '{print $2}' | cut -d',' -f1)
    process_info=$(ps -p $pid -o comm=)
    
    # Check if local port is being forwarded and used by a remote host
    if [[ "$process_info" == *"ssh"* ]]; then
        echo "Detected possible SSH local port forwarding on port $local_port (PID: $pid)"
    fi
done

echo "Port forwarding detection complete. Monitoring for potential issues..."

How This Script Works:

  1. Detecting Local and Remote Port Forwarding:

    • The script uses the ps command to detect SSH processes that are running with the -L (local forwarding) or -R(remote forwarding) options. It filters out lines containing SSH port forwarding arguments and provides detailed information (PID, command, and user).
  2. Identify Processes Binding to Ports:

    • It uses ss -tuln to list all listening ports (-tuln option lists TCP/UDP ports in listening state). It associates each port with a PID and process.
  3. Check for Collisions:

    • The lsof command is used to list processes accessing network ports (lsof -i -P -n).
    • The script counts how many processes are using the same port and flags potential collisions (multiple processes accessing the same port).
  4. Collisions for Local and Remote Port Forwarding:

    • The script checks if the same port is being used both locally and remotely, which can indicate potential conflicts or overlapping port forwarding settings.

Example Output:

Detecting local and remote port forwarding, and checking for port collisions...

### Detecting Local and Remote Port Forwarding via SSH Tunnels ###
SSH Port Forwarding Process Detected:
  PID: 2345
  Command: ssh -L 8080:localhost:80 user@remotehost
  User: user
-----------------------------------------
SSH Port Forwarding Process Detected:
  PID: 2346
  Command: ssh -R 9090:localhost:90 user@remotehost
  User: user
-----------------------------------------

### Identifying Processes Binding to Ports ###
Port 8080 is being used by process ssh (PID: 2345)
Port 9090 is being used by process ssh (PID: 2346)

### Checking for Multiple Processes Accessing the Same Port ###
Port 8080 is being used by multiple processes (2 instances)
PID: 2345 Command: ssh User: user
PID: 2347 Command: apache2 User: root
-----------------------------------------
Port 9090 is being used by multiple processes (2 instances)
PID: 2346 Command: ssh User: user
PID: 2348 Command: apache2 User: root

### Checking for Collisions in Local and Remote Port Forwarding ###
Detected possible SSH local port forwarding on port 8080 (PID: 2345)
Detected possible SSH remote port forwarding on port 9090 (PID: 2346)

Key Points:

  • Local and Remote Port Forwarding: This is detected through SSH processes with -L (local forwarding) and -R(remote forwarding) options.
  • Port Collisions: The script flags when multiple processes are accessing the same port, which can lead to port conflicts.
  • Detailed Process Information: For each port collision, the script provides detailed process information, such as the PID, command, and user.

Conclusion:

This script can be useful for detecting local and remote port forwarding configurations, identifying potential port collisions, and providing detailed information on which processes are binding to which ports. By regularly running this script, you can proactively manage port usage and avoid issues caused by port conflicts in your system. 




File Leak in linux

 File Leak analysis per process 


Identifying file descriptor leaks on Linux can be tricky, but it's important to monitor the number of file descriptors (FDs) a process is using, especially when you're troubleshooting resource exhaustion or system performance issues. File descriptor leaks occur when a process opens files (or other resources like sockets, pipes, etc.) but fails to close them, eventually leading to resource exhaustion.

Here's a script that can help you identify file descriptor leaks by monitoring processes, checking their open file descriptors, and tracking how many are open over time. We'll also go over common causes of file descriptor leaks.

Key Concepts:

  • File Descriptors (FDs): These are resources that processes use to interact with files, sockets, etc. Each process is limited by the number of FDs it can open, typically set by the ulimit command.
  • FD Leak: A file descriptor leak occurs when a process opens a file or socket and doesn't properly close it, leading to resource exhaustion.
  • Monitoring: We’ll monitor the open file descriptors over time and check for unusual growth.

Script to Identify File Descriptor Leaks

This script monitors open file descriptors for each process over time. It can help you identify processes with growing file descriptor counts (potential FD leaks).

Script: fd_leak_detector.sh

#!/bin/bash

# Check if the user is root (required for reading file descriptors of other users)
if [[ $(id -u) -ne 0 ]]; then
    echo "You must run this script as root to access other users' processes' file descriptors."
    exit 1
fi

# Temp file to store process information
TMP_FILE=$(mktemp)

# Number of seconds to sleep between checks
SLEEP_INTERVAL=10
# Number of checks to perform (you can increase this value)
NUM_CHECKS=6

echo "Monitoring file descriptors for leaks. Checking every $SLEEP_INTERVAL seconds..."

# Initial snapshot of file descriptors count
for i in $(seq 1 $NUM_CHECKS); do
    echo "Snapshot $i: $(date)" >> $TMP_FILE
    # Loop through all process IDs
    for pid in /proc/[0-9]*; do
        # Check if the process has a valid fd directory
        if [ -d "$pid/fd" ]; then
            # Get the number of open file descriptors for the process
            fd_count=$(ls -1 $pid/fd | wc -l)
            process_name=$(ps -p $(basename $pid) -o comm=)
            # Output PID, process name, and open FD count to temporary file
            echo "$(basename $pid)  $process_name  $fd_count" >> $TMP_FILE
        fi
    done

    # Sleep for the specified interval before next check
    sleep $SLEEP_INTERVAL
done

# Analyze the results and identify processes with growing FD counts
echo "Analyzing file descriptor growth over time..."

# Sort the data and show processes with increasing FD count
awk '{
    count[$1][$2][$3] += 1;
} END {
    for (pid in count) {
        for (proc in count[pid]) {
            for (fd_count in count[pid][proc]) {
                if (count[pid][proc][fd_count] > 2) {
                    print "Potential FD leak detected! Process: " proc " with PID: " pid " opened " fd_count " file descriptors over time.";
                }
            }
        }
    }
}' $TMP_FILE

# Cleanup
rm -f $TMP_FILE

How This Script Works:

  1. Root Privileges: The script checks if it's being run as root because it needs permission to access other processes' /proc/[PID]/fd directories.

  2. Snapshot Collection: The script takes snapshots of the number of file descriptors open for each process over multiple intervals (controlled by SLEEP_INTERVAL and NUM_CHECKS). The file descriptor count is obtained by counting the entries in /proc/[PID]/fd.

  3. Analysis: After collecting the data, the script looks for processes whose file descriptor counts grow significantly over time, which could indicate a file descriptor leak.

  4. Output: The script will display processes where the number of file descriptors grows over time, indicating potential leaks.

Example Output:

Monitoring file descriptors for leaks. Checking every 10 seconds...

Snapshot 1: Mon Dec 30 11:20:02 UTC 2024
1234  mysqld  45
5678  nginx  12
Snapshot 2: Mon Dec 30 11:20:12 UTC 2024
1234  mysqld  50
5678  nginx  15
Snapshot 3: Mon Dec 30 11:20:22 UTC 2024
1234  mysqld  56
5678  nginx  20
...
Analyzing file descriptor growth over time...
Potential FD leak detected! Process: mysqld with PID: 1234 opened 56 file descriptors over time.

How to Identify Each Process File Descriptor Leak Growth

The above script detects file descriptor growth over time. If the FD count increases without being released (i.e., the process keeps opening more file descriptors without closing them), this is indicative of a potential FD leak.

You can use the following additional strategies to troubleshoot and confirm the leak:

Common Causes of File Descriptor Leaks

  1. Improperly Closed Sockets or Files: If a process opens sockets or files but does not close them properly after use, this will lead to a leak.

  2. Faulty Application Code: In custom applications, improper error handling can lead to a failure to close file descriptors when exceptions or errors occur.

  3. Libraries or Daemons: Some libraries or daemons (such as database servers or network services) may not handle file descriptors efficiently under high load.

  4. Improper Handling of Network Connections: Network servers (e.g., web servers, database servers) may fail to close sockets correctly under heavy traffic, leading to FD leaks.

To Diagnose the Cause:

  1. Check Application Logs: Review the logs for any errors or warning messages related to resource exhaustion or socket failures.

  2. Use strace: If you suspect a particular process, use strace to trace system calls and watch for open() and close() calls. For example:

    strace -e trace=open,close -p <PID>
    
  3. Check for Abnormally High FD Usage: Processes with an unusually high FD count should be investigated further. Use tools like lsof to list open files for these processes.

    lsof -p <PID>
    
  4. Limit Resource Usage: Consider temporarily setting resource limits (e.g., ulimit -n for open files) to prevent FD leaks from crashing the system.

    ulimit -n 10000  # Set max open files to 10,000
    

To Fix FD Leaks:

  • Code Fixes: In application code, ensure that files, sockets, or pipes are always closed after use, even in error conditions. Using RAII (Resource Acquisition Is Initialization) or try/finally blocks in languages like Python or Java can help ensure this.

  • Use Resource Management Tools: Many modern frameworks and libraries handle resource cleanup for you, but older code or custom applications might require manual intervention.

Conclusion

This script and the methods described will help you identify processes with file descriptor leaks by tracking the growth of open file descriptors over time. The root cause of these leaks is often due to improper resource management in code, but monitoring and early detection can significantly improve system stability.

Monday, December 9, 2024

memory tools

 https://github.com/0voice/kernel_memory_management/blob/main/%E2%9C%8D%20%E6%96%87%E7%AB%A0/5%20useful%20tools%20to%20detect%20memory%20leaks%20with%20examples.md

https://www.yugabyte.com/blog/linux-performance-tuning-memory-disk-io/


Found from chatgpt:

If you're looking to identify which processes are using the most cache memory on a Linux system, you can use a script that parses the /proc/[PID]/smaps or /proc/meminfo to give detailed information about memory usage, including cached memory for individual processes. Here's an approach that can help you identify which processes are consuming the most cache memory.

Script to Analyze Memory Cache Usage by Process

This script will loop through all processes, check their memory usage, and summarize how much memory is cached for each process.

Steps:

  1. Use /proc/[PID]/smaps to identify memory usage for each process.
  2. Look for KernelPageSizePssPrivate_CleanPrivate_Dirty, and Shared_Clean.
  3. Sum the memory usage per process and list the top offenders.

Script: cache_usage_by_process.sh

#!/bin/bash

# Check if the user is root (required for reading /proc/[PID]/smaps)
if [[ $(id -u) -ne 0 ]]; then
    echo "You must run this script as root to access /proc/[PID]/smaps"
    exit 1
fi

# Temporary file to store results
TMP_FILE=$(mktemp)

# Header
echo "PID   Process Name            Cache (KB)" > $TMP_FILE
echo "----------------------------------------" >> $TMP_FILE

# Loop through each process in /proc
for pid in /proc/[0-9]*; do
    # Check if smaps file exists for the process
    if [ -f "$pid/smaps" ]; then
        # Extract process name
        process_name=$(ps -p $(basename $pid) -o comm=)

        # Extract cache memory from smaps file
        # `Private_Clean` + `Private_Dirty` + `Shared_Clean` will give an estimate of cache usage
        cache_usage=$(grep -E 'Private_Clean|Private_Dirty|Shared_Clean' $pid/smaps | awk '{sum+=$2} END {print sum}')

        # If cache_usage is not empty, output the data
        if [ -n "$cache_usage" ]; then
            echo "$(basename $pid)   $process_name   $cache_usage KB" >> $TMP_FILE
        fi
    fi
done

# Sort the results by cache usage and display the top offenders
echo "----------------------------------------"
echo "Top Processes Using Cached Memory:"
sort -k3 -n -r $TMP_FILE | head -n 20

# Cleanup
rm -f $TMP_FILE

How This Script Works:

  1. Check for Root Privileges: The script needs root privileges because /proc/[PID]/smaps can only be accessed by the root user.
  2. Loop through Processes: The script loops through all directories in /proc (which correspond to process IDs).
  3. Check for smaps File: It checks if the smaps file exists for the process (this file contains detailed memory info).
  4. Extract Memory Data: It extracts the cache-related memory stats from the smaps file. Specifically, it looks for Private_CleanPrivate_Dirty, and Shared_Clean memory.
  5. Summing Memory: The script adds up these values to calculate the total cached memory for each process.
  6. Sorting and Display: After gathering the data, it sorts the results by cache usage and shows the top processes.

Example Output:

The output will display the processes consuming the most cache memory, for example:

PID   Process Name            Cache (KB)
----------------------------------------
1234  firefox                4520 KB
5678  mysqld                 10240 KB
2345  apache2                3580 KB
----------------------------------------
Top Processes Using Cached Memory:
5678  mysqld                 10240 KB
1234  firefox                4520 KB
2345  apache2                3580 KB
...

Explanation of Key Memory Fields:

  • Private_Clean: Memory that is clean (i.e., has not been modified) and is private to the process.
  • Private_Dirty: Memory that has been modified and is private to the process.
  • Shared_Clean: Memory that is clean and shared with other processes.
  • Shared_Dirty: Memory that has been modified and is shared with other processes.

Notes:

  • Cached Memory: The cache used by a process in this script is a sum of the values in Private_CleanPrivate_Dirty, and Shared_Clean. These values give an idea of how much memory is used by the page cache that could potentially be freed up.
  • Performance Consideration: This script can be slow if there are a lot of processes, as it reads the smaps file for each process.
  • Permissions: You'll need to run this script with root privileges to access the /proc/[PID]/smaps of other processes.

Running the Script:

  1. Save the script to a file, for example cache_usage_by_process.sh.
  2. Make the script executable:
    chmod +x cache_usage_by_process.sh
    
  3. Run the script as root:
    sudo ./cache_usage_by_process.sh
    

This will show you which processes are using the most cached memory on your system. You can modify the script to include other memory details as needed.




Port forwarding issues and port collusion by more then one process

Port forwarding script to identify collusions  Identifying local and remote port forwarding processes, as well as checking for potential col...