HP Mini Machine Auto-Telnet Inspection Script v2.0

I. Overview

In pursuit of slacking off at work while minimizing exploitation by the boss, and in pursuit of earning a paycheck guilt-free during idle moments, I spent several days piecing together a set of scripts that automatically telnet into hosts, relay inspection results back, and generate a PDF report after sorting everything out.

II. Script Purpose

1. Store host passwords and usernames in the script to enable automatic telnet into the hosts. (Once the script is complete, you can encrypt it with software to eliminate the risk of passwords being seen by idle onlookers.)
2. Executable commands are stored in variables, so inspection content can be added or removed at will. (As for the PDF report, additional filtering and re-sorting are required.)
3. Since it is automated, in addition to automatic execution, it can also filter out correct items and keep only error logs, making results clear at a glance.
4. The generated PDF report can be emailed directly to the BOSS, saving effort and trouble. (This is the most important point.)

III. Script Structure

1. auto_check.sh — iterates over hosts and calls various script commands.
2. host.exp — executes expect commands; simply put, it makes remote commands run locally.
3. console.exp — similar in function to host.exp, automatically typing in carriage returns and such, except its target is the small-machine MP card.
4. Others. Extract a single feature and write it standalone along the way, to pad the word count.

IV. Brief Note

Mission accomplished, wahaha — now I can devote myself wholeheartedly to fixing computers. (Applause should be heard here.)

It always feels like time stretches on endlessly. What joy is there in being alive, what sorrow in dying? Regardless of what the meaning of life may be, sitting around waiting to die is certainly not the most reasonable path. Get out more, look around more — it may not lead to enlightenment, but a mouse-eye view will surely bring about self-destruction.

This weekend, with so much time, I only went out to eat two meals, slept a total of eight or nine hours, kept staring at letters with my eyes while my mind ran through loops, and even wondered over meals whether that part could be written better this way. In a word, skills are limited, and forcing it through is exhausting. I wonder how much extra search volume I added to Baidu these past two days.

The computer played "Love Is You and Me" and "West Sea Love Song" on loop from the start, and I even hooked up a small external speaker — not sure whether it could be heard from outside at night, so I'll leave that part unmentioned. That said, accompanied by the soprano and Dao Ge's world-weary tones, my train of thought flowed with astonishing smoothness. Of course, there were also the awkward English prompts, the not-very-targeted variables, the awk/sed commands and shell arrays I wasn't fully proficient with — so let's each interpret them in our own way and learn as we go.

V. BUGs

1. Parts of the generated PDF file do not display in their intended positions;
2. The vmstat results are not reflected in the PDF report;
3. The code is lengthy and repetitive;
4. Some inspection features are not covered.

Attached below is the script code; where privacy is involved, multiple \* are used as substitutes.

Bash

1、auto_check.sh
#! /bin/bash
# auto_check.sh
# Create by Kitaluft 8-2-14
# Copyright (c) 2014 Kitaluft. All right reserved.
HOST=host
USER=user
PASSWD=passwd
[ -e ./host.log ] && rm -rf ./host.log
rm -rf *.log
SAR="echo 'SAR BEGIN'
for I in u d w
do
sar -\$I 1 | grep -e usr -e busy -e swpin
sar -\$I 2 2 | grep -i average | sort -rk 3 | head -1
done
echo 'SAR END'"
VMSTAT="echo 'VMSTAT BEGIN'
vmstat | head -2; vmstat 2 2 | tail -1
echo 'VMSTAT END'"
IOSCAN="echo 'HW BEGIN'
ioscan -fn | grep -i -e \"no_hw\" -e error -e \\
unknown -e unclaimed || echo \"All Hardware OK.\"
echo 'HW END'"
VG="echo 'VG BEGIN'
vgdisplay -v | grep -i -e unavailable -e stale || echo \"All VGs OK\"
echo 'VG END'"
BDF="echo 'FS BEGIN'
bdf | sed -n '/[8,9][0-9]%/p' || echo \"All file system OK\"
echo 'FS END'"
LOG="echo 'LOG BEGIN'
for I in /var/adm/syslog/syslog.log /etc/rc.log /etc/shutdownlog
do
cat \$I | grep -E \"`date +'%b %d'`.*`date +%Y`\" \\
| grep -i -e error -e warning -e mistake -e \\
notice -e \"timed out\" -e \"scsi reset\" -e \"file system full\" -e fail \\
-e notification -e instance -e offline || echo \"\$I OK\"
done
echo 'LOG END'"
DMESG="echo 'DMESG BEGIN'
dmesg | grep -i -e error -e warning -e mistake \\
-e notice -e \"timed out\" || echo \"Dmesg OK.\"
echo 'DMESG END'"
for I in 1 2 3 4 5 6
do
    HOST=1.1.1.$I
    USER=root
    PASSWD="passwd"
    case $I in
        3)
            USER="user"
            PASSWD="password"
            ;;
        18)
            PASSWD="123456"
            ;;
        20)
            PASSWD="abcdef"
            ;;
    esac
    echo "Auto check $HOST@$USER, Please waiting..." | tee -a ./host.log
    ./host.exp $HOST $USER $PASSWD "$SAR" "$VMSTAT" "$IOSCAN" \\
        "$VG" "$BDF" "$LOG" "$DMESG" | sed -e '/spawn telnet/,/]#\|QUIT...........0/d' \\
        -e '/logout\|Connection closed by foreign host.\|exit/d' \\
        -e 's/.*]$ //g' -e '/.*]#\|echo\|There are running jobs./d' \\
        -e '/for I\|> do/,/done/d' -e '/ioscan\|vmstat\|dmesg/d' \\
        | tee ./host.log
    for I in SAR VMSTAT HW VG FS LOG DMESG
    do
        sed -n '/'$I' BEGIN/,/'$I' END/p' ./host.log \\
            | sed -e 's/Average/'$HOST'/g' \\
            -e 's/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/host/g' >> "$I".log
    done
done
for I in 1 2 3 4 5
do
    HOST="2.2.2.$I"
    echo "Auto check $HOST's MP, please waiting..." | tee -a ./host.log
    ./console.exp $HOST | sed -n '/Switching to \\
        alert level 3 filter.\|Alert threshold level 3 \\
        filter will be applied./,/MP MAIN MENU/p' | sed '/MP:/d' \\
    | tee -a ./host.log
#   | grep -E -e `date +"%d %b %Y"` -e `date +%m/%d/%Y`
#   tee -a ./console.tmp
done
unset HOST USER PASSWD SAR VMSTAT IOSCAN VG BDF LOG DMESG
COL=7
ROW=0
FILE=( SAR.log SAR.log SAR.log $10 )
KEY=( usr swpin device $11 )
NUM=( 1 1 1 $12 )
TAB_COUNT=( l l l l )
COUNT=9
for((I=0;I<"$(($COUNT/3))";I++))
do
    HEAD[I]=`sed -n '/'${KEY[I]}'/p' ${FILE[I]} | head -1 | sed 's/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/host/'`
    TABLE[I]=`sed -n '/'${KEY[I]}'/,+'${NUM[I]}'{//b; p}' ${FILE[I]}`
    echo "${HEAD[I]}
    ${TABLE[I]}" > tmp
    ROW=`echo "${HEAD[I]}" | awk '{print NF}'`
    TAB_COUNT[I]=`for((i=0;i<="$ROW";i++))
    do
        if [ $i -ne $ROW ];then
            echo -n "|l"
        else
            echo -n "|"
        fi
    done`
    TAB_CONTENT[I]=`for((i=1;i<="$COL";i++))
    do
        for((j=1;j<="$ROW";j++))
            do
                sed -n ''$i'p' tmp | awk '{printf("%s",
$j')}'
                if [ $j -ne $ROW ];then
                    echo -n " & "
                else
                    echo "\\ \\hline"
                fi
            done
    done`
done
COL=7
ROW=5
FILE=( null VG.log DMESG.log HW.log FS.log )
HOST=( host 1.1.1.1 1.1.1.3 1.1.1.4 1.1.1.5 1.1.6 )
VG=( vg )
DMESG=( dmesg )
FS=( filesystem )
HW=( hardware )
vg_begin=`sed -n '/BEGIN/=' VG.log`
vg_end=`sed -n '/END/=' VG.log`
VG_BEGIN=( null $vg_begin )
VG_END=( null $vg_end )
dmesg_begin=`sed -n '/BEGIN/=' DMESG.log`
dmesg_end=`sed -n '/END/=' DMESG.log`
DMESG_BEGIN=( null $dmesg_begin )
DMESG_END=( null $dmesg_end )
fs_begin=`sed -n '/BEGIN/=' FS.log`
fs_end=`sed -n '/END/=' FS.log`
FS_BEGIN=( null $fs_begin )
FS_END=( null $fs_end )
hw_begin=`sed -n '/BEGIN/=' HW.log`
hw_end=`sed -n '/END/=' HW.log`
HW_BEGIN=( null $hw_begin )
HW_END=( null $hw_end )
for((i=1;i<$COL;i++))
do
    if [ "$[${VG_END[i]}-${VG_BEGIN[i]}]" -eq 2 ];then
        VG[i]="ok"
    else
        VG[i]="error"
    fi
    if [ "$[${DMESG_END[i]}-${DMESG_BEGIN[i]}]" -eq 2 ];then
        DMESG[i]="ok"
    else
        DMESG[i]="error"
    fi
    if [ "$[${FS_END[i]}-${FS_BEGIN[i]}]" -eq 2 ];then
        FS[i]="ok"
    else
        FS[i]="error"
    fi
    if [ "$[${HW_END[i]}-${HW_BEGIN[i]}]" -eq 2 ];then
        HW[i]="ok"
    else
        HW[i]="error"
    fi
done
TABLE=`for((i=0;i<$COL;i++))
do
    echo -ne "${HOST[i]} ${VG[i]} ${DMESG[i]} ${FS[i]} ${HW[i]}\n"
done`
echo "$TABLE" > tmp
TAB_COUNT4=`for((i=0;i<="$ROW";i++))
do
    if [ $i -ne $ROW ];then
        echo -n "|l"
    else
        echo -n "|"
    fi
done`
TAB_CONTENT4=`for((i=1;i<="$COL";i++))
do
    for((j=1;j<="$ROW";j++))
        do
            sed -n ''$i'p' tmp | awk '{printf("%s",
$j')}'
            if [ $j -ne $ROW ];then
                echo -n " & "
            else
                echo "\\ \\hline"
            fi
        done
done`
ERROR=`for((i=1;i<"$COL";i++))
do
    if [ "${VG[i]}" = "error" ];then
        echo ${HOST[i]}
        sed -n ''${VG_BEGIN[i]}','${VG_END[i]}'p' VG.log \\
            | sed -e 's/VG BEGIN/VG errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
    if [ "${DMESG[i]}" = "error" ];then
        sed -n ''${DMESG_BEGIN[i]}','${DMESG_END[i]}'p' DMESG.log \\
            | sed -e 's/DMESG BEGIN/dmesg errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
    if [ "${FS[i]}" = "error" ];then
        sed -n ''${FS_BEGIN[i]}','${FS_END[i]}'p' FS.log \\
            | sed -e 's/FS BEGIN/filesystem errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
    if [ "${HW[i]}" = "error" ];then
        sed -n ''${HW_BEGIN[i]}','${HW_END[i]}'p' HW.log \\
            | sed -e 's/HW BEGIN/hardware errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
done`
echo "\\documentclass{article}
\\author{Kitaluft}
\\title{HP-UX AUTO CHECK REPORT}
\\begin{document}
\\begin{table}
\\begin{tabular}{${TAB_COUNT[0]}}
\\hline
${TAB_CONTENT[0]}
\\end{tabular}
\\end{table}
\\begin{table}
\\begin{tabular}{${TAB_COUNT[1]}}
\\hline
${TAB_CONTENT[1]}
\\end{tabular}
\\end{table}
\\begin{table}
\\begin{tabular}{${TAB_COUNT[2]}}
\\hline
${TAB_CONTENT[2]}
\\end{tabular}
\\end{table}
\\begin{table}
\\begin{tabular}{$TAB_COUNT4}
\\hline
$TAB_CONTENT4
\\end{tabular}
\\end{table}
\\begin{table}
\\end{table}
$ERROR
\\end{document}" | sed -e 's/\\ /\\\\ /g' -e '/.*/s/$/\\\\/g' \\
    -e '/^\\/s/\\\\//g' -e '/hline/s/hline\\/hline/g' \\
    -e '/hline/s/hline\\/hline/g' -e 's/%/\\%/g' \\
    -e 's/_//g' -e '/^$/d' -e '/{}/{n; d;}'  > tmp.tex
latex tmp.tex >/dev/null 2>&1
dvipdf tmp.dvi host_check_report.pdf 1>/dev/null 2>&1
rm tmp*
unset FILE COL ROW TAB_COUNT TAB_CONTENT KEY NUM
exit 0
2、console.exp
#! /usr/bin/expect
# console.exp
# Create by Kitaluft 8-2-14
# Copyright (c) 2014 Kitaluft. All right reserved.
set timeout -1
set HOST [lindex $argv 0]
spawn telnet $HOST
expect "MP login:"
send "Admin\n"
expect "MP password"
send "Admin\n"
expect "MP>"
send "sl\n"
expect {
    "MP:VW>" { send "sel\n" }
    "*Quit:" { send "e" }
}
expect "*>"
send "t"
sleep .5
send "a"
sleep .5
send "3"
sleep .5
send "\n"
sleep .5
send "\n"
sleep .5
send "\n"
sleep .5
send "\n"
sleep .5
send "\n"
expect "*>"
send ""
expect "MP>"
send "x\n"
expect eof
exit 0
3、host.exp
#! /usr/bin/expect
# host.exp
# Create by Kitaluft 8-2-14.
# Copyright (c) 2014 Kitaluft. All right reserved.
set timeout -1
set HOST [lindex $argv 0]
set USER [lindex $argv 1]
set PASSWD [lindex $argv 2]
for {set I 3} {$I<$argc} {incr I} {
    spawn telnet $HOST
    expect login:
    send "$USER\r"
    expect Password:
    send "$PASSWD\r"
    expect {
    "                                       QUIT...........0" {send "\n"}
    "]#" {send "\n"}
    }
    expect {
    "]$ " {send "[lindex $argv $I]\n"}
    "]#" {send "[lindex $argv $I]\n"}
    }
    expect {
    "]$ " {send "exit\n"
        exp_continue}
    "]#" {send "exit\n"
        exp_continue}
    }
}
exit 0
4、make_table.sh
Extract vg/log/dmesg/hardware/filesystem info and generate a table
#! /bin/bash
# make_table.sh
# Create by Kitaluft 8-3-14.
# Copyright (c) 2014 Kitaluft. All right reserved.
COL=7
ROW=5
FILE=( null VG.log DMESG.log HW.log FS.log )
HOST=( host 76.1.40.1 76.1.40.3 76.1.40.4 76.1.40.5 76.1.40.18 76.1.40.20 )
VG=( vg )
DMESG=( dmesg )
FS=( filesystem )
HW=( hardware )
vg_begin=`sed -n '/BEGIN/=' VG.log`
vg_end=`sed -n '/END/=' VG.log`
VG_BEGIN=( null $vg_begin )
VG_END=( null $vg_end )
dmesg_begin=`sed -n '/BEGIN/=' DMESG.log`
dmesg_end=`sed -n '/END/=' DMESG.log`
DMESG_BEGIN=( null $dmesg_begin )
DMESG_END=( null $dmesg_end )
fs_begin=`sed -n '/BEGIN/=' FS.log`
fs_end=`sed -n '/END/=' FS.log`
FS_BEGIN=( null $fs_begin )
FS_END=( null $fs_end )
hw_begin=`sed -n '/BEGIN/=' HW.log`
hw_end=`sed -n '/END/=' HW.log`
HW_BEGIN=( null $hw_begin )
HW_END=( null $hw_end )
for((i=1;i<$COL;i++))
do
    if [ "$[${VG_END[i]}-${VG_BEGIN[i]}]" -eq 2 ];then
        VG[i]="ok"
    else
        VG[i]="error"
    fi
    if [ "$[${DMESG_END[i]}-${DMESG_BEGIN[i]}]" -eq 2 ];then
        DMESG[i]="ok"
    else
        DMESG[i]="error"
    fi
    if [ "$[${FS_END[i]}-${FS_BEGIN[i]}]" -eq 2 ];then
        FS[i]="ok"
    else
        FS[i]="error"
    fi
    if [ "$[${HW_END[i]}-${HW_BEGIN[i]}]" -eq 2 ];then
        HW[i]="ok"
    else
        HW[i]="error"
    fi
done
TABLE=`for((i=0;i<$COL;i++))
do
    echo -ne "${HOST[i]} ${VG[i]} ${DMESG[i]} ${FS[i]} ${HW[i]}\n"
done`
echo "$TABLE" > tmp.log
TAB_COUNT=`for((i=0;i<="$ROW";i++))
do
    if [ $i -ne $ROW ];then
        echo -n "|l"
    else
        echo -n "|"
    fi
done`
TAB_Con\=`for((i=1;i<="$COL";i++))
do
    for((j=1;j<="$ROW";j++))
        do
            sed -n ''$i'p' tmp.log | awk '{printf("%s",
$j')}'
            if [ $j -ne $ROW ];then
                echo -n " & "
            else
                echo "\\ \\hline"
            fi
        done
done`
ERROR=`for((i=1;i<"$COL";i++))
do
    if [ "${VG[i]}" = "error" ];then
        echo ${HOST[i]}
        sed -n ''${VG_BEGIN[i]}','${VG_END[i]}'p' VG.log \\
            | sed -e 's/VG BEGIN/VG errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
    if [ "${DMESG[i]}" = "error" ];then
        sed -n ''${DMESG_BEGIN[i]}','${DMESG_END[i]}'p' DMESG.log \\
            | sed -e 's/DMESG BEGIN/dmesg errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
    if [ "${FS[i]}" = "error" ];then
        sed -n ''${FS_BEGIN[i]}','${FS_END[i]}'p' FS.log \\
            | sed -e 's/FS BEGIN/filesystem errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
    if [ "${HW[i]}" = "error" ];then
        sed -n ''${HW_BEGIN[i]}','${HW_END[i]}'p' HW.log \\
            | sed -e 's/HW BEGIN/hardware errors:/g' \\
            -e '/END/d' -e '/OK/d'
    fi
done`
ERROR=`echo "$ERROR" | sed '/.*/s/$/\\\\/g'`
echo "$ERROR"
echo "\\documentclass{article}
\\author{Kitaluft}
\\title{HP-UX AUTO CHECK REPORT}
\\begin{document}
\\begin{table}
\\begin{tabular}{$TAB_COUNT}
\\hline
$TAB_CONTENT
\\end{tabular}
\\end{table}
$ERROR
\\end{document}" | sed -e 's/\\ /\\\\ /g' -e '/.*/s/$/\\\\/g' \\
    -e '/^\\/s/\\\\//g' -e '/hline/s/hline\\/hline/g' \\
    -e '/hline/s/hline\\/hline/g' -e 's/%/\\%/g' \\
    -e 's/_//g' -e '/^$/d' -e '/{}/{n; d;}'  > tmp.tex
latex tmp.tex >/dev/null 2>&1
dvipdf tmp.dvi running.pdf 1>/dev/null 2>&1
# rm tmp.*
unset FILE COL ROW TAB_COUNT TAB_CONTENT KEY NUM
exit 0
5、text2pdf.sh
Convert sar output into a table
#! /bin/bash
# text2pdf.sh
# Create by Kitaluft 8-3-14.
# Copyright (c) 2014 Kitaluft. All right reserved.
if [ $# -lt 3 ];then
    echo -e "\nUsage: $0 [source file] [key word] [number]\n"
    echo -e "eg: \ta b c d\n\t1 2 3 4\nthen 'key word'=a 'number'=1\n"
    exit 1
else
    if (($#%3));then
        echo -e "Usage Error."
        echo -e "\nUsage: $0 [source file1] [key word1] \
[number1] [source file2] [key word2] [number2]...\n"
        exit 2
    fi
fi
COL=7
ROW=0
FILE=( $1 $4 $7 $10 )
KEY=( $2 $5 $8 $11 )
NUM=( $3 $6 $9 $12 )
TAB_COUNT=( l l l l )
for((I=0;I<"$(($#/3))";I++))
do
#   HEAD[I]=`sed -n '/'${KEY[I]}'/p' ${FILE[I]} | head -1 | sed 's/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/host/'`
#   TABLE[I]=`sed -n '/'${KEY[I]}'/,+'${NUM[I]}'{//b; p}' ${FILE[I]}`
#   echo "${HEAD[I]}
#   ${TABLE[I]}" > tmp
    cat tmp.log > tmp
    ROW=2
    #   ROW=`echo "${HEAD[I]}" | awk '{print NF}'`
    TAB_COUNT[I]=`for((i=0;i<="$ROW";i++))
    do
        if [ $i -ne $ROW ];then
            echo -n "|l"
        else
            echo -n "|"
        fi
    done`
    TAB_CONTENT[I]=`for((i=1;i<="$COL";i++))
    do
        for((j=1;j<="$ROW";j++))
            do
                sed -n ''$i'p' tmp | awk '{printf("%s",
$j')}'
                if [ $j -ne $ROW ];then
                    echo -n " & "
                else
                    echo "\\ \\hline"
                fi
            done
    done`
done
echo "\\documentclass{article}
\\author{Kitaluft}
\\title{HP-UX AUTO CHECK REPORT}
\\begin{document}
\\begin{table}
\\begin{tabular}{${TAB_COUNT[0]}}
\\hline
${TAB_CONTENT[0]}
\\end{tabular}
\\end{table}
\\begin{table}
\\begin{tabular}{${TAB_COUNT[1]}}
\\hline
${TAB_CONTENT[1]}
\\end{tabular}
\\end{table}
\\begin{table}
\\begin{tabular}{${TAB_COUNT[2]}}
\\hline
${TAB_CONTENT[2]}
\\end{tabular}
\\end{table}
\\end{document}" | sed -e 's/\\ /\\\\ /g' -e 's/%/\\%/g' -e '/^$/d' -e '/{}/{n; d;}'  > tmp.tex
latex tmp.tex
#>/dev/null 2>&1
dvipdf tmp.dvi sar.pdf
#1>/dev/null 2>&1
# rm tmp.*
unset FILE COL ROW TAB_COUNT TAB_CONTENT KEY NUM
exit 0