NMAP Automation

After I retired from state government IT security work, I was surprised when I started as a federal contractor to see how many of the security practices that were common due to the state IT shoe string security budget also being prevalent in the federal space. One of those practices is paying security professionals to be copy and paste machines where the data being copied is the output of some security tool and the destination is an Excel spreadsheet. In the federal position, I worked with one of the Jira experts from the lead contract firm who automated ingestion of the Nessus scan results into Jira tickets. While that was a big time and sanity saver for me, it ended up leaving me a bit under challenged and I went back into incident response but in the private sector. Luckily, vulnerability scanning and management is not in my job duties, but I still get tapped by some our new team members who are tasked with that responsibility to help make their jobs easier through automation and scripting. The most recent ask was to look at a way to automate nmap scanning to produce a file that could easily be ingested into, you guessed it, Excel.

The current practice is running weekly nmap scans on specific regions for specific ports. That in itself is easy enough with command line nmap and grepable file output options, but with a little bit of scripting and files for targets and ports with some python thrown in to get the output to be pure csv – we ended up with something that might be worth sharing if your situation is similar. While my preferred approach would have been to scan the full range of targets for all of the ports and grepping for the regions and ports, the current tracking spreadsheet is more tailored for specific network scans where the scan activity is limited by port and region.

nmap can take a file as input for the targets, but not for ports to be scanned. A simple solution is to call a Linux command to read a file after the -p port parameter, but that expects a properly formatted comma separated port list. Maintaining port list files is much easier if we can list the ports as one port per line.

The nmap command we ended up with is:

nmap -Pn -n -sS -iL targets/"$targets" -p $(sed -z 's/\n/,/g;s/,$/\n/' < ports/"$ports") -oG output/"$filename"

The parameters are $filename $targets and $ports that are provided as arguments to the script call. The nmap parameters are standard, but the port file is being inserted through a couple of calls to sed with the ports file as source. The first sed removes the newlines and replaces them with a comma. The second sed removes the last newline of the file to avoid a blank port or space showing up in the command. In this next portion of the script, we leverage a python script nmaptocsv that converts nmap output to csv.

python nmaptocsv/nmaptocsv.py -i output/"$filename" -d ',' -f ip-fqdn-port-protocol-service-version-os > output/"$filename".temp

Finally we need a little more clean up to get to pure csv.

cat output/"$filename".temp | sed '/^$/d' | sed 's/\"//g' | awk 'NR == 1; NR > 1 {print $0 | "sort -n"}' | grep -E "VERSION|tcp|udp" > output/"$filename".csv

Sed is used to delete the quotes and blank lines from the output. While awk is used to preserve the header line and sort the rest of the file. grep is used to limit the output to the header and hosts/lines showing open ports. For the full code, follow this link. Comments are welcome as I am not a shell script master, but can cobble something together when needed.

Here is a before and after sample from the straight nmap output, to the temp after the nmaptocsv, and the final csv.

NMAP Output:

Host: 10.10.12.131 () Status: Up
Host: 10.10.12.131 () Ports: 21/closed/tcp//ftp///, 22/open/tcp//ssh///, 25/closed/tcp//smtp///, 80/open/tcp//http///, 110/closed/tcp//pop3///, 137/closed/tcp//netbios-ns///, 139/closed/tcp//netbios-ssn///, 443/closed/tcp//https///, 445/closed/tcp//microsoft-ds///, 3389/closed/tcp//ms-wbt-server///, 8008/closed/tcp//http///, 8080/closed/tcp//http-proxy///

Nmaptocsv Output:
"IP","FQDN","PORT","PROTOCOL","SERVICE","VERSION","OS"
"10.10.12.131","","22","tcp","ssh","",""
"10.10.12.131","","80","tcp","http","",""

Final CSV Output:
IP,FQDN,PORT,PROTOCOL,SERVICE,VERSION,OS
10.10.12.131,,22,tcp,ssh,,
10.10.12.131,,80,tcp,http,,


Leave a Reply

Your email address will not be published. Required fields are marked *