timbrown.us

After I retired from state government IT security work, I was surprised when I started as a federal contractor to see how many of the security practices that were common due to the state IT shoe string security budget also being prevalent in the federal space. One of those practices is paying security professionals to be copy and paste machines where the data being copied is the output of some security tool and the destination is an Excel spreadsheet. In the federal position, I worked with one of the Jira experts from the lead contract firm who automated ingestion of the Nessus scan results into Jira tickets. While that was a big time and sanity saver for me, it ended up leaving me a bit under challenged and I went back into incident response but in the private sector. Luckily, vulnerability scanning and management is not in my job duties, but I still get tapped by some our new team members who are tasked with that responsibility to help make their jobs easier through automation and scripting. The most recent ask was to look at a way to automate nmap scanning to produce a file that could easily be ingested into, you guessed it, Excel.

The current practice is running weekly nmap scans on specific regions for specific ports. That in itself is easy enough with command line nmap and grepable file output options, but with a little bit of scripting and files for targets and ports with some python thrown in to get the output to be pure csv – we ended up with something that might be worth sharing if your situation is similar. While my preferred approach would have been to scan the full range of targets for all of the ports and grepping for the regions and ports, the current tracking spreadsheet is more tailored for specific network scans where the scan activity is limited by port and region.

nmap can take a file as input for the targets, but not for ports to be scanned. A simple solution is to call a Linux command to read a file after the -p port parameter, but that expects a properly formatted comma separated port list. Maintaining port list files is much easier if we can list the ports as one port per line.

The nmap command we ended up with is:

nmap -Pn -n -sS -iL targets/"$targets" -p $(sed -z 's/\n/,/g;s/,$/\n/' < ports/"$ports") -oG output/"$filename"

The parameters are $filename $targets and $ports that are provided as arguments to the script call. The nmap parameters are standard, but the port file is being inserted through a couple of calls to sed with the ports file as source. The first sed removes the newlines and replaces them with a comma. The second sed removes the last newline of the file to avoid a blank port or space showing up in the command. In this next portion of the script, we leverage a python script nmaptocsv that converts nmap output to csv.

python nmaptocsv/nmaptocsv.py -i output/"$filename" -d ',' -f ip-fqdn-port-protocol-service-version-os > output/"$filename".temp

Finally we need a little more clean up to get to pure csv.

cat output/"$filename".temp | sed '/^$/d' | sed 's/\"//g' | awk 'NR == 1; NR > 1 {print $0 | "sort -n"}' | grep -E "VERSION|tcp|udp" > output/"$filename".csv

Sed is used to delete the quotes and blank lines from the output. While awk is used to preserve the header line and sort the rest of the file. grep is used to limit the output to the header and hosts/lines showing open ports. For the full code, follow this link. Comments are welcome as I am not a shell script master, but can cobble something together when needed.

Here is a before and after sample from the straight nmap output, to the temp after the nmaptocsv, and the final csv.

NMAP Output:
…
Host: 10.10.12.131 () Status: Up
Host: 10.10.12.131 () Ports: 21/closed/tcp//ftp///, 22/open/tcp//ssh///, 25/closed/tcp//smtp///, 80/open/tcp//http///, 110/closed/tcp//pop3///, 137/closed/tcp//netbios-ns///, 139/closed/tcp//netbios-ssn///, 443/closed/tcp//https///, 445/closed/tcp//microsoft-ds///, 3389/closed/tcp//ms-wbt-server///, 8008/closed/tcp//http///, 8080/closed/tcp//http-proxy///
…

Nmaptocsv Output:
"IP","FQDN","PORT","PROTOCOL","SERVICE","VERSION","OS"
"10.10.12.131","","22","tcp","ssh","",""
"10.10.12.131","","80","tcp","http","",""

Final CSV Output:
IP,FQDN,PORT,PROTOCOL,SERVICE,VERSION,OS
10.10.12.131,,22,tcp,ssh,,
10.10.12.131,,80,tcp,http,,

My AMD Athon – II system has been running as a LAMP server for over eight years at this point.  I think I probably started with 14.04 and most recently updated to Ubuntu 18.04 Bionic Beaver. Over time, I played with alternatives to Apache like Nginx and Lighttpd, but always seemed to end up going back to Apache.  With an eight year old system, I could sort of start to tell it was losing its ability to process the new larger images and render them quickly in netPhotoGraphics, formerly ZenPhoto 2.0, by Stephen Billard.  WordPress itself never seemed to complain, but you have to admit this is pretty simple blog. I wasn’t looking forward to replicating the settings and tweaks I had made to the system over the past eight years.  Any externally listening port had security protections and monitoring along with lots of custom shell scripts to do things like look up GeoIP locations and fix the image file timestamps to match the time shot from the EXIF data.  While I could start from scratch and eventually get back to a similar state with a new server, I wanted to preserve the older system, the content, and my tweaks; but do it in the simplest way possible.  After a little research on converting physical to virtual methods, I decided to just try and create a VirtualBox virtual disk image (VDI) using the VboxManage tool to convert the existing hard drive from the Athlon – II server.

To complete this project, I needed some new hardware.  The case was bought a couple of years ago at a good discount from a local PC parts store, Intrex. The box had been pretty banged up in shipping, but I can’t really locate anything more than a tiny ding on one of the side panels.  For the internals, I felt like I wanted to stick with AMD given eight years of good service from the current server. So, I did some research on several sites and used the PC builder parts site to get a feel for pricing.  My initial order included these parts to populate the Antec P100 case.

I ended up having a 2TB Western Digital Black drive to install as the primary disk, but decided to splurge for a 1TB Western Digital Black SSD since the Gigabyte motherboard supported it and I leveraged it for the OS and the 2TB for the VM drive.

For a video card, I had a Nvidia GeForce GT 710B PCIE card in another system I am retiring and moved that card into this system.  I installed Ubuntu 18.04 Bionic Beaver and VirtualBox after building up the system and letting it burn in over the weekend.  The next task was the actual conversion and setting up the virtual machine to use the VDI converted from the actual hard drive from the original server.  I will save you the heartache of thinking you can have the original drive hooked up to your new system for the conversion via USB.  It is just too slow, especially when the original was a 1TB disk with a 512GB root, 1GB swap, and the rest set up as the web content partition.  I tried it, but it was going to take like 60+ hours to make the conversion.  So the original drive was connected to the new server’s SATA controller and the VBoxManage raw disk to vdi conversion executed.

$ sudo VBoxManage convertfromraw /dev/sdc LAMP.vdi –format VDI

Connected to the SATA controller and writing my LAMP.vdi to the internal 2TB conventional disk, the conversion of the 1TB disk to a VDI took about three hours.  Once I had the vdi, I opened VirtualBox and pointed to the LAMP.vdi file as the hard drive for the new virtual machine. 
(Note: You will have to change ownership to your user account or grant permissions on the completed VDI file to use it in VirtualBox.)  Then I tried to replicate the hardware settings of the original system as close as possible.  Mainly the number of processor cores and memory.  I also defined a share so I could move files more easily between the host and the LAMP guest.  I installed another Gb NIC in the system so I could bridge the VM adapter to it and let the LAMP system use the same IP address that it had before without any NAT going on.  As I expected, the initial boot up of the VM failed; but I knew the /etc/fstab was going to be an issue as I had video and music drives that were not part of the migration plan.  After getting dumped into the recovery shell, I edited the fstab file and remarked out the two mounts for those drives.  Another reboot and I was able to sign in.  I went ahead and manually entered the IP address in the NIC config and disabled and re-enabled the connection using the Ubuntu GUI.  At this point, a quick check of trying to bring up the WordPress blog and the photo gallery resulted in the having the default Apache installation page rendered.  To remedy this issue, I did a sudo systemctl reload apache2 and cleared my browser’s cache and both sites rendered correctly on the next attempt.

So far, I am very happy with the new server and plan to add some old OS VM’s in the next few weeks.  One, I will create from an old XP laptop that I have to keep around to program a couple of ham radio HT’s. Eventually, I will add another large drive to the host computer and start using it or another VM to act as a media server and file share.  I suspect that it will take a lot to start taxing the resources on the new system for sometime to come.

Behind the scenes:  The recap above concerning the actual conversion of the disk into a VDI and booting the VM for the first time is from a repeated effort after I foobared some permissions in the /var folder after having a working converted VM.  What only took about 30 minutes to get going the second time around (not counting the VDI conversion), actually took several hours the first time as I could not get past Apache rendering its default page until clearing browser caches after using systemctl to reload the Apache configuration once the IP address was set correctly.  Also, I actually did try and use a dd raw image of the original disk I had created by writing the DD file to an external USB drive connected to the original server.  It was slow and painful and the VBoxManage convertfromraw did not succeed using the DD file as the source.  What I used to the get the VM going the first try, was DD cloned drive of the original and connecting it to the SATA controller to do the convert from raw.  For the final effort as blogged, I pulled the original drive from the Athlon system, connected it directly to the new server’s SATA controller, skipped making a DD image and just converted from the raw disk.  If you have physical access to the new server, this method should be pretty foolproof. But my experience with open source and Linux tells me your experience trying this could vary greatly.

Originally published 8Aug2014 updated below – I would have to call myself and audiophile from a pretty young age as my dad was always recording the band concerts and coming home and playing them on a decently high end stereo at the time.  My Dad and also had the pleasure of knowing Harry Grasser, a violinist, that would share some great classical LP pieces he had come across.  I ended up leaning toward early 80’s contemporary Jazz after growing up on Pop and southern Rock & Roll.  This was not from a dislike of those genres, but more of an escape.  In the early 80’s I was DJ’ing at a local roller skating rink, Sportsworld of Cary.  After playing Pop and a lot of R&B, the chill out music of choice, thanks to Jerry Rose, became a lot of what we call smooth jazz today.  At night, I could not turn up the stereo to get my fix, so I would fall asleep most nights listing to some music through  my headphones.  Back then there was not a large market place and you generally chose between Koss, Sony or some of the new Walkman style phones.  I ended up with some Koss Pro-4a’s and wore them out between heavy use and age.  Since then I floated between ear buds of various sorts and some cheap around ear styles that I did not care if they broke as I was using them each night when I went to bed.  Yes, I still listen to something every night as a fall asleep.  But these days the tracks are mostly binaural beats or self-hypnosis focused on various things from pain control to body asleep – mind awake.  With binaural beats there is a desire hear the low bass frequencies as they form the core of the tones sent to each ear at different frequencies to produce a harmonic within the auditory function of the brain which appears to sound like a beat.  People tend not believe this part of the binaural beat phenomenon, but if you pull the headphone off of one ear the beat produced by the different tones to each ear goes away.  So back to headphones.

Most of the cheap headphones I have bought have been really cheap Sentry brand from BigLots.  Like I said, I was just buying something that worked that I did not care about as I might pull the cord out or break the ear piece off while sleeping.  The specifications on the the Sentry around ear Studio model was 20Hz – 20,000Hz which is fairly standard for an average pair of phones or “cans” today.  Eventually I came across some V-Moda LP’s at Costco and got those, but needless to say, they did not get  bed time duty.  They are some pretty good headphones, but I found the around ear cushions were just shy of going around my ears which limits them to short term use.  Sound quality is definitely better than the cheap Sentry stuff and the ability to use them with the second cord that includes a mic that works with an iPhone is a nice feature and the storage case is added plus.  But like I said, these are a little over the top for use while falling asleep. So I kept looking at reviews on Amazon and everything was about $80+ for something with 4 stars, but then I came across the Panasonic RP-HTF600’s (600’s for short).  Most every review gave these guys 5 stars with very few low ratings.  The specs for the 600’s are 10Hz – 27,000Hz at $29.99 price point.  I have been listening to the 600’s about six months now and they find their way from the bed room to the computer room so I can listen to music, binaural beats or whatever else I decide to play through them.  I have played with EQ settings and find that I like the sound they produce without any EQ adjustments.  I have not really found any frequencies where they fail to produce a clear sound.  The mids are not overbearing or muffled and the highs are very crisp with super smooth bass.  One of the best traits so far are the new deeper tones I hear when listening to binaural beats.  Some of the self hypnosis sessions that I have listened to for years have been rejuvenated like old people in the movie “Cocoon”.   These are Amazon prices as of May 18, 2014.  But you said $50, well the additional cost comes from Beyerdynamics replacement velour ear cushions.  While the original ear cushions are okay, they would get a little sweaty being  pleather.  The Beyerdynamics velour cushions were recommended in a couple of reviews.  And I am very glad I added them to the order.  So far these cans have holding up on their own four over six months.   I still want to try and make them a little more closed versus semi-open by inserting some blue tack on the inside vents; leaving only a small opening for ventilation.  If I make that change, I will post an update on the improvements or detriments.

Some other sets I own in addition to the V-Moda and Panasonics are: Bose Mobile in-ear (clearance shelf at Sam’s Club), Bose OE (Target clearance item), Monoprice DJ headphones,  Radio Shack Realistic Pro-35a (Titanium) also sold as Koss KTX Pro 1 on Amazon, and a pair of Koss from Walmart that have been discontinued.  The Bose in-ear are good with my iPhone, but they are not going to win any contests even though I do find the microphone better than most in-ear mobile headsets.  I have the OE Bose at work and find they are bassy with very good highs if my old ears can hear them, but they are missing some mid range that make the overall sound experience a little muddy. The Radio Shack Pro-35a could almost be a standard reference which is saying a lot when you think about the current Koss version costs less than $12.  I picked up the Monoprice headphones on a daily deal or sale and got two more pairs for my nieces as gifts.  They were 10 and 13 at the time I gave them to the girls and they have not broken either set in two years.  I like the Monoprice as far as sound experience goes, but the headband tended to cut into my head and the pleather ear cushions caused some sweating.  I made two mods to the Monoprice to improve them.  One, I took the ear cushions I removed from the Panasonics and put them on the Monoprice set and I took a strip of rubber cut from an old bicycle inner tube to create suspension headband inside of the original poorly padded head band.  If you take a look at the Koss Pro1 photo, the homemade inner tube head band looks sort of like that on my modified Monoprice cans.

No doubt I will probably keep looking at new cans and probably buy some higher end models if I come into some gift certificates or money.  But unless I am a producer behind a studio sound board for a professional musician, I can’t see spending upwards of $150 for my old ears.  So now I wait for the new Audio-Technica ATH-M50x to come down to my price range.

UPDATE 20Nov2017 – Venturing out of the $50 range & $10
So it has been a little over three years since I blogged on headphones and considering additional headphones now in the stable; I think it is time for an update.  I finally picked up some of the Audio-Technica ATH-M50x when they were on Prime Day or a Black Friday sale.  I also grabbed some Bose Around Ear QC-25’s from the Bose outlet.  And I had the opportunity to try some Bose SoundTrue Ultra in-ear headphones after my Bose in-ear Mobile’s were stolen from the DJ booth at the local skating rink where do my every Wednesday 80’s night gig.  The SoundTrue Ultra in-ear were totally isolating like I was wearing noise reduction plugs for the gun range.  So I ended up finding the Bose ebay store and picked up another pair of the old in-ear Mobile model.  When I made the return of the SoundTrue Ultra by taking advantage of the Bose return policy, I updated to the new Soundwave Companion around neck speakers from Bose.  If you remember the old Bones that you could wear around your neck, think of them updated with bluetooth and some decent sounding speakers with bass ports.  If you are considering buying Bose stuff from the outlets, the return policy is 30 days for refunds and 90 days for an exchange.  The latest find for a true set of cans are some Sennheiser HD 380 pro’s I found at the local pawn shop and negotiated down to $50.  For alternative to the Bose Mobile’s, I also picked up some of the highly rated Panasonic Ergo-Fit in-ear, which I have to admit have pretty impressive sound for an under $10 price point.

A quick grade (A-F) on each one while listening to…
1. Barber -Adagio for Strings (Overall sense of sound field capacity)
2. Delta Rae – If I loved You (Ability to carry vocals)
3. Lindsey Stirling – Crystalize (Bass response and mid-range separation)

• Panasonic Ergo-Fit in-ear:
  1. – B (good overall sound picture for style of headphone)
  2.- C (volume seemed weak, but decent vocal renditions)
  3.- B (amazing bass response for in-ears)
• Sennheiser HD 380 pro around ear:
  1. – C (seemed to be a little hot in the midrange – correctable with a little EQ)
  2.- A (very clear definitive vocals with full coverage of the light raspiness of her voice)
  3.- A (great bass response with overpowering the violin even with the ultra low frequency bass at 3:05)
• Audio-Technica ATH-M50x around ear:
  1. – A (very even across all ranges – best at carrying the crescendo up the 6:00 minute mark)
  2.- B (vocalist seemed a little distant, but good balance between instruments and vocals)
  3.- B (separation between low frequencies and violin was good and bass was a tad better than the 380’s)
• Bose QC-25 around ear noise cancelling connected via Bolle & Raven
  AirMod Wireless Bluetooth Adapter:
• 1. – A (the Bose haters will say what they will, but these render this adagio with the gracefulness it deserves)
  2.- A (clear vocals and almost overbearing with full band and backup vocals)
  3.- A (crisp violin with ability to carry the bass off without distorting the other frequencies)

At some point, I will do a proper review of the Bose Soundwave Companion.  I sit in a semi-open work environment and have these on most of the day for listing to music and I can usually keep them at a low enough volume that no one around me hears the sound as the speakers are directed up at my ears.  I guess the only issue is, as they become unnoticeable to me, I get some looks when I forget to take them off and leave my desk when headed out for lunch or a break.  I have been told they look like a mini neck rest for travel.  When the volume is up on these they are pretty decent for full coverage of most all of the frequencies, but the mobile app does allow for cutting bass if needed.