I’m happy to report that, with a lot of help, I was able to get a basic framework in place and working yesterday for our new Mobile POS effort to connect to a store’s IAP. We’ll be onboarding these iPod units with ClearPass OnBoard, downloading unique cert per device as well as network settings to enforce the use of EAP TLS. Then with the same SSID the device will auto-connect with a different role on the IAP.
Couple things I still need to work on:
1. Why isn’t forced redirect working for the onboarding role specified on the IAP (ClearPass is handing it back to IAP correctly)?
2. Need to set up API account on AirWatch MDM and configure CPPM to point to it, then lock down the authentication to require the device to be enrolled in the MDM.
3. Lock down firewall rules on the IAP for the onboarding and mobile-pos roles. If you have a captive portal enforcement redirecting to an external site, do you have to allow traffic to that site? Or is it inferred automatically that traffic is allowed?
What am I forgetting? Any hints/tips/tricks? Thanks to @sethfiermonti and others for the help!
Swack
Twitter: @swackhap
Archives
VMware View Problems with 64-bit Windows 7 Virtual Desktop
We’ve been growing our Virtual Desktop Infrastructure (VDI) quite a bit lately, and as a result I’ve taken ownership of a shiny new Windows 7 64-bit virtual desktop. Unlike the 32-bit Win7 VM I used before, though, this one has been giving me trouble.
The trouble starts when I am trying to reconnect to the already booted VM from a machine other than the last one I was on. Specifically, I use Windows 8 64-bit at work on a Dell tower with 4 monitors (two dual-monitor graphics cards). I use my VDI VM all the time from that machine on all four monitors. I also have a Macbook Pro (MBP) that I take to meetings and use outside the office.
Sometimes (not always) when I re-connect to my VM from my MBP I get a black screen with a mouse cursor and nothing else. After waiting a minute, I either disconnect or quit the View application and re-launch. Reconnecting the second time gives me an error indicating that desktop resources are busy. When this happens I cannot even connect via RDP, let alone through the usual way via the View broker. I attempt to restart the guest OS through vCenter but it never actually reboots unless I power cycle the VM in vCenter.
I worked with VMware Support but unfortunately haven’t been able to fully solve the problem. The View support folks have looked thoroughly at our setup and don’t see anything that could be causing problems. They handed me off to another group that was able to analyze a crash dump of my VM after the problem occurred, but they could only tell me that it appeared the VM was trying to use 3D rendering services of some sort (if I remember correctly).
As a workaround, I now re-size my View window on my desktop before disconnecting so it is intentionally smaller than the laptop from which I usually connect. This seems to have helped but it’s rather frustrating. No other users have reported having the same issue, but there are currently no other VDI users with more than 2 screens. I should also point out that I’ve observed the same behavior when I connect from my home Windows 7 machine. It doesn’t seem to matter if I’m connecting to the internal View servers that only use AD authentication or if I use the Secure Gateway View server that requires 2-factor authentication and tunnels secure PCoIP.
Based on all the evidence it seems my problem is related to having 4 monitors, but VMware support has been unable to identify the root cause and neither have I. If you have ideas, I’d love to hear them. Hit me up on Twitter (@swackhap).
Network Disruption Causes vCenter DB Corruption
First off, I am NOT a VMware expert by any stretch of the imagination. I AM however learning a lot working with some smart folks in virtualized servers and desktops.
A network engineer (who shall remain nameless) was making some changes to the network infrastructure last night and unfortunately experienced an outage. Due to an ongoing network migration from Cat6500 to Nexus 7k/5k/2k, all ESX hosts are now connected to Nexus FEX but iSCSI storage is still on old Cat6500. Outage basically cut connectivity between Nexus-connected hosts and iSCSI storage.
As users started trying to login to their desktops in the morning, we started getting reports of problems. Our VDI vCenter showed 4 of our 20+ hosts disconnected or not responding. We ended up power-cycling those, one at a time, and once they came up we were able to re-connect them back into vCenter.
The next big problem was that the profile server, which runs as a VM in the VDI infrastructure, was hung while attempting to migrate. We rebooted vCenter which orphaned the profile server, but we found we were unable to browse the particular LUN where that VM’s datastore existed to add it back into vCenter. At that point, we engaged VMware support and spent several hours on WebEx troubleshooting storage connectivity problems (tail -f /var/log/vmkernel and some other stuff). By the time I left in the early afternoon we had identified half a dozen hosts that seemed to be having iSCSI problems based on what VMware Support was seeing in the logs, and we rebooted those hosts one at a time to minimize end-user impact.
I had to leave before the fun was all over, but found out afterwards that apparently a couple of the hosts got duplicates of the datastore IDs on them when they recovered from the outage overnight. Once that happened, the database was somehow corrupted with the wrong datastore information. It was apparently cleared by removing the two particular hosts from vCenter and adding them back in, thus giving them new datastore information.
Like I said, I’m not a VMware expert but I’m learning more each day. You ever experience something like this? Who else is doing VDI? Leave your comments below or find me on Twitter (@swackhap).
Cisco Live Tips and Tricks
Hard to believe it’s been over a year since my last post here. As I’ve learned in life though, sometimes you have to forgive yourself for your failings (in this case, not blogging for a while) and then you can continue to improve on yourself.
I recently attended Cisco Live 2012 in San Diego. After attending 9 times (thereabouts), I figured I’d share some ideas/thoughts/tips.
First off, have a 10-foot extension cord when traveling and when attending sessions. Many breakout sessions and labs are in rooms that have power strips available, but some do not. If your extension cord has a 3-prong plug, have a 3-prong to 2-prong adapter with you just in case you need to plug into an old outlet.
The World of Solutions (WoS) is the area where Cisco and their partners set up booths with all sorts of goodies. The first night it may be okay to wander a bit, but at some point you need to HAVE A PLAN. Look over the list of exhibitors. Think about your goals for the conference. Are there particular problems at work that you’re trying to solve? The WoS is THE PLACE to find the solution. Print a map of the booths and circle the ones you want to visit. Then cross them off after you’ve been there. Stay focused!
Some of my favorite places in the World of Solutions:
- Walk-In Hands-On Labs – Great place to spend a few minutes learning new skills and practicing configurations on a plethora of systems.
- Cisco Booth – Incredible opportunity to learn about almost every product/system/solution that they sell.
- Social Media Hub – For the first time this year, the folks behind all the social networking for the event, such as the @CiscoLive Twitter account, set up shop to show off the top Tweeters and give people a place to lounge for a bit.
- Technical Solutions Clinic – Basically an engineer’s Heaven-on-Earth, there are several dozen whiteboards surrounded by some of Cisco’s smartest Technical Marketing Engineers and TAC folks. What problem did you have at work you’ve been trying to fix? They’ll solve it for you.
The Cisco Live mobile app makes navigating the conference a snap. View your schedule of sessions, browse WoS exhibitor listings and conference maps, and complete evaluations of sessions you’ve attended, right on your phone or tablet. The evaluations are incredibly important and Cisco takes them very seriously.
I’m very excited to have attended Cisco Live once again, and hope to continue doing so. I consider a week at Cisco Live equivalent to about 3 weeks worth of training.
If you have any questions, comment below or hit me up on Twitter (@swackhap). Cheers!
RSA SecurID Soft Token for iPhone – A Better Deployment Method
Working in a retail environment makes you think really hard about security, especially in light of what happened with TJ Maxx a few years ago. Using credit cards in retail is a privilege that we only get to keep if we follow the Payment Card Industry Data Security Standard (PCI DSS). One of the requirements of PCI is related to two-factor authentication for remote-access to your corporate network, and one solution for this is RSA’s SecurID authentication product.
RSA SecurID supports many form factors, both hardware fobs/cards and software-based on PCs and mobile devices. This post focuses on mobile device soft tokens, particularly iPhones.
For quite some time, the process to get a soft token on an iPhone looked something like this:
- User downloads RSA app from App Store
- Administrator log in to RSA SecurID appliance and assign soft token to user
- Generate CT-KIP credentials for web download, e-mail special link to user
- Connect user’s iPhone to internal corporate network
- Have user open e-mail on the native iPhone app and tap the link
- iPhone communicates directly with RSA appliance
- Token is now present on iPhone
Step 4 is required because of the way RSA has locked down its current appliance. The only way for an iPhone to connect to the RSA appliance from outside the corporate firewall would be to somehow expose the appliance itself to the Internet, either directly or through a Microsoft ISA proxy server. This is one of my big gripes about the appliance, but it’s a great solution for the most part.
The most recent update to RSA’s iPhone app has greatly improved the token deployment process. Now the process looks like this:
- User downloads RSA app from App Store (no change)
- Administrator log in to RSA SecurID appliance and assign soft token to user (no change)
- Issue token file (.sdtid) and save to desktop
- Use RSA-provided TokenConverter.exe on command line to convert .sdtid file to a long string of characters, then e-mail that to user
- Have user open e-mail on the native iPhone app and tap the link (no change)
- Token is now present on iPhone
The new method precludes the requirement for the iPhone to communicate directly with the appliance, which is a huge improvement. The TokenConverter.exe is available for download from RSA’s website for both Windows and Linux, and also works with Android and Windows Mobile, though I’m not sure if it works yet for Windows Phone 7. Of course, the same token deployment process I’ve described above works for any iOS device (iPod Touch, iPad).
Switch Flooding 101 – Troubleshooting Case Study
Remember the first time you learned the basics of bridging? Dig deep in your memory and think back to the basics. With helpful verification from my co-workers and Aaron Conaway (on Twitter as @aconaway), I verified that some “crazy” behavior I saw today on our network was, in fact, “normal,” albeit undesired.
I’ve been troubleshooting some very strange behaviors on our network lately. I suspect some (all?) of them have to do with our fairly old Cisco Catalyst 6500s with Sup2’s and Sup1a’s in our data center, as well as the dinosaur Catalyst 2948 access switches in our closets. There are times when our monitoring system throws alerts saying it can’t ping certain devices. But minutes later, things return to normal. (Don’t you just love intermittent problems?) One tool that any good network engineer will consider when dealing with such a problem is a packet capture product such as the ever-popular Wireshark.
When I fired up Wireshark on my desktop computer, I had to filter through the muck to see what was going on. By “muck” I’m referring to the traffic I don’t care about, such as the traffic my box is generating, as well as broadcast and multicast. I slowly added more and more exceptions to my capture filter (see below) to narrow the scope of my capture.
My Wireshark Capture Filter: not host [my IP address] and not host [directed broadcast for my subnet] and not broadcast and not host 239.255.255.250 and not host 224.0.0.2 and not host 224.0.0.251 and not host 230.0.0.4 and not host 224.1.0.38 and not ether proto 0x0806 [for CDP] and not ether host 01:00:0c:cc:cc:cc [for HSRP] and not host 224.0.0.252 and not host 228.7.6.9 and not host 224.0.1.60 and not host 224.0.0.1 and not host 224.0.0.252 and not stp and not host 224.0.0.13 and not host 224.0.0.22
Once I filtered out enough to see more clearly, I noticed a TON of syslog (UDP 514) traffic destined for another host on my subnet. After scratching my head and consulting with co-workers, I started looking at the mac-address tables (or CAM tables). My upstream switch didn’t have a CAM table entry for the mac address of the syslog server. Neither did it’s upstream switch. In fact, the Cat 6500 directly connect to the syslog server didn’t have a CAM table entry for it.
Checking the timeouts for the CAM table on one of the CatOS switches gave us this:
CatOS-Switch> (enable) sh cam agingtime
VLAN 1 aging time = 300 sec
VLAN 2 aging time = 300 sec
VLAN 9 aging time = 300 sec
VLAN 17 aging time = 300 sec
VLAN 18 aging time = 300 sec
VLAN 20 aging time = 300 sec
VLAN 21 aging time = 300 sec
VLAN 25 aging time = 300 sec
[snip]
Similarly, the Cat6500 running Native IOS showed this:
NativeIOS-Switch#sh mac-address-table aging-time
Vlan Aging Time
—- ———-
Global 300
no vlan age other than global age configured
Apparently, this syslog server is so quiet, so stealthy, that it doesn’t transmit ANY traffic for more than 5 minutes (300 sec) at a time. After 5 minutes, the CAM table entries timeout, and all traffic destined for that server gets flooded to every port in the VLAN throughout our trunked network.
One way to prevent the flooding would be to put static CAM table entries in all the affected switches. Perhaps an easier solution is to configure the syslog server to generate some traffic at least every 5 minutes or less.
I’m not sure if the flooding is causing the other strange behaviors we’re seeing on our network, but this has been a good learning experience and reminder for me about the basics of Layer-2 networking.
Any other troubleshooting ideas you would use for a situation like this? Comment here and/or hit me up on Twitter (@swackhap).
Contacts Consolidation
I don’t know about you, but I have contacts everywhere. I’ve got Exchange with Outlook at work, Google Contacts (to go along with Gmail and Google Voice), Facebook, Twitter, and Linked In. There may be others but I spent about 30 minutes and pulled together all my current contacts from all these sources last night. Here’s how I did it:
- Outlook: Exported all contacts as a CSV file. Cleaned it up and imported into Google Contacts.
- Facebook: I found a post that explained how to use a Yahoo account to import Facebook contacts. I then exported as a CSV and, again, imported into Google Contacts.
- Linked In: Under the Contacts listing, there’s an easy-to-use “Export Connections” link. Exported to CSV and, you guessed it, imported into Google Contacts.
- Twitter: Found a nice service called MyTweeple.com that has a handy tool to export all contacts to a CSV file. Imported into Google Contacts.
By now you see a pattern developing. Since I use Gmail and Google Voice so heavily, Google Contacts is a natural repository for all my contacts. It also allowed me to import custom column fields, like “TwitterName”, so I have all my tweeps listed in my Google Contacts with their “twittername” as a Note attached to their details.
Another great thing about Google Contacts is that it is great at finding and merging duplicate contacts. As you might guess, there are many people that I follow on multiple social networks, so merging duplicates is a must for me.
How do you keep your contacts organized?
Find me on Twitter at @swackhap.
Don’t Drink and Drive; DO Geekout and Drive
I’ve been listening to Pandora on my iPhone while driving to and from work for weeks now, and I love it. I am very musically oriented. But I’ve saturated myself with awesome music for now. I wanted something different to occupy my time in the car. So I started searching for some interesting technical podcasts to listen to. Here’s some great ones that I found:
Packet Pushers Podcast (http://packetpushers.net/) – Roundtable of network engineers talking about the week’s happenings in the networking industry
Tech News Today (http://twit.tv/tnt) – Amusing daily look at technology news from different sources, quite professionally done
vChat (http://www.vmwarevideos.com/vchat) – Fantastic discussions about VMWare
Wireless LAN Professionals (http://wirelesslanprofessionals.com/category/podcasts/wlw/) – Helps me keep up with wireless technology in the enterprise
What other podcasts do you recommend? Tell me on Twitter @swackhap!
Swacknet2 