|
|
ICC Home / Members / Meetings / Peer Support / Documentation / Projects
Minutes of June 10th, 2010 ITAC-NI Meeting: |
Link to ACTION ITEMS from meeting AGENDA: CALL TO ORDER: This meeting was scheduled in CSE E507 at 1:30 pm on Thursday, June 10th and was made available via videoconference with live-streaming and recording for future playback. Prior announcement was not made via the Net-Managers-L list, so our broader audience may have been unaware of the meeting. The meeting was called to order by ITAC-NI chairman, Dan Miller, Network Coordinator of CNS Network Services. ATTENDEES: Twelve people attended this meeting locally and there was one attendee via Polycom videoconference. There are no records of how many may have listened into the stream via a web browser using the web interface. Nine members were present: Charles Benjamin, Ron Cigna (via Polycom), Dan Cromer, Tim Fitzpatrick, Craig Gorme, Shawn Lander, Steve Lasley, Tom Livoti, and Dan Miller. Five members were absent: Clint Collins, Erik Deumens, Margaret Fields, Chris Leopold, and Handsford (Ty) Tyler. Four visitors participated as well: Matt Grover, Todd Hester, Brad O'Hara, and Dave Pokorney. Viewing the recording You may view the recording via the web at http://128.227.156.84:7734. Currently, you will need to click on the "Top-level folder" link, then the "watch" link next to the "ITAC-NI Meeting_10Jun10_13.15" item. Generally these recordings are moved into the ITAC-NI folder shortly after each meeting. Cross-platform access may not be available; on the Windows platform you will have to install the Codian codec. Audio archive An archive of audio from the meeting is available. 1) IntroductionsBefore getting to approval of the minutes, Dan Miller wanted to introduce Brad O'Hara who will be replacing the retiring Marcus Morgan. 2) Approve prior minutesNo corrections or additions were offered and the minutes were consequently approved. 3) CNS request to sunset 802.11b wireless in order to improve overall performance3-1) The case for discontinuing 802.11b wireless serviceMatt Grover began the discussion by passing out a handout which graphed the last four weeks of wireless usage broken down by the various protocol types:
Matt noted that there is an inherent speed limitation just from running in 802.11b compatibility mode and that the actual connection of a client over that protocol further drops the throughput potential considerably for all users. Consequently, CNS wants to determine a "sunset" date upon which they will discontinue offering 802.11b service. 3-2) 802.11b currently represents only 0.05% of our wireless usageSince the proposed discontinuation will affect some users, CNS studied the numbers using various wireless protocols currently, as indicated in the above graph, to determine the extent of that. A report run from April 18th to May 13th captured the last few weeks of the Spring semester indicated 196 unique 802.11b clients out of rough 41,000 overall; this comes to about 0.04% actually using 802.11b. Another analysis done during the early Summer term indentified 136 unique 802.11b clients out of roughly 25,000 overall (~0.05%). Based on low usage and the potential advantages overall, CNS is considering dropping 802.11b around the end of the Fall term. 3-3) Potential concernsTom Livoti mentioned a couple of concerns. If CNS is using any of the Cisco 7920 wireless phones, he pointed out that they only support the 802.11b protocol. Tom also said that they had some specialized medical equipment, particularly at Shands, that uses wireless and still only supports that older protocol. FDA rules make for slow changes in such devices because the manufacturers have to submit any changes to them for approval and that can be a long drawn-out process. That said, Tom is has been pushing to discontinue 802.11b for quite some time and is in favor of moving in that direction. 3-4) Could a minimal 802.11b presence be maintained?Dan Cromer asked if 802.11b couldn't remain in one or two locations so that users with no other option might take advantage of that. Matt explained that doing this would entail utilizing a separate controller because the radio settings are essentially controller-wide. Additionally, the data indicate that current 802.11b users are spread randomly across campus; finding a single location which would help any significant numbers of them would be difficult. Dan Miller added that controllers are very expensive and have a capacity of 158 wireless access points (WAPs) in redundant mode; that would be quite an investment just to keep 802.11b going in a few locations. Frankly, it would be less expensive to buy all those users new adapters. 3-5) Advertising the removal of 802.11b serviceTim Fitzpatrick asked how this discontinuation might best be advertised. Matt suggested placing a notice on the login splash page that would inform users of the upcoming change and letting them know of the need to have 802.11g or 802.11n capable adapters by that date. Dan Miller added that they need to get the UF Computing Help Desk plugged into this change and also distribute notice via the NET-MANAGERS-L list. Matt noted that information exists for identifying the 802.11b users. They could be sent directed e-mails notifying them ahead of time about the plans to discontinue the service. Tim suggested that one-on-one communication should be left to the bitter end for when individuals are making pleas for exceptions. Dan Cromer suggested creating a wiki/web page to point folks at that would detail the reason this is being done and the options available for those who might be affected. 3-6) Other mitigation efforts already in placeCharles Benjamin asked Matt if they have considered turning off some of the lower speeds for 802.11b. Matt responded that they already had; 1 Mbit/s and 2 Mbit/s were the first to go and he believes they have it set currently to only allow 11 Mbit/s. Shutting off all the lower-rate signaling provided dramatic overall speed improvements at the time. 3-7) Unanimous approval recommending the removal of 802.11b service near year's endThe committee voted unanimously to recommend that 802.11b service be retired on or about the end of 2010. 4) CNS plan to provision a new SSID using WPA24-1) Current "UFW" wireless describedMatt Grover explained that our current "UFW" wireless connection is wide-open and un-encrypted with authentication being done via a captive web portal using GatorLink credentials. The logon process is encrypted via SSL on the web portal but the layer 2 connection with the WAP is not encrypted. The preferred mode, and what CNS recommends to faculty and staff (and students when they get the chance), is to use a VPN which provides encryption. 4-2) Plans for additional WPA2 secure wireless serviceThe other things CNS has been looking at are the 802.11i iterations, beginning with WPA and now WPA2. CNS wants to move forward with WPA2 both to provide encryption as well as to address an issue with smartphone-type devices. Such devices need to login via the portal and the problem arises because these devices go to sleep after about 10 minutes or so. When the device wakes up the user has to log back in. WPA2 provides the ability at a low level for a lot of these devices to be able to cache the credentials. The big win would be that such devices could reconnect automatically upon wakeup without the user having to go through the "horrid" login screens. This issue was the primary driving force behind this move, but having encryption is a nice plus as well. This can be delivered over existing equipment, they just have to put in an additional Service Set identifier (SSID). The big question is what should they name this new SSID. This will actually be the third global SSID; in addition to UFW there is another non-broadcast SSID currently which is used by WiFI VoIP phones. This third SSID will probably be non-broadcast as well but available for people to configure and use. 4-3) HSC has been providing secure wireless for about two yearsTom Livoti said they are already doing this at the Health Science Center (HSC). Their secure SSID is not broadcast, but the MAC of the device does have to be registered in IPAM. Craig Gorme added that they also have a public SSID for students. This requires GatorLink credentials for access and a posture assessment is done. If the device does not pass that assessment the user is directed into a separate VLAN which only provides access to resources by which they may remedy their non-compliance issues. 4-4) Posture assessment less of a concern with managed machinesMatt said that the big unknown currently for CNS is how to tie this into posture assessment. If CNS was to implement WPA2 at this point, posture assessment would have to be bypassed completely. Tom responded that they do not do posture assessment on their secured wireless; they assume anyone connecting via that method are utilizing a managed device; that gives HealthNet some comfort that they are being patched correctly. Tom said he believes some things are still best managed by policy rather than by technology at this time. 4-5) Clarification of the multiple SSID supportSteve Lasley asked for a clarification on whether each access point could handle multiple SSIDs. Matt replied that this was indeed the case, but those don't all necessarily have to be global. There are some locations on campus where a small group of WAPs get their own SSID which is not visible outside that area. They envision one day having yet another SSID available for UPD use campus-wide. 4-6) Smartphone device users will not have to re-enter credentials at wake-upDan Miller pointed out that he had "misspoken" in his preliminary agenda that this system would utilize GLauth; that is not the case. Matt added that while a separate mechanism is used, a login is still required. Craig pointed out that after the initial login users would not have to re-enter their credentials until they next changed their GatorLink password; this will make access for smartphone users MUCH easier. 4-7) SSID naming suggestionsSome discussion arose over what to name this SSID. UFWS, UFWSEC, and UF-SECURE were among the suggestions made, but it was noted that whatever choice was made it would only be published on an internally facing web site. Matt pointed out that it wasn't for security purposes that this new SSID would not be broadcast. 4-8) Implementation date is dependent on RADIUS configuration tweakingMatt was asked when WPA2 might be implemented; he responded that he didn't know because the holdup was basically on the Remote Authentication Dial In User Service (RADIUS) end of things over which he does not have control. Our redundant RADIUS system is managed by Chris Griffin. Chris is currently tweaking that for other purposes; that work needs to be completed before we can move ahead with WPA2. We are running FreeRADIUS, and version 2 has made some changes that require revision testing to assure stability. RADIUS requires quite a bit of custom coding on the backend and this makes version upgrades a bit tricky. That said, all agreed that this should go ahead as quickly as CNS was able, as it would be a welcome improvement. 5) Update on plans for NAC for campus wireless5-1) Improvements made to central wirelessMatt reported that wireless has basically just been keeping pace with the build-out of Wall Plate. They have been adding controllers and have a total of five blades now in each of two chassis. The build-out is fully redundant and rather than loading a controller up to its maximum number of WAPs the capacity is split between controllers in separate chassis and locations. Thus, should one blade or chassis fail the other could continue to handle the entire load. The NAC is built in a somewhat similar manner. 5-2) Moving to seating density as a criterion for WAP placementCNS just added two new blades which brings current capacity up to around 1500 WAPs. That increase is partially due to preparing for providing higher density wireless for classroom spaces. Rather than providing minimal geographic coverage, they are now aiming for a density of about 1 WAP for every 25 users; so, they are moving from using signal strength as the criterion for WAP placement to a consideration of seating density. Matt provided the example of Matherly Hall which had fairly full geographic coverage with six WAPs; they have now gone back through and picked about 48 locations in which to add new WAPs in that building. 5-3) Multiple vendors offer high-density wireless solutionsCraig asked if CNS had looked at the Maru products. HealthNet and security at HSC have had really good luck with using that. Tom noted that they were doing 145 simultaneous connections in two adjacent rooms using that with six WAPs total. Matt responded that he had read about Maru, Aruba and other options. They all have their takes on the high-density problem and Matt has read contradictory whitepapers touting one or another. Tom said that HealthNet basically just picked one (Maru) and has been pretty happy with it. 5-4) Channel bonding and coverage vs. throughput issuesCraig asked how 802.11n affected the number of channels available. Matt answered that 802.11n involves channel modeling. In the 2.4GHz spectrum you only have 3 channels anyway, so the use of bonded channels is basically limited to the 5GHz spectrum. CNS feels that having a wider number of channels available across the spectrum provides better overall coverage as the controller can better space those out. They favor that method in general over bonding channels for higher top-end throughput. If higher throughput was needed in specialized situations they might reconsider that, however. 5-5) Potential tripling of WAPs counts is not unusualTim summarized that we might be looking at tripling our WAP counts under this new model from roughly 1000 to 3000. Tim asked if HealthNet had done this. Tom responded that they currently have about 1300 WAPs and recent expansions have indeed roughly tripled their counts. Although HealthNet has been careful to specify that mission critical applications should not depend on wireless, the usage continues to grow. 5-6) 802.11a being used to isolate mobile VoIP from interferenceHe noted that Shands (NOT HealthNet) is putting mission critical equipment on 802.11a currently; they had to use 802.11a because it was already so noisy on the other bands. Matt said that they are using 802.11a for VoIP deployments for that same reason and Tom said HealthNet was looking at the same thing. 5-7) Balancing WAPs in relation to controllers and geographic locations; how that impacts mobilityWhen you bring up a new WAP it will attach to whatever controller has the most space on it. That is not optimal, however, especially in large deployments. One downside involves roaming. While roaming works well within a common mobility group, there are limits to how many controllers can be in such a group. Though that number has increased, there are also RF groups which determine when WAPs automatically do their channel alignment via an internal database. There is a limit on how many WAPs can be in an RF group. Consequently, CNS takes their WAPs and groups them into coherent areas where the likelihood of roaming is high. This gets all those WAPs on the same controllers so there is no requirement for traffic to be crossing off to other controllers. It makes for a better overall experience. 5-8) Newsgroup for staying abreast of breaking issuesMatt Grover indicated that this newsgroup is a good place to keep in touch with various wireless issues: the Educause listserv for wireless discussion. 5-9) New management platform: WCSCNS has moved to a new management platform. Matt said that CNS had previously been using Airwave which was acquired by Aruba about two and one-half years ago. That worked very well for a long time because it was good at managing the autonomous systems and the Cisco products were terrible for a long time. When the Cisco product got to about version 6 it was a completely different beast. CNS was approaching 1000 WAPs and the Airwave license was for that number; consequently, they re-evaluated things and decided to go with the Cisco Wireless Control System (WCS); it was a lot cleaner, easier to use, more intuitive and had a faster response time. The handout Matt supplied earlier was generated with WCS. Tom said that HealthNet has been using WCS since version 4. 5-10) Broad plans for future wireless expansionDan Miller indicated that future plans include expanding coverage into the western regions of campus where there is currently little or no networking. This may include a meshed hybrid model to get around having to buy fiber to connect small buildings; those are options which are being considered. 5-11) NAC being coordinated with UF IT SecurityThe use of NAC for UF wireless had last been mentioned at our November 2009 meeting. Dan said that CNS has made some progress with NAC and it is back on the front burner. Matt reported that they had moved to Cisco NAC back when they were looking for a replacement for the old Bluesocket system. All the new controller-based wireless goes into that NAC system. It is currently still doing the old portal login stuff. Matt has been looking at some of the reporting available with it and he met with the UF IT Security group about this recently. The plan is that CNS will keep the equipment running and the UF IT Security group will provide the actual day-to-day profile management. Security has begun playing with the tools to see what is available. The first anticipated step would be to turn on a reporting level with no remediation. They want to avoid adverse user impact. 5-12) End user issues with NAC implementationCraig mentioned that in having to deal with NAC at HSC he had found two main issues. One is that when a software vendor revises a patch, sometime the NAC will report the client as non compliant even though they may be at current patch levels; this is generally due to values recorded in the registry. The other issue is that early adopters of new versions can be detected as non-compliant until the NAC is made aware of the new release. They have consequently had situation where a user had to downgrade in order to gain access. Matt mentioned that John Sawyer is in close contact with Avi Baumstein on these matters. There is a Clean Access Listserv that discusses such issues and various workarounds. 5-13) NAC may eventually make its way to wired portsTom asked if the plan was to use NAC just for wireless. Matt responded that the thought is to eventually make it available for wired ports if people wanted that. There are public access wired ports across campus with are still held by the Bluesockets. CNS has another set of NACs coming in now that they are putting in for out-of-band to handle all this edge stuff that will perform VLAN shifting on the ports. 5-14) DHNet using Safe Connect for posture assessmentCharles mentioned that DHNet has implemented a product called Safe Connect from Impulse which they now use for posture assessment. They had looked at Bradford, Cisco and Impulse and went with the latter because it didn't involve buying a bunch of appliances and was thus less expensive. This product makes sure the OS is up-to-date, that the machine is running antivirus software, etc. DHNet provides a grace period during which the user must install antivirus and get their OS up-to-date. This puts up a splash screen web page instructing the user where to go to update things. If that is not done by the end of the grace period that client is moved to a VLAN where they can only reach remediation resources. They implemented this with Summer A and it has been very positive; they had very few UFIRT notices during that period. It will be interesting to see how things go with Summer B when new students begin to arrive. Tom said that HealthNet handles things similarly but does not provide a grace period; non-compliant machines are immediately quarantined. They stopped short of providing remedial services because of the liability involved with working on non-HSC equipment. 5-15) Slow, gradual phase-in of NAC posture control foreseen for UF wirelessMatt said that UF IT Security plans to phase things in very gradually. Initially they will just check for malicious agents that could cause harm to the network. Matt mentioned that getting the UF Computing Help Desk up-to-speed will be very important prior to going further with posture assessment, as they will be the ones dealing with issues that arise as a result. 6) Other topics6-1) Update on centrally provided SSL certificatesThis matter had last been discussed at our March 2010 meeting. 6-1-1) Review of previous investigations Craig noted that Chris Leopold and he were tasked with investigating the best route to take for obtaining security certificates for the campus as a whole. Doing it ourselves via Open Source or Microsoft was determined to be too expensive. Partnering with a certificate company was another option which was investigated; a wide variation in pricing structures was found across the companies which offered that. In doing this research Craig discovered that the UFAD and CNS already have a relationship with one vendor; they pay a certain amount each year and get a certain number of certs back. 6-1-2) New and attractive option recently made available In the meantime an organization called InCommon is planning to offer certs to higher education for a minimal cost beginning this summer. It would appear this would cost UF roughly $15,000 a year for unlimited SSL certificates. Estimates are that UF is spending roughly $90,000 a year currently, so this would be a significant overall savings. 6-1-3) Recommendation from our committee would assist in financial justification Tim Fitzpatrick felt that a recommendation from the committee that stated the financial case would be very useful. He asked about the mechanics of actually acquiring the certs and whether or not it would require central administration for that aspect. Craig believed that as long as a certain subset of people, say the network managers, were authorized to request these that it could be distributed with each unit acquiring their own via a web portal. He would have to investigate further to know for sure if that was possible. 6-1-4) Concerns for a stable and sustainable funding source Would the recent Microsoft Server Licensing be a model to emulate? Dan Cromer noted that UFAD had gone through a similar situation where Microsoft Core Server licenses were purchased centrally in a move which was estimated to save roughly $80,000 per year. However the funding is handled, Dan feels that this same model could be applied to the our situation with certificates. Microsoft Server Licensing based on one-time funds Tim Fitzpatrick commented on this saying that there is a lot of wishful thinking that would add the eCAL and Server licensing costs to the RCM Microsoft Campus Agreement. Tim cautioned that there is no mechanism or process to make that happen right now; come next October when that bill is due, somebody is going to have to pay for it if they want to continue these services. Continued concerns over eCAL portion of the Microsoft License Dan Cromer said he is convinced the eCAL portion is already covered but Tim believes that Mike Conlon, who feels these are important services, has managed to pay for these things via one-time monies in the hope that somehow someway some magic will occur and it will be paid for from the top. Tim has been through the UF RCM budget process for the last six months and the base Microsoft Campus Agreement was a major step to get bundled into RCM but these two add-ons are not part of that. Tim believes that people are very relieved that the base is now funded, which is a good $600,000 plus story. There is an additional $150,000 that is hanging loose. 6-1-5) Additional investigation plans Getting back to the topic at hand, Craig said that we would need to figure out a reliable continuing funding source if we went the InCommon route. Tim commented that if we are talking about $15,000 with a 6x ROI, CNS would fund it. Tim does not have the $150,000 needed for these Microsoft additions, however. Craig said that he will contact InCommon and get the cost and other details and report back to the committee. 6-2) Wake-on-LAN6-1-1) WoL solution sought for end users using RDP On behalf of the Green IT initiative, Dan Cromer is hoping to find a technical solution to providing Wake-On-LAN (WoL) functionality across subnets. 6-2-2) Directed broadcasts deemed a security concern Matt Grover responded that the problem is the mechanism of delivery. WoL works via a "Magic packet" that uses directed broadcasts. In the late 1990s Smurf attacks and Fraggle attacks became very popular denial-of-service (DoS) attacks. Since that time almost all the router vendors have gone to a default configuration of not allowing directed broadcasts. While you can turn it on, people generally don't do it. 6-2-3) Agent-based solutions exist CNS has kept directed broadcasts turned off on the routers for some time and people have come forward several times wanting to do WoL. CNS has responded to those by saying you can do WoL but you can't do it with directed broadcasts. There are solutions, however, which are unicast based. Cisco's EnergyWise is one product that does this. Microsoft SCCM and 1e are other examples which provide a unicast solution. All of these basically utilize an agent which resides on the local subnet along with a central control console. A web application could potentially tie into the central console to control this all and let individuals wake up specific machines. The central console then sends out signals to the local agents which are embedded in the subnets; this is how it gets around the directed broadcast issue. The agent on the subnet then sends out the WoL broadcast packet. 6-2-4) NIC-based solutions coming in a few years There is a new generation of NICs coming out now where the network card itself actually carries a small IP stack and the card will be able to be communicated with even when the machine is off. Those aren't out yet, but that may provide the next generation of WoL. 6-2-5) Providing WoL capabilities to end users might entail additional complications Matt added that the use of WoL by computer administrators for patching, etc. was one thing, but to open that up to end users so they could wake their machines so they can remote desktop in might be a much more complicated matter. 6-2-6) VDI suggested as the enterprise solution Tom suggested that a VDI solution would be a better solution. While Dan Cromer agreed that VDI might be a long-term strategic goal, he would still appreciate an investigation of the WoL options. Matt said that there are various ways of doing this but they all come down to having a local agent. There are solutions out there and it is simply a matter of choosing one and pursuing it. Dan Miller said there is also the question of cost because you need an agent on each subnet. Matt said that this was the case unless you went for a homebrew solution; that raised the question of whether one really wanted to promote such solutions in an enterprise environment. 6-2-7) Working homegrown solution could be better documented Placing an agent on a PC which was maintained by a local admin seems like a workable solution and that is what Elwood Aust's solution uses. Shawn Lander suggested it would be worthwhile to document that solution on the web somewhere and then have CNS point folks to that when approached about WoL solutions. That would encourage some standard practice even if an enterprise-level solution was not practical at this time. 6-2-8) DHNet has agent-less solution which might be investigated Charles mentioned that he believed DHNet has already adopted a scripted solution for doing this that did not require an agent but rather involved increasing the flush timer on their arp cache for their routers. He said that he would get the details of that and send it to the committee. 6-2-9) Other do-it-yourself solutions being investigated Shawn added that he had done some research and located a wolcmd.exe that creates that Magic packet and forwards it off to the machine and wakes it up across subnet boundaries. They tried it once and it worked but he has not done any further testing. He does, however, want to do a whole lot more investigation. Matt suggested that this likely works via the method that Charles mentioned. He suspects you would need a very long hold time on the arp table; otherwise the packet would be meaningless by the time it got there. Shawn was thinking that this could be scripted and used in a web application that met Dan Cromer's desire to allow end users to wake up their own machine for the purposes of RDP. The meeting was adjourned just a bit late. Action Items
Next MeetingJuly 8, 2010 |
last edited 15 June 2010 by Steve Lasley
