|
|
ICC Home / Members / Meetings / Peer Support / Documentation / Projects
Minutes of March 12th, 2009 ITAC-NI Meeting: |
Link to ACTION ITEMS from meeting AGENDA: CALL TO ORDER: This meeting was scheduled in CSE E507 at 1:00 pm on Thursday, March 12th and was made available via videoconference with live-streaming and recording for future playback. Prior announcement was made via the Net-Managers-L list. The meeting was called to order by ITAC-NI chairman, Dan Miller, Network Coordinator of CNS Network Services. ATTENDEES: Twelve people attended this meeting locally. There were two attendees via Polycom videoconference and there are no records of how many may have listened into the stream via a web browser using the web interface. Eight members were present: Clint Collins, Dan Cromer (via Polycom), Tim Fitzpatrick, Stephen Kostewicz, Steve Lasley, Tom Livoti, Dan Miller, and Handsford (Ty) Tyler. Six members were absent: Charles Benjamin, Erik Deumens, Craig Gorme, Shawn Lander, Chris Leopold, and Bernard Mair. Six visitors participated as well: Avi Baumstein, Dennis Brown (via Polycom), Jim Hranicky, David Huelsman, Todd Hester, and Marcus Morgan. Viewing the recording You may view the recording via the web at http://128.227.156.84:7734. Currently, you will need to click on the "Top-level folder" link, then the "watch" link next to the "ITAC-NI Meeting_12Mar09_12.15" item. This will likely be moved into the ITAC-NI folder shortly. Cross-platform access may not be available; on the Windows platform you will have to install the Codian codec. Audio archive An archive of audio from the meeting is available. 1) Approve prior minutesNo corrections or additions were offered and the minutes were approved without further comment. 2) HealthNet IPAM updateAvi Baumstein was on hand at Dan Miller's request to provide an overview of the IPAM project.
2-1) IPAM project overview2-1-1) The origins of IP management at HSC Avi first started working at the Health Science Center shortly after the creation of HealthNet. At the beginning there was little coordination; each group received a hunk of IP addresses which were deployed manually to individual machines. There were continual changes which necessitated revisiting each machine to be renumbered. Before long, Avi implemented a Bootp server for his own use, which helped with IP management considerably and others started doing the same thing. Then in the late 90's, HealthNet in conjunction with Shands implemented a DHCP server that covered the entire network. One of the unique things they decided to do at the time was to require that MAC addresses be registered rather than offering leases to anyone who plugged in. Gary Bennett created a web-based script which authorized people to go in and add reservations. That system has been in use now for over a decade. 2-1-2) Reservation data grew bad over time Conversations about replacing this system had occurred a number of times in the past without result. When they started looking at their network security requirements a few years ago, a number of problems were discovered which increased the interest in improving their IP management system. One was that the data in the system became bad; there was a process to add information to the system but not to update or remove it. 2-1-3) Security incident response was hampered When security incidents arise, the best way to find things on the network is to look through the DCHP data and figure out who it was who submitted the MAC address reservation. They can then go find that person and have them locate the device in question. This was made difficult by all that bad data and the fact that computers get moved between departments over time. Avi would have to remember the history of the folks who created the reservations and try to figure out where they were at the time--as people move around as well. Avi was practically the only one who could figure these things out because he was the only one who remembered who worked where at what times. This led them to work seriously with Shands to put in some requirements and find something to replace this. 2-1-4) Security zones implementation Along that time they also began looking at other network security initiatives. The other major concept they began developing was that of security zones; the idea being that there would be different zones in the network which were aligned with firewall rules at the HealthNet boundary. They decided that the zones would have to be based on the hardware rather than the user's identity--which is where most products focus their attention. They found that individuals would use different computers for different purposes. The computing needs and requirements for protection differed by computer, not by the individual using them. This led them to consider adding a field to the MAC address database which listed the zone in which the device should reside. The thought was that they could then somehow figure out a way for the network to apply a device to its proper zone. At first they didn't really know how to do that, but after a year's consideration they got the idea of tying this into the IPAM system, making that system the collection point for that information. So this was another of the drivers behind this entire project and was why the security office became involved with it: their concerns about incident response and getting good data for their zone registration system. The security office has been helping to coordinate the work on the Health Sciences Center side, working with and training the various units, while Shands has been doing the technical implementation. 2-1-5) The search for a management solution They looked at every IP address management system they could find and none of them seemed well suited for HealthNet's environment. They required the capability for delegating responsibility to their various IT groups for adding, updating and maintaining the MAC reservation database. Support for this feature was difficult to find because the available products are all intended for either extremely hierarchical corporate environments or particularly for large ISPs. The system they finally decided on is from Bluecat Networks as it came the closest to matching their needs. 2-1-6) Policy support Next they began looking at policies which would be required to support this. About a year ago they implemented some policies and standards regarding Network Security Authority and Responsibility. These detailed who can have addresses, what they can be used for, how you get them, how you must keep information related to these things up-to-date, etc. That ensured that they had well documented backing for project implementation. 2-1-7) Recent temporary hold-up They have been working with Bluecat professional services since sometime last Fall to get the system up and running; Gary Bennett has been doing the bulk of that by far. The cutover was originally planned for February 15th but performance issues in the user interface were discovered that pushed things back. Those issues have since been corrected via a code update and cutover is now scheduled for March 22nd. This very morning 30 IT folks were assembled for a stress test of the system performance; most operations were handled fairly well. Multiple full-database searches were able to bog things down, but not at usage levels anywhere near those expected in production. 2-1-8) The data cleanup process One of the main aspects of this implementation has involved data cleanup. Units were asked to develop an inventory of every device connected to the network by filling out a spreadsheet which was provided. They started with over 100,000 records in the database. They then removed any devices which had not received a lease within the last six months. That alone excluded a large number of records. They then reconciled the results from their inventory request against the remaining database records to arrive at lists of devices that had and had not been claimed. The orphan list was distributed back to the units so they could have the chance to identify any devices which may have slipped their notice. After several iterations of this they arrived at what they believe is a very clean set of data of what devices are actually in use with very good information of who is responsible for them. This process has allowed them to remove 70-80,000 old devices that are no longer in use, and it is this cleaned-up information which they are loading into the IPAM system. 2-1-9) Required and optional fields support customization The minimum registration information required includes: Although the above fields are required to be maintained for each device, there are extra optional fields available for use as well. Some units already have extensive inventory systems and are using a field as a pointer back into their own systems. Others are satisfied with this new database and plan to use it as their complete inventory system. 2-1-10) Support for monitoring sensitive/critical systems Tim asked if there was a step in the registration process where they ask what type of data will be stored on the device. Avi responded that this was one of the required fields in the registration record. They also ask for the criticality of the device, which they define as the recoverability objective: does it have to be able to be recovered in less than 24 hours, 48 hours, etc. This ties into incident response where they maintain a "crucial systems registry". The criticality of a device may affect how best they respond to an incident. Tom added that devices with high recoverability objectives require a DR; they have to have something in writing on site to justify the device being a critical device. This has obviously slowed down interest in registering devices as critical. There is another field which addresses the data sensitivity. They ask does this device store complete SSN, credit card numbers, patient records and those kinds of things. This is listed as a volume: 0-10, 10-100, 100-1000, etc. This all helps with the security analysis in making sure devices don't get assigned to an inappropriate zone. 2-1-11) Future DNS management potential The new IPAM system also handles DNS and has some capabilities for delegating that. They do not plan to use that feature initially, but may look at that down-the-road along with trialing some dynamic DNS to see if that is a good idea.
2-2) Follow-up questions from the committee2-2-1) What are the building blocks of the IPAM system? Dan Miller saw on the Project Overview that "Adonis" is the DHCP service and "Proteus" is the IP management interface. He wasn't sure what "IP management interface" meant in relation to everything else and asked if Avi could elaborate on the building blocks and how they fit together. The Adonis appliance Avi responded that Bluecat Networks is the company which has two main products. The very first product they had was Adonis which is an ISC DHCP server wrapped inside an appliance. Their initial management interface was a fairly rudimentary Java applet which you could use to make changes. The Proteus appliance Then Bluecat added the Proteus IPAM appliance which is a very extensive database. You can use this very flexibly to build out IP networks of any shape or size, create supporting DHCP pools for those networks, and assign devices according to detailed specifications. It does the same thing regarding DNS, providing a graphical view of zones where you can create records within the zones via a web GUI. The unique aspect of Bluecat which they really liked was that it also has this concept of a MAC address record; that is what allowed them to add devices to DHCP pools as well as provide extra fields for maintaining device information. The Proteus database updates the Adonis DHCP server The real meat of the Proteus system is that database and the user interface behind it. You make all the changes on the Proteus and at specific "deployment" intervals it then pushes all that information down via XML transfer (or something similar) to the Adonis appliances. The Adonis has a service that picks up that data, turns it into the necessary dhcp.conf and zone files, starts the daemon, and that kind of thing. The other neat thing about the system is that they have implemented failover among multiple Adonis appliances so you get true DHCP failover. So far in their testing that aspect looks very solid. 2-2-2) Does connection location affect the process? From plug-in to connectivity Tim Fitzpatrick asked if every device to be connected to the network has to have its MAC address registered in advance and Avi replied that this was the case. He further asked that, since those MAC addresses are associated with one or more zones or pools being utilized for various purposes, that when a device connects it had better connect within an authorized zone or it will not get access? Avi said that a device can plug into the network anywhere. The system will select which zone a device needs to be in and place it in the correct VLAN. Once a device gets assigned that VLAN it can do its DHCP broadcast and can get an IP address. So the access of a device is constrained and controlled by the VLAN to which it is dynamically assigned. A foreign MAC address will not get an IP address and therefore will not receive connectivity. Rudimentary Network Access Control Avi summarized this by saying the system acts as a rudimentary form of NAC--not nearly as foolproof as a real one, but it will certainly slow people down as far as plugging in devices without the assistance of their IT support person. 2-2-3) Where is the overlap of wireless vs. wired in all this? Dan Miller asked about wireless NAT and whether that was separate from the IPAM which covers the wired network--or is there overlap? Avi responded that the overlap is that the IPAM system will be providing DHCP services to wireless devices. They have two wireless VLANs currently. The "hnet-public" VLAN goes through the NAC for which posture assessment is done and users have to log in via their Gatorlink. Devices using this do not need to have their MAC addresses registered. They also have an "hnet-secure wireless" network which is 802.1x protected. This also requires Gatorlink authentication, via a supplicant, but devices using this network do have to be registered. The idea is that the assistance of an IT person is needed in order to access this network; it is not something which a user can do for themselves. Once an IT person has registered a device, they have in effect taken responsibility for the security and all other matters related to that device. All the IT support people who have attended the training and been approved will be given access to the web interface of the Proteon for registering and updating their devices. As systems move around, there will be methods for transferring responsibility for individual devices as well. 2-2-4) Where is posture assessment done? Dan Miller asked if posture assessment was only on the public wireless network. Avi confirmed that to be the case. The idea there was twofold. They didn't want to bite off too much at once--posture assessment on all the students was a big enough undertaking. The other theory was that since the devices on the secure network are managed by an IT group, we should have a lot fewer problems with the kinds of things which would cause a device to fail posture such as missing patches and antivirus updates. 2-2-5) Is there any IP Spoofing protection? Clint Collins asked if the system involved any protections against IP spoofing. Avi responded that it does not. A sophisticated attacker would not have much difficulty bypassing the system except on the wireless where the WCS has a setting whereby one can deny any network access unless you get a address from DHCP. If someone connected to the wired network and started up a sniffer they could just look at what is in use and pick an IP address. Clint asked about key exchanges but Avi responded that they are trying to stick with stuff that is currently on their machines. This may eventually work towards something more sophisticated, but it is already a big jump just going this first step. Something like 802.1x may be an option in a couple of years once it becomes a bit more reliable. 2-2-6) Flexible DHCP option groups eliminate need for unit-run DHCP Ty mentioned that this system eliminates the last of the excuses people may have had for running their own DHCP. Avi responded that this was a good point. They have a couple of units with special DHCP options requirements which have been running their own DHCP. One of the nice things about the Bluecat system is that a group of MAC addresses can get its own bundle of DHCP options. This allows unit IT staff to control the options on their devices no matter where they plug in; this was a really appealing feature of the Bluecat system. It turns out that there is less and less call for that sort of thing nowadays, but there are still a few instances and the phones are a good example (though there are other ways around that as well). 2-2-7) How do VoIP phones relate to IPAM? Dan Miller asked if the phones were entered into IPAM. Avi responded that the voice VLANs do not require that the devices be registered to get an IP address. IPAM is the DHCP system for the entire complex, however. When a phone is plugged in, the Cisco Discovery Protocol (CDP) process places that device into the voice VLAN which limits the device to just voice traffic. A device chained off the phone's data point goes into the regular data VLAN.
2-3) Security zone implementation2-3-1) The zones When the IT person registers a MAC address there is a drop-down field where they select the appropriate security zone: 2-3-2) Zone access descriptions Devices in the Open zone get public addresses and more-or-less complete internet access. Devices in the Protected zone get (in almost cases) private IPs with full outgoing access but no incoming access. Avi refers to this as the equivalent of being "behind a Linksys router" and is where the bulk of their devices go. The Closed zone provides absolutely no internet access; this is used for devices which have no reason for internet access like printers, lab equipment, medical devices, etc. This would also be the correct zone for back-end servers and for purely internal front-end servers as well. Each of these zones is implemented as a particular VLAN with their own distinct IP ranges. All of the VLANs for the various zones are in contiguous maskable ranges and the firewall is implemented based on those ranges. Rather than having to enter individual firewall rules for each device they have five big buckets of things which have five groupings of firewall rules. 2-3-3) MAB implementation HealthNet utilizes Cisco MAC Authentication Bypass (MAB) which is a part of 802.1x which has been broken out. It works something like the following. You plug a device in. The switch makes a RADIUS request with the MAC address as the username. The RADIUS server then responds with "Authentication OK" and the name of the VLAN which you should be placed in. The switch then assigns that particular port to that VLAN. If the device is not in the database, it gets dropped into the default VLAN which goes nowhere and the device will not get an IP address. Consequently, you can't even just plug in and guess an IP address; at this point you have to be in the system in order for you to get an IP address. You can still spoof a MAC address, of course. 2-3-4) Having few systems remaining in the Open zone was a pleasant surprise On the HSC side of things they have ended up with about 12,000 devices and out of those they had on the order of 200 registered in the Open zone. Avi finds it very exciting that only this small portion of their machines will have incoming internet access. Originally, Avi had thought the portion might be more like 25%; but when you think about it, that is about the right number of devices needed to account for mail and web servers. The rest of their devices can all be Protected. Marcus Morgan commented that most publicly visible machines don't really need to be publicly visible; people just think they do. Stephen Kostewicz said that it didn't take much convincing once this was available to get people to move many things to the Protected zone. 2-3-5) Policies encourage Open zone use only when necessary Ty pointed out that additional policy has been put in place for servers requiring more stringent monitoring. Avi noted that this was the "stick" side of things as opposed to the "carrot". They had revised their Host Security Policy last year to specify additional security evaluation that must be applied to machines in the Open network security zone. This places such machines in the same category as those storing or providing concurrent access to Restricted information. That helped people make those difficult decisions of whether or not it was worth leaving various machines in the Open zone; doing so provided not only extra risk but also extra work. 2-3-6) Details of other zones Tim asked for details of the other zones. One is the Campus zone, which allows sessions to be initiated from campus. There are some cross units like Physical Plant that are supported by IT service up on campus. They need to remotely manage things, for example. On the other hand they have units like the Student Healthcare Center where they have groups up on campus that need to be able to access things down in the Health Science Center. This amounts to an extension of the Protected zone, which covers the entire campus. There is also a Building Management zone for things like Johnson Control, Siemens, Lenel, etc. They are starting to put fire alarms on the network. This zone is not expected to be dynamically assigned, but rather will be fixed. These involve devices that are generally bolted down and should not move around. They also consider Wireless to be another zone though it again is something which is not dynamic. In general they haven't enabled policies for doing different things based on how you authenticate to the wireless. The exception is for authentication via a Gatorlink Guest account; those are denied access to the internal zones.
2-4) Guest access discussionDan Miller mentioned that CNS is in discussions with IFAS concerning wireless guest access for IFAS remote sites. Dan asked if Avi envisioned enhancing their Guest access or providing another class of guest access for people who need access to internal things. Avi responded that they have seen no requirement for that so far. Separate external network is one option In general, the Guest access is for vendors, guest lecturers, meeting attendees and the like where they want to access their home networks. They really have no need to get to the internal network. They have looked at possibly providing some other sort of guest access for true guests. Tom mentioned that Shands is doing that now via a Cox POP they have brought in with an SSID that only goes to that. So people who are getting on that wireless are outside the network proper--they could be anybody. They are still considering whether they want to extend that into HealthNet or not. VetMed is building a new Animal Hospital and they will likely do that there for folks in the waiting rooms. They are looking at the Orthopedic Institute as well because they are a clinic. Gatorlink guest accounts are another option Dan clarified that IFAS is looking at providing access for people who have signed up for a weekend course at an extension office. Clint said that they handle such things by creating Gatorlink guest accounts for each individual. Avi said that the good thing about that approach is that you can tie some identity to it. They puzzled about how to handle guests for quite some time. When you have large public areas you tend to not want to tell people that they have to go to a particular place and go through a registration process in order to use the network. On the other hand, there is no good way to authenticate someone unless they do present themselves to you. The only other option at that point is checking credentials or something like that which brings in a whole other set of problems. Security concerns may weigh heaviest They decided that the whole idea of this initiative was to make their network more secure, not less secure. If we start allowing anyone onto the network, then that is less secure and we don't want to go there. Ty mentioned that they also have a requirement whereby if someone is misbehaving on the network, they need to be able to find out who they are. Tom mentioned that the alternative is to get an outside vendor and put hotspots everywhere.
2-5) The IPAM process revisitedTim asked if the following overview was an accurate description. IT staff register devices, plug in the proper fields, and those devices get slotted into zones that are immediately operational. The local admins thus define where things go. Security staff then review certain of these based on the fields which are set and ask local IT staff to take additional steps which they already know they should be taking. Avi responded that this was basically correct, but if they are requesting to go in the Open zone, then that requires prior approval. Settings such as the indication for a need for rapid restore response would trigger a follow-up process such as confirming the existence of a DR plan.
2-6) Zone implementation pilotThe HSC Security Office will be piloting the zone implementation shortly. Assuming that goes well they will then begin plans to roll it out. The rollout will involve installing new VLANs, IP ranges, etc. There will be mapping for a set of VLANs which implement the same set of zones in each building. VLAN names will be checked rather than VLAN IDs; as long as the VLANs are named consistently then the whole thing should work. As things are rolled out into the new zones which are new networks, in most cases they will be getting private IP addresses. The real dance here will be keeping the NAT pool full enough. The trick will be trying to get enough public IPs to throw at the NAT pool. You can't take the public IP away from a building until you have addresses in the NAT pool; but you can't put addresses in the NAT pool until you take your IP addresses away from somewhere. Once the Dental building is done, they should have "starting seed" for that process.
2-7) Miscellaneous security discussions2-7-1) Defining private information Clint asked Avi if phone numbers not tied to other information were considered private information; this question was based on his having a potential client who did not want their phone numbers to stay on the PBX when calls are made to them. Avi responded that this was not the case under Florida law. Stephen said that there may still be some disclosure required; he advised that this be discussed with Susan Blair. 2-7-2) Security and networking at the VA There was some additional discussion about security and networking at the VA and how their stricter needs affect certain of our relationships. Action Items
Next MeetingThe next regular meeting is tentatively scheduled for Thursday, April 9th. |
last edited 15 March 2009 by Steve Lasley