ICC Home  /  Members  /  Meetings  /  Peer Support  /  Documentation  /  Projects

Minutes of May 14th, 2009 ITAC-NI Meeting:

back to ITAC-NI minutes index

    Link to ACTION ITEMS from meeting


    1. Approve prior minutes
    2. OIT Action Plan new committees coming soon
    3. CNS plan to roll out production IPv6 on the network
    4. CNS WP update


    This meeting was scheduled in CSE E507 at 1:30 pm on Thursday, May 14th and was made available via videoconference with live-streaming and recording for future playback. Prior announcement via the Net-Managers-L list was overlooked this month. The meeting was called to order by ITAC-NI chairman, Dan Miller, Network Coordinator of CNS Network Services.

    ATTENDEES: Fourteen people attended this meeting locally. There were two attendees via Polycom videoconference but there are no records of how many may have listened into the stream via a web browser using the web interface.

    Seven members were present: Charles Benjamin, Dan Cromer, Tim Fitzpatrick, Stephen Kostewicz (via Polycom), Steve Lasley, Tom Livoti, and Dan Miller.

    Seven members were absent: Clint Collins, Erik Deumens, Craig Gorme, Shawn Lander, Chris Leopold, Bernard Mair, and Handsford (Ty) Tyler.

    Seven visitors participated as well: Elwood Aust, Dennis Brown (via Polycom), Chris Griffin, Todd Hester, Mark McCallister, Marcus Morgan, and Tim Nance.

    Viewing the recording

    You may view the recording via the web at Currently, you will need to click on the "Top-level folder" link, then the "watch" link next to the "ITAC-NI Meeting_14May09_13.15" item. This will likely be moved into the ITAC-NI folder shortly. Cross-platform access may not be available; on the Windows platform you will have to install the Codian codec.

    Audio archive

    An archive of audio from the meeting is available.

    1) Approve prior minutes

    No corrections or additions were offered and the minutes were approved without further comment.

    2) OIT Action Plan new committees coming soon

    This will be the last meeting of the ITAC-NI and Dan Miller thanked everyone for being so helpful and constructive during his tenure as chairman which began in October of 2007. He stated that it had been a pleasure working with the committee and that he felt there had been many good and fruitful discussions.

    As Tim Fitzpatrick recalled from the planning document, there will be four topical IT advisory committees formed to address the areas of: Academic Technology, Infrastructure, Enterprise Systems, and Security. Three of these four are similar to existing committees, and the new Infrastructure committee is expected to encompass the functions of this ITAC Network Infrastructure committee while adding other technology infrastructure not included as part of Academic Technology or Enterprise Systems.

    3) CNS plan to roll out production IPv6 on the network

    Dan Miller introduced a new "Discussion Draft" outlining the CNS view of IPv6.

    3-1) Background

    Dan Miller mentioned that this looks very similar to what was discussed here previously in January. There is a background statement which affirms that we have been using IPv6 for many years--at least in a trial mode. There hadn't really been much of a demand for moving ahead prior, but now ARIN has been requesting large institutions to begin developing and maintaining some IPv6 presence.

    3-2) Purpose and scope

    CNS believes the first step should be to roll out IPv6 to the campus backbone, after which that service would be available to HealthNet, Shands, DHNet and other building networks. All of the edge deployment of IPv6 would be dependent upon security and management tools which would need to be in place to handle such traffic. The current discussion draft does not discuss the implementation of IPv6 beyond the campus core network.

    Tim Fitzpatrick related that he had received a notice from ARIN referring to our campus address space and listing some new rules and guidelines; Dr. Machen received a similar letter referring to another address space. Those letters led to discussion within CNS on whether it was time to move ahead and if so, what that would mean. That is what prompted this discussion topic here today. Tim is certain there is a whole lot more to IPv6 and how it would be used than what is covered in the discussion draft, but this was Tim's attempt to elicit discussion and learn more from the wider networking community on the topic.

    Dan Miller said that he had brought along Marcus Morgan and Chris Griffin to talk about some of the details. One of the things CNS had been waiting on was the official delegation from ARIN through FLR. That happened in February. Dan then turned it over to Marcus to discuss the allocation and our options there.

    3-3) UF's allocation

    Marcus Morgan explained that ARIN wants organizations to obtain their allocations from their service providers who would in-turn deal directly with ARIN on their behalf. FLR is our primary service provider and thus is the group through which our allocation was requested. ARIN does not want to deal with our level of granularity directly, but more importantly, they want to ensure better routing aggregation by handling things via the service providers.

    Marcus said our allocation will be a /48. An IPv6 address is 16 bytes (128 bits) long. The last 8 bytes are really local to the device; a device will typically autoconfigure and place the equivalent of its MAC address in that location. Consequently, you can think of an IPv6 address as being 8 bytes (64 bits) long, composed of the first 8 of 16. There are 48 network bits in a /48 allocation providing us 16 bits of usable addressing. This provides a tremendous amount of address space, equivalent to 256 times what we have in our Class B 128.227.x.x currently. This should handle our needs for decades.

    3-4) Approach

    Dan Miller said that CNS has to begin with the address plan. In terms of rollout they are going to start at the Internet boundary (IPOP) router--what they call the External WAN (EWAN) router. They will perform a code upgrade and IPv6 rollout. The Core and Nexus will follow until the UF backbone is covered.

    The Datacenter is a special case, especially with regards to Firewall Services Modules and server load balancing modules. Those remain a work in progress and CNS will have to see what support comes from their network vendors regarding such items. DNS/DHCP is an integral part of the solution which will need to be worked on.

    3-5) Architecture

    Dan Miller said that IPv6 for the most part is just viewed as another protocol running on the core network. Fortunately, we have hardware that will handle IPv6 appropriately.

    Chris Griffin pointed out that we had IPv6 on a portion of our core already. That has been routed into the CNS test lab since 2002 using a portion of a /48 obtained from Internet2. While CNS has gained experience from routing IPv6 for quite a while, they did not want to roll out that /48 to campus tying us into the Internet2 network. UF needs the more direct aggressive aggregation which FLR can provide and that is where we should get our space.

    Chris said that IPv6 is already running on our EWAN, one of the Nexus routers, and one of our Core routers which takes it to the lab where they have had it for many years. Dan Miller asked if the intent was to return the Internet2 space and Chris responded that it was.

    Chris mentioned that there is still a lot of discussion about how to handle multihoming in the IPv6 world; that is a problem which is yet to be solved. There may come a time when we would get a Provider Independent Address, but once that address reached campus a section of the address would be translated to a provider dependent address. There are at least three competing standards for that and it is still in discussion how that may play out.

    Tim said that it appeared to him that the CNS plan being discussed involved mainly "readiness" for IPv6 by assuring our Core can route that protocol. He asked if there were any uses for IPv6 yet which were being considered. Chris responded that we have test hosts in the lab, but that is basically it currently. Tim wanted to know what those who will be utilizing IPv6 might need to be doing in parallel with this Core rollout.

    3-6) Considerations

    • Addressing plans: Dan Miller said that the FLR /48 recently provided to UFL should cover main campus, DHNet and possibly HealthNet. Shands potentially could be served by a separate /48 request. through FLR. It is the expectation that UFL would be the owner of record for all IPv6 space on campus.
    • Further rollout: IPv6 may be extended to HealthNet and DHNet as soon as the Core upgrades are complete--it will be a matter of coordination and testing.
    • User responsibilities: It will be the responsibility of users to ensure that IPv6 works correctly with their systems' hardware and software. That applies to network hardware and software for other providers as well.
    • Nexus code upgrades: These are dependent on an internal CNS project to move WiSM blades to a dedicated chassis. That is expected to happen within the next few months.
    • The CNS Datacenter: The IPv6 rollout to the Datacenter will not initially support advanced services, but they could provide basic connectivity for single-attached hosts.
    • IDS: IDS is a concern regarding support for IPv6. Support there will be necessary in order to rollout IPv6 to significant user populations. The Datacenter also has special considerations. They are covered in terms of sniffing capability, so from the network point-of-view it is just another protocol.
    • NMS systems:
    • The current network management systems will need to account for IPv6 as well.

    3-7) Questions

    Is there a NAT IPv6 to IPv4 conversion that can be used? Will a client have to support one or the other? (Dan Cromer)

    Chris Griffin responded that there are several techniques which may be used. One is the "dual stack" approach where clients support both protocols. The protocol actually used is then largely dependent on the DNS answer provided when a record is requested. If you get a AAAA (quad-A) record rather than the normal A (or CNAME) record result and you have IPv6 enabled, then the system will use IPv6 to try to connect with the foreign host.

    There are several translation mechanisms available as well. There is IPv6 compatible IPv4 addressing where everything internally is IPv4 and when you leave campus part of your IPv4 address gets stuck on an IPv6 address and sent out. There is automatic tunneling. There is a general NAT solution. The approach we intend to use, however, is the dual stack approach.

    Marcus said that the initial goal is to get some significant portion of the university web space so that they are addressable externally via IPv6. We'll be able to have IPv6 presence with things like that, which is what ARIN has called for at this point.

    What are the issues with public vs. private addresses with regards to IPv6? (Dan Cromer)

    Marcus responded that there is a boundary. What really protects your private space, assuming it is NATed so it is addressable by the Internet, is a set of ACLs which defines how that private space will be treated. Marcus assumes we can define an IPv6 space which will be analogous to what we are doing currently with IPv4 so that it is similarly treated and has the same ACLs.

    Tim raised the question of standards or guidelines regarding the use of private space by workstations. Marcus responded that we currently have a standard which says workstations should be on private IP. Tim pointed out that this standard is not strictly enforced, however; he asked whether we thought it should be. Should we reassert the standard and develop and implement to conform with it? Marcus believe we should as there is a real benefit from restricting your vast population of workstations.

    Dan Miller clarified that under IPv6 we will specify a subnet which will be appropriate for use by workstations and we will have access control filters similar to the way the 10.x.x.x space is handled currently. The main point is that it is not the network numbers themselves that provides the security--it is the ACLs.

    There are several ways of handling addressing with IPv6. Has CNS settled on a particular embedding method or is that still open for discussion? (Charles Benjamin)

    Chris Griffin responded that the IPv6 address is generally built by converting the MAC address to a modified EUI-64 format. Your network level address is then concatenated in front of that. DNS dynamic registration is generally used to register the hostname to DNS because typing in these extremely long IPv6 addresses will obviously not be something anyone will want to do.

    IPv4 compatible IPv6 addressing is generally done on the border router in a NAT-like fashion. In such a case the client would have the IPv4 stack and the addressing would be made compatible with IPv6 as it leaves the network. We don't expect to go that route, but rather hope to use the dual stack approach. Marcus added that he suspects we will utilize autoconfiguration where the device builds its address and the adjacent router provides the network part of that.

    Charles said that the private vs. public question relates to routing because you are not supposed to route private IP addresses out to the Internet. He mentioned having seen that happen on one occasion however. Charles suspected that we can see to it that "private" IPv6 addresses are not routed out, but we will still need to do some sort of NAT.

    Chris Griffin responded that the way he would prefer to handle that is to implement access controls on the private IPv6 addresses so there is no translation but there is the same stateful security. You will be able to get an in-bound request as the result of an outbound connection. Unless you put an ACL in front of it, once the standard IOS NAT engine builds the translation essentially anything can come back in (unless you are doing PAT). CNS has that running across the Firewall Services Module, which is basically a pit with a state engine in front of it. The security you see for all these devices that are talking across the Internet and which pretty much always have translations built, is that state engine sitting in front which says "I don't recognize this traffic; it isn't for you." and drops it. What they propose is to have that same state engine sit in front of a range of those IPv6 addresses, so you know that when you are within this range you are protected.

    Charles mentioned that the first book he ever saw on IPv6 was way back in 1997, which tells one just how complicated this new addressing scheme will be to implement. Chris added that the RFCs on how to implement this have finally slowed from every few weeks down to one every 3-4 months. Marcus said that he felt delaying this over the last five years or so has helped us a lot; we have accumulated some good experience with this technology over that period.

    Charles asked if it was preferred that our web servers were the first devices to use IPv6. When Marcus replied that this was the call, Tim clarified that this discussion is a brainstorming session to figure things out rather than anything prescriptive. Marcus responded that by "call" he meant that this is was what ARIN has asked us to do. Charles suggested another way of putting this would be to say that UF needs to have an IPv6 presence on the Internet.

    What is the routing of IPv4 and IPv6 between Cox and FLR? (Charles Benjamin)

    Chris Griffin responded that currently we do IPv6 with FLR. FLR does IPv6 out to Internet2 and the National LambdaRail. That is the current extent of IPv6 and Chris did not know what Cox's plans were in that regard. FLR is currently bringing up IPv6 commodity peering in several different modes. One is via Internet2 which has a commodity peering service which they offer; they peer with a few content providers which support IPv6.

    The second way FLR is going about this is to talk with some of their direct peers such as Google, Limelight and a few of the hosted providers which do offer IPv6 services and we are turning those services up. The hope is that this will give IPv6 implementers some place to go in order to help get things out of "test mode". A lot of the content providers have been encouraged by the regional registries to roll out IPv6 services and they are starting to do that. Once we get IPv6 on our backbone we will be able to go to http://google.com and it will actually run over IPv6.

    The plans for actual true commodity Internet services are on a slow ramp-up because a lot of the services are being provided via tunnels. CNS has made the engineering decision not to engage in IPv6 tunneling because it provides a much worse path than would a native IPv4 path. The last thing we want is to have people turn on IPv6 and for it get slow; that would lead to a poor perception.

    What are HealthNet's plans regarding IPv6? (Tim Fitzpatrick)

    Tom Livoti said they really had no plans at this time. They have been moving very quickly to private IPv4 and retaining very little public IP space. He believes the Health Center can probably get away with just a /24 of public IP space. The first time he saw mention of IPv6 here was at the January meeting and without further mention he felt he had enough to do. Tim Nance added that IPv6 was pretty far back on the burner at Shands as well.

    Basically, Tom is trying to figure out the need. He doesn't want to go down that path unless it is necessary. Dan Miller responded that the hope is that the costs will be minimal, only relating to the labor involved, because the modern operating systems are supporting dual stack. The hardware is more and more supporting IPv6 just as it would IPv4.

    Chris Griffin spoke to the need, which he said lies away from this campus. ARIN has reported that there is somewhere in the range of 18-24 months of IPv4 space left; once that happens they will no longer have any space to hand out and will be forced to supply only IPv6. There will be groups who will only get IPv6 and not only will they be unreachable to us, but we will be unreachable by them. As a university that is not a desirable outcome.

    Tim stated that in the past we had visited this topic on occasion but had noted no pressing need. Now we are revisiting this to see if the situation is changing. Tom asked about translating at the boundary as a fallback so that we could be IPv4 internally while still providing an IPv6 presence to the outside world. Chris Griffin said that this was a possible option but the downside is that application translation becomes a lot more complicated. IPv6 has this notion of embedded chained headers which can determine how intermediate systems and even end systems treat the communication. This comes into play especially with encryption; we will see a lot more transport layer encryption popping up because support for that is built into the stack.

    Marcus pointed out that IPv4 will continue to be routed for 15-20 years or more even after it runs out. Consequently, HealthNet's move to mostly private IP with a little bit of public space is a very reasonable course of action. He added though, that they might want to look at dual homing some of their web servers because they will need to have that presence to the IPv6 community.

    Charles said that he was confident that Housing could support IPv6 via one of their web servers as that became desired. Regarding desktop use of IPv6, Chris Griffin noted a concern with Microsoft Vista and IPv6 in that it can build an automatic tunnel when it gets a quad-A response. They actually have observed this happening where the machine chose a path through some tunnel server in Europe instead of taking the nice direct path. CNS will have to address that and put out a paper on how to turn off this "auto-tunneling" feature.

    What are the security implications of IPv6? (Tim Fitzpatrick)

    Dan Miller responded that security tools would need to be IPv6 capable. The ACLs will also have to be developed so that they perform the same function. Beyond that, Dan guessed that the only issue was having to wrestle with the same long addresses that the network engineers were dealing with.

    When we are ready to enable a web server for IPv6 what do we do? (Charles Benjamin)

    Chris responded that a unit should request an IPv6 delegation to get the routing turned on. When Charles followed up by asking for a timeframe until things were ready, Dan Miller said that it was difficult to say at this time; a year would seem very reasonable and doable, however.

    Should we consider a central CNS-supported IPv6 dynamic DNS/DHCP server? (Dan Cromer)

    Dan Miller restated this by saying that the need to move applications like DHCP to IPv6 may provide another reason for people to consider moving to a centrally provided service.

    Tim said that today's purpose was to solicit comments like that and to say that we are thinking about enabling these capabilities on our Core. Such things beg for a plan and we shouldn't go off and do this without a plan. The next review would be about an implementation plan which would include cost and scheduling details. What he didn't want to happen was for his staff to answer his questions with a plan that the UF networking community hadn't had a chance to comment on. That was why Tim carefully steered Dan Miller to make this a discussion here today and from the results of the discussion to come back with a plan. That is also why Tim asked about the implications of IPv6 for security.

    Dan Cromer then asked about the implications for DHCP. We need to discover the implications and offer some options for dealing with those. When Marcus mentioned that Dan Cromer's comment was really looking toward the deployment of the workstation and we really aren't there yet, Dan responded that this is a window of opportunity for dealing with things collegially and collectively rather than having each unit do its "own thing". Dan wanted to encourage the whole of idea of centralization...pause for a better word (offered: consolidation [Tim], cooperation [Tom], standardization [Dan Miller]). Marcus pointed out that we will need to make a great deal of careful study before we implement anything at the level of workstations; that is completely different than saying here is a web server which we want to make available to the world.

    Charles Benjamin added that not having their own DHCP servers wouldn't work for them because of their P2P validation processes.

    Tim said that he believed Dan Cromer's point was that as we move to IPv6 we should consider the opportunities involved and seize them where appropriate. Tom responded that the need for DHCP service for this looks to be far off since web servers facing this would have their addresses statically allocated in any case. Dan Cromer mentioned that the provision of static addresses via DHCP MAC reservations was a common practice within IFAS. Marcus said that what we really need is a self-service DHCP server so that each unit could have their own span of control and make that work as they wanted. Tom responded that the Health Science Center and Shands have implemented a DHCP system which is like that. He added, however, that these systems aren't generally well adapted to delegating DHCP management; they have managed to make this work within their system but it has been painful.

    Chris Griffin pointed out that the way end stations are typically done in the IPv6 world is using a technically stable sort of configuration. To get your address you don't actually talk to the DHCP server. Your address is built dynamically via the router advertizing its network using a broadcast process termed IPv6 network discovery. The end station hears the network discovery and utilizes an algorithm which builds the EUI-64 portion of the address from the local MAC address by concatenated that with the network portion. The end station then autoregisters that address with DNS saying "here is my name". Typically, it uses DHCP only to learn about DNS servers, Time-of-day servers, and those kinds of secondary systems. DHCP is still required and is useful under IPv6, but it will not serve the primary role it does with IPv4 whereby if it is not working you don't get an address.

    Are there any in-built security advantages or considerations to the IPv6 protocol? (Dan Miller)

    Chris Griffin responded that while some people might say yes, he would probably say no. Originally IPsec was designed for IPv6 and then shoehorned into IPv4; however the same restrictions to IPv6 apply to IPv4. You need a certificate authority before IPsec can work and those kinds of things. In the end you are still dealing with a transport level address--that is pretty much all that is changing.

    There are some restrictions because the address is now longer. Some hardware platforms may not have the ability to do deep inspection on an IPv6 packet. On hardware firewalls you might start seeing some restrictions where you would have to upgrade those. From a usability perspective as a network engineer, it is going to be difficult to poke holes in ACLs based on full IPv6 addresses. Right now they normally specify allowed addresses to various locations. They are going to have to figure out some way of assigning more reasonable addresses for machines which need holes poked. Everyday end-user stations can just float around with their really long addresses. It is more of a technique issue. Dan Cromer noted that nmap would probably take longer to scan and Chris responded that security through obscurity would have some role to play there.

    3-8) The next step

    Tim stated that he believed CNS should take the input heard today along with other bright ideas which may be forming and turn this into what could be a project charter or something similar. He knew that Mike Conlon had already suggested bringing together a group to brainstorm. Tim found today's discussion as a good starting point for feedback.

    4) CNS WP update

    Todd Hester reported that as of May 1st CNS was supporting a little over 21,000 data ports and almost 2,800 VoIP phones. Projecting forward they have several projects nearing completion which should be finished by July of this year including: Entomology, Norman, Academic Advising, the stadium (Facilities). That will add another 2,600 data ports and 600 phones. They also have two new constructions projects: the Law Advocacy building and the Southwest Garage and office area. Those are going to add another about 200 data points and 100 VoIP phones. They are in the early planning stages currently talking to Physics, Yon, Bartram-Carr, Fifield, New Engineering, Grinter, Materials, McCarty A & B, Williamson, Turlington, Black, Phelps. If those all move forward they will add another 6,500 data ports and 1,700 VoIP phones. So if they stay on schedule, by January of 2010 CNS will be supporting about 30,000 data ports and just over 6,000 phones total.

    They are projecting that they will top out the end of next fiscal year probably around 35,000 data ports and 8,000 phones.

    Tim stated that there had been a lot of concern that the budget crunch would constrain this roll out in year three. In fact, it would appear that most people are continuing to sign up and it looks to be another busy year.

    Dan Cromer asked if Entomology had started yet. Steve Lasley responded that their wiring remediation had just been completed and that the switches were going to be put in place on the 23rd.

    5) Other discussion

    5-1) SIP compatibility

    Charles Benjamin asked if CNS was looking at SIP at all. Todd asked to respond informally to that saying that he believed the next release of Call Manager would support SIP. The technology will be there but Todd didn't believe they had formally discussed implementation yet. Tom mentioned that their current technology does not support SIP to the trunks. Tim asked if SIP would allow us a much broader selection of handsets. Chris Griffin responded that this was true hypothetically, but we need to make sure that the handset can utilize our system.

    Dan Cromer mentioned Office Communications Server is now a project underway with the Active Directory and UF Exchange groups. Tom advised being very careful with the Microsoft products as they have written some of the codecs themselves and are very proprietary in what they are pushing out rather than relying on G.711 or G.723. Tom said he would rather go to Asterisk than follow Microsoft down that path.

    5-2) Status of the self-service network management web interface

    Steve asked about where things were with this. Marcus responded that this is almost ready to go and he believed it should be available within the next few weeks. He said this would allow local IT support to view their devices and get statistics on them.

    5-3) Cisco energy management

    Charles asked if CNS was looking at the new Cisco energy management for their PoE devices such as the VoIP phones. Dan Miller said that they saw the 3560's come out and he understood that some of the EnergyWise framework still needed to be developed to support that. Charles said that they were buying the V2s now as they were the same price and ran the same code. Dan Miller said that once the technology is there the question is what kind of policy do we want to turn off phones to save power vs. people expecting their phones to work. There is some discussion which needs to happen.

    Action Items

    1. None


    Next Meeting

    This is expected to be the last meeting of the ITAC-NI.

last edited 7 October 2009 by Steve Lasley