ICC Home / Members / Meetings / Peer Support / Documentation / Projects

Minutes of January 10, 2008 ITAC-NI Meeting:

Link to ACTION ITEMS from meeting

AGENDA:

Approve prior minutes
ITAC-NI membership changes
TSM use of the network on and off campus
Cox Internet update
Final discussion and vote on recommendation to not allow external network devices in Wall-Plate buildings
other Discussion

CALL TO ORDER:

This meeting was held in CSE E507 at 1:00 pm on Thursday, January 10th and was made available via videoconference, with live-streaming, and recording for future playback. Prior announcement was made via the Net-Managers-L list (though that notice was sent only 10 minutes prior to the meeting starting). The meeting was called to order by ITAC-NI chairman, Dan Miller, Network Coordinator of CNS Network Services.

ATTENDEES: Sixteen people attended this meeting locally. There were two attendees via Polycom videoconference but there are no records of how many may have listened into the stream via the web interface. Again, we may want to clarify that latter option, as there may be considerable interest in viewing the meeting in real-time via a simple web browser.

Twelve members were present: Charles Benjamin, Clint Collins, Dan Cromer, Erik Deumens, Tim Fitzpatrick, Craig Gorme, Stephen Kostewicz, Shawn Lander, Steve Lasley, Bernard Mair, Dan Miller, and Handsford (Ty) Tyler.

Two members were absent: Chris Leopold and Tom Livoti.

Six visitors were present as well: Stan Anders, Dan Hawn, Todd Hester, Allen Rout, Dave Pokorney (via Polycom), and Jan van der Aa (via Polycom).

Viewing the recording

You may view the recording via the web at http://128.227.156.84:7734. You will need to click on the "Top-level folder" link, then the "watch" link next to the "ITAC-NI Meeting 1/8/08" item. Cross-platform access may not be available; on the Windows platform you will have to install the Codian codec.

Audio archive

An archive of audio from the meeting is available though the first few minutes were inadvertently omitted.

1) Approve prior minutes

No corrections or additions were offered and the minutes were approved without further comment.

2) ITAC-NI membership changes

We welcomed, Charles Benjamin who replaces Mark Hill for Housing, and Dr. Bernard Mair from the Department of Mathematics who replaces Dr. Jack Sabin for CLAS.

3) TSM use of the network on and off campus (Allen Rout)

Allen Rout was on hand to give us an overview of the Automated Backup Services at CNS.

3-1) What the backup service does and how it functions

Allen explained that since 1997 when CNS was NERDC they have used Adstart Distributed Storage Manager (ADSM), now called Tivoli Storage Manager (TSM), to perform their internal backups. As they developed that service internally, Allen came up with the idea of letting other people use it as well. TSM is one of a set of backup packages now known as disc-to-disc-to-tape backup. Rather than as with systems like Backup Exec which generally talk directly to a tape drive somewhere across the network, you are actually speaking to discs in the CNS machine room when you do backups with TSM. The advantage of this is that it permits many more backups to be running than for which you possess tape drives, which are generally non-preemptable resources.

There are currently about 1000 different machines, both within and outside CNS, backing up to their TSM infrastructure. They pass 2TB of data on a slow day up to 6-7TB on a fast day. All that data is obtained over the UF core network and is all sent to Atlanta on an essentially daily basis. The way their TSM service is architected, they extend a great deal of control to the clients who sit at the far end of the pipe. That allows local control of the administration and retention policies which affect a client's backups. This means Allen doesn't have to do that work, though he can if desired. There is a charge-back structure (see bottom of page) for this service on an on-going basis which sustains the service in its entirety--including Allen's salary.

3-2) The network impact of this service

There is a dedicated 1GB pipe from CNS to Atlanta and this backup service fills that pipe on an occasional basis nightly. The major way that the network between here and Atlanta affects Allen for these backups is that it doesn't. He can treat a tape drive in a machine room in rented space in Atlanta as though it is local. He can send 60-70MB per second to that tape drive on a consistent streaming basis. This means there is a bunch of abstraction and juggling that Allen does not need to do.

As his workload on a daily basis goes up, all he has to do is allocate more network bandwidth and he can theoretically run as many of those 70MB/sec streams as the busses on the two ends can tolerate. In the not-too-distant future, Allen plans to split his Gainesville infrastructure onto two separate hosts, which would mean he would have plenty of buss bandwidth here to entirely choke that network.

On campus, all the networks are sufficiently broad so that, unless someone has a great deal of data on the other side of a small link, nobody ever notices. Three hundred different machines write data straight to the discs in CNS at peak times during the night. Allen throttles the number of sessions based on the cap which he is willing to permit on each of his TSM server instances so that there is no performance bottleneck at his end.

3-3) Questions and Answers

Q.01: Who is your largest user? (Tim Fitzpatrick)

A.01: The largest single user is the ERP organization (Bridges). They constitute somewhat less than 50% of the total usage. Until very recently, the single largest individual client was an SQL database--though they have now figured out how to make that stop happening. The largest of the PeopleSoft production databases now holds that honor, and CNS receives a roughly 300MB full backup of that every other night.

Q.02: How has the volume of your user-base changed over the last year or two? (Tim Fitzpatrick)

A.02: There has been roughly 80% growth in nightly traffic over the past year and recent trends come closer to doubling on a yearly basis. Allen wanted to point out, however, that the long-term behavior of this service is dominated by unique events. When someone comes up with a new thing they want to do, we get a new trend line which is stable for a month or so until someone else makes a change.

Q.03: But over the past 2-3 years you have probably doubled twice? (Tim Fitzpatrick)

A.03: Yes.

Q.04: What has happened to the [charging] rates? (Tim Fitzpatrick)

A.04: They have gone down--a lot. The last rate cut was 40%; the one before that was ~25%. Thus the rates have decreased by substantially more than 50% over the last year to year-and-a-half.

Q.05: What kind of machines are you backing up? Are they mostly servers in departments, or are they individual machines? (Erick Deumens)

A.05: Those are all over the map. CNS provides backups for 27 different platforms (CIRCA's DECs; workstations; AIX, HP, Netware, SGI and Sun servers; lots of Windows). Probably 15-20% are workstations, but the backups are mostly comprised of servers and databases.

Q.06: Do you backup WebCT through this service? (Tim Fitzpatrick)

A.06: Yes. WebCT is currently running about 1.4TB on its full backups which are done weekly. Their logs are captured in-between as well. After Bridges, WebCT is the next largest customer.

Q.07: So, in theory TSM is an "enterprise systems" backup, but in practice there is a lot of college and departmental level use? (Tim Fitzpatrick)

A.07: Absolutely.

Q.08: It was my understanding on the TSM that you only have to do one full backup and after that it just picks up changed files? (Craig Gorme)

A.08: In most backup schemes for which admins are familiar, you do a full backup and then you some or other math to figure out on successive occasions what things you need to pick up each time. But eventually, by any measure, that count gets sufficiently large that you just give it up and do another full backup. You might call that a "full and incremental" or "full and differential", "father-grandfather-son", "level 0-level 1-level 2", etc.

TSM operates on a scheme called "incrementals forever" or "progressive backups". At its core, TSM has a large database which records, for every machine that talks to TSM and for every file that is sent, what that file looked like the last time TSM received a version of it. This includes all the metadata concerning the files and you can even do checksums if you feel like it. What that means is, the very first time you do a backup, you send everything but it is all on an incremental basis. TSM downloads a database of everything which it has for your machines and if this is the first time it has talked to this particular machine it reports nothing--a very short list. Then on the machine to be backed up, the client walks the file system and makes individual decisions which, coincidentally, are all "yes I need this". The next time it comes up it gets a very long list of things that it has and then makes the same set of decisions--but this time it only decides to send a very small set of things. Because of this, you never need to do another full backup or copy an unchanged file over the network ever again--at least not for technical reasons. There may be some policy reasons for doing so.

Q.09: So why did you just say that WebCT does a full backup weekly? (Craig Gorme)

A.09: File system backup, retention and maintenance are in many ways fundamentally different from what is needed for databases. Databases have two major sources for backup: you have a snapshot of the entire thing, and you have the transactional logs. It is possible to cut a full and then retain only logs forever. Allen mentioned having one customer who had done that with their Exchange Server for eight months. When it eventually came time for them to restore, they found that the restore required more time than they determined was appropriate for their policies going forward. So they modified that habit and they cut fulls with a frequency reflecting their recovery needs.

That is why WebCT cuts a full on a weekly basis; they have taken a look at the recovery time and determined that rolling through a week's worth of logs does not substantially extend the recovery time which would be needed in case of a disaster. Allen encourages all those who are backing up databases to make that evaluation so that they can balance the costs of the backups with the recovery times.

Q.10: The pricing model is that you charge for what is being stored, but typically it was loaded once and it resides there. Then you charge for the daily incrementals--the traffic. Right? (Tim Fitzpatrick)

A.10: The charging algorithm has two major terms in it: transfer and storage. Allen maintains a fairly detailed cost model that represents what it takes on an on-going basis to maintain the service. That's Allen's numerator. And then he has these units that are related to the amount of activity that is going on for the server. These include units that are easy for the clients to understand: how much data you send, and how much are you choosing to retain. That the denominator. About every six months Allen refigures the math on that and, since we are increasing in usage very rapidly, there is a substantial reduction in charge per unit every time we look at it.

Q.11: But I could guess that with copy it once (fulls) and incrementals ever after, the major cost would be in storage and not in transfer. Is that accurate? (Tim Fitzpatrick)

A.11: [No.] What we attempt to do with each revision of the charging algorithm is make it so neither one of those terms utterly dominates, but neither one of them is significant. We balance those in order to help people understand the impact their behavior has on the infrastructure. If we permit transfer to utterly dominate, then people will turn up their retention parameters all the way and not worry about it--because it doesn't cost anything to keep it around. On the other hand, if we make it so that the storage utterly dominates, then people will throw away things which, in fact, are valuable to them--because that is where all their expenses lie. So the most correct answer to "Where is most of the cost--transfer or storage?" is "We try to balance it." As user behavior changes, we try to re-tune it so both of those considerations are relevant for most of our customers. Of course, we do have people whose behavior is out on the tails, in whatever direction, and there is no way that we can help that.

The detailed structure of the balance within the charging algorithm is more about people than it is about the technical structure underlying it because it is impossible to draw a precise correspondence with a particular piece of activity related only to storage and not to transfer.

Q.12: Could you tell us a little bit about the encryption? (Clint Collins)

A.12: There are several different layers at which the TSM service is capable of encrypting your data. The one to which Allen attracts the most attention is client encryption. This means that data which you consider to be important is encrypted by one of a variety of algorithms, including AES, on your client before it ever leaves your controlled environment. Thus the data is encrypted on the tapes in CNS, across the wire to Atlanta, and sitting in the machine room in Atlanta. Anyone who gets hold of it not only has to make sense of it without the TSM server, but also has to break the encryption. So, for people who have data which they feel they really need to secure, that is what Allen suggests that they do. This leaves it entirely under their control and even Allen can't get at it.

Q.13: This is a comment rather than a question. Housing has been very pleased with this service and when we needed some cross-training within our team, Allen came down and did an excellent job of that. (Charles Benjamin)

A.13: Thanks.

Q.14: Where is the data stored in Atlanta? (Stephen Kostewicz)

A.14: There is a facility called Telex which styles itself a "Telco Hotel". This is the same facility in which the Florida LambdaRail debouches into the National LambdaRail. It also connects with the Southern Crossroads (SoX) network there. That proximity was the overwhelming basis for the selection of that site as our off-site location because, instead of having to pull fiber somewhere across town, we could just get within-the-building connections. The other site which was considered was somewhere in Miami, which is probably a marginally more secure location because they built it like a bomb shelter after hurricane Andrew; somehow, Allen didn't feel comfortable telling folks that off-site backups were kept in Miami, however. The site in Atlanta is a commercial facility. We pay rent and have a cage. Allen mentioned having cameras in there which he watches from time-to-time.

Q.15: Do you have any medium to large server rooms outside of data centers as customers? (Dan Miller)

A.15: CISE's infrastructure is a good example of that. PHHP is another.

Q.16: Have you heard any reports of networking problems due to traffic levels from your service? (Dan Miller)

A.16: No. TSM is very good at smoking out bottlenecks though. It has helped locate sites which have had servers connected at the wrong duplex setting for example. But there have been no problems related to traffic levels.

4) Cox Internet update

4-1) Background and current status

Dan Miller related that CNS recently met with Cox Communications regarding the CNS/Cox Internet peering service to review some failures that had occurred. There were two failures in quick succession which could be categorized as the "best failures" which they have ever had, in the sense that the fail-over worked as designed. In the past they had some very unfortunate fail-over modes which caused us to be "black-holed" to a significant portion of the Internet. That was a very bad thing and it had definitely raised some concerns with management that led to several meetings. The recent meeting was a follow-up and Dan believes we have agreed to go forward with Cox based on some more engineering meetings which should be scheduled on an on-going basis to keep each party's network engineers in close contact. We hope they will continue to improve and we will stay with them at least through our contract.

4-2) UF has a variety of Internet connection pathways

Tim asked Dan Miller to describe the variety of commodity Internet paths and the redundancy they provide us. Dan responded that UF has two main providers, with FLR serving the largest bandwidth at 350mbps. FLR, in turn, receives Internet service from two tier-one carriers, Level 3 Communications and Cogent Communications; and those are physically diverse through the FLR network. Thus there is a great deal of redundancy and resiliency on the FLR side. Then as an extra failsafe path, we have the Cox service at 300mbps currently. We recently (within the last year) negotiated to get double the data rate at the price which we had been paying. Tim responded that the added value of Cox to UF is the local peering (the exchange of zero-cost traffic between consenting parties).

4-3) Redundancy reminder

Dan Miller wanted to remind everyone that if the SSRB network is completely disconnected, as happened about two months ago, the rest of the campus should still have full Internet capability through the Centrex facility. We have redundancy at both the Core and EWAN level; so there should be no ill-effects noticed--other than, of course, the fact that the nameserver has gone away. Dan mentioned that there will be continuing discussions regarding "opportunities for improvement" in that regard.

4-4) SONET connection?

Charles Benjamin asked if the SONET connection is still there with GRUCom. Dan Miller responded that this OC-12 use is pretty much gone. Both Cox and GRU come into the SSRB machine room at 10gbps via Ethernet; they each have Ethernet rings that they are building out.

4-5) AT&T SMARTRing-hosted services

The SMARTRing is a method for delivering local loop access from the AT&T CO (formerly the BellSouth Central Office) to UF. We have three nodes on the ring which are in the AT&T CO, SSRB and Centrex for redundancy. That is all it really does. We could put Internet-type traffic on there, but in reality the bandwidth is much lower on the SMARTRing. The big pipes usually come in as dedicated pipes directly into routers.

The SMARTRing hosts our VoIP PRIs, our dial-up PRIs, the IFAS Distance Education T1s, and some other miscellaneous T1s. In the future we may decide to put SIP/IP dial tone over the SMARTRing. That kind of traffic can go on there and we recently renegotiated the SMARTRing so that we have another 2-year term at OC-12 rates. It is pretty lightly used currently.

4-6) FLR peering opportunities

Dave Pokorney commented on the peering opportunities which FLR has brought to UF. We are seeing about 60% of all commercial Internet traffic taking place over our peers. So for the Internet access that we are procuring from FLR we can assume there is about 50% above that which is coming through peers. Tim responded that this has been wonderful for us because with FLR taking half of our traffic and sending it to other places through peering arrangements, then that is traffic that doesn't have to be paid for through an Internet Service Provider (ISP).

4-7) Negotiating for IFAS REC connections

Dan Cromer asked if we have any connectivity with AT&T or GRU. Dan Miller responded that we peer locally with GRU. GRU is not a major Internet service provider, at least on the level which UF would need. In fact, Dan Miller believes they had been talking to FLR about the possible purchase of Internet service. Dan Miller did not believe AT&T has been a major player; he wasn't sure if that was because of price. He knew that there had been situations where we have been unable to get good peering relationships with them; he believed FLR may have made some progress with them in that regard. Dave Pokorney responded to Dan Cromer that AT&T is becoming increasingly more difficult to peer with.

Tim responded to Dan Cromer that, for the RECs who are currently point-to-point, CNS believes the path to improvement is affordable local loop connections in those remote locations onto the FLR point-of-presence and piping them back that way.

Dan Cromer agreed with that, but we still have to deal with somebody for the last mile and dealing with AT&T in particular has been tough. Except for Bright House Networks, who has sharpened their pencil pretty well, costs have been prohibitive. Tim mentioned that they had approached BellSouth about this bundle of IFAS circuits and had gotten their attention, but no results; then BellSouth turned into AT&T, so maybe there is another possibility there. Dan Cromer mentioned that as we get more agreements with Bright House, AT&T may feel the competition and be more willing to negotiate.

Dave Pokorney mentioned to Dan Cromer that the quote for Immokalee came pretty close to IFAS's budget, so additional providers are starting to surface. Dan Cromer expressed his appreciation for Dave and Dan Miller's work on that.

5) Final discussion and vote on recommendation to not allow external network devices in Wall-Plate buildings

5-1) Background

This committee spent most of the last meeting (December) discussing the CNS Network Edge Protection plan and a recommendation to disallow external network devices in Wall-Plate networks. Due to the lack of a quorum at that meeting, a vote on such a recommendation was tabled until this meeting.

Dan Miller had sent a CNS proposed "ITAC-NI Recommendation on External Network Devices in Wall-Plate Networks" to the committee at about 5:30 pm the day prior to this January meeting--along with the proposed agenda. This recommendation, which is listed following, drew several e-mailed responses which are also documented below.

CNS proposed ITAC-NI Recommendation on External Network Devices in Wall-Plate Networks

Whereas enterprise network technologies are becoming more complex, and Whereas network availability is becoming more crucial to enterprise operation, and Whereas CNS can provide temporary network devices in Wall-Plate networks in a timely fashion, and Whereas CNS can provide technical bench switches for building large numbers of PCs at once in Wall-Plate networks, and Whereas peer Networks (HealthNet and DHNet) do not allow external devices on their networks as standard practice, This ITAC-NI committee recommends that external network devices such as hubs, switches, routers and wireless APs not managed by CNS not be allowed in Wall-Plate networks.

If this is approved, CNS would resume the deployment of port-security to protect the network. An automated notification system would need to be in place when port security is deployed, and it would send email to the local administrator and notify CNS as well. If a device with multiple MAC addresses is connected to an edge port on a Wall-Plate network, that port will be automatically disabled. The port will reactivate after 15 minutes, and will continue to be disabled until the external network device is removed. More details on temporary and tech bench switches from the CNS Wall-Plate web page: http://www.cns.ufl.edu/wallplate/wallplate-faq.shtml#spider-hubs If there is ever a special/temporary need for a hub, you will need to discuss the justification with us. They may be used as a temporary stop- gap until proper wiring to the closet can be established. All spaces served by the Wall-Plate project should have jacks terminated in the TRs, and a one-to-one port to machine ratio. VoIP phones act as pass-through connections, and thus take no additional ports or wiring. The Wall-Plate project supports "bench switches" in IT staff locations for dynamic activities such as new PC deployment and workbench trouble-shooting. These are standard Cisco switches like the rest of the Wall-Plate installation, but without the redundant network path and UPS which switches in the closets have. Bench switches will be handled on a case-by-case basis.

E-mailed Response #1: A departmental perspective from beyond the Wall-plate

Steve Lasley posted a response to the proposed recommendation via e-mail to the committee early this morning:

Dear colleagues,
   As I stated at our last ITAC-NI meeting, my perspective on the 
proposal behind today's agenda item 5 comes from my position as 
IT support staff at a typical academic department on campus (UF/IFAS 
Entomology). In my opinion the proposal before us begs for further cost/
benefit analysis. I would be concerned with supporting such a 
recommendation prior to evidence-based estimates of the resultant value to 
units relative to the inherent costs (both in dollars and flexibility). 
Furthermore, flexibility concerns will be difficult to address without 
first developing details of the processes to be implemented for requesting, 
approving, documenting and maintaining exceptions--temporary or otherwise.
   Another concern with the proposal as stated is that the granularity 
inherent in the available port-security controls is currently insufficient 
for a workable automated enforcement of the proposed policy. Also, HealthNet 
and DHNet may not necessarily be good peer models for comparison to UF in 
general, as they could be argued to support a more stringent set of network 
needs, goals and constraints--particularly in regard to legal concerns. In 
other words, I believe we may be going too far too fast with this. Would our 
overall network accessibility via the Wall-plate suffer greatly without this 
additional measure at this time?
   Consequently, I would recommend that this proposal undergo further review 
and consideration. In the meantime, I would like to suggest instead that 
exception requests, approvals, documentation and maintenance processes be 
fully developed (as they will be needed in any case), and that we then 
consider using those to require unit IT staff to register any external 
network devices under their control until such time that their elimination 
is shown to provide positive overall value to the unit and/or the UF 
organization as a whole.
Thank you for your consideration,
Steve

E-mailed Response #2: Correction and suggested change to original proposal

Charles Benjamin offered the following response:

All,

The following statement is incorrect:

Whereas peer Networks (HealthNet and DHNet) do not allow external devices on 
their networks,

There are exceptions were Housing does allow external devices connected to 
client ports of DHNet. One example is in Diamond Village because each 
apartment only has one Ethernet connection, they are permitted to connect a 
switch or wireless router for multiple connections per apartment.

The other issue is we are currently collaborating with CNS to install an 
ASA5510 in the VP Office to form a VPN tunnel between the VP Office in Tigert 
and Housing.  Appliances such as a ASA5500 which can function as a VPN 
concentrator or Firewall is not explicitly listed and should not fall into 
the category of "devices such as"

E-mailed Response #3: General agreement plus additional issues to address

Chris Leopold offered the following response:

All,

Unfortunately, I will be out Thursday and Friday to attend my Aunt's Funeral 
and will not be able to go to today's meeting. So, I would like to share my 
thoughts on the discussion of external network devices in Wall-Plate 
networks.

It has been my layer2 experience that user installed "switches" have caused 
nothing but problems. Therefore, I will support the concept of port-security 
within the Wall-Plate model. However, at the same time, I do share many of 
the concerns that Steve Lasley has expressed. In light of those concerns, I 
would vote "for" the ITAC-NI recommendation as written. If the proposal is 
modified, I authorize Steve Lasley to vote as my proxy.
 
I do have other issues that I would like to see this committee address. For 
example, the issue of network security and how the unit's ISM fits into that 
equation. Basically, would the ISM have access to the Wall-Plate switch or 
other management webpage to disable offending ports or track down MAC 
addresses?  I suppose this can be a discussion for another day.
 
CRLE

5-2) A quick edit of the recommendation

Dan Miller began discussion by reading through the proposed recommendation and noted a change which had already been made based on the input of Charles Benjamin. Namely, the phrase "as standard practice" has been added to the end of the last "Whereas" clause.

5-3) Is the recommendation too non-specific?

Charles Benjamin wanted to request another change because he felt the proposal was not specific enough as to exactly what devices would not be allowed. Charles specifically was concerned that the Cisco ASA 5510 Adaptive Security Appliance, which they are currently testing to provide firewall and VPN service between Housing and the VP Office in Tigert, not be included in this list. Dan Miller responded that exceptions would always be considered, so he didn't see this as a problem.

5-4) Recommendation vs. Policy or Standards documents

Ty asked for clarification from Dan Miller that this is a recommendation and not a draft of the policy itself. Dan responded that this was indeed the case and that the policy would have to come from the CIO. Ty replied that the policy certainly needs to be written, but he then suggested to Charles that his ASA 5510 deployment would not be a violation in any case; rather Charles is working with the providers at CNS on this--which is exactly what he should be doing. It is not like Housing is putting some personal device on the network. Charles still believed that there should be a very specific list of disallowed devices within the proposal; he noted that we want to serve our clients and not make it difficult for them. The flip-side of that would be a list of what devices were allowed.

Craig asked what we would do then as technology changed and device names changed or new devices came along. Charles responded that the list could be modified as necessary. Ty said you simply can't make a list of allowed devices; besides, this is just a recommendation that a policy be written. Policies should intentionally be left unspecific as they are intended to stand on their own and last a long time without modification. Policies should be very difficult to change and should require very high level review. You don't want to write a policy where, if somebody comes up with a new device in six months, you will want to change it.

Ty commented that the policy should articulate some sort of concept about what it is you want to do so you can evaluate a specific activity in light of the policy. If you want to have a list as a "standard", that is how that is handled at the Health Science Center. Standards can change without such a high level of review. Some discussion ensued on whether this recommendation should contain a specific list or not. Dan Miller said that he believed the wording made it clear that these are just examples of the types of equipment that would not be allowed. Ty suggested that it was not very important to wordsmith a recommendation; what will be important is to wordsmith the actual policy.

Tim Fitzpatrick agreed with TY and said that this should have been stated as a policy and/or a standard and then they should have asked us to review that. Tim indicated that he will do a follow-up translation of the recommendation for presentation at a future date. Tim believed, however, that the concept was pretty clear: don't extend the edge of the network with various devices. Tim will see if they can't craft the language into a proper document, but was still looking for agreement on the intent of the recommendation--even if we don't yet have the language correct currently.

5-5) Will mesh networking complicate things?

Allen Rout mentioned it is extremely likely that we will start seeing people bringing OLPC devices onto campus very soon. The interest in mesh networking that is being engendered by that technology is likely to leak over to a lot of other devices that people do not conceptualize as routers. But, none-the-less, by bringing their laptop onto campus and hooking up to WIPA, they will be doing something which could be construed as extending the edge of that network. Allen mentioned that he doesn't have a solution, but he does think the issue will arise this semester.

Dan Miller believes he does have a response to that. Let's take a step back from the current recommendation and look at how we ended up here, namely that CNS would like to continue to roll out the port security commands at the edge of the network. One of the main challenges, the one which we just can't get by at this time, is when people plug in switches and hubs at the edge of the network. That gives us multiple MAC addresses (plus there are other ways we can detect traffic that is coming through an unintentional loop) that should not be there. We have the same issue that Allen is describing as coming with mesh networking happening today with laptops that may have wireless enabled and bridging turned on. The port security commands should provide some level of automatic protection against the deleterious effects of those types of inadvertent backdoor connections. Dan didn't know that we could handle mesh networking today, but maybe by the time it hits in large numbers we will have something. That is the intent of this vote.

5-6) MAC address limit settings for port security

Craig Gorme asked about MAC address limit settings for port security. He was considering some of the things he has done and how port security might affect that. For example, he will hook up a hub between his workstation and his laptop to sniff traffic on his connection. That would cause two MAC addresses on that port. Dan Miller responded that we had started with a necessary minimum of three MAC addresses in order for an IP phone to work correctly. Then we realized that someone might plug in a laptop and then swap it out with another laptop for troubleshooting through the port on their phone--so we ended up at four. Wherever we end up, either at four or five, that should handle the occasional use. Still, Dan would suggest that the networking group should be able to provide sniffing traces should you have problems.

5-7) Bench switches for local unit IT staff

Clint Collins asked for some clarification on the bench switch issue, since this is something they use a lot. Clint wondered if there was any situation where CNS might deny providing one of those. The current recommendation wording specifies the need to discuss justification for those. Clint wanted to know if it was just a matter of asking.

Dan Miller turned the question over to Todd Hester for an answer. Todd replied that it is a case of asking. The bench switch is designed for your technical staff, people who are building or troubleshooting PCs. If that is what you are trying to accomplish, it is realistic that you be able to do that. Tim asked for clarification as to whether that is considered a temporary solution, or whether we have clients who have permanent bench switches. Todd responded that the CIRCA area has a permanent bench switch and Dave Pokorney mentioned the College of Business as being another example. Tim asked if bench switches would be available on a permanent basis or only on rare occasions. Todd responded that when there is an office whose clear function is solely technical support, then a tech bench could be supplied on a permanent basis.

5-8) Limits to an external device ban

Charles turned discussion back to his concerns regarding the ASA 5510, that CNS not decide six months down the road that this is a non-supported device and require that it be removed. Tim responded that this conversation needed to be taken off-line. Tim stated that CNS has agreed to a six-month evaluation project and we need many more documented evaluation criteria so we can measure those (beginning, middle and end) and see at the end of the evaluation period if what we set out to do did in fact avoid all the problems and accomplished all of the objectives. If it does that, Tim can guarantee Charles that this document will not be prohibitive or preventative for what Charles is requesting.

Tim stated that Ty has suggested we can recommend this matter in concept but we cannot approve a policy document because this is not presented here as a policy or standards document. Tim believes we need to come back with the policy or standards document, but we can still ask for agreement or disagreement in concept.

5-9) Can wireless deployment ameliorate the need for wiring remediation at units?

Dan Cromer brought up what Steve had said. While Dan supports limiting external network devices from a security standpoint, for economic reasons IFAS has implemented workgroup hubs/switches which have provided years of trouble-free service. Dan wanted to express that concern; he has one over in the Livestock Pavilion, for example.

Dan Miller asked if the Livestock Pavilion was on the Wall-plate currently and Dan Cromer replied that it was not. Dan Miller responded that, when it does become Wall-plate wireless will be available. As of just yesterday, CNS has agreed to move forward with the new 802.11n access points from Cisco as soon as possible--perhaps in as soon as two months. These should provide far superior data rates compared to what you see today in wireless and we would hope that occasional light and intermittent use could be handled through the wireless network.

Dan Miller also wanted to point to Chris Leopold's comments via e-mail where he says that these layer-2 devices cause problems. There are examples where there are no problems, but there are plenty of other examples where there are; it only takes a few problems to "ruin your day". One final comment Dan Miller had on the cost issue is that this is an opportunity for improvement. If this recommendation were passed and policy were made at a higher level, that would give unit IT staff ammunition for some one-time money to do some necessary wiring upgrades in each Wall-plate building as it comes on-line. Then you would be covered and you would have a solid network in each building; so, this could be viewed as an opportunity for improvement.

Charles Benjamin mentioned that 802.11n will likely not get ratified until late 2009 and asked Dan Miller if that was a concern. Dan Miller replied that they had done some extensive research into the subject and they feel that now is a good time to move forward. We have five units which we just received last week which we are going to be doing some testing on. Charles queried about the client side drivers. Dan Miller replied that this may be a good topic for discussion at the next meeting, but there are three main benefits of 802.11n technology and two of them are attainable with the use of MIMO APs without having the 802.11n client. Also, referring back to the standards, we have heard that the current draft is (with a reasonable level of confidence) likely to go forward. In other words, most laptops are coming out now with draft 2.0 wireless NICs and we are led to believe that software upgrades should bring them to compliance. Ty responded that they had made that same assessment at the Health Science Center.

5-10) Will port security turn unit support staff into end-users?

In response to what Chris Leopold had said Steve Lasley wanted to point out that Chris had referred to "user installed" devices. It is already against the UF AUP (section on Network Infrastructure/Routing) for users to extend the network. The point Steve wanted to make was that we have units with IT support staff, and this policy is essentially turning them into "end-users" with regards to the network. That is a concern as it could be argued to be a waste of existing university resources at the unit level. Some of the details which Steve does not yet see specified involve how we will handle exception requests, approvals, documentation and maintenance processes. He would like those to be fully developed (as they will be needed in any case).

Craig Gorme commented that the details regarding exceptions could be elements of a standards document. Ty suggested that, when the policy is written up, we simply address the matter of exceptions. All the policy needs to say is that there will be a procedure or process for obtaining exceptions to this policy. Steve agreed with Ty on that policy matter. He pointed out, however, that we are going to need a pretty hefty system to keep track of the exceptions. Consequently, Steve suggests that we go ahead with the development of that, and in the meantime allow IT staff to register exceptions.

5-11) Request that exception details be spelled out

Dan Miller responded that, just as in the case Charles raised earlier, CNS will be open to discussing exceptions on a case-by-case basis. Shawn Lander, however, agreed with Steve concerning the need to get procedures documented in writing, rather just state that IT support staff "will need to discuss the justification" with CNS. That documentation should address such things as who do you go to in order to obtain an exception, what types of exceptions will absolutely be over-ruled, who reviews the matter after initial discussion, and similar things. Craig added "how long a request would take" to that list. Steve mentioned, and Shawn agreed, that allowed exceptions should also require renewal so that assignments don't grow stale over time.

Dan Miller wanted to point out, as stated in the paragraphs following the proposed recommendation, that the bench and temporary switches aren't really exceptions to this recommendation. Dan Miller believed it had been mentioned at the last meeting that temporary switches could usually be provided on short order--maybe one business day would be a reasonable response time for that. Todd Hester pointed out that this would depend on whether the network had to be modified or not. If it is a service similar to what is already being offered in that area and doesn't affect the core (for which we would have to catch a maintenance window), then Todd felt there wouldn't be any problems with that.

Tim Fitzpatrick was to add that, if the concern is that standard operating procedure ought to be documented so that there are clear expectations on all sides, then he completely agrees. If, rather, the concern is that we write 100 pages, dotting the i's and crossing the t's, on the presumption that we are all bureaucratic self-serving monopolistic whatevers, that doesn't set very well with him. Tim then tasked Todd Hester with writing up the procedures.

5-12) Bench switch clarification

Clint wanted to clarify that CNS was okay with the concept of an agreed upon permanent bench switch for IT support staff. Tim responded that such had been the case since their very first pioneer Wall-plate customer, the College of Business; this was back when they were paying rather than having it provided for free. Nowadays there is the incremental cost of "what do I do with the wiring required if I have to replace a hub"; but the offset to that is that you no longer have to buy the switches and VoIP phones are being provided at half-price should you elect for that service. There are a ton of incentives that are being provided from the top so that we can all "kind-of win-win together". Tim would like to have the presumption of good faith intentions on all sides.

5-13) Cooperative effort to improve the UF network

Erik Deumens concurred that this is all part of an effort to try and build a new network that has a much higher level of reliability. You want to have a structure where you can debug and test and you don't want people to arbitrarily modify that. Erik believes that what we are trying to do at UF is much more flexible than what he has seen at other universities where exceptions may not be allowed at all. CNS is willing to allow exceptions as long as they knows how to do it and can manage it. So, Erik believes the general idea is definitely right.

5-14) Tasking out of Policy and Standards codification

Tim believed he heard a consensus that we are moving in the right direction and that we need to codify this as a policy and/or a standard, with some suggestions from the discussion today and we need to append to it some procedural implementation kinds of things. Rather than call for a vote today, he suggested that we get those together and bring it back next time we meet. This suggestion was approved without objection.

Tim said that a lot of this allows port security to be turned on and operate in an automatic fashion. We are saying that, on a short-term basis, you can plug in this and that. The procedure that Todd is going to write up basically will say that you had better tell us in advance because we have automatic processes which will shut you down otherwise. Craig asked if the standard will specify the number of MAC addresses which will be allowed, say 4-5. Dan Miller and Tim both responded that such a setting would be reasonable.

5-15) VMware and multiple MAC addresses

Charles mentioned that Housing is seeing an increase in the use of VMware by students. That means there are multiple MAC addresses when operating in bridged mode (the default VM networking arrangement that allows your VM to talk to the outside world via your host machine's network card). Dan Miller responded that with a modest number of MAC addresses allowed on a port typical VMware use is not impacted. The ESX automatic failover for high availability servers on the other hand is a major challenge to handle; we will just have to make an exception there and turn off port security for those devices. That would typically be in server rooms, however; the consumer-level host-level VMware should typically not be a problem.

6) Other Discussion

6-1) Websense as future discussion topic

Charles said that Housing is filtering web use via a product called Websense. He wanted to know if others on the committee were interested in him presenting on that topic at an upcoming meeting. Dan Miller agreed that we should plan on that, perhaps in March. Craig mentioned that Shands uses Surfcontrol (which was obtained by Websense by last October) on all the Shands-controlled subnets in the Health Science Center. Craig summarized that Shands doesn't want their employees visiting much of anything.

6-2) Content blocking in general

Tim mentioned that Operations Analysis, sponsored he believes by Ed Poppel, is exploring content filtering as well. Though Tim hadn't been in discussions on the matter, he had heard that there is some concern on the academic side that maybe we shouldn't be doing content filtering period. That is not an issue or debate in which Tim wishes to get involved. With Operations Analysis considering this, an organization which has many administrative units on campus that would be subject to filtering, someone at the higher levels is going to have to deal with this as a policy matter.

Dan Cromer responded that he did not believe we should have a policy that says zero content filtering, because he believes that this is appropriate for those kinds of things which are counter to our AUP--for example Napster or some other similar or illegal download service. Dan believes we ought rather to say there is going to be some limit to content. Allen Rout pointed to the distinction which exists between protocol blocking and content blocking. Saying we don't let Usenet through is different from saying we only carry the "good" groups. Allen has dealt with that particular question for a long time. Charles mentioned that the bottom line is that we have finite resources and we need to manage those appropriately. Tim mentioned that our AUP would be a much more relevant criterion to use for that than a freedom of speech consideration.

6-3) 802.11n wireless as future discussion topic

Craig asked that 802.11 discussion be added as a topic for an upcoming meeting as well. Dan Miller thought we might be able to fit that in next time, but will see how it goes.

Action Items

Subscribe Dan Miller, ITAC-NI chair, to all other ITAC committee lists for collaboration purposes (pending from previous meeting).
Draft a committee position statement on our need for a multi-year plan for reclaiming IPv4 address space (pending from previous meeting).
Discuss the issue of network security and how a unit's ISM fits into that equation. Will the ISM have access to the Wall-Plate switch or other management web page to disable offending ports or track down MAC addresses?
Re-work recommendation into policy and standards document format and represent to the committee at February's meeting.
Schedule times for miscellaneous agenda topics including: 2nd-site redundancy plans, the MRI network, UF Exchange project, 802.11n wireless, IPv6, IPv4 space usage, and content/site blocking.

Next Meeting

The next regular meeting is tentatively scheduled for Thursday, February 14th.

last edited 17 January 2008 by Steve Lasley