ICC Meeting:

IFAS COMPUTER COORDINATORS
(ICC)

NOTES FROM April 10th 2009 REGULAR MEETING

A meeting of the ICC was held on Friday, April 10th, 2009 in the ICS conference room. The meeting was chaired and called to order by Steve Lasley at about 10:00 am.

PRESENT: Fourteen members participated.
 
Remote participants: Dan Christophy, Dan Cromer, Dean Delker, Patrick Pettus, Louise Ryan, and Mitch Thompson.
 
On-site participants: Benjamin Beach, Bill Black, Dennis Brown, Stewart Collins, Lance Cozart, Francis Ferguson, Wayne Hyde, Steve Lasley.
 
Guests: Dan Miller and James Moore of CNS Network Services.
 

STREAMING AUDIO: available here.

NOTES:

Agendas were distributed and the sign-up sheet was passed around.

Report from the chairman

Member news:

Member news...

Please join Steve in congratulating Brian Gray on winning an IFAS Superior Accomplishment Award.

Recap since last meeting:

As per his usual procedure, Steve pointed folks to the notes of the last meeting, without going into any details.

Special Topic: WAN Service Level Objectives (SLO) agreement

Background (previous WAN transition discussion)

IFAS made the decision to outsource its WAN support to CNS during the first half of 2008 and James Moore was hired full-time in August as our main point of contact for that service. Since that time, James has been busy getting a handle on what is out there at our various CEOs and RECs and recently CNS has been developing a Service Level Objectives document in an attempt to define the various roles and duties of CNS and IFAS within this arrangement. This SLO is currently in draft version six which was made available to the ICC on Monday, March 30th.

Introduction

Steve expressed his belief that the very fact that IFAS will have a known slice of funds allocated on a recurring basis for the upkeep of our WAN is a most promising trend. In the past we just had to scramble to fix it when something broke, and upgraded when funds were available at the end of a fiscal year. The other encouraging aspect of this is that the details are being discussed openly prior here.

The question of funding

Steve believed the question "What is this going to cost us?" might be foremost in the minds of the RECs and others. The primary challenge, in his opinion, is that the CNS equipment budget is tight for the number of locations we are trying to support and maintain. As the SLO says, it seems clear that there will be other costs outside the $50k/yr which will need to come from somewhere in IFAS.

Since Dr. Joyce could not attend our meeting, and it was uncertain even if Dan Cromer could, Steve asked Dan prior if he could provide some perspective on how any extra financial responsibilities might be allocated between IFAS and the units--and how improvements might be prioritized across our various WAN needs. Here was Dan's response:

On the CNS side of things, Steve had asked Dan Miller if he could provide approximate current cost estimates for the sorts of items they will be specifying which a CEO or REC might want/need to subsidize.

Prior discussion via the ICC-L

Discussion of the SLO had begun via the ICC-L prior to the meeting. Here are some of those exchanges:

Comments and questions from Louise Ryan, NW District IT Support - 3/30/2009

Response from Dan Miller, CNS Network Coordinator - 3/30/2009

Steve ended his topic introduction by pointing out that another of the other major issues requiring coordination (among a host of minor ones) is the matter of communication between CNS and IT support in the field. At that point, Steve turned things over to Dan Miller for his perspective.

CNS SLO overview

Dan Miller began by mentioning that the SLO document is not intended to be an ironclad agreement covering every detail of the service. Rather, it is meant to outline in sufficient detail who is doing what.

Scope

Dan Miller related that a scope section was included in the SLO to provide an opportunity via negotiation with the Executive Associate VP for units to opt-out if that was their desire.

Roles and Responsibilities

This section of the SLO provides descriptions of the various groups of people involved and what they do. The IFAS Technicians involved includes the five district support people but also includes certain IT staff at the various RECs. CNS wants to get an explicit list of IFAS IT Technicians from the point of view of this agreement. Those will be on an e-mail list and CNS will work with them very closely.

Chris Leopold's group will provide a number of important application level services as well, and IFAS FacOps will continue to be involved in WAN facilities. Outside contractors may play a larger role depending on how things work out and VCS will be closely involved in coordination of the videoconferencing aspects.

Dan Miller related that Fedro Zazueta had presented a number of suggestions to the IFAS Administrative Council this week regarding the reliability of our videoconferencing system. The major points related there included assuring that Polycom units are left on 24/7 at all locations and that there is some trained staff available and on-site during any scheduled videoconference. Patrick Pettus responded that he thought this was well received. Steve notes that his Chairman was sufficiently impressed to come and discuss the matter with him--so it appears to have been effective in at least that single instance.

Dan Miller noted that the purpose of this SLO is similar to what VCS is trying to do with videoconferences. They are trying to formalize the support model as much as possible in order to improve overall services. CNS wants to bring us to a better place where the network is easier to maintain and is more reliable.

Design standards

Dan Miller noted that what is included here is just a high-level overview of current design standards. There will be a separate design standards and procedures document which will get into more of the details. That latter document is something which CNS will share closely with the IFAS Technicians.

Funding Model

Dan Miller mentioned that CNS was very careful to point out various types of situations where IFAS would need to spend money.

Problem Remediation

Finally, Dan mentioned that the SLO contained some overall outline of how problem remediation responsibilities would be assessed and handled.

Questions from Apopka

Mitch Thompson had presented the following questions via e-mail:

Problem remediation and unsupported devices

Regarding questions 1 and 2, Dan Miller responded that CNS is going to be very clear about what devices are on the network. They should all be documented and CNS is still in the process of doing that currently. CNS will be monitoring and alerting on those devices. As the SLO states, CNS wants to be the ones who handle the network design and change management. That means that CNS needs to be aware of any changes that go on in the network. Unless it is an absolute emergency, nothing should be plugged into the network without prior consultation to CNS. Even in the case of emergency they have on-call engineers who can be called to try and work with people.

Steve asked Dan to contrast this with the Wallplate in that regard. Dan responded that the Wallplate support model is at a higher level because there is more money behind it. The biggest difference is the equipment funds; CNS has a $5/port/month budgeting model much of which comes directly from the Provost's office. Staffing levels is another large part of the picture however as well. There is a relatively built-out model for Wallplate with three teams of field engineers who respond on-site. CNS had to go to a more distributed model with the IFAS WAN service.

Consumer grade devices will gradually be replaced

Regarding adding devices to the network, James Moore wanted to clarify that this didn't involve things like moving hosts around. It was more an issue of not adding WAPs or unsupported devices like network hubs. Dan Miller agreed that CNS definitely wants to get away from that but they also realize that it will take some time to get there. They don't have the money to replace everything year one; they're not even close in fact. Nor do they have the time. Consequently, CNS is going to do best effort support of things like Linksys and HP switches, but the lower the quality of the equipment the less interested CNS is going to be in making it work--rather, they would tend to suggest a replacement. Consumer grade equipment will be high on their list to go. HPs can probably be lived with for a number of years.

Mitch Thompson commented via e-mail:

When Ben Beach raised a point about county offices in this regard, Dan Miller said that this was not going to be a case of "rip out that Linksys right now". CNS will work with the technicians to determine what is out there and what is supported; if something new comes up they will try to determine how that happened and what they can do to get rid of it. They will be less accepting of new consumer grade devices once their discovery phase is complete and they have everything documented. Dan said that this is what he is referring to in the Problem Remediation section of the SLO where unsupported network devices are mentioned. In the long run, CNS cannot support undefined devices; they need to know precisely what is where.

Guest access

Mitch's question about network access for non-Gatorlink guest users arises continually. Steve pointed out that the IFAS clientele is very broad and remote sites support many extension-type activities involving brief but large groups of visitors. Dan Miller responded that he has had numerous discussions on that topic with Dan Cromer and with Warren Curry. There have been changes to the Gatorlink guest standards so that "guest" is going to work for the sorts of circumstances to which Mitch refers. The direction is toward "Departmental Associate" supported Gatorlink access for longer term "guests" and the Gatorlink Guest account for short-term needs. The latter may be created individually by those having the UP_PA_GA_GUEST_SINGLE role or in bulk by those having the UP_PA_GA_GUEST_BULK role. That is accomplished via the UF web portal. If you do not have those roles, you can have your Directory Security Administrator request it on your behalf. When that happens you will see "Gatorlink Account Management" item added to your web portal menu.

Distance education access is a separate matter

Dan Miller added that there is a move within Distance Education to make it very easy for people to sign up for continuing education or distance education classes. While Dan feels making such a process simple is a good thing, he is concerned that we not tie that to providing network access. Except for those on the county networks, all our users are on UF network address space; We need to adhere to UF AUPs and security practices so that we know who is using the network.

What about separate Internet-only access?

Steve asked about providing Internet only access that was segregated from the rest of the network. Steve wondered if that wouldn't solve some of the issues. Dan Miller said that this could be done via a split tunnel on a VPN; they have discussed that primarily as an option for when they are dealing with circuit problems, however. The best connection which the RECs can hope for is an FLR circuit which is 100MB point-to-point. That is where we hope to go. Buying external internet connectivity would be an extra cost item that would add considerable complexity.

Guest account administration deemed cumbersome

Louise Ryan commented that she didn't understand how Reitz Union Hotel where guests are provided a Gatorlink guest account for network access. We need to improve our Guest account creation procedures. Dan suggested it would be simple to create those in bulk via an Excel spreadsheet. The sponsor of the event can be in charge of creating those; it is not a "self" identity, but rather a "representative" identity. Dan suggested that it was simply a matter of developing the mechanism and training folks on its use. Dan Cromer, however, seemed to believe that bulk accounts could be prepared ahead of time. The problem with that is the process currently requires knowing specific contact information about these individuals, including first and last name, phone number and e-mail address. Steve believes Dan is underestimating the administrative staffing costs involved. The information entered certainly cannot be validated in any fashion, so the value of doing this is suspect in any case. It seems to Steve that some other solution should continue to be investigated. Trying to fit a square peg into a round hole just doesn't make sense. Mitch supplied a comment via e-mail:

While driver's licenses are not involved in Guest account creation per se, Dan Cromer had suggested that those might be utilized by event sponsors to ensure identities. Mitch has a point regarding the overhead and inconvenience inherent in the process for both local staff and the participants. Dan Miller said that CNS isn't going to immediately rip out what IFAS has in place, but that Gatorlink guest accounts and streamlining the processes for their creation is the long term solution which is being investigated. We are going to have to see how much progress we can make on this in the next year, but we can't just have wide-open access. If that doesn't pan out, then a separate circuit might be an option; that would involve considerable extra cost, however.

County networks

Ben Beach asked for clarification regarding county networks. James Moore responded that we are only directly concerned with the UF network space with regards to security and tying usage to an individual. Users on county network space would be the concern of the county and not our direct concern.

A mix of networks at the CEOs

Dan Miller said it was his understanding that we had cases where CEOs included a mix of county and UF networks while others were supported over county networks only; CNS is only going to support the network needs of folks on the UF network. They may talk to them a little bit if they are having problems, and IFAS IT will have to provide desktop support, but those folks will not be plugged into a network device which CNS manages.

Ben provided the example of Nassau County. They have a DSL connection which CNS arranged for but which is paid by Nassau County. James suggested that where there is a DSL connection, even one that is county funded, that CNS would likely have a managed router there. Ben said that this was not the case at the Nassau CEO. When he installed a Polycom unit there he had to remote into the DSL router and configure IP forwarding for that.

Stewart Collins asked about Ruskin, which is has a satellite office for the Fisheries and Aquatic Sciences department which he supports. They use that facility as a teaching lab, among other things. James responded that CNS has a managed router at that site. We are still in the stages where CNS is getting to know which IT support staff has interests at which sites; conversely, we still have support staff that aren't aware of James Moore and the support he can provide regarding networking to remote sites.

Videoconferencing endpoint documentation concerns regarding the network

Lance Cozart mentioned that it would be good to have a list of which CEOs were UF vs. County supported. Regarding our distributed Polycom assets, such a list would help him in understanding which units might be more difficult to manage. Lance is in a situation where he needs to support our remote Polycom sites. In particular, folks at remote sites sometimes change the Polycom configuration and break things; in those cases he needs to get things set back. Knowing more about the connections would help him greatly.

James responded that Ben Beach has arranged a place on SharePoint for the district support folks to post information regarding the various CEOs and that there was a spreadsheet there which he would try to clean up a little, but which could provide Lance with the information he sought. Lance added that it would be good to have VCS plugged-in to any network changes which might occur so they can update their own database accordingly. Patrick Pettus responded that he has been working closely with James and that he didn't believe that aspect will be problematic.

James said that they just ordered new Foundry (recently purchased by Brocade) switches for some remote sites; while they have much more experience with Cisco, it was decided that Foundry was the best lower-cost solution for IFAS. Now they are working to learn more about those switches, testing feature sets, and figuring out nuances regarding CLI syntax and the like. Once they have that James will work in lock-step with Patrick Pettus so that VCS has captured configurations for each site. CNS is aware that videoconferencing is a high-priority application on the IFAS WAN

Lance said the other item they needed to track were the connection speed recommendations for videoconferencing at the various CEOs. The 512kbps requirement included in the SLO will impact a number of our sites. James mentioned that he has had about 8 tickets over the last week dealing with Polycom issues at Volusia County, Hillsborough County, and Sumter County. In getting with the provider at Volusia, for example, he has found that they can actually upgrade their connection while saving $15 a month. He is discovering some locations which are on "best effort" connections currently which really need to be upgraded to at least a 512kbps uplink. Dan Miller added that they particularly wanted to rule out 256kbps connections for use with videoconferencing. James pointed out that it isn't just the feeds but the configuration that is a concern as well: things like static NAT settings and NAT awareness on firewalls must take the Polycom units into consideration.

Videoconferencing network troubleshooting can be tricky

As an example of the sorts of things they deal with, James related that at Sumter County they ran into an issue that had something to do with the older equipment in place there currently. When they do a static NAT translation on a router IPSEC does not take any of the traffic from a statically NATed host and apply that to the tunnel back to campus. That is actually good for a DSL site with 512kbps up because the encryption overhead is a good thing to avoid. But even though the payload takes that unencrypted path, sometimes a control frame gets applied to the tunnel, so it gets there late and the bridge drops the call. Consequently, they have changed the configuration to filter communications from the Polycom through the tunnel in order to fix that particular issue. Also, at Sumter, every time they turn on the Polycom they see error messages on the LAN side; consequently, the router and patch cable are being replaced there Monday by Francis Ferguson, in order to eliminate those issues.

Dennis Brown asked if the network connection to the Hastings farm was 256kbps and whether there was any prospect for upgrading there. Ben confirmed that the downtown office was on a DSL with 512kbps, but the farm was still at 256kbps. The farm is about six miles from the downtown office in the middle of a potato field.

Connection upgrade plans for RECs

Regarding the RECs James related that he has a meeting next week with FLR and they have proposals back from Verizon, MCI, Deltacom, Quest, Embarq, Sprint, Florida Power & Light, and AT&T. The numbers are coming back promising. He is a bit concerned over potential connection charges for our copper-based connection sites like Homestead, and they are doing site inspections which are causing a bit of a holdup. James has a lot of questions for those providers in this meeting next week and they are going to ask them to show a bit more than they usually show folks so we understand how their MPLS backbones are configured, what VRF we are going to land on, and do some testing before we agree to anything. We are looking at maybe a two week turn-around on having a hard answer there. Cost-wise, however, things are looking good and James believes all the RECs will be able to get new connectivity. One of the new FLR agendas is peering connections with all these providers and Brighthouse is one of those. That promises to improve routing paths among our various locations and could positively impact our video performance at some locations.

County firewall concerns

Lance asked about county networks that might be blocking certain protocols. James related that his experiences with counties have varied widely. On the one hand, Bay County was quite friendly and open to him while Okaloosa did not want him there. Francis added that Orange, Volusia and to some degree Marion County were also quite resistant to IFAS or UF IT visits. Dan Miller said that CNS would negotiate with the network providers on behalf of VCS and IFAS WAN users/technicians at sites that were county-only sites or sites which were mixed county/CNS. CNS talks their language and can perhaps get a little further into at least defining the situation so CNS understands things. Where CEOs have the finances available, perhaps a DSL can be pulled in with a managed router. CNS can standardize things in such cases, but county run sites will continue to be all over the place as they have no interest in standardizing among themselves. Francis pointed out that some places like Orange county were interested in using the Polycoms, however, so that might provide some impetus for getting them to standardize such things.

Clarification on the deployment of Wallplate

Stewart Collins noted that Fisheries way out in the Millhopper area recently went Wallplate and he wondered if there were plans for that at any RECs. Dan Miller responded that Wallplate is only for "on campus" though there is some flexibility about what "on campus" means; that depends on funding, classroom education, type of circuit, distance, difficulty in getting there, etc. That allowed Fisheries to be included in Wallplate, but this would not be extended to any RECs. On the other hand, if a location is eligible for Wallplate, then that is the only CNS support option. Outside that area the option is IFAS WAN and units are in unless they opt-out.

Communication and notification between CNS and IFAS technicians

Steve tried to bring discussion back to the issue of how communication and notification between CNS and IFAS technicians would be handled. That seemed to be a concern of many and Steve wondered how much of that process or the expectations there should be included in this SLO. Dan Miller expressed his opinion that such details did not belong in the SLO, but rather would come down to standards, procedures and change management. The SLO is meant to be general though there is mention of that in there.

Nagios monitoring

Ben Beach had raised this issue to Dan Miller previously and Dan has talked to their monitoring group regarding problem notification procedures. They utilize Nagios to monitor all network devices. Those are categorized as critical or non-critical depending on whether or not they would wake up an engineer in the middle of the night if the automated system saw something go away. Most IFAS sites are considered non-critical. CNS monitors 24/7 but waits until the next business morning to get on it unless someone from IFAS calls and is willing to go on-site to help CNS troubleshoot. In many cases it is a power outage or circuit outage, which gets corrected pretty quickly.

Outage notification via e-mail

Nagios can send out e-mail notifications, and it was suggested that the easiest answer to Ben's concerns was to put the IFAS Technicians into the Nagios system as getting e-mails when their sites go up or down. They will probably start off with problems at any IFAS site sending notifications to all IFAS techs; he is not sure how long it will take them to get to a more granular approach. These sorts of programming matters often take some time to implement, but James noted that if IFAS has particular needs, CNS is ready to consider and address those. They are also considering giving some IFAS technicians the ability to log into Nagios to see the alerts and what is up/down should they wish to check.

Hurricane procedure planning

Ben raised the issue of a hurricane situation where we need to know who is down so we can get in touch with them. He has experienced situations where folks in county offices had no phones or cell phone connections; Ben needs to know what got hit and what didn't so they would know if they have to make a site visit to assess damage. Dan Miller mentioned that CNS has some hurricane procedures specific to securing the data center and a little bit for the campus network. We probably ought to consider what is special and different about hurricane scenarios.

Business-hour versus off-hour procedures

James said that if something goes down in Nagios during the day, CNS responds by generating a ticket which immediately goes to a dispatcher who has to physically make contact with an engineer and ask if they are handing the ticket. James supports the IFAS WAN but there are a lot of other technicians on call to take tickets for the IFAS WAN (or there will be very soon). Procedures internal to CNS will see to fixes during normal business hours; the person who handles that ticket just needs to know who to call, so that is a training issue for CNS internally: call the IFAS tech, call the site, verify power, etc. Where the e-mail from Nagios issue came in was in response to afterhours support. What if the IFAS Tech wants to be notified during the night of what to expect when they come in at 8 in the AM. CNS doesn't expect IFAS Techs to respond, but it might improve things if IFAS techs were e-mailed in real time. During the day, CNS will definitely call the IFAS techs in the case of any outage.

IFAS Technician initiated tickets

There are times as well when IFAS technicians will have to submit a ticket to CNS, where Nagios will not see an outage. For example, a routing problem might be preventing a user from getting to a particular web site or a Polycom is not working because it has lost its gateway. You can call in and operations will fill the ticket out for you if you are in the field. You can try to get a hold of James, but if you don't you can call the main number.

Dan Miller mentioned that they will have a well-defined list of "IFAS Technicians" who are responsible for assisting with troubleshooting at remote sites. They will also maintain a list of site contacts; if there is a problem they will try and contact the local site to find out what is going on. There will be a difference in how CNS interacts with sites that have local contacts as opposed to IFAS Technicians.

Ensuring that the most useful reporting paths are used

Regarding Mitch Thompson's question on direct contact to CNS, Dan Miller emphasized that unit weren't required to go through the IFAS Help Desk system, but that it was highly recommended. What they want to avoid is having unit staff calling CNS for desktop support, because that is something which CNS cannot provide. Doing so would needlessly burden CNS and slow down a solution to the problem as well. Unit staff need to be instructed how to route issues in various circumstances. If they cannot reach their district or RECs support person then the IFAS Help Desk is likely the next best level of support.

CNS and IFAS Remedy systems to be integrated

Dan Miller mentioned that work is being done on programming the Remedy system to permit interaction between the CNS and IFAS Remedy interfaces. This will allow the Remedy system to be used to refer tickets either way as necessary. Dan Cromer and Dan Miller are in alignment with wanting to track all the issues via Remedy so that problem types and their frequency can be documented and analyzed. Until that interconnection is in place, CNS will use e-mail notification in cases where a CNS ticket calls for action by an IFAS Technician and CNS will continue to track those results in their own ticketing system; that's the easiest way they can do such things today.

Opting-out of the IFAS WAN

Dan Miller wanted to point out that any unit wishing to opt out would still be responsible for authenticating users on the UF network. The security requirements, the AUP and the IFAS reporting requirements all still apply and that is clearly stated in the SLO.

Temporary network devices

Dan Miller related that they hadn't really discussed in detail temporary routers or switches. He supposed that if a strong enough argument was made CNS could come up with a predefined preconfigured device which could sit on a shelf until needed. An IFAS Technician might then be able to deploy those upon occasions where high density connections were required. In general, however, CNS would prefer wireless to support such intermittent needs.

Mobile users

In situations where people come and go, we need to know who is connected. This is true even if it is to a temporary switch. Such ports, wireless or wired, will need to be connected to the authentication system and users will have to log in with Gatorlink.

Cisco NAC to be our authentication system

The long-term goal of CNS will be to use the Cisco NAC system which they have just purchased for UF campus use. That is where the on-campus Bluesocket authentication will be migrating, a process which has already begun. There are a couple more phases in that project and part three (perhaps by year-end) will involve the capability to take wired ports back to the central NAC servers. At that point CNS can provide this service on wired ports. There is a cost associated with this; it is approximately $20 per concurrent use. We currently have about 4500 wireless users at any one time and that is how many licenses we need. CNS will cover that cost out of central funds. There is also a $3 per concurrent user maintenance cost for the NAC system which CNS is covering as well.

Gatorlink access by "Departmental Associate" affiliation for related non-UF employees

Stewart mentioned that their facility hosts an outside state agency, the Florida Fish and Wildlife Conservation Commission in a couple of buildings. There is wireless there but these folks do not have Gatorlink accounts. Dan Cromer responded that the Directory Coordinator for Fisheries could create Gatorlink IDs for those personnel and relate them as "Departmental Associate". They could then create their own Gatorlink accounts and get wireless access.

Port security and the IFAS WAN

Steve asked if port security was being considered on IFAS WAN switch ports to control proliferation of workgroup hubs. Dan Miller responded that they have not finalized those standards yet, but he imagined that they will have some level of MAC address limiting to prevent loops. They see more loop issues on campus because the networks are larger, but it is still a concern and it would be more of a concern at RECs than at CEOs. James added that this would be handled on a case by case basis, but if more port density is needed there should be a way to fill that need without adding unmanaged hubs. Steve suggested that Mitch's concern regarding hubs related to the same concern he had with Wallplate, where CNS essentially relegates IT staff to the level of "user" on their own network. Some level of distributed management should be allowed for in order to allow local staff to address needs more economically where necessary.

A cooperative attitude should be the norm

Dan Miller said that their requirements will not be extremely strict as they are with Wallplate. The IFAS WAN is supported to the switch, not to the Wallplate. Local staff will be managing the cable plants in the case of the IFAS WAN; that is not the case with Wallplate. CNS will consult and are definitely interested in seeing the cable plant clean, organized and documented--as that will help everybody in maintaining the network. It is not the responsibility of CNS, however, and they will not be enforcing anything in that regard. They can leave ports turned on. Their only concern is that the local contacts and/or IFAS Technicians can explicitly identify who is using each port in the case of a security problem. If someone sends a death threat to the President, the FBI will come knocking on our door and we need to know. If it is port supporting a mobile audience then we need to take steps sometime in the next year or so to move that to an authentication system.

James said that if wiring cost was a primary issue and a hub deployment was determined the only financially feasible solution at the time, then CNS would at least appreciate knowing that the device was there so they could document it. Dan Miller said that he is not happy about allowing those but he has to recognize some of that may be necessary at this time. What he would like to move to, however, is providing a switch port for each device. That is where the CNS budget comes in. Other than the wiring, the RECs would only be hit financially if they wanted to upgrade to GB speeds. CNS believes they have funding now to meet the needs of all 10/100 wired hosts and they would very much prefer to see that all go back to managed switch ports. If there are to be exceptions then CNS would want to know about them and document them.

Switch replacement cycle

Francis Ferguson commented that a 9-year replacement cycle seemed a bit long. Dan Miller agreed, but funding does not permit the 5-year cycle they would prefer. They had gone through a cost modeling project where they compared HP, Foundry, and Cisco. Cisco costs would have necessitated a bit over 10-year replacement cycle. HP was just a bit less than Foundry, but Foundry had considerably greater features and compatibility with Cisco at the time they signed the contact.

Dan Miller said he would much prefer a 5-year life cycle, but this is still better than what IFAS had before, in general. If a location has money to subsidize getting their site fully up to speed then CNS would welcome that help. Dan's original concern with taking on the IFAS WAN was the lack of a hardware budget. They then worked out that a $50,000 chunk would be allocated specifically to hardware. That seemed minimal but acceptable. Dan wanted also to emphasize that this includes enterprise level equipment everywhere; including switch ports, WAPs and routers. They inherited a lot of end-of-support Cisco router equipment with this project and replacing that will be one of their main early priorities both this year and next year.

Equipment costs

Steve asked Dan Miller to provide an estimate of equipment upgrade costs so that units could use those to estimate whether or not opting-out made sense from a financial standpoint. Dan Miller was very sure that no unit could provide the same or better level of equipment and support outside of the IFAS WAN support agreement without a greater cost.

Specific prices

A 24 port Foundry switch with fiber uplinks costs $657. A 48 port switch costs $1097. These are 10/100 with 1GB ups. If an IFAS site wanted to have GB to the edge, the upgrade cost would be $440 for a 24 port switch and $880 for a 48 port switch. The standard access point today is the Cisco 1242; with an antenna and power injector, these cost about $470. If a remote site wishes to upgrade to 802.11n with the "a" radios, the cost would be about $1000 total or about $4-500 more. There are some changes coming with the WAP models where CNS is hoping to go from the 1252 to the 1142 so you don't have to buy external antennas which run roughly $200. They are still working through the specific prices on that.

Cost shouldn't be a reason for opting out

Dan Miller stated that he hoped that no unit would opt-out. He believes that standardizing on the same support at all locations would make life easier for everyone. It would allow remote units to use their time and money to focus on other things. If a site wished to opt out CNS would be glad to do a differential cost comparison to help with that decision. Dan would wager that the Foundry solution subsidized by CNS would be cheaper than any solution units could provide for themselves.

Control issues may be another concern

Steve added that the other issue that might relate to a decision to opt-out would be the matter of local control. James mentioned that CNS is willing to work on local control options. He gave the example of Mike Ryabin at Ft. Lauderdale who wants the capability of moving phones or voice VLAN ports. CNS can configure ports so that local staff can control such things simply by moving patch cables around on the switch. There will also be tools made available just as on Wallplate. They will supply graphs and provide tools so local staff can look at ports and see what VLANs are assigned, etc. They can view traffic and errors on the switches. There is also nmon and things like that which can be rolled out to give local staff control over top talkers and pulling people from the network and things like that.

Dan Miller mentioned that the Wallplate had a plan a few years ago to develop an application that would allow local admins to go in and change VLAN assignments to ports on the fly. They realized, however, that they can provide good service in almost every case. The only time that might become an issue would be in the case of an emergency and there were in fact very few emergencies. If there is an emergency CNS has an on-call engineer 24/7. Consequently, they backed away from doing that on Wallplate, saying they can give you the VLAN you need on the port that you need when you need it. If you need it now, then call us. The IFAS WAN is a little different, but what they are talking about doing as James said is that unused ports can be assigned a mix of VLANs, local staff can be made aware of what VLANs are available on which spare ports, and they can move things around by changing where the patch cord in connected. CNS would not encourage them to move cables often and cause a lot of churn, however, because that would disrupt historical data which would be useful for troubleshooting. VCS would, of course, want Polycom units to remain on a given port as well. But CNS believes this is a reasonable solution that would provide a level of local control.

Lance pointed out that Polycom units at some sites are kept on a cart in a closet. They need to get UPS-backed power and network to those closets so the systems are always up and running so they can be monitored and maintained.

Travel and shipping budgets

Shipping budgets currently don't exist

Francis Ferguson asked about the specification that IFAS Technicians "must have adequate travel and shipping budget". He and Bill Black both stated that they don't have a "shipping budget". Dan Cromer said that shipping can be arranged but that he needed to investigate the details. He assumed that district staff could use their P-Cards at the UPS store for doing that. Dan Cromer doesn't want any huge cost surprises there, but believes that any reasonable needs can be met.

Travel budgets have been shrinking

Bill Black mentioned concern that his travel budget comes from the district and has been regularly reduced over time. Dan Cromer responded that travel to support the districts should be coming from the district budget (except for places like Ona). If the district extension director has cut that to the point where they can't travel to fix things then IFAS IT will do its best to cover that; this is something which Dan would negotiate with the district directors. We can't have a group down, however, simply because the travel budget is gone. James commented that some of this can be handled w/o travel if they have someone on-site who they believe competent to be talked through assisting with a solution.

GB copper for the MPS may be shoehorned-in

Wayne Hyde asked if the Foundry switches being deployed had any GB copper ports which could be used for the multi-purpose servers. James responded that they all have dual GB copper ports. James said that any of those not being used for uplink can be utilized for such connections. Dan Miller said that while Wallplate is by the port, the IFAS WAN is by the switch, by the site, by the year.

The rationale behind the SLO

James said that early on CNS didn't feel they wanted to support the UPSes and the replacement of batteries out of concerns for eating too far into the equipment budget. They were trying to keep the replacement cycle as short as possible. It is not that they don't want to spend CNS money on certain things, but rather that they have to be cautious about how they spend IFAS's $50,000/yr in hardware money. The SLO is not a vehicle for CNS to sidestep responsibilities in the service they provide the IFAS WAN. The SLO is just so that CNS can present to IFAS what they think are IFAS's expectations of them and what CNS can deliver. Then CNS can get feedback on questions that might arise or aspects which should be added. If CNS's target goal doesn't match 1-for-1 with what IFAS's expectations are then CNS is bound to fail. The SLO is a good medium for all of us to say here is what you want us to deliver and this is what we think we can deliver.

District support overlaps

Francis Ferguson related that district support was not cut strictly along district lines. Francis and Bill each handle several counties out of Kevin Hill's South District. He wanted to make sure that CNS was aware of this so that they would know which of them to contact for which county. James replied that the spreadsheet Ben had put together had some of that in there.

District support handles aspects at certain RECs

Bill also mentioned that he picks up the Ona and Ruskin RECs. Ben added that every district support tech is supposed to co-support the surrounding districts. Dan Miller asked if REC IT staff helped out the district staff and was told that this did not occur as a general rule. They are paid from REC funds, not IFAS IT or district funds, so it is outside their scope. Bill Black mentioned that there is an MPS and DC at Lake Alfred that he manages as well, though he doesn't have any jurisdiction at CREC. Balm and Plant City are other like examples where he supports DCs and MPS servers at RECs.

Ticket handling changes to support upcoming deployments

James said that this information was useful for them to document because other CNS engineers would soon be handling IFAS WAN tickets in tandem in order to free James up a bit to work deployments at Belle Glade, Ft. Lauderdale, etc. The router upgrades will take CNS through the summer at least because those will not terminate the DSL connections and James has to coordinate with the various vendors to get a modem in place at each location.

Dan Miller mentioned that Belle Glade will be the first site to get a complete lifecycle upgrade of all their switches; this was done because they have a major expansion coming up shortly. Ft. Lauderdale will likely take the remainder of this year's budget and then they will start looking ahead. James said that once they get their overall plan in place for lifecycle replacement there will be transparency. They will be meeting with Dan Cromer and central IT and that will likely be presented to the ICC as an opportunity for feedback.

Ben mentioned that he was going to start a contact site on SharePoint so the remote techs can all update their contact information along with the sites they support.

[Ben created both a discussion board and contact list on the ICC SharePoint site as a place to follow up on discussions from our meetings, and generate an up-to-date list of all the IT support personnel and the sites for which they are responsible. See the "ICC Topics for Discussion" and "IT Contacts" items in the left-hand Quick-launch bar.]

Dan Miller said that he would rather have that information go on the CNS IFAS WAN Wiki. James said that this is the long-term plan for centralizing collaboration, but that it has been a bit slow in getting going.

Wrap-up on WAN SLO discussion

Steve thanked Dan and James for coming to discuss this with us today. He hoped that they got some good feedback and was sure that the ICC now had a better idea on what the IFAS WAN service entails. He noted that a number of the RECs were not represented here today but hoped that they would chime in via the ICC-L or via private e-mail with any questions or concerns which they had.

VoIP

Steve failed to mention this at the meeting, but a new Voice over IP (VoIP) Telephone Standard was announced March 24th. Since up until now VoIP has required Wallplate, Steve wonders how that might relate to the proposed CNS coordination of VoIP at remote locations. Dan Cromer had responded to Steve's e-mail query on this as follows:

REGEDIT4

; ----------------------------------------------------
; 
; REMOVE the IE-SPYAD: Internet Explorer Restricted Sites List
; 

[-HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\Domains]

[-HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\Ranges]

; --------------------------------------------------------------------------------