|
|
ICC Meeting: |
IFAS COMPUTER COORDINATORS
|
Prepared questions from the ICC--with answers by Todd Hester:
|
Wall-Plate history
[slides one-three] John Madey displayed a PowerPoint Presentation and began with a brief history of the Wall-Plate which has been documented elsewhere. John mentioned that they had done phone leasing at one time but have stopped that now. CNS is currently providing incentive pricing for VoIP phones at the time units are approached regarding participation (ref. the Implementation Schedule).
Wall-Plate overview
[slide four] John related that the Wall-Plate (WP) project is basically a life-cycle replacement for your switches, providing the initial up-front costs of upgrading your network as well as maintenance, monitoring and troubleshooting 24/7. Along with WP they also provide basic wireless services covering your entire building. If you require more intensive wireless access (say for a classroom where two dozen students might be using it simultaneously) then the unit will have to fund that separately. To receive VoIP, you must be a WP client because CNS must manage the network connectivity in order to guarantee the VoIP service.
Project contacts
At this point John handed things off to Todd Hester for him to describe the Wall-Plate networking aspects. Todd passed out a WP project contact list. Todd mentioned the Wall-Plate web site and its associated FAQs, saying that he planned to add the above questions and responses to those after today's meeting. Todd's three team leaders, Mary Byrd, Sheard Goodwin and Plant Rodgers each run a four-member team that will be the folks doing the actual installation work for the various buildings. Rosa Jackson is the Customer Service Representative (CSR) who (along with her team) will work with you over the telephone to arrange some of the implementation details. Judy Hulton provides coordination with outside wiring contractors for any wiring remediation which is required. Todd also provided links where units can submit telecom work orders and network trouble tickets and work requests.
Tim Fitzpatrick noted a couple of other key players which had been omitted from the contact list. Bruce Engstrom is not merely the processor of telecom work orders, but also assists John after the initial contact. Barbara Sawyer is their outside plant person who takes care of fiber to the building should that be needed. Barb Sedesse is the manager-level person to whom Rosa reports; Barb makes sure that the customer service and billing sides of the VoIP service are maintained and handles the budget of this multi-million dollar project.
Project workflow
Todd then presented a general WP Workflow Chart (get Visio Viewer). Each job is a little different, but this chart describes generally how things go when they come to your building. It begins with the initial meeting where they provide details on WP so a unit can make their decision to participate or not. The column on the left details the wiring side of things while the right-hand column does the same for the telecom portion. Thus there will be two groups working in parallel: one group will collect information about your network and the other will be doing the same for the telecom side. These two groups coordinate as indicated by the middle column. Todd emphasized that our assistance in getting all the informational details settled correctly is very important as well; the more information they have up-front, the better they can be at tailoring the system to your local needs.
In terms of actual equipment roll-out, they usually come in Sunday or Tuesday mornings during maintenance windows to start swapping out the electronics in your closets. Once all the network is fully deployed, tested and is running then they will start deploying the telephones. Those usually go out at a rate of about 40/week. Phones will generally be deployed on a Monday and they will sit beside your existing phones. The existing phone continues to work through that week as will the VoIP phone; that provides a week of parallel operation whereby an individual can continue to use their own phone should they need to, while having the opportunity to try out and learn the new VoIP phone. Also during that week, if you wish, the CSRs will provide hands-on training to demonstrate the features of the VoIP phones. Assuming everything has gone well, on the Friday of that week a cut-over will occur where your old phone number goes dead and you continue to use the VoIP phone from that point forward.
Chris Leopold asked if there would be the opportunity to renumber the VoIP phone to your old number. Todd responded that if you currently have a phone number then that can be forwarded to your VoIP phone. There will be a temporary number on your VoIP phone while it is first deployed, and on the Friday cut-over that number would change to that of your old phone.
Dan Cromer interjected that he and Tim had stressed to Dr. Joyce that this project was proceeding on a building-by-building basis. In some cases that means that multiple departments must coordinate the decision to opt in or out (Fifield for example); one cannot participate without the others and that will have to be negotiated at the unit head level. Dan also stressed that, for the time-being, this is only for on-campus locations. At some point this will include RECs, but that is a very long-term matter at this time.
Todd continued on, saying that the 40 phone/week schedule would continue until the entire roll-out was complete. Following the final deployment of the phones they have a closure meeting to ascertain that all the local needs have been addressed, and then they close out the work orders.
Wall-Plate requirements
Todd then referred to another handout detailing the requirements for participation. It is very important that we are in compliance with UF Acceptable Use Policy. Hubs and switches or any other devices which extend the network beyond the wall plate must be removed. Todd said that they understand there is a lot of policy involved in that and he would return to the topic later. Units are responsible for the installation of all the data cables and Todd said this would be discussed later as well. If you have servers plugged into telecommunications rooms, then CNS will need to work with you to get those out. Exceptions may be made for the newer larger 10' x 12' telecom rooms, but there is generally too much of a heat load in the smaller rooms to support servers there as well.
As the Wall-Plate is rolled out CNS will address every connection you have currently and will figure in a growth factor of 10%. Once you reach the point where you need a new switch there will be a one-time charge to pay for the switch. Todd said that with the Wall-Plate, you basically buy the first switch (what you have in place today) and CNS will take care of you from then on; the same thing goes for future growth. Tim Fitzpatrick interjected that [ignoring what units already have in place] CNS buys the first bunch of switches. Todd said that exceptions are possible and if you have difficulties working with some of these requirements he asked that we present those to them; CNS would review those and see what alternatives might be negotiated. This latter point is documented in their Service Level Objectives document. The exception request procedure is detailed in there.
Wall-Plate port pricing
Todd then moved on to another handout detailing Wall-Plate port pricing and specifying where the additional costs come in. 10/100 Mb ports are provided at no charge. Class 2 Power over Ethernet (PoE) switch upgrades for supporting VoIP are also provided at no extra cost. Class 2 PoE provides 7 watts of power; if you need to meet Class 3 requirements (15W) then there may be some additional charges. Chris Leopold asked about the use of power injectors. Todd responded that if you are doing things like tilt/swivel cameras, then you will probably have to use a power injector; if you provide those there would be no charge. Upgrading to Gigabit Ethernet involves a 1-time up-front charge of $600 per 24 ports; this can be done on a switch-by-switch basis. Mark Ross commented that if a single user required Gigabit then one entire switch would need to be upgraded.
If you require Gigabit plus PoE, that raises the upgrade costs to $1200 per 24 ports. Todd recommended against that because you are also going to need a Gigabit telephone if you are running those and that is very expensive ($400+ per phone). There are some Class 3 telephones (such as the 7931G); again that would involve some additional charges which would have to be looked at on a case-by-case basis. The handout also details the per switch cost of expansion beyond what you have currently; those range from $2400 for a 24 port 10/100 switch to $3600 for a 10/100/1000 PoE Class 2 switch. Those are one-time charges and only are applied when you fill up all the ports in your closet and need to expand. Tim added that these are pass-through prices; CNS is not making a profit on those items.
Dennis Brown remarked that he felt everyone would like to move to Gigabit data if they could afford it. Since the Gigabit phones are so expensive, Dennis asked if CNS would consider sequestering phones and data onto separate switches. Todd responded that this has been done, but it expands the number of data ports beyond your current usage by not sharing a single phone and data line. That would lead to additional costs via the expansion port pricing algorithm. With low phone densities (relative to data ports) it would likely be less expensive to share Gigabit ports via the extra cost of Gigabit phones.
Dennis then mentioned that they have some small buildings with very few users. He wondered what sorts of switches would be deployed in such cases. Todd replied that there was an 8-port version of the switch (roughly $1000) which could be deployed in such instances; if you have more than 6 drops at a particular site they would likely upgrade that to the 24-port switch however to allow for future expansion.
Tech bench switches
Steve Lasley asked about the tech "bench switches". Todd replied that those were for technical staff and would go into your secure non-public work area. These are for technicians who have a lot of machines coming and going as a flow but which do not remain on a permanent basis. This is something which you can set on your desk/bench and plug into as needed. Since they are not secured in the closet, CNS does not monitor them on a 24/7 basis to make sure they are still working. Since it is not a permanent connection, they will trust us to call and notify them of any problems. These will not be on a UPS. Those can be set-up as an ordinary switch or they be configured specially; for example, they can have additional ACLs or filters applied to emulate a firewall. The switches are typically 8-port unless a greater need exists. Mark Ross asked if multiple VLANs could be enabled on those, for example to allow WIPA on some ports. Todd responded that currently they supply a single customer VLAN but that could be requested as an exception.
Unit preparations for the Wall-Plate
Steve raised the question of what we could do as IT support folks to prepare for this. He asked if there was something like an Excel template which we could enter information into. John responded that they do have a spreadsheet for the VoIP deployment to record the user information including the type of phone desired and the phone number you wish it to have. Winnie Lante had pointed out before the meeting started that her unit, which recently completed Wall-Plate deployment, had difficulty at closing in reconciling what was done with what was specified. Having things clearly worked out ahead of time would seem like a good way of avoiding that.
John gave the example of Dennis Brown who has been working with Judy Hulton for a quote on remedial wiring; he mentioned that those are things which we could be taking a look at now. On that topic, Dennis replied later that the quote Judy gave them was considerably higher than what they had been paying previously from Complete Network Solutions ($150/port). This led Dennis to believe that direct negotiation with contactors for wiring remediation might be preferable. John responded that he would rather units buy straight from the contracter. CNS has Judy for consultation and CNS will actually bid it out when it comes time to do the job, but they would prefer that units handle this for themselves if possible.
Steve pointed out that a great deal of work at the unit level must go into the first little arrow on the workflow chart before a unit can make an informed decision to opt in or out.
Tim added that figuring out the phone counts and locations has been problematic; if CNS can get units to start working on that in advance it would be very helpful to everyone. Also, Tim pointed out that the memorandum of understanding (upper-left) was misplaced on this chart. It deals with both the Wall-Plate and the Telecom aspects and should be moved down in the workflow a few steps as well to the point where the phone and network info come together and we can really tell a decision maker this is the deal and this is what we are all committed to on both sides. Steve added that this could be examined again at the end to ensure that this is what occurred as well.
John mentioned that the other thing they do up front at the admin level is to provide a cost analysis and investment summary which includes any costs or savings which might accrue from moving to VoIP.
Custom phone considerations
Mark confirmed from Steve that Entomology has an aging PBX system. Plant Pathology has a rather new system, however, which involves CAT5 wiring back to the phone (not data) closet. Mark wondered if CNS would reuse that. Todd responded that they would have to look at that as every building is different, but CAT5 cabling meets their specs for that. CNS might be able to run a tie-cable between the two closets as one option or use the existing phone closet as an auxiliary data closet, but it depends on the specifics.
BPOP to Core fiber connections
Steve inquired whether Gigabit fiber connection between the BPOP and the core was assured. Todd responded that they believe they are able to deliver Gigabit to all of the IFAS buildings--if not now, within the next 24 months. Steve indicated that he understood that Entomology currently has the wrong kind of fiber run for that.
As a side-note on the fiber question which Steve raised, John added that they have a matrix which they consult whereby if you have "X" number of phones, your building will get a dual fiber connection to the core. Entomology is way over that number so CNS will provide a dual route to the building.
Wireless access points
Steve asked again about wireless as well, asking if the basic service included just one access point per closet. Todd responded that there will be access points deployed sufficient to provide basic coverage throughout the building. They are beginning to roll out the new tri-transmitter light-weight Cisco APs. These units can handle 802.11abgn signals; they expect to be able to support 802.11n, though there are some details yet to work out on that. Dennis asked if each AP had a single connection back to the switch. Todd replied that this was indeed the case and that a single AP can typically support 12-24; that will vary by usage however. If you have a high-density classroom where you will potentially have a large number of users on all at the same time, you may need to deploy an additional AP at the cost of roughly $1000. Mark and Chris raised the issue of RF dead spots, but Todd repeated that they would ensure basic coverage over the entire building.
On to VoIP
John said that the WP was the meat of the project and VoIP is really just a service riding on that infrastructure. Todd countered that getting the telephone right is very important as well and John agreed. Tim stated one final time so there would be no mistake: fiber to the building is the problem of CNS and it at their cost; wiring within the building is the responsibility of the unit.
History of VoIP on campus
[slides six-eight] John then began the telecom portion of his presentation. Initial deployment was back in 2003 with new buildings on campus--the first being Rinker Hall. At that time we had two gateways with PRI connections back to what was BellSouth at that time--now AT&T. Since then the system has grown and we now have a number of Call Managers running on servers. Unity is the voicemail component and that has grown into a redundant system. Today the PRIs ride on a SmartRing back to the central office downtown. We have backups, and backups to those backups.
[slide nine] CNS started with new buildings and now any new building uses VoIP.
[slide ten] While there was still a $5 port charge, CNS had an initiative to convert Centrex users (high cost per line) to VoIP. Microbiology was one of the first buildings to migrate over to VoIP under that plan. Numerous others have since migrated similarly. Todd mentioned that when Microbiology migrated they were a little nervous about using the data port on the phone; consequently their existing data network is still in place. CNS wants to go back through and migrate them over to the phone ports.
Current VoIP roll-outs
[slide eleven] John then showed a slide of buildings being done this current year; approximately the top half of those shown have been completed.
VoIP phone options
[slide twelve] The 7940G is the phone model recommended for most people and its subsidized cost is $140. There is a data port on each phone which supports a local data connection. The wire currently going to your computer would thus go from the wall-plate to the phone and then on to the computer. CNS wants to converge the telecom and data networks onto one network. Mark asked if these were SIP phones and John responded that they were Cisco proprietary currently.
[slides thirteen-fourteen] Tim asked how many VoIP phones we had currently and Todd responded that the current number was 2644. Most of these are the 7904G model, but some are the more expensive 7960G models. Ben Beach asked for clarification on Gigabit data connections with phones and Todd replied that either of the two models shown would throttle that back to 100Mb at its data port. One would need a special phone to pass the higher speed.
Conference phones and FAXes
[slides fourteen-fifteen] John then showed a slide of the basic conference phone. This phone runs on IP and is actually built by Polycom for Cisco. Mark asked if there was an Analog Telephone Adapter (ATA) for those who had already spent thousands of dollars on analog conference phones. John replied that there was and that it cost about $150. You can use that on your FAX machines as well.
Wireless VoIP phones
[slide sixteen] John showed a slide of a wireless VoIP phone which is currently in use for the Library and the Law School. The engineers want to hold off deploying any more, however, until they get the design such that you can roam across campus. This uses the wireless access points and seems to work well at a cost of about $450. According to John, it's usually the IT guys who use these because they are always roaming around. It has all the features of your regular phone.
VoIP phone configuration
[slide seventeen] There is a browser interface for configuring your phone. This is where you can set up things like speed dials. You can use this interface remotely to forward your office calls to your cell phone and the potential exists for developing other XML applications for this interface. There is also a hook into Outlook now so you can use that to look up phone numbers.
VoIP Attendant Console
[slide eighteen] Steve mentioned that the receptionists in his unit have busy lamp fields and asked what would replace that function within the VoIP system. This can be done by an application run on the local PC in conjunction with a 7950G phone called the VoIP Attendant Console. John mentioned that they have a couple of these at University Financial Services. They have heavy call volumes there and this allows them to see who is on their phone, based on an icon which is displayed. John said that he believed this application costs just a one-time setup charge of $25.
The IP Call Center application
[slide ninteen] When you call 392-HELP you are dialing into their IP Call Center. That is the system they use to track the number of callers coming in, how many people are in queue, how long they have been waiting, etc. This system allows the Help Desk to be a campus-wide solution which brings in Bridges, University Financial Services and others. Again, they are trying to get more of a campus-wide solution instead of each department going on their own.
SoftPhone
[slide twenty] Another reason we are going VoIP is to enable applications such as the SoftPhone. Todd mentioned that he had used this and really enjoyed the convenience of taking his local phone number with him even when traveling outside the state; plus there is no long distance charge between the laptop and campus. It doesn't give you long distance to the world however. This is a program (~$100) which installs on your laptop and works in conjunction with a USB handset (~$60). In a hotel room with a high-speed Internet connection it works just as well as the local phone would.
John mentioned we could get toll-free long distance calling from these if we could get everyone on campus to agree to a $24.99 charge. Tim mentioned that they will be looking for options like that, but they were "kinda busy" right now :-). John said that, while we currently have the SmartRing with PRIs, that will eventually become sub-trunking and when you do that you are just paying for a bundled service and you would have the free unlimited toll free long distance calling. John thinks this is a couple of years away. Tim said that the reality is that a dollar savings here and a nickel spent over there are not perceived as balancing each other out. If you try to address issues where some may win and some may lose by taking a certain course--central management has never been very good at attempting to equalize that.
Dan Cromer asked if there was a timeline concerning when the SoftPhone would be available and whether people have to be on WP to use that. Right now you need to have a VoIP office phone on your desk to make use of this because the SoftPhone has to be associated with a number on your campus VoIP phone.
Long distance phone billing
Dennis Brown asked how departments would be billed for long distance. CNS provides number-based charge detail for each phone just as they do currently (though key systems and PBXs must deal with that in-house due to the sharing of lines).
Video Telephony
[slide twenty-one] Video Telephony is currently quite expensive but will likely one day be the norm. This application is available, but is roughly $1500 per seat.
Presentation Summary
[slide twenty-two] This slide lists reasons CNS is attempting to converge our data and telephone networks to maximize UF's investment, lower operating costs and provide greater possibility for new services down-the-road.
[slide twenty-three] That finished the presentation on VoIP and John then quickly addressed the scheduling of IFAS buildings by showing a slide of those currently done or in progress as well as the expected "first approach" dates for each of the rest.
Tim wanted to clarify that the dates listed on this slide are the dates of first contact. It has been their experience that a building with several hundred ports and a hundred or more phones would require roughly six months to complete.
Miscellaneous questions
Chris Leopold asked if there had been any discussions about merging the Call Manager at Lake Alfred with any Call Managers here; this could reduce the long distance call costs between UF and CREC. John said that HealthNet has a Call Manager and CNS will likely be talking to them first.
Dennis asked, since there is a five-year refresh cycle planned, would it be a reasonable expectation that when the next cycle came that Gigabit would be the new standard and would be used for replacements via central funding. Todd thought that was very likely, but reminded Dennis that they are on a fixed yearly budget and port estimates have inflated greatly from original estimates. Mark pointed out that the phone would be the unit responsibility, however, and if they were not replaced as well, they would still throttle a Gigabyte connection down to 100 on their data port. Of course, in 5 years we might have SoftPhones there instead.
Mark asked about continuing costs for VoIP phones. Each phone costs a line charge $12.50 per month compared to basic Centrex line charges of $19.50. There is also a key system surcharge of an additional $7 so users of those are paying their fair share. Steve noted that the estimated costs of joining the Wall-Plate for Entomology are very roughly $15,000 for wiring remediation, $15,000 for the phone sets, and $15,000 a year for line charges (this on somewhat over 200 phones). John responded that owners of key systems had addressed their phone costs up front; if you bought a new key system in the last 5 years CNS is not going to tell you to rip it out. This is the direction in which they want campus to move, however. McCarty D was like a microcosm of campus with a mixture of various key systems and Centrex lines. Across the entire building the move to VoIP is going to save money. Some units there will save money and some units may have to pay a little bit more.
Thank you's
Steve thanked John, Todd and Tim for coming and let them know that the information which they supplied here would be made available to all, including those unable to attend today.
Recommendations for Barracuda management
Background
The ICC has had previous discussions on the inadvisability of blocking potential spam. In a March 6th meeting with Mike Conlon, Chris Leopold and Andrew Cary were told that the Barracuda default spam score settings would not be changed. Dr. Conlon is happy with the settings as they are and said he would not be persuaded to consider blocking as a problem unless IFAS could confirm that at least 20% of blocked messages were considered false positives via direct verification with the intended recipients. IT/SA staff believes, and Steve agrees, that this is unreasonable. Rather, any form of blocking is unnecessary and potentially detrimental. Consequently, Steve would like to propose a recommendation from the ICC for changes that he feels would greatly alleviate, though not completely remove, the potential for e-mail not being delivered.
Steve briefly reviewed his draft proposal and then solicited comments. He pointed out that IFAS used to deliver all e-mails it received (short of infected messages) to either the Inbox or the Junk E-mail folder. While users might assume that the quarantine now works similarly, most are unaware that we now have blocking in place. Steve said he would appreciate any input on making this recommendation better. He also said that he did not know exactly where we would take our recommendation. It might go to ITPAC or it might be more efficient to consult directly with Dr. Mark Rieger who is the IFAS representative on the UF Exchange Advisory Committee.
Note: the membership list posted appears out-of-date; a better source of members might be found by searching the GAL for "UF Exchange Advisory Committee". Excepting Brenda Stevens, who is the owner of the list and is Mike Conlon's Executive Secretary, the membership of that mailing list contains the following individuals:
Granularity of default settings control
Dennis asked if the default scores could be set individually for IFAS. Steve responded that we had looked into that. Originally, Dwight Jesseman believed they could be, but it turned out that there are technical limitations which require that IFAS accept the defaults for the "@ufl.edu" domain--consequently, if those are to change they would have to change for an audience broader than IFAS. It seems reasonable that the defaults reflect the wishes of the average user and that those wanting things handled differently can modify their settings individually. We will need good documentation on that, however, so that users know how and why to make changes and how to do it safely. IFAS currently does represent about 75% of the overall usage, however.
Chris Leopold responded that it was his understanding that, by providing a list of e-mail addresses, other departments have had custom settings set via script. If it turns out that our users generally do not like blocking, Chris believes we might consider that route should we fail in convincing Dr. Conlon that the defaults should be changed. The problem then, however, is that managing new accounts in the future would require attending to yet another item on a check-list of setup procedures. Joe Gasper responded that it would be cheaper to just buy your own Barracuda--obviously a drastic and unpalatable solution.
Example instance of faculty blocking issues
Chris related that an IFAS researcher had contacted the IFAS Help Desk and others indicating his concern that a number of e-mails were not being delivered. This researcher even went so far as to say "It is now at a level where we are losing funding and publishing opportunities because of this." Since such a serious allegation demands a careful inspection of the available data, this led Chris to request and analyze the Barracuda logs for this individual (sanitized). You may look at the data yourself, but here is Chris's factual analysis:
There were 119 blocked e-mails during the 49 day period and depending on how you “subjectively” rate spam we are looking at false-positive rates of 7.5% - 21.8%. Because rating spam is subjective, I have expressed the false-positive rating as a range rather a fixed number. In this case, the range was derived from two premises. Premise A - you believe companies like Circuit City, Delta, Amazon and Days Inn send un-solicited e-mails. Premise B - you believe companies like Circuit City, Delta, Amazon and Days Inn sends only solicited e-mails. That translates into 7.5% false-positive rate for the former and 21.8% false-positive rate for the later. If this user chooses to adjust the blocked rating, this user would have to deal with an average of 2.42 extra e-mails a day delivered to his Quarantine mailbox. Being in the Quarantine mailbox, this user would have the opportunity to white list the sending domain’s e-mail address - eventually leading to the reduction of mail that would be Quarantined. |
Those messages which comprise the 7.5% group mentioned include voice messages (mpegs) from the researcher's home phone system and two Requests for Proposals (RFPs).
Chris Leopold mentioned that this is a sample size of one and by itself cannot be extrapolated across all our users. Steve responded, however, that so far every ICC person he knows who has investigated this matter has been able to determine that e-mail they wished to receive has been blocked; those same people believe that a slightly longer quarantine list is a valid trade-off for removing blocking. The real question is what would best serve the majority of our users, however, not just us as IT support people. The problem there is that we are at least aware of the issue and able to investigate; many of our users are not.
Problems with white listing
Wayne Hyde pointed out that, while the quarantine provides a fairly simply method of white listing, it does so by the complete address. White listing by domain is often more effective (many newsletter senders use basically random addresses which change with each mailing), but would take extra work and understanding on the part of our users. Blocking, however, makes white listing extremely unmanageable; as how can you white list addresses or domains if you aren't even aware of the need? In going through the logs which UF Exchange has provided, Wayne has discovered numerous reservation notifications which were blocked; those come from basically random addresses and white listing those on a prophylactic basis is next to impossible. White listing at the domain level has some additional drawbacks; domains are easy to spoof and newsletters which you may wish to receive often come from domains which also generate spam.
But CNS was already blocking, right?
Chris said that user education will be necessary and that ICCers should make their users aware: e-mail is being blocked. While Dr. Conlon counters with the fact that IFAS has been blocked by CNS for over a year, the CNS Spamhaus blocking he refers to is not analogous as it is RFC compliant; the sender in that case receives a bounce and can retry contact via other means.
How do we proceed?
Mark Ross was concerned with what this might do to the load on the Help Desk and UF Exchange staff should this became common knowledge. To consider that one must understand that when a user reports missing e-mail, the reasons usually include:
Consequently, there are steps which local admins need to check when a user reports missing e-mail:
The final item in the above list is the last resort because it requires the time of and assistance from the UF Exchange staff. A goal would be to minimize the need for Barracuda log requests. If a user is operating under the default settings and they raise concerns, and assuming the usual checks don't pan out, transaction logs are the only option for continued investigation. The burdensome nature of these requests is felt across all involved. Steve believes that alone is a strong argument for reconsidering the defaults, as the negative consequences of making quarantine our primary default method would appear to be negligible for the great majority. Blocks are not distributed evenly, however, and Wayne has done estimates that indicate less then 2% of the mailboxes account for more than 28% of the blocks; there is some small percentage of users who would see a very significant increase in their quarantine lists should the defaults be changed to what we have been suggesting.
The real question is would informed users be willing to review a longer quarantine list as a trade-off for not having messages blocked? That is difficult to judge without providing widespread notification of what we are beginning to believe is a real problem. Steve is hoping this can be addressed without the negative consequences such notification might evoke; the concern there is that people not begin wholesale adjustment of the default settings because the consequences are quite complex for an end-user to decipher. It would be much preferable if the defaults met the needs of the majority.
User feedback
Both Dennis Brown and Winnie Lante mentioned having had the opportunity to query their users and in both cases their users all agree that they want the opportunity to see ALL the mail. They want to at least know that something has not been stripped that they will not be able to find. Winnie had one user (an Assistant Department Chair) that absolutely insisted that the Barracuda be turned off completely. He was furious because there was a message which was blocked that he had been sitting there all afternoon waiting for.
Would our users have to wade through long quarantine lists if the default block score was moved to 9?
Regarding the burden which a change in defaults might place on our users, Mike Conlon has said that when he "asks people if they would rather review 100 e-mails or 5, most say 5." Mike is an atypical e-mail user, however, and at current rates it would be impossible for the majority of users to see anything like 100 per day.
In actuality, from February 11th to March 11th the UF Exchange blocked a total of 503,000 messages. If you divide that by the number of days and the number of users, that amounts to 3.5 extra e-mails/day which would go into quarantine (on average per user). Joe Gasper has provided further interesting numbers suggesting that quarantine sizes wouldn't bloat inordinately were we to move from blocking to quarantine. Note below that Business Affair's per user quarantine sizes over the last 5 months are only 4.11. Note also that they appear to actually block a larger percent with their much less stringent block settings than we do with ours.
The default settings on Business Affairs's Barracuda (with an estimated less than 100 having changed from those defaults) are:
| Tag: | 10 |
| Quarantine: | 3 |
| Block: | 9 |
A summary of Joe Gasper's data follows (updated on April 9th to include data through April 7th):
| Source | non-spam | blocked | quarantined | tagged | viruses | bad recipients | total |
| UF (Feb11-Apr7) | 54.2% | 34.5% | 4.8% | 1.29% | 0.010% | 5.1% | 2591124 |
| BA (5 month aver.) | 39.6% | 40.1% | 17.1% | 0.04% | 0.014% | 2.6% | 6365045 |
| Daily Aver. Per User | non-spam | blocked | quarantined | tagged | viruses |
| UF (Feb11-Apr7) | 5.20 | 3.30 | 0.46 | 0.12 | 0.001 |
| BA (5 month aver.) | 9.51 | 9.61 | 4.11 | 0.010 | 0.003 |
What If UF Exchange disabled "Tagging" and "Block by Score"?
| UF | non-spam | blocked | quarantined | tagged | viruses |
| Daily Aver. Per User | 5.32 | 2.84 | 0.53 | 0.00 | 0.002 |
Note that Joe Gasper doesn't believe turning off blocking (putting it all the way to 10) is really necessary; a value of 9 might give an actual < 1% false positive level which most people might find acceptable.
Note from future: An additional interesting fact was provided to Joe Gasper by Mike Kanofsky on March 21st. Only 14% of blocks are due to spam scoring. This further suggests that moving the default block score to 9 would have minimal effect on quarantine levels for the great majority.
How do our settings compare to those at other educational institutions?
Mike Conlon insists that the current default values work fine and that they were carefully and specifically chosen to work in the users' best interest--users who don't want to be bothered with monitoring quarantines and who expect the system to make the best choices for them. If the current defaults were clearly superior, one might expect other organizations would follow suit or have arrived at similar settings on their own. What sorts of blocking scores are used elsewhere?
The average blocking used by all the above is 7.8 compared to the much more aggressive 3.5 which UF is currently using. Joe Gasper has supplied an even longer list of settings used at various institutions:
Summary of that longer list of Barracuda settings at other EDUs:
|
One may note that one-third of the institutions surveyed do not block at all. The inverse is that two-thirds do block. The average block setting of those, however, is much less aggressive than the 3.5 value which UF Exchange is using as the default and the lowest blocking score among those was set to a value of 6--again, much less aggressive than the settings we currently have as our defaults.
Other questions
Via e-mail, Mitch Thompson asked if the Barracuda should be smart enough to white list DOMAINS a user sends to. The consensus was that this was not a feature of the Barracuda. Chris said that the Barracuda does support an "auto-white listing" feature whereby if you send an e-mail to a person, then the domain of that address could potentially be added to a global white list. Wayne speculated that the Outlook plug-in might do that, but he didn't know.
Mitch had an additional question about setting global IFAS / Ag friendly white list terms, such as “publication” “entomology” “budget”. How often does spam contain certain IFAS friendly terms? E-mails with these terms should have a better rating. Joe Gasper responded that, while there is a global list for which you might do some editing, this would probably not be recommended.
What about having to look in multiple places for false-positives?
Joe Gasper suggested that, since people are complaining about having multiple places to review for false positives that we could turned off the Junk E-mail folder completely -- or alternatively set a rule which would move their Barracuda spam reports there.
Can users make these changes reliably themselves?
Dan suggested that we could instruct users to change their settings from the defaults. Steve pointed out that, while we may be forced to that, it is far from ideal. So far he has pointed only one user to his Barracuda configuration recommendations. When he checked how that user followed those instructions, he found that they had noticed the in-built "Recommended" settings which Barracuda advertizes and used those setting instead!
Steve wishes we could take a binary editor to that page and change those values to list the UFAD system defaults.
Recommended course of action
Dan Cromer suggested that Steve and Joe Gasper be invited to accompany Mark Rieger and him to the next UF Exchange Advisory committee meeting (held monthly) so we could discuss our concerns and available options. Steve agreed that we should get with Dr. Rieger and discuss the matter with that intent.
Mike Conlon plans to attend our next meeting
Steve wanted everyone to know that Mike Conlon intends to present on this issue at the April ICC meeting. By that time he expects to have a white paper finished on the tools and the rationale for their configuration. Despite our input he insists that only a small handful of IFAS system admins are unhappy with the default settings. He has also said: "Our own analysis looks like the systems are operating as designed."
The outsourcing of IFAS IT to CNS
Steve told Dan Cromer that he realized Dan wasn't comfortable with discussing this matter but that he simply had to ask about the proposed outsourcing of IFAS IT to CNS. Dan responded that the process is in place and a proposal has been received from CNS. On the face of it the proposal would appear to save funds so it will be up to Dr. Joyce and Dr. Cheek to make the decision.
Will a decision be made without technical review of the proposal?
Steve asked if that decision would be made just on the input from CNS and from Dan. Dan responded that CNS took the input from Chris Leopold. Then Steve asked if Dr. Joyce and Dr. Cheek planned to trust the CNS version without running that by Chris? Chris mentioned that he asked for a copy but is yet to receive that. Dan responded that Chris would be reviewing it for validation that the service CNS proposes to provide match what Chris had indicated IFAS is currently enjoying.
Steve then expressed some deep concerns. As an example, he asked Dan to consider all the effort it was taking under a centralized model to investigate the options we might have to address the e-mail blocking issue. Dan countered that we would retain the same support people, but that they would just work for CNS. Steve responded that when services are moved out of house, things change; we lose control of things and Steve is absolutely convinced it will not be business as usual.
Dan said that he can't speak for Dr. Joyce, but that he was quite confident that Dr. Joyce would provide the proposal to Chris to verify that the figures listed were correct. Dan suggested that, perhaps, we could bring the same document back to the ICC for review and comment; then the ICC could make a recommendation. Dan would support that process, but he also knows that the decision of whether or not to do that lies ultimately with Dr. Cheek.
Does Dan Cromer support the proposal?
Steve asked if Dan had seen the proposal and Dan replied that he had. Then Steve asked Dan if he thought that accepting the proposal would be in the best interest of IFAS and Dan responded that he did. Dan related that they are talking about transferring four positions (Chris Leopold, Wayne Hyde, Mark Ross and Andrew Carey) as well as the unfilled position formerly held by Jenny Brewer. Dan believes that we would continue to work with the same four people which we do currently and would see no loss of service or accessibility whether these folks were paid for by IFAS or CNS. Steve was flabbergasted and unable to fathom that Dan really believed that. Steve then asked if that has been the case with Dwight. Steve didn't want to speak for Dwight and by no means is anything less than happy with the effort which Dwight continually provides, but happens to know that Dwight believes as we do that the customer comes first; we can see for ourselves how much affect that support has had for our cause. Not having seen it, Steve has grave concerns over the wisdom of accepting this proposal.
ICC input?
When Dan said that the ICC could make their own recommendations Steve asked him about when we would ever get the chance. When Steve raised the issue at ITPAC he got the distinct impression that nobody wanted him to talk about it there. Dan responded that the topic is sensitive politically because higher administration may want to move away from having service units directly in IFAS; they may want to outsource. It is outsourcing to us, but it "cross-sourcing" (or whatever you want to call it) to UF. Steve said that he didn't think it was wise to consider such a course with asking the clients for input. Steve said that he was a client and his users are clients; shouldn't their input be solicited and listened to? The only reason not to would be that the decision has already been made and no amount of input could or would change that.
Mari Jayne Frederick responded that the problem was that Steve is trying to think logically; in instances like this logic is not supported. MJ said that even though Steve may not have been listened to at ITPAC, maybe if we work together through our directors and chairs, explaining what our issues and concerns are, then perhaps those higher authorities could reinforce what we are trying to say.
Steve shared his concern that we won't get the opportunity for input (similarly to how the Exchange transfer occurred). If Steve hadn't raised this at ITPAC (which he felt a little bit sheepish about doing) and written about it, then there would be absolutely no indication that any of this was taking place at all. Steve gets the distinct impression that administration wants to limit discussion on this.
Long-term concerns
Mark mentioned that this needs to be thought out long-range, because it is his opinion that once we get a few years down the road our costs will increase tremendously and we will have no options but to pay the price or forego the services. CNS charges for services granularly by resources utilized; this is a much different model than IFAS IT has been operating under up to this time; that change could have enormous consequences for our departments--especially under these tight budget times. Chris Leopold responded that, from private discussions with CNS, the services that his group supports currently will not be the same services they will support should this proposal be accepted. Instead of clustered Win2k8 file services, those will be supplied by a NAS appliance. The services will be scaled by the need of CNS to handle not just IFAS but others as well. We will be just another customer instead of looking after our own. Wendy Williams asked if there was an associated Service Level Agreement and Dan responded that there was.
Steve asked Dan directly if we would get the chance to look at this proposal. Dan responded that he hoped so, but that he was only the Acting IT Director. Steve then asked Dan if he could press administration with the notion that it would be wise for us to have it for review. Dan said that he would certainly raise our concerns. He pointed out that the IMM which speaks of the responsibilities of the ICC makes it clear that such a review would be well within our purview. Dan suggested that Steve write an e-mail to Dr. Joyce on behalf of the ICC saying that we are concerned about this and asking if it would be possible for us to have any input into the matter before a decision is reached. The problem Steve sees with that is we don't officially even know about this; Steve gets the feeling that he isn't supposed to know. Dan said he couldn't comment on that because he did not know if that was the intent or not. Steve responded that he had a pretty strong feeling that it was because he had asked Dan about this matter at the December ICC meeting and Dan mentioned nothing at all about it, even though Dan knew this was in progress.
Fears of service degradation
Ben Beach reminded Dan again of how things are currently transpiring with Exchange and blocking. Dan responded that Ben wasn't being moved and departmental support staff weren't being moved--implying that this shouldn't concern us. Wendy asked if it included web servers and Dan said that it did. Dan asked why we would care when we went to an IP address or web page whether the server was at CNS or in the IFAS machine room. Wendy responded that she might care greatly whenever a problem arose. Currently, Wendy can call Mark and work out the problem. Dan insisted that Wendy would still call Mark in the same fashion. Steve said it was more likely that we would have to go through a ticket system and that our issue could and would sit in a queue somewhere. The Remedy system still isn't meeting the needs of IFAS (for either the users or the support folks) after considerable trying; the potential for that becoming mandatory at the higher level is thus concerning.
Ben asked what was in place to prevent CNS from taking the excellent equipment we have recently assembled and using that for their own purposes while redeploying our services on a less expensive and less robust platform. Dan said that this should be addressed in the SLA [who is checking on that?], but that we shouldn't care what hardware was being used behind the scenes as long as we were given the level of service we require. Dan related that Dr. Joyce has said it has to be the same service at lesser cost, to which Steve responded that our switch to UF Exchange saved no money (in fact it effectively cost us some portion of an .FTE because Dwight used to do many other things for us besides just e-mail) and we are seeing with the blocking issue that accommodations can be much more difficult to obtain after a service is outsourced and shared by a wider audience. Dan argued that this was just a single issue; Steve agreed, but believes the example to be quite valid. Dan said that this is a policy matter and Steve responded that it was a policy matter that we wouldn't have to deal with if we still had the service in-house. Dan responded that this then was something which we could bring to Dr. Joyce's attention.
Steve related that units have many such difficulties dealing with issues even at the IFAS level; that is why so many units fight tooth-and-nail to ensure that services are kept at the unit level so that unit needs can be met. We've worked to where we have a good relationship with IFAS IT (in many cases) and now Dan seems anxious to give that away. How is it going to affect us? Steve said he knows it will affect us. Is our service level going to stay the same? Steve said he knows it will go down in many aspects, as he has seen it happen again and again over the years. Dan corrected Steve saying that he didn't "know" that, but rather that Steve just "thinks" it will. Steve said he wasn't looking forward to telling Dan "I told you so" after-the-fact.
Can this proposal make good economic sense?
Joe Gasper asked if this move was expected to save something like $200,000; if not, why would we want to go through the monumental effort this will entail? Dan responded that he believed the savings promised were more on the order of $50,000/year which is still not "chicken feed". Dan said that the idea is when you had an issue you would call the same person you do now. Joe Gasper finished Dan's sentence and added that they would be busy with a non-IFAS customer. Dan said that these people would be dedicated to IFAS. Joe Gasper asked how CNS can possibly save any money when they have to add another person to provide the same level of service. Dan replied that they are looking for economies of scale, particularly with the hardware; if you can have two or three departments on the same blade center then you can save money on the resource. Steve mentioned that we have already achieved economies of scale on our own hardware and Joe Gasper added that he has taken advantage of that. Joe Gasper hopes that somebody can pay him to get out of that arrangement and asked who was going to give him back the $50,000 he had invested in the IFAS hardware. Dan said that he believed these were valid concerns and that Steve should write an e-mail to Dr. Joyce and request that the ICC have input in any decision.
Why should IFAS be so generous in funding centralized start-ups during such budget-tight times?
Steve told Dan that it seemed to him that the one person who might be fighting hardest against this was Dan himself--but he seemed to support it. In answer to that, Dan appeared to change course 180 degree from his prior statements and said that he really didn't. When we pointed out that he had just said he thought this move was better for IFAS, Dan replied that he did--he believed this would save money for IFAS. Steve pointed out, and Wendy concurred, that saving money isn't necessarily the best thing.
Wendy said that administration is not going to listen to anyone else but Dan on this. Dan then admitted that he has already told Dr. Joyce that he doesn't want this to happen. Dan said that he had already written his opinion to Dr. Joyce that anytime we give anyway direct control we give away certain degrees of freedom--as in the Barracuda issues. Dan said that he still strongly believes it is better for UF overall to have a central Exchange service even if it isn't going to save IFAS much. To that Steve responded that IFAS has helped UF get AD going and we have helped them get Exchange going; do we have to jump start their web and file services as well--even if it provides IFAS no real advantage and may even end up costing more for a service which we will have less control over? Joe Gasper added the question of whether we are going to help them get a Windows shop going as well. CNS has had since 2000 to build a Windows shop and has demonstrated little interest.
Personal views of the Acting Director of IT for IFAS
Steve said he had heard several opposing quotes from Dan, but that he preferred to believe the last one where Dan said that he didn't want this to happen. Dan reiterated that he would personally prefer that the proposal not be implemented. Dan went on to say that he believes we have a better culture in IFAS and he could say this even if Tim Fitzpatrick was present. Dan believes we have a better work ethic in IFAS. Dan knows we still have memories of difficult dealings with CNS and some of that may still be going on, but Dan also believes that Tim is trying to change CNS. Steve added that it spoke a good deal that three of their staff, including Tim himself, were here today. Joe Gasper said that he agreed that CNS does well with some things such as the Wall-Plate. Joe loves that service, thinks it is wonderful, and wants it everywhere. That is something which they have done which is very important; but they haven't and general don't do Windows to a significant extent. IFAS is a Windows shop pure and simple.
Dan responded that Tim is trying to move there. Mark reiterated his concern that CNS was going to take over our services and then start charging us for every little thing--services which we would have provided for ourselves at little or no cost.
MJ asked Steve that, if Dan was the only one whom Dr. Joyce would listen to, would it be possible for the ICC to draft a recommendation and go with Dan to see Dr. Joyce about that. Steve responded that he wasn't sure how things would transpire, but that he certainly would draft an e-mail to Dr. Joyce requesting the chance to review and have feedback concerning the proposal prior to any decision being made. It all depends on how receptive Dr. Joyce is to that. MJ added that if Dr. Joyce supports what we are trying to say and that Steve is representing us as our chairman, it might be more effective. Dan replied that he was willing to do that.
Going back to what he had said and what Wendy had said, Dan noted that sometimes you run the risk of losing a number of intangibles in the quest to save money. That is why Dan feels that IFAS is big enough to have its own IT department with server administration. Dan can see that there are economies of scale from Tim's viewpoint. Dan also mentioned that Mark Hoit and Mike Conlon believe there are a lot of funds being wasted on IT across UF campus--though not necessarily in IFAS who have been successfully consolidating their own units since the integration of IT in 1997.
IFAS would be giving up a good thing
Joe Gasper responded that he used to be in IFAS back in the 90's and it was a fight between all the various departments at the time. Joe believes we have done a wonderful job of gaining trust and building a central core service that he would be scared to just up and give away; it will certainly appear as if it has been given away. You are going to do everything you can to say same service same everything, but there is going to be a different phone number to call, you might be talking to a different person--they do have a couple of other Windows people over at CNS. IFAS has a really good central IT service, one of the largest on campus. That's pretty good and there are lots of other units that could do that and consolidate. IFAS has done that already.
Is this proposal the result of pressure at the UF level?
Dan said there is conflict going on. One of Dr. Cheek's goals is to focus efforts on teaching, research and extension. Dr. Cheek's preference would be to outsource IT even at the same costs (assuming we get the same level of service) and not have to worry about it directly. Joe Gasper suggested that we would likely have a greater worry with outsourcing as we can't have any of those programs without an excellent IT service. Ben mentioned that CNS is a "for survival" organization; if some faculty member has a new system they wish to deploy it is going to cost that faculty for each individual item involved. That is quite different than how things are handled currently. Mark mentioned a bigger issue in that IFAS is not just on campus; Chris travels quite a bit. That is part of the intangibles which Mark suspects CNS will be unwilling to take on.
Comprehensive IT risk assessments will be REQUIRED soon
Steve mentioned that he wasn't clear on whether or not he needed to attend Achilles training. Dan replied that he was hoping that Wayne would attend and then direct the various IFAS IT units on how to proceed. Joe Gasper said that if you are running your own file or web server within your own department then he thinks Achilles might be useful; if you are merely using central services he believes you are taken care of.
Other Project-related agenda items were skipped this month because time did not permit.
Videoconferencing topics
Dan had a couple of things he wanted to say regarding videoconferencing. He believes we should all know how to setup an ad hoc videoconference. You should also realize that people can join via phone (see section at bottom of that page). Steve has also updated the ICC meetings page to indicate some of these things and to point people to the Video Conference Event Listing for determining the conference IDs of scheduled VCs.
Other Operation-related agenda items were skipped this month because time did not permit.
The meeting ran a bit long and was adjourned at about 12:15pm.
