IFAS COMPUTER COORDINATORS
Message from Mike Kanofsky to the CCC list:
1/1/11 11:11:11 pm (how often do you see that date/time stamp... ok twice today)
As some of you may have heard, I will be leaving UF after 16 years of employment. I have accepted a position with Microsoft as a Premier Field Engineer.
It wasn't an easy decision to make. I did a lot of soul-searching for several weeks before making my decision (I really looked everywhere and couldn't find my soul anywhere! :-P)
Over the years I've worked with some of the most brilliant and talented individuals, it is these individuals that have taught me and challenged me over the years to help build the core of UFAD and the underlying services that keep it going. (It has also helped that I refuse to be beaten by inanimate objects :))
I can only hope that in my future career with Microsoft that I have the opportunity to work with organizations and individuals that have the same camaraderie that we have at UF.
The most ironic part of this is that now that we moved into the East Campus Office Building with CNS (which is only 1.5 miles from my house) I'll probably be passing the office on my way to the airport!
Again, I want to thank everyone that I've worked with over the years. It's been a blast!
I'll be working out of my home so I'm not moving, I'll just be the 'new guy' for a while and be traveling. Hopefully, I won't have to camp out in airports that are snowed in!
I'll be in touch and will try and help UF through the transition as much as I can. (before I get the MS implants...)
Determining what tasks need transitioning
Iain responded that the immediate issue is determining exactly what things Mike has his hands in--and that is lot of stuff as we all know. James has begun working on a lot of that and Luis Molina has considerable expertise in Active Directory. Erik Schmidt also has a lot of UFAD expertise. OSG has different people working on the various areas and feels they do have the personnel to step into those roles until a replacement is located. They do need, however, to re-engage those across campus who have been working directly with Mike.
Preparations are complete for posting
Near to mid-term, Iain has developed a new updated job description for Mike's position; that hadn't been updated since Mike moved into UFAD about seven years ago. That description is now more accurate and includes such things as OCS; it now covers those aspects in which we will want a new candidate to excel. The vacant position will be posted any day now and will be advertized on Monster, The Chronicle of Higher Education, Educause, and a couple of other places including Windows Hied. Iain wants to make sure he gets the attention of those individuals out there who may be a good fit for us.
Has OSG considered leaning on expertise elsewhere at UF?
Ben Beach asked if Iain had considered branching out to other departments to tap the considerable expertise available right here at UF. He suggested that some cooperative relationships might provide additional support and expertise that would be useful to OSG. Iain said that this was not something that had been considered but that he could see where such a thing might be of use.
How urgent is the need to transition to a 2008 R2 native domain?
Kevin Hill related that prior to Iain's arrival today there had been a discussion about a rush need for finishing Domain Controller deployment prior to Mike leaving. The thought was that we need to retire older DCs immediately so we can move UFAD to a Windows 2008 R2 native domain. The transition to Lync is just one reason that must happen, but Kevin wanted to know if the urgency was real or if that could be delayed somewhat in order to provide him more time.
Iain encouraged Kevin to move forward with the deployment as we would like to get that upgrade done as quickly as possible. Iain would like to get as much of that project wrapped up as possible before Mike leaves; if not, of course they will continue on after. Iain definitely sees no reason to wait, however. Kevin responded that he would be more comfortable waiting until Mike's replacement was in place before making this important transition. It is not that Kevin is trying to delay things, rather he is concerned that Mike's leaving should cause an "emergency" need to rush matters.
What is the timeframe for filling the position and what is the approach?
Kevin asked what timeframe was expected on filling Mike's position and whether we were looking to fill that from inside or from without. Iain was completely open in saying that James Oulman is an internal candidate. James is very interested in moving into a full Windows, Exchange and Active Directory role. He has a vast amount of expertise there and Luis has been encouraging him to apply for the position.
If that happens, Iain's full intent is to rewrite James's position to be Active Directory and Exchange and repost that. Iain mentioned this because he doesn't want people to think OSG would be a person down should James take the position. That will not be the case as we would still seek another hire with this same skill set. Either of those can basically be pursued in parallel as they already have rewrites for both positions and he expects those to be posted within the next few days.
Iain said that while he will proceed as fast as he can, we all understand that the hiring process will take a while. Iain said that he believes Kevin's concern related to that gap; OSG has both Luis and James to help fill that. Keith can also assist, not necessarily with the nitty-gritty DC work, but with a lot of the day-to-day work that comes in. Erik is helping as well and Iain feels we have a lot of good team members to cover this. The projects that Mike is working on may not proceed as quickly because new contexts will have to be learned by various individuals. We can keep working on things; they just may initially take a bit longer than prior.
More staff but also more duties: will this move be synergistic?
Steve commented that Iain has more staff now plus a whole lot more things to manage. Steve asked if Iain felt this was a winning situation overall for OSG and that a synergy might form from this integration. Iain believes there is a great synergy there. He gave the example of their extensive virtual hosting services where Microsoft-focused hosting provides greater revenue than that for other systems. OSG has huge customers in the library and DCE for Windows file sharing services (20TB for the library) and MS SQL is really huge as well. James and Keith and others have been doing that for a while already and are used to working in that area.
Iain said that things are working the other way as well. Luis expressed interest in learning more about Proofpoint so that when something comes in he can get an end-to-end view of it and not have to hand it off to someone else.
Are the OSG file hosting solutions appliance- or Windows 2008-based?
Chris Leopold asked if the file services being hosted for the library were on an appliance box or if it was actually a Windows clustered environment. Iain responded that OSG actually offers both, but the library chose to utilize OSG's EMC storage appliance which is integrated with UFAD. That appliance lacks a minimal feature subset, notably DFSR, but the library did not require that; they were more interested in the redundancy provided. One unit is housed at SSRB and another at CSE. Each consists of three blades which equate to really large servers at each spot so it is triple redundant at the box within each site and if there is a site failure it will pick it up at the other site.
Naturally, OSG can accomplish a very similar thing using Windows clustering. OSG has other customers that want Windows Server 2008 R2 machines and they tailor offerings to that as needed.
How will continued innovation around Microsoft products be addressed?
Chris Leopold also inquired about how future Microsoft-based service offerings might considered and handled. Within IFAS we discuss needs here at the ICC and then raise policy decisions to ITPAC for consideration as appropriate. When directions have been set, then the IFAS IT Server Administration group handles the implementation. Moving ahead, should IFAS decide that some new service is needed--say Direct Access as a VPN replacement--how might that best be raised to OSG for consideration? Should these things be brought to Dan Cromer and then from him to Iain or Elias?
Elias quickly responded that he does not expect to be involved with decisions on that level. Rather he has encouraged Dan to contact the service providers directly for resolution; Elias feels that Dan is very good at that. He said that he gets involved only if people are not playing nicely with each other or if we need to make an investment for which we have no resources.
Access to decisions and decision making
The new IT governance structure
Elias then briefly described the new governance organization and passed around handouts of its overall structure. The governance has a "Shared IT Infrastructure" group where Elias encourages everyone to bring whatever issues we believe the organization should address. Elias would like to leverage that governance wherever possible.
The Campus IT Directors group
Elias has also established a group called the "Campus IT Directors", of which Dan Cromer and Wendy Williams are currently IFAS members. That group meets now on a monthly basis and is also very much encouraged to bring whatever issues they think are important for group discussion. There are a number of different venues which have been established for people to communicate back and forth with the IT Organization to ensure that the services they provide meet expectations.
Hopes for more rapid adoption of innovation around Microsoft solutions
Steve pointed out that in the past a number of innovations at UF began as grassroots movements here at IFAS; getting from there to centrally-based services was difficult but ultimately successful. Steve is excited that the opportunity for a strong centralized group now exists with experience in the Microsoft solutions upon which IFAS has built and depends. He hopes this might lead to centrally supported implementations much more quickly in the future. One example is the current interest in SCCM; it would be great to involve OSG in that as early and often as is possible.
OSG team meetings will be open
Steve asked if there would be other conduits of information available other than through the committee and organizational structure. Are or will there be open meetings that interested parties might attend to keep abreast of current OSG projects? Iain responded that OSG has team meetings on Monday out at the East Campus which everyone is welcome to attend either in person or via LiveMeeting. Steve said that sounded very promising, as the ICC has interest in building new relationships in whatever ways we can.
How will transparency be handled with the topical committees?
Elias asked Steve if he had any ideas on how UF IT might better engage the ICC than how they are doing currently. Steve responded that he had heard the term "transparent" used in referring to the new governance structure. One of the things Steve feels is necessary for that is to be able to see what these various committees are doing, which Steve believes would entail the keeping of good meeting minutes.
Steve asked if it is the plan to have a secretary for each of these committees that will publish what is going on or is that outside of our means? Elias responded that we cannot afford secretaries and asked that Steve please not suggest that to the groups. We are, however, asking them to keep notes; we can't have minutes because then it becomes a public record conversation which we are trying to avoid. You have to invite the media and there are legal terms which we just cannot use according to the UF General Counsel--though Elias admitted he was not clear on the entire context behind that. However, notes will be kept for these meetings and those notes will be posted on the web site for each of these groups. That should provide an idea of what the conversation is all about and what the issues are which are being brought forward.
A web-based forum for input to the committees is planned
Ben Beach asked if that web site would include a forum where any IT worker might be able to make comments or suggestions? Elias said that yes, there will be an opportunity for people to provide information electronically to those committees and for them to take it under consideration.
IT staffers are likely to be caught by surprise from time to time
Steve pointed out that what has tended to happen in the past is that something is decided and we are the last to hear about it, even though we are the ones who have to implement it. Elias responded that this will very likely happen again. This is a large place and you are not going to be able to touch base with every single person all the time in the right sequence. There are going to be times where you find out something that you probably should have found out earlier. The scale and sequences within which events have to take place sometimes means things just have to go.
Elias said he is not going to pretend that the governance structure will solve all the problems; rather this is going to provide a venue for us to talk as collaboratively across the campus as we possibly can. Elias added that he is open to adjusting things as is appropriate as we move forward. If people have different and better ways of accomplishing things, Elias is all for it.
An example surprise notice: the Electronic Copy - Print Output Cost Reduction proposal
Steve asked if we could use as an example the "Electronic Copy - Print Output Cost Reduction" program which just became public yesterday. The notice forwarded to our Departmental Chairs and Directors from the deans along with Dan Cromer's forward to the ICC-L were basically the first any of us had heard of this. It seems to be something that is a mandatory thing which will be quite sweeping in scope. Should we expect things like that often in the future or was this surprise the result of its arrival ahead of governance development?
This was not an IT initiative
Elias responded that this is not an IT initiative; he takes the blame for many things, but this is not one of those. Elias explained how this proposed project happened. University leadership has the obligation to engage our units when such issues come to the table. A presentation on this initiative from the CFO was made to the President's Cabinet and all the Senior VPs and VPs were present.
A year and a half ago, before Elias came here, the university engaged a consultant to inventory how much printing we do and how many copiers/printers/all-in-one type devices we had. That work was done by the Procurement Office. The consultant came back and made a recommendation about a different model to manage printing on campus. That model suggests that, if we change things, the university will save $3 million. That presentation was made to the senior leadership.
It is Elias's job to say, wait a minute--I can't sign off on this yet until I go talk to my staff. It is not the problem of the person who presented this, rather it is Elias's job to either get feedback from his staff to see what this might take or to say he knows enough, it is fine, and he will just sell it to his staff. That's how things occur.
This proposal was not mishandled
Elias doesn't believe this was really done the wrong way. The VPs who are responsible for their areas should make the call as to how much they want to engage in these decisions with those they represent. The President looked at it and asked if everyone was okay with it and they all said we are good to go. UF is a big place. If you want to make a decision and you have to talk to every group, committee, and person on this campus then nothing would ever get done. We all know this from having been on committees where a lot of discussion occurs but no decisions are made; that is higher education for you.
Understanding higher education
Elias sees this all the time; sometimes thirty-six to one is a tie in higher education. You hear everybody, they all express their opinion; you trust the leadership that they are going to be held accountable for the decisions they make. If they work, great; if they don't then someone will have to do something about it. So, that is how that decision was made and Elias is not sure how else you would do it. If you are the CFO and made that presentation, what else would you do?
Ben Beach said that they might include someone from IT. If it was anything that could have an impact on IT Ben would want to have someone from IT at that meeting. Elias pointed out that he was there at that presentation last week.
The message may have been misinterpreted
The message that came out of the meeting was that it is being implemented and all the details have been figured out; that, however, is not what was presented. What was presented is a model which provides the opportunity to save $3 million. We still have to have an Invitation to Negotiate (ITN) for someone to come on campus and really prove that.
An ITN will work out the details
Elias commented that we all have dealt with consultants; they sell you everything but what they need to be selling you. Elias told Dr. Machen after the meeting that taking that $3 million and dividing by some number--that is probably closer to the truth. But in his defense, Machen said there is another process which we will go through. We will do an ITN; someone will come on campus and look at all the details of location, networking, support, etc. The only thing which was agreed to was a model to, for the most part, outsource printing. The university will move to a model where we will no longer own these big copiers. They will be provided by the vendor and instead we will be paying by the click.
The rationale behind the proposed new model
According to the consultant that did this work prior to Elias's arrival, the average cost per print on this campus is $.076. The consultant is saying they could bring that cost down to $.016. That's a lot of money, so if you are the President of the university, this is something about which you would wish to hear more details. That's where we are. Elias said that other important point is that the money being saved under this proposal would stay with the units. The university is not trying to grab that and retain it centrally. Looking at it that way, Elias believes this is worth looking into further; let someone come in and demonstrate these savings are feasible. Elias believes that at that time there will be more engagement about how we are actually going to move on to supporting it.
Budgets are a continuing worry and cost cutting will remain an important topic
That is how these things happen. Will more of these happen in the future? Probably. The university is trying to see right now if we have an immediate budget problem. At every cabinet meeting people are waiting to hear whether or not we have a budget cut coming. Everybody is going to be looking around to see where we can save money while continue to provide services.
What will be the scope of this new printing model?
Wendy Williams asked if this printing initiative would include computer labs where students print. Elias responded that he did not know the answer to that. This is one of the questions which was followed up on the next day by Fedro Zazueta. Lisa Deal with Procurement was the person who really coordinated with the consultants and Fedro has a meeting scheduled with her for next week to see exactly what is in that $3 million.
Dennis Brown asked if this program was meant to cover not just copiers but also individuals' printers as well. Elias did not believe this includes desktop printers. He thinks rather this refers to the "big devices" out in the units. Dennis then asked who the vender was and Elias explained that we don't have one. The next step is to do an ITN and select a vendor.
Wayne Hyde asked if the savings possible from replacing individual printers with shared network printers had been investigated. Elias said that the idea is to eliminate as many of those individual printers as we can, but he has been in high education long enough to know that a lot of faculty are not going to like this. Elias believes they are going to try to go after the "big rocks" with this.
As an example, Elias mentioned that he doesn't have a printer in his office and hasn't had for about twelve years--retrieving output is the only exercise he gets. Some people are not going to want this but there are other people really in need of special services: the College of Architecture and other units where they need plotters and other devices. This will not be a one size fits all deal. The ITN process will dig in to flesh out those details. This will give UF the opportunity to invite vendors and talk things out with them before deciding which way to go.
This program is expected to be limited to the Gainesville campus
In response to another question from Dennis, Kevin Hill reported that a qualification came out yesterday that this proposal would be limited to the local Gainesville campus.
Elias did not know the answer to that but said there will be a committee identified to work on the ITN and he knows IFAS will be a part of that.
Summary on the printer topic as an example collaborative process
So those are the details on the printer proposal. The good news about governance is that it creates a venue for participation; the bad news about governance is that it can become bureaucratic. Consequently, some sort of a balance must be struck. Those processes will have to be evolved and the risks must be managed.
Like everything in life, nothing is 100%. If it is 80%, then Elias is going to be a happy guy; but if it is 20% then obviously it doesn't work. There will be things that are going to be missed and decisions will be made that people will find out about afterward. We have to accept those things for what they are and manage them.
Elias reminded us that the overwhelming majority of such proposals come from people with good intentions. They may overlook something because their role is not as broad, but they do have good intentions. In Elias's experience, one is able to talk to those 99% with the good intentions and they will be willing to re-evaluate in the light of additional evidence. Of course, there is the other 1% and you must deal with them as well. That's how it's going to be and that is what we must work through to get things done.
The balance between diversity and centralization
Dennis Brown said that he wanted to obtain a better idea of what direction the campus might be heading technology-wise. He mentioned that IFAS has been very Microsoft-centered and he was curious to know what trends Elias saw throughout the rest of campus. Elias responded that nearly every technology you can imagine exists somewhere on the UF campus. UF is very diverse and, rightly or wrongly, very decentralized. On the Enterprise side, Elias said that we are trying to be as diverse as we possibly can while still be able to support all these things. Elias is seeking some type of balance between diversity and centralization where some technology will be standardized--but it is not going to be every single technology. We will still have to support innovations and meet the emerging needs of our user base.
Elias is not married to a particular technology
Right now we have a pretty diverse portfolio. As we move forward, with every initiative we adopt we will evaluate which technology best meets the needs. Elias is not religious on any technology; he is not a Microsoft guy or Apple or Sun. He has never been that way, does not believe in that way, and doesn't think any organization should be that way. He thinks we should look at each problem and evaluate the best solution. If that is Microsoft, so be it; if it is not; so be it. We will be as diverse as we possibly can as long as we can still support it.
Engaging vendors in roadmap conversations
Elias said that one of the things they are going to do is engage our major vendors in strategic conversations. This afternoon, for example, Microsoft is going to be in Elias's office for two hours giving him a quick look at their roadmap. To the extent it doesn't turn out to be a sales pitch, we will probably open up that conversation to larger audiences on campus. Dell was here yesterday and Oracle is scheduled soon to provide a roadmap conversation. All of that is being done in the hope of being engaged with where the technology is going and how we position ourselves with those roadmaps.
Elias is well aware that a lot of innovation doesn't necessarily happen within centralized organizations. Centralized organizations are built to standardize and leverage economies of scale. That allows for the scaling of systems and provides insurance that they are robust. Once in a while central organizations are innovative, but in general, innovation occurs out in the units.
The Active Directory conversation is a very good example of grassroots effort where a service was evaluated and its potential for widespread usefulness became apparent. Let's throw out weight behind that and make it happen. To Elias, that is where a central organization provides value; it provides a means of supporting such things and making them reality.
But you obviously cannot do them all, so you pick and choose where they fit into the roadmap of the university and the roadmap of where those technologies are going.
Summarization and final discussion
Elias summarized this discussion by answering the question, are you guys going to find out sometimes that a decision has been made and you didn't know about it? Yes, you are. This happens even to Elias who sits at the highest levels of the university. Then you find out that these people only have eight hours in the day and have seven thousand people to talk to; it's okay. With good intentions you move on. Elias will engage the people as well as our major partners in strategic conversation and that process has already begun.
Steve thanked both of our guests and commented good naturedly that he had been a little concerned back when Elias mentioned getting involved only when folks aren't getting along. Steve hoped that this wasn't Elias's main motivation for being here today. Elias responded that he really believes in hiring good people and then getting out of their way. Iain does not need Elias; what Iain knows Elias will never learn and honestly does not want to learn. That is what Elias had meant; he tries to provide a supporting role in helping them do their jobs. He truly believes leadership is not about having people follow you as much as it is having people believe in what you are trying to do and doing it together.
Elias is not looking to make every decision and he is not looking to be every place. These people know what they are doing and are pretty good at what they do. If they need his help he will help them; if not, he will get out of their way. That is the way he likes it to be and he tells people all the time: if you come to him and complain about each other then one of you is not doing your job; if so, he will have to do your job. He tries to minimize that so (joking) nobody wants to come to his office.
He also tells people to pick up the phone and call somebody. IT people are very good at drafting e-mails. You hit the send button and think, well, I spoke my mind...you didn't hear me... Elias said that we all know that e-mail is the worst form of communication if you want to solve any problem. Consequently, Elias has a "two e-mail" rule. If a problem doesn't get solved within two e-mails he picks up the phone. He expects us to pick up the phone and talk to him and he encourages his people to do the same thing. If you have a problem with any of the services, call the provider and ask what is going on--as opposed to deferring to some hierarchy for the issue to get resolved. Elias tries to get out of the way of his staff and he suspects they like it that way.
The IT org chart
Elias also provided an organizational chart to help us visualize the organization that he is building. There is one change not noted: Brian Beach has decided to take on a new responsibility outside UF and is returning to the private sector. Brian is the Senior VP who is Elias's boss until January 25th. After that, Elias will report directly to the President as Brian's position will not be replaced.
Other than that, this chart is what we look like today; how things will look next week Elias does not know.
Topical committees have begun to meet
Steve had noticed that the chairmen of the various topical committees have now been posted and asked if they have started to meet. Elias responded that three of the six have met already for the first time. Education and Outreach has met, Research Computing has not. Administrative Systems and Web Services have each met but Security and Shared Infrastructure have not.
Steve mentioned that he hasn't heard much yet from our new IT Security and Compliance head, Rob Adams. Steve asked about how he is getting along. Elias responded that Rob is still trying to find his way around campus; Elias knows from experience that it is like drinking from fire hose at first. There is a lot of stuff coming your way; it is a new culture and a new system that requires considerable adapting. While attempting to learn that, Rob has been thrown into the middle of managing a Federal subpoena served to UF investigating the manner in which research funds have been managed here. This is an enormous lawsuit similar to what Duke and others have already gone through. Rob is representing us currently with the General Counsel; the good news is that he just finished doing this with Duke so he has the experience to do that here.
Elias encouraged the ICC to invite Rob to an upcoming meeting; he is sure Rob would be more than happy to do so. Iain said that Rob is a really good guy and very personable. Iain's team has actually been working with him a lot on this investigation. He knows that Ron and his team are swamped with that because of all the information which is being asked for.
Steve thanked our guests and mentioned having met Elias prior at an ITPAC, but that this was his first opportunity to meet Iain; for some reason their paths had never crossed before but he was very glad to have finally made his acquaintance. Steve mentioned that he had been in contact with James Oulman for the first time recently concerning the transition from Barracuda to Proofpoint and had been very impressed with James's openness and responsiveness. Iain responded that James is a very good guy and OSG feels lucky to have him. Iain mentioned it has really been fun to watch James and Luis work together. Luis kind of picked up our Exchange 2010 infrastructure from Dwight and Luis had some slight changes to suggest on the architecture. Then as James dug into it more he asked why are we doing these double proxies and these other things. It has been fun watching them figure out how to streamline things.
Elias thanked the ICC for giving them some time this morning and he was sure we would meet again various places on campus. In the meantime, he asked that we let them know and keep them engaged. Elias and Iain then left.
After thoughts from the discussion with Elias and Iain
Kevin Hill wanted to note that his concerns remain about OSG favoring open source solutions over Microsoft solutions. For Kevin, CNS and OSG have always been sort of a "black hole". He doesn't know people there and the IT/SA group doesn't know that whole team yet. Kevin is concerned that access to the solution providers will be all too cumbersome and he fears that Remedy ("which takes two minutes to load and then you don't know what the heck to do with it") or some similar interface will not be conducive the sort of collaboration that leads to rapid problem solutions.
Kevin added that communication is the key and the relationship with the folks supporting us is absolutely critical. It is important to tier 3 support to be able to feel they have a handle on their environment and be able to deal with an issue without having to send in a ticket and have to wonder who is looking at it and when they will look at it.
Patience and cooperation will pay off as we learn new ways to collaborate
Chris Leopold responded that he shared Kevin's sentiment, but suggested that we first give OSG a chance to get things organized regarding this transition before we lay on the criticism.
Dennis Brown noted that OSG does have an email address (email@example.com) and that is generally how he contacts them with good results. A forms-based contact method is also available. Steve would like to note that the "UFAD Team" distribution group (firstname.lastname@example.org) has added a number of OSG folks now as well, including James, Keith, and Iain. Dennis and Ben also pointed out that Elias has urged us to pick up the phone as well.
Steve noted that Iain has expressed the desire to engage with us and he felt that was further indicated by his willingness to be here today. We have started a conversation and Steve tried to set expectations that it could actually be great to have stronger centralized Microsoft support. Steve reminded folks we have been saying for years that the MAG group didn't have enough people or support from central administration. This is just the first step and it will be an evolving process which we will have to continue to work at and improve. Steve reminded folks that this move has stressed OSG nearly as much as it has the MAG group, but they are clearly taking steps to deal with it and they deserve our patience and assistance in working things out.
Concerns with CNS not limited to OSG
Kevin provided the example of the trials he has seen in getting a new circuit installed at Immokalee--though this relates to CNS Network Systems and not OSG. They started with that back in 2009 when the money was available and it has now been well over a year without results. He didn't mean to lay blame on anyone in particular, but the fact remains that it is just not getting done; Kevin feels that if it had been left to the REC it would have been done in less than a month. The bureaucracy that comes with all this chain of command seems paralyzing to progress.
When Steve joked that it is now "UF & Immokalee" [along the lines of "UF & Shands"] Kevin responded that he had stayed home for this meeting today so that he could guarantee a good connection.
New Microsoft Campus Agreement signed December 16th
We are still awaiting exact details, but Dan Cromer had disseminated the following to the ICC-L:
Message to the ICC-L from Dan Cromer:
Great news from Elias this afternoon, the Microsoft Campus Agreement was signed yesterday to include faculty, staff, Shands, and students, with student OS and Office included, along with core CALS, if I understood correctly it also includes OCS CALS for students. The good news for students is they get to keep their license when they leave. Formal announcements are being composed as I write this.
The formal announcement was apparently delayed and on January 6th Dan sent out another message providing further details:
Message to the ICC-L from Dan Cromer:
At a UF IT Directors meeting this afternoon, it was announced that formal announcement about the Microsoft contract would come soon, as soon as the media was available in the UF Bookstores and Hub, which is expected tomorrow or Monday. It's already shown on the ISIS site, http://www.isis.ufl.edu/, and we were told the most frequent question for the UF Help Desk was "When will the Microsoft media be available?". I understand the media cost will be $15 for Win 7 and Office 2010. They will keep track of which students buy, as only one purchase per student will be allowed at the media-only price. It includes OS upgrade, Office for MS or Mac, core CAL, Windows CAL, SharePoint CAL, and Exchange CAL. However, nothing has been decided yet about supporting students with Exchange, that's a discussion for later.
When Joe Gasper responded asking why OCS was not in the list of licensed CALs for students, Dan responded "I would have hoped to have OCS/Lync as part of the contract, and it may be, but just not mentioned in the announcement I saw."
After some other discussion on the CCC list, Erik Schmidt cleared up many of the details via the following post:
Message to the CCC from Erik Schmidt:
A formal announcement is expected from the CIO's office shortly but yes, the 'Student Desktop Bundle' has been added to the campus agreements. This will bring Windows and Office into the hands of students for only the cost of the media, just like the work-at-home (WAH) program for faculty and staff. The only difference is that the WAH program terminates licensing when an individual leaves employment at UF and the student program provides 'perpetual use' after a student *graduates*. (No graduation = no perpetual use.) As with previous agreements, I will continue to handle enterprise software distribution through the MVLS and James Hardemon will handle retail software distribution through the bookstore.
It is very likely that the announcement was delayed pending the arrival of additional media. Not much sense having a run on media when there isn't enough stock to support it!
All other facets of the campus agreement were renewed and remain in place, such as E-CAL and the Server Platform Agreement, which allows unlimited use of any version of Windows Server on campus-owned machines.
Please let me know off-list if there are any specific questions about this licensing.
Steve noted his great relief upon reading this posting from Erik because the question of whether or not eCALs would continue for us has now been answered. Andrew cautioned, however, that while we will continue to have eCALs for faculty and staff, students do not. Consequently, students who are not also employees are not licensed for Office Communicator.
Dan Cromer stated that once the Exchange 2010 migration is well under way he has a commitment from Iain that he will let Luis work on Lync. This is one reason we need to get to Windows 2008 R2 native mode for UFAD. Lync will allow federation with outside groups that should permit students to participate in meetings held here via Lync. Student will thus not be able to host meetings, but they certainly can attend them.
Videoconferencing topics (previous discussion)
Connecting to the bridge via Office Communicator
Steve noted that he had received a couple of questions on how to connect to the bridge for the ICC meetings using Office Communicator. The definitive description of that process is available on the UF IT Wiki under the Office Communicator topic within the section labeled "Office Communicator with videoconferencing bridges".
Steve would also like to note that the web streaming instructions he sent out were incorrect because this year the conferences are being held on the second bridge. The ICC Meeting has now been updated to reflect that, but basically, VCs with Conference IDs of 7831xxx are at 188.8.131.52 and VCs with conference IDs of 7832xxx are at 184.108.40.206. He apologizes for any inconvenience.
Francis Ferguson asked how to go about renewing the maintenance on Polycom systems. Dan Cromer responded that this could be handled by contacting Lance Cozart with the serial number of the endpoint. Lance will negotiate with the various vendors to determine the cost.
Office Communicator infrastructure status (previous discussion)
No updates available...
Recording lectures for Distance Education (previous discussion)
Considering AMMS for Accordent management
No updates available...
WAN transition to CNS (previous discussion)
Updates from James Moore
James was unavailable for his usual update this month, but Kevin Hill hoped for a status update on the new circuit which has been pending at Immokalee. Dan Cromer responded that this process has been a real "ping-pong" game between UF Purchasing and Comcast on the wording of the contract. Dan understands the ball is now in Comcast's hands to come up with the final agreement; UF Purchasing is ready and raring to go otherwise. The PO was cut for the PAETEC circuit for Belle Glade, but they didn't have a signed contract. The vendor wanted to put in a penalty clause for early cancellation and Dan had to work with Purchasing to okay that since it initially raised some flags.
New wireless access policy!
Dan Cromer reported that Rob Adams, UF's new security guy, has decided to allow a shared password provide access to wireless. While this is particularly good news for the RECs, it should make certain things easier--especially at remote sites. A password can be generated for wireless access that is changed, perhaps, for each meeting or at some other efficacious period. This would allow prior practices to continue without the need to create individual GatorLink guest accounts for all wireless users. Dan is sure this is welcome news.
Steve asked how important this topic was to folks and Andrew Carey responded that it had been important mainly to Dwight from the standpoint of the central Exchange organization. Steve noted that it can also cause confusion for the users. He explained to the more recent folks that IFAS had accounts back in the IFASDOM days and email addresses from those still remain as aliases. A number of people had different usernames under IFASDOM then they do under GatorLink. Users are often confused that one can't universally change an "ifas.ufl.edu" or "mail.ifas.ufl.edu" address to its "ufl.edu" equivalent by just changing the domain portion.
Chris Leopold suggested that we should at least get people started unsubscribing from lists that they might have joined via these other domains. That would be a necessary step in cleaning things up in any case.
Steve wants to remind everyone of the "UF_PA_IDM_NETMGR" role which will allow you to set NMB for your users. Your Department Security Administrator can do that for you.
One-time GatorLink name changes halted temporarily by Sakai design flaw
Dennis Brown raised the point and Dan Cromer confirmed that Sakai has utilized GatorLink usernames rather than UFIDs as the key it uses across its database. Consequently, individuals wanting to make use of the "one-time" GatorLink name change which has been allowed prior will have to wait as that would prevent them from being able to use Sakai. [See Known Issues]
UF moving to BigBlueButton
Dan Cromer mentioned in passing that UF is looking into replacing Elluminate with BigBlueButton
There has been a hitch in UF's LOA1 plans
Steve asked Dan Cromer if they were still considering allowing the use of Sakai for outside collaboration--at least until the LOA1 implementation was completed. Dan responded that the LOA1 project has run into a snag. It turns out that the UF definition of LOA2 doesn't match well with that of the Internet2 group and that has caused UF to defer development at this time of a means for anyone to generate Gatorlink credentials via the web. Dan never addressed the original question and Steve regrets not following up.
Better access to ongoing discussions?
Steve asked how the general IT community might find out about these sorts of things. Where are they discussed? Are they written down anywhere? Dan responded that there may be some notes from meetings but these things are still basically preliminary discussions. As soon as there gets to be more agreement about the course we are going to take, he assumes details will be published.
Steve wondered why broader input isn't sought regarding what sorts of things our faculty, students, staff want to be able to do. Dan pointed out that small committees get together to figure out basic things with the concept that each member represents some group and is keeping their best interests in mind. It is a distraction to have too many inputs to such processes.
Steve wants to keep this on our agendas in case discussion seems warranted.
UF Exchange upgrade
No updates available...
Barracuda load issues - replacing with Proofpoint
All ICC members were moved away from the Barracuda to Proofpoint on December 13th. We should each have received a welcome letter from James Oulman:
Message from James Oulman:
I have created Proofpoint user accounts for each of you and carried over your safe/block list settings from the Barracuda. Proofpoint end-user documentation can be found off of Managing Email at http://www.mail.ufl.edu. Please let me know if you have any questions that are not covered by the documentation.
The URL to access your quarantines is https://quarantine.mail.ufl.edu and provided you get spam between now and tomorrow morning you should receive a digest report @ 7AM just as you did with the Barracuda.
Thank you for your help testing and again please feel free to email me with any questions or concerns.
After several days of not seeing any messages quarantined, Steve began a brief discussion on the ICC-L. James Oulman joined that list and responded to questions:
Message to the ICC-L from James Oulman:
Spam scores in Proofpoint tend to either be very high or very low. I am attaching a csv that shows the distribution data for the last 7 days. These stats to not reflect actual message volume as much of it is blocked using DNS blacklists at the connection phase.
Additionally if you would like generate probable spam score (79) you can add !=xxxxtest1xxxx!, minus the = symbol, to the content of the message. We use this for internal testing. When the departmental migrations begin we will use a special header to generate the required scores.
Micah, if you can send me the X-Proofpoint-Spam headers to me I may be able to help you filter these types of messages out. Proofpoint treats bulk mail differently than say spam, phish or adult content. We are not utilizing the bulk scoring at this time since it tends to cast a wide net and some of what it classifies end-users actually request. You can create email client rules that test against this score and move them to Junk.
As Steve said I just joined the list and may have missed other messages in the thread. Please let me know if you have any other questions or comments.
There seemed to be a bit of confusion on the part of some as to what the user-controllable settings did exactly, but as mentioned in the welcome letter, documentation is now available on-line covering Using Proofpoint and Proofpoint SPAM Settings and Scoring.
Dan sent out an IFAS-wide announcement just as he had proposed:
Message to the IFAS-Announce-L from Dan Cromer:
UF Computing and Networking Services (CNS) is planning a change to the programs used for filtering spam, to begin for IFAS on January 23rd. In short, the conversion should not have a noticeable impact on you after conversion if you take no action. However, the new program, called ProofPoint, has a different interface, and improves your choices about what mail you want delivered or filtered. Documentation about ProofPoint is available at http://www.mail.ufl.edu/usingproofpoint.shtml.
All members of the IFAS Computer Coordinators (ICC) group were converted to ProofPoint by December 12 for early testing; your local computer support person or the IFAS Help Desk can answer any questions you have about it.
The actual implementation now appears to be scheduled for one week prior to the end of this month.
Steve Lasley admitted to having not closely looked at the Proofpoint settings levels since he had seen no increase in spam to the inbox and little to nothing in quarantine. Gasper, however, raised the point via the ICC-L that the controls are quite limited compared to what we had experienced with the Barracuda. The four available control settings are also poorly named:
Message to the ICC-L from Joe Gasper:
With current ProofPoint (PP) spam profiles, the inbox of a user will not see anything different between the settings "Low Spam Protection", "Medium Spam Protection", and "High Spam Protection". In all 3 settings, their inbox will see the same amount of spam (scored below 50). All 'cuda users and administrators understand currently that if a user wants "Higher" spam protection (i.e., fewer spams in their inbox), we set the quarantine score to smaller number - and more spam (and non-spam) will be placed in their quarantine reports. Why would users not all choose "Higher" as labeled currently in PP? In the currently labeled PP "Higher" setting, that would not reduce the number of spams in their inbox, it would only reduce the number of messages listed in their quarantine report (and likely, delete real messages they wanted - how long and hard of a fight was it to change the central 'cuda boxes to stop deleting non-spam messages? That is the point of quarantine, users whitelist and eventually they rarely need to look at the report). When we are talking "Protection", I think it would be best that it relates to the quantity of spam in a user's inbox - currently the 3 levels just determine how many messages are listed in their spam reports.
Suggestions on naming and scoring:
And the "Off" setting is not off as current Exchange/'cuda users would expect. Off would be as follows:
We have gone through the need of a true off before and I believe there are departments and users that use that setting.
I don't know if the difference between my suggested "Less", "Standard", and "More" are enough to warrant the need for "Less" and "More" (I leave that to James to help us understand).
If there is some agreement here and/or need for more discussion, we can bring this forward at next Wednesday's Tier 2 meeting.
Steve began to wonder why we had significantly more messages quarantined by the Barracuda than are being quarantined by Proofpoint given the following flow:
Proofpoint Scores 100 98 --------\ \ \ 80 -----\ \ \ NOW: Proofpoint Quarantined (default settings) \ / \ / \ / \ 50 --------/ \ \ PREVIOUSLY: Barracuda Blocked/Quarantined/Delivered / / / / / / / 0 ------/
It does seems odd that the Barracuda would have quarantined several times more messages than Proofpoint does given that Proofpoint had first crack and was discarding the 80-98 range prior but is now sending them to quarantine.
James responded to earlier ICC-L comments publicly:
Message to the ICC-L from James Oulman:
Sorry for taking so long to respond. I have asked Kiem to add a topic to Wednesdays Tier-2 agenda to discuss the spam profile naming conventions and behaviors.
I have gotten a few questions about what Proofpoint is doing right now in the smtp.ufl.edu to Barracuda mail flow so I'll touch on that first. The majority of mail is evaluated using a default policy we defined when we started using Proofpoint 2 years ago. The scoring/behavior is as follows:
The X-UFL-Spam-Level header added by the system for messages scoring in the probable range is evaluated by some systems downstream. For example, Gatorlink mail matches on this header and places the messages in the users Spam folder. Other departments that smtp.ufl.edu provides MX for may also use this score. Neither Exchange or the Barracudas use this header. The Barracuda's evaluate messages that Proofpoint scored in the 0-79 range.
The score distribution in Proofpoint tends to be very high or low. I've attached a csv that shows the score distribution for the past 7 days. We have found over time that messages scoring < 50 are most likely not spam which is why the profiles do not make any evaluation on them.
The intention of the protection levels was to give users a good set of policies that would allow them to receive more or less spam to their quarantines. This would allow those that want to receive less spam to use the "high protection policy" and those who do not want to lose any mail to use the "low protection policy". We're trying to keep most spam out of the quarantine but keep the possibility of messages getting thrown away lower in the medium policy.
The off profile was created for departments that do not wish to receive quarantine notifications but still allow some level of protection against spam. The off profile works like disabling spam scoring and notification in the Barracuda. Messages scoring 0-79 are delivered to the users Inbox and the X-UFL-Spam-Level header is added.
I agree that there are improvements that can be made and am looking forward to discussing this and getting everyone's input on Wednesday.
Please let me know if you have any questions or concerns about any of the above.
Steve wants to point out that James has been very open and responsive about this service. Just one indication of this is that he increased the quarantine retention time almost immediately once it was pointed out that the Barracuda settings were longer.
Summary of this week's Exchange Tier 2 and Exchange Policy meetings
They want to eliminate the "Highest spam protection policy" setting because they feel messages scored between 80 and 98 should always go into the quarantine. For security reasons they also did not want messages scored 50 or above to be delivered to our inboxes; neither do they feel allowing one to disable the Quarantine is a good idea. The default will remain as "Medium spam protection policy" and it is suggested that only two options will be offered: Medium and Low. This was approved by the Policy committee and is being passed onto to Tim Fitzpatrick for consideration.
The wording of those two options unfortunately is not adjustable at this time nor is it clear from this page that "Default means "Medium". It will be investigated whether or not this page can be customized.
Dan would like to propose that an advanced "Custom" setting be considered as well along the lines of the Barracuda that would allow setting the various cutoff scores individually between blocking, quarantine and delivery.
Wayne Hyde noted that blocking includes much more that simply those messages scored > 98 (previously >80). DNS blacklisting and virus protection likely account for the vast majority of total emails blocked.
What surprises Steve is that messages scoring between 80 and 89 were formerly blocked and should now show up in the quarantine; surprisingly there is still very little in quarantine--somewhere in the neighborhood of six times less than with Barracuda for Steve (he has saved all the Barracuda and Proofpoint reports since day 1). It is almost like we are arguing about how to better control "nothing" because there is nothing there to control. Wayne agreed that they might as well just turn off the Quarantine because it really isn't doing anything.
Centralized FAX service via Exchange (previous discussion)
No updates available...
Steve wants to keep this on the agenda for future reference.
IT survey is coming (previous discussion)
No updates available...
Outsourcing of DE course development (previous discussion)
Updates not available...
New web cluster
More about cleanup of IF-SRV-WEB prior to the move is to come, including that wmv and wma files on the video server need to be moved to media/video server
No updates available...
No updates available...
No updates available...
There continues to be no progress on the documentation which was to happen prior to announcement. Since this has never been formally announced, the matter remains on the agenda as a standing item.
Ad hoc UF SCCM Support Group formed
Steve noted that a new UF SCCM list has been created (send a message with the command "SUBSCRIBE SCCM-L Firstname Lastname" in the body to email@example.com) and an ad hoc group has developed in order to share information about SCCM. An inaugural SCCM Support Group meeting was held Thursday from 2-4PM in McCarty D Room G001 which was attended by many across campus--including our own Andrew Carey, Kevin Hill and Dennis Brown.
Steve asked Andrew how the meeting went and Andrew responded that he believed the meeting was useful. It was more of a discussion get together than it was a meeting for planning a centralized SCCM structure. IT at the Provost's Office has done considerable work with SCCM as has CLAS and it was helpful to Andrew to hear about some of the things they are doing. Apparently, a good deal of information has now been posted on the UF IT Wiki concerning what they have done and how they've done it.
A centralized SCCM hierarchical structure?
Kevin Hill said he did not know many of the people there but wondered if OSG had any representation. Andrew responded that Erik Schmidt and Mike Kanofsky were there as well as David Burdette from CNS. The reason Kevin asked was that Erik Schmidt had raised the possibility prior of hosting a "root" SCCM server which could be the start of a useful hierarchical structure for SCCM here at UF.
Andrew responded that he believed getting an UF hierarchy started is going to take a bit more grassroots effort on our part. Getting the Provost's Office and CLAS behind that effort might be the best means to that end, but Andrew isn't sure how disruptive that might be to their ongoing work or how amenable they might be to that.
Chris Leopold noted that the SCCM product is still not very mature and wondering if holding off on things for a bit might not be a good idea. Andrew responded that if we begin discussions about a collaborative SCCM structure now, by the time SCCM 2012 is ready to roll UF may then be ready to move.
No updates available...
Re-enabling the Windows firewall (prior discussion)
No updates available...
Services Documentation: Is a Wiki the way? (prior discussion)
No updates available...
Restoration of back-ups on the file server
Wayne Hyde intends to document and announce proper usage as time permits.
Membership of ". IFAS-ICC" email distribution group to be narrowed to ICC members only (previous discussion)
Steve will keep this as a standing item on our agendas for now as a reminder. The ICC distribution list is more targeted and restricted to IFAS IT support folks only.
No updates available...
Steve had left this on the agenda in case further discussion was deemed warranted.
Steve wants to keep this topic on our radar.
Moving away from the IFAS VPN service (previous discussion)
Steve assumes that moving our VPN to private IP is waiting on Wayne Hyde finding the time to implement.
VDI desktops as admin workstations (previous discussion)
This is another cool service that Wayne has in progress and which is awaiting sufficient time to pursue further.
No updates available...
Computer compliance tool in production (previous discussion)
Updates not available...
You are reminded to please take the time to read and implement the new standards. If you have any questions get with Wayne or Steve.
This is yet another matter for which finding time for implementation is proving difficult. Steve wants folks to remember that Andrew Carey has a good plan for dealing with this which he simply has had no time to address. In the meantime, it would be very good of each OU Admin to consider mimicking the proposed plan manually by keeping their own records and deleting any computer object which have been disabled for 90 or more days; Wayne's Power Tools can identify those. Steve has finally begun doing that for his own unit and it has made his view within ADUC much more agreeable.
Core Services status (previous discussion)
DPM machine issues now resolved
Chris Leopold reported that there were a number of problems initially with the DPM server that have finally been resolved. Three drives were lost in one array (RAID 6--Uh-oh!) and two drives in the other. They were ping-ponged among three different Dell technicians trying to solve the issues. In solving this, a motherboard was replaced as was a riser. There were several firmware flashes involved and the RAID controller was replaced twice along with driver upgrades. The bottom line is that it is working and luckily no snapshots were lost in the process and nothing had to be recovered from tape.
Steve noted that when the rest of us out in the units are thinking everything is just going smoothly, centrally the IT server administrators are often sweating bullets. Chris agreed that this was not a pleasant process, but the end outcome has been fine. This did affect the enduser's ability to restore previous versions of files for at least one week, but backups were still available should they have been needed.
No updates available...
Status of SharePoint services (prior discussion)
IFAS migrating to centralized MOSS
No updates available...
Nothing further was available on this topic at this time.
The January Microsoft patches included only two bulletins. The first bulletin is rated as "Important" and affects Windows Vista while the second bulletin has an aggregate of rating of "Critical" and affects all supported versions of Windows.
McAfee provides podcasts on the highlights of each month's offerings and another podcast summary of these patches is provided by "Security Bulletins for the regular IT guy".
MS Office News update
No updates available...
Job Matrix Update status
This is here as a standing topic--no discussion this month.
No updates available...
UAC settings egregious for users?
No updates available...
PDF-Xchange (prior discussion)
No updates available...
A number of our usual topics were deferred to next month's meeting due to lack of time. The meeting was adjourned on-time at about noon.