ICC Meeting:

IFAS COMPUTER COORDINATORS
(ICC)

NOTES FROM December 10th 2004       REGULAR MEETING

A meeting of the ICC was held on Friday, December 10th, 2004. The meeting was chaired and called to order by Steve Lasley, at 10:10 a.m. in the ICS conference room.

PRESENT: Seventeen members participated. Remote participants: ??? On-site participants: Mike Armstrong, David Bauldree, Dennis Brown, Marcus Cathey, Dan Cromer, Marion Douglas, Joe Hayden, Kevin Hill, Tom Hintz, Chris Hughes, Dwight Jesseman, Jack Kramer, Winnie Lante, Chris Leopold, Steve Lasley, Mark Ross and Joe Spooner.

STREAMING AUDIO: part one and part two .

NOTES:

Agendas were distributed and the meeting was called to order by Steve Lasley at 10:10 am.

There were two new ICC members that were introduced in their absence: Paul Falor has been at the School of Natural Resources and Environment since May, but has been involved with computers for around 8 years now. He is very interested in network topologies, SAP, and all web technologies. Being the only IT guy over at SNRE, his responsibilities are at many ends of many different spectrums; "but", Paul says, "at least it stays interesting." Greg Wilson has been an Information Specialist/Graphics Editor/Webmaster with the UF/IFAS Office of Conferences & Institutes for almost 10 years now. He started out as OPS and is self-taught. Greg is the "go to" guy for computer issues at OCI, although he jokes that "he has them fooled."

As usual, Steve pointed out that he always includes notes to the last ICC meeting on the agendas to encourage members to recap where we are in our discussions.

The meeting started out with a discussion on making the ICC security group (currently at ad.ufl.edu/UF/Departments/IFAS/-Co-Managed/Campus/ENTNEM/Groups/IF-ICC) into a distribution list. Chris Hughes had proposed this originally and Dwight answered some questions regarding this since Chris was late to the meeting. Using this distribution list would allow us to leverage Exchange and Outlook in assisting how we communicate and schedule our activities. The main downside would be the issue of archiving our interactions via this mechanism. Currently we use the ICC-L listserv which automatically archives our posts. We will have to investigate methods of retaining that archive feature with the new mechanism. It might be possible to somehow include the listserv as a member of the distribution list so that it archived the messages, but not have any actual members subscribed to the listserv itself. Dan Cromer would like to see something like this for all IFAS listserv lists.

Joe Spooner confirmed that he has volunteered to chair the ICC IT Charter Subcommittee. Joe has been working with Chris Hughes to draft an outline, but noted that much depends on the settling-in of the incoming administration and how receptive they may be to such proposals. Consequently, Joe believes it will be a month or so before they are ready to really begin work in earnest. Kevin Hill mentioned that the Windows Hi-ed list would be a good resource for information on aspects that are unique to educational institutions in this regard.

There was discussion of methods for end user maintenance and other informational notifications from IT. Steve Lasley noted that this function seems to be missing currently from IFAS IT and that having a webmaster is one necessary component. The overwhelming majority of the ICC feels that IT administration has been ineffective for years in addressing this issue. The only evidence that this is being at all considered is occasional lip service agreeing to its importance (when pressed) accompanied by saying that it must wait for one reason or another. There has been nothing offered by IT administration that addresses a strategy for obtaining the necessary support from IFAS administration for this. Jack Kramer expressed his belief that SharePoint services could help greatly with this. Dan Cromer stated Joe Joyce has the position of webmaster/content manager as a high IT priority although there is no known public statement regarding that to which we can point. Dan believes SharePoint is a related but higher priority need, but that the problem with both would be funding--especially of the human resources necessary. Dan believes that we should build the service and leave content management to ICS under the direction of Ashley Wood.

One problem with all this, even if it were being aggressively pursued, is that it doesn't address the issue of how IT subject matter experts provide documentation for dissemination--but rather presupposes that a good delivery system is all that is necessary. (Personal note: I believe that the content will always have to be generated by the subject matter experts, in this case the IT service admins, and that there is no point in waiting on the delivery method for getting the content creation aspects in order).

Dan Cromer stated that the VP fund reserve is being eaten up by hurricane cleanup efforts. The majority of the ICC feels that such excuses are especially impertinent in this instance; we would expect that most, if not all funds for cleanup will be reimbursed by FEMA and should have no long term effect on overall funds available. Dan also mentioned that he sees IT as being in an unfavorable trade-off position with faculty positions in regard to funding. Mike Armstrong countered that the strong IT support offered by CREC has been instrumental in attracting quality faculty members. Mike finds that good IT support is rapidly becoming an important issue to prospective hires. Joe Hayden concurred with Mike. He sees an increasing demand for technology throughout IFAS facilties. Dan said that it is too early to judge the emphasis that the new IFAS administration will place on IT; we would be much more comfortable, however, if he spoke of his plans for focusing the new administration on what are deemed important IT needs for servicing IFAS faculty, staff and students. The ICC feels that IFAS IT requires and deserves leadership that speaks to how IT goals can and will be accomplished rather than continually to how goals must be postponed and forward movement delayed. The ICC unanimously desires IT leadership that openly and publicly speaks on IT's behalf and works to see that they have the support necessary to do the best job possible--funding considerations not withstanding. If the IT Charter is deemed a means to this, the ICC wants an IT administration that gets actively involved in the development and promotion of that. We do not want an IT administration that seems so willing at every turn to propose an absolute bare-bones budget that will do nothing but propagate the problems under which IFAS IT, and consequently our users, have suffered for many years.

Discussion then turned to the issue of maintenance notifications. Chris Leopold described the anatomy of a maintenance window as follows:

• What: Maintenance Window Announcements
• When: If service disruption is expected on the big five (File, Print, SQL, Web, or Exchange), the announcement will give targeted audience three days advanced notice.
• Regular: Every Sunday 6am - 10am and Tuesday 5am - 7am
• Emergency/User Requested: Monday - Saturday (Anytime)
• Client HF/Patch (to be discussed at ICC): Wednesday 6pm - Midnight
• Who:
• Outage 15min or less - ICC announcement by various staff members
• Outage more than 15min - IFAS-ALL accouncement by Dan Cromer
• How: Website, E-mail, and possibly by network broadcast messages

From: itnslan@ifas.ufl.edu
Subject: Server Maintenance - December 2 (9:35 am - 12:00 pm)
 
~ Message Body ~
 
For a more detailed description of this maintenance please
check the ITNS Website.
 
(points to something like http://www.it.ifas.ufl.edu/operations/plan-12-2.html
and contains a detailed plan as to the order of the systems being operated 
on (this forces the introduction of planning) and a brief reason as to why - 
Chris this is where a custom CMDB would be really nice - something to 
think about - *** this also includes a description posted from an online form 
that Kirby would have filled out explaining the activity - just a copy and 
paste after a quick phone call/email to confirm the plans if needed).
 
Servers Affected
**********************************
IF-FS-SRVTASTE
 
Maintenace Activities
***********************************
Custom
 
Comments/Questions?
***********************************
Email: itnslan@ifas.ufl.edu
Phone: 392-HELP

While this information that Chris presented pertained primarily and specifically to ICC announcements, a discussion ensued on the continuing issue of IFAS-ALL announcements. All agreed that such messages must be carefully considered and worded. The crux of the problem, however, seems to be that we don't currently have any good mechanism to more carefully target our user base. That is necessary if such messages are to be formulated in the most effective manner. Kevin Hill and Jack Kramer both mentioned that they often will target IFAS-ALL message follow-ups to specific portions of their users that may be affected, explaining the consequences in more detail. Mike Armstrong suggested that the ICC could perform this function on behalf of IT by filtering and passing along information that came as more technical announcements from IT to the ICC-L. This would miss a large portion of the user-base, however, that did not have their own IT staff. Mark Ross suggested that these could be handled by dividing that portion of the user-base among Helpdesk personnel. A more personal, one-on-one style relationship could then begin to be developed between each enduser and their respective support person. It was believed that, if aggressively pursued by the HelpDesk, such a plan could foster improved two-way communications that could reap great rewards in improving our notification process. Dan agreed to pursue implementation of such a scheme with Tom Hintz and the HelpDesk--as well as with the District Support staff. The ICC will continue to monitor progress on this issue and do its best to ensure that an implementation actually occurs.

The issue of unit IT contacts was also raised again, as it has been many times in the past. Dan feels that this is best handled via an IMM from administration--something that he believes must wait on the upcoming new administration, although Dan says that Joe Joyce has expressed IMM review and revision as a priority. It was pointed out that delegating the responsibility is not the same as seeing that the job gets accomplished. This has been a problem with the IFAS and UF directory liaisons as well.

Joe Hayden related that Facilities Planning and Operations had sucessfully implemented a contact scheme for each of their sites via a mandate from Mike Martin. Joe has a contact list for each site and that list goes through a series of contacts ending ultimately with the chair/director. Going the other way, each site has an 800 number sites can call to receive support. Joe said this scheme was working; though it had caused some heated discussion, it had been enforced by IFAS administration from above and was meeting its intended need. Chris Hughes was extremely interested as this is exactly what is needed by the AD project to support the remote DCs. Chris had been unable, in working either through or around Dan, to get these necessary contacts arranged. Dan did not offer any explanation as to what efforts he had made in this regard or how he had failed to discover that such a contact scheme was already in place for IFAS. Dan did, however, believe that putting IT under this same scheme should be no problem since it already exists. The ICC urges IT administration to proceed on that with all due haste and, again, will continue to monitor progress on this issue to ensure that an implementation actually occurs.

Note: streaming part one ended at this point. The remainder of the meeting is contained in part two.

Wrapping up what started as a discussion on maintenance window announcements, Steve suggested that Chris Leopold move forward as much as possible with what he had presented regarding announcements to the ICC-L. Chris agreed that he would do so. Tom Hintz did add the fact that he believed we should have an opt-in list for IT announcements, something that Steve has supported all along and has gotten approval for from ITPAC. Many oppose this because it is not a complete solution; Steve admits that, but feels it should still be considered as another useful component in an overall information delivery scheme.

Discussion then moved on to the long-standing issue of the need for an IFAS IT problem and solution tracking system. Dwight reported that Fran McDonnell from UF HelpDesk has agreed to a cooperative arrangement whereby IFAS would be integrated into their Remedy trouble ticket system. With this system the end user has the ability to either enter a trouble ticket themselves into the system via a web interface, or to contact the HelpDesk via e-mail or phone and have a ticket started on their behalf. The tickets could then be routed to the appropriate person for resolution. The UF HelpDesk has three levels of support. In this cooperative scheme, the IFAS HelpDesk would be tier-two support. The UF HelpDesk would handle all Gatorlink resets, UF portal questions, and other UF level service problems. IFAS-level issues would be routed to the IFAS HelpDesk as tier-two support. Support can use this system to e-mail end users and the message will contain the ticket information in the subject header. An email reply from the user goes back into the Remedy system and can track additional information concerning the problem.

There is client software (that has an associated cost) that can be used to access the system, but CIRCA has a programmer that has used PERL to build a web interface via APIs that Remedy exposes, which is generally used for access. Incorporation of IFAS into the Remedy system will require some programming of the web interface by CIRCA. They will be meeting with Dwight next week to see what changes need to be made and expect to be able to be able to accomplish that by the end of February. This would initially be without cost, though that might change at some future date. Reporting capabilities are built into the client software and you can also use Crystal Reports to build your own. They are willing to share with us the reports they have written and to allow us to modify reports to meet our specific needs. Mike Armstrong was concerned about the level of detail that an end user might see in following up on a problem they had submitted.

Chris Leopold inquired as to the status of CIRCA's unified HelpDesk initiative, which had apparently been languishing for some time. Chris Hughes said that Fran had indicated that this was continuing and an upcoming meeting was slated on the issue. Fran had invited Chris to participate. Tom Hintz was encouraged to contact Fran and become involved in that effort.

Moving on to the next agenda item, Chris Leopold reported that he had a list of 10 people who were supposedly qualified for the open IT/SA IIS/SQL support position. He has not examined those yet, however, but expects to have an update by next week.

At this point we moved onto AD matters. Kevin Hill gave a quick run-down of what had transpired at yesterday's ICC-AD subcommittee meeting. Kevin reported that we had gone over our migration and mop up plans for addressing the year-end migration completion date. He also noted that we had covered some concerns regarding administrative organization and funding of the UFAD project. The committee is concerned that UFAD be managed in a manner that can provide 24/7 support in a secure and stable UF-wide environment. Specifically, the committee drafted and delivered a letter to Marc Hoit, Interim Associate Provost for Information Technology. As follow-up to this Chris Hughes noted that the lack of UFAD funding and support is extremely critical and continues to cause grave problems. Chris relayed that Dr. Hoit would soon be meeting with Jimmy Cheek to specifically address these issues. Chris told Dan that we need to make sure that Dr. Cheek is informed of the UFAD support issue and its importance to IFAS IT. From Dan's lack of response, the ICC is very concerned that IT administration is not being proactive with the issue in seeing that proper steps are being taken by IFAS administration to address the issues. This is yet another in an increasingly long line of examples of how the ICC feels IT administration is not supporting IT requirements. Matters in which IT adminstration should be actively involved are again being left to the ICC for their own best attempts at resolution. IT administration continues to function primarily as roadblock between the ICC and ITPAC recommendation processes on one hand and IFAS administrative action on the other.

Chris Hughes went over again the important events that are going to occur with the migration over the next 10 days:

• Friday, December 10th at 5:30 PM: Another migration run will begin by rebooting machines, hitting them with the agent to migrate permissions, and finally using the netdom command to bring them into UFAD.
• Saturday, December 11th: Hopefully, another run.
• Sunday, December 12th: Possibly, another run.
• Monday, December 13th: By the start of work, Chris will get out a list of failed machines that have problems (turned off, firewalled, incorrect permissions, etc.) that will need to be resolved quickly--either manually migrated or touched by a support person. At 5:30 will be another run that is meant to get those remaining machines.
• Tuesday, December 14th: Same as Monday.
• Wednesday, December 15th: Same as Tuesday.
• Thursday, December 16th: A join command will be put in the IFASDOM login script (via NETDOM). Chris also plans to include in this script the addition of the IF-ADMINS groups to the Local Administrators group of the machine being migrated (great work there!).
• Friday, December 17th: Same as Thursday.
• Saturday, December 18th: Will poke at problem machines that are still there to see if there is anything that can be done to get them to move over.
• Sunday, December 19th: Same as Saturday.
• Monday, December 20th: The major ITNS servers will be migrated-NT-File, webservers and SQL servers. Continue with manual migrations.
• Tuesday, December 21th: Test shutoff of IFASDOM 4am-7am to see what breaks. After that we will just continue to keep turning IFASDOM on and off until all problems are resolved.

Chris Hughes wanted to mention that all IFASDOM BDC's really, really, really need to be kept turned off. Additionally, Chris is concerned about four 56k sites that are continuing to be a problem accessing well enough to do the necessary copying.

Steve Lasley raised concern to Dan that there were AD-related items which the ICC-AD subcommittee believed had been budgeted, but for which the funds are now not there. The ICC-AD committee had the following exchange with Dr. Joyce which led to our understanding that $90k would be allocated from salary savings to cover AD costs each fiscal year continuing onwards, as per our budget projection. It should be mentioned that our subcommittee had felt the need to reach an agreement directly with Dr. Joyce on this because all efforts at getting IT administration to work on our behalf in the matters of budgeting had proved futile.

-----Original Message-----
From: Joyce, Joe C. 
Sent: Friday, March 26, 2004 2:45 PM
To: Hill, Kevin U.
Subject: RE: IFAS AD Project budget meeting summary

Kevin:  Pretty close but remember I can't make that kind of additional 
commitment without ok by VP and Deans.  Money Dan has in his pocket 
is committed to this, it's the extra I've got to navigate.  Also I'm not sure 
about the charge back part yet.

Thanks for hanging with this.

Joe

-----Original Message-----
From: Hill, Kevin U. 
Sent: Friday, March 26, 2004 2:34 PM
To: Joyce, Joe C.
Cc: Cromer, Daniel H; Leopold, Christian R.; Hughes, Chris J.
Subject: IFAS AD Project budget meeting summary


Dr. Joyce,

Thank you for meeting with us this morning to firm up funding details for 
the IFAS migration to Windows 2003 Active Directory. Because nearly 
a year has elapsed since our last budget meeting, our purpose was to 
re-establish the nature and extent of financial commitment to the project, 
and to disclose changes in the request that have evolved since the last 
approval.

As a follow-up, I've assembled a brief summary of key statements and 
observations that you made during the meeting. Please let me know if 
I misinterpreted or missed an important point.

- The Active Directory project will be funded including remote sites.
- $50K of current FY IFAS-IT budget are available to fund immediate AD 
   project costs (present - 31Jun04)
- $90K (IT salary savings) will be allocated to fund FY2004 AD project 
   costs (Jul04 - ongoing)
- IT should pursue a lease option for remote hardware to spread costs 
   over 3 years. 
- Remote units (RECs & CEOs) will support hardware lease costs via 
   an annual chargeback.
- Core IT budget needs are recognized as separate and ongoing and 
   will be addressed.
- The AD Committee should make immediate plans to present a project 
   overview to Department Heads.

Thanks again for your support of the project. We look forward to 
implementing the plan to update the IFAS computing environment.

- Kevin

__________________________________________
Kevin U. Hill, MCSE-W2K
Coordinator, Computer Applications
University of Florida / IFAS - SWFREC
2686 State Road 29 North
Immokalee, FL 34142
PH 941-658-3439 | SC 974-3439 |
FAX 941-658-3469
------------------------------------------ 

Dan responded that we needed to realize that there was no budget for AD and that all monies go into one big IT pot. He further stated that he did not get the salary savings as indicated. Although Pete Kearny's salary of $72k was given and spent, the additional monies were expected to come from the benefits portion allocated to his position. Those were not given as expected and thus the shortage. Dan also indicated that there were unexpected overages on the DE budget that left IFAS IT $30k in the hole on those. Dan plans on speaking to Dr. Joyce soon about these matters, but seemed to already have given up on fighting for the full $90k allocation as previously budgeted. This is particularly disappointing because, unlike many projects (DE included), the AD project has been run strictly within the proposed budget and is (over great hardships) being kept on schedule as well.

Steve continued by referring to an earlier comment, that Dan did not want to go into future projects (like content management or SharePoint) without assuring that sufficient resources were allocated for those to be successful. The AD project had been planned extensively over a three year process. We spent a long time trying to assure that we had the resources to manage this project and, quite frankly had thought we did. Now we find out this was not the case. We could have planned differently, had we known or been kept informed of budget status details, but current IT administration seems to only provide feedback on those when specifically pressed. While we understand that ERP had delayed the ability to judge current fund status, Chris Hughes had kept careful track of the AD expenditures and we were confident that this would not be an issue. What came as a complete surprise to us was that the funds had not been allocated as we had been led to believe. The committee was in no way advised as to the details of this matter until this very meeting--and yet it was mentioned at this meeting that this knowledge had been available since as early as August.

Chris Hughes brought up another expense that we will have to cover--namely CALS for Windows and Exchange (and possibley Office) licenses for IFAS employees using county-owned machines. This necessary because the UF Microsoft Campus Agreement did not cover all employees as was originally believed.

John Sawyer was not available to give a security status report, but Chris Leopold spoke briefly to the new understanding that has been reached by CNS with regard to the wallplate initiative. Most of the issues have been resolved by the new CNS policy on what is now called "shared access" rather than "shared management". CNS had made a policy that said they considered VLANs (802.1Q trunking) to be a core service; hence, sites that did not have a CNS managed device at the B-POP could not take advantage of such services. (The most recent DDD memo in this regard can be found here.) This was initially viewed as a way of forcing units into the wallplate initiative, as IFAS has about 70 buildings on campus that do not have CNS supplied Cisco equipment at their B-POPs. It has since been arranged that the definition of a B-POP in this regard will be more flexibly construed to mean primary switches and not "workgroup" switches--a classification into which many of the IFAS building would fall.

Although nobody is really arguing that the wallplate initiative couldn't lead to improved networking service, the question of funding remains to be resolved. The current costs were based on what it actually cost the College of Business to implement that scheme. The problem remains that we can't ask IFAS units to pay for this service without them being given funds to do so; many units, if not all, can not afford that.

The ICC then briefly discussed office media distribution and network Administrative vs. Shared CD and update points. John Sawyer was not available to discuss the various installation point options and time considerations required moving on quickly from this topic in order to complete our agenda.

Dwight Jesseman spoke briefly about his implementation of changes to the Exchange environment as discussed at the last ICC-AD meeting (changes to GroupShield and distribution list naming conventions, for example, have been done). Dwight is currently in the process of creating additional stores in the storage groups that exist for various levels of quota implementation. These stores and groups will now have an alphabetical naming convention. Dwight sized the quotas to give us some breathing room, but after some time, we can expect users to begin reporting that they are getting quota exceeded messages. At 80% utilization a warning notice is sent, at 90% another notice is sent that mail receipt will stop, and finally at 100% of quota utilization the user is able to neither send nor receive. Whether such persons can be granted more space vs. being required to clean out their accounts depends on their role, but IT remains committed to providing storage space proportional to the actual work needs of its users.

Dwight then spoke on backup options with IT/SA. There are currently four individuals for whom IT is supplying backup services: Wayne Hyde, Steve Lasley, Richard Phelan, and Mark Ross. This service is being supplied in exchange for those people providing an agent license for Veritas Backup Exec (item #563084 via CDWG for $145.00) and media (Data Cartridge for LTO-2 Tape Drives - 5-Pack via Dell for $228.36). Dwight will arrange a meeting soon with those involved to discuss managment access.

Jack Kramer asked about backups on the multi-purpose servers. This has been implemented locally via mirroring for the 40GB available on each and some of that information is pulled off to Gainesville on a regular basis. We would have the capability of pulling off data remotely in the case of an impending disaster such as an anticipated hurricane hit.

Dwight also mentioned that Report of Accomplishments (ROAs) are looming, so he revisited the availability of the webstats site. Registration is required there in order to provide Dwight with an email contact point to those using that service. The credentials are documented within the secure portion of the ICC website (IF-ADMN credentials required for access). Dwight asked that we all make our faculty aware of this service as it will assist a number of them in their ROA preparations.

Mike Armstrong intends to place CREC logs on a share off the multi-purpose server so that Dwight may access those.

The meeting ran late and was adjourned at roughly 12:10pm after which a number of the ICC members (9 actually) attended a holiday dining experience at the Olive Garden thanks to Dennis Brown's coordination efforts. Great fun was had by all and we hope to hold additional after-meeting dine-outs in the future.