IFAS COMPUTER COORDINATORS
(ICC)
NOTES FROM November 9th 2007 REGULAR MEETING
A meeting of the ICC was held on Friday, November 9th, 2007. We were back in our old haunts, the ICS conference room, which had been unavailable the past two months due to renovation work. The meeting was chaired and called to order by Steve Lasley on time at 10:00 am.
PRESENT: Seventeen members participated.
Remote participants: Bill Black, Chris Fooshee, Chris Leopold, Joel Parlin, Mike Ryabin, Louise Ryan and D. A. Walker.
On-site participants:
David Baudree, Dennis Brown, Dan Cromer, Andrew Carey, Wayne Hyde, Winnie Lante, Steve Lasley, Ligia Ortega, Mark Ross and Wendy Williams.
STREAMING AUDIO: available here
NOTES:
Agendas were distributed, and the meeting began roughly on time at about 10:01am. The ICS conference room lost power about 21 minutes into the meeting and this caused our connection to the remote participants to be disconnected for roughly 20 minutes or so.
Remote participation trials:
Steve would like to thank Lance Cozart for his help in setting up the Polycom system for our use today. Lance is redoing the entire configuration of the ICS conference room and has installed power connections, network drops and computer-to-Polycom/projector connections directly into the conference table top. The new system includes an equipment rack in the back corner of the room which houses the Polycom, a WAP, a VGA matrix switch, and a laptop with wireless keyboard and mouse. He also plans an easy-to-use switching system that will allow users to control the various pieces by function desired rather than needing to understand the complex switching setup. Since that menu system is not yet in place, Lance kindly provided Steve a tutorial on manual setup and was on-site prior to the meeting to assist in getting us going as well.
Except for the loss of power, remote participation went well, with Steve using People+Content to alternate between showing his desktop and showing the local camera. Steve would appreciate hearing back on how best to orchestrate the switching. Would it be best for us to generally stay on content until a question is asked from a remote site?
New members:
Steve noted that Marten Gehman has joined AEC as OPS Computer Support. Rick Noble will be resigning his IT support position there at the end of the year.
Recap since last meeting:
As per his usual procedure, Steve pointed folks to the notes of the last meeting, without going into any details.
Policy
Next ITPAC meeting is November 14th
Steve noted that the November ITPAC meeting is next Wednesday. There are several items which the ICC will be raising there amongst what promises to be a crowded agenda.
E-mail alias removal
Based on discussions which we had over the ICC-L and at our last meeting, Steve had drafted a proposed “recommendations for removing IFAS e-mail aliases” for consideration by the ICC. After going over that briefly, Steve requested feedback from the other members.
Dennis Brown wondered how difficult it would be to implement a method for allowing people to view and then opt-out of certain of their aliases. Essentially, this can be handled manually and, in fact, has been done for some folks. Obviously, anyone can use Outlook to view their aliases:
and Scott Owens can go in manually and remove extra aliases on request.
Dan Cromer mentioned that this could also be automated somewhat if that was deemed appropriate, though the programming of a secure web application for that might be a bit tricky. Steve pointed out that implementation details were intentionally left out of the recommendation due to the fact that we were merely seeking policy at this time.
In absence of dissent, it was decided that the ICC would approve these recommendations as written and that Steve would present them to ITPAC on the 14th.
UF-IFAS Web Policy and UF-IFAS Domain Name Policy drafts for review and recommendation
Steve noted and Ligia Ortega confirmed that there have been no changes to these drafts since our last meeting and that no one had contacted either Ligia or Diana Hagan with any changes. Apparently, ICCers are happy with the policy documents as written. Ligia and Diana will be on hand at ITPAC to present these proposals for consideration by IFAS administration.
UF IT Advisory Committee for Network Infrastructure meeting
Steve gave a brief rundown from yesterday's ITAC-NI meeting. You are encouraged to check the minutes for the complete details. The agenda covered the following items:
- SSRB going off-line
As Mike Kanofsky put it to the CCC on Wednesday, "Who would have figured that with dual network connections, dual switches, dual power supplies, and dual core routers that everything could be brought down by a copier!". Details of what happened and how to prevent reoccurrences were discussed.
- UFL.EDU to UF.EDU branding project - Christine Schoaff
Everyone has no doubt noticed the heavy discussion of this issue on the CCC list which Allen Rout began yesterday. Most everyone in the IT community seems to feel that this proposal has very few positives and a huge (nearly inestimable) number of downsides.
- Update on Wall-Plate roll out progress and schedule - Todd Hester
Steve noted that the wall-plate rollout is behind schedule. Although they hope to get back on track, Steve wonders if more resources will not be required in the short-term in order to meet the long term goal of a three-year replacement cycle on our network equipment. Already, the number of units opting for VoIP and the number of ports required overall have increased greatly beyond prior expectations.
- Videoconferencing of the meetings
Chris Leopold asked if any decision was made on whether to continue videoconferencing for these meetings. Steve said that this was the last topic of discussion and that, due to lack of objection, it seemed that we would continue to do so. The live stream was supposed to be announced to the Net-Managers list, but that did not happen this month. Steve will try and see that this announcement is made in December in case folks want to tune in.
Steve noted that audio from this ITAC-NI meeting is now available.
November 8th UFAD meeting
Steve mentioned that he also recorded audio from recorded yesterday's UFAD meeting. The audio quality from that was none-too-good, but it was a very useful and informative meeting, and you might find that worth listening to--particularly for Mike Conlon's "State of UFAD" message and Dwight Jesseman's description of the UF Exchange service.
One subject which we might want to discuss at the December ICC meeting is the plan to have a 365 day folder retention period for default folders. There will be a "Managed Folder" for each user where they may move messages in order to avoid this automatic deletion, and sub-folders may be created within that as desired. Hopefully, Dwight can be available for discussion.
Dwight is still working on the amazingly detailed design documentation, but he kindly provided a draft project design document in Visio format (ufad\if-admn credentials required). A free Visio viewer is available for download. Please realize that this is still a work in progress.
IT Governance sub-committee status report
Steve noted that UF now has appointed a CFO, who will be starting with us at the beginning of the new year. Perhaps this will get us one step closer to settling the CIO issue and getting on with IT restructuring plans.
UF Calendar Project
Prior UF Calendar Project discussion. There is no news to report on this matter. As mentioned the last several meetings, Steve believes the project is basically awaiting the needed commitment of resources; whether that will materialize is not known.
Extension Calendar
Apparently, Chris Fooshee will be giving a demo of Meeting Room Manager at next week's ITPAC meeting. If you can't make it to the meeting, the notes should eventually provide details for the curious.
The UF Exchange Project
As mentioned above, Steve hopes to have Dwight Jesseman available to give us an update on this at the next ICC meeting.
The Wallplate Project
Steve is still awaiting the proper time and method to use in addressing his own department's cost concerns regarding the wallplate project. At yesterday's ITAC-NI meeting, he did mention to Todd Hester that many departments might appreciate a much greater lead time between first contact and wall-plate implementation. Steve feels that the considerable costs involved for VoIP handsets and in remedial wiring needs make it important to inform units ASAP about what their potential costs might be.
Recommendation: autogroups for *selected* roles
This is generally not discussed (and wasn't again this month) but is being kept on the agenda for future consideration. Basic role autogroups are now in place within UFAD.
Split DNS solution for UFAD problems
Steve mentioned that he had thought of raising this matter at yesterday's UFAD meeting, but felt it might be too off-topic for most. As important as this matter seems to Steve, UFAD certainly has their hands full with other projects. We might have to bide our time on this.
As a standing issue, Steve would like to remind folks that he is investigating alternate solutions for the split DNS issues. Initial tests look promising.
If anyone else would like to help test this, please get with Steve. Since a split-DNS solution is not expected to happen any time soon, this might really help out with our user experience in the meantime.
Projects
SharePoint Test Site
Prior SharePoint discussion. Ben Beach was unavailable, but Steve asked if anyone knew of any progress with this project. Dan Cromer had Steve go to http://my.ifas.ufl.edu to see if it was up and running. Multiple authentications were necessary; according to Mark Ross, this is due to the multiple sources which Ben in pulling info from for this site. In any case, nobody really had any news to report at this time. Steve noted that this is really a huge project and that Ben obviously has many other duties which make it all the more difficult for him to proceed quickly on this.
Virtualization of Core Services
Steve asked Wayne Hyde to tell the group about all the new hardware for the SAN and virtualization of the majority of our services.
Testing the new equipment
Wayne reported that we now have the new Dell/EMC CX3-40c with iSCSI and FC support
SAN installed along with the new Dell PowerEdge 2950 Servers for the ESX virtual server cluster. Wayne is currently doing a bunch of disk I/O and CPU tests, "basically trying to pound it into the dirt and make it cry". Wayne reports that the system is more than up to the stress and has been "laughing" at his puny attempts to bring it to its knees.
Wayne expects to do another couple weeks of testing before proceeding. The first set of services which he needs to get off the old server cluster is the file server. That is going to take a bit of work, however, because they are still trying to decide if they want a clustered file server (active/passive or active/active) and how they are going to do backups. That is a bit of work, plus our file server is up for replacement in six months and we are evaluating whether to go to Windows Server 2008 64 bit -- the whole nine yards. We need to decide if we want to stay with Veritas for backups or use Microsoft DPM.
Upcoming advantages
After we move around some file server data, we can migrate the VMs off the current virtual server onto the new cluster. Then we can reformat ESX01 and add it to the cluster. That will give us quite a bit of horsepower and capability for the VMs that we have. The cluster will give us four host servers running all our virtual servers. If one host goes down, everything will fail-over to the other three. Even if two go down at the same time, we will still have the capacity to run all the critical VMs.
Maintenance is going to be great. If you need to do maintenance on one of the hosts, you place it into maintenance mode and it migrates all the VMs using "VMotion". The VMs continue to run while they are being moved over to the new host. Once all the VMs are off that physical host, you can patch it or whatever is needed and then bring it back up. It has distributed resource scheduling so it will even out the load. You could patch all the host servers in this fashion one-by-one. You also can add additional capacity to your cluster without affecting your running VMs (guests); this makes things very flexible. This sort of maintenance relates to the either the hardware or ESX itself; Windows server patching (of VMs) will still require rebooting (virtual) services, however.
Of course the SAN will give us an enormous amount of disk I/O bandwidth to play with, which is very nice as well. That is going to make life a lot easier for the IT/SA group.
IF-SRV-FILE03 will be used for volume shadow copy space and we need to figure out how best to handle that. There are just a lot of issues to be considered before arriving at the best way to provide highly-available services for IFAS. Since IF-SRV-FILE02 is due for replacement in May, we are going to try to replace it with a clustered file server running Windows Server 2008. There are many options to evaluate. We can use Volume Shadow Copy and storage space on the SAN or use DPM to make replicas of the data via volume shadow snapshots so users will still be able to restore their own deleted files.
The switches in the server room are now configured in a matrix in an attempt to make everything redundant. Critical servers will have two NICs each and those will be plugged into two different switches (one active, one passive) so if one switch goes down, the servers will still have connectivity. If the main backbone switch which feeds all the other switches goes down, then we will obviously have an outage--at least from the outside world. The goal is to get everything redundant and highly available.
Steve mentioned that Mike Conlon had talked about redundancy with regards to our ERP systems. If something took out the machine room on the East Campus, he estimated that we would be facing at least a two month re-build time. The data would be safe, as it is backed-up, but the systems themselves would be enormously laborious to replace and reconfigure. Wayne noted that our new SAN configuration has two storage processors and fiber connections to two different fiber switches. Wayne was surprised to find that the ERP SAN is only on a single fiber switch--they only have a single fabric. Consequently, if they lost their fabric the entire enterprise data network would be down.
Go Live Date?
Dan Cromer asked about when Wayne was going to certify the new hardware "production ready"; he wanted an estimate of how long before this is all in actual use. Wayne said that good estimates were difficult to make. We just got the licenses for the backup software yesterday. We just received the tape upgrades for one of our tape libraries yesterday as well. The new tapes will be put in on Tuesday and Wayne still needs to test the new backup software. The ESX cluster and the SAN have passed all Wayne's tests over the past two weeks with flying colors. The next big step is to migrate things off IF-SRV-FILE03, because that system has the PowerVault MD1000 which Wayne needs to connect to the backup server for VM backups (among other things). Consequently, there are several steps Wayne needs to take in order to get everything going. The first step is to migrate the ICS folks off of IF-SRV-ICSFS01 onto IF-SRV-FILE02 and use the ICS file server as IF-SRV-FILE01. Then he needs to migrate the data off of IF-SRV-FILE03 onto IF-SRV-FILE01. Once that is stable then everything else will fall into place as fast as Wayne can move VMs off the old server onto the new cluster. At that point Mark Ross will be happy because Wayne will be able to create a bunch of VMs for him.
Dan again tried to get a specific date from Wayne, but Wayne mentioned some other difficult to estimate issues, including the restructuring of shares on the ICS file server. Apparently, that is currently quite a mess and will need a bit of work to get into reasonable order. Numerous shares will have to be collapsed into one data share. The other issue is that ICS has a bunch of Macs using those shares--which may have something to do with how loosely permissioned some of those are currently.
Wayne is still evaluating what sort of RAID groups work best and how much I/O we need for the storage used by our various services. For the file server, he has pretty much decided to use either 1 or 2 TB storage groups; this will be an improvement (for maintenance and backup) to our current monolithic 7 TB volume configuration there. We may still have quotas to prevent excessive use, but our storage capacity will definitely be improved. We will have 60 fiber channel drives. Wayne will likely wipe out the test configuration and start from scratch, because the Dell folks didn't exactly follow EMC best practices in the original configuration.
Macintosh Fileserver Access Issues
The other issue with our file server is that Macs cannot access DFS. There are a couple of possible solutions there. The first is to buy Thursby's ADmitMac for about $100 per client. This will let them see DFS, though Mark Ross mentioned seeing numerous issues within their support forums. Without that Macs can still connect via SMB directly to the fileserver share (which will break when things get moved), but using DFS will not work apparently.
Mark said that WebDAV is another option, but one has to be careful to map into the structure directly, because mapping to the root does not work due to the vast number of resources for which access rights must be enumerated. Wayne mentioned that if we move to an active/passive file server structure we can use the cluster name to provide some flexibility for the Macs; then the cluster will take care of any necessary rerouting to the physical servers themselves. That would be similar to DFS, but using straight SMB. If we go with active/active (which means we have two file servers which split the load) then, if one of the servers fails, the other server manages the disks and they are brought back up on that remaining server. Under that configuration, you still have a cluster name, but you also have virtual names for each of the two file servers which can remain static regardless of changes to the actual machine names used. That will allow some flexibility under that configuration as well, should we decide to go that route.
Steve mentioned his surprise that DFS had such client-side issues, as he figured all aspects of DFS were handled by the server side. Mark replied that DFS was basically a server-side redirect of SMB, but Apple clients still can't handle that as of now. Wayne said that Samba is a bit behind the Windows client in this aspect and speculated that Apple didn't want to roll this support in directly because they didn't want either to "kill off" Thursby or to buy them out--similar to Microsoft's prior stance on anti-virus software for Windows.
In summary, Wayne said that Mac clients could either buy the Thursby client or, hopefully, our cluster will take care of most issues regarding named access to our file server from the Mac. In the latter case, and if we go with an active/active configuration, Macs would still have to know which of the two file servers they were using. Windows users should still be set to go to the \\ad.ufl.edu\ifas DFS share for the flexibility that provides.
Improvements visible to our users
Steve asked whether our users would notice any of these improvements. Wayne mentioned that restoration of Volume Shadow Copy (ufad\if-admn credentials required), which will allow users to recover deleted or altered files without support staff interventions, is likely the major noticeable change. The rest of the advantages, from the user standpoint, will simply be better quality of service with less downtime for maintenance and when certain problems occur. The goal is five nines or as close to that as we can.
IFAS WebDAV implementation
Still no movement has occurred in getting this documented.
Vista TAP and Vista Deployment via SMS and WDS
This topic was not covered, although it will eventually need to be addressed. Currently, most folks are waiting for application and hardware support on Vista to improve.
Re-enabling the Windows firewall
This topic was also skipped, but will remain on our project list for eventual consideration.
Exit processes, NMB and permission removal
Prior exit procedure discussion. We skipped this item for now, since there was no news to relate. Steve wants to keep this on our radar however.
Operations
ePO version 4 is awaiting deployment
ePO version 4.0 is awaiting preparation of the new ESX cluster prior to deployment. This will eventually provide a noticeable improvement for OU Admins, however.
Polycom: private IPs, maintenance and contingency planning
Steve asked about how the Polycom deployment was going and whether DHCP was generally being used for configuring IP addresses for those. Dan feels that it is going pretty well considering the large number of devices we are deploying and the wide variety of networking configurations we must deal with at the various county offices.
Regarding DHCP, Dan said that it was being used in most cases. At locations where a particular public IP address is supplied by the telephone company for access to a single device, Dan feels it a "no-brainer" to apply a static address on the Polycom unit itself. Dan also mentioned that there were various problems with individual counties; for example, Orange county wanted to be supplied with a list of all the IP addresses which might want to access the Polycom there--presumably for firewall purposes. Obviously, that is not reasonable to do and we need to negotiate around that problem.
Wayne mentioned that Polycom has Global Management System software that might be useful for managing our numerous Polycom units, though he wasn't sure of the costs involved. Dan mentioned that once these are set up and are configured properly, they should keep on working (assuming they are not changed). Dan also mentioned that there is a web interface on each of the Polycoms for making system changes remotely as well. As long as a configuration change doesn't render a system unreachable remotely, that method should permit handling many issues which arise.
Mark Ross expressed his concern that too many folks know the system passwords on these devices; centralized management would make things more reliable. Dan responded that IFAS has a distributed and diversified support system where there are individuals at all sites which can handle some of these matters. Steve mentioned that this distributed system works well when close contact is kept with the central folks and those working relationships are well maintained.
Dan expressed his appreciation to the district support folks for the good job they are doing with this rollout. Steve said that it had been mentioned prior that deployment would be starting in the north and moving down throughout the state. Dan replied that Pete Vergot and Mark Rieger, along with himself, were the three who basically decided on what equipment should be purchased with the ~$800K which IFAS had available to spend for this project. Pete was very anxious to get the equipment deployed in his NW district, and he supported Louise Ryan in making that district a "guinea pig" for working out many of the deployment details. Hopefully, the knowledge learned there can be applied to assist with the other deployments throughout the state. One of the things they are doing is putting in higher speed DSL circuits at various locations where that might not have been available prior.
Louise Ryan reported that she has about four Polycom units left to deploy--those are at locations where they needed to change the telephone service. A. D. Walker mentioned that Live Oak is still pending as well; they are waiting on a monitor to use with the device in the farm conference room there. Bill Black reported that his had just started with his first unit yesterday in DeSoto county. Bill had picked his units up just Tuesday. He expects some challenges with regards to connectivity as some of the DSL links are not up to par.
Note: this is roughly the time at which power was restored to ICS and we got back on-line with the Polycom.
Patching updates
November is going to be a light month with regard to MS patches. There will be only two critical patches for Windows and, apparently, none for Office.
Wayne reported that there is an "AntiPiracy Macrovision Bug" which will affect all our Windows platforms. There is not really much we can do about this one. Note that Apple Quicktime is now up to version 7.3, with earlier versions having security issues. Similarly, Adobe Reader is up to 8.1.1. Real Player has issues, though the Enterprise version has fewer extras than the consumer version and is much less open to exploit. Third-party apps are now perhaps our biggest patching pain due to the current lack of centralized methods for that.
Office 2007 issues update
There was no discussion on this matter during the current meeting.
Job Matrix Update status
This matter was not discussed.
Admin Helper Script and IE7 update
This item is being left on the agendas so it remains on our radar. Steve suspects that this issue will come back to the forefront as Vista deployment proceeds.
Remedy system status
We again skipped discussion of the woes with this system. Again, Steve feels that the basic problems could be addressed and resolved for the most part if a person was assigned to following through with that.
Other news
VoIP options at remote locations
Mike Ryabin asked about VoIP options at Ft. Lauderdale. Chris Leopold felt that the options there were dependent on having a high-speed link to campus. Dan said that Ft. Lauderdale is not in the plan for a high-speed link due to funding issues. Apopka, Balm, Lake Alfred and Plant City are getting or have gotten their highs-peed links. Dan has applied for a high-speed link for Fisheries and Aquatic Sciences here in town as well. He is also asking for funds to connect Ft. Pierce. Dan believes that Ft. Lauderdale and Jay will be next on the list, but funding is very tight right now. Dan mentioned receiving a quote of $36K/yr for Immokalee, and nobody seems to have money they wish to apply to that.
Data protection at remote sites
Wayne mentioned that we have data protection issues at remote sites. Currently on our multi-purpose servers, we backup the local data drive to a separate volume on that same machine locally. With the faster sites, Wayne would like to use DFS replication to pull that data centrally. DFS can be configured to limit bandwidth needs, and using that could improve data backup for many of our remote file servers. The slow sites should look at getting an external NAS device to take home or whatever; that might provide minimal protection for those.
Collapsing our various web site domains
Mark mentioned that he is busy locating various web sites which potentially can be reorganized under existing departmental sites. Steve noted that Entomology is one of the units that has many individual sites which need to be moved. Steve mentioned that this fragmentation was due to the way in which new site requests have traditionally been advertised. Mark is working on a new request page which will help with that and Steve is learning the details of what is involved with transitioning so he can work with the various site owners and get their cooperation for doing that. "Permanent" (status code 301) redirects will be necessary and some folks will resist the necessary URL changes. Mark mentioned that, for Steve, he has pulled off FPSE and now Steve can control access to subfolders within the Entomology site just as he would do on the file server. This offloads the support capability to the unit and relieves Mark of the day-to-day access maintenance. So far it looks like a great solution. Steve likes it because he can still blame Mark if there is a problem :-), but can fix most things himself.
The meeting was adjourned a quite early, at about 11:10.
|
