You are here: Home / Blog / posts / Pixelworkers of CAES DO / Moving servers and the lessons we learned

Moving servers and the lessons we learned

by randles — published Jan 13, 2011 01:40 PM, last modified Jan 28, 2014 02:54 PM

We have one final web server running on the "bare" hardware, the rest are hyper-v VMs and for months we have wanted to visualize it.  Several weeks ago I outlined a plan for how to do this, tested it and then had Trish test it.  Each time we discovered another step, fine tuned our troubleshooting skills, and work out all of Plone and Zope quirks.  Armed with our proven step by step guide we decided that the change over was going to happen this Wednesday 1/12/11 at 3:00PM and this was our plan:

  1. Double check and repeat migration instructions
  2. Email the hostclerk (they run the schools DNS entries)
  3. Email every web site editor for the effected sites
  4. Post the outage on computing.caes.ucdavis.edu
  5. Coordinate with our friends at the Held Desk / Infrastructure who keep us up and running
  6. Blocked off 24 hours in case anything goes wrong

We forgot to add one thing on the list:Plone is not dependable or consistent

  1. permissions, zeoserver just did not want to stay up and running. This was in the error logs
    2011-01-03T02:10:13 INFO ZEO.runzeo (43359) created PID file '/usr/local/Plone/zeocluster/server/var/ZEO.pid'
    ------
    2011-01-03T02:10:13 INFO ZEO.runzeo (43359) opening storage '1' using FileStorage
    ------
    2011-01-03T02:10:13 INFO ZEO.runzeo (43359) removed PID file '/usr/local/Plone/zeocluster/server/var/ZEO.pid'
    ------
    2011-01-03T02:10:13 INFO root sleep 8 to avoid rapid restarts
    ------
    2011-01-03T02:10:13 INFO root pid 43359: exit status 1
    ------
    2011-01-03T02:10:21 INFO root spawned process pid=43360
    (To see this error message i used tail /usr/local/Plone/zeocluster/client1/log/Z2.log)
    In the end it was a BSD permissions issue fixed by a chown -R plone:plone * when in the /usr/local/Plone folder
  2. Cleanup: In the same directory as our Data.fs lives several other files (namely Data.fs.index, Data.fs.lock, Data.fs.temp) all of which needed to be deleted before Plone and Zope can start
  3. To fast!: With all of our practice and clear off the old sites we actually "completed" (see bellow) the migration in half the time we had expected it to take. Ordinarily this does not sound like a problem except in our case the hostclerk was not going to update the DNS entries to point to our new site until the following day and we could not request it to be done that day because it was after 5:00. Trish came to the rescue and disabled the /login_form on the old server so we could bring up both without fearing our users could edit the old site. 
  4. Kupu: Today after congratulating ourselves on a speedy job well done we received an email stating that our users could not edit the body text of the website.  After some frantic googling, checking multiple platforms, every plone site, redoing the java script registries, cleaning all of the caches and still no luck, we fixed the problem by reinstalling kupu from the zmi in portal_quickinstaller for each site

In the end it was not too painful, we are no longer dependent on old hardware and we learned about Plone along the way

When:

Where:

Contact