On Friday afternoon, ShootQ went down unexpectedly. The service is back up and running and we advise our users to read this announcement for further explanation and instructions.
What happened?
Our system administration team was alerted to an imminent drive failure. To initiate the repair, the ShootQ team activated our backup system. At 1:30 p.m. PT, while the repair of our primary system was underway, our backup system simultaneously failed, causing the system outage. The likelihood of both our primary and backup systems failing simultaneously was low, but unfortunately, it did occur.
To restore service as fast as possible, we performed a recovery from our secondary backup, which was made at 1:30 am PT July 18, 2014. We were able to restore functionality by 1:00 am PT July 19, 2014 but the data was reverted to the state at the time of the backup. This means any data that was entered/created between 1:30 am and approximately 1:30 pm PT on July 18 has been irrevocably lost. The fact that this has happened is very upsetting to us, and we understand and sincerely apologize for the frustration and inconvenience this will cause to our users.
What are next steps?
We recommend you review any actions/updates/changes that you did on ShootQ between 1:30 am - 1:30 pm PT on Friday, July 18, 2014.
Here are steps to resolve any changes made to the following:
- A client booked and signed a contract (the client will need to re-sign the contract)
- Your studio signed a contract (you will need to re-sign the contract)
- A proposal/order was sent to the client to sign (it will need to be re-sent)
- A client paid online or you did a manual payment (you'll need to record a manual payment)
- You received a new lead (copy/paste the email notification using Lead Via Email function under the Shoots tab or enter the shoot manually)
- A new shoot was created (the shoot will have to be re-created)
Information that will need to be re-entered into the system may include (but not limited to) the following:
- Any information that was updated on a shoot (relationships, shoot location, remarks, etc)
- Events/Appointments that were added to a shoot
- Any change to the shoot’s type (i.e. changed from Portrait to Boudoir)
- Any email that was archived with the shoot
- Any contact information that was entered or modified
- Any files uploaded to a shoot
- A workflow that was assigned to a shoot (or standalone tasks that were created)
- Workflow dates that were modified
- Workflows that were put on hold/resumed
- A lead status that was changed
- A lead who was marked as dead, deleted, or shared via QClub
- Any job that was Canceled or Marked Completed
- ProSelect or Redcart invoices that were imported
- Membership plan or account info that was updated
- Any Setting under the Settings tab that was modified/changed/created
- Calendars that were initially synced to the ShootQ calendar
- A questionnaire was sent or responded to (please send the questionnaire again)
- Emails that were sent from the shoot will already have been sent and delivered. However, the sent email will not be logged with the shoot.
- New shoots synced with Pictage may be broken and will have to be removed on Pictage and re-synced on ShootQ.
We will be monitoring P3 payments that were scheduled to be run on July 18, 2014 to ensure that any payments are properly collected and recorded. You can verify yourself by looking up the payment on your Pictage account statement.
If you purchased marketplace items, please contact us to have them re-installed.
We will continue to conduct a thorough investigation into the root cause of the failure and will implement corrective actions to reduce or eliminate the likelihood of such an occurrence in the future. Again, we apologize and if there is anything we can do to help please contact us via chat messenger or submit a support ticket. We're available Monday-Friday, 8am to 5pm Pacific Time.
3 Comments