VIP Appliance Appears to be Locked Up

Owned by Anand Rajaraman
Last updated: Dec 02, 2019
3 min read

Symptoms

  • User reports that system is unavailable remotely

  • When user looks at physical SpotMonitor or has monitor, keyboard, and mouse connected to the VIP Appliance, the screen appears to be frozen one on image (if SpotMonitor is up) or shows the login screen but they can't move the mouse or type in the fields

  • The screen may also appear blank, or stuck on a screen that says "Performing System Update"

  • The power light is on and the fans are running, but there's no hard drive light activity

Solution

  1. Try restarting the system by pushing the power button, waiting 30 seconds, then pushing it again.

  2. Does the system come back up to the Sign In screen (with local monitor), or is it once again available remotely?

  3. If yes, then goto Monitor shows a blank, black screen, or shows error issue.

  4. If the system start after a reboot:

    1. Check the system history in Salesforce by clicking the serial number and viewing other cases listed on the system. Has this system had previous cases showing a lockup?

    2. Repeated lockups can be caused by environmental issues (excessive dust or heat, inadequate power) or software glitches. Ask about the installation:

      1. Does the VIP Appliance have adequate ventilation and protection from dust? If not, this is the most likely cause for the lockup. Refer them to the Installation and Configuration Guide for appropriate environmental specs.

      2. Is it plugged into a UPS? If not, they should verify that the system is plugged into an outlet that is capable of providing 400W of power.

    3. If the environment is OK, we need to take a look at logs.

      1. Login and open SystemManager.

      2. Copy log files. Depending on when the system locked up, there could be a gap in the logs. Select to view by file and choose the 10-20 most recent files available (the dates are in the file names).

      3. If the customer reports that they've been onsite before to reboot due to a lockup, but they didn't call support, try to get logs from around that lockup date as well.

      4. Save the logs to a PC and go to http://uploadlogs.3vr.com to send them to 3VR for analysis.

    4. The logs may show that the lockup is due to an underlying hardware issue:

      1. If this is already the second instance of a lockup within a short period of time (~1 month), issue an RMA.

      2. If there are no other lockup cases for this system, but the customer reports that they've frequently had to reboot due to lockups but didn't call support, talk to Sara or Gordy about whether to issue an RMA.

      3. We sometimes see lockups that happen after a periodic system maintenance reboot (aka internally as a "magic day lockup"). You'll see the system start to come back up after a magic day reboot but just stop at some point, until it is restarted. If you see this signature, be sure to update the bug: http://bugzilla.3vr.net/show_bug.cgi?id=18369

      4. Evidence of a HikVision failure may be provided by the following fatal error occurring for PipelineManager.exe

ForcedApplicationRestartException

 PipelineManager: MPEG-4 encoder card failure/unknown state

 FrameAcqSDK: Error setting frame structure for analog input port 0: c0000016

 No stack trace included

If the environment is OK, this is the first lockup reported, and the logs don't show a clear resolvable reason for the lockup, we will keep an eye on the system to make sure it doesn't happen a second time.