Server Side Troubleshooting

Owned by Pauline Mbabu
Last updated: Apr 08, 2020 by Martin Ndegwa
3 min read

 

Note: These steps apply to troubleshooting an opensrp server.

For this Document we shall  be using a boresha-aya staging server to show examples.

Troubleshooting different issues.

1. Server Unavailability (Error code 4xx)

Case 1: Client is not able to access the server 

  1. You can use the following ways to check if the service is available.

    1. Navigate to the server URL/IP 

      1. Use a locally installed browser to navigate to the server URL

        https://boresha-afya-stage.smartregister.org/opensrp/ or 63.32.56.42

    2. Ping the URL/IP address i.e

      ping 63.32.56.42

  2. Depending on the results of step 1, if the server returns a response that denotes that the service is available you can attempt any of the following steps

    1. Connect to the server, you can use an ssh client for this:

      ssh user@serverip.com

    2. Check that the actual service in the server is  running.

      1. Use systemctl to check that OpenSRP , OpenMRSand/ nginx services are running. You can use these commands 

      2. Use systemctl to start or restart any of the services that are down

    3. If the services are running , confirm that all that port 8080 is allowing all incoming traffic to the server

d. You can then read the logs for the services if all the services are running, to access server side logs use either of the 2 commands

i. For tomcat logs , these are OpenSRP/OpenMRS logs, navigate apps logs, you can use the command below

 

ii.Confirm that the services are running and the server is okay. If the server is down, you can infer the reason from the log files.Some common causes for service shutting down are:

1. JVM out of memory

a. You can use htop to have a UI representation of memory use

2. Disk out of space

a. Check disk space via

i. If disk is out of space

1. You can delete server logs.(Do this for both OpenSRP and OpenMRS)

a. cd/home/openrsp/tomcat-opensrp/logs. 

d. Delete logs

2. Increase physical disk space 

3. Tomcat has shut down unexpectedly

4. Overload on number of errors reported by the  JVM

3. If service is unavailable: This would be denoted  by a 404 error response by the browser or ping not returned by the server. Confirm that you have an active internet connection to any of the other well known services, a common solution for this would be to navigate to www.google,com and confirm your device is still connected to the internet. If you can access the internet but not your server,

a. You can check if a connection to the specified server is available using an ssh client. If this can be achieved then attempt to login and try the steps listed above for service available(step 2)

b. In the event you can neither receive a response from the server nor connect via a ssh client, you might need to check that the server is online in this case

i. Check that the server is running. Log in to your AWS/Google cloud/Azure/Ali Cloud/Digital Ocean console and confirm that the server is online

 

More

Determining which server version of code is deployed.
ssh into the server and run the command
less /usr/lib/collectd/plugins/monitored-apps-list.sh

Related articles