Nemo Documentation

Appearance

Introduction ​

This document outlines the procedures for backing up and restoring a Nemo deployment. Regular backups protect against data loss, corruption, and other unforeseen issues, while a well-defined restore process ensures that you can quickly recover and resume operations in the event of a failure.

By following the procedures outlined in this document, you can safeguard your data, minimize downtime, and ensure that your Nemo application remains a robust and dependable component of your voice network.

Overview ​

The Nemo application is located under /opt/nemo/ with configuration files in /opt/nemo/etc/. It relies on MongoDB database, whose data files are located under /data/db/. For central servers, it is mandatory to backup both the database and the application.

Backup Procedure ​

  1. Ensure the database is running and check with systemctl status mongod:
[root@nemo-server ~]# systemctl status mongod
* mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-04-18 15:43:14 CEST; 2 months 23 days ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 1249 (mongod)
   CGroup: /system.slice/mongod.service
           `-1249 /usr/bin/mongod -f /etc/mongod.conf

  1. Stop the Nemo application with systemctl stop nemo.

  2. Ensure that the application is not running anymore with systemctl status nemo:

[root@nemo-server ~]# systemctl status nemo
* nemo.service - SYSV: Nemo GUI
   Loaded: loaded (/etc/rc.d/init.d/nemo; bad; vendor preset: disabled)
   Active: inactive (dead) since Thu 2024-07-11 15:07:25 CEST; 54s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 21026 ExecStop=/etc/rc.d/init.d/nemo stop (code=exited, status=0/SUCCESS)
  Process: 20486 ExecStart=/etc/rc.d/init.d/nemo start (code=exited, status=0/SUCCESS)
  1. Navigate to a directory with enough space to hold the database dump. On standard Nemo deployments, the directory/filesystem /data/backup is set up to perform these backups. Create a brand new directory and navigate to it:
[root@nemo-server ~]# cd /data/backup/
[root@nemo-server backup]# mkdir production_backup_20240711
[root@nemo-server backup]# cd production_backup_20240711
[root@nemo-server production_backup_20240711]#
  1. Dump the database contents with the command mongodump. This will create a new directory dump under the current directory:
[root@nemo-server production_backup_20240711]# mongodump
2024-07-11T15:16:21.687+0200	writing admin.system.version to
2024-07-11T15:16:21.689+0200	done dumping admin.system.version (1 document)
2024-07-11T15:16:21.690+0200	writing calls.sip to
2024-07-11T15:16:21.690+0200	writing calls.rtp to
2024-07-11T15:16:21.690+0200	writing nemo.counters to
2024-07-11T15:16:21.690+0200	writing nemo.licenses_status to
2024-07-11T15:16:21.732+0200	done dumping nemo.licenses_status (2036 documents)
2024-07-11T15:16:21.732+0200	writing stats.arrays_bwg to
2024-07-11T15:16:22.864+0200	done dumping stats.arrays_bwg (452 documents)
...

[root@nemo-server production_backup_20240711]# ls
dump
  1. Copy over the Nemo application to the current directory:
[root@nemo-server production_backup_20240711]# cp -a /opt/nemo .
[root@nemo-server production_backup_20240711]# ls
dump  nemo
  1. Create a compressed archive file with tar zcvf backup_filename.tar.gz *:
[root@nemo-server production_backup_20240711]# tar zcvf production_backup_20240711.tar.gz *
dump/
dump/nemo/
dump/nemo/netnetsmxrcs_sbcs.metadata.json
dump/nemo/settings.metadata.json
dump/nemo/sonus_gws.metadata.json
...

Once the backup file is copied to a remote storage, it can be removed from the backup directory, along with the directory dump and nemo previously created.

Restore Procedure ​

  1. Navigate to a directory with enough space to uncompress the backup. On standard Nemo deployments, the directory/filesystem /data/backup is set up to perform these operations. Create a brand new directory and navigate to it:
[root@nemo-server ~]# cd /data/backup/
[root@nemo-server backup]# mkdir production_restore_20240711
[root@nemo-server backup]# cd production_restore_20240711
[root@nemo-server production_restore_20240711]#
  1. Uncompress the backup with tar xvf backup_filename.tar.gz:
[root@nemo-server production_restore_20240711]# tar zxvf production_backup_20240711.tar.gz
dump/
dump/nemo/
dump/nemo/netnetsmxrcs_sbcs.metadata.json
...
[root@nemo-server production_restore_20240711]# ls
dump  nemo  production_backup_20240711.tar.gz

  1. Stop the Nemo application with systemctl stop nemo.

  2. Ensure that the application is not running anymore with systemctl status nemo:

[root@nemo-server ~]# systemctl status nemo
* nemo.service - SYSV: Nemo GUI
   Loaded: loaded (/etc/rc.d/init.d/nemo; bad; vendor preset: disabled)
   Active: inactive (dead) since Thu 2024-07-11 15:07:25 CEST; 54s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 21026 ExecStop=/etc/rc.d/init.d/nemo stop (code=exited, status=0/SUCCESS)
  Process: 20486 ExecStart=/etc/rc.d/init.d/nemo start (code=exited, status=0/SUCCESS)

  1. Stop the MongoDB database with systemctl stop mongod.

  2. Ensure that the application is not running anymore with systemctl status mongod:

[root@nemo-server ~]# systemctl status mongod
* mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2024-07-11 15:34:39 CEST; 4s ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 1249 (code=exited, status=0/SUCCESS)

  1. Remove all MongoDB data files with rm -rf /data/db/*.

  2. Restart the MongoDB database with systemctl start mongod.

  3. Restore the database with the command mongorestore dump/:

[root@nemo-server production_restore_20240711]# mongorestore dump/
2024-07-11T15:42:37.292+0200	preparing collections to restore from
2024-07-11T15:42:37.298+0200	reading metadata for nemo.counters from dump/nemo/counters.metadata.json
2024-07-11T15:42:37.299+0200	restoring nemo.counters from dump/nemo/counters.bson
2024-07-11T15:42:37.302+0200	reading metadata for stats.arrays_bwg from dump/stats/arrays_bwg.metadata.json
2024-07-11T15:42:37.302+0200	restoring stats.arrays_bwg from dump/stats/arrays_bwg.bson
2024-07-11T15:42:37.312+0200	reading metadata for calls.sip from dump/calls/sip.metadata.json
2024-07-11T15:42:37.312+0200	restoring calls.sip from dump/calls/sip.bson
...

  1. Copy over the Nemo application backup to /opt/ with cp -a nemo /opt/:

  2. Restart the Nemo application with sysytemctl start nemo.

  3. Ensure the application is running with systemctl status nemo:

[root@nemo-server production_restore_20240711]# systemctl status nemo
* nemo.service - SYSV: Nemo GUI
   Loaded: loaded (/etc/rc.d/init.d/nemo; bad; vendor preset: disabled)
   Active: active (running) since Thu 2024-07-11 16:07:46 CEST; 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 3233 ExecStop=/etc/rc.d/init.d/nemo stop (code=killed, signal=KILL)
  Process: 3333 ExecStart=/etc/rc.d/init.d/nemo start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nemo.service
           |-3342 /opt/nemo/bin/python /opt/nemo/bin/nemo-watchdog
           |-3343 /opt/nemo/bin/python /opt/nemo/bin/nemo-queue-runner 0 2
           |-3345 /opt/nemo/bin/python /opt/nemo/bin/nemo-queue-runner 1 2
           |-3346 /opt/nemo/bin/python /opt/nemo/bin/nemo-broadworks-cdr-xml-collector
           |-3347 /opt/nemo/bin/python /opt/nemo/bin/nemo-gui
           |-3348 /opt/nemo/bin/python /opt/nemo/bin/nemo-rest
           |-3350 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 0 4 broadsoft
           |-3355 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 1 4 broadsoft
           |-3357 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 2 4 broadsoft
           |-3360 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 3 4 broadsoft
           |-3361 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 0 4 capture
           |-3362 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 1 4 capture
           |-3364 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 2 4 capture
           |-3367 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 3 4 capture
           |-3369 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 0 4 netnetsd
           |-3371 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 1 4 netnetsd
           |-3374 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 2 4 netnetsd
           |-3376 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-engine updateStatistics 3 4 netnetsd
           |-3377 /opt/nemo/bin/python /opt/nemo/bin/nemo-reporting-engine
           |-3379 /opt/nemo/bin/python /opt/nemo/bin/nemo-anomalies-engine
           |-3382 /opt/nemo/bin/python /opt/nemo/bin/nemo-stats-export-engine
           |-3384 /opt/nemo/bin/python /opt/nemo/bin/nemo-cdr-export-engine
           |-3387 /opt/nemo/bin/python /opt/nemo/bin/nemo-health-monitor
           |-3390 /opt/nemo/bin/nemo-capture-orchestrator
           `-3397 /opt/nemo/bin/python /opt/nemo/bin/nemo-ipfix-collector

Jul 11 16:07:46 nemo-server systemd[1]: Starting SYSV: Nemo GUI...
Jul 11 16:07:46 nemo-server nemo[3333]: Starting nemo watchdog: [  OK  ]
Jul 11 16:07:46 nemo-server systemd[1]: Started SYSV: Nemo GUI.
Warning: nemo.service changed on disk. Run 'systemctl daemon-reload' to reload units.