Quick step-by-step tutorial on a NixOS Linux VM#

In this tutorial, we will see how to setup SSH Monitor in order to monitor a virtual machine (VM).
Here, the VM will both be the target machine to monitor, and the machine hosting SSH Monitor.

We will do so, using the SSH Monitor training VM on NixOS Linux, in order to make this tutorial even nicer.

We will also do so using Nix in order to make this tutorial a lot quicker.

Important

In this turorial, a lot of corners have been cut in order to “ease” the overall process.

In particular, in this tutorial:

The host is also monitoring the target (i.e. “self-monitoring”).
The root user is passwordless and is used for monitoring purpose, which is obviously not recommended for production/real-worl use!

This allows to ignore some steps:

Those steps are ignored only to make this tutorial as quick and easy as possible.
They are otherwise always recommended and should not be skipped for production/real-world use!

0. Prerequisites#

Ideally, install the latest version of VirtualBox. At least v7+.
Download the SSH Monitor training VM on NixOS (the SSH-Monitor-training-VM-on-NixOS.ova file)
and import it with VirtualBox (File → Import appliance, or Ctrl+I).
You don’t need to modify the VM settings, the .ova file comes with the VM pre-configured.
Start the VM.

Troubleshooting

If VirtualBox doesn’t launch the VM and throws the VERR_NAT_REDIR_SETUP error,
then this is probably because a program is using TCP and/or UDP port 5064,
or because a program is using TCP and/or UDP port 5065.

You can check what program is using those ports, e.g. on Linux with: $ sudo netstat -tunlp.

Sometimes, CSS or Phoebus can still be using those ports even after being closed.
In this case, do not hesitate to kill their associated Java process(es).

0.1. SSH access to the VM, from Linux#

On the Linux computer used to run VirtualBox, please follow the below instructions:

Check that you can access gitlab (hosting SSH Monitor) using SSH:
```
$ ssh -T git@gitlab.com
Welcome to GitLab, @you-are-such-a-nice-person!
```
- If you don’t have access, then please follow the Gitlab documentation about how to use SSH keys to communicate with GitLab.
Check that your GitLab SSH key is included in your SSH agent:
```
$ ssh-add -l
```
Check that the ForwardAgent option is not set to no, in $HOME/.ssh/config. E.g.:
```
Host localhost
  ForwardAgent yes
  ...
```
Check that the ForwardAgent option is not set to no, in /etc/ssh/sshd_config.
Connect to the VM via SSH like so:
```
$ ssh -A -p 2222 root@localhost
```
On the VM launched by VirtualBox,
check that you can still access gitlab using SSH:
```
VM $ ssh -T git@gitlab.com
Welcome to GitLab, @you-are-such-a-nice-person!
```
- If you don’t have access, then you somehow probably have a ForwardAgent issue.
  Please refeer to the GitLab documentation about how to troubleshoot SSH.

On the VM launched by VirtualBox,
try to SSH to yourself in order to add yourself to the ~/.ssh/known_hosts file
(this is important because otherwise, errors will appears when running your SSH Monitor IOC program):

VM $ ssh root@localhost

  The authenticity of host localhost (::1) can t be established.
  ED25519 key fingerprint is SHA256:<...>
  This key is not known by any other names.
  Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
  Warning: Permanently added localhost (ED25519) to the list of known hosts.

🎉 You are ready to start the tutorial!

0.2. SSH access to the VM, from Windows#

On the Windows computer used to run VirtualBox, please follow the below instructions:

Open a terminal (CMD or PowerShell) and check that OpenSSH is installed, e.g. like so:
```
$ ssh -V
```
OpenSSH should be installed by default on Windows
(since Windows 10 build 1809 and Windows server 2019).
See the OpenSSH for Windows overview for more details.

If OpenSSH is not installed, then you will have to install it yourself.
Follow the official microsoft prerequisites (and prerequisites check).

Make sure the ssh-agent service is enabled. E.g. run in a PowerShell terminal:

# By default the ssh-agent service is disabled. Configure it to start automatically.
# Make sure you're running as an Administrator.
Get-Service ssh-agent | Set-Service -StartupType Automatic

# Start the service
Start-Service ssh-agent

# This should return a status of Running
Get-Service ssh-agent

# Now load your key file (the one associated to your GitLab account) into ssh-agent
ssh-add $env:USERPROFILE\.ssh\<your-ssh-key-file>

Troubleshooting

After running the ssh-add command,
if you are confronted to the No such file or directory error message,
then you will have to specify the full absolute path to your SSH key file by hand.
E.g. like so:

  ssh-add C:\Users\username\.ssh\id_rsa

Troubleshooting

After running the ssh-add command,
if you are confronted to an error message similar to the one below:

  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  Permissions for 'C:\\Users\\username\\.ssh\\id_rsa' are too open.
  It is required that your private key files are NOT accessible by others.
  This private key will be ignored.

Then you should change the permissions of the SSH key file.
Follow this link for help about how to do it.

For more details,
see the official microsoft documentation about how to start the ssh-agent service
(at the end of the “User key generation” section).

Check that you can access gitlab (hosting SSH Monitor) using SSH:
```
$ ssh -T git@gitlab.com
Welcome to GitLab, @you-are-such-a-nice-person!
```
- If you don’t have access,
  then please follow the Gitlab documentation about how to use SSH keys to communicate with GitLab.
Check that your GitLab SSH key is included in your SSH agent:
```
$ ssh-add -l
```
Check that the ForwardAgent option is not set to no,
in the config file located in the .ssh directory of your home. E.g.:
```
Host localhost
  ForwardAgent yes
  ...
```
Connect to the VM via SSH like so:
```
$ ssh -A -p 2222 root@localhost
```
On the VM launched by VirtualBox,
check that you can still access gitlab using SSH:
```
VM $ ssh -T git@gitlab.com
Welcome to GitLab, @you-are-such-a-nice-person!
```
- If you don’t have access, then you somehow probably have a ForwardAgent issue.
  Please refeer to the GitLab documentation about how to troubleshoot SSH.

VM $ ssh root@localhost

  The authenticity of host localhost (::1) can t be established.
  ED25519 key fingerprint is SHA256:<...>
  This key is not known by any other names.
  Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
  Warning: Permanently added localhost (ED25519) to the list of known hosts.

🎉 You are ready to start the tutorial!

Important

From now on, the rest of the turorial happens on the VM!
All the following commands should be run inside the VM, not on the computer hosting it.

1. Create your Top#

See also

This section is inspired from the EPNix documentation.
Feel free to check this documentation for more details.

Create the directory of your future Top:

$ mkdir -p /tops/mySelfMonitorTop
$ cd /tops/mySelfMonitorTop

Create the Top at that location:

$ nix flake new -t 'git+ssh://git@github.com/epics-extensions/EPNix.git' .

Refresh the EPNix development environment:
```
$ nix develop --command echo "OK"
```
Warning

This command can take a long time to complete (around 10mn),
because https://nix-cache.extra.cea.fr is very slow at the moment.

Modify your ./flake.nix file like so:

  {
-   description = "EPICS IOC for <...>";
+   description = "EPICS Top, using SSH Monitor in order to monitor the local machine hosting its associated IOC program.";
+
  ...
        epnix = {
          inherit inputs;

          # Change this to be the name of your EPICS Top
          # ---
-         meta.name = "my-top";
+         meta.name = "mySelfMonitorTop";
  ...

Initialize Git in your Top, and start tracking your files:

$ git init --initial-branch=main # initial branch name set to `main` in order to be compliant with GitHub and GitLab defaults
$ git add -N .
$ nix develop --command echo OK

Create your App:

$ nix develop --command makeBaseApp.pl -t ioc selfMonitor
$ nix develop --command makeBaseApp.pl -a linux-x86_64 -i -t ioc -p selfMonitor SelfMonitor

Track the new files with Git:
```
$ git add -N .
```
Update your EPICS ./configure/RELEASE.local through EPNix:
```
$ nix develop --command eregen
```
Make sure that your ./configure folder is all good:
```
$ nix develop --command check-config
```
At this point, your Top should build correctly:
```
$ nix build -L
```

2. Import SSH Monitor#

See also

This section is inspired from the EPNix documentation,
and from the install, update, remove how-to guide.
Feel free to check those documentations for more details.

Add sshmonitor to your ./flake.nix:

  {
    description = "EPICS Top, using SSH Monitor in order to monitor the local machine hosting its associated IOC program.";

    inputs.flake-utils.url = "github:numtide/flake-utils";
-   inputs.epnix.url = "github:epics-extensions/epnix/nixos-xx.xx";
+   inputs.epnix.follows = "sshmonitor/epnix";

    # If you have a support module as a separate EPNix repository,
    # add it as an input here:
    # ---
    #inputs.mySupportModule = {
    #  url = "git+ssh://git@my-server.org/me/exampleApp.git";
    #  inputs.epnix.follows = "epnix";
    #};

    # If you have an "App" as a separate repository,
    # add it as an input here:
    # ---
    #inputs.exampleApp = {
    #  url = "git+ssh://git@my-server.org/me/exampleApp.git";
    #  flake = false;
    #};

+   inputs.sshmonitor = {
+     url = "git+ssh://git@gitlab.com/sshmonitor/sshmonitor.git";
+   };
+
    outputs = {
      self,
      flake-utils,
      epnix,
      ...
    } @ inputs: let
-     myEpnixConfig = {pkgs, ...}: {
+     myEpnixConfig = {pkgs, lib, ...}: {
        # Set your EPNix options here
        # ---

        # If you have a support module as a separate EPNix repository,
        # uncomment this line to make the package available:
        # ---
        #nixpkgs.overlays = [inputs.mySupportModule.overlays.default];

+       nixpkgs.overlays = [
+ 
+         # Let `sshmonitor` be available in `pkgs.epnix.support`
+         inputs.sshmonitor.overlays.default
+ 
+         # Select the same EPICS base than sshmonitor:
+         (_final: prev: {
+           epnix = prev.epnix.extend (_final: prev: {
+             epics-base = builtins.elemAt (builtins.elemAt (lib.filter (x: (builtins.typeOf x) == "list") inputs.sshmonitor.packages.x86_64-linux.build.buildInputs) 0) 0;
+           });
+         })
+ 
+       ];
+
        epnix = {
          inherit inputs;

          # Change this to be the name of your EPICS top
          # ---
          meta.name = "mySelfMonitorTop";

          # You can choose the version of EPICS-base here:
          # ---
          #epics-base.releaseBranch = "3"; # Defaults to "7"

          # Add one of the supported modules here:
          # ---
          #support.modules = with pkgs.epnix.support; [ StreamDevice ];
+         support.modules = with pkgs.epnix.support; [ sshmonitor ];
  ...

Update your EPICS ./configure/RELEASE.local through EPNix:

$ nix develop --command eregen

Make sure that your ./configure folder is all good:

$ nix develop --command check-config

Refresh the EPNix development environment:

$ nix develop --command echo OK

Import SSH Monitor’s .template and .substitutions in ./selfMonitorApp/Db/Makefile:

  TOP=../..
  include $(TOP)/configure/CONFIG
  #----------------------------------------
  #  ADD MACRO DEFINITIONS AFTER THIS LINE
  
  #----------------------------------------------------
  # Create and install (or just install) into <top>/db
  # databases, templates, substitutions like this
  #DB += xxx.db
+
+ DB_INSTALLS += $(EPNIX_SSHMONITOR)/db/ssh_monitor_core.template
+ DB_INSTALLS += $(wildcard $(EPNIX_SSHMONITOR)/db/*.substitutions)
  
  #----------------------------------------------------
  # If <anyname>.db template is not named <anyname>*.template add
  # <anyname>_template = <templatename>
  
  include $(TOP)/configure/RULES
  #----------------------------------------
  #  ADD RULES AFTER THIS LINE

Import SSH Monitor’s .dbd and library in ./selfMonitorApp/src/Makefile:

  TOP=../..
  
  include $(TOP)/configure/CONFIG
  #----------------------------------------
  #  ADD MACRO DEFINITIONS AFTER THIS LINE
  #=============================
  
  #=============================
  # Build the IOC application
  
  PROD_IOC = selfMonitor
  # selfMonitor.dbd will be created and installed
  DBD += selfMonitor.dbd
  
  # selfMonitor.dbd will be made up from these files:
  selfMonitor_DBD += base.dbd
+ selfMonitor_DBD += system.dbd
  
  # Include dbd files from all support applications:
  #selfMonitor_DBD += xxx.dbd
+ selfMonitor_DBD += sshmonitorSupport.dbd
  
  # Add all the support libraries needed by this IOC
  #selfMonitor_LIBS += xxx
+ selfMonitor_LIBS += sshmonitorSupport
  
  # selfMonitor_registerRecordDeviceDriver.cpp derives from selfMonitor.dbd
  selfMonitor_SRCS += selfMonitor_registerRecordDeviceDriver.cpp
  
  # Build the main IOC entry point on workstation OSs.
  selfMonitor_SRCS_DEFAULT += selfMonitorMain.cpp
  selfMonitor_SRCS_vxWorks += -nil-
  
  # Add support from base/src/vxWorks if needed
  #selfMonitor_OBJS_vxWorks += $(EPICS_BASE_BIN)/vxComLibrary
  
  # Finally link to the EPICS Base libraries
  selfMonitor_LIBS += $(EPICS_BASE_IOC_LIBS)
  
  #===========================
  
  include $(TOP)/configure/RULES
  #----------------------------------------
  #  ADD RULES AFTER THIS LINE

Make sure that your Top still builds correctly:

$ nix build -L

3. Configure your `.cmd` and `.substitutions`#

See also

This section is inspired from the .cmd and .substitutions how-to guide.
Feel free to check this documentation for more details.

Edit ./iocBoot/iocSelfMonitor/st.cmd, in order for this file to have to following content:

#!../../bin/linux-x86_64/selfMonitor

< envPaths

epicsEnvSet("IOCSH_PS1", "Self Monitor> ")
epicsEnvSet("EPICS_CAS_SERVER_PORT", "5064")

epicsEnvSet("PREFIX", "tests:")
epicsEnvSet("SSH_OPTS_AND_ARGS", "ssh root@localhost -o BatchMode=yes -o PasswordAuthentication=no -o ConnectionAttempts=3 -x -o ControlMaster=auto -o ControlPath=~/.ssh/%C -o ControlPersist=60s")

dbLoadDatabase("${EPNIX_SSHMONITOR}/dbd/menuScan.dbd")
dbLoadDatabase("${TOP}/dbd/selfMonitor.dbd")

selfMonitor_registerRecordDeviceDriver(pdbbase)

var(dbRecordsOnceOnly, 1)

dbLoadTemplate("${TOP}/db/processors_monitoring.substitutions", "PREFIX_MACRO=${PREFIX}, SSH_OPTS_AND_ARGS_MACRO=${SSH_OPTS_AND_ARGS}, SCAN_MACRO=1 minute")
dbLoadTemplate("${TOP}/db/memory_monitoring.substitutions", "PREFIX_MACRO=${PREFIX}, SSH_OPTS_AND_ARGS_MACRO=${SSH_OPTS_AND_ARGS}, SCAN_MACRO=1 minute")

iocInit()

For more details about this .cmd configuration, see the .cmd and .substitutions how-to guide.

Note that for the moment, the only loaded .substitutions files are
processors_monitoring.substitutions and memory_monitoring.substitutions,
for processors and memory monitoring.

Don’t forget to make your .cmd executable:

$ chmod +x ./iocBoot/iocSelfMonitor/st.cmd

Important

In a few words, how does it work internally?

Let’s look at the processors_monitoring.substitutions file.
When we take a closer look e.g. to the PV_NAME_SCALAR_1 macro
and the SCALAR_1_INSTRUCTION macro,
we can see that the SSH shell instruction used to retrieve the number of CPUs is:

lscpu | grep '^CPU(s):' | awk '{print $2}'

This instruction is passed to the SSH Monitor core .template file: ssh_monitor_core.template,
where it is processed by an aSubRecord.

Note that the SSH options and arguments, sourroundig the SSH shell instruction,
are specified with the SSH_OPTS_AND_ARGS macro.
The combination of the SSH options and arguments and the SSH shell instruction
gives us the full SSH command.

Every SSH command is defined like so.

4. Run your IOC program#

Build your Top:
```
$ nix build -L
```

Run your IOC program:

$ cd ./result/iocBoot/iocSelfMonitor
$ ./st.cmd

Troubleshooting

When running your IOC program, if you encounter errors messages similar to the ones below:

Host key verification failed.
SSH Monitor ERROR: the full SSH command including the shell instruction (ssh root@localhost -o BatchMode=yes -o PasswordAuthentication=no -o ConnectionAttempts=3 -x -o ControlMaster=auto -o ControlPath=~/.ssh/%C -o ControlPersist=60s "free -m | grep 'Swap:' | awk '{print \$2}'" >| /root/.cache/sshmonitor/root_tests:mem:ASubCore) returns an error when executed!

Host key verification failed.
SSH Monitor ERROR: the full SSH command including the shell instruction (ssh root@localhost -o BatchMode=yes -o PasswordAuthentication=no -o ConnectionAttempts=3 -x -o ControlMaster=auto -o ControlPath=~/.ssh/%C -o ControlPersist=60s "free -m | grep 'Swap:' | awk '{print \$3}'" >| /root/.cache/sshmonitor/root_tests:mem:ASubCore) returns an error when executed!

Host key verification failed.
SSH Monitor ERROR: the full SSH command including the shell instruction (ssh root@localhost -o BatchMode=yes -o PasswordAuthentication=no -o ConnectionAttempts=3 -x -o ControlMaster=auto -o ControlPath=~/.ssh/%C -o ControlPersist=60s "free -m | grep 'Swap:' | awk '{print \$4}'" >| /root/.cache/sshmonitor/root_tests:mem:ASubCore) returns an error when executed!

then you should SSH to yourself in order to add yourself to the known hosts:

$ ssh root@localhost

The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:<...>
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.

Another possible explanation is that the SSH_OPTS_AND_ARGS macro – specified in the .cmd file – contains e.g. a syntax error.

Troubleshooting

When running your IOC program, if you encounter the following message:

muxclient: master hello exchange failed

then you can ignore it.

This might be due to the fact that your are forwarding your ssh-agent to the VM with more than one SSH key.

See the first note of section 5.1 of the following link:
20241007123652/https://docs.releng.linuxfoundation.org/_/downloads/common-packer/en/latest/pdf/

See also

For more troubleshooting, see the troubleshooting guide.

5. Check the GUI#

On the computer used to launch VirtualBox (not on the VM),
please retrieve SSH Monitor default GUIs: gui.
- E.g. on Linux:
```
$ cd /tmp
$ git archive --remote git@gitlab.com:sshmonitor/sshmonitor.git master | tar -xvf - gui
```
Run Phoebus or CSS in order to display the GUI.
- If using Phoebus, check if its associated settings.ini file
  includes either 127.0.0.1 or localhost in the org.phoebus.pv.ca/addr_list variable.
- If using CSS, check if the “Address List” property, located in
  Edit → Preference → CSS Core → Data Sources → Channel Access → Address List,
  includes either 127.0.0.1 or localhost.
Execute main_display.bob (if using Phoebus) or run main_display.opi (if using Phoebus).
Note that for the moment, the only loaded .substitutions files are
processors_monitoring.substitutions and memory_monitoring.substitutions,
for processors and memory monitoring.
So only the “Memory monitoring” and “Processors” views will find connected PVs.

6. Success#

🎉 Congratulations! Your are now monitoring your own VM through SSH Monitor!

7. Exercise: Monitor things more securely#

Like said in the introduction: in this turorial, a lot of corners have been cut in order to “ease” the overall process.

It’s time to do things the right way, by following the below how-to guides:

8. Exercise: Monitor your own things#

In order to specify your own shell instruction’s and retrieve the exact data you want, the way you want,
follow the below how-to guide:

Custom shell instructions how-to

∞. Monitor even more things#

A lot more can be done with SSH Monitor,
do not hesite to explore other how-to guides and explanations to find out!

Quick step-by-step tutorial on a NixOS Linux VM

Contents

Quick step-by-step tutorial on a NixOS Linux VM#

0. Prerequisites#

0.1. SSH access to the VM, from Linux#

0.2. SSH access to the VM, from Windows#

1. Create your Top#

2. Import SSH Monitor#

3. Configure your .cmd and .substitutions#

4. Run your IOC program#

5. Check the GUI#

6. Success#

7. Exercise: Monitor things more securely#

8. Exercise: Monitor your own things#

∞. Monitor even more things#

3. Configure your `.cmd` and `.substitutions`#