How to configure SAM to monitor containers or local instances ?

Question

Question

Sylvain Guilbaud · Feb 25, 2022

#System Alerting and Monitoring (SAM) #InterSystems IRIS

Monitoring IRIS through SAM (sam:1.0.0.115)

While I've used to make it work in the past, I'm not able any more to fix the already faced issue : "State: Unreachable"

Despite all the ip-addresses attempts (or hostname = host.docker.internal) and while "/api/monitor/metrics" is running well...

Could someone provide good practices to make it work cleanly ?

Product version: IRIS 2022.1

$ZV: IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2022.1 (Build 114U) Mon Jan 31 2022 01:58:06 EST

Discussion (13)1

Log in or sign up to continue

Sylvain Guilbaud Feb 28, 2022 to Sylvain Guilbaud

docker exec -ti -u root sam-prometheus-1 sh

/prometheus # wget http://172.20.10.3:50004/api/monitor/metrics -O iris-a-metrics

Connecting to 172.20.10.3:50004 (172.20.10.3:50004)

saving to 'iris-a-metrics'

iris-a-metrics       100% |*************************************************************************************************|  8584  0:00:00 ETA

'iris-a-metrics' saved

/prometheus # wget http://172.20.10.3:50014/api/monitor/metrics -O iris-b-metrics

Connecting to 172.20.10.3:50014 (172.20.10.3:50014)

saving to 'iris-b-metrics'

iris-b-metrics       100% |*************************************************************************************************|  7901  0:00:00 ETA

'iris-b-metrics' saved

/prometheus # wget http://172.20.10.3:52773/api/monitor/metrics -O iris-health-metrics

Connecting to 172.20.10.3:52773 (172.20.10.3:52773)

saving to 'iris-health-metrics'

iris-health-metrics  100% |*************************************************************************************************| 24567  0:00:00 ETA

'iris-health-metrics' saved

/prometheus # wget http://172.20.10.3:52774/api/monitor/metrics -O iris-metrics

Connecting to 172.20.10.3:52774 (172.20.10.3:52774)

saving to 'iris-metrics'

iris-metrics         100% |*************************************************************************************************|  6110  0:00:00 ETA

'iris-metrics' saved

0 0

score 0 · Answer 1 · 2022-02-25T08:11:49-05:00

It's not a big issue, to make it work. Just keep in mind that both containers, with SAM and monitored system, have to be in the same docker's network. If you run both images in with docker-compose, they by default will get the same network, so, in most cases, you may use IRIS service's name from docker-compose.yml as a hostname, and it will be able to connect to it internally inside that network.

If you need access from dockerized SAM to a dockerized IRIS, both working in different networks, you have to publish ports for IRIS, so, it should be available from the host, but from the docker, your SAM has to know the host's address and host.docker.internal is a way, to give it to him. And port should be the same as published when docker container is running.

When docker container is started with -p param, which says how to publish the port, possible to say, which IP to select, or select all for instance -p 0.0.0.0:52273:52273

docker ps, may help with understanding how the container is available

8d8977f9d5f8   intersystemsdc/iris-community:preview                          "/tini -- /iris-main…"   6 days ago       Up 3 days (healthy)       0.0.0.0:1972->1972/tcp, 2188/tcp, 53773/tcp, 0.0.0.0:52773->52773/tcp, 54773/tcp                           iris

if it shows something like 127.0.0.1:52773->52773, it will only be available as localhost or 127.0.0.1 and will not be accessible from another container and network, because of host.docker.internal will refer to another ip

Could you explain, how you running all the parts?

score 0 · Answer 2 · 2022-02-25T09:41:28-05:00

Thanks Dmitry for your reply.

Actually, I know all of this ; that's why I don't understand why it's not working any more...

gh repo clone intersystems-community/sam

```
cd sam
```
```
tar xvzf sam-1.0.0.115-unix.tar.gz
```
```
cd sam-1.0.0.115-unix
```
```
./start.sh
```

Then I create a cluster + a target on my local instance (non-container) :

iris list irishealth                

Configuration 'IRISHEALTH'

directory:    /Users/guilbaud/is/irishealth

versionid:    2021.2.0.649.0

datadir:      /Users/guilbaud/is/irishealth

conf file:    iris.cpf  (SuperServer port = 61773, WebServer = 52773)

status:       running, since Fri Feb 25 15:35:32 2022

state:        ok

product:      InterSystems IRISHealth

I check that /api/monitor/metrics runs well :

curl http://127.0.0.1:52773/api/monitor/metrics -o metrics

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 17634  100 17634    0     0  14174      0  0:00:01  0:00:01 --:--:-- 14383

score 0 · Answer 3 · 2022-02-25T09:50:48-05:00

For containers, I'm using docker-compose.

I've tried with SAM in the same yml file to get everything in the same network, but, nothing is working (0.0.0.0, etc.)

score 0 · Answer 4 · 2022-02-25T11:03:00-05:00

127.0.0.1 will not work, due to it's still different host, and in case of this sam bundle, it should point to sam's iris. And will not work it, because, it should be available for prometheus container, not just for IRIS container for sam. So, you have to use, it's name as a host iris

And to access, some IRIS external to docker, you may use host.docker.internal hostname, or docker.for.mac.localhost with Docker on macOS

So, the first line is points to SAM's IRIS, and second to an IRIS outside

I think host.docker.internal was added in some recent versions of Docker, so, probably check the version of Docker

score 0 · Answer 5 · 2022-02-25T11:37:19-05:00

I'm running the very last version of Docker : 4.5.0 (74594)

docker version

score 0 · Answer 6 · 2022-02-25T12:01:00-05:00

Sylvain Guilbaud Feb 25, 2022 to Dmitry Maslennikov

docker.for.mac.localhost : same result

0 0

score 0 · Answer 7 · 2022-02-25T12:08:50-05:00

Sylvain Guilbaud Feb 25, 2022 to Dmitry Maslennikov

Same result with host.docker.internal :

0 0

score 0 · Answer 8 · 2022-02-25T12:45:25-05:00

So, host.docker.internal is 192.168.65.2 and it's the same network as in settings, but this IP is used to get access from the virtual docker's environment

Could you try to use one of your real host's IP, which comes from Wi-Fi for instance

my wifi gives me IP 192.168.1.170, and it works

score 0 · Answer 9 · 2022-02-28T11:47:55-05:00

ping and wget works well from SAM containers (tested from prometheus or nginx containers) ; I'm able to reach each IRIS instances (containers and non-containers).

But from the SAM UI, I'm still facing unreachable errors.

score 0 · Answer 10 · 2022-02-28T11:47:17-05:00

Dmitry Maslennikov Feb 28, 2022 to Sylvain Guilbaud

Maybe prometheus logs, have something that may help

0 0

score 0 · Answer 11 · 2022-02-28T11:56:55-05:00

docker logs sam-prometheus-1

level=info ts=2022-02-28T16:14:51.209Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:335 host_details="(Linux 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021 x86_64 aa300025820c (none))"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"

level=info ts=2022-02-28T16:14:51.215Z caller=web.go:514 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2022-02-28T16:14:51.214Z caller=main.go:667 msg="Starting TSDB ..."

level=info ts=2022-02-28T16:14:51.225Z caller=head.go:575 component=tsdb msg="replaying WAL, this may take awhile"

level=info ts=2022-02-28T16:14:51.227Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0

level=info ts=2022-02-28T16:14:51.227Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=1.4381ms

level=info ts=2022-02-28T16:14:51.228Z caller=main.go:683 fs_type=EXT4_SUPER_MAGIC

level=info ts=2022-02-28T16:14:51.228Z caller=main.go:684 msg="TSDB started"

level=info ts=2022-02-28T16:14:51.228Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

ts=2022-02-28T16:14:51.234Z caller=dedupe.go:112 component=remote level=info remote_name=43bf89 url=http://iris:52773/api/sam/private/db/write msg="starting WAL watcher" queue=43bf89

ts=2022-02-28T16:14:51.234Z caller=dedupe.go:112 component=remote level=info remote_name=43bf89 url=http://iris:52773/api/sam/private/db/write msg="replaying WAL" queue=43bf89

level=info ts=2022-02-28T16:14:51.239Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:14:51.239Z caller=main.go:635 msg="Server is ready to receive web requests."

ts=2022-02-28T16:14:59.846Z caller=dedupe.go:112 component=remote level=info remote_name=43bf89 url=http://iris:52773/api/sam/private/db/write msg="done replaying WAL" duration=8.6116179s

level=info ts=2022-02-28T16:36:46.583Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:36:46.592Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:14.966Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:14.974Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:37.074Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:37.081Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:53.001Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:53.007Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

score 0 · Answer 12 · 2022-02-25T11:35:01-05:00

After a reboot I'm now again able to reach 1 local instance (out of 2) and 0 containers + the IRIS-SAM instance.

docker-compose.yml

docker ps

docker ps

CONTAINER ID   IMAGE                                                                COMMAND                  CREATED              STATUS                                 PORTS                                                                               NAMES

5ee1a03673cb   nginx:1.17.9-alpine                                                  "nginx -g 'daemon of…"   About a minute ago   Up About a minute                      80/tcp, 0.0.0.0:8080->8080/tcp                                                      sam-nginx

cce6e64a4fd6   grafana/grafana:6.7.1                                                "/run.sh"                About a minute ago   Up About a minute                      3000/tcp                                                                            sam-grafana

bbadfa5c326e   prom/alertmanager:v0.20.0                                            "/bin/alertmanager -…"   About a minute ago   Up About a minute                      9093/tcp                                                                            sam-alertmanager

89df4b965a3b   containers.intersystems.com/intersystems/webgateway:2022.1.0.131.0   "/startWebGateway"       About a minute ago   Up About a minute (healthy)            0.0.0.0:50200->80/tcp, 0.0.0.0:50243->443/tcp                                       webgateway

260b51880b60   prom/prometheus:v2.17.1                                              "/bin/prometheus --w…"   About a minute ago   Up About a minute                      9090/tcp                                                                            sam-prometheus

961b84b9ff4a   iris:2022.1.0.114.0                                                  "/iris-main"             About a minute ago   Up About a minute (health: starting)   2188/tcp, 53773/tcp, 54773/tcp, 0.0.0.0:50015->1972/tcp, 0.0.0.0:50014->52773/tcp   iris-b

b0c23098794a   postgres:13.4-alpine3.14                                             "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)            0.0.0.0:50006->5432/tcp                                                             postgres

4cb33c8c36e3   store/intersystems/sam:1.0.0.115                                     "/iris-main"             About a minute ago   Up About a minute (healthy)            2188/tcp, 51773/tcp, 52773/tcp, 53773/tcp, 54773/tcp                                sam-iris

54e117a4e856   iris:2022.1.0.114.0                                                  "/iris-main"             About a minute ago   Up About a minute (healthy)            2188/tcp, 53773/tcp, 54773/tcp, 0.0.0.0:50005->1972/tcp, 0.0.0.0:50004->52773/tcp   iris-a

2c5133d95693   containers.intersystems.com/intersystems/arbiter:2022.1.0.131.0      "/arbiterEntryPoint.…"   About a minute ago   Up About a minute (healthy)            0.0.0.0:50100->2188/tcp                                                             arbiter

307e59e12fdc   mcr.microsoft.com/mssql/server:2019-latest                           "/opt/mssql/bin/perm…"   About a minute ago   Up About a minute                      0.0.0.0:50007->1433/tcp                                                             mssql

isc_prometheus.yml

alerting:

alertmanagers:

- static_configs:

- targets:

- alertmanager:9093

global:

evaluation_interval: 15s

scrape_interval: 15s

remote_read:

- url: http://iris:52773/api/sam/private/db/read

remote_write:

- url: http://iris:52773/api/sam/private/db/write

rule_files:

- ./isc_alert_rules.yml

scrape_configs:

- job_name: SAM

metrics_path: /api/monitor/metrics

scheme: http

static_configs:

- labels:

cluster: "2"

targets:

- 192.168.1.100:52774

- 192.168.1.100:52773

- labels:

cluster: "3"

targets:

- 192.168.1.100:50004

- 192.168.1.100:50014

- 0.0.0.0:50004

- 0.0.0.0:50014

- labels:

cluster: "4"

targets:

- 192.168.1.100:8080